Ansible NTP-GPS

Here’s a fun project to create an NTP server for systems where there is no Internet connectivity. I recently set up a weather station on a farm and there is no connectivity to the “World Wide Web”.

I use two Raspberry Pis; one runs Node-Red and does all the coordinating functions and the other runs Graphite/Grafana for pretty charts of the data being collected from various sensors.

In that environment, when the power goes out, the Raspis lose their minds and can’t find a time signal. The result is some unknown time which seems to be random. In any case, it’s wrong and this solution works great.

This build uses a Raspberry Pi as an NTP Server and an Adafruit Ultimate GPS Breakout V3 board which communicates with the Raspi through serial pins of the GPIO pins.

I am learning Ansible so I designed the build as an Ansible Role and an associated playbook. The Role has been posted on Github and also available as in Ansible Galaxy.

At the bottom of the readme on github you can lookup some references that I used to learn how to do this.

Have fun.

Installing OpenWrt on a Western Digital N600

OpenWrt is a Linux operating system targeting embedded devices like WiFi routers.  It is more stable, offers more features, is more secure and, because it has a large community base, has better support than almost any of the commercial routers. It’s design philosophy is to provide the ability to use devices in ways not originally envisioned.

Years ago, I started using OpenWrt because, prior to the arrival of the micro-computers like the Raspberry Pi, I could load python into an Asus WiFi router and create my own xbee gateway. Memory on those routers was very limited but by mounting a thumb drive into the USB port I could load python into that partition and run it from there. I learned these tricks from Jeff Keyser (http://mightyohm.com) and LadyAda (http://AdaFruit.com) and their work with the Asus WL-520gU and the Tweet-a-Watt.

WD My Net N600

These days I use OpenWrt as my standard OS for any new WiFi routers I install. In fact I usually check the devices which are supported by OpenWrt before buying a new one. My recent favorite is the Western Digital My Net N600/N750. I’ve been able to pick them up for as little as $20. These are the basic steps I use to flash OpenWrt on a the N600.

You can find very detailed approaches to flash OpenWrt on the N600 here but the following steps seem to work best. The general idea is to first flash an intermediate firmware named Gargoyle, then install the OpenWrt software using Gargoyle. Here we go…

n600-reset-button
Special paperclip tool inserted into the reset button hole.
  • Find the location of the reset button hole on the bottom of the N600 and push a bent paperclip into the hole. There is a small micro-switch under the hole which one can feel with the paperclip. Hold the button down with the paperclip for about 15 to 30 seconds.
  • In your browser, access http://192.168.1.1 and you should see a primitive dialog to upload firmware.
  • Select the gargoyle file from the Downloads directory, and click to upload it. It will take a few minutes so let it run it’s course.
  • When the N600 reboots it will come back with new firmware, Gargoyle. Log in with user: root, password: password.
  • Find the System menu and then the Firmware update option.
  • Select the OpenWrt file from the Downloads directory, and click to upload it. It will take a few minutes but when it reboots it will come up in OpenWrt mode.
  • Enter a new root password and proceed to the Overview menu option.

From here you’re on your own to set up the wifi router as you need. The N600 has 2 radios, 2.4 GHz and 5.0 GHz. You can set up your basic dual band secured access point or allow SSH for access to the command line so you can use it in ways not originally envisioned by the manufacturer.

In my home automation setup, I use an N600 as a separate access point which receives connections from all my wifi based devices. Mostly I use the Wemos D1 Mini breakout for an ESP8266 chip. I also run 2 Raspberry PIs:

  1. the central controller server running Node-Red
  2. the data server running Graphite/Grafana for pretty trend charts

Have fun.

From Pachube to Cosm to Xively – the death of DIY?

diy

Xively, formerly Cosm, and in the beginning, named Pachube, has gone a little wacky.  They have a new article on their site , The Dangers of DIY, declaring “the era of DIY has passed“.    To be fair their angle is about helping organizations productize and monetize their IoT products but please, let’s not be so cynical.

DIY is here to stay.

I used Pachube when I was just learning about Arduinos and how to hack wifi routers with OpenWRT, running python scripts to collect sensor data and send it to “the cloud”.  I was in contact with a brilliant fellow named Usman Haque, founder of Pachube, and had the opportunity to help with a simple javascript issue.  I was skeptical when they renamed the service to Cosm, and then finally left them when LogMeIn took over and renamed the service to Xively.  I was generously grand-fathered in to my free account, but the uncertainty left me less than enthusiastic about their service.

Now, in my home automation system, I’m running all open source software.  I keep all my data on my own Raspberry Pis in a round robin database (Whisper). I use Node-Red as the central coordinating component and I use Graphite/Grafana as the charting and data analysis component.

My last DIY integration was adding a chatbot system using a Raspberry Pi, Rocket.chat, and Hubot to manually control lights and to be informed of any changes in the state of the system.  The next phase will be to move into voice control.  Fun awaits.

Yes, the reports of the death of DIY are greatly exaggerated…

Home Automation Chatbot with Rocket.chat & Hubot

Getting ready for next weekend’s 2016 Maker Faire.

Again, as a non-commercial hobbyist, I have been accepted for a booth at this year’s event where I will show various aspects of a DIY home automation system. There are sensors and switches and blinking lights that turn on and off.  Very automagical.

I’ve presented my system at the Maker Faire for the last 5 years and each year I show some new aspect I’ve added. I started with a system that just collected data. Then I added a coordinating piece, Node-RED, that provides a signal routing function (they call it a component for the wiring of the IoT – the Internet of Things).

Selection_008.png

Node-RED flow: mqtt sensor subscription routing to graphite charting

This year my new piece is a bot that runs in a chat system. In the software development world, the concept of using chat to communicate with one’s workmates has taken off. The new twist is the use of a bot (a software robot). The bot is typically programmed to assist the developers and to notify them of any ongoing issues.

My addition is to include a stand-alone home chat system called Rocket.chat that provides a way to “converse” (i.e., chat) with the home automation system. The “bot” software is from the folks at Github called Hubot.  The attached image shows the beginnings of how that might look and at the end, the light does in fact turn on. Woo hoo!

Selection_007.png

A typical chat with Iris, my home automation chatbot.

I have named my chatbot, Iris, who was, in Greek mythology, the messenger of the gods.

The Rocket.chat software, as is, would be useful for family members but the chatbot addition makes it easy to keep up with what’s going on in the home.

The conversation is started with the call to Iris

Iris list rooms – and Iris responds with a list of the rooms found in the home automation repository.

Iris turn on light in kitchen – and Iris responds by looking up the specifics about how the light in the kitchen is physically turned on.  It does so by connecting to the Node-RED coordinating software which in turn makes the appropriate calls.

Of course, there are commercial versions of this with voice recognition like Apple’s Siri, or Amazon’s Alexa (using their Echo device) but here I have a system, fully open source, running without the need of any cloud connection…

Perhaps, next year, after I get the text-based system working smoothly, I might add voice recognition. But one step at a time.

If you’re in the area and you have some time, come on out to the Maker Faire. It’s an inspirational event; a gathering of people who bring their collective use of tech and art for all to share. You might be overwhelmed but you won’t be disappointed.

http://makerfaire.com – My booth name is The Sense of Things

Home Automation System – Slide Show

Last week I did a presentation of my Home Automation System to a group of folks ManyLabs.org – Here’s the link...

About ManyLabs.org:   an open science skunkworks. We make and support open tools for science and education, and provide space for a network of passionate inventors to realize their ideas.

Puppeting a Home Automation Gateway

The Puppet Master Host

The Puppet Master Host

I’ve been playing with Puppet, an open-source configuration management utility.  It is designed to provide a process for installing software onto a remote host.  One can set up a “Puppet Master” (I think the concept comes from a classic anime film titled Ghost In the Shell)  Then from a remote host a Puppet agent component connects to the Puppet Master and installs configured packages.

My goal is to develop a puppet system where I can build up, from a stock version of Debian or Raspian, a fully loaded gateway within a few minutes.  Usually it can take an hour or more to set up a new gateway.

There’s much more work to be done on this project but I thought I’d put what I have so far on github .  Basically it is set up to install the following software:

  • Apache 2.4.7
  • MySQL 5.5.44
  • PHP 5.5.9
  • node 0.12.7
  • npm 2.11.3
  • Node-RED
  • Graphite 0.9.13
  • Grafana 2.1.3
  • [mochad 0.1.16 – this is for X-10 module access]

Notes are in the README.

Setup for MakerFaire 2015

At the Tinaja Labs booth, called The Sense of Things, I had a full IoT system (Internet of Things) in place. Front to back I started with wireless sensors, to a Raspberry Pi that collected sensor data, to a Beaglebone Black running Node-Red (as a signal routing application), to newly introduced inexpensive wi-fi modules (the ESP8266), and finally to old-school X-10 light switches.

I’ve always described my setup as a home automation system that is built on a wireless sensor network.  I first heard about the concept of the Internet of Things (IoT) in 2009.  Home automation and wireless sensor networks are now included in the overarching concept of IoT and includes M2M (machine to machine) and industrial applications like manufacturing systems.  IoT describes all the things that can be connected through internet technologies with sensors on one end, actuators on the other end and coordination software in the middle.

Overview of the IoT system

Download the OS image here. Read instructions here and learn about how to burn an OS image to a MicroSD card using WinDiskImager32.  More info about setting up a MicroSD card here.

The system in place includes examples of typical components of an IoT system.  On one end I had wireless sensors (based on XBee Series 1 radios), in the middle I had connecting and coordination system software (Node-Red), and on the other end I had charting and analysis (Graphite/Grafana) and actuation (wi-fi based LED control).  Pretty much, that covers the basic areas of home automation.

Code for this setup is on Github at:  TinajaLabs/makerfaire2015

Sensor PCB with XBee

the MS-1 PCB

My MS-1; a Multi-Sensor PCB w XBee.

I designed this PCB a few years ago and it still works great.  I deployed these in every room of our home.  It has a 3.3V regulator, an XBee Radio, a temperature sensor (tmp36), a light sensor (CdS cell), and headers for 2 more sensors (mostly I’ve hooked up a hall effect sensor and an FSR, force sensitive sensor).  I also included a rail to rail buffer chip to create a clean separation between the sensors.  In hind-sight I probably didn’t need that.

The sensor board schematic and the PCB layout:

MS-1 PCB Layout

MS-1 Schematic


Using XCTU, a configuration tool (available for Windows, Mac, or Linux) from Digi, I configured the XBee Series 1 radio end points like this:

AP=2
CE=0
MY=
ID=1111 (same for all radios in your system)
CH=0C (same for all radios in your system)

Wi-fi Capacitive Touch Button

Download the OS image here. Read instructions here and learn about how to burn an OS image to a MicroSD card using WinDiskImager32.  More info about setting up a MicroSD card here.

This is a new sensor setup, breadboarded, that uses a capacitive touch switch (Standalone Toggle Capacitive Touch Sensor Breakout – AT42QT1012) and an ESP8266 wi-fi PCB.  The capacitive switch responds to touch and toggles a high/low voltage  on the GPIO01 pin of the ESP8266.  The ESP8266 is programmed to send the switch’s state over wi-fi to a web service on the Node-Red software.  More about that in the Node-Red section below.
django-tagging 0.3.1 or greater

 CapSwitch-esp8266_bb

Here’s the Fritzing breadboard of the capacitive switch hooked up to an ESP8266. Find the code I used code on Github.

The Gateway “Stack”

The picture below shows what I call my gateway rack (constructed from a CD caddy and CD sized pieces of colored Plexiglas from Tap Plastics).  All of the functionality could probably be included on any one of these devices, but for the sake of experimentation and separation of functions, I set up 3 micro-computers:

  1. Raspberry Pi B+ (on red Plexiglass):  Has a GPIO board from Ciseco called the Slice of Pi that supports an xbee coordinator configured radio.  It is running  Raspbian OS and a python script that collects the serial data being collected by the XBee radio.  The metric data is collected, bundled up and formatted, and sent to a Node-Red TCP Socket module.  It also has an application named mochad-0.1.16 that provides X-10 device connectivity.
  2. Beaglebone Black (on blue Plexiglass): Running Debian Wheezy and installed with Node-Red (requires Node.js; should already be installed on Wheezy)
  3. Odroid C-1 (on orange Plexiglas): Running Ubuntu 14.04.2 LTS with Graphite and Grafana installed to provide a data store and data charting for all the sensor data.

20150802_135321

When loading up an OS on these gateway devices, the basic process is to load an image file onto a MicroSD card that stands in as the computer’s bootable hard disk.  I prefer using Win32 Disk Imager and I found ApplePi-Baker to be useful on Mac OS X.  There’s a great overview of different methods here:  http://elinux.org/RPi_Easy_SD_Card_Setup

Sensor Gateway

Download the OS image here. Read instructions here and learn about how to burn an OS image to a MicroSD card using WinDiskImager32.  More info about setting up a MicroSD card here.

The Raspberry Pi sensor gateway has a GPIO PCB from Ciseco, the Slice of Pi, that supports an xbee radio.  This XBee radio is configured to be the coordinator radio for all my XBee Sensor PCBs (described above).  There is a python script, sensorgate2.py, that reads the data being received from the XBee radio via a serial port.  Sample code on Github.

Using a software configuration tool from Digi (manufacturer of XBee radios) called X-CTU, and an XBee FTDI device (from AdaFruit, or Sparkfun – FTDI friend needed), you can set the configuration of the XBee modules.  Here are the critical settings for my setup:

AP=2
CE=1
MY=1234
ID=1111 (same for all radios in your system)
CH=0C (same for all radios in your system)

To start the python script whenever the Raspi boots up, you’ll need an init script.  Find a sample on Github.

Once you place this file in /etc/init.d, run this command:

# update-rc.d tinaja_sensor defaults

Once this is set up, the python script will start-up at boot-time, or you can manually start and stop the script when you make changes like this:

# service tinaja_sensor stop
# service tinaja_sensor start

Node-Red Gateway

This gateway was built using a BeagleBone Black, Revision B.  It is running the OS, BeagleBoard.org BeagleBone Debian.

Download the OS image here. Read instructions here and learn about how to burn an OS image to a MicroSD card using WinDiskImager32.  More info about setting up a MicroSD card here.

Download the OS image hereRead instructions here and learn about how to burn an OS image to a MicroSD card using WinDiskImager32.  More info about setting up a MicroSD card here.

Node-Red is a web based application with a drag-and-drop user interface that makes it possible to build data flows by dragging various components on to the design surface (http web services, tcp socket listeners, MQTT, functional blocks, format converters, etc.) and connecting them with rubber-band connectors that direct the “flow” of data between components. node-red-sample1

This gateway device is responsible for managing the connectivity and the logical flow between the sensors and actuators.  All of the sensors in my home direct their metric data to a web service or a simple TCP Socket which are listening within Node-Red.   From Node-Red, the data can be directed to various destinations such as a data store, a chart, an actuator such as a relay, a light switch, an alerting system, or even a controller for central heating and cooling.  These definitions are called flows and are store on disk in JSON format.

Node-Red is built on Node.js (based on Javascript) and the BeagleBone Debian Image comes with that pre-installed.  The best way to get Node-Red installed is to follow the instructions for BeagleBone Black on the Node-Red web site:  http://nodered.org/docs/hardware/beagleboneblack.html

If you’d like to use a Raspberry Pi instead of BeagleBone Black you can look at a specialized distro called TheThingBox – http://thethingbox.io/.  It is an OS image pre-loaded with Node-Red.

Node-Red Flows

A Node-Red flow is a definition of a collection of nodes modules which are strung together in Node-Red.  You can read about flows on the Node-RED site here.  and see a sampling of available nodes and flows on the Node-RED web site.  You can find some of my flows on Github.

Charting and Analytics Server

This device is an Odroid C1 running Ubuntu 14.04.  It’s basically a Raspberry Pi clone with a quad core CPU and 1G of RAM.  It can run from a MicroSD as the other micro-computers or, it can run significantly faster ( and significantly more expensively) on an eMMC memory card.  You can purchase these devices in the US from a company called Ameridroid.com.

I have utilized this server to host our data storage and to provide our charting and analysis services.  For that I use Graphite, a python based Django web application which includes a TCP socket listener, named Carbon, for all the data input and a round robin database (RRD) known as Whisper.  Graphite at its core is a capable charting application but it looks kind of dry.  To see cool looking charts with an easy to design web interface I use Grafana.

Graphite

graphite-chart

Graphite Chart Style

Grafana Chart Style

Graphite is a comprehensive application and it has a bunch of moving parts.  The basic process of setting this up is to install Graphite the various supporting components:

  • Carbon, the TCP Socket listener for all the data
  • Whisper – the round robin database
  • Graphite – the web application runs in Apache and has these prerequisites:

You can read the Graphite install instructions here.

Grafana

grafana-chart3

Grafana Chart Style

Next comes Grafana.  It is a fork of a component that comes with an application known as ELK (Elasticsearch, Logstash. Kibana).  I’ll let you figure out from which component it is derived.

When I originally installed Grafana it was with version 1.8 and it required the addition of Elasticsearch and Java.  Version 2.x does not have the Elasticsearch/Java requirement so I would recommend 2.x.  You can read about installing Grafana here.  The simple approach requires 3 commands:

# wget https://grafanarel.s3.amazonaws.com/builds/grafana_2.1.2_amd64.deb
# sudo apt-get install adduser libfontconfig
# sudo dpkg -i grafana_2.1.2.deb

Wifi Controlled Light (LED)

This setup uses an ESP8266 wi-fi device as a web server which is waiting for a URL web service request that toggles an LED on/off.  Of course the LED stands in for a relay or anything you’d like to switch on and off. You can find the code I used here.esp8266_LED_Controller

Here’s the Fritzing breadboard diagram.

What’s Next?

From here going forward, I’ve got some more projects to work on. One is to have all the system events logged and to have a voice system that announces the events; kitchen light on, garage door opened, and so on.  It’s a precursor  to a voice control system.  I’d like to set up a wall switch replacement that contains environmental sensors (temp, humidity, light, presence) and capacitor touch switches for a simple control surface.  I’d also like to manage conditioned air to each room in the house based on metrics received from the wall plate sensors in each room.  In other words it would look like a dozen very inexpensive thermostats distributed all around the house and controlling the air conditioning on a room to room basis.

If you get here and find this article meaningful but find something I could improve upon, please let me know.  Thanks.

Maker Faire 2015 is over

Another Maker Faire, my 5th, the 10th in the Bay Area.  It is done, it was fun, and I am exhausted.  Now the post Maker Faire ennui has set in so I will let it go for a few days and relax as my voice recuperates.

My booth, The Sense of Things, had an extra side show this year.  In addition to my home automation demo, I included a small table of LED strands triggered through Arduino Micros by various sensors.  I first built this set for the East Bay Mini-Maker Faire at the Park Day School in Oakland.  Because it’s a middle school, I tried to create an experience for middle-schoolers where LEDs respond directly to interactively programmed sensors.

The 5 sensors were:

FSR + Arduino-Micro + NeoPixel LEDs

FSR + Arduino-Micro + NeoPixel LEDs

It was a hit.  Children of all ages were drawn to the LEDs like moths to a flame and seemed delighted at what they experienced.  Occasionally I would intervene to challenge the participants to think of what they might do with the sensors.  The source code for the sensor demo is available on github.

I have two very generous friends, Terence Shek and Larry Quantz who stood-in when I needed a break and who would graciously engage the curious attendees.

TheSenseOfThings_ChrisJefferies

On Sunday I was presented with an Editors Choice Award from Jordan Bunker, technical editor at Maker Media, for my IoT (Internet of Things) demo. I never really thought about receiving an award at Maker Faire but I was surprised by it and surprised at how it made me feel like all my research, technical struggles, and late nights were appreciated.  Thanks to Jordan and Maker Media team members, and the other Makers that put their heart and soul into the Greatest Show and Tell on Earth.

LapPi – Raspberry Pi as laptop

Here’s a really beautiful assembly of a DIY laptop with Raspberry Pi at its core from Instructables.  It requires some woodworking skills as well as computer assembly.

http://www.instructables.com/id/LapPi-A-Raspberry-Pi-Netbook/