An IoT-connected PowerPoint multi-device show!

Today I was honoured to have the opportunity to discuss the challenges and opportunities IoT brings to the world of technical documentation in a keynote for the inaugural Cambridge meet-up of the Write The Docs community.

One of the topics I covered was how IoT enables human-computer interaction across a much broader range of devices than the traditional screen, keyboard and pointing device – the 4th generation user interface as Steve Wilson describes it – leading to richer, more natural, and more immersive experiences for users of applications (in the broadest sense).

To help illustrate this in the context of a presentation I decided to extend the slideshow beyond the projector and slide clicker and include lights and buttons to create a more entertaining experience. For example I talked about some of the opportunities devices such as Amazon Echo could create. As I reached the relevant slide a set of LEDs came on to illuminate an actual Echo device on a stand on the stage. I also had a large push-button illuminated with controllable multi-coloured LEDs; this became a software-defined push-button which did different things during the course of the presentation including advancing slides, turning off the LEDs, and so on. The LED colour and action assigned to the button were controlled based on the slide being projected.

As usual I used Octoblu as the IoT automation platform. A key input to the flow was a trigger for each time the PowerPoint presentation advanced a slide. To do this I created a PowerPoint macro named “OnSlideShowPageChange” (called in the obvious manner by the application) which sent a HTTP POST to an Octoblu trigger with the current slide number and the total number of slides from the presentation.

Sub OnSlideShowPageChange(ByVal SSW As SlideShowWindow)
    Set objHTTP = CreateObject("MSXML2.ServerXMLHTTP")
    URL = "https://triggers.octoblu.com/v2/flows/<UUID>/triggers/<UUID>"
    objHTTP.Open "POST", URL, False
    objHTTP.setRequestHeader "User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
    objHTTP.setRequestHeader "Content-Type", "application/json"
    objHTTP.send "{""currentSlide"":" & SSW.View.CurrentShowPosition & ",""numSlides"":" & SSW.Presentation.Slides.Count & "}"
End Sub
To drive PowerPoint I created a new Octoblu connector that runs on my laptop that can remote control a PowerPoint slideshow. This is a very simple connector that really just adds an Octoblu shim on top of the excellent slideshow NPM library. The connector can do a few things but the most useful is the “GotoSlide” endpoint which can take a slide number, or one of “next”, “prev”, “first” or “last”. A simple example of using it in a flow is:
pptnode
(BTW if you want to use my connectors see https://github.com/jamesbulpin/my-meshblu-connectors for how to add my custom connector repository to your own Octoblu account. All the custom connectors in this blog post can be found in that repo.)
The flow looks rather more complex that it really is:
pptflowannotated
There are essentially five main parts to the flow:
  1. The trigger from the PowerPoint macro (telling us which slide is being shown) setting the state of a string of WS2811 LEDs (using my WS2811 Raspberry Pi connector which is also in the repo mentioned above) based on which slide is being shown. These LEDs illuminate the Echo and a few other props. The JSON templates contain configuration to tell the WS2811 connector which LEDs to turn on to illuminate the relevant prop. When the final slide is shown the entire string is put into a moving pattern (just for fun 🙂 ).
  2. The slide number is also used to show a slide progress bar on a second string of WS2811 LEDs. The connector takes the current slide number and the total number of slides. With the LEDs in the string mapping to slides in the presentation in a one-to-one manner, it turns LEDS green for slides already shown, red for those not yet shown, and yellow for the currently showing slide.
  3. The slide number is used to define the colour of the illuminated push-button (or to turn it off entirely) and to send it a string which the button will return (see below) if pushed – this makes the function of the button dynamically configurable depending on which slide is being shown.
  4. If the push-button is pushed it sends the string configured in #3 above. This part of the flow dispatches messages depending on that string. If it’s “ON” all the LEDs on both strings are put into a moving pattern and “OFF” turns all LEDs off. Otherwise it’s interpreted as a slideshow command (e.g. “next”) and routed to the PowerPoint connector described earlier.
  5. An additional effect of the button push in the “ON” and “OFF” cases is setting the button to do the opposite, i.e. if the push was “ON” then the button is reconfigured for “OFF”.

An aside on the WS2811 connector: This works by specifying a mode (via a parameter on the message) and, depending on the mode, some additional parameters such as colour or slide number. Currently the available modes are:

  • “off” – what you’d expect
  • “solid” (or “color”) – display a solid colour on all LEDs – requires the “color” parameter (can be a word such as “red” or hash hex value such as “#00553e”)
  • “slide” – the slide progress bar described above – requires “slide” and “slidemax” parameters
  • “colorwheel” – a moving and colour-changing dynamic pattern
  • “twinkle” – colorwheel modulated with a random twinkling (each LED turning on and off randomly)
  • “percent” – show a percentage bargraph (“VU meter” style) on the LEDs – requires the “percent” parameter, turns on the first percent% of the LEDs
  • “direct” – takes a JSON object containing a description of which LEDs should be which colour: pass the object (e.g. using the JSON Template node and passing {{msg}} in the parameter) containing a key named “groups” which is a list of objects with “color” (string color name/hascode) and “leds” (a list of integer LED index with the first LED on the wire being zero) parameters.

e.g.:

{
  "groups":[
    {
      "color":"blue",
      "leds":[1,3,5,9]
    },
    {
      "color":"red",
      "leds":[0,10,132]
    }
  ]
}

(This WS2811 connector, developed in collaboration with John Moody, is a better version of the one I described in a previous post – see that post for info on the wiring.)

Now of course there are ways to do similar things to all of this without IoT but like many things with IoT it’s the removal of barriers of cost, accessibility and vendor compatibility that make an IoT approach interesting.

If you’re at Citrix Synergy 2017 in Orlando later this month be sure to join me and a growing list of IoT and automation experts for SYN401 to see what else is possible with the Octoblu platform. I can guarantee that the PowerPoint (there won’t be much – it’s a far more interactive session than that!) will be even more IoT than in today’s event!

Advertisements

Displaying messages on my Christmas tree lights

IoT and Christmas go well together – all those festive lights and sounds are just asking to be connected up to the Internet and automated! My own Christmas tree is of course no exception and, as I wrote about over on the Citrix blogs, now sports a string of individually-addressable LEDs controlled by a Raspberry Pi which is in turn managed by Octoblu.

But I decided to go a step further than simply displaying some fun flashing patterns and colours on the tree: seeing as every LED can be individually managed, if I could determine the exact position on the tree of each LED I could come up with geometric patterns or even use it as a sort of dot matrix display to show messages.

iotreecalibratewebcamMy solution was to connect a webcam to the Raspberry Pi and write a script that turned each LED on one by one, captured an image from the webcam, and found the coordinates of the bright spot on the image. After the script ran through all 100 LEDs it then normalised the coordinates to a 100 x 100 grid and output a list of coordinates that I could then use in my LED controller program. The code is a bit gross, being a mash-up of various bits of other scripts quickly bolted together, including a web server that I used to view the image while positioning the camera.

iotreecalibrateTo visually check the output I wrote a quick bit of HTML/JavaScript that used the pre-normalised coordinates to overlay blobs on top of a captured image from the webcam (with all LEDs lit) – I could then see if the blobs lined up with the LEDs. As you can see in the image here there is at least one misalignment caused by the reflection of the light on a nearby white surface.

So armed with a set of coordinates for the LEDs I then extended my existing WS2811 LED control program (see my post on the Citrix blog for more on this, and on the hardware setup) to use coordinates, instead of just LED sequence number, for its patterns.

Firstly I created a simple set of test patterns that moved a vertical line horizontally across the tree, a horizontal line vertically and then the two different diagonals. I also updated the VU meter effect to use Y coordinates instead of LED sequence number.

However the most fun piece was rendering text on the tree. To do this I found a suitable encoding of a dot matrix font online (there are loads out there) and morphed it into a suitable form to include in Node.js. I then wrote a very simple rendering function that runs over a string of text one character at a time, using the font configuration to determine which pixels in each column of the 5×7 matrix should be illuminated. The end result was an array with an entry per column (for the entire message) with the entry being an encoding of which row LEDs should be illuminated. I also created a similar array to record which colour to use for each column – in this case populated to cycle through red, green and blue on a character-by-character basis (this helps the eye to see the characters better).

var dmmessage = " XMAS";

var dmarray;
var dmcolor;
var colorcycle = 255;
function renderMessage() {
  dmarray = new Uint32Array(dmmessage.length * 7 + 5);
  dmcolor = new Uint32Array(dmmessage.length * 7 + 5);
  for (var i = 0; i < dmmessage.length; i++) {
    var fonttablebase = (dmmessage.charCodeAt(i) - 0x20) * 5;
    for (var j = 0; j < 5; j++) {
      dmarray[i*7+j] = font5x7[fonttablebase + j];
      dmcolor[i*7+j] = colorcycle;
    }
    if (colorcycle == 0xff0000) {
      colorcycle = 255;
    } else {
      colorcycle = colorcycle << 8;
    }
  }
}

In the main part of the WS2811 driver program, where a loop iterates over the 100 LEDs choosing which colour to display for each, the code uses the incrementing offset variable (increments once per 100-LED update cycle, every 80ms) to index into the column array, offset by the X coordinate of the LED. A quantised Y coordinate of the LED is used to look up the row pixel on/off data from the array entry – the quantisation effectively creating 7 rows across the surface of the tree.

function dotmatrix_y(y) {
  if (y < 25.0) return 8;
  if (y < 40.0) return 0;
  if (y < 50.0) return 1;
  if (y < 60.0) return 2;
  if (y < 70.0) return 3;
  if (y < 80.0) return 4;
  if (y < 90.0) return 5;
  return 6;
}

function dotmatrix(pos) {
  var x = xy.xy[pos][0];
  var ydot = dotmatrix_y(xy.xy[pos][1]);

  var idx = Math.floor(((offset + x)/10)%dmarray.length);
  var column = dmarray[idx];

  if ((1<<ydot)&column) {
    return dmcolor[idx];
  }
  return 0;
}

I added a new parameter to the Octoblu message to allow the display string to be set. I updated the Octoblu flow and Alexa skill to allow a command such as “Alexa, tell the Christmas tree to say Hello”.

Alexa intent schema (new part in bold, full code here):

{
  "intents": [
    {
      "intent": "ModeIntent",
      "slots": [
        {
          "name": "mode",
          "type": "LIST_OF_MODES"
        }
      ]
    },
    {
      "intent": "ColorIntent",
      "slots": [
        {
          "name": "color",
          "type": "LIST_OF_COLORS"
        }
      ]
    },
    {
      "intent": "TextIntent",
      "slots": [
        {
          "name": "text",
          "type": "LIST_OF_WORDS"
        }
      ]
    },
    {
      "intent": "AskOctobluIntent"
    },
    {
      "intent": "SorryIntent"
    }
  ]
}

AWS Lambda intent handler (new part in bold, full code here):

def on_intent(intent_request, session):
    """ Called when the user specifies an intent for this skill """

    print("on_intent requestId=" + intent_request['requestId'] +
          ", sessionId=" + session['sessionId'])

    intent = intent_request['intent']
    intent_name = intent_request['intent']['name']
    
    mode = None
    color = None
    text = None

    session_attributes = {}
    speech_output = None

    # Dispatch to your skill's intent handlers
    if intent_name == "ModeIntent":
        if ('mode' in intent['slots']) and ('value' in intent['slots']['mode']):
            mode = intent['slots']['mode']['value']
    elif intent_name == "ColorIntent":
        if ('color' in intent['slots']) and ('value' in intent['slots']['color']):
            color = intent['slots']['color']['value']
            mode = "solid"
    elif intent_name == "TextIntent":
        if ('text' in intent['slots']) and ('value' in intent['slots']['text']):
            text = intent['slots']['text']['value']
            mode = "dotmatrix"
    elif intent_name == "AskOctobluIntent":
        mode = "octoblu"
    elif intent_name == "SorryIntent":
        speech_output = "I'm sorry James, I can't do that"
    else:
        raise ValueError("Invalid intent")

    obmsg = {"debug":intent}
    if mode:
        obmsg["mode"] = mode
    if color:
        obmsg["color"] = color
    if text:
        obmsg["text"] = text
    url = octoblu_trigger
    data = urllib.urlencode(obmsg)
    req = urllib2.Request(url, data)
    response = urllib2.urlopen(req)
    the_page = response.read()
    
    if not speech_output:
        speech_output = "OK"
    return build_response(session_attributes, build_speechlet_response(speech_output))

WS2811 Octoblu message handler (new part in bold):

conn.on('ready', function(rdata){
  console.log('UUID AUTHENTICATED!');
  console.log(rdata);

  clearTimeout(connectionTimer);
  connectionTimer = undefined;

  conn.update({
    "uuid": meshbluJSON.uuid,
    "token": meshbluJSON.token,
    "messageSchema": MESSAGE_SCHEMA,
  });

  conn.on('message', function(data){
    console.log('Octoblu message received');
    console.log(data);
    mode = data.mode;
    color = ("color" in data)?data.color:null;
    if (("text" in data) && data.text) {
      dmmessage = " " + data.text.toUpperCase();
      renderMessage();
      offset = 0;
    }
    handle_message(mode, color)
  });

});

Octoblu connects together the HTTPS POST request from the Alexa skill handler with the WS2811 driver program.

iotreeoctoblu

And there we have it – Alexa-controlled scrolling message display on a Christmas tree thanks to Raspberry Pi and Octoblu!

 

Automating HDMI input switching

I recently wrote about automating TV power control using IR LEDs. This enabled me to turn a TV and amplifier on and off with Amazon Echo and as part of other automation flows. However, with a cable TV box, Kodi media player, PlayStation 3 and Google Cast I still needed to choose the TV’s input source manually. I’d like to be able to say “Alexa, turn on cable”, or “Alexa, turn on Google Cast” and have it not only turn on the TV and amp as it can now, but also switch to the correct input.

One option would be to take advantage of automatic HDMI switching driven from a source coming online – but with things like the Cast and Kodi player being always-on this made this a no-go.

Another option was to use an external HDMI switcher box with an IR remote control and blast IR commands at this in the same way as for the TV itself. In fact this was my original plan and I carefully selected a HDMI switcher that was not only getting decent reviews, but also had a remote control button per channel (simply having a “next input” cycle button was not suitable because I’d have no way to know when I got to the input I wanted). However when I received the switcher I thought I’d try something different, just for fun.

Hardware

This particular switcher has an LED for each input, showing which is selected, and a single push button to cycle through the inputs. Having opened up the box by prising off the plastic feet and unscrewing the 4 small screws underneath I took a good look at the circuit board. I observed that the LEDs were wired with the cathodes connected to ground, therefore the anodes would be at a positive voltage when the LED was lit – this is good enough to directly drive a Raspberry Pi GPIO input pin in pull-up mode. By connecting all 5 LED anodes to GPIO pins I could easily tell which input was selected. The push button was also connected to ground, suggesting it was being used in a pull-up fashion. I took a gamble that I could get away with driving this directly from a Raspberry Pi GPIO output without any further interfacing. To make life even easier all of the LED and switch pins were through-hole soldered into the switcher’s PCB (i.e. not surface mount like most of the board) meaning small wires could be fairly easily soldered onto them.

hdmiwiring

Even more conveniently the arrangement of the box, although very compact, had plenty of room to bring in a 9 core cable (5 LEDs, 1 switch, 1 ground and two unused wires) leading to a neat appearance. To route the cable out I drilled a 5mm hole close to the IR socket on the edge of the box and routed the wires over the top side of the PCB.

On the Raspberry Pi end I simply chose 6 unused GPIO pins and connected the LEDs and switch input as well as connecting ground on the HDMI box to a ground pin on the GPIO header.

Software

I wanted to use the same Raspberry Pi running LibreELEC and Kodi that I was already using to drive the IR emitters. This meant extending the existing Kodi add-on with the capability to read the HDMI switcher’s LED status and drive its push button. The general strategy was to have an incoming MQTT message set the desired input number (1-5) and then send button push pulses until the associated LED lit up. One potential gotcha here is that this switcher skips over inputs that do not have an active HDMI source, therefore if the MQTT message requests an input with either nothing connected, or with a source that’s not switched on, the button pushing could go on forever. To avoid this I limited the number of button push attempts per MQTT message to 10.

The prerequisite was to add the “Script – Raspberry Pi Tools” Kodi add-on (search the registry from the Kodi add-on manager UI) to add the GPIO Python library.

The code is pretty simple. At start of day I set up GPIO – the first 5 pins are the inputs from the switcher’s LEDs and the final pin is the output to the push button:

sys.path.append('/storage/.kodi/addons/virtual.rpi-tools/lib')
import RPi.GPIO as GPIO

# Setup pins for HDMI switcher control
GPIO.setmode(GPIO.BCM)
GPIO.setup(21, GPIO.IN, pull_up_down = GPIO.PUD_UP) 
GPIO.setup(8, GPIO.IN, pull_up_down = GPIO.PUD_UP) 
GPIO.setup(16, GPIO.IN, pull_up_down = GPIO.PUD_UP) 
GPIO.setup(12, GPIO.IN, pull_up_down = GPIO.PUD_UP) 
GPIO.setup(7, GPIO.IN, pull_up_down = GPIO.PUD_UP)
GPIO.setup(20, GPIO.OUT)
GPIO.output(20, 1)

And initialise some global variables to record the desired input number and the count of button push attempts:

hdmiinput = 1
hdmiattempts = -1

In the MQTT message handler I added another sub-topic case:

            elif ll[2] == "HDMI":
                try:
                    hdmiinput = int(msg.payload)
                    hdmiattempts = 0
                except:
                    pass

And in the main loop (which cycles every 0.5s):

        if (hdmiattempts > -1) and (hdmiattempts < 10):
            for i in range(4):
                hdminow = 0
                if GPIO.input(21): hdminow = 1
                if GPIO.input(8): hdminow = 2
                if GPIO.input(16): hdminow = 3
                if GPIO.input(12): hdminow = 4
                if GPIO.input(7): hdminow = 5
                log("HDMI switcher currently on %s, want %s" % (hdminow, hdmiinput))
                if hdminow == hdmiinput:
                    hdmiattempts = -1
                    break
                else:
                    log("Sending HDMI switch button trigger")
                    GPIO.output(20, 0)
                    time.sleep(0.1)
                    GPIO.output(20, 1)
                    time.sleep(0.1)
                    hdmiattempts = hdmiattempts + 1

The full Kodi add-on code can be found here. It’s quick and dirty, but it works.

Automation with Echo

So now I have a way to change HDMI input to the TV by sending a MQTT message such as “KODI/Lounge/HDMI=4“. To use this with Amazon Echo I extended my existing TV control with specific cases for each of the 4 inputs in use. The configuration for the Alexa smart home skill adapter (see Controlling custom lighting with Amazon Echo and a skill adapter for more on this) sends a single MQTT command into my home broker:

                {
                    "applianceId":"loungetvcable",
                    "manufacturerName":"James Bulpin",
                    "modelName":"LWRF",
                    "version":"v0.1",
                    "friendlyName":"Virgin Media",
                    "friendlyDescription":"Living room TV and amp on cable TV",
                    "isReachable":True,
                    "actions":[
                        "turnOn",
                        "turnOff"
                    ],
                    "additionalApplianceDetails":{
                        "mqttTopics":"compound/loungetvcable"
                    }
                },
                {
                    "applianceId":"loungetvcast",
                    "manufacturerName":"James Bulpin",
                    "modelName":"LWRF",
                    "version":"v0.1",
                    "friendlyName":"Chrome Cast",
                    "friendlyDescription":"Living room TV and amp on Google Cast",
                    "isReachable":True,
                    "actions":[
                        "turnOn",
                        "turnOff"
                    ],
                    "additionalApplianceDetails":{
                        "mqttTopics":"compound/loungetvcast"
                    }
                },
                {
                    "applianceId":"loungetvps3",
                    "manufacturerName":"James Bulpin",
                    "modelName":"LWRF",
                    "version":"v0.1",
                    "friendlyName":"PS3",
                    "friendlyDescription":"Living room TV and amp on PS3",
                    "isReachable":True,
                    "actions":[
                        "turnOn",
                        "turnOff"
                    ],
                    "additionalApplianceDetails":{
                        "mqttTopics":"compound/loungetvps3"
                    }
                },
                {
                    "applianceId":"loungetvkodi",
                    "manufacturerName":"James Bulpin",
                    "modelName":"LWRF",
                    "version":"v0.1",
                    "friendlyName":"Kodi",
                    "friendlyDescription":"Living room TV and amp on Kodi",
                    "isReachable":True,
                    "actions":[
                        "turnOn",
                        "turnOff"
                    ],
                    "additionalApplianceDetails":{
                        "mqttTopics":"compound/loungetvkodi"
                    }
                },

(See this gist for the context – these rules are just added to the list)

Rules in the rules engine then turn this into TV power control and in the “on” case, also sends the HDMI control message:

  case 'compound/loungetvcable':
    resources.mqtt.publish("KODI/Lounge/TV", message.toString());
    if (message.toString() == "on") {
      resources.mqtt.publish("KODI/Lounge/HDMI", "1");
    }
    break;
  case 'compound/loungetvcast':
    resources.mqtt.publish("KODI/Lounge/TV", message.toString());
    if (message.toString() == "on") {
      resources.mqtt.publish("KODI/Lounge/HDMI", "2");
    }
    break;
  case 'compound/loungetvps3':
    resources.mqtt.publish("KODI/Lounge/TV", message.toString());
    if (message.toString() == "on") {
      resources.mqtt.publish("KODI/Lounge/HDMI", "5");
    }
    break;
  case 'compound/loungetvkodi':
    resources.mqtt.publish("KODI/Lounge/TV", message.toString());
    if (message.toString() == "on") {
      resources.mqtt.publish("KODI/Lounge/HDMI", "4");
    }
    break;

In closing

By adding explicit input selection to the set of MQTT commands I can send I can now build more complex automation actions that not only turn things on or off but can also select inputs. This enables a single Echo command such as “Alexa, turn on Chrome Cast” to get all the necessary devices into the right state to achieve the desired outcome of the TV and amp being on and displaying the output of the Cast.

hdmisidebar

 

Using infrared control from Amazon Echo

A while back I built a skill adapter to allow Amazon Echo’s smart home skill system to manage my LightwaveRF lights via my custom Octoblu+MQTT system. But why stop there? If I can say “Alexa, turn on the living room light” why can’t I also say “Alexa, turn on the TV“? With IoT mains power sockets such as Belkin Wemo or LightwaveRF I could power on and off the TV but I prefer to leave the TV in standby (yes, I know this uses power and kills the planet). Instead I decided to solve the problem by using infrared emitters glued to the front of the devices I wanted to control and allowing Alexa to control these.

Hardware

ircircuit2Each TV in my house is co-located with a Raspberry Pi running the Kodi media player. Therefore it made sense to add the IR emitter capability to these rather than add more hardware. There are various options for emitting IR including USB IR blasters and professional systems design for commercial AV use-cases. I chose a homebrew route and built my own emitters using an IR LED, a NPN transistor and a couple of resistors. The whole thing connects to a Raspberry Pi GPIO pin. Using a transistor allows for higher LED currents than can be used directly from the GPIO pin and allows multiple emitters to be connected to the same GPIO pin, e.g. to connect several home entertainment devices.

irhead2

I wired the four components together in free space (no board) and connected to a three core cable. I used offcuts of cable sheath and heatshrink tubing to insulate and encapsulate everything. The end result is an IR LED poking out the top of black blob on the end of a black cable. It looks a bit ugly close-up but once attached to a black TV case it’s hardly noticeable.

I connected the other end of the cable to the GPIO header on the Raspberry Pi using +5V, GND and GPIO17 (this is the default for the LIRC Pi GPIO drive) pins. I then positioned and superglued the emitter onto the front of the TV such that it could see the TV’s IR receiver but without completely blocking it.

irtv

Kodi interface

The Raspberry Pi runs the LibreELEC distribution of Kodi. To set this up to use GPIO-driven IR required me to SSH to the Pi as root and edit /flash/config.txt to add a line “dtoverlay=lirc-rpi” – this is needed to load the kernel driver for Raspberry Pi GPIO for the Linux IR subsystem. I then needed to download the specific IR configurations I needed for my devices. For example for my Sony Bravia TV I ran:

 wget -O "/storage/.config/lircd.conf" \
  'https://sourceforge.net/p/lirc-remotes/code/ci/master/tree/remotes/sony/RM-ED009.lircd.conf?format=raw'

For controlling multiple devices multiple remote control configs can be concatenated into the lircd.conf file.

After a reboot it was time to code a Kodi add-on. This add-on runs as a service and connects to my home MQTT broker in order to receive published messages (more of which later). The MQTT message handler spawns the “irsend” command (comes with the LibreELEC distribution) to send the named IR command(s). For example, the following code is used to control the TV and an amplifier at the same time (using two IR emitters wired to the same GPIO pin):

# The callback for when a PUBLISH message is received from the server.
def on_message(client, userdata, msg):
    global player, hdmiattempts, hdmiinput

    try:
        msgpayload = str(msg.payload)
        print(msg.topic+" "+msgpayload)
        ll = msg.topic.split("/")
        if len(ll) > 2:
            if ll[2] == "TV":
                log("Incoming TV command: " + msgpayload)
                ircmds = []
                if msgpayload.lower() == "off":
                    ircmds.append(["Sony_RM-ED009-12", "KEY_POWER", 5])
                    ircmds.append(["Marantz_RMC-73", "Amp_Standby", 1])
                elif msgpayload.lower() == "on":
                    ircmds.append(["Sony_RM-ED009-12", "KEY_POWER", 5])
                    ircmds.append(["Marantz_RMC-73", "Amp_Standby", 1])
                else:
                    ll = msgpayload.split("/")
                    ircmds.append([ll[0],ll[1],1])
                if len(ircmds) > 0:
                    log("Sending IR commands: " + str(ircmds))
                    for ircmd in ircmds:
                        cmd = "/usr/bin/irsend --count %u -d /run/lirc/lircd-lirc0 SEND_ONCE %s %s" % (ircmd[2], ircmd[0], ircmd[1])
                        os.system(cmd)
                        time.sleep(1)
    except Exception, e:
        log("MQTT handler exception: " + str(e))

I developed the add-on in-place on the Raspberry Pi (as opposed to building a distributable package and importing it through Kodi’s UI). To do this I firstly created a directory:

# mkdir /storage/.kodi/addons/service.mqtt.jamesbulpin

Within this I created two files:

And a subdirectory “resources” containing:

Overall the structure was this:

LoungeTV:~/.kodi/addons/service.mqtt.jamesbulpin # find . -type f
./resources/__init__.py
./resources/settings.xml
./resources/language/english/strings.xml
./resources/paho/__init__.py
./resources/paho/mqtt/__init__.py
./resources/paho/mqtt/client.py
./resources/paho/mqtt/subscribe.py
./resources/paho/mqtt/publish.py
./mqttservice.py
./addon.xml

The full code can be found at https://bitbucket.org/jbulpin/kodihomeautomation – note that this includes hard-coded IR devices and commands so would need customisation before use.

kodiconfigThe easiest way to get Kodi to discover the add-on is to reboot. It may also be necessary to enable the add-on via the “My add-ons” menu in Kodi. The settings dialog can also be accessed from here – this allows setting of the MQTT broker IP address and of the name that will be used in the MQTT topic the add-on subscribes to – if this is set to “Lounge” then the topic prefix is “KODI/Lounge“.

With the add-on running I tested it by manually sending MQTT messages:

# mosquitto_pub -m on -t KODI/Lounge/TV

Control with Amazon Echo

So far I’ve described how to get from a MQTT message through to an infrared control action on a TV. The next step was to have the MQTT message be sent in response to an Amazon Echo voice command. Happily this was very easy because I’d already built infrastructure to send MQTT commands (albeit to manage lighting) from Alexa. To add TV control was just a matter of adding another entry to the list of devices returned by the skill adapter when a discovery was initiated, and then initiating a discovery (“Alexa, discover my smart home devices“).

Example of an existing entry:

                {
                    "applianceId":"DiningRoomLights",
                    "manufacturerName":"James Bulpin",
                    "modelName":"LWRF",
                    "version":"v0.1",
                    "friendlyName":"Dining room lights",
                    "friendlyDescription":"Fireplace lights in the dining room",
                    "isReachable":True,
                    "actions":[
                        "turnOn",
                        "turnOff"
                    ],
                    "additionalApplianceDetails":{
                        "mqttTopics":"Light/Dining/Fireplace"
                    }
                },

New entry for the TV:

                {
                    "applianceId":"loungetv",
                    "manufacturerName":"James Bulpin",
                    "modelName":"LWRF",
                    "version":"v0.1",
                    "friendlyName":"TV",
                    "friendlyDescription":"Living room TV and amp",
                    "isReachable":True,
                    "actions":[
                        "turnOn",
                        "turnOff"
                    ],
                    "additionalApplianceDetails":{
                        "mqttTopics":"KODI/Lounge/TV"
                    }
                },

You can see this just uses the KODI MQTT message instead of a lighting message. Alexa has no idea it’s a TV rather than a light that’s being controlled. The entire skill adapter code can be seen here.

Of course it’s also possible to control multiple devices from a single Alexa command, just by sending multiple MQTT messages.

An enhancement

This all works well but you may have noticed that the IR commands to turn the TV on and off are the same – this is because that’s usually how IR remote controls work. This means that any command, via Alexa or otherwise, will actually toggle the current power state rather than move to the defined state. In most cases this isn’t a problem, for example why would you ask Alexa to turn the TV on if it’s on already? However for other more complicated automation activities I’d like to be able to control the TV without knowing whether it’s on or off already (e.g. I want an “all off” button or timer for a bedroom that will turn off all lights and the TV no matter what was already on).

To do this I added an input to sense if the TV was on – power being available on a TV USB port was my chosen method. I used a small opto-isolator, driven by the USB power, with the other side connected to a GPIO input.

To use this in the Kodi add-on I first had to add the “Script – Raspberry Pi Tools” add-on (search the registry from the Kodi add-on manager UI to find this) to add the GPIO Python library. To use this from my add-on code I added:

sys.path.append('/storage/.kodi/addons/virtual.rpi-tools/lib')
import RPi.GPIO as GPIO

# Set up pin for TV power monitoring (active low)
GPIO.setmode(GPIO.BCM)
GPIO.setup(23, GPIO.IN)

Then added simple checks in the MQTT message handler to only send IR commands if the current state is not the commanded state:

                if msgpayload.lower() == "off":
                    if GPIO.input(23) == 0:
                        # Only send if the TV is on - (GPIO pin is low)
                        ircmds.append(["Vestel_TV", "OFF", 5])
                elif msgpayload.lower() == "on":
                    if GPIO.input(23) == 1:
                        # Only send if the TV is off - (GPIO pin is high)
                        ircmds.append(["Vestel_TV", "OFF", 5])

In a future blog post I’ll show how I extended this TV control to add automatic input source selection by interfacing to, and automating a HDMI switcher box.

alexatv

Hybrid local/cloud style home automation

When I first started automating items in my home I used a local (“on-prem” if you like) point solution for security lighting. This had a connection to an internet service (Google Calendar) but only in a “pull” manner, and only for that specific use-case. As time went on I wanted to enable a broader set of use-cases, some of which required other internet-connected services in both pull and push directions. I experimented with using Octoblu, with the local LightwaveRF controller being connected to it – all the automation therefore was being performed in the cloud. As I looked at adding additional devices (PIR detectors, reed switches, IR blasters, and so on) I encountered a number of failures and delays caused by this reliance on my domestic internet connection and the (at the time) patchy reliability of Octoblu (it’s far more solid now).

My solution was to use a “hybrid” model with a local message broker and set of services running inside the home, and a connection to Octoblu for cases when external services were involved. Overall this means that truly local operations, such as PIRs turning on lights depending on the position of various reed switches, can be kept within my home network with no reliance on my internet connection or any external service. In cases where external stimuli are used (e.g. for connecting Amazon Echo’s smart home stuff) or external actions are required (e.g. sending a text message using Twilio) the bridge between the two worlds allows a message to be forwarded in either direction.

The on-prem piece

homeautopiThe local system is based on a MQTT broker running on a Raspberry Pi. This provides a publish-subscribe messaging system that various services connect to. The broker and many of the services run on the same Raspberry Pi however the broker also listens on a TCP port to allow other Raspberry Pis (and potentially other devices) in the home to join the party. I also have a Redis key-value-pair (KVP) store which is used to store state such as presence information or whether it’s day or night.

As an illustration of how the system is used: the LightwaveRF radio-controlled lighting system is connected via a small node.js service that subscribes to lighting messages (e.g. “Light/Kitchen/Main=on“) and sends the appropriate command to the local LightwaveRF hub which then emits the 433MHz radio command. When I later added a Philips Hue system alongside the LightwaveRF one I created a similar node.js service that also subscribes to lighting messages and makes calls to the Hue hub’s API as necessary. In both cases the services ignore messages about lights they don’t control. This all means other services, such as the PIR detector, or the rules engine, can simply publish a lighting control message without worrying about whether it’s a LightwaveRF, Hue, or (perhaps in the future) other system managed locally or via a cloud service.

Other services that publish and/or subscribe to MQTT messages include:

  • A Bluetooth Low Energy (BLE) advertisement monitor – this is used to determine the presence of absence of various things
  • A Google calendar interface that can update local KVP state, or generate messages, based on scheduled events
  • A GPIO service that enables things like PIRs and push buttons to be attached to Raspberry Pis
  • A timer service that other services and rules can use to provide timeouts, e.g. to turn off PIR-controlled lights after a set delay
  • Plug-ins for the Kodi media player to control Kodi itself as well as IR emitters for nearby TVs and other devices (stay tuned for a blog post on this)

homeautoruleA core part of the whole system is the rules engine. This is a service that subscribes to MQTT messages, implements rules, and publishes MQTT messages as necessary. It doesn’t directly interface to any device. An example of a rule is managing lighting for the basement stairs: inputs to the rule are two PIRs (via the GPIO service), the value of the day/night KVP, and the timeout message from a named timer (from the timer service). If a PIR triggers and it is night (from the KVP) then a lighting message (“Light/Stairs/Basement=on“) is published; when the PIR stops detecting, a timer control message is published (“Timer/basement_stairs_lighting/reset=<time in 60s>“) to start the timer; receipt of the timer’s timeout message causes the rule to emit an “off” lighting message.

The Octoblu bridge

homeautodiagram

So, what if I want to control stuff in the home from somewhere else? Or if I need to send a message to a cloud-connected device or service? That’s where the Octoblu connector comes in. I built a basic connector that runs as a node.js service on the main MQTT Raspberry Pi. This subscribes to a subset of MQTT topics and has the ability to publish arbitrary messages into the on-prem system. It also acts as an Octoblu connector, maintaining a bidirectional connection into Octoblu. This allows my entire home automation world to appear as a thing in Octoblu so I can send messages to it, which leads to MQTT published messages, and receive messages from it, which came from MQTT subscribed messages.

As an example, I can remote-control a light using a button on a web page by connecting an Octoblu “trigger” (a thing that can respond to a HTTPS POST and then emit a message into the Octoblu platform) to the home automation thing, and ensure the message payload includes the right MQTT topic (e.g. “Light/Kitchen/Main“) and message (e.g. “on“). The connector routes this message into my home as a published MQTT message that the LightwaveRF or Hue service will act on. This is the core mechanism I used when connecting my home lighting up to Amazon Echo using an Alexa smart home skill adapter.

In the other direction an on-prem service, such as the rules engine, can send a message to Octoblu just by publishing to a suitable topic (currently this means it has a “Octoblu/” prefix) via the local MQTT broker. An example of this is a security feature that sends me a text message if a particular combination of stimuli are seen. The rules engine publishes a message “Octoblu/alert/<detail>=<status>“, the Octoblu connector receives this via its MQTT subscription and sends it into Octoblu. The Octoblu flow then decides how to process this message which may end up with a call to the Twilio thing to send a text message.

Other examples of using the Octoblu connection include:

  • Sending events for sunrise and sunset, based on pulling data from a weather API (saves having to drill walls to install a light sensor)
  • Routing messages from Alexa skills handlers (these run in the cloud as AWS Lambda functions so I need a way to route the message back across my firewall)

In closing

I’ve ended up with a hybrid-cloud style of IoT management for my home automation, not by design, but by evolution. I find that being able to combine local automation – with its low latency and lack of reliance on external connectivity and services, with a powerful cloud-based automation platform able to send and receive from a variety of cloud/internet services, is a best-of-both-worlds solution.

homeautooctoblu

Hacking Big Mouth Billy Bass – part 2/3

A couple of weeks ago there was a lot of interest on social media in a video showing a Big Mouth Billy Bass animatronic novelty device seemingly take on the role of Amazon Echo, including animating its mouth and body in sync with the speech. With my recent exploits in connecting strange crap to Octoblu I decided to have a go at automating and Internet-connecting Billy Bass.

In this three-part blog series I’ll cover:

  1. Reverse-engineering Billy Bass and automating its movements based on audio input
  2. (This part) – connecting Billy Bass to the Octoblu IoT automation platform and synthesising speech for it
  3. Controlling Billy Bass with Amazon Echo and Slack

Recap and introduction

In the first part of this blog series I described how I added an Arduino to Big Mouth Billy Bass to enable it to be controlled by commands over USB-serial as well as automatically moving its mouth in sync with audio input. In this instalment I’ll describe how I connected this to the Internet using the Octoblu IoT automation platform.

The Octoblu connector

“Things” can be connected to Octoblu via “connectors”. Connectors run somewhere that they can talk to the “thing” as well as talk to Meshblu, Octoblu’s messaging platform. In this case my connector runs on a Raspberry Pi. The overall operation of the connector is that it listens for incoming Meshblu messages which contain a base64 encoded mp3 file. Each received message is added to a local FIFO queue (to avoid trying to play multiple messages simultaneously). When a message is played from the queue the connector first sends a USB-serial message to the Billy Bass Arduino to tell it to move the fish’s head. It then creates a temporary mp3 file and base64-decodes the mp3 content from the message into it. Next the connector spawns an external mp3 player program on the Pi which plays the mp3 file and on completion the callback in the connector gives the fish’s tail a quick flip before returning all body parts to their at-rest position.

I created the connector using a simple node.js program based on the v1 (now deprecated) Meshblu NPM library (“npm install meshblue@1.34.1”). This is very much prototype code with hard-coded items and without much error handling. To use the meshblu library I needed to create a “generic device” from Octoblu’s “All Things” list: having clicked on “GENERIC DEVICE” in the “OTHER” section, I clicked the “CONNECT GENERIC DEVICE” button; leaving the selection as “Register a new thing”, entering a name for the device (“My Billy Bass” here) and leaving the “type” field empty; I clicked “CONNECT GENERIC DEVICE” again; and found the new “My Billy Bass” device in the list of my things. I clicked on the device to bring up its configuration page. From here I copied the UUID, and clicked “GENERATE TOKEN” to generate and show an authentication token – I copied this also. I then added both the UUID and token to a file named meshblu.json (see the comment in the code for an example) alongside the connector.

connectorsetup

As well as meshblu, I also needed to npm-install child_process, fs, serialport and tempfile. Having connected the Arduino to the Pi and ensured its serial interface was available as /dev/ttyACM0, and having connected the fish’s audio input (this goes to the amplifier and to the Arduino’s analogue input) to the Pi’s headphone socket, I ran “node piaudio.js”. This confirmed I was connected to Meshblu but this won’t do anything useful without an Octoblu flow.

piaudiojs

The Octoblu flow

So now I’ve got a fish that can talk and move its mouth, head and tail by playing an mp3 file sent to it via Meshblu. To make this do something worthwhile meant building an Octoblu flow to create a suitable mp3 file containing synthesised speech. To do this I used the text-to-speech web service Voice RSS. This service provides an easy-to-use API service to turn text into speech in a variety of formats, languages and accents. It has a free tier that allows for up to 350 requests per day. I had looked at doing text-to-speech locally on the Pi using a tool such as Festival, however I found the Voice RSS output to be much higher quality, and using a web service allows for more future flexibility than having everything locked down on the Pi.

The flow begins by generating the text that Voice RSS will later convert to speech. In this flow I have two ways to do this: firstly, for testing, I am hard-coding two test phrases using “compose” tools (these are just assigning the text strings to a key named “text” that will later be extracted in the HTTP API call to Voice RSS). The second mechanism is to receive the text via a HTTPS POST call into an Octoblu trigger tool (it’s named “Call from Alexa skill” in this flow – you’ll see this in use in the video at the end of this post as well as in the third part of this series). This then gets put into the same “text” key as above.

billbassflow

The Voice RSS API is very easy to drive from Octoblu. I simply used a HTTP GET tool configured with the various API parameters, including my voicerss.org API key. The text to convert is taken from the “text” field of the incoming message using Octoblu’s “moustache” syntax (“{{msg.text}}”).

voicerss

collatemp3The output from the HTTP GET is base64 encoded mp3 which is carried in the message as the only payload. The “compose” tool that follows the HTTP GET simply puts this payload into a key named “mp3” to match what the connector is expecting (the reason for doing this is to allow extension with other data/metadata, such as customised motor commands, in the future). This is then sent to the “My Billy Bass” device (the one I configured earlier) with the “Use Incoming Message” option enabled (this causes the message we just created with the mp3 data to be sent directly to the connector).

Summary

Having connected my Arduino-connected Big Mouth Billy Bass to a Raspberry Pi and to Octoblu I now have a HTTPS POST webhook I can call with some text that will be sent to Voice RSS to be synthesised into speech. This will then be sent to Billy Bass via my Meshblu connector and the fish will speak and move its mouth, head and tail to sync with the speech.

In the third and final part of this blog series I’ll use these mechanisms to connect Billy Bass to Amazon Echo (as in the video below) and Slack.

Code

 

 

 

Getting started with home automation

20160908_195003I’ve always had an interest in electronics and software, and in connecting the two together. I guess it started in my pre-teen years when I figured out I could use the cassette motor relay on my family’s BBC Micro to turn LEDs and buzzers on and off, and it grew as I learned to use the BBC’s 8 bit “user port” with more sophisticated electronics. I’ve used a variety of technologies and gadgets over the years, including various microcontrollers (PIC in particular) accessed via serial (RS232 or RS423), Elexol’s USBIO24 GPIO devices and, more recently, various Arduino and Raspberry Pi models. I’ve built a number of different things such as controls and displays for my flight simulator, computer control and feedback interfaces for my model railway, and miscellaneous gadgets of varying levels of utility. But until recently an area I’ve not explored is home automation – a topic well aligned with my experience and interests. That changed about 4 years ago when the problem of needing landlord permission to run cables and drill holes disappeared and I became a homeowner.

Like most people who venture into home automation my starting point (and mostly still the extent of my experience) was lighting. The automation of lights is both relatively straightforward and safe – there are no moving items under control of my software so no bug can lead to overheated motors or physical damage.

20160908_185431A decade ago I’d probably have started by buying a bunch of relays or triacs and working back from that (in fact I still have a device much like this that I fear to plug in) however I decided that it was time to leave the mains voltage to the professionals and concentrate on the low voltage stuff and the software. When I looked around for suitable hardware I was particularly impressed by the LightwaveRF range. The core of the range is a set of dimmers and relays in various form-factors including inline (which for the light switch types mean they need a leakage current through the lights to power them – something I’m not keen on) and plug-in units. All the dimmers and relays are controlled by 433MHz RF, either locally from handheld remotes or wall-mounted buttons, or via the network using an Ethernet to RF bridge (called a “Wi-Fi Link” despite having no Wi-Fi capability!).

20160908_185332The “Wi-Fi link” is designed to be used with LightwaveRF’s cloud service to enable control of the lights from mobile apps or via REST APIs. However, it can also be used directly by sending suitably formatted UDP packets to it.

When I first started using LightwaveRF my priority was to use the system to replace timer-based security lights simulating occupation while away from home. I initially started with Paul Clarke’s excellent “lightwaverf” tool which can drive the Wi-Fi Link from either command line invocations or from events in a Google calendar. I ran this tool on an early-model Raspberry Pi sitting in the cellar. I coupled this with an IFTTT recipe that runs daily and populates a dedicated Google calendar with a sunset event which was the basis for triggering the switching on of some of house lights.

20160908_185404This was all fine until Google withdrew the ability to share a calendar privately as an XML download. This was the motivation to build a more flexible system that would enable more use-cases for automatic light control (these will be the subject of future blogs). I looked at using Octoblu, an IoT automation platform acquired by Citrix. It didn’t have support for LightwaveRF but I did end up building a simple gateway to allow me to use the two technologies together. More importantly it was, at the time, not particularly stable (it’s much better now) so I ended up building a local MQTT-based system with a bunch of individual services connected via the MQTT broker (all running on the same Raspberry Pi as the previous system). One of these services is the LightwaveRF service which subscribes to messages about light controls and sends the appropriate UDP commands to the LightwaveRF Wi-Fi Link device (turns out there’s a NPM library for this). Another service is the calendar service which uses the Google calendar APIs to regularly pull down the same automation events as I was using with the previous system, but this time using my Google account and an API key, thus avoiding the problem of the missing private XML calendar. This service then issues lighting control command via messages published to the MQTT broker. By adopting the same calendar format as used previously I was able to continue to use the IFTTT recipe to populate the sunset events and enable/disable the security lights based on my travel plans

I’ve since added more services to the MQTT environment enabling more lighting automation use-cases – keep an eye out for more blog posts on these in the future.