An IoT-connected PowerPoint multi-device show!

Today I was honoured to have the opportunity to discuss the challenges and opportunities IoT brings to the world of technical documentation in a keynote for the inaugural Cambridge meet-up of the Write The Docs community.

One of the topics I covered was how IoT enables human-computer interaction across a much broader range of devices than the traditional screen, keyboard and pointing device – the 4th generation user interface as Steve Wilson describes it – leading to richer, more natural, and more immersive experiences for users of applications (in the broadest sense).

To help illustrate this in the context of a presentation I decided to extend the slideshow beyond the projector and slide clicker and include lights and buttons to create a more entertaining experience. For example I talked about some of the opportunities devices such as Amazon Echo could create. As I reached the relevant slide a set of LEDs came on to illuminate an actual Echo device on a stand on the stage. I also had a large push-button illuminated with controllable multi-coloured LEDs; this became a software-defined push-button which did different things during the course of the presentation including advancing slides, turning off the LEDs, and so on. The LED colour and action assigned to the button were controlled based on the slide being projected.

As usual I used Octoblu as the IoT automation platform. A key input to the flow was a trigger for each time the PowerPoint presentation advanced a slide. To do this I created a PowerPoint macro named “OnSlideShowPageChange” (called in the obvious manner by the application) which sent a HTTP POST to an Octoblu trigger with the current slide number and the total number of slides from the presentation.

Sub OnSlideShowPageChange(ByVal SSW As SlideShowWindow)
    Set objHTTP = CreateObject("MSXML2.ServerXMLHTTP")
    URL = "https://triggers.octoblu.com/v2/flows/<UUID>/triggers/<UUID>"
    objHTTP.Open "POST", URL, False
    objHTTP.setRequestHeader "User-Agent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.0)"
    objHTTP.setRequestHeader "Content-Type", "application/json"
    objHTTP.send "{""currentSlide"":" & SSW.View.CurrentShowPosition & ",""numSlides"":" & SSW.Presentation.Slides.Count & "}"
End Sub
To drive PowerPoint I created a new Octoblu connector that runs on my laptop that can remote control a PowerPoint slideshow. This is a very simple connector that really just adds an Octoblu shim on top of the excellent slideshow NPM library. The connector can do a few things but the most useful is the “GotoSlide” endpoint which can take a slide number, or one of “next”, “prev”, “first” or “last”. A simple example of using it in a flow is:
pptnode
(BTW if you want to use my connectors see https://github.com/jamesbulpin/my-meshblu-connectors for how to add my custom connector repository to your own Octoblu account. All the custom connectors in this blog post can be found in that repo.)
The flow looks rather more complex that it really is:
pptflowannotated
There are essentially five main parts to the flow:
  1. The trigger from the PowerPoint macro (telling us which slide is being shown) setting the state of a string of WS2811 LEDs (using my WS2811 Raspberry Pi connector which is also in the repo mentioned above) based on which slide is being shown. These LEDs illuminate the Echo and a few other props. The JSON templates contain configuration to tell the WS2811 connector which LEDs to turn on to illuminate the relevant prop. When the final slide is shown the entire string is put into a moving pattern (just for fun 🙂 ).
  2. The slide number is also used to show a slide progress bar on a second string of WS2811 LEDs. The connector takes the current slide number and the total number of slides. With the LEDs in the string mapping to slides in the presentation in a one-to-one manner, it turns LEDS green for slides already shown, red for those not yet shown, and yellow for the currently showing slide.
  3. The slide number is used to define the colour of the illuminated push-button (or to turn it off entirely) and to send it a string which the button will return (see below) if pushed – this makes the function of the button dynamically configurable depending on which slide is being shown.
  4. If the push-button is pushed it sends the string configured in #3 above. This part of the flow dispatches messages depending on that string. If it’s “ON” all the LEDs on both strings are put into a moving pattern and “OFF” turns all LEDs off. Otherwise it’s interpreted as a slideshow command (e.g. “next”) and routed to the PowerPoint connector described earlier.
  5. An additional effect of the button push in the “ON” and “OFF” cases is setting the button to do the opposite, i.e. if the push was “ON” then the button is reconfigured for “OFF”.

An aside on the WS2811 connector: This works by specifying a mode (via a parameter on the message) and, depending on the mode, some additional parameters such as colour or slide number. Currently the available modes are:

  • “off” – what you’d expect
  • “solid” (or “color”) – display a solid colour on all LEDs – requires the “color” parameter (can be a word such as “red” or hash hex value such as “#00553e”)
  • “slide” – the slide progress bar described above – requires “slide” and “slidemax” parameters
  • “colorwheel” – a moving and colour-changing dynamic pattern
  • “twinkle” – colorwheel modulated with a random twinkling (each LED turning on and off randomly)
  • “percent” – show a percentage bargraph (“VU meter” style) on the LEDs – requires the “percent” parameter, turns on the first percent% of the LEDs
  • “direct” – takes a JSON object containing a description of which LEDs should be which colour: pass the object (e.g. using the JSON Template node and passing {{msg}} in the parameter) containing a key named “groups” which is a list of objects with “color” (string color name/hascode) and “leds” (a list of integer LED index with the first LED on the wire being zero) parameters.

e.g.:

{
  "groups":[
    {
      "color":"blue",
      "leds":[1,3,5,9]
    },
    {
      "color":"red",
      "leds":[0,10,132]
    }
  ]
}

(This WS2811 connector, developed in collaboration with John Moody, is a better version of the one I described in a previous post – see that post for info on the wiring.)

Now of course there are ways to do similar things to all of this without IoT but like many things with IoT it’s the removal of barriers of cost, accessibility and vendor compatibility that make an IoT approach interesting.

If you’re at Citrix Synergy 2017 in Orlando later this month be sure to join me and a growing list of IoT and automation experts for SYN401 to see what else is possible with the Octoblu platform. I can guarantee that the PowerPoint (there won’t be much – it’s a far more interactive session than that!) will be even more IoT than in today’s event!

Displaying messages on my Christmas tree lights

IoT and Christmas go well together – all those festive lights and sounds are just asking to be connected up to the Internet and automated! My own Christmas tree is of course no exception and, as I wrote about over on the Citrix blogs, now sports a string of individually-addressable LEDs controlled by a Raspberry Pi which is in turn managed by Octoblu.

But I decided to go a step further than simply displaying some fun flashing patterns and colours on the tree: seeing as every LED can be individually managed, if I could determine the exact position on the tree of each LED I could come up with geometric patterns or even use it as a sort of dot matrix display to show messages.

iotreecalibratewebcamMy solution was to connect a webcam to the Raspberry Pi and write a script that turned each LED on one by one, captured an image from the webcam, and found the coordinates of the bright spot on the image. After the script ran through all 100 LEDs it then normalised the coordinates to a 100 x 100 grid and output a list of coordinates that I could then use in my LED controller program. The code is a bit gross, being a mash-up of various bits of other scripts quickly bolted together, including a web server that I used to view the image while positioning the camera.

iotreecalibrateTo visually check the output I wrote a quick bit of HTML/JavaScript that used the pre-normalised coordinates to overlay blobs on top of a captured image from the webcam (with all LEDs lit) – I could then see if the blobs lined up with the LEDs. As you can see in the image here there is at least one misalignment caused by the reflection of the light on a nearby white surface.

So armed with a set of coordinates for the LEDs I then extended my existing WS2811 LED control program (see my post on the Citrix blog for more on this, and on the hardware setup) to use coordinates, instead of just LED sequence number, for its patterns.

Firstly I created a simple set of test patterns that moved a vertical line horizontally across the tree, a horizontal line vertically and then the two different diagonals. I also updated the VU meter effect to use Y coordinates instead of LED sequence number.

However the most fun piece was rendering text on the tree. To do this I found a suitable encoding of a dot matrix font online (there are loads out there) and morphed it into a suitable form to include in Node.js. I then wrote a very simple rendering function that runs over a string of text one character at a time, using the font configuration to determine which pixels in each column of the 5×7 matrix should be illuminated. The end result was an array with an entry per column (for the entire message) with the entry being an encoding of which row LEDs should be illuminated. I also created a similar array to record which colour to use for each column – in this case populated to cycle through red, green and blue on a character-by-character basis (this helps the eye to see the characters better).

var dmmessage = " XMAS";

var dmarray;
var dmcolor;
var colorcycle = 255;
function renderMessage() {
  dmarray = new Uint32Array(dmmessage.length * 7 + 5);
  dmcolor = new Uint32Array(dmmessage.length * 7 + 5);
  for (var i = 0; i < dmmessage.length; i++) {
    var fonttablebase = (dmmessage.charCodeAt(i) - 0x20) * 5;
    for (var j = 0; j < 5; j++) {
      dmarray[i*7+j] = font5x7[fonttablebase + j];
      dmcolor[i*7+j] = colorcycle;
    }
    if (colorcycle == 0xff0000) {
      colorcycle = 255;
    } else {
      colorcycle = colorcycle << 8;
    }
  }
}

In the main part of the WS2811 driver program, where a loop iterates over the 100 LEDs choosing which colour to display for each, the code uses the incrementing offset variable (increments once per 100-LED update cycle, every 80ms) to index into the column array, offset by the X coordinate of the LED. A quantised Y coordinate of the LED is used to look up the row pixel on/off data from the array entry – the quantisation effectively creating 7 rows across the surface of the tree.

function dotmatrix_y(y) {
  if (y < 25.0) return 8;
  if (y < 40.0) return 0;
  if (y < 50.0) return 1;
  if (y < 60.0) return 2;
  if (y < 70.0) return 3;
  if (y < 80.0) return 4;
  if (y < 90.0) return 5;
  return 6;
}

function dotmatrix(pos) {
  var x = xy.xy[pos][0];
  var ydot = dotmatrix_y(xy.xy[pos][1]);

  var idx = Math.floor(((offset + x)/10)%dmarray.length);
  var column = dmarray[idx];

  if ((1<<ydot)&column) {
    return dmcolor[idx];
  }
  return 0;
}

I added a new parameter to the Octoblu message to allow the display string to be set. I updated the Octoblu flow and Alexa skill to allow a command such as “Alexa, tell the Christmas tree to say Hello”.

Alexa intent schema (new part in bold, full code here):

{
  "intents": [
    {
      "intent": "ModeIntent",
      "slots": [
        {
          "name": "mode",
          "type": "LIST_OF_MODES"
        }
      ]
    },
    {
      "intent": "ColorIntent",
      "slots": [
        {
          "name": "color",
          "type": "LIST_OF_COLORS"
        }
      ]
    },
    {
      "intent": "TextIntent",
      "slots": [
        {
          "name": "text",
          "type": "LIST_OF_WORDS"
        }
      ]
    },
    {
      "intent": "AskOctobluIntent"
    },
    {
      "intent": "SorryIntent"
    }
  ]
}

AWS Lambda intent handler (new part in bold, full code here):

def on_intent(intent_request, session):
    """ Called when the user specifies an intent for this skill """

    print("on_intent requestId=" + intent_request['requestId'] +
          ", sessionId=" + session['sessionId'])

    intent = intent_request['intent']
    intent_name = intent_request['intent']['name']
    
    mode = None
    color = None
    text = None

    session_attributes = {}
    speech_output = None

    # Dispatch to your skill's intent handlers
    if intent_name == "ModeIntent":
        if ('mode' in intent['slots']) and ('value' in intent['slots']['mode']):
            mode = intent['slots']['mode']['value']
    elif intent_name == "ColorIntent":
        if ('color' in intent['slots']) and ('value' in intent['slots']['color']):
            color = intent['slots']['color']['value']
            mode = "solid"
    elif intent_name == "TextIntent":
        if ('text' in intent['slots']) and ('value' in intent['slots']['text']):
            text = intent['slots']['text']['value']
            mode = "dotmatrix"
    elif intent_name == "AskOctobluIntent":
        mode = "octoblu"
    elif intent_name == "SorryIntent":
        speech_output = "I'm sorry James, I can't do that"
    else:
        raise ValueError("Invalid intent")

    obmsg = {"debug":intent}
    if mode:
        obmsg["mode"] = mode
    if color:
        obmsg["color"] = color
    if text:
        obmsg["text"] = text
    url = octoblu_trigger
    data = urllib.urlencode(obmsg)
    req = urllib2.Request(url, data)
    response = urllib2.urlopen(req)
    the_page = response.read()
    
    if not speech_output:
        speech_output = "OK"
    return build_response(session_attributes, build_speechlet_response(speech_output))

WS2811 Octoblu message handler (new part in bold):

conn.on('ready', function(rdata){
  console.log('UUID AUTHENTICATED!');
  console.log(rdata);

  clearTimeout(connectionTimer);
  connectionTimer = undefined;

  conn.update({
    "uuid": meshbluJSON.uuid,
    "token": meshbluJSON.token,
    "messageSchema": MESSAGE_SCHEMA,
  });

  conn.on('message', function(data){
    console.log('Octoblu message received');
    console.log(data);
    mode = data.mode;
    color = ("color" in data)?data.color:null;
    if (("text" in data) && data.text) {
      dmmessage = " " + data.text.toUpperCase();
      renderMessage();
      offset = 0;
    }
    handle_message(mode, color)
  });

});

Octoblu connects together the HTTPS POST request from the Alexa skill handler with the WS2811 driver program.

iotreeoctoblu

And there we have it – Alexa-controlled scrolling message display on a Christmas tree thanks to Raspberry Pi and Octoblu!

 

Hybrid local/cloud style home automation

When I first started automating items in my home I used a local (“on-prem” if you like) point solution for security lighting. This had a connection to an internet service (Google Calendar) but only in a “pull” manner, and only for that specific use-case. As time went on I wanted to enable a broader set of use-cases, some of which required other internet-connected services in both pull and push directions. I experimented with using Octoblu, with the local LightwaveRF controller being connected to it – all the automation therefore was being performed in the cloud. As I looked at adding additional devices (PIR detectors, reed switches, IR blasters, and so on) I encountered a number of failures and delays caused by this reliance on my domestic internet connection and the (at the time) patchy reliability of Octoblu (it’s far more solid now).

My solution was to use a “hybrid” model with a local message broker and set of services running inside the home, and a connection to Octoblu for cases when external services were involved. Overall this means that truly local operations, such as PIRs turning on lights depending on the position of various reed switches, can be kept within my home network with no reliance on my internet connection or any external service. In cases where external stimuli are used (e.g. for connecting Amazon Echo’s smart home stuff) or external actions are required (e.g. sending a text message using Twilio) the bridge between the two worlds allows a message to be forwarded in either direction.

The on-prem piece

homeautopiThe local system is based on a MQTT broker running on a Raspberry Pi. This provides a publish-subscribe messaging system that various services connect to. The broker and many of the services run on the same Raspberry Pi however the broker also listens on a TCP port to allow other Raspberry Pis (and potentially other devices) in the home to join the party. I also have a Redis key-value-pair (KVP) store which is used to store state such as presence information or whether it’s day or night.

As an illustration of how the system is used: the LightwaveRF radio-controlled lighting system is connected via a small node.js service that subscribes to lighting messages (e.g. “Light/Kitchen/Main=on“) and sends the appropriate command to the local LightwaveRF hub which then emits the 433MHz radio command. When I later added a Philips Hue system alongside the LightwaveRF one I created a similar node.js service that also subscribes to lighting messages and makes calls to the Hue hub’s API as necessary. In both cases the services ignore messages about lights they don’t control. This all means other services, such as the PIR detector, or the rules engine, can simply publish a lighting control message without worrying about whether it’s a LightwaveRF, Hue, or (perhaps in the future) other system managed locally or via a cloud service.

Other services that publish and/or subscribe to MQTT messages include:

  • A Bluetooth Low Energy (BLE) advertisement monitor – this is used to determine the presence of absence of various things
  • A Google calendar interface that can update local KVP state, or generate messages, based on scheduled events
  • A GPIO service that enables things like PIRs and push buttons to be attached to Raspberry Pis
  • A timer service that other services and rules can use to provide timeouts, e.g. to turn off PIR-controlled lights after a set delay
  • Plug-ins for the Kodi media player to control Kodi itself as well as IR emitters for nearby TVs and other devices (stay tuned for a blog post on this)

homeautoruleA core part of the whole system is the rules engine. This is a service that subscribes to MQTT messages, implements rules, and publishes MQTT messages as necessary. It doesn’t directly interface to any device. An example of a rule is managing lighting for the basement stairs: inputs to the rule are two PIRs (via the GPIO service), the value of the day/night KVP, and the timeout message from a named timer (from the timer service). If a PIR triggers and it is night (from the KVP) then a lighting message (“Light/Stairs/Basement=on“) is published; when the PIR stops detecting, a timer control message is published (“Timer/basement_stairs_lighting/reset=<time in 60s>“) to start the timer; receipt of the timer’s timeout message causes the rule to emit an “off” lighting message.

The Octoblu bridge

homeautodiagram

So, what if I want to control stuff in the home from somewhere else? Or if I need to send a message to a cloud-connected device or service? That’s where the Octoblu connector comes in. I built a basic connector that runs as a node.js service on the main MQTT Raspberry Pi. This subscribes to a subset of MQTT topics and has the ability to publish arbitrary messages into the on-prem system. It also acts as an Octoblu connector, maintaining a bidirectional connection into Octoblu. This allows my entire home automation world to appear as a thing in Octoblu so I can send messages to it, which leads to MQTT published messages, and receive messages from it, which came from MQTT subscribed messages.

As an example, I can remote-control a light using a button on a web page by connecting an Octoblu “trigger” (a thing that can respond to a HTTPS POST and then emit a message into the Octoblu platform) to the home automation thing, and ensure the message payload includes the right MQTT topic (e.g. “Light/Kitchen/Main“) and message (e.g. “on“). The connector routes this message into my home as a published MQTT message that the LightwaveRF or Hue service will act on. This is the core mechanism I used when connecting my home lighting up to Amazon Echo using an Alexa smart home skill adapter.

In the other direction an on-prem service, such as the rules engine, can send a message to Octoblu just by publishing to a suitable topic (currently this means it has a “Octoblu/” prefix) via the local MQTT broker. An example of this is a security feature that sends me a text message if a particular combination of stimuli are seen. The rules engine publishes a message “Octoblu/alert/<detail>=<status>“, the Octoblu connector receives this via its MQTT subscription and sends it into Octoblu. The Octoblu flow then decides how to process this message which may end up with a call to the Twilio thing to send a text message.

Other examples of using the Octoblu connection include:

  • Sending events for sunrise and sunset, based on pulling data from a weather API (saves having to drill walls to install a light sensor)
  • Routing messages from Alexa skills handlers (these run in the cloud as AWS Lambda functions so I need a way to route the message back across my firewall)

In closing

I’ve ended up with a hybrid-cloud style of IoT management for my home automation, not by design, but by evolution. I find that being able to combine local automation – with its low latency and lack of reliance on external connectivity and services, with a powerful cloud-based automation platform able to send and receive from a variety of cloud/internet services, is a best-of-both-worlds solution.

homeautooctoblu

Hacking Big Mouth Billy Bass – part 3/3

 

A few weeks ago there was a lot of interest on social media in a video showing a Big Mouth Billy Bass animatronic novelty device seemingly take on the role of Amazon Echo, including animating its mouth and body in sync with the speech. With my recent exploits in connecting strange crap to Octoblu I decided to have a go at automating and Internet-connecting Billy Bass.

In this three-part blog series I’ll cover:

  1. Reverse-engineering Billy Bass and automating its movements based on audio input
  2. Connecting Billy Bass to the Octoblu IoT automation platform and synthesising speech for it
  3. (This part) controlling Billy Bass with Amazon Echo and Slack

Recap and introduction

In the first part of this blog series I described how I added an Arduino to Big Mouth Billy Bass to enable it to be controlled by commands over USB-serial as well as automatically moving its mouth in sync with audio input. In the second part I described how I connected this to the Internet using the Octoblu IoT automation platform. In this final part I’ll describe how I used the Octoblu integration to control the fish from Amazon Echo and Slack.

Where we got to last time was an Octoblu flow that could take a message containing a text string, send it to the Voice RSS text-to-speech service, then send to resulting mp3 to the Billy Bass fish to play via its speaker and move motors in sync. The examples in this blog post look at how to generate a message with suitable text in order to feed this flow.

Amazon Echo

The goal here was to create a “skill” for Amazon Echo to allow voice control of the fish, e.g. saying “Alexa, tell Billy Bass to say hello world” would send a message of “hello world” to the text-to-speech service as described in the previous part of this blog post series.

In the Octoblu web interface I edited my “Billy Bass” flow to add a trigger node to act as a HTTPS POST endpoint for the Alexa scripts to call – I named this “Call from Alexa skill”. I connected this to a “Compose” node which is just there to tidy up the incoming message from the POST data into the form the rest of the flow is expecting: it creates a key named “text” with value “{{msg.data.text}}”.

billybassalexatrigger

I’ve previously created a smart home skill adapter for Echo – creating a custom skill is broadly similar so I won’t repeat all the details again but will highlight the key parts specific to fishy conversations.

Firstly I created a new Alexa skill in the Amazon Developer Console using the custom skill type and an invocation name of “billy bass”. I then defined an interaction model for the skill – this describes the forms of wording that Echo will recognise and defines slots that are, in effect, variables that will be populated when voice recognition takes places. The intent schema defines just one intent – asking the fish to say something:

The slot is the word(s) that we’re asking the fish to say. The slot can either be a constrained type (such as a date, or fixed, enumerated list) or, as is the case here, an open-ended list. I defined the LIST_OF_SAYINGS slot type as a custom slot type with three simple examples: “hello world | how are you today | what time is it”.

The final part of the interaction model is to provide some sample utterances which tell Alexa what kind of phrasing maps to the intents (in theory this doesn’t need to be an exhaustive list of all ways in which a user may state the intention). Here I used the single sample: “BillySay to say {Saying}” where “BillySay” corresponds to the intent and “{Saying}” is a placeholder for the open-ended slot.

Overall this looks like:

billybassechomodel

alexabpThe next step was to create an AWS Lambda function to handle the skill. This is called each time Alexa matches an intent on this skill. I used a Python 2.7 blueprint for an Alexa skill. The blueprint prompts to create a trigger from the Alexa Skills Kit to Lambda – let it create this. In the Lambda function configuration page I edited the code (see below) and set up the role – I used the same role as for my home skill adapter (see here for details).

The core part of the code is the on_intent function which simply takes the text that Alexa recognised as being the “Saying” slot in the intent in the schema defined above and sends it to my Octoblu trigger as a HTTPS POST with the text being in the “text” field of the posted message. The skill ID (“amzn1.ask.skill.<UUID>”) can be found in the skill details in the Amazon Developer Console. The Octoblu trigger URL can be found from the “thing inspector” in the Octoblu web interface having clicked on the relevant Trigger node (the “Call from Alexa skill” one here).

Having saved and published the Lambda function I added its ARN (this should be displayed on the Lambda function’s pages in the AWS console) to the Alexa skill via the configuration page in the Amazon Developer Console.

And that’s pretty much it. After restarting the Octoblu flow I was able to say to the Echo “Alexa, tell Billy Bass to say hello world” – this led to Alexa matching the skill and intent I defined above then calling the Lambda function with the intent and its slot value (the words the fish has been asked to say). This function then called Octoblu via a HTTPS POST to the trigger and this then got passed to Voice RSS for text-to-speech and then back to Octoblu to be routed to the connector running on the Raspberry Pi and ultimately to the fish itself. With the fish and the Echo next to each other you can get the fish to talk to Alexa, e.g. say “Alexa, tell Billy Bass to say Alexa what time is it” – the fish will say “Alexa what time is it” and then the Echo will answer.

Slack

I’ve been playing with Slack as a way to have human interaction with Octoblu flows. Adding a way to control the Big Mouth Billy Bass seemed like a natural next step 🙂

Luckily Octoblu already has a good way to connect to Slack using a streaming “slurry” thing. I set this up using a private test Slack account I already had by first creating a bot account and extracting an API token for it (see Slack docs for details on doing this). I added the bot to each Slack channel I wanted it to respond to. I then navigated Octoblu’s web interface to “All Things” and selecting the “Slack Streaming Bot (beta)” thing. Having clicked on this I was asked to authorize access and then to provide the bot’s token.

Having saved this thing I then went to my existing “Billy Bass” flow, added the new Slack thing and clicked the “UPDATE PERMISSIONS” button to enable Slack to send messages to this flow. I routed Slack messages through a non-null filter on “{{msg.data.text}}” to ensure there is a parsable Slack message in there, then through a function node to look for Slack messages that start with “Fish:” and extract the rest of the message to forward to to the text-to-speech service an ultimately to the fish itself. The second filter node removed any messages that didn’t match the “Fish:” prefix.

slackfish

So now I can type into a Slack channel “Fish: hello world” and the Billy Bass will say “hello world.

In closing

So there we have it – an IoT talking fish integrated with Amazon Echo and Slack using Octoblu, Raspberry Pi, Arduino, Voice RSS and a handful of electronic components. Clearly not the most practical IoT device ever created but hopefully an illustration of the power of these tools.

 

Hacking Big Mouth Billy Bass – part 2/3

A couple of weeks ago there was a lot of interest on social media in a video showing a Big Mouth Billy Bass animatronic novelty device seemingly take on the role of Amazon Echo, including animating its mouth and body in sync with the speech. With my recent exploits in connecting strange crap to Octoblu I decided to have a go at automating and Internet-connecting Billy Bass.

In this three-part blog series I’ll cover:

  1. Reverse-engineering Billy Bass and automating its movements based on audio input
  2. (This part) – connecting Billy Bass to the Octoblu IoT automation platform and synthesising speech for it
  3. Controlling Billy Bass with Amazon Echo and Slack

Recap and introduction

In the first part of this blog series I described how I added an Arduino to Big Mouth Billy Bass to enable it to be controlled by commands over USB-serial as well as automatically moving its mouth in sync with audio input. In this instalment I’ll describe how I connected this to the Internet using the Octoblu IoT automation platform.

The Octoblu connector

“Things” can be connected to Octoblu via “connectors”. Connectors run somewhere that they can talk to the “thing” as well as talk to Meshblu, Octoblu’s messaging platform. In this case my connector runs on a Raspberry Pi. The overall operation of the connector is that it listens for incoming Meshblu messages which contain a base64 encoded mp3 file. Each received message is added to a local FIFO queue (to avoid trying to play multiple messages simultaneously). When a message is played from the queue the connector first sends a USB-serial message to the Billy Bass Arduino to tell it to move the fish’s head. It then creates a temporary mp3 file and base64-decodes the mp3 content from the message into it. Next the connector spawns an external mp3 player program on the Pi which plays the mp3 file and on completion the callback in the connector gives the fish’s tail a quick flip before returning all body parts to their at-rest position.

I created the connector using a simple node.js program based on the v1 (now deprecated) Meshblu NPM library (“npm install meshblue@1.34.1”). This is very much prototype code with hard-coded items and without much error handling. To use the meshblu library I needed to create a “generic device” from Octoblu’s “All Things” list: having clicked on “GENERIC DEVICE” in the “OTHER” section, I clicked the “CONNECT GENERIC DEVICE” button; leaving the selection as “Register a new thing”, entering a name for the device (“My Billy Bass” here) and leaving the “type” field empty; I clicked “CONNECT GENERIC DEVICE” again; and found the new “My Billy Bass” device in the list of my things. I clicked on the device to bring up its configuration page. From here I copied the UUID, and clicked “GENERATE TOKEN” to generate and show an authentication token – I copied this also. I then added both the UUID and token to a file named meshblu.json (see the comment in the code for an example) alongside the connector.

connectorsetup

As well as meshblu, I also needed to npm-install child_process, fs, serialport and tempfile. Having connected the Arduino to the Pi and ensured its serial interface was available as /dev/ttyACM0, and having connected the fish’s audio input (this goes to the amplifier and to the Arduino’s analogue input) to the Pi’s headphone socket, I ran “node piaudio.js”. This confirmed I was connected to Meshblu but this won’t do anything useful without an Octoblu flow.

piaudiojs

The Octoblu flow

So now I’ve got a fish that can talk and move its mouth, head and tail by playing an mp3 file sent to it via Meshblu. To make this do something worthwhile meant building an Octoblu flow to create a suitable mp3 file containing synthesised speech. To do this I used the text-to-speech web service Voice RSS. This service provides an easy-to-use API service to turn text into speech in a variety of formats, languages and accents. It has a free tier that allows for up to 350 requests per day. I had looked at doing text-to-speech locally on the Pi using a tool such as Festival, however I found the Voice RSS output to be much higher quality, and using a web service allows for more future flexibility than having everything locked down on the Pi.

The flow begins by generating the text that Voice RSS will later convert to speech. In this flow I have two ways to do this: firstly, for testing, I am hard-coding two test phrases using “compose” tools (these are just assigning the text strings to a key named “text” that will later be extracted in the HTTP API call to Voice RSS). The second mechanism is to receive the text via a HTTPS POST call into an Octoblu trigger tool (it’s named “Call from Alexa skill” in this flow – you’ll see this in use in the video at the end of this post as well as in the third part of this series). This then gets put into the same “text” key as above.

billbassflow

The Voice RSS API is very easy to drive from Octoblu. I simply used a HTTP GET tool configured with the various API parameters, including my voicerss.org API key. The text to convert is taken from the “text” field of the incoming message using Octoblu’s “moustache” syntax (“{{msg.text}}”).

voicerss

collatemp3The output from the HTTP GET is base64 encoded mp3 which is carried in the message as the only payload. The “compose” tool that follows the HTTP GET simply puts this payload into a key named “mp3” to match what the connector is expecting (the reason for doing this is to allow extension with other data/metadata, such as customised motor commands, in the future). This is then sent to the “My Billy Bass” device (the one I configured earlier) with the “Use Incoming Message” option enabled (this causes the message we just created with the mp3 data to be sent directly to the connector).

Summary

Having connected my Arduino-connected Big Mouth Billy Bass to a Raspberry Pi and to Octoblu I now have a HTTPS POST webhook I can call with some text that will be sent to Voice RSS to be synthesised into speech. This will then be sent to Billy Bass via my Meshblu connector and the fish will speak and move its mouth, head and tail to sync with the speech.

In the third and final part of this blog series I’ll use these mechanisms to connect Billy Bass to Amazon Echo (as in the video below) and Slack.

Code

 

 

 

Connecting the BS button to Octoblu

This didn’t start out as a sweary project. The original idea was to connect a big red button to Octoblu to allow it to start an online meeting, replicating what Chris Matthieu did with a BT.TN in this video. I thought it would be fun to get an actual Staples Easy Button and wire it up via a Raspberry Pi or similar. However, I’ve not been able to find one of these big red buttons available in the UK so I hunted around for alternatives. I came across a range of similar devices on Amazon with various words printed on the button and a set of recorded sounds appropriate to the printed word that would be played on a push of the button. I decided on this particular version, which has some, er, colourful things to say.

My original plan was to discard all the electronics except for the push button but on hearing the amusing, if slightly rude, things the device had to say I thought it would be a shame to strip it of its ability to answer back. So instead my unidirectional button->Octoblu plan became a bidirectional one with the ability to send a message to Octoblu when the button is pressed and to have Octoblu be able to send a message to the button to cause it to utter one of its delightful musings.

img_9031First step was to take a look inside to see how it worked and how much space I had to play with. I was pleasantly surprised at how easy it was to disassemble the device in a non-destructive manner. Removing three screws allowed me to easily separate the main body from the upper casing and push button. Care was necessary because the two halves are joined by two wires leading to the LED within the button – these broke from the PCB several times during the project.

The inspection gave me some key information to plan the build:

  1. The device runs on 3V from two batteries in a compartment in the base.
  2. There is quite a bit of empty space in the unit.
  3. The switch is part of the PCB with a clear trace from it to the main integrated circuit, ideally placed to be cut and soldered to.

img_9047u421Next I decided on an interfacing mechanism. I wanted to make the unit USB-connected to give me the flexibility of using it either with a Raspberry Pi or with a Windows NUC in the office. This meant using an Arduino or similar device with a USB interface. An Arduino Micro would have been a good choice but I don’t have any in stock right now and this was a “right now” project. Instead I dug out an old USBmicro.com U421 device – this is a simple 16 pin USB-connected GPIO device that is sufficiently small to fit within the battery compartment of the button, having removed the battery contacts and the central dividing wall.

Next job was powering the device so it can play its sounds and light up its LED. The unit normally works off two AAA batteries, i.e. 3V. The U421 is a 5V device. It’s probably the case that the button’s electronics would be fine with 5V but I didn’t want to blow it up and spoil my evening IoT playtime so I put three diodes in series to drop sufficient voltage to be safe (yes, I know this isn’t exactly elegant, but it works).

img_9043For the input side of things, the physical button push leading to a message to Octoblu, I simply cut the trace on the PCB between the push switch contacts and the device’s main IC and scratched off a patch of PCB laquer to be able to solder a wire directly to the PCB trace and hence connect to the switch contact (the other end of which was ground). This then connects to a GPIO pin on the U421 using a pull-up.

The output function, simulating a key press to trigger the device to play a sound, was done by connecting a GPIO pin configured as an output to the other side of the severed PCB trace connected to the device’s IC. I put a 4k7 resistor between the two chips to deal with voltage mismatches.

img_9065While coding up the software to talk to the U421 (more of which later) I realised that the dumb nature of this unit was forcing me to use a painfully tight polling loop to avoid missing button pushes. Later U421 revisions have a latching feature to help with this problem but this particular device can only be polled. I worked around this in hardware by inserting a one-shot timer (using a standard 555 monostable circuit with a 22k resistor and 47uF capacitor giving about a second of pulse) between the push button and the U421 to extend the time the the effect of the button push could be detected by software and therefore allowing a less aggressive polling frequency. The 555 circuit was assembed in free-space (gross, I know) and tucked into a gap beneath the circuit board inside the button device.

The software to drive this is currently in very raw prototype form. I’m currently using node.js with the old v1.x meshblu library to talk to Octoblu and the very comprehensive “usb” NPM package to interface with the U421 over USB. On the button side the script runs a 100ms polling loop to read the relevant GPIO pin. On a state change corresponding to a push it sends a message to Octoblu. On the sound/LED activation side it listens for messages from Octoblu and on receipt, toggles the output GPIO pin down and up to simulate a button push.

bsflowIn Octoblu I created a new generic device and copied its UUID and token to the local meshblu.json config file alongside my node.js program. To test both the send and receive function I created a flow that on receipt of a message (i.e. someone has pushed the button) sends it to a delay node that then forwards it back to the generic device 3 seconds later leading to a message back to the node.js program which triggers the button device to play a sounds and flash its LED. And it works!

img_9068

Controlling custom lighting with Amazon Echo and a skill adapter

ctb2adfxeaaptil-jpg-largeAmazon launched Echo in the UK today – it’s been a long wait! My pre-ordered Echo arrived this morning and my first priority was to get it (her?) to control my home lighting. As I’ve previously written about, my LightwaveRF lights are currently managed by a custom set of scripts communicating within the house using MQTT pub-sub. This means that Amazon Echo (or Alexa, as I’ll refer to it/her from now on) doesn’t know how to interface with them like she does with Philips Hue, for example.

Luckily Amazon has made available its Smart Home Skill API which allows individuals and home automation vendors to provide “skill adapters” for the various home automation “skills” Alexa already has. This means it is possible to say “Alexa, turn on the bedroom light” and have Alexa use whatever underlying system and technology you have to execute the command. This is preferable to defining a new custom skill because it avoids the need to use a skill trigger word (e.g. “Alexa, turn on the bedroom light using LightwaveRF”). AWS Lambda is used to provide the concrete implementation for the abstract actions.

octoblualexaIn my case the skill adapter will make a HTTPS call to an Octoblu trigger node passing details of the required action (essentially just a MQTT message with the topic being the light to control and the message body being the action (on or off)). The Octoblu flow then messes about a bit with the JSON structure before passing the message to an existing Meshblu device that connects my home MQTT world with Octoblu. In reality I’m using Octoblu and Meshblu here as firewall-bridging plumbing to get a MQTT message from the Lambda function into my home environment.

Having signed up for my Amazon developer account I followed Amazon’s steps to create a Smart Home Skill. This started by creating a new skill (of type “Smart Home Skill API”) in the Alexa Skills Kit section of the Amazon Developer Console – I chose to name this skill “James’s Lights”.

To provide the actual implementation I created a Python 2.7 Lambda function named “JamesLightsSkillAdapter” in AWS (using the eu-west-1 (Ireland) region to co-locate with the Alexa voice service for the UK) based on the alexa-smart-home-skill-adapter blueprint. I based the code on the template provided in the “steps to create” page above. For the role I selected “Create new role from template(s)”.

The code handles two separate types of message from the Alexa service:

  1. Device discovery – this is an action initiated by the end user from the Alexa app (or by voice) to get an inventory of devices that Alexa can control. In the Lambda function this is implemented by returning a big blob of JSON with an entry for each device. The “friendlyName” item being the words Alexa will recognise to control the device. I’m using the additionalApplianceDetails item to record the MQTT topic that will be used to control this device. My initial prototype implementation hard-codes the entire inventory in the Lambda function – clearly not a long term solution!
  2. TurnOnRequest and TurnOffRequest commands – these are issued by the Alexa service when Alexa is asked to turn a device on or off and the device is recognised as one in the inventory. The Lambda function is called with the relevant JSON blob and therefore my code can pull out the previously configured MQTT topic and send that as part of a HTTPS POST to the Octoblu trigger mentioned above.

I tested the Lambda function using the “Alexa Smart Home – Control” sample event template, manually crafting the JSON messages to match the ones the Alexa service will be sending. After testing it’s important to make sure to enable the Alexa trigger for the Lambda function.

echoskill

Back in the Developer Portal I configured the skill to use an endpoint in AWS Europe using the Lambda ARN created above. As this will be a private skill adapter I didn’t really need account linking (this is how a regular user would link their Echo with, for example, their Philips Hue account) but the console wouldn’t let me proceed without setting it up. Therefore I followed this great blog post’s instructions on how to use Amazon itself as the linked account.

file-28-09-2016-21-56-25Having saved all the skill configuration and enabling testing I then used the Alexa app on my iPad to add this newly created skill (in the “Your skills” section) and logged in with my Amazon account as the account linking step. From there is was a simple matter of letting the app and Alexa discover the devices and then Alexa was good to go.

I can now say to Alexa: “Alexa, turn the lounge light on” and a second or so later it’ll come on and Alexa will say “OK”. What happens under the hood is more interesting though:

  1. The Alexa cloud service processes the message, figuring out its a smart home control command
  2. The Alexa service looks through my discovered device list and identifies the “lounge light” is one that is controlled via this skill adapter.
  3. The Alexa service makes a call the my AWS Lambda function with a JSON message including the configuration for the requested light as well as the “TurnOnRequest” command.
  4. My Lambda function makes a HTTPS POST call to the Octoblu trigger with a MQTT-like message including the topic string for the requested light and the “on” message.
  5. The Ocotblu flow forwards this message via Meshblu to a simple Meshblu connector I have running at home.
  6. My Meshblue connector publishes the messaage to my local MQTT broker.
  7. The LightwaveRF script also running at home and subscribed to “Light/#” messages picks up the message and looks up the room/device codes which it then send via UDP to the LightwaveRF bridge.
  8. The LightwaveRF bridge sends the appropriate 433MHz transmission which is picked up by the paired light fitting and the power is switch on.

As this is all highly customized to me I’ll be leaving the app in its testing state, not putting it forward for certification (which would certainly fail!) and publication.

Future work

Right now the implementation hard-codes my device list as well as the URL of my Octoblu trigger. I’d like to, at the very least, make the device list dynamically generated from the equivalent configuration inside my home MQTT environment.

What I’ve built here is basically an Alexa to MQTT bridge. This means I’m not limited to 1-to-1 control of individual lights. With the right MQTT messages in the device discovery JSON blob I could also control groups of lights, timed sequences, or anything else I care to code up.