A couple of weeks ago there was a lot of interest on social media in a video showing a Big Mouth Billy Bass animatronic novelty device seemingly take on the role of Amazon Echo, including animating its mouth and body in sync with the speech. With my recent exploits in connecting strange crap to Octoblu I decided to have a go at automating and Internet-connecting Billy Bass.
In this three-part blog series I’ll cover:
- (This part) – reverse-engineering Billy Bass and automating its movements based on audio input
- Connecting Billy Bass to the Octoblu IoT automation platform and synthesising speech for it
- Controlling Billy Bass with Amazon Echo and Slack
Reverse-engineering Billy Bass
Looking around the web suggests there are at least a couple of revisions of the Billy Bass product; for my hack I bought a brand new instance (yet another device to have its warranty voided within minutes of getting it home!) – the details here apply to this revision, YMMV with older variants.
Accessing the internals was easy – six screws on the back allow the back panel to be separated from the main part of the device. The front part of the device houses the mechanical parts that move the fish’s mouth, head and tail; a push button and light sensor to activate the device; and the main circuit board. The rear panel contains the battery box (four AA batteries with a 4.5V connection as well as the primary 6V one); a switch to select whether the device is button- or motion-activated; and the loudspeaker. The two halves are connected by a small wiring bundle that plugs into the main board.
The device is controlled by two 6V motors: one for the mouth (hidden inside the body of the fish) and one for the head and tail (easily visible and accessible from inside the case). In all cases the motors drive the given body part against a spring return, therefore power is applied to move, and hold, the body part, then power is removed to allow the body part to spring back. Partial power is used to slow the advance or return. The practice of using power against springs to hold the body part in the extended position means that the motor is stalled – I measured the stall current at about 0.8A for each motor. This is particularly interesting for this battery-powered device that spends quite a bit of its active time with the head at full deflection, and thus with the head motor stalled and therefore consuming 0.8A continuously.
The mouth is easy to use – apply power to open and stay open, remove power to close. The head and tail are more complicated: they are driven by a single motor with one direction moving the tail and one moving the head; both using a spring return. This means that it is not possible to move the head and tail at the same time, or have both in their extended position simultaneously.
The mouth motor is connected with the grey (positive) and red (negative) wires, and the head/tail motor is connected with the black and orange wires with black being positive to move the head and orange being positive to move the tail.
Using a GoPro and watching the video back frame-by-frame gave the following approximate timings for deflection and spring return of each body part (at the full 6V drive):
- Mouth open 0.25s, spring closed 0.2s
- Head move 0.4s, spring back 0.35s
- Tail move 0.25s, spring back 0.2s
The unit has a single circuit board consisting of an epoxy-encapsulated part (probably an ASIC or microcontroller), a set of transistors for motor drive, and other components associated with the light sensors and so on. It is the motor drive that’s of interest here.
The mouth motor is driven using a basic single NPN transistor drive circuit. The head/tail motor is driven by a classic H-bridge driver consisting of two D882 NPN, and two D772 PNP transistors. Two of the four control inputs are wired back into the bridge to simplify the manner in which the bridge is used: there are two inputs from the device’s control logic – one to drive the head and one to drive the tail. (It is absolutely important not to drive both of these at the same time, otherwise you’ll end up with all four transistors connecting 6V directly to ground and the inevitable failure will ensue. This is easier to do by accident than you might think, especially if you connect these to an Arduino before programming it where the pull-up resistors can provide a positive drive signal.)
The following diagram shows the motor drive circuit with PCB annotations.
Interfacing Billy Bass with an Arduino
The reverse-engineering established that Billy Bass’s motors are driven by three analogue signals, one for each of the mouth, head and tail. I chose to drive these with an Arduino Uno R3 using the PWM quasi-analogue outputs. The first task was to connect up the Arduino. Luckily the 100 ohm base resistors provided a simple solution – by removing these from the PCB I was presented with ready-made solder pads to which I could connect wires. Removing these surface-mount devices with a soldering iron was easy because of their small size, smaller than the face of the tip of the soldering iron bit. For each of the three drive connections I connected a wire to the PCB, via an in-line 100 ohm resistor, to a PWM pin of the Arduino. I also connected the other solder pads (the side of the original resistor connected to the Billy Bass device’s encapsulated logic) to three of the Arduino’s digital input pins – this allows for reading what the fish’s logic wants the motors to do so I can preserve the original Billy Bass behaviour in addition to adding my own driving functions.
The PWM approach gave me mixed success with anything less than full power (100% duty cycle PWM) leading to a screaming noise from the motors – this is most likely because the Arduino PWM frequency is sufficiently low, and I have no electronic smoothing of the signal, and therefore the motors are seeing the constant on-and-off of the power. A to-do item is to add low-pass filters to the PWM output to turn them into something more akin to true analogue outputs. For now I’m driving the motors at either full power or completely off.
Controlling the motors and hence the body movements of the fish is now simply a matter of writing 255 or 0 to the relevant analogue output pin. In the Arduino sketch I wrapped these pin writes in functions to firstly ensure that both head and tail are never activated at the same time (that’ll lead to a melt-down of the drive transistors) and to add a auto-off timeout to each motor to ensure it doesn’t stay activated permanently.
In common with my other Arduino-interfaced projects I added a simple text-based USB-serial command protocol to control the motors (e.g. sending “MOUTH OPEN” will activate the mouth motor) – see the code for more on this. This interface will be used by the Octoblu connector described in the second part of this blog series. I also added a pass-through mode which samples the signals coming from the Billy Bass logic and drives the motors with those values – i.e. emulating the out-of-the-box operation of the singing fish.
My plan was to control head and tail movements by explicit commands over the USB-serial connection. The mouth was to be controlled implicitly by the audio speech that the device is emitting. This decision does limit the fidelity of the mouth movement (if you think about it, mouth movement usually starts before any sound is emitted, thus anything that is driven from audio will seem to lag – what we’ll end up with will look move like a sock puppet style of mouth motion rather than something accurate, however for a novelty plastic fish this is probably sufficient), but it does mean that no external pre-processing is necessary and the interface to the device is simple: send it audio and it’ll move in a roughly synchronised manner.
The primary input to the Arduino therefore becomes the input audio signal. I’m using a line/headphone level input so that I can later connect the device to a Raspberry Pi’s audio output. The input uses a basic DC bias circuit with an input isolation capacitor as per the diagram below. Some trial and error was needed to establish the range of input values corresponding to peaks and troughs in speech. Ideally an adjustable pre-amp would be used here to allow tuning to different input amplitudes, however for now the output from a Raspberry Pi headphone socket gives a swing of plus/minus 40 from an at-rest value of around 512 on the Arduino’s 0-1023 analogue-to-digital convertor scale. For this simple hack I’m assuming any swing greater than 10 be an active syllable.
The lip sync code is really simple (and an area where improvement is definitely possible!) – the input signal is sampled on every loop of the Arduino sketch’s main loop. After 100 samples of greater than 10 units (either side of the at-rest value) the syllable is determined to have started and the mouth is opened; after 100 samples below 10 units of deviation the syllable is determined to have ended and the mouth is released to spring back. The effect is OK, but not great. I suspect, based on my sock puppet analogy, it may work better if the sense is reversed, i.e. the mouth is closed as a syllable is spoken, however this requires that the next syllable is anticipated so that the mouth can be opened in preparation.
Using the loudspeaker
So to recap: we now have motor control for the Billy Bass, with USB-serial commands for the head and tail, and automatic control of the mouth based on the audio being sent to the device via a line-level input. The final piece here is to connect that audio input to the Billy Bass speaker to ensure the speech sounds like it’s coming from the fish.
My initial hope was to use the existing electronics in the device to act as an audio amplifier, however unlike the motor driver which was easily accessible, the speaker is connected directly into the epoxy-encased blob on the PCB and hence I don’t have an easy way to inject audio into the amplifier part of the circuit. Instead I built a small amplifier based on a MC34119 IC I happened to have around, connected to the existing speaker in the fish. However, somewhere along the line I managed to kill the speaker in an apparently unrecoverable manner.
The quick fix was to grab a cheap set of PC speakers from the local PC store and rip out both a speaker and the amplifier board (another device whose warranty was voided within minutes of getting it home!). The amp is powered from the “Vin” pin on the Arduino, which in turn is powered by USB from the Raspberry Pi to which it is connected. I simply connected the audio input to the Arduino (see above) to the audio input to the amplifier such that the same input drives both. Other than that the audio amplifier works in exactly the same way as it was designed to, albeit outside of its plastic case and with one of the stereo channels only (I’m going to drive this all with a mono waveform anyway so it manners little).
Putting it all together
My goal is to have the extra pieces of electronics, the Arduino, audio circuit and audio amplifier, all sit inside the Billy Bass case so it looks like it did before I started hacking it. Although there is quite a bit of spare space inside the case the large sizes of the Arduino and the replacement loudspeaker mean this is not currently possible. On my to-do list I have a task to get an Arduino mini and a smaller, mylar speaker to make everything fit. For now I’ve just added a couple of brackets to let the back panel stand-off from the front panel by an inch or so – so at least the device looks right from the front.
The end result then is a Big Mouth Billy Bass that has two cables coming out of it: a USB cable which provides power for everything except the motors (that still comes from the batteries – PSU to be added later) and takes motor movement commands over USB-serial, primarily for the head and tail movements; and a 3.5mm audio cable that plugs into any reasonable headphone or line-out socket, which supplies the audio which is played from the fish’s loudspeaker and which moves the mouth via the lip sync functionality.
The second part of this blog series will look at how I connected this to the Octoblu IoT automation platform and used a text-to-speech web service to make it an Internet-connected speech device. In the third part I’ll show some examples of using Octoblu and the fish with other Internet-connected devices and services.