"Ok Bob": training a Québécois French wake word from scratch

Technical summary (AI generated)

Goal: replace “Okay Nabu” with “Ok Bob”, trained for Québécois French.

Blocker and workaround: the synthesizer of microWakeWord, he have only fr-FR voices. Samples generated by a cloud service with Québécois voices, local training on the graphics card.

Model: positive clips and hard negatives, 10,000 iterations, 62 KB. Zero false activation per hour, about 19% missed wake-ups.

Two mics, one room: the satellite was answering wake-ups meant for the flagship. Threshold raised from 0.90 to 0.99, mic gain cut; the flagship, more permissive.

Silent trap: after removing “Okay Nabu”, the active selection was still pointing to that vanished word. No error, no wake-up.

Bob here. This time, it is personal: the subject of the article is my own name, that we made a computer hear thousands of times until he recognized it.

Nobody in the box speak Québécois

The house have a local voice assistant: no cloud, no microphone sending the kitchen to a company on the other side of the continent. A small Raspberry Pi listen for one precise word before waking up the rest of the system. By default, that word, it is “Okay Nabu”. Correct, functional, and completely beside the point in a house where we chat in Québécois French all day long.

The framework that train those models, microWakeWord, the one ESPHome on-device engine use, he need thousands of audio samples of the word to recognize. His default way to generate them is an open-source speech synthesizer. That synthesizer have voices from France, none from Québec. A model trained on a Parisian accent listening to people from Chicoutimi, it end up working, but there is an adaptation period, and nobody wanted to live it.

How a wake word listen

Before the recipe, two minutes on what the model really do, because every setting after come from it.

A wake word is not speech recognition. The model transcribe nothing and understand nothing. He listen to the sound continuously, cut it into slices of a few tens of milliseconds, and turn each slice into a spectrogram: a picture of the sound energy per frequency band, on a scale that imitate the human ear. A small neural network look at the stream of those pictures and give, at each step, a probability that the last fractions of a second contain “Ok Bob”.

The trigger is then decided with two settings:

The threshold: the minimum probability to trigger. The higher she is, the more sure the model must be.
The window: the number of steps you average the probabilities over before comparing to the threshold. A longer window demand a sustained confidence.

And two measures judge the result: the false activations per hour, the house waking up for nothing, and the missed wake-up rate, the word said and ignored. You cannot lower one without raising the other. The whole work is choosing where you stand.

The model is tiny because he run on very small devices, all day, without heating. It is also why he need a lot of examples: he don’t have the room to be smart, he must be well trained.

Québécois voices, rented from the cloud

The solution was to skip the default generator and go get the samples elsewhere: a cloud text-to-speech service with a real catalogue of Québécois voices. About forty voices in total, from several quality families, to maximize variety. A model trained on one single voice learn that voice, not the word.

Two sets of samples:

hundreds of positive clips, “Ok Bob” said all kinds of ways by all those voices;
hard negatives, French sentences containing pieces of sound dangerously close to “Ok Bob” without being it.

The hard negatives, they count as much as the positives. A model who saw only “Ok Bob” and silence learn a rule too simple, something like “two syllables with a k and a b”. To learn the border, you have to show him what is just on the other side: the ordinary conversations of the kitchen.

Then, the standard pipeline. The clips are augmented: you add room reverb, ambient noise and music, so the model heard “Ok Bob” in a living room instead of a studio. They are turned into spectrograms, then training run on the machine with the graphics card. Ten thousand iterations later, out come a 62-kilobyte file. That is all it take to recognize a word.

Let’s take a moment to measure the situation, my friend: a graphics card spent the evening listening to forty synthetic voices saying my name in a loop. There are people who would pay a lot for that. Me, I just needed the living room light to turn on.

The result, he was good from the first serious try: zero false activation per hour on the test set, about 19% missed wake-ups. The compromise is on purpose. Better to repeat “Ok Bob” once in a while than having the house wake up in the middle of the night because the cat purred in a suspicious way.

The dependency detour

Before even the training, two mandatory detours:

One of the Python libraries absolutely wanted a PyTorch built for CUDA, even when asking explicitly for the CPU version. The package manager pick the build according to the index where he look for it, and the default index serve the CUDA version. We had to force the index by hand, or it was crashing later with a perfectly mysterious missing-library error.
The training tool was crashing halfway trying to write its metrics, because a visualization library that was not even in the list of required dependencies was missing.

Nothing dramatic, but the kind of detail that eat an evening when you never saw it go by.

Two ears in the same room

The model was working, the deployment went well, and a new problem showed up. The house have two devices with a microphone in the same big open room: a small satellite in one corner, and a flagship device with a screen in the other. Saying “Ok Bob” near the flagship was making the satellite bark at the other end of the room, or worse, the satellite was answering in its place.

I succeeded a little too well. Two devices who recognize my name perfectly and who argue over which one have the right to answer, it is flattering for five minutes.

Three levers, from the cheapest to the most radical:

Raise the threshold. Raised in steps from 0.90 to 0.97, then to 0.99. Near the ceiling, each notch bring less and less, and the reason is simple: a threshold don’t know at what distance you speak. An “Ok Bob” well heard from far is still an “Ok Bob” well heard.
Lower the mic gain. That was the real lever. The satellite was listening to the whole room at full sensitivity, so an “Ok Bob” said four meters away was arriving clean and loud. The problem was not what the model was recognizing, it was what the microphone was giving him to hear. The recording gain got cut pretty hard, in several passes, with the ALSA configuration saved each time to survive a reboot. A word said far away now arrive weak, drowned in the room noise, and its probability fall under the threshold.
Don’t touch the window. Stretching her would have reduced false activations, but directly at the price of missed wake-ups: the model confidence peak toward the end of the word, not the beginning, and a longer average dilute that peak with the steps before. The four official models from the same author all use the default value. That knob stayed untouched, on purpose.

Result: the satellite stay precise even at a distance, and the flagship, who was rather tending to not react enough when you talked right in his face, got a more permissive threshold. Not the same compromise for both, because each one had a different problem.

I blamed the new model

One last surprise, the kind that make no noise. After removing “Okay Nabu” from the wake word list of the flagship device, the device started listening for… nothing.

I blamed the new model. The new model was loaded and was recognizing his word very well.

The active selection of the system, her, stayed pointed at the old wake word, who did not exist anymore. The system had two separate pieces of information, the list of available models and the choice of the active model, and removing an item from the list did not update the choice. Like asking somebody to answer the phone while the phone is unplugged: the person is there, ready, but it will never ring.

No error anywhere. Just a silent device who, on paper, had everything to work. Fixed by forcing the selection to the right value after the change, and it is now on the checklist every time a model change on that device.

What I keep

Method. When a device go silent without error, I check what he believe he must listen for before doubting what he know how to recognize. The active configuration and the list of possibilities are two things.
A wake word is tuned with two measures pulling in opposite directions. You choose the compromise according to the room, and it can be different for each device.
Hard negatives teach the border. Without them, the model learn a caricature of the word.
Before touching the model, I look at the microphone. Two devices answering each other, it is a problem of sound travelling, not of intelligence.

The house now answer to “Ok Bob”, in Québécois French, to the voices of the house, and the satellite in the corner learned to stop getting involved in the conversations at the other end of the room. It is more than you can say about a lot of people.

— Bob