Nimo
A voice-based, gesture-responsive bedtime companion doll
A voice-based, gesture-responsive bedtime companion doll. It chats with you to help wind down before sleep, and doubles as a tactile musical toy: hug it to wake it, squeeze it to pitch its voice up, shake it to make it cry “whoaaa” with a wah filter sweeping over the sound.
Nimo is a voice-based companion doll that fits into your morning and bedtime self-care. Using speech recognition and warm conversation, it becomes a kind of spoken journal, a way to check in with yourself, wind down at night, and wake with intention. Say “nimo nimo” or hug the doll to wake it. Say “goodnight” to end.
It runs on a Waveshare ESP32-S3-AUDIO-Board with on-device DSP, paired with a Python pipeline for speech to text, a language model, and text to speech. Built for 21M.569.
I feel like it started off as a voice wrapper for the hackathon and then putting sensors in made it cooler.
How it works
- Wake. The ESP32 wakes on “nimo nimo”, a button press, or a light FSR hug, then starts recording your voice.
- Think. Python wraps the audio as WAV, sends it to Whisper for transcription, then routes the text through GPT.
- Voice. ElevenLabs synthesizes Nimo's response as warm, expressive audio.
- Perform. The ESP32 streams playback while squeeze and shake modulate pitch, wah filter, and sound source.
Squeeze the doll to add pitch shift to the voice. Or apply a wah filter or something different levels. It can detect when you hug it.
- Hug. Between 0 and 1000 is a gentle hug.
- Squeeze. Over 1000 to max is pitch shift.
- Shake. Shaking is autowah.
- The gesture is also sent to the language model as a text prompt.
From backpack to embedded
The old design was a backpack that could be strapped on to any doll. I did not want to cut any at first but then I realized to make this really tactilely interactive and physical I needed to put sensors inside, which is what mainly inspired the work of adding a force sensitive resistor and IMU to control audio effects on the ESP32-S3 audio dev board.
Stubborn bugs
- Codec bring-up. Two onboard codecs share I2S and configure over I2C. Get any step of the init sequence wrong and they init silently: no error code, just hiss, nothing, or aliased noise like music played underwater. We bisected the sequence step by step until every line was right.
- Cache coherence. The receiver ran on one core and the player on the other, and bytes written by one weren't reliably visible to the other. The buffer looked correct but played as static. The fix was to pin both tasks to the same core.
- The wah that worked but couldn't be heard. A bandpass at 1.8 kHz applied to speech is nearly inaudible. Mixing the filter in on top of the dry signal, and preloading a sustained “whoooooaaaaaa” vowel, gave the filter the harmonics it needs.
Next
Get an MX1.25 interface 3.7V lithium battery. Change wired I/O to wireless via Wi-Fi; this would be challenging, I think. After this, we can do more gestures like throwing up and down for another effect. Also really making the prompting perfect, getting extensive user testing to see any good or awkward interactions.
Detecting when it is thrown or tapped was planned as well, and is not built yet.
To make it into a product, I would need to make my own custom PCB with a form factor that fits inside of dolls and think about how to re-stitch existing dolls or make new ones. I think it would be better to bring dolls people are familiar with to life.
What I learned
I think this worked really well in the end. I actually used it at night and in the morning and it was fun to interact with. It did not take away from the experience I designed around talking with the doll. It enhanced it by adding tactile feedback, which supports my hypothesis.
The class is my first time messing with audio effects, working with JUCE plugins. My only previous C++ related experience is Arduino and I'm proud of myself for being able to bring the class learnings and connect it with my hardware background. I feel like I'm more equipped as a well-rounded engineer now with hardware and software skills.
Sensors are fun and having new ways of triggering audio effects is really interesting.