Five versions of the voice, what went wrong in each, and how I fixed it. For the short version, see the main Voice Agent page.
| Version | What I built | What went wrong | Result |
|---|---|---|---|
| 1. Record, transcribe, answer, speak | A relay on my app builder's built-in AI | The microphone switched off while the guide spoke, so nobody could interrupt it. Some recordings couldn't be read, and every turn used build credits | Partly worked |
| 2. The browser's free speech tools | Listening and speaking in the browser, with only the answer written on the server | A robotic voice that ended my turn after about 0.35 seconds of silence. Interrupting did nothing | Didn't work |
| 3. Gemini's free tier plus my own interruption detection | A more natural voice, and a rule to stop talking when the microphone hears something clearly louder | The free tier was often busy. The backup apology played in the browser voice, the microphone heard it as the learner, and the guide kept interrupting itself | Didn't work |
| 4. Claude writes the answers | Claude as the same brain for chat and voice, with retries and lighter backup models | At first every answer began with "Say in a warm, friendly, conversational tone…", because the style note was sent to the voice as text to read. Once fixed, answers were better but still slow | Partly worked |
| 5. A real-time voice agent | Deepgram's Voice Agent for listening and speaking in one live connection, with Claude writing the answers. Version 4 stays as the backup | Answers start quickly, the voice sounds natural, and it stops as soon as you speak | Live |
I also prototyped a speech-to-speech version on Gemini Live and shelved it, because it depended on the same busy free tier.
| What I noticed | Cause |
|---|---|
| Slow answers, and being cut off | Each turn went through four steps in a row, with a fixed silence timer at the start |
| A robotic voice appearing mid-conversation | The free tier failing and the browser voice filling in |
| Couldn't interrupt, and apologising in a loop | Echo: my detection couldn't tell the learner from the guide's own voice |
| A flickering orb | The animation restarting on every state change |
Only the orb was a bug I could simply fix. The rest came from how the pipeline was built.
| Option | Cost for a demo | Decision |
|---|---|---|
| Deepgram Voice Agent | $200 free starter credit, no card, Claude available inside it | Chosen |
| OpenAI Realtime | Paid only | No |
| ElevenLabs Agents | 15 free minutes a month, then paid, with the language model billed separately | No |
| Gemini Live | Free, but on the same busy free tier | Shelved |
flowchart LR
subgraph Before["Before: a relay of four steps"]
A["Record until 0.7 s of silence"] --> B["Transcribe"] --> C["Claude answers"] --> D["Speak whole reply"] --> E["Play"]
end
subgraph After["After: one live connection"]
M["Microphone streams"] --> G["Deepgram Voice Agent<br>listens, Claude thinks, speaks"] --> S["Speaker starts while writing"]
G -. "learner starts speaking: playback stops" .-> S
end
| The relay (versions 1 to 4) | Real-time agent (version 5) | |
|---|---|---|
| When the reply starts | After every step has finished | While the answer is still being written |
| Knowing the learner has stopped | A fixed silence timer | The service detects the end of the turn |
| Interrupting | My own loudness rules, easily fooled by echo | The service reports when the learner starts speaking |
| Points of failure | Four, two of them on a busy free tier | One, with the relay as a backup |
When the backup voice can't understand someone, it still says "Sorry, I missed that" in the robotic browser voice. It doesn't happen often now, but it breaks the guide's character.