I Built a Voice-Based Daily Reflection Companion under 15 Minutes Using Agora Agents SDK

I expected to spend a weekend on working on my vision. Instead, I had a voice agent asking me "How was your day?" in under 15 minutes. What surprised me most wasn't the speed - it was that when I interrupted the voice AI agent I named "Compass" mid-sentence to change my answer, she just stopped and listened. No stuttering, no doubled audio, no ghost speech finishing in the background. It just worked, out of the box, without a single line of interruption-handling code on my end. If you've tried to build a voice agent from scratch, you know the pipeline looks deceptively simple on a whiteboard: speech in, text to LLM, speech out. The reality is messier. You need a WebRTC or WebSocket layer to stream audio in real time. You need to integrate STT (and handle partial transcripts), stream tokens to TTS, manage token refresh, handle network retries, detect when the user starts speaking mid-sentence, and somehow prevent the agent from continuing its TTS output while the user is already replying. That's before you write a single line of actual product logic. The Agora Agents SDK removes every item on that list. It's built on top of Agora's existing RTC infrastructure, which is the same real-time communications network that powers video calling for hundreds of millions of users. The SDK wraps that into a Python (or TypeScript or Go) library where you describe what your agent should do, not how the audio pipeline should work. Daily Reflection Companion Backend: Python + FastAPI Agent pipeline: Deepgram STT → OpenAI GPT-4o-mini → MiniMax TTS Frontend: HTML/JS + Agora Web SDK (no build tooling) Install pip install agora-agents fastapi uvicorn[standard] python-dotenv openai That's it. The SDK installs in under 10 seconds. Imports are under agora_agent (note: no hyphen at import time). Credentials You'll Need Create a free account @console.agora.io Create a project, and enable the Conversational AI feature. You'll get an App ID and an App Certificate. You also need an OpenAI API key (for the LLM and for generating the post-session summary) and DeepGram API key Copy your .env.example to .env: AGORA_APP_ID=your_app_id AGORA_APP_CERTIFICATE=your_certificate OPENAI_API_KEY=sk-… DEEPGRAM_API_KEY=your_deepgram_key Note: The Agora console's Conversational AI toggle is not prominently labelled. I spent about 4 minutes finding it - it lives under Project Settings → Features. Once enabled, everything worked on the first try. The builder chain agent.py: from agora_agent import ( Agent, Agora, Area, DeepgramSTT, OpenAI as AgoraOpenAI, MiniMaxTTS, expires_in_hours, ) client = Agora(area=Area.US, app_id=app_id, app_certificate=app_certificate) agent = ( Agent(client=client, turn_detection={"language": "en-US"}) .with_stt(DeepgramSTT(model="nova-3", language="en")) .with_llm( AgoraOpenAI( model="gpt-4o-mini", system_messages=[{"role": "system", "content": REFLECTION_SYSTEM_PROMPT}], greeting_message="Hello! I'm Compass. How was your day today?", failure_message="I didn't quite catch that. Could you say that again?", max_history=50, params={"max_tokens": 150, "temperature": 0.75}, ) ) .with_tts(MiniMaxTTS(model="speech_2_6_turbo", voice_id="English_captivating_female1")) ) Every line is intentional: turn_detection={"language": "en-US"} - tells the VAD (Voice Activity Detection) which language's speech patterns to use for end-of-turn detection. This directly affects how quickly the agent recognises you've finished speaking. max_history=50 - the agent's LLM receives up to 50 turns of conversation as context. Critical for a reflection agent that needs to remember what was said early in the session. max_tokens=150 - voice replies need to be short. Capping at 150 tokens enforces this at the model level, not just the prompt. greeting_message - the agent speaks this immediately when the session starts, without waiting for the user to speak first. No extra session.say() call needed. The reflection flow is driven entirely by the LLM system prompt. No explicit state machine, no conditional logic in the server code. The prompt instructs the agent to move through five phases naturally: PHASE 1 - DAILY CHECK-IN: Ask one meaningful follow-up after the user's response. PHASE 2 - MEANINGFUL MOMENT: Explore one significant experience. PHASE 3 - GRATITUDE: Ask for 2–3 things they're thankful for. PHASE 4 - TOMORROW'S INTENTION: Ask for one thing they'd like to carry forward. PHASE 5 - CLOSING: Offer a warm, personalised goodbye. VOICE RULES: Keep all responses under 35 words. One question per turn. No markdown. The "under 35 words" constraint was the most important prompt engineering decision. Voice synthesis doesn't render markdown, and long replies feel like lectures, not conversations. Starting a Session session = agent.create_session( channel=f"reflection-{int(time.time())}", agent_uid="999", remote_uids=["*"], idle_timeout=90, expires_in=expires_in_hours(1), ) agent_id = session.start() session.start() is a single blocking call that provisions the agent, connects it to the RTC channel, and returns an agent_id. The channel name is how the browser's Agora Web SDK joins the same audio room. After the conversation ends, I call session.get_history() to retrieve the transcript and send it to GPT-4o-mini with a structured summary prompt: history = session.get_history() # Format turns into a readable transcript, then: response = openai_client.chat.completions.create( model="gpt-4o-mini", messages=[ {"role": "system", "content": SUMMARY_PROMPT}, {"role": "user", "content": f"Transcript:\n\n{transcript}"}, ], max_tokens=200, ) return response.choices[0].message.content The summary appears on-screen after the session ends - a written record of what you reflected on. This is the feature that makes this more than a demo. The browser uses the Agora Web SDK (loaded from CDN - no npm, no webpack) to join the same RTC channel: // Create client and join rtcClient = AgoraRTC.createClient({ mode: "rtc", codec: "vp8" }); await rtcClient.join(app_id, channel, token || null, uid); // Publish microphone localMicTrack = await AgoraRTC.createMicrophoneAudioTrack({ encoderConfig: "speech_standard", }); await rtcClient.publish(localMicTrack); // Subscribe to agent's audio and play it rtcClient.on("user-published", async (user, mediaType) => { if (mediaType !== "audio") return; await rtcClient.subscribe(user, "audio"); user.audioTrack.play(); }); Three async calls: join, publish, subscribe. That's the entire WebRTC layer. Once the pipeline runs, the question that matters is: does it feel like a real conversation? Turn-taking Interruption test Conversation quality Built with Agora Agents SDK (Python) + OpenAI GPT-4o-mini + Deepgram STT + MiniMax TTS What's genuinely great session.think() (not demonstrated in this project's MVP but available) is a powerful primitive: you can inject mid-session instructions into the LLM without the agent speaking them aloud. For a more sophisticated reflection agent, this could be used to nudge the agent toward a specific phase based on time elapsed. What could be better The gap between quickstart and production. **The CLI quickstart (agora init) scaffolds a working app fast, but the gap from that template to understanding *why each piece exists is steep. Better intermediate documentation (not just API reference, not just quickstart) would help. **get_history() response format. **The method exists and works, but the response shape isn't clearly documented. I had to handle three possible formats defensively. This is the kind of thing that adds 30 minutes to an otherwise 5-minute task. Error specificity. When I accidentally misconfigured my App Certificate, the error was a generic 401. A message like "App Certificate mismatch - check AGORA_APP_CERTIFICATE in your environment" would have saved 10 minutes of debugging. These are fixable problems, not fundamental ones. The core pipeline is rock-solid. I started this build expecting to spend most of my time on infrastructure. Instead, I spent most of it on the part that matters - the conversation design, the system prompt, the reflection flow. That's the correct trade-off, and the SDK made it possible. The Agora Agents SDK doesn't replace the REST API - it's built on top of it, and REST stays fully supported. What it does is remove the RTC plumbing so you can focus on what your agent should say and feel like, not on how audio bytes travel between browser and model. For a real-time voice product where interruption, latency, and audio quality matter, the infrastructure Agora provides is serious. For a developer who wants to build that product in an afternoon rather than a week, this SDK is the fastest path I've found. pip install agora-agents fastapi uvicorn[standard] python-dotenv openai cp .env.example .env # Fill in your API keys uvicorn main:app - reload # Open http://localhost:8000 GitHub (Python SDK): https://github.com/AgoraIO/agora-agents-python #VoiceAI #AIagents #Agora #ConversationalAI #ConvoAI #OpenAI #Agora #TTS #STT