Building Bharat Buddy: My 10-Day Voice AI Journey with Murf Falcon

What if learning didn't require typing? What if a student could simply speak to an AI tutor in Hindi, English, or Hinglish, ask questions naturally, practice problems, and even get connected to a specialist when needed? That was the idea behind Bharat Buddy, the voice agent I built during 10 Days of Voice Agents — VoiceForBharat Edition. Over these 10 days, I went from a basic voice assistant to an agent that can remember users, use tools, make outbound calls, escalate problems to humans, track call analytics, and hand conversations over to specialist agents. This post is about what I built, what went wrong, what I learned, and how someone else can start building their own voice agent. 🚀 What is Bharat Buddy? Bharat Buddy is an AI-powered voice tutor designed for students. The goal was to make learning feel more like talking to a helpful teacher rather than interacting with a traditional chatbot. A student can speak naturally: "Mujhe percentage samajh nahi aa raha." or: "Can you explain this maths problem?" or even: "Bhai, mujhe Hinglish mein samjha." Bharat Buddy can respond conversationally and adapt to the user's language. The project focuses on: 🇮🇳 Hindi The idea was simple: Make AI learning more natural, accessible and conversational. 🎯 Why Voice? Most AI applications still expect users to: Type → Wait → Read → Type again But speaking is much more natural. For a student, saying: "What is 20 percent of 500?" is much easier than typing a detailed prompt. Voice also makes the interaction feel more human. Instead of staring at a chatbot, you can actually talk to your tutor. That's what I wanted Bharat Buddy to feel like. 🏗️ How Bharat Buddy Works The basic architecture looks like this: USER │ │ Voice ▼ ┌──────────────────┐ │ Speech-to-Text │ └────────┬─────────┘ │ ▼ ┌──────────────────┐ │ LLM │ │ Bharat Buddy │ └────────┬─────────┘ │ ┌─────────┼──────────┐ │ │ │ ▼ ▼ ▼ Memory Tools Specialist Agent │ │ │ └─────────┼──────────┘ │ ▼ ┌──────────────────┐ │ Text-to-Speech │ │ Murf Falcon │ └────────┬─────────┘ │ ▼ USER The project uses LiveKit for real-time voice communication, an LLM for reasoning, and Murf Falcon for voice generation. Murf AI 🎙️ Murf Falcon One of the most important parts of the project was making the AI actually sound conversational. I used Murf Falcon, the fastest TTS API mentioned in the challenge, to generate the agent's voice. This made the interaction much more natural than simply displaying text. The agent can speak responses such as: "Namaste! I'm Bharat Buddy. What would you like to learn today?" and continue the conversation naturally. The voice layer became a major part of the experience rather than just an additional feature. 🧠 Memory A useful assistant shouldn't completely forget you after every conversation. So I added memory capabilities to Bharat Buddy. This allows the system to maintain useful user context and make future conversations more meaningful. The important lesson here was that memory should be useful, not excessive. A voice agent doesn't need to remember every single sentence. It should remember information that actually improves future interactions. 🛠️ Tools Another important part of the project was giving the agent access to tools. Instead of making the LLM pretend it knows everything, tools allow the agent to perform specific actions. For example: User This makes the agent more reliable and gives it capabilities beyond simple conversation. 📞 Outbound Calls Bharat Buddy also gained the ability to make outbound calls. This was one of the more interesting parts of the challenge because the agent moved beyond: "User calls AI" to: "AI can initiate a call." This opened up possibilities such as: Follow-ups 🧑‍💼 Human Escalation One of the biggest lessons I learned was: An AI agent should know when it should stop being the expert. So I added a human escalation system. For example, if the user needs help that the agent cannot safely handle, the agent can create a request for a human. The workflow became: User needs human help The escalation request contains only useful information rather than dumping the entire conversation. This was especially important from a privacy perspective. 📊 Call Analytics Dashboard On Day 8, I built a simple analytics system. The dashboard tracks: Total Calls How many calls have been recorded. Successful Calls Calls where the defined success condition was achieved. Failed Calls Calls where the success condition wasn't reached. The data is stored using SQLite and exposed through an API that the dashboard can consume. For example: { "total": 7, The important part is that these aren't hardcoded numbers. They come from actual agent activity. 🧮 Maths Practice Specialist One of my favourite features was adding a specialist agent. Instead of making Bharat Buddy an expert at everything, I created a separate: Maths Practice Specialist Its job is focused specifically on: Arithmetic The flow looks like: Student The user doesn't have to explain the entire problem again. This taught me an important architectural concept: A good AI system doesn't necessarily need one giant agent. Multiple focused agents can often be easier to control. 🌐 Hindi + English + Hinglish Another important part of Bharat Buddy was language flexibility. The agent is designed to respond according to the user's language: English → English Hindi → Hindi Hinglish → Hinglish For example: User: "Percentage ka concept simple language mein samjhao." Bharat Buddy can respond naturally in Hinglish rather than suddenly switching to formal English. For an India-focused voice assistant, this makes a huge difference. 🎨 Frontend The frontend provides the interface through which users interact with the voice agent. The goal wasn't to create an unnecessarily complicated UI. It focuses on showing the agent's state and making the voice interaction obvious. The overall experience is: Open Agent Honestly, the project didn't work perfectly on the first try. One of the most frustrating problems happened while implementing the specialist handoff. I initially received an error related to the tool schema: invalid JSON schema for tool After fixing that, another problem appeared: TypeError: The model was trying to call the handoff function, but the function definition and the arguments expected by the Python function didn't match. The important lesson: When using function calling, the tool schema and the actual Python function signature must agree. For example, if the function expects: async def handoff_to_maths_specialist(reason: str): then the model needs to provide a reason argument. This sounds small, but it can completely break the voice pipeline. 🔐 Another Important Lesson: Secrets While building the project, I also learned how important it is to keep API credentials outside the source code. API keys should be stored in environment variables such as: .env.local and should never be committed to GitHub. My .gitignore contains rules for environment files and database files so that sensitive/local data isn't accidentally uploaded. This is especially important when publishing an AI project publicly. 🧰 Tech Stack The main technologies used in Bharat Buddy include: Technology Purpose If you want to experiment with the project, start by cloning the repository: Bharat Buddy — GitHub Repository / murf-livekit-starter 🇮🇳 Bharat Buddy — AI Voice Agent A multilingual AI voice assistant built during 10 Days of Voice Agents — VoiceForBharat Edition, powered by Murf Falcon and LiveKit. Bharat Buddy is a real-time AI voice agent designed to make learning and everyday assistance more natural through voice. It can understand and respond in English, Hindi and Hinglish, remember returning users, use tools, make outbound calls, escalate problems to humans, track call analytics and hand conversations to specialist agents. 🎯 The Problem Many users are more comfortable speaking than typing, especially when using technology in their everyday language. Bharat Buddy explores how a voice-first AI assistant can make digital learning and assistance more accessible by allowing users to simply speak naturally instead of typing everything. The project focuses on: Natural voice conversations Indian languages and Hinglish Learning assistance Human escalation when AI needs help Specialist agents for focused… View on GitHub Then create your environment and install the project dependencies. For example: git clone https://github.com/Rishabhpandey2007/murf-livekit-starter.git cd murf-livekit-starter cd backend uv sync Create your local environment file: .env.local Add the required API credentials there. Do not put your real API keys directly into Python files or commit .env.local to GitHub. Then start the agent according to the project instructions. Once the agent is running, open the browser client, connect to the voice session, and start speaking. 🧪 Testing the Agent I tested different paths during development. Normal conversation Bharat Buddy: Bharat Buddy: Maths Specialist: Before this challenge, I mostly thought about AI assistants as: Input → LLM → Output After building Bharat Buddy, I realized a production-style voice agent is much more than that. You need to think about: Real-time audio The LLM is only one component. The real engineering challenge is making all the components work together reliably. 🔮 What I Would Build Next Bharat Buddy is still a work in progress. Some things I'd like to add next: 📱 Better mobile experience Make the voice experience easier to use on low-end smartphones. 📚 Study Material Integration Allow students to upload notes and ask questions about them. 📊 Better Analytics Track: Success rate For example: Bharat Buddy Eventually, I'd like Bharat Buddy to support more Indian languages and make voice-based learning accessible to more students. 💡 Final Thoughts The biggest thing I learned from this challenge is that building a voice agent isn't just about making an AI talk. It's about designing a system that knows: when to answer, when to use a tool, when to remember, when to call a specialist, when to ask a human, and most importantly, when not to pretend it knows something. Bharat Buddy started as an idea for a voice tutor. After 10 days, it became a much bigger experiment in building real-time, conversational AI systems. I'm still learning, but this challenge gave me a much better understanding of what it takes to turn an AI idea into a working voice product. 🔗 Project GitHub: Built during: Voice: Murf Falcon Tech: Python + LiveKit + LLM + Murf Falcon + SQLite 🙌 Thanks A big thanks to Murf AI for organizing the 10 Days of Voice Agents — VoiceForBharat Edition challenge. Building something every day for 10 days, debugging it, breaking it, fixing it, and finally documenting the journey was a genuinely valuable experience. 10 days. One voice agent. A lot of debugging. 🚀 VoiceForBharat #10DaysOfVoiceAgents #MurfAI #VoiceAI #GenerativeAI #AI #Python #LiveKit #ConversationalAI #BuildInPublic