Building Agentic Voice AI for Career Counseling

Agentic Voice AI for Career Counseling: A Technical Deep Dive

This presentation details the architecture and implementation of an AI-powered voice platform designed for career counseling, specifically targeting children. The project aimed to transform an offline career counseling platform serving 20,000 students into an authentic AI experience within a two-month timeframe.

Project Genesis and Challenges

• Origin: The project originated from an agency client seeking to digitize their offline career counseling services.
• Core Problem: Children struggle with traditional dashboards and forms for career guidance. They need an intuitive, voice-first interaction.
• Key Challenge: Developing a system that could handle complex career counseling tasks (aptitude tests, academic record analysis, Holland codes, college recommendations) through voice, while being engaging and easy for children to use.
• Technical Constraints: Limited client-side resources (RAM) necessitated efficient, lightweight processing.

Architectural Components and Solutions

1. Voice Interaction & Wake Word Detection:

   • Client-Side Processing: Implemented on the client-side to manage limited resources.
   • getUserMedia API: Used for continuous audio streaming.
   • Audio Context: Managed to retain only recent conversation snippets (e.g., 10-5 seconds) to prevent memory bloat.
   • Pico Voice: Utilized for lightweight, pattern-matching wake word detection (e.g., "Hey Mohit", "Hey Counselor"). It achieves high accuracy (99.99%) by matching predefined patterns rather than processing every word.
   • Service Workers: Employed to run audio processing in the background, even when the tab is closed.

2. AI Orchestration and Interaction:

   • Live Kit: An open-source framework used to orchestrate voice, physical, and visual AI. For this project, the focus was on the voice agent.
   • Pipeline Stages: Speech-to-Text (STT), LLM Layer, Text-to-Speech (TTS).
   • LLM Layer Enhancements:
     • Tools: Integration of various tools for specific actions (web search, navigation, form filling).
     • Context Provider: To maintain conversational context.
     • Super Memory: A system to remember past conversations, extract key points, and build a knowledge graph. This allows the AI to recall user preferences (e.g., "hates math") and provide more personalized guidance.
   • Langchain: Used for LLM orchestration, enabling the creation of predetermined, deterministic flows for career counseling, mimicking a human counselor's process.

3. Tool Calling and UI Integration:

   • Live Kit RPC: Preferred over traditional MCP for client-side tool calls within a WebRTC connection, enabling direct interaction with the front end.
   • DOM Manipulation: AI can trigger actions like highlighting elements, navigating pages, and filling forms.
   • Voice Navigation Hook: Allows the AI to navigate the UI based on user voice commands (e.g., "go to settings").
   • UI Context Awareness:
     • High-Level Schema Layer: Tracks current page, available pages, and defines schemas for navigation.
     • Element Registration: Hooks are created to register UI elements, allowing the AI to reference and interact with them (e.g., highlight, add borders).
   • OpenAPI Spec & Swagger JSON: Used to automate the generation of type-safe TypeScript hooks for REST APIs, ensuring consistency between front-end and back-end.
   • Orwell: A tool used in conjunction with Codegen to convert Swagger JSON into TypeScript hooks, improving developer experience and preventing integration errors.

4. Scalability and Infrastructure:

   • Monorepo Architecture: Used to manage multiple services (Python, React, etc.) and share packages efficiently.
   • Serverless Infrastructure: Built on Cloudflare, utilizing workers, workflows, and queues for high scalability and cost-effectiveness.

Results and Key Takeaways

• Pilot Program: A pilot run with 5,000 students across three schools in Bangalore yielded significant positive results.
• Performance Metrics:
  • Over 100+ AI calls handled.
  • 68% completion rate for the entire career counseling process.
  • Significant increase in fulfillment ratio compared to offline methods.
• Business Impact: Reduced costs, increased business, and improved profits for the client.

Core Principles for AI Implementation

• Contextual Appropriateness: Voice agents are not a universal solution; choose the right tool for the specific problem.
• Multi-modal Approach: Combine voice with UI elements (triggers, highlighting) for a richer user experience.
• Continuous Learning: Always seek better solutions and explore new technologies. Don't rely solely on known methods.

Personal Project: Memory App

• A multi-platform application (Instagram bot, WhatsApp bot, Chrome extension, mobile apps) designed to solve the problem of digital clutter and unorganized saved content.
• Architected using a monorepo structure.
• Resources (HLD diagrams, READMEs) are available for those interested in production-level AI integration and multi-agent loops.