This repository demonstrates advanced, agentic patterns built on top of the Realtime API, currently focusing on the implementation of a Conversation Assistant. The goal of this project is to help users participate more effectively in conversations. See the Product Requirements Document (PRD.md) for full details.

The core framework demonstrates:

• Sequential agent handoffs according to a defined agent graph (taking inspiration from OpenAI Swarm)
• Background escalation to more intelligent models like o1-mini for high-stakes decisions

You can use this repo to understand how to build complex, multi-agent realtime voice applications.

Note: The screenshot may reflect an earlier version of the UI.

• This is a Next.js typescript app
• Install dependencies with npm i
• Add your OPENAI_API_KEY to your env
• Start the server with npm run dev or npm run build-check
• Open your browser to http://localhost:3000 to see the app. It should automatically connect to the adhdAssistant Agent Set.

The primary agent configuration is now focused on the ADHD Assistant, located in src/app/agentConfigs/adhdAssistant/index.ts.

Previous examples like simpleExample, customerServiceRetail, and frontDeskAuthentication have been removed to streamline focus on the current project goal.

// Example snippet from src/app/agentConfigs/adhdAssistant/index.ts
import { AgentConfig } from "@/app/types";
import { injectTransferTools } from "../utils";

// Define agents for ADHD Assistant workflow
const monitorAgent: AgentConfig = {
    // ... configuration ...
};

const responseAgent: AgentConfig = {
    // ... configuration ...
};

// ... other agents ...

const agents = injectTransferTools([/* ...adhd agents... */]);

export default agents;

• While the current focus is the ADHD assistant, the underlying framework is reusable. You can adapt the structure in src/app/agentConfigs/ to create new agent sets. Add your new config to src/app/agentConfigs/index.ts to make it selectable in the UI's "Scenario" dropdown.
• For help creating prompts, refer to the metaprompt here or the Voice Agent Metaprompter GPT.

• Scenario/Agent Selection: Select the adhdAssistant scenario (or others you add) and specific agents using the dropdowns.
• Transcript: Located on the left, showing the conversation flow, including user/assistant messages, tool calls, and agent changes. Click to expand details.
• Agent Answers: A dedicated panel (center-right) now displays responses specifically from the responseAgent for clarity. This panel can be collapsed/expanded.
• Dashboard: The right panel, formerly "Logs", now serves as a Dashboard, displaying detailed client/server events, potentially including token usage, agent steps, and timing information in the future. Click events to see the full payload. This panel can be collapsed/expanded.
• Bottom Toolbar: Controls for connecting/disconnecting, muting/unmuting the microphone (Push-to-talk has been removed), and toggling the visibility of the Agent Answers and Dashboard panels.

This section tracks the progress towards implementing the features outlined in the PRD.md.

• Noah MacCallum - noahmacca
• Ilan Bigio - ibigio