# Specification: Eventbuzz Virtual Concierge (Text & LiveKit Voice) ## 1. System Overview & Architecture Build a multi-modal (Text & Voice) AI concierge agent for the **Eventbuzz** platform. The system uses an **Agentic Tool-Calling pattern** where a single LLM call acts as both intent recognizer and response generator. ```text ┌──────────────┐ │ Client / Web │ │ (Text / App) │ └──────┬───────┘ │ (HTTP/WS with JWT) ▼ ┌──────────────┐ ┌────────────────────────────────────────────────────────┐ │ LiveKit │ │ Concierge Agent Gateway │ │ Voice Agent ├─►│ 1. Extract & Verify Auth (User ID, Event ID) │ │ (SIP/WebRTC) │ │ 2. Construct LLM Context & Bind User-Scoped Tools │ └──────────────┘ └──────────────────────────┬─────────────────────────────┘ │ ▼ ┌────────────────────┐ │ LLM (Tool Calling) │ └──────────┬─────────┘ │ ┌───────────────────────┴───────────────────────┐ ▼ ▼ [Personal Info Intent] [Event Knowledge Intent] Tool: `get_my_registration` Tool: `search_event_faq` Backend API / DB query Vector DB / Hybrid Search (Hard-scoped to Auth Context) (Scoped to Event ID) │ │ └───────────────────────┬───────────────────────┘ ▼ ┌───────────────────────┐ │ LLM Synthesis Call │ └───────────┬───────────┘ ▼ Formatted Text / Audio Response ``` --- ## 2. Core Project Structure ```text src/ ├── agent/ │ ├── prompts.ts # System instructions and tone configuration │ ├── runner.ts # Core LLM chat and tool execution loop │ └── tools/ │ ├── index.ts # Tool registry and dynamic context binding │ ├── personal.tools.ts # Ticket, registration, and user-specific queries │ └── knowledge.tools.ts # RAG / FAQ retrieval tools ├── auth/ │ └── context.ts # Session extraction (userId, eventId, role) ├── livekit/ │ └── voice_worker.ts # LiveKit Agents SDK worker integration ├── rag/ │ └── retriever.ts # Vector search or semantic retrieval service └── server.ts # WebSocket / HTTP entry point for text chat ``` --- ## 3. Tool Declarations & Context Binding To prevent privilege escalation or data leakage, **the LLM must never supply the `userId` as an argument**. The execution layer injects user identity from the authenticated session. ### Tool Schemas (Provided to LLM) ```typescript export const conciergeTools = [ { name: "get_my_registration_and_tickets", description: "Retrieves the current attendee's ticket tier, QR code data, payment status, and registered sessions. Requires no input arguments.", parameters: { type: "object", properties: {}, required: [] } }, { name: "search_event_info", description: "Searches event details such as venue policies, schedules, speaker bios, dress code, Wi-Fi details, and general FAQs.", parameters: { type: "object", properties: { query: { type: "string", description: "Specific search term or question regarding the event." } }, required: ["query"] } }, { name: "get_agenda_schedule", description: "Fetches schedule items for a specific date, stage, or track.", parameters: { type: "object", properties: { date: { type: "string", description: "Date in YYYY-MM-DD format (optional)" }, track: { type: "string", description: "Track or stage name (optional)" } }, required: [] } } ]; ``` ### Context-Bound Tool Handlers ```typescript export interface SessionContext { userId: string; eventId: string; } export function createToolExecutors(ctx: SessionContext) { return { get_my_registration_and_tickets: async () => { // Injects ctx.userId automatically; LLM cannot spoof another user return await db.registrations.findFirst({ where: { userId: ctx.userId, eventId: ctx.eventId }, include: { tickets: true, assignedSessions: true } }); }, search_event_info: async ({ query }: { query: string }) => { return await vectorStore.similaritySearch(query, { k: 4, filter: { eventId: ctx.eventId } }); }, get_agenda_schedule: async (filters: { date?: string; track?: string }) => { return await db.sessions.findMany({ where: { eventId: ctx.eventId, ...(filters.date && { date: filters.date }), ...(filters.track && { track: filters.track }) } }); } }; } ``` --- ## 4. System Prompt Design ```text You are the Eventbuzz Virtual Concierge for the event identified as {{eventName}}. Your role is to assist participants with both logistical event questions and personal ticketing/registration details. Tone & Demeanor: - Welcoming, clear, concise, and professional. - For voice delivery (via LiveKit), keep responses brief (under 3 sentences when possible) and avoid formatting characters like markdown tables, bullets, or raw URLs. Tool Rules: 1. Whenever the user asks about their own status, tickets, check-in QR, or seat, invoke `get_my_registration_and_tickets`. 2. When the user asks general event questions (parking, dress code, speaker line-ups), invoke `search_event_info`. 3. NEVER assume personal registration details or make up ticket numbers. If a tool returns no record, inform the participant politely and suggest visiting the physical Help Desk. ``` --- ## 5. Execution Loop (Text & Voice) ```typescript export async function runConciergeTurn( messages: ChatMessage[], ctx: SessionContext ) { const toolExecutors = createToolExecutors(ctx); // First call: LLM determines whether to respond directly or invoke a tool const response = await llm.chat({ messages, tools: conciergeTools, toolChoice: "auto" }); const toolCalls = response.message.tool_calls; // Case 1: Direct response (chit-chat or clarifications) if (!toolCalls || toolCalls.length === 0) { return response.message.content; } // Case 2: Intent matched to tool const toolResults = await Promise.all( toolCalls.map(async (call) => { const fn = toolExecutors[call.function.name]; if (!fn) throw new Error(`Unknown tool: ${call.function.name}`); const args = JSON.parse(call.function.arguments || "{}"); const result = await fn(args); return { tool_call_id: call.id, role: "tool", name: call.function.name, content: JSON.stringify(result) }; }) ); // Synthesis call: Feed tool outputs back to LLM for final natural response const finalResponse = await llm.chat({ messages: [ ...messages, response.message, ...toolResults ] }); return finalResponse.message.content; } ``` --- ## 6. LiveKit Voice Integration Spec Connect the agent to LiveKit rooms using the LiveKit Agents framework: 1. **Room Entry & Token Validation:** When dispatching the voice worker to a participant's room, pass `userId` and `eventId` inside the LiveKit participant metadata or worker job metadata. 2. **STT & TTS Pipeline:** * **STT:** Deepgram / Whisper stream. * **LLM Engine:** Hook `runConciergeTurn` directly into the agent callback. * **TTS:** Cartesia / ElevenLabs stream with low-latency configuration. 3. **Interrupt Handling:** Ensure LiveKit's turn-detection/VAD (Voice Activity Detection) cancels inflight LLM synthesis if the participant interrupts mid-sentence. --- ## 7. Step-by-Step Implementation Tasks for Cursor ```text - [ ] Task 1: Setup data layer for event knowledge and registrations - Create mock/read queries for `registrations`, `tickets`, and `sessions`. - Implement basic RAG retriever filtered by `eventId`. - [ ] Task 2: Implement Tool Definitions and Executor Mapping - Define JSON schemas in `src/agent/tools/`. - Implement context-injection factory in `src/agent/tools/index.ts`. - [ ] Task 3: Build the Agent Orchestrator - Implement `runConciergeTurn` in `src/agent/runner.ts`. - Add conversational state management (in-memory or Redis session store). - [ ] Task 4: Expose Text Chat API - Create a POST `/api/concierge/chat` route with JWT validation. - Test via curl: 1. General query ("Where is the keynote?") 2. Personal query ("Do I have a VIP pass?") - [ ] Task 5: Implement LiveKit Voice Worker - Configure `livekit-agents` worker in `src/livekit/voice_worker.ts`. - Bind participant metadata to `SessionContext`. - Ensure tool outputs format cleanly for speech (no bullet points or markdown tables). ```