10 KiB
10 KiB
Specification: Eventbuzz Virtual Concierge (Text & LiveKit Voice)
1. System Overview & Architecture
Build a multi-modal (Text & Voice) AI concierge agent for the Eventbuzz platform. The system uses an Agentic Tool-Calling pattern where a single LLM call acts as both intent recognizer and response generator.
┌──────────────┐
│ Client / Web │
│ (Text / App) │
└──────┬───────┘
│ (HTTP/WS with JWT)
▼
┌──────────────┐ ┌────────────────────────────────────────────────────────┐
│ LiveKit │ │ Concierge Agent Gateway │
│ Voice Agent ├─►│ 1. Extract & Verify Auth (User ID, Event ID) │
│ (SIP/WebRTC) │ │ 2. Construct LLM Context & Bind User-Scoped Tools │
└──────────────┘ └──────────────────────────┬─────────────────────────────┘
│
▼
┌────────────────────┐
│ LLM (Tool Calling) │
└──────────┬─────────┘
│
┌───────────────────────┴───────────────────────┐
▼ ▼
[Personal Info Intent] [Event Knowledge Intent]
Tool: `get_my_registration` Tool: `search_event_faq`
Backend API / DB query Vector DB / Hybrid Search
(Hard-scoped to Auth Context) (Scoped to Event ID)
│ │
└───────────────────────┬───────────────────────┘
▼
┌───────────────────────┐
│ LLM Synthesis Call │
└───────────┬───────────┘
▼
Formatted Text / Audio Response
2. Core Project Structure
src/
├── agent/
│ ├── prompts.ts # System instructions and tone configuration
│ ├── runner.ts # Core LLM chat and tool execution loop
│ └── tools/
│ ├── index.ts # Tool registry and dynamic context binding
│ ├── personal.tools.ts # Ticket, registration, and user-specific queries
│ └── knowledge.tools.ts # RAG / FAQ retrieval tools
├── auth/
│ └── context.ts # Session extraction (userId, eventId, role)
├── livekit/
│ └── voice_worker.ts # LiveKit Agents SDK worker integration
├── rag/
│ └── retriever.ts # Vector search or semantic retrieval service
└── server.ts # WebSocket / HTTP entry point for text chat
3. Tool Declarations & Context Binding
To prevent privilege escalation or data leakage, the LLM must never supply the userId as an argument. The execution layer injects user identity from the authenticated session.
Tool Schemas (Provided to LLM)
export const conciergeTools = [
{
name: "get_my_registration_and_tickets",
description: "Retrieves the current attendee's ticket tier, QR code data, payment status, and registered sessions. Requires no input arguments.",
parameters: {
type: "object",
properties: {},
required: []
}
},
{
name: "search_event_info",
description: "Searches event details such as venue policies, schedules, speaker bios, dress code, Wi-Fi details, and general FAQs.",
parameters: {
type: "object",
properties: {
query: {
type: "string",
description: "Specific search term or question regarding the event."
}
},
required: ["query"]
}
},
{
name: "get_agenda_schedule",
description: "Fetches schedule items for a specific date, stage, or track.",
parameters: {
type: "object",
properties: {
date: { type: "string", description: "Date in YYYY-MM-DD format (optional)" },
track: { type: "string", description: "Track or stage name (optional)" }
},
required: []
}
}
];
Context-Bound Tool Handlers
export interface SessionContext {
userId: string;
eventId: string;
}
export function createToolExecutors(ctx: SessionContext) {
return {
get_my_registration_and_tickets: async () => {
// Injects ctx.userId automatically; LLM cannot spoof another user
return await db.registrations.findFirst({
where: { userId: ctx.userId, eventId: ctx.eventId },
include: { tickets: true, assignedSessions: true }
});
},
search_event_info: async ({ query }: { query: string }) => {
return await vectorStore.similaritySearch(query, {
k: 4,
filter: { eventId: ctx.eventId }
});
},
get_agenda_schedule: async (filters: { date?: string; track?: string }) => {
return await db.sessions.findMany({
where: {
eventId: ctx.eventId,
...(filters.date && { date: filters.date }),
...(filters.track && { track: filters.track })
}
});
}
};
}
4. System Prompt Design
You are the Eventbuzz Virtual Concierge for the event identified as {{eventName}}.
Your role is to assist participants with both logistical event questions and personal ticketing/registration details.
Tone & Demeanor:
- Welcoming, clear, concise, and professional.
- For voice delivery (via LiveKit), keep responses brief (under 3 sentences when possible) and avoid formatting characters like markdown tables, bullets, or raw URLs.
Tool Rules:
1. Whenever the user asks about their own status, tickets, check-in QR, or seat, invoke `get_my_registration_and_tickets`.
2. When the user asks general event questions (parking, dress code, speaker line-ups), invoke `search_event_info`.
3. NEVER assume personal registration details or make up ticket numbers. If a tool returns no record, inform the participant politely and suggest visiting the physical Help Desk.
5. Execution Loop (Text & Voice)
export async function runConciergeTurn(
messages: ChatMessage[],
ctx: SessionContext
) {
const toolExecutors = createToolExecutors(ctx);
// First call: LLM determines whether to respond directly or invoke a tool
const response = await llm.chat({
messages,
tools: conciergeTools,
toolChoice: "auto"
});
const toolCalls = response.message.tool_calls;
// Case 1: Direct response (chit-chat or clarifications)
if (!toolCalls || toolCalls.length === 0) {
return response.message.content;
}
// Case 2: Intent matched to tool
const toolResults = await Promise.all(
toolCalls.map(async (call) => {
const fn = toolExecutors[call.function.name];
if (!fn) throw new Error(`Unknown tool: ${call.function.name}`);
const args = JSON.parse(call.function.arguments || "{}");
const result = await fn(args);
return {
tool_call_id: call.id,
role: "tool",
name: call.function.name,
content: JSON.stringify(result)
};
})
);
// Synthesis call: Feed tool outputs back to LLM for final natural response
const finalResponse = await llm.chat({
messages: [
...messages,
response.message,
...toolResults
]
});
return finalResponse.message.content;
}
6. LiveKit Voice Integration Spec
Connect the agent to LiveKit rooms using the LiveKit Agents framework:
- Room Entry & Token Validation: When dispatching the voice worker to a participant's room, pass
userIdandeventIdinside the LiveKit participant metadata or worker job metadata. - STT & TTS Pipeline:
- STT: Deepgram / Whisper stream.
- LLM Engine: Hook
runConciergeTurndirectly into the agent callback. - TTS: Cartesia / ElevenLabs stream with low-latency configuration.
- Interrupt Handling: Ensure LiveKit's turn-detection/VAD (Voice Activity Detection) cancels inflight LLM synthesis if the participant interrupts mid-sentence.
7. Step-by-Step Implementation Tasks for Cursor
- [ ] Task 1: Setup data layer for event knowledge and registrations
- Create mock/read queries for `registrations`, `tickets`, and `sessions`.
- Implement basic RAG retriever filtered by `eventId`.
- [ ] Task 2: Implement Tool Definitions and Executor Mapping
- Define JSON schemas in `src/agent/tools/`.
- Implement context-injection factory in `src/agent/tools/index.ts`.
- [ ] Task 3: Build the Agent Orchestrator
- Implement `runConciergeTurn` in `src/agent/runner.ts`.
- Add conversational state management (in-memory or Redis session store).
- [ ] Task 4: Expose Text Chat API
- Create a POST `/api/concierge/chat` route with JWT validation.
- Test via curl:
1. General query ("Where is the keynote?")
2. Personal query ("Do I have a VIP pass?")
- [ ] Task 5: Implement LiveKit Voice Worker
- Configure `livekit-agents` worker in `src/livekit/voice_worker.ts`.
- Bind participant metadata to `SessionContext`.
- Ensure tool outputs format cleanly for speech (no bullet points or markdown tables).