Eventbuzz_Concierge/eventbuzz-concierge-implementation-plan.md

# Specification: Eventbuzz Virtual Concierge (Text & LiveKit Voice)

## 1. System Overview & Architecture

Build a multi-modal (Text & Voice) AI concierge agent for the **Eventbuzz** platform. The system uses an **Agentic Tool-Calling pattern** where a single LLM call acts as both intent recognizer and response generator.

```text
                  ┌──────────────┐
                  │ Client / Web │
                  │ (Text / App) │
                  └──────┬───────┘
                         │ (HTTP/WS with JWT)
                         ▼
┌──────────────┐  ┌────────────────────────────────────────────────────────┐
│  LiveKit     │  │ Concierge Agent Gateway                              │
│ Voice Agent  ├─►│ 1. Extract & Verify Auth (User ID, Event ID)          │
│ (SIP/WebRTC) │  │ 2. Construct LLM Context & Bind User-Scoped Tools    │
└──────────────┘  └──────────────────────────┬─────────────────────────────┘
                                             │
                                             ▼
                                  ┌────────────────────┐
                                  │ LLM (Tool Calling) │
                                  └──────────┬─────────┘
                                             │
                     ┌───────────────────────┴───────────────────────┐
                     ▼                                               ▼
         [Personal Info Intent]                           [Event Knowledge Intent]
         Tool: `get_my_registration`                      Tool: `search_event_faq`
         Backend API / DB query                           Vector DB / Hybrid Search
         (Hard-scoped to Auth Context)                    (Scoped to Event ID)
                     │                                               │
                     └───────────────────────┬───────────────────────┘
                                             ▼
                                 ┌───────────────────────┐
                                 │ LLM Synthesis Call    │
                                 └───────────┬───────────┘
                                             ▼
                                  Formatted Text / Audio Response
```

---

## 2. Core Project Structure

```text
src/
├── agent/
│   ├── prompts.ts              # System instructions and tone configuration
│   ├── runner.ts               # Core LLM chat and tool execution loop
│   └── tools/
│       ├── index.ts            # Tool registry and dynamic context binding
│       ├── personal.tools.ts   # Ticket, registration, and user-specific queries
│       └── knowledge.tools.ts  # RAG / FAQ retrieval tools
├── auth/
│   └── context.ts              # Session extraction (userId, eventId, role)
├── livekit/
│   └── voice_worker.ts         # LiveKit Agents SDK worker integration
├── rag/
│   └── retriever.ts            # Vector search or semantic retrieval service
└── server.ts                   # WebSocket / HTTP entry point for text chat
```

---

## 3. Tool Declarations & Context Binding

To prevent privilege escalation or data leakage, **the LLM must never supply the `userId` as an argument**. The execution layer injects user identity from the authenticated session.

### Tool Schemas (Provided to LLM)

```typescript
export const conciergeTools = [
  {
    name: "get_my_registration_and_tickets",
    description: "Retrieves the current attendee's ticket tier, QR code data, payment status, and registered sessions. Requires no input arguments.",
    parameters: {
      type: "object",
      properties: {},
      required: []
    }
  },
  {
    name: "search_event_info",
    description: "Searches event details such as venue policies, schedules, speaker bios, dress code, Wi-Fi details, and general FAQs.",
    parameters: {
      type: "object",
      properties: {
        query: {
          type: "string",
          description: "Specific search term or question regarding the event."
        }
      },
      required: ["query"]
    }
  },
  {
    name: "get_agenda_schedule",
    description: "Fetches schedule items for a specific date, stage, or track.",
    parameters: {
      type: "object",
      properties: {
        date: { type: "string", description: "Date in YYYY-MM-DD format (optional)" },
        track: { type: "string", description: "Track or stage name (optional)" }
      },
      required: []
    }
  }
];
```

### Context-Bound Tool Handlers

```typescript
export interface SessionContext {
  userId: string;
  eventId: string;
}

export function createToolExecutors(ctx: SessionContext) {
  return {
    get_my_registration_and_tickets: async () => {
      // Injects ctx.userId automatically; LLM cannot spoof another user
      return await db.registrations.findFirst({
        where: { userId: ctx.userId, eventId: ctx.eventId },
        include: { tickets: true, assignedSessions: true }
      });
    },

    search_event_info: async ({ query }: { query: string }) => {
      return await vectorStore.similaritySearch(query, {
        k: 4,
        filter: { eventId: ctx.eventId }
      });
    },

    get_agenda_schedule: async (filters: { date?: string; track?: string }) => {
      return await db.sessions.findMany({
        where: {
          eventId: ctx.eventId,
          ...(filters.date && { date: filters.date }),
          ...(filters.track && { track: filters.track })
        }
      });
    }
  };
}
```

---

## 4. System Prompt Design

```text
You are the Eventbuzz Virtual Concierge for the event identified as {{eventName}}.
Your role is to assist participants with both logistical event questions and personal ticketing/registration details.

Tone & Demeanor:
- Welcoming, clear, concise, and professional.
- For voice delivery (via LiveKit), keep responses brief (under 3 sentences when possible) and avoid formatting characters like markdown tables, bullets, or raw URLs.

Tool Rules:
1. Whenever the user asks about their own status, tickets, check-in QR, or seat, invoke `get_my_registration_and_tickets`.
2. When the user asks general event questions (parking, dress code, speaker line-ups), invoke `search_event_info`.
3. NEVER assume personal registration details or make up ticket numbers. If a tool returns no record, inform the participant politely and suggest visiting the physical Help Desk.
```

---

## 5. Execution Loop (Text & Voice)

```typescript
export async function runConciergeTurn(
  messages: ChatMessage[],
  ctx: SessionContext
) {
  const toolExecutors = createToolExecutors(ctx);

  // First call: LLM determines whether to respond directly or invoke a tool
  const response = await llm.chat({
    messages,
    tools: conciergeTools,
    toolChoice: "auto"
  });

  const toolCalls = response.message.tool_calls;

  // Case 1: Direct response (chit-chat or clarifications)
  if (!toolCalls || toolCalls.length === 0) {
    return response.message.content;
  }

  // Case 2: Intent matched to tool
  const toolResults = await Promise.all(
    toolCalls.map(async (call) => {
      const fn = toolExecutors[call.function.name];
      if (!fn) throw new Error(`Unknown tool: ${call.function.name}`);
      
      const args = JSON.parse(call.function.arguments || "{}");
      const result = await fn(args);

      return {
        tool_call_id: call.id,
        role: "tool",
        name: call.function.name,
        content: JSON.stringify(result)
      };
    })
  );

  // Synthesis call: Feed tool outputs back to LLM for final natural response
  const finalResponse = await llm.chat({
    messages: [
      ...messages,
      response.message,
      ...toolResults
    ]
  });

  return finalResponse.message.content;
}
```

---

## 6. LiveKit Voice Integration Spec

Connect the agent to LiveKit rooms using the LiveKit Agents framework:

1. **Room Entry & Token Validation:** When dispatching the voice worker to a participant's room, pass `userId` and `eventId` inside the LiveKit participant metadata or worker job metadata.
2. **STT & TTS Pipeline:**
   * **STT:** Deepgram / Whisper stream.
   * **LLM Engine:** Hook `runConciergeTurn` directly into the agent callback.
   * **TTS:** Cartesia / ElevenLabs stream with low-latency configuration.
3. **Interrupt Handling:** Ensure LiveKit's turn-detection/VAD (Voice Activity Detection) cancels inflight LLM synthesis if the participant interrupts mid-sentence.

---

## 7. Step-by-Step Implementation Tasks for Cursor

```text
- [ ] Task 1: Setup data layer for event knowledge and registrations
  - Create mock/read queries for `registrations`, `tickets`, and `sessions`.
  - Implement basic RAG retriever filtered by `eventId`.

- [ ] Task 2: Implement Tool Definitions and Executor Mapping
  - Define JSON schemas in `src/agent/tools/`.
  - Implement context-injection factory in `src/agent/tools/index.ts`.

- [ ] Task 3: Build the Agent Orchestrator
  - Implement `runConciergeTurn` in `src/agent/runner.ts`.
  - Add conversational state management (in-memory or Redis session store).

- [ ] Task 4: Expose Text Chat API
  - Create a POST `/api/concierge/chat` route with JWT validation.
  - Test via curl: 
      1. General query ("Where is the keynote?")
      2. Personal query ("Do I have a VIP pass?")

- [ ] Task 5: Implement LiveKit Voice Worker
  - Configure `livekit-agents` worker in `src/livekit/voice_worker.ts`.
  - Bind participant metadata to `SessionContext`.
  - Ensure tool outputs format cleanly for speech (no bullet points or markdown tables).
```