Stratawiz
/
TalkingBot
6
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
TalkingBot/voice_rag_system_architectu...

4.5 KiB
Raw Permalink Blame History

Real-Time Voice RAG Architecture Blueprint

Overview

This document outlines the technical design for a Voice-enabled Retrieval-Augmented Generation (RAG) system. The application allows users to upload multiple document formats (PDF, Word) and interact with the content via real-time, low-latency voice conversations.

The core of this system balances a heavy, asynchronous multimodal ingestion pipeline with a lightweight, ultra-low latency real-time voice routing pipeline.

1. Core Technology Stack

  • LLM Engine: AWS Bedrock (Claude 4.5 Haiku) - Chosen for its exceptional speed and reasoning capabilities, ideal for latency-sensitive voice agents.
  • Voice/WebRTC Orchestration: LiveKit - Handles real-time audio streaming, Voice Activity Detection (VAD), and turn-taking orchestration.
  • Document Parsing: Unstructured.io or Amazon Textract - Essential for parsing complex layouts, tables, and images from PDFs/Word docs.
  • Vector Database: Weaviate, Pinecone, or Amazon OpenSearch Serverless.
  • Embedding Model: Amazon Titan Multimodal Embeddings G1 (via Bedrock).
  • Speech-to-Text (STT): Deepgram or Amazon Transcribe (streaming).
  • Text-to-Speech (TTS): ElevenLabs or Cartesia (low-latency streaming).

2. System Components & Workflows

A. Asynchronous Document Ingestion Pipeline

This pipeline runs when a user uploads a booklet. It processes the document into searchable embeddings.

  1. Extraction: The raw document (PDF/Word) is sent to the layout-aware parser (e.g., Unstructured.io). It separates the document into text chunks, markdown tables, and image references.
  2. Embedding: Text, tables, and image metadata are sent to the Multimodal Embedding model.
  3. Storage: The resulting vectors are pushed to the Vector DB.
    • Metadata attached: Document ID, page number, chunk type (text/table/image), and access control lists (ACL) if supporting multiple users.

B. Real-Time Voice RAG Pipeline

This is the synchronous loop that occurs while the user is actively speaking.

  1. Audio Input: User speaks into the client application. LiveKit streams the raw audio via WebRTC to the backend Agent Worker.
  2. Transcription (STT): The Agent Worker pipes the audio stream to Deepgram for real-time transcription.
  3. RAG Retrieval: Once the user finishes speaking (detected by LiveKit's VAD), the transcript is converted into an embedding. The Vector DB is queried using Hybrid Search (Vector + Keyword) to retrieve the top K most relevant document chunks.
  4. LLM Execution: The Agent Worker constructs a prompt containing the system instructions, the retrieved document chunks, and the user's spoken transcript, sending it to AWS Bedrock (Claude 4.5 Haiku).
  5. Audio Generation (TTS): As Claude streams back its response token-by-token, the Agent Worker buffers the text into complete sentences and streams them to the TTS provider (ElevenLabs/Cartesia).
  6. Audio Output: The synthesized audio chunks are streamed back over the LiveKit WebRTC connection to the user's device.

3. Latency Optimization Strategies

To achieve natural, human-like conversation (target latency: < 800ms), implement the following optimizations:

  • Sentence-Boundary Streaming: Do not wait for the LLM to finish generating the entire response. Pipe text to the TTS engine as soon as a sentence-ending punctuation mark (., ?, !) is emitted.
  • Semantic Routing / Intent Classification: Use a lightweight, fast local classifier or regex to detect conversational filler (e.g., "hello", "thank you"). Skip the Vector DB retrieval step for these utterances to save 200-300ms.
  • Interruption Handling (Barge-in): Utilize LiveKit's built-in Voice Activity Detection (VAD). If the user starts speaking while the agent is playing audio, the system must immediately signal the TTS and LLM streams to halt, flushing the audio buffer.
  • Pre-fetching Connections: Keep persistent WebSocket connections open with the STT and TTS providers in your worker environment so no handshake time is lost during a turn.

4. Next Steps for Implementation

  1. Phase 1: Set up a standard text RAG pipeline with Unstructured and Bedrock (no voice yet) to validate retrieval quality on your specific booklets.
  2. Phase 2: Set up a basic LiveKit agent using their Python or Node.js framework, connecting STT, Claude 4.5 Haiku (without RAG), and TTS.
  3. Phase 3: Merge the two systems by injecting the Phase 1 retrieval logic into the Phase 2 agent's LLM generation step.