EventBuzz_Chatbot/chatbot_plan.md

This execution plan covers everything needed to build, deploy, and scale the Event Buzz Virtual Concierge SaaS Chatbot as a decoupled microservice running alongside your primary Node.js and Managed MongoDB stack on DigitalOcean.

---

🛠️ Step 1: System Infrastructure & Architecture Design

Before writing any application logic, you must isolate this high-concurrency chatbot service from your main ticketing web platform so that peak event traffic doesn't degrade your core registration experience.

                  [ Meta WhatsApp Cloud API ]
                               │ (HTTPS Post Webhooks)
                               ▼
                    [ DigitalOcean Droplet ]
                               │
                        [ Nginx Proxy ]
                               │
                       [ PM2 Cluster Mode ]
                               │
                    [ Node.js Webhook Server ]
                               │
                    [ BullMQ / Redis Queue ]
                               │
                     [ Background Worker ]
                    /          |           \
                   /           |            \
                  ▼            ▼             ▼
       [ OpenAI API ]   [ DO Managed Mongo ]  [ Ext. Location APIs ]
       (Embeddings/LLM) (Shared Cluster DB)  (Google Places/Maps)

1.1 Infrastructure Components

• Compute Layer: Spin up a dedicated DigitalOcean Droplet (Ubuntu 24.04, minimum 2 vCPUs / 4GB RAM to comfortably process in-memory mathematical arrays).
• Process Manager: Use PM2 running in cluster mode (pm2 start server.js -i max) to natively scale across all available virtual CPU cores.
• Message Broker: Install Redis directly on the droplet or use a managed instance. This acts as the backbone for your queue architecture (BullMQ) to prevent webhook timeouts.
• Reverse Proxy: Configure Nginx to handle SSL termination (via Let's Encrypt Certbot) and proxy incoming WhatsApp webhooks from port 443 to your internal Node.js port.

1.2 Shared Database Connections

Configure a dedicated, isolated database client instance connecting to your existing DigitalOcean Managed MongoDB cluster.

• Add a strict connection pool parameter (maxPoolSize: 50) to handle rapid parallel worker queries.
• Create a separate database name (e.g., eventbuzz_whatsapp) within the same cluster to logically isolate chatbot data from ticketing collections while retaining shared cluster resources.

---

🗄️ Step 2: Exact MongoDB Data Schemas

Implement these exact Mongoose schemas to track configuration, session routing states, and the multi-tenant knowledge chunks.

2.1 Knowledge Source Configuration Schema (KnowledgeSource)

Tracks the individual assets uploaded by the organizer and controls the text verification states.

const mongoose = require('mongoose');

const KnowledgeSourceSchema = new mongoose.Schema({
  eventId: { type: mongoose.Schema.Types.ObjectId, required: true, index: true },
  type: { type: String, enum: ['url', 'pdf'], required: true },
  sourcePath: { type: String, required: true }, // URL string or S3/DO Spaces file key
  displayName: { type: String, required: true },
  status: { 
    type: String, 
    enum: ['PENDING', 'PROCESSING', 'READY', 'FAILED'], 
    default: 'PENDING' 
  },
  errorMessage: { type: String, default: null },
  lastSyncedAt: { type: Date, default: Date.now }
}, { timestamps: true });

module.exports = mongoose.model('KnowledgeSource', KnowledgeSourceSchema);

2.2 Granular Vector Text Chunk Schema (ContextChunk)

Stores individual text blocks alongside their mathematical representation array for localized context parsing.

const ContextChunkSchema = new mongoose.Schema({
  eventId: { type: mongoose.Schema.Types.ObjectId, required: true, index: true },
  sourceId: { type: mongoose.Schema.Types.ObjectId, ref: 'KnowledgeSource', required: true, index: true },
  rawText: { type: String, required: true },
  embedding: { type: [Number], required: true }, // Storing 1536 float values natively
  isActive: { type: Boolean, default: false, index: true } // False until organizer explicitly approves via preview
});

module.exports = mongoose.model('ContextChunk', ContextChunkSchema);

2.3 Interactive User Session State Schema (ChatSubscriber)

Manages the individual conversation paths, data access masks, and tracking rules for attendees.

const ChatSubscriberSchema = new mongoose.Schema({
  eventId: { type: mongoose.Schema.Types.ObjectId, required: true, index: true },
  attendeeId: { type: mongoose.Schema.Types.ObjectId, default: null }, // Linked when ticket matches
  phoneNumber: { type: String, required: true, index: true }, // E.164 international format
  sessionState: { 
    type: String, 
    enum: ['WELCOME_FORK', 'AWAITING_TICKET', 'MAIN_MENU', 'LOCAL_GUIDE_MODE'], 
    default: 'WELCOME_FORK' 
  },
  metaContext: {
    guestModeEnabled: { type: Boolean, default: false },
    lastMenuInteraction: { type: Date, default: Date.now }
  }
}, { timestamps: true });

// Ensures unique session footprint per phone number per event
ChatSubscriberSchema.index({ eventId: 1, phoneNumber: 1 }, { unique: true });

module.exports = mongoose.model('ChatSubscriber', ChatSubscriberSchema);

---

⚙️ Step 3: The Content Ingestion Pipeline (Scraping, Chunking & Verification)

When an event organizer triggers a sync or uploads documents, the backend processes it asynchronously using an execution queue.

[ Dashboard Input ] ──> [ BullMQ Job ] ──> [ Text Extractor ] ──> [ Token Splitter ] ──> [ Vector Array Generation ] ──> [ Staging Database ]

3.1 Text Extraction Rules

• For URLs: Use a structured engine like cheerio or playwright. Inject a global sanitizer middleware to systematically drop <script>, <style>, <nav>, <footer>, and <form> components. Extract clean semantic content inside <article>, <h1>-<h6>, <p>, and <li> items.
• For PDFs: Stream files via standard read streams into pdf-parse. Standardize layout anomalies by replacing consecutive whitespace, tabs, and duplicate carriage returns with a single space profile.

3.2 Mathematical Text Chunking Logic

Implement a fixed-size window sliding chunker:

• Target Size: Maximum of 500 characters per text chunk.
• Overlap Margin: Exactly 50 characters of overlap between sequential blocks to prevent losing contextual nuance at boundary lines.
• Metadata Attachment: Map every individual chunk array back to its parent sourceId and eventId.

3.3 Embeddings Vector API Integration

Send text blocks to the OpenAI API endpoint using the text-embedding-3-small model architecture to generate 1536-dimensional arrays. Save these vector arrays straight to the ContextChunk collection with isActive: false.

3.4 The Dashboard Preview UI Layer

Build an interface inside the Event Buzz organizer management panel that exposes the staged records:

• Read all items from ContextChunk where sourceId === selection and isActive === false.
• Render these chunks inside an editable text grid component.
• Provide clean, individual inline Delete overrides and text area modifiers.
• The Publish Action: On click of "Approve & Train Bot", switch isActive: true globally for verified chunks.

---

📥 Step 4: Webhook Event-Driven Pipeline & In-Memory RAG Engine

This core logic processes live incoming interactions under 200ms while maintaining zero context cross-contamination between events.

4.1 Phase 1: Rapid Acknowledge & Queue Integration

Meta requires an instant response to prevent duplicate webhook retries. Your endpoint must do nothing but drop the message payload into Redis.

// server.js - Fastify / Express Entry Hook
const { Queue } = require('bullmq');
const inboundQueue = new Queue('whatsapp-inbound-processor');

app.post('/webhook/whatsapp', async (request, reply) => {
  const payload = request.body;

  // Instantly protect webhook validity with Meta
  reply.status(200).send('OK');

  const value = payload.entry?.[0]?.changes?.[0]?.value;
  if (value && value.messages?.[0]) {
    await inboundQueue.add('process-message', {
      phoneId: value.metadata.phone_number_id,
      messageData: value.messages[0],
      profileName: value.contacts?.[0]?.profile?.name || 'Guest'
    });
  }
});

4.2 Phase 2: Context Resolution & User State Assessment

Inside the execution worker processing the Redis queue items, run the lookup sequence:

// worker.js
const { Worker } = require('bullmq');
const math = require('mathjs'); // For high-speed matrix cosine operations

const worker = new Worker('whatsapp-inbound-processor', async (job) => {
  const { phoneId, messageData, profileName } = job.data;
  const userPhone = messageData.from;
  const incomingRawText = messageData.text?.body?.trim();

  // 1. Resolve exact Event tracking via incoming target metadata
  const targetEvent = await Event.findOne({ "conciergeSettings.wabaPhoneNumberId": phoneId });
  if (!targetEvent) return;

  // 2. Fetch or initialize User Subscriber Session State
  let subscriber = await ChatSubscriber.findOne({ eventId: targetEvent._id, phoneNumber: userPhone });
  if (!subscriber) {
    subscriber = await ChatSubscriber.create({
      eventId: targetEvent._id,
      phoneNumber: userPhone,
      sessionState: 'WELCOME_FORK'
    });
    return await sendWelcomeGreeting(userPhone, targetEvent, profileName);
  }

  // 3. Evaluate User Action State Machine Route
  await processStateRouting(subscriber, targetEvent, messageData, incomingRawText);
});

4.3 Phase 3: High-Speed In-Memory Cosine Similarity Calculation

When a text query falls back to document scanning, perform the exact vector evaluation step locally inside Node.js memory context:

async function executeLocalRAG(eventId, userQueryString) {
  // 1. Fetch user question vector embedding array from OpenAI
  const queryEmbedding = await getOpenAIEmbedding(userQueryString);

  // 2. Load the isolated vector profile cache for this specific event from MongoDB
  const activeEventChunks = await ContextChunk.find({ eventId, isActive: true }).lean();
  
  if (activeEventChunks.length === 0) return null;

  // 3. Compute Cosine Similarity metrics against active memory set
  const rankedChunks = activeEventChunks.map(chunk => {
    // Cosine Similarity Formula: A • B / (||A|| * ||B||)
    const dotProduct = math.dot(queryEmbedding, chunk.embedding);
    const normA = math.norm(queryEmbedding);
    const normB = math.norm(chunk.embedding);
    const similarity = dotProduct / (normA * normB);
    return { text: chunk.rawText, score: similarity };
  });

  // 4. Extract top 3 matched components
  rankedChunks.sort((a, b) => b.score - a.score);
  return rankedChunks.slice(0, 3).map(c => c.text).join("\n\n");
}

---

🧭 Step 5: Attendee Interaction State Machine & Routing Logic

Manage user intent, authentication state checks, and context masking based on subscription parameters.

                             [ User Message ]
                                    │
                  ┌─────────────────┴─────────────────┐
                  ▼                                   ▼
        [ State: WELCOME_FORK ]             [ State: GUEST_MODE / MAIN_MENU ]
         - Look for Interactive Buttons       - Standard Request Verification
         - Link Ticket OR Guest Route         - Intercept Profile Queries

5.1 Interactive State Logic Implementations

Route A: WELCOME_FORK Context Evaluator

• If user clicks interactive button ID action_connect_ticket: Update sessionState: 'AWAITING_TICKET'. Dispatch text template: "Please type your Event Buzz Ticket ID or your registered email address below."
• If user clicks interactive button ID action_guest_explore: Update sessionState: 'MAIN_MENU', set metaContext.guestModeEnabled: true. Fire main structural menu block.

Route B: AWAITING_TICKET State Processing

• Capture incoming string text values. Query Event Buzz core registration collection schema.
• Match Found: Update ChatSubscriber record, write matching database internal reference key to attendeeId, drop user tracking state to MAIN_MENU. Dispatch confirmation template: "Account validated! Your digital check-in access is ready."
• Match Failed: Deliver error notification payload: "Ticket lookup failed. Please re-enter code or choose to switch to guest access path." Append text selection buttons [Re-try Ticket Registration] and [Browse as Guest].

Route C: MAIN_MENU & GUEST_MODE Operational Rules

• Protected Inquiries Interception: If a user in guest status submits requests mapping keywords like "ticket", "my seat", "badge", or "schedule", intercept the text logic without executing an OpenAI lookup. Reply directly with a standard system template message highlighting registration requirements.
• Global Access Processing: Execute standard internal context matching (such as Wi-Fi queries) and trigger local RAG retrieval for location, logistics, and tourism inputs.

---

🧹 Step 6: Automated Infrastructure Cleaning Pipeline (Data Lifecycle)

Protect system speed, lower storage overhead, and prevent database growth issues inside your DigitalOcean infrastructure by setting automated cleanup boundaries.

6.1 Lifecycle Rules Setup

Build an independent, recurring worker function managed via BullMQ repeatable schedules configured using a localized cron expression to trigger every single morning at exactly 02:00 AM UTC.

6.2 Data Removal Process Flow

When the execution criteria kicks off, your backend performs a cascading removal query sequence:

// daily-purge-cron.js
const moment = require('moment');

async function executeDataPurge() {
  // Target events wrapped precisely 7 days ago
  const thresholdDate = moment().subtract(7, 'days').toDate();

  const expiredEvents = await Event.find({
    endDate: { $lt: thresholdDate },
    "conciergeSettings.status": { $ne: "ARCHIVED" }
  });

  for (const event of expiredEvents) {
    const eventId = event._id;

    // 1. Purge heavy binary array vector blocks completely
    await ContextChunk.deleteMany({ eventId });

    // 2. Wipe tracking source operational logs
    await KnowledgeSource.deleteMany({ eventId });

    // 3. Strip out temporal user subscriber conversation profiles
    await ChatSubscriber.deleteMany({ eventId });

    // 4. Log summary footprint variables into parent metrics object for historical reporting
    event.conciergeSettings.status = "ARCHIVED";
    await event.save();
    
    console.log(`Successfully purged chatbot dataset for Event ID: ${eventId}`);
  }
}

---

🚀 Step 7: Complete Deployment & Testing Plan

Follow this structural blueprint sequence to roll out your feature build smoothly into your production environments.

7.1 Local Verification & Integration Strategy

1. Build out individual background queue worker frameworks on local development configurations.
2. Use Ngrok to provision a secured tunnel mapping directly down to local port instances (ngrok http 3000).
3. Connect the generated secure URL endpoint within your Meta Developer App Management Panel profile under Webhook Subscriptions.
4. Manually trigger testing profiles executing variations of text patterns to confirm state routing accuracy.

7.2 DigitalOcean Droplet Production Setup Matrix

1. Clone the isolated chatbot service repository package directory straight into the droplet target instance.
2. Configure environment file parameters (.env) specifying secure environment token objects:

MONGO_URI=mongodb+srv://<do_managed_credentials>
REDIS_URL=redis://127.0.0.1:6333
OPENAI_API_KEY=sk-proj-...
META_WHATSAPP_TOKEN=EAAB...

3. Use PM2 configuration definitions to instantiate persistent executions.
4. Finalize Nginx mapping setups pointing standard external endpoints securely right into internal routing endpoints.