368 lines
16 KiB
Markdown
368 lines
16 KiB
Markdown
This execution plan covers everything needed to build, deploy, and scale the **Event Buzz Virtual Concierge SaaS Chatbot** as a decoupled microservice running alongside your primary Node.js and Managed MongoDB stack on DigitalOcean.
|
|
|
|
---
|
|
|
|
## 🛠️ Step 1: System Infrastructure & Architecture Design
|
|
|
|
Before writing any application logic, you must isolate this high-concurrency chatbot service from your main ticketing web platform so that peak event traffic doesn't degrade your core registration experience.
|
|
|
|
```
|
|
[ Meta WhatsApp Cloud API ]
|
|
│ (HTTPS Post Webhooks)
|
|
▼
|
|
[ DigitalOcean Droplet ]
|
|
│
|
|
[ Nginx Proxy ]
|
|
│
|
|
[ PM2 Cluster Mode ]
|
|
│
|
|
[ Node.js Webhook Server ]
|
|
│
|
|
[ BullMQ / Redis Queue ]
|
|
│
|
|
[ Background Worker ]
|
|
/ | \
|
|
/ | \
|
|
▼ ▼ ▼
|
|
[ OpenAI API ] [ DO Managed Mongo ] [ Ext. Location APIs ]
|
|
(Embeddings/LLM) (Shared Cluster DB) (Google Places/Maps)
|
|
|
|
```
|
|
|
|
### 1.1 Infrastructure Components
|
|
|
|
* **Compute Layer:** Spin up a dedicated DigitalOcean Droplet (Ubuntu 24.04, minimum 2 vCPUs / 4GB RAM to comfortably process in-memory mathematical arrays).
|
|
* **Process Manager:** Use **PM2** running in cluster mode (`pm2 start server.js -i max`) to natively scale across all available virtual CPU cores.
|
|
* **Message Broker:** Install **Redis** directly on the droplet or use a managed instance. This acts as the backbone for your queue architecture (**BullMQ**) to prevent webhook timeouts.
|
|
* **Reverse Proxy:** Configure **Nginx** to handle SSL termination (via Let's Encrypt Certbot) and proxy incoming WhatsApp webhooks from port `443` to your internal Node.js port.
|
|
|
|
### 1.2 Shared Database Connections
|
|
|
|
Configure a dedicated, isolated database client instance connecting to your existing DigitalOcean Managed MongoDB cluster.
|
|
|
|
* Add a strict connection pool parameter (`maxPoolSize: 50`) to handle rapid parallel worker queries.
|
|
* Create a separate database name (e.g., `eventbuzz_whatsapp`) within the same cluster to logically isolate chatbot data from ticketing collections while retaining shared cluster resources.
|
|
|
|
---
|
|
|
|
## 🗄️ Step 2: Exact MongoDB Data Schemas
|
|
|
|
Implement these exact Mongoose schemas to track configuration, session routing states, and the multi-tenant knowledge chunks.
|
|
|
|
### 2.1 Knowledge Source Configuration Schema (`KnowledgeSource`)
|
|
|
|
Tracks the individual assets uploaded by the organizer and controls the text verification states.
|
|
|
|
```javascript
|
|
const mongoose = require('mongoose');
|
|
|
|
const KnowledgeSourceSchema = new mongoose.Schema({
|
|
eventId: { type: mongoose.Schema.Types.ObjectId, required: true, index: true },
|
|
type: { type: String, enum: ['url', 'pdf'], required: true },
|
|
sourcePath: { type: String, required: true }, // URL string or S3/DO Spaces file key
|
|
displayName: { type: String, required: true },
|
|
status: {
|
|
type: String,
|
|
enum: ['PENDING', 'PROCESSING', 'READY', 'FAILED'],
|
|
default: 'PENDING'
|
|
},
|
|
errorMessage: { type: String, default: null },
|
|
lastSyncedAt: { type: Date, default: Date.now }
|
|
}, { timestamps: true });
|
|
|
|
module.exports = mongoose.model('KnowledgeSource', KnowledgeSourceSchema);
|
|
|
|
```
|
|
|
|
### 2.2 Granular Vector Text Chunk Schema (`ContextChunk`)
|
|
|
|
Stores individual text blocks alongside their mathematical representation array for localized context parsing.
|
|
|
|
```javascript
|
|
const ContextChunkSchema = new mongoose.Schema({
|
|
eventId: { type: mongoose.Schema.Types.ObjectId, required: true, index: true },
|
|
sourceId: { type: mongoose.Schema.Types.ObjectId, ref: 'KnowledgeSource', required: true, index: true },
|
|
rawText: { type: String, required: true },
|
|
embedding: { type: [Number], required: true }, // Storing 1536 float values natively
|
|
isActive: { type: Boolean, default: false, index: true } // False until organizer explicitly approves via preview
|
|
});
|
|
|
|
module.exports = mongoose.model('ContextChunk', ContextChunkSchema);
|
|
|
|
```
|
|
|
|
### 2.3 Interactive User Session State Schema (`ChatSubscriber`)
|
|
|
|
Manages the individual conversation paths, data access masks, and tracking rules for attendees.
|
|
|
|
```javascript
|
|
const ChatSubscriberSchema = new mongoose.Schema({
|
|
eventId: { type: mongoose.Schema.Types.ObjectId, required: true, index: true },
|
|
attendeeId: { type: mongoose.Schema.Types.ObjectId, default: null }, // Linked when ticket matches
|
|
phoneNumber: { type: String, required: true, index: true }, // E.164 international format
|
|
sessionState: {
|
|
type: String,
|
|
enum: ['WELCOME_FORK', 'AWAITING_TICKET', 'MAIN_MENU', 'LOCAL_GUIDE_MODE'],
|
|
default: 'WELCOME_FORK'
|
|
},
|
|
metaContext: {
|
|
guestModeEnabled: { type: Boolean, default: false },
|
|
lastMenuInteraction: { type: Date, default: Date.now }
|
|
}
|
|
}, { timestamps: true });
|
|
|
|
// Ensures unique session footprint per phone number per event
|
|
ChatSubscriberSchema.index({ eventId: 1, phoneNumber: 1 }, { unique: true });
|
|
|
|
module.exports = mongoose.model('ChatSubscriber', ChatSubscriberSchema);
|
|
|
|
```
|
|
|
|
---
|
|
|
|
## ⚙️ Step 3: The Content Ingestion Pipeline (Scraping, Chunking & Verification)
|
|
|
|
When an event organizer triggers a sync or uploads documents, the backend processes it asynchronously using an execution queue.
|
|
|
|
```
|
|
[ Dashboard Input ] ──> [ BullMQ Job ] ──> [ Text Extractor ] ──> [ Token Splitter ] ──> [ Vector Array Generation ] ──> [ Staging Database ]
|
|
|
|
```
|
|
|
|
### 3.1 Text Extraction Rules
|
|
|
|
* **For URLs:** Use a structured engine like `cheerio` or `playwright`. Inject a global sanitizer middleware to systematically drop `<script>`, `<style>`, `<nav>`, `<footer>`, and `<form>` components. Extract clean semantic content inside `<article>`, `<h1>-<h6>`, `<p>`, and `<li>` items.
|
|
* **For PDFs:** Stream files via standard read streams into `pdf-parse`. Standardize layout anomalies by replacing consecutive whitespace, tabs, and duplicate carriage returns with a single space profile.
|
|
|
|
### 3.2 Mathematical Text Chunking Logic
|
|
|
|
Implement a fixed-size window sliding chunker:
|
|
|
|
* **Target Size:** Maximum of 500 characters per text chunk.
|
|
* **Overlap Margin:** Exactly 50 characters of overlap between sequential blocks to prevent losing contextual nuance at boundary lines.
|
|
* **Metadata Attachment:** Map every individual chunk array back to its parent `sourceId` and `eventId`.
|
|
|
|
### 3.3 Embeddings Vector API Integration
|
|
|
|
Send text blocks to the OpenAI API endpoint using the `text-embedding-3-small` model architecture to generate 1536-dimensional arrays. Save these vector arrays straight to the `ContextChunk` collection with `isActive: false`.
|
|
|
|
### 3.4 The Dashboard Preview UI Layer
|
|
|
|
Build an interface inside the Event Buzz organizer management panel that exposes the staged records:
|
|
|
|
* Read all items from `ContextChunk` where `sourceId === selection` and `isActive === false`.
|
|
* Render these chunks inside an editable text grid component.
|
|
* Provide clean, individual inline `Delete` overrides and text area modifiers.
|
|
* **The Publish Action:** On click of "Approve & Train Bot", switch `isActive: true` globally for verified chunks.
|
|
|
|
---
|
|
|
|
## 📥 Step 4: Webhook Event-Driven Pipeline & In-Memory RAG Engine
|
|
|
|
This core logic processes live incoming interactions under 200ms while maintaining zero context cross-contamination between events.
|
|
|
|
### 4.1 Phase 1: Rapid Acknowledge & Queue Integration
|
|
|
|
Meta requires an instant response to prevent duplicate webhook retries. Your endpoint must do nothing but drop the message payload into Redis.
|
|
|
|
```javascript
|
|
// server.js - Fastify / Express Entry Hook
|
|
const { Queue } = require('bullmq');
|
|
const inboundQueue = new Queue('whatsapp-inbound-processor');
|
|
|
|
app.post('/webhook/whatsapp', async (request, reply) => {
|
|
const payload = request.body;
|
|
|
|
// Instantly protect webhook validity with Meta
|
|
reply.status(200).send('OK');
|
|
|
|
const value = payload.entry?.[0]?.changes?.[0]?.value;
|
|
if (value && value.messages?.[0]) {
|
|
await inboundQueue.add('process-message', {
|
|
phoneId: value.metadata.phone_number_id,
|
|
messageData: value.messages[0],
|
|
profileName: value.contacts?.[0]?.profile?.name || 'Guest'
|
|
});
|
|
}
|
|
});
|
|
|
|
```
|
|
|
|
### 4.2 Phase 2: Context Resolution & User State Assessment
|
|
|
|
Inside the execution worker processing the Redis queue items, run the lookup sequence:
|
|
|
|
```javascript
|
|
// worker.js
|
|
const { Worker } = require('bullmq');
|
|
const math = require('mathjs'); // For high-speed matrix cosine operations
|
|
|
|
const worker = new Worker('whatsapp-inbound-processor', async (job) => {
|
|
const { phoneId, messageData, profileName } = job.data;
|
|
const userPhone = messageData.from;
|
|
const incomingRawText = messageData.text?.body?.trim();
|
|
|
|
// 1. Resolve exact Event tracking via incoming target metadata
|
|
const targetEvent = await Event.findOne({ "conciergeSettings.wabaPhoneNumberId": phoneId });
|
|
if (!targetEvent) return;
|
|
|
|
// 2. Fetch or initialize User Subscriber Session State
|
|
let subscriber = await ChatSubscriber.findOne({ eventId: targetEvent._id, phoneNumber: userPhone });
|
|
if (!subscriber) {
|
|
subscriber = await ChatSubscriber.create({
|
|
eventId: targetEvent._id,
|
|
phoneNumber: userPhone,
|
|
sessionState: 'WELCOME_FORK'
|
|
});
|
|
return await sendWelcomeGreeting(userPhone, targetEvent, profileName);
|
|
}
|
|
|
|
// 3. Evaluate User Action State Machine Route
|
|
await processStateRouting(subscriber, targetEvent, messageData, incomingRawText);
|
|
});
|
|
|
|
```
|
|
|
|
### 4.3 Phase 3: High-Speed In-Memory Cosine Similarity Calculation
|
|
|
|
When a text query falls back to document scanning, perform the exact vector evaluation step locally inside Node.js memory context:
|
|
|
|
```javascript
|
|
async function executeLocalRAG(eventId, userQueryString) {
|
|
// 1. Fetch user question vector embedding array from OpenAI
|
|
const queryEmbedding = await getOpenAIEmbedding(userQueryString);
|
|
|
|
// 2. Load the isolated vector profile cache for this specific event from MongoDB
|
|
const activeEventChunks = await ContextChunk.find({ eventId, isActive: true }).lean();
|
|
|
|
if (activeEventChunks.length === 0) return null;
|
|
|
|
// 3. Compute Cosine Similarity metrics against active memory set
|
|
const rankedChunks = activeEventChunks.map(chunk => {
|
|
// Cosine Similarity Formula: A • B / (||A|| * ||B||)
|
|
const dotProduct = math.dot(queryEmbedding, chunk.embedding);
|
|
const normA = math.norm(queryEmbedding);
|
|
const normB = math.norm(chunk.embedding);
|
|
const similarity = dotProduct / (normA * normB);
|
|
return { text: chunk.rawText, score: similarity };
|
|
});
|
|
|
|
// 4. Extract top 3 matched components
|
|
rankedChunks.sort((a, b) => b.score - a.score);
|
|
return rankedChunks.slice(0, 3).map(c => c.text).join("\n\n");
|
|
}
|
|
|
|
```
|
|
|
|
---
|
|
|
|
## 🧭 Step 5: Attendee Interaction State Machine & Routing Logic
|
|
|
|
Manage user intent, authentication state checks, and context masking based on subscription parameters.
|
|
|
|
```
|
|
[ User Message ]
|
|
│
|
|
┌─────────────────┴─────────────────┐
|
|
▼ ▼
|
|
[ State: WELCOME_FORK ] [ State: GUEST_MODE / MAIN_MENU ]
|
|
- Look for Interactive Buttons - Standard Request Verification
|
|
- Link Ticket OR Guest Route - Intercept Profile Queries
|
|
|
|
```
|
|
|
|
### 5.1 Interactive State Logic Implementations
|
|
|
|
#### Route A: `WELCOME_FORK` Context Evaluator
|
|
|
|
* If user clicks interactive button ID `action_connect_ticket`: Update `sessionState: 'AWAITING_TICKET'`. Dispatch text template: *"Please type your Event Buzz Ticket ID or your registered email address below."*
|
|
* If user clicks interactive button ID `action_guest_explore`: Update `sessionState: 'MAIN_MENU'`, set `metaContext.guestModeEnabled: true`. Fire main structural menu block.
|
|
|
|
#### Route B: `AWAITING_TICKET` State Processing
|
|
|
|
* Capture incoming string text values. Query Event Buzz core registration collection schema.
|
|
* **Match Found:** Update `ChatSubscriber` record, write matching database internal reference key to `attendeeId`, drop user tracking state to `MAIN_MENU`. Dispatch confirmation template: *"Account validated! Your digital check-in access is ready."*
|
|
* **Match Failed:** Deliver error notification payload: *"Ticket lookup failed. Please re-enter code or choose to switch to guest access path."* Append text selection buttons `[Re-try Ticket Registration]` and `[Browse as Guest]`.
|
|
|
|
#### Route C: `MAIN_MENU` & `GUEST_MODE` Operational Rules
|
|
|
|
* **Protected Inquiries Interception:** If a user in guest status submits requests mapping keywords like `"ticket"`, `"my seat"`, `"badge"`, or `"schedule"`, intercept the text logic without executing an OpenAI lookup. Reply directly with a standard system template message highlighting registration requirements.
|
|
* **Global Access Processing:** Execute standard internal context matching (such as Wi-Fi queries) and trigger local RAG retrieval for location, logistics, and tourism inputs.
|
|
|
|
---
|
|
|
|
## 🧹 Step 6: Automated Infrastructure Cleaning Pipeline (Data Lifecycle)
|
|
|
|
Protect system speed, lower storage overhead, and prevent database growth issues inside your DigitalOcean infrastructure by setting automated cleanup boundaries.
|
|
|
|
### 6.1 Lifecycle Rules Setup
|
|
|
|
Build an independent, recurring worker function managed via **BullMQ repeatable schedules** configured using a localized cron expression to trigger every single morning at exactly `02:00 AM UTC`.
|
|
|
|
### 6.2 Data Removal Process Flow
|
|
|
|
When the execution criteria kicks off, your backend performs a cascading removal query sequence:
|
|
|
|
```javascript
|
|
// daily-purge-cron.js
|
|
const moment = require('moment');
|
|
|
|
async function executeDataPurge() {
|
|
// Target events wrapped precisely 7 days ago
|
|
const thresholdDate = moment().subtract(7, 'days').toDate();
|
|
|
|
const expiredEvents = await Event.find({
|
|
endDate: { $lt: thresholdDate },
|
|
"conciergeSettings.status": { $ne: "ARCHIVED" }
|
|
});
|
|
|
|
for (const event of expiredEvents) {
|
|
const eventId = event._id;
|
|
|
|
// 1. Purge heavy binary array vector blocks completely
|
|
await ContextChunk.deleteMany({ eventId });
|
|
|
|
// 2. Wipe tracking source operational logs
|
|
await KnowledgeSource.deleteMany({ eventId });
|
|
|
|
// 3. Strip out temporal user subscriber conversation profiles
|
|
await ChatSubscriber.deleteMany({ eventId });
|
|
|
|
// 4. Log summary footprint variables into parent metrics object for historical reporting
|
|
event.conciergeSettings.status = "ARCHIVED";
|
|
await event.save();
|
|
|
|
console.log(`Successfully purged chatbot dataset for Event ID: ${eventId}`);
|
|
}
|
|
}
|
|
|
|
```
|
|
|
|
---
|
|
|
|
## 🚀 Step 7: Complete Deployment & Testing Plan
|
|
|
|
Follow this structural blueprint sequence to roll out your feature build smoothly into your production environments.
|
|
|
|
### 7.1 Local Verification & Integration Strategy
|
|
|
|
1. Build out individual background queue worker frameworks on local development configurations.
|
|
2. Use **Ngrok** to provision a secured tunnel mapping directly down to local port instances (`ngrok http 3000`).
|
|
3. Connect the generated secure URL endpoint within your Meta Developer App Management Panel profile under Webhook Subscriptions.
|
|
4. Manually trigger testing profiles executing variations of text patterns to confirm state routing accuracy.
|
|
|
|
### 7.2 DigitalOcean Droplet Production Setup Matrix
|
|
|
|
1. Clone the isolated chatbot service repository package directory straight into the droplet target instance.
|
|
2. Configure environment file parameters (`.env`) specifying secure environment token objects:
|
|
```env
|
|
MONGO_URI=mongodb+srv://<do_managed_credentials>
|
|
REDIS_URL=redis://127.0.0.1:6333
|
|
OPENAI_API_KEY=sk-proj-...
|
|
META_WHATSAPP_TOKEN=EAAB...
|
|
|
|
```
|
|
|
|
|
|
3. Use PM2 configuration definitions to instantiate persistent executions.
|
|
4. Finalize Nginx mapping setups pointing standard external endpoints securely right into internal routing endpoints. |