Stratawiz
/
TalkingBot
6
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
TalkingBot/ingestion/README.md

5.2 KiB
Raw Blame History

PDF ingestion

CommonJS pipeline that turns a PDF booklet into cleaned text, chapters, summaries, token stats, a Transcribe vocabulary, and a versioned OpenSearch index.

The Hapi upload path is unchanged. It still writes the legacy talkingbooklet index. This package writes a new index and does not delete the old one.

Setup

cd ingestion
npm install
cp .env.example .env

Node 20 or newer. The CLI reads ingestion/.env first, then fills any unset keys from backend/.env. Model ids come from the environment. Do not hardcode them.

BEDROCK_MODEL_EMBED for this account is amazon.titan-embed-text-v2:0 with EMBED_DIMENSIONS=1024. The embedder sends the Titan request body (inputText, dimensions, normalize) and rejects any other model id.

Commands

node src/cli.js run --book-id 3 --pdf ./book.pdf
node src/cli.js run --book-id 3 --pdf ./book.pdf --until clean
node src/cli.js run --book-id 3 --pdf ./book.pdf --textract-json ./raw.json
node src/cli.js run --book-id 3 --pdf ./book.pdf --local --prefer-text-layer --until structure
node src/cli.js stats --book-id 3
node src/cli.js reindex --book-id 3
node src/cli.js verify --book-id 3 --query "where does the river go" --chapter 2
node src/cli.js migrate plan
node src/cli.js migrate run --book-id 3 --dry-run
node src/cli.js migrate cutover
node src/cli.js migrate rollback

--book-id is booklets.id. The row must already exist. The CLI does not insert booklets and does not change title or file_url.

--local stores files under LOCAL_DATA_DIR and a JSON book record. It does not call AWS, MySQL, or OpenSearch.

Re-running without --force skips stages whose artifacts exist, including embeddings when meta/indexed.json matches INGEST_VERSION.

What gets stored

Objects (S3, or the local data directory):

books/{bookId}/source.pdf
books/{bookId}/textract/raw_*.json
books/{bookId}/clean/pages.jsonl
books/{bookId}/clean/book_marked.txt
books/{bookId}/structure/chapters.json
books/{bookId}/summaries/chapters.json
books/{bookId}/summaries/book_card.json
books/{bookId}/meta/stats.json
books/{bookId}/transcribe/vocab.txt

booklets.status stays processing, completed, or failed, so the dashboard and voice room keep working. completed is set only at READY.

New nullable columns, added on backend startup and again by the CLI:

ingestion_status, page_count, token_count, token_count_method, chapter_count, chapter_detection_method, author, detected_title, error, artifact_keys, index_name, index_version, chunk_count, embed_model, updated_at.

ingestion_status moves through UPLOADED, OCR_DONE, CLEANED, STRUCTURED, SUMMARIZED, INDEXED, READY, and FAILED.

OpenSearch

The live domain is Amazon OpenSearch Service 2.7 in ap-south-1, with fine-grained username and password access. It is not Serverless. Custom document ids and delete_by_query are supported, so a re-ingest bulk-writes new docs and then deletes that book's docs from any other ingest_version.

The legacy index talkingbooklet has booklet_id, chunk_id, text, page, and vector_field (nmslib, cosine, 1024). Leave it in place. The voice agent still queries it.

The new physical index is {OPENSEARCH_INDEX_ALIAS}_{OPENSEARCH_INDEX_VERSION}, for example books_current_v2. Reads and writes for this pipeline go through the alias. Documents have level chunk, chapter, or book, plus chapter and page fields. The vector field is embedding (Lucene HNSW, cosine). nmslib stays only on the old index because it cannot filter inside kNN.

Search pipelines are not available on OpenSearch 2.7 (GET /_search/pipeline returns 400; the API arrived in 2.9). verify runs a kNN query and a separate chapter filter. Scores are not fused. Mapping changes require a new OPENSEARCH_INDEX_VERSION and a reindex. ensureIndex never calls putMapping on an existing index.

Migration

  1. migrate plan is read-only. It lists legacy booklet_id values and which books already have pages.jsonl.
  2. migrate run rebuilds those books from stored artifacts into the new index. It does not read old chunk text as source. --dry-run writes nothing. One book's failure does not stop the others. Results go to migration_report.json.
  3. After you check verify, migrate cutover swaps the alias in one _aliases request.
  4. migrate rollback points the alias at the previous index.

Cutover does not switch the voice agent. Pointing chat or voice at the alias is a later change.

IAM

The runtime role needs:

  • S3 read and write on the book prefix in AWS_BUCKET
  • textract:StartDocumentAnalysis and textract:GetDocumentAnalysis
  • bedrock:InvokeModel and bedrock:CountTokens (Converse and CountTokens use these)
  • transcribe:CreateVocabulary and transcribe:GetVocabulary only if you pass --create-vocabulary
  • es:ESHttp* on the domain for the new index, alias, bulk, and delete-by-query

OpenSearch access for this domain is the existing fine-grained user (OPENSEARCH_USER / OPENSEARCH_PASSWORD), not SigV4.

Conversation lives in chat-engine/. This CLI does not start a chat session.

Tests

npm test
npm run lint

Tests mock AWS and OpenSearch. They do not call the live domain.