106 lines
5.2 KiB
Markdown
106 lines
5.2 KiB
Markdown
# PDF ingestion
|
|
|
|
CommonJS pipeline that turns a PDF booklet into cleaned text, chapters, summaries, token stats, a Transcribe vocabulary, and a versioned OpenSearch index.
|
|
|
|
The Hapi upload path is unchanged. It still writes the legacy `talkingbooklet` index. This package writes a new index and does not delete the old one.
|
|
|
|
## Setup
|
|
|
|
```bash
|
|
cd ingestion
|
|
npm install
|
|
cp .env.example .env
|
|
```
|
|
|
|
Node 20 or newer. The CLI reads `ingestion/.env` first, then fills any unset keys from `backend/.env`. Model ids come from the environment. Do not hardcode them.
|
|
|
|
`BEDROCK_MODEL_EMBED` for this account is `amazon.titan-embed-text-v2:0` with `EMBED_DIMENSIONS=1024`. The embedder sends the Titan request body (`inputText`, `dimensions`, `normalize`) and rejects any other model id.
|
|
|
|
## Commands
|
|
|
|
```bash
|
|
node src/cli.js run --book-id 3 --pdf ./book.pdf
|
|
node src/cli.js run --book-id 3 --pdf ./book.pdf --until clean
|
|
node src/cli.js run --book-id 3 --pdf ./book.pdf --textract-json ./raw.json
|
|
node src/cli.js run --book-id 3 --pdf ./book.pdf --local --prefer-text-layer --until structure
|
|
node src/cli.js stats --book-id 3
|
|
node src/cli.js reindex --book-id 3
|
|
node src/cli.js verify --book-id 3 --query "where does the river go" --chapter 2
|
|
node src/cli.js migrate plan
|
|
node src/cli.js migrate run --book-id 3 --dry-run
|
|
node src/cli.js migrate cutover
|
|
node src/cli.js migrate rollback
|
|
```
|
|
|
|
`--book-id` is `booklets.id`. The row must already exist. The CLI does not insert booklets and does not change `title` or `file_url`.
|
|
|
|
`--local` stores files under `LOCAL_DATA_DIR` and a JSON book record. It does not call AWS, MySQL, or OpenSearch.
|
|
|
|
Re-running without `--force` skips stages whose artifacts exist, including embeddings when `meta/indexed.json` matches `INGEST_VERSION`.
|
|
|
|
## What gets stored
|
|
|
|
Objects (S3, or the local data directory):
|
|
|
|
```
|
|
books/{bookId}/source.pdf
|
|
books/{bookId}/textract/raw_*.json
|
|
books/{bookId}/clean/pages.jsonl
|
|
books/{bookId}/clean/book_marked.txt
|
|
books/{bookId}/structure/chapters.json
|
|
books/{bookId}/summaries/chapters.json
|
|
books/{bookId}/summaries/book_card.json
|
|
books/{bookId}/meta/stats.json
|
|
books/{bookId}/transcribe/vocab.txt
|
|
```
|
|
|
|
`booklets.status` stays `processing`, `completed`, or `failed`, so the dashboard and voice room keep working. `completed` is set only at `READY`.
|
|
|
|
New nullable columns, added on backend startup and again by the CLI:
|
|
|
|
`ingestion_status`, `page_count`, `token_count`, `token_count_method`, `chapter_count`, `chapter_detection_method`, `author`, `detected_title`, `error`, `artifact_keys`, `index_name`, `index_version`, `chunk_count`, `embed_model`, `updated_at`.
|
|
|
|
`ingestion_status` moves through `UPLOADED`, `OCR_DONE`, `CLEANED`, `STRUCTURED`, `SUMMARIZED`, `INDEXED`, `READY`, and `FAILED`.
|
|
|
|
## OpenSearch
|
|
|
|
The live domain is Amazon OpenSearch Service 2.7 in `ap-south-1`, with fine-grained username and password access. It is not Serverless. Custom document ids and `delete_by_query` are supported, so a re-ingest bulk-writes new docs and then deletes that book's docs from any other `ingest_version`.
|
|
|
|
The legacy index `talkingbooklet` has `booklet_id`, `chunk_id`, `text`, `page`, and `vector_field` (nmslib, cosine, 1024). Leave it in place. The voice agent still queries it.
|
|
|
|
The new physical index is `{OPENSEARCH_INDEX_ALIAS}_{OPENSEARCH_INDEX_VERSION}`, for example `books_current_v2`. Reads and writes for this pipeline go through the alias. Documents have `level` `chunk`, `chapter`, or `book`, plus chapter and page fields. The vector field is `embedding` (Lucene HNSW, cosine). `nmslib` stays only on the old index because it cannot filter inside kNN.
|
|
|
|
Search pipelines are not available on OpenSearch 2.7 (`GET /_search/pipeline` returns 400; the API arrived in 2.9). `verify` runs a kNN query and a separate chapter filter. Scores are not fused. Mapping changes require a new `OPENSEARCH_INDEX_VERSION` and a reindex. `ensureIndex` never calls `putMapping` on an existing index.
|
|
|
|
## Migration
|
|
|
|
1. `migrate plan` is read-only. It lists legacy `booklet_id` values and which books already have `pages.jsonl`.
|
|
2. `migrate run` rebuilds those books from stored artifacts into the new index. It does not read old chunk text as source. `--dry-run` writes nothing. One book's failure does not stop the others. Results go to `migration_report.json`.
|
|
3. After you check `verify`, `migrate cutover` swaps the alias in one `_aliases` request.
|
|
4. `migrate rollback` points the alias at the previous index.
|
|
|
|
Cutover does not switch the voice agent. Pointing chat or voice at the alias is a later change.
|
|
|
|
## IAM
|
|
|
|
The runtime role needs:
|
|
|
|
- S3 read and write on the book prefix in `AWS_BUCKET`
|
|
- `textract:StartDocumentAnalysis` and `textract:GetDocumentAnalysis`
|
|
- `bedrock:InvokeModel` and `bedrock:CountTokens` (Converse and CountTokens use these)
|
|
- `transcribe:CreateVocabulary` and `transcribe:GetVocabulary` only if you pass `--create-vocabulary`
|
|
- `es:ESHttp*` on the domain for the new index, alias, bulk, and delete-by-query
|
|
|
|
OpenSearch access for this domain is the existing fine-grained user (`OPENSEARCH_USER` / `OPENSEARCH_PASSWORD`), not SigV4.
|
|
|
|
Conversation lives in `chat-engine/`. This CLI does not start a chat session.
|
|
|
|
## Tests
|
|
|
|
```bash
|
|
npm test
|
|
npm run lint
|
|
```
|
|
|
|
Tests mock AWS and OpenSearch. They do not call the live domain.
|