# PDF ingestion

CommonJS pipeline that turns a PDF booklet into cleaned text, chapters, summaries, token stats, a Transcribe vocabulary, and a versioned OpenSearch index.

The Hapi upload path is unchanged. It still writes the legacy `talkingbooklet` index. This package writes a new index and does not delete the old one.

## Setup

```bash
cd ingestion
npm install
cp .env.example .env
```

Node 20 or newer. The CLI reads `ingestion/.env` first, then fills any unset keys from `backend/.env`. Model ids come from the environment. Do not hardcode them.

`BEDROCK_MODEL_EMBED` for this account is `amazon.titan-embed-text-v2:0` with `EMBED_DIMENSIONS=1024`. The embedder sends the Titan request body (`inputText`, `dimensions`, `normalize`) and rejects any other model id.

## Commands

```bash
node src/cli.js run --book-id 3 --pdf ./book.pdf
node src/cli.js run --book-id 3 --pdf ./book.pdf --until clean
node src/cli.js run --book-id 3 --pdf ./book.pdf --textract-json ./raw.json
node src/cli.js run --book-id 3 --pdf ./book.pdf --local --prefer-text-layer --until structure
node src/cli.js stats --book-id 3
node src/cli.js reindex --book-id 3
node src/cli.js verify --book-id 3 --query "where does the river go" --chapter 2
node src/cli.js migrate plan
node src/cli.js migrate run --book-id 3 --dry-run
node src/cli.js migrate cutover
node src/cli.js migrate rollback
```

`--book-id` is `booklets.id`. The row must already exist. The CLI does not insert booklets and does not change `title` or `file_url`.

`--local` stores files under `LOCAL_DATA_DIR` and a JSON book record. It does not call AWS, MySQL, or OpenSearch.

Re-running without `--force` skips stages whose artifacts exist, including embeddings when `meta/indexed.json` matches `INGEST_VERSION`.

## What gets stored

Objects (S3, or the local data directory):

```
books/{bookId}/source.pdf
books/{bookId}/textract/raw_*.json
books/{bookId}/clean/pages.jsonl
books/{bookId}/clean/book_marked.txt
books/{bookId}/structure/chapters.json
books/{bookId}/summaries/chapters.json
books/{bookId}/summaries/book_card.json
books/{bookId}/meta/stats.json
books/{bookId}/transcribe/vocab.txt
```

`booklets.status` stays `processing`, `completed`, or `failed`, so the dashboard and voice room keep working. `completed` is set only at `READY`.

New nullable columns, added on backend startup and again by the CLI:

`ingestion_status`, `page_count`, `token_count`, `token_count_method`, `chapter_count`, `chapter_detection_method`, `author`, `detected_title`, `error`, `artifact_keys`, `index_name`, `index_version`, `chunk_count`, `embed_model`, `updated_at`.

`ingestion_status` moves through `UPLOADED`, `OCR_DONE`, `CLEANED`, `STRUCTURED`, `SUMMARIZED`, `INDEXED`, `READY`, and `FAILED`.

## OpenSearch

The live domain is Amazon OpenSearch Service 2.7 in `ap-south-1`, with fine-grained username and password access. It is not Serverless. Custom document ids and `delete_by_query` are supported, so a re-ingest bulk-writes new docs and then deletes that book's docs from any other `ingest_version`.

The legacy index `talkingbooklet` has `booklet_id`, `chunk_id`, `text`, `page`, and `vector_field` (nmslib, cosine, 1024). Leave it in place. The voice agent still queries it.

The new physical index is `{OPENSEARCH_INDEX_ALIAS}_{OPENSEARCH_INDEX_VERSION}`, for example `books_current_v2`. Reads and writes for this pipeline go through the alias. Documents have `level` `chunk`, `chapter`, or `book`, plus chapter and page fields. The vector field is `embedding` (Lucene HNSW, cosine). `nmslib` stays only on the old index because it cannot filter inside kNN.

Search pipelines are not available on OpenSearch 2.7 (`GET /_search/pipeline` returns 400; the API arrived in 2.9). `verify` runs a kNN query and a separate chapter filter. Scores are not fused. Mapping changes require a new `OPENSEARCH_INDEX_VERSION` and a reindex. `ensureIndex` never calls `putMapping` on an existing index.

## Migration

1. `migrate plan` is read-only. It lists legacy `booklet_id` values and which books already have `pages.jsonl`.
2. `migrate run` rebuilds those books from stored artifacts into the new index. It does not read old chunk text as source. `--dry-run` writes nothing. One book's failure does not stop the others. Results go to `migration_report.json`.
3. After you check `verify`, `migrate cutover` swaps the alias in one `_aliases` request.
4. `migrate rollback` points the alias at the previous index.

Cutover does not switch the voice agent. Pointing chat or voice at the alias is a later change.

## IAM

The runtime role needs:

- S3 read and write on the book prefix in `AWS_BUCKET`
- `textract:StartDocumentAnalysis` and `textract:GetDocumentAnalysis`
- `bedrock:InvokeModel` and `bedrock:CountTokens` (Converse and CountTokens use these)
- `transcribe:CreateVocabulary` and `transcribe:GetVocabulary` only if you pass `--create-vocabulary`
- `es:ESHttp*` on the domain for the new index, alias, bulk, and delete-by-query

OpenSearch access for this domain is the existing fine-grained user (`OPENSEARCH_USER` / `OPENSEARCH_PASSWORD`), not SigV4.

Conversation lives in `chat-engine/`. This CLI does not start a chat session.

## Tests

```bash
npm test
npm run lint
```

Tests mock AWS and OpenSearch. They do not call the live domain.