# Runbook

The API runs under PM2 as `ppaimt-crest-api` on port 4040. Nginx serves the frontend and proxies `/api`. Do not open port 4040 to the internet.

## Production checklist

- [ ] `NODE_ENV=production`, `TRUST_PROXY=true`, and `CORS_ORIGIN` is the public site origin.
- [ ] `JWT_SECRET`, `TOKEN_PEPPER`, and `DATA_ENCRYPTION_KEY` are set from Secrets Manager and are not in git.
- [ ] `S3_BUCKET` is private, encrypted, and versioning is enabled.
- [ ] RDS automated backups are on, with a retention of 7 to 35 days. The instance is not publicly accessible.
- [ ] `GET /api/health` returns `status: ok` through Nginx.
- [ ] Certbot renewal is installed (`sudo certbot renew --dry-run`).
- [ ] CloudWatch alarms below exist, and someone is subscribed.
- [ ] A restore drill has been run on a copy, not on the live database.
- [ ] `VITE_SUPPORT_EMAIL` was set before the frontend build.
- [ ] The CREST team has signed [UAT.md](UAT.md).

## Health

`GET /api/health` runs `SELECT 1`. When `S3_BUCKET` is set it also calls HeadBucket. A failure returns 503. An empty bucket means local disk storage, and storage is `skipped`.

The response includes `X-Request-Id`. PM2 logs the same id. Point the CloudWatch agent at the PM2 log files.

## Alarms

Create these in CloudWatch. There is no Sentry project.

- API 5xx count above 5 in 5 minutes, from the Nginx or application log metric filter.
- Health check failed, from a Route 53 or load balancer check of `/api/health`.
- RDS CPU above 80 percent for 10 minutes.
- RDS free storage below 5 GB.

## Backups and restore

Turn on RDS automated backups and keep them for 7 to 35 days. Enable S3 versioning:

```bash
aws s3api put-bucket-versioning --bucket YOUR_BUCKET_NAME --versioning-configuration Status=Enabled
```

Restore drill, onto a new instance:

1. In RDS, restore a snapshot to a new instance. Do not replace the live instance.
2. Point a spare `.env` at that instance and run the API on a host that is not serving physicians.
3. Open one known application and confirm the name, PAN, and a survey answer.
4. Delete the restored instance when the check is done.

## Adding an admin

On the server:

```bash
cd /var/www/ppaimt-crest
ADMIN_EMAIL="name@crest.example" ADMIN_PASSWORD="a-long-password" npm run seed:admin
```

The password must be at least 8 characters. An existing email is left unchanged.

## Resending a survey link

In the admin application page, use Resend survey link. That revokes unused survey tokens and emails a new one. A completed survey is not reminded and cannot be opened again.

## Rotating secrets

`JWT_SECRET` signs the admin cookie. Changing it signs everyone out. Generate a new value of at least 32 characters, put it in Secrets Manager, update `backend/.env`, and `pm2 reload ecosystem.config.cjs`.

`TOKEN_PEPPER` hashes resume and survey tokens. Changing it invalidates links that have not been used. Tell physicians to use the latest email, or resend the survey link from the admin page.

`DATA_ENCRYPTION_KEY` encrypts PAN and bank account numbers. Replacing it does not rewrite old rows by itself. To rotate:

1. Put the current key in `DATA_ENCRYPTION_KEY_PREVIOUS` and the new key in `DATA_ENCRYPTION_KEY`.
2. From `backend/`, run `npm run reencrypt`.
3. Remove `DATA_ENCRYPTION_KEY_PREVIOUS`, reload PM2, and confirm an admin detail page still shows a full PAN.
4. Store only the new key in Secrets Manager.

Do not run that command against the live database until a snapshot exists.

## Deploy

GitHub Actions workflow Deploy SSHs to the instance and runs `scripts/deploy.sh`, which pulls, installs, builds, migrates, and reloads PM2. You can run the same script on the server.

## Load test

`npm run load --workspace backend` sends a few seconds of traffic to `http://127.0.0.1:4000`. It refuses any other host. Do not point it at production.

## Playwright

`npm run test:e2e` starts the API against the test database and runs the physician journey locally. Pointing that suite at staging is a separate run with `PLAYWRIGHT` base URL changes in a staging copy of the config, after the staging database and signature are ready. This repository does not sign off staging.