PPPAIMT-Crest/docs/RUNBOOK.md

Runbook

The API runs under PM2 as ppaimt-crest-api on port 4040. Nginx serves the frontend and proxies /api. Do not open port 4040 to the internet.

Production checklist

• NODE_ENV=production, TRUST_PROXY=true, and CORS_ORIGIN is the public site origin.
• JWT_SECRET, TOKEN_PEPPER, and DATA_ENCRYPTION_KEY are set from Secrets Manager and are not in git.
• S3_BUCKET is private, encrypted, and versioning is enabled.
• RDS automated backups are on, with a retention of 7 to 35 days. The instance is not publicly accessible.
• GET /api/health returns status: ok through Nginx.
• Certbot renewal is installed (sudo certbot renew --dry-run).
• CloudWatch alarms below exist, and someone is subscribed.
• A restore drill has been run on a copy, not on the live database.
• VITE_SUPPORT_EMAIL was set before the frontend build.
• The CREST team has signed UAT.md.

Health

GET /api/health runs SELECT 1. When S3_BUCKET is set it also calls HeadBucket. A failure returns 503. An empty bucket means local disk storage, and storage is skipped.

The response includes X-Request-Id. PM2 logs the same id. Point the CloudWatch agent at the PM2 log files.

Alarms

Create these in CloudWatch. There is no Sentry project.

• API 5xx count above 5 in 5 minutes, from the Nginx or application log metric filter.
• Health check failed, from a Route 53 or load balancer check of /api/health.
• RDS CPU above 80 percent for 10 minutes.
• RDS free storage below 5 GB.

Backups and restore

Turn on RDS automated backups and keep them for 7 to 35 days. Enable S3 versioning:

aws s3api put-bucket-versioning --bucket YOUR_BUCKET_NAME --versioning-configuration Status=Enabled

Restore drill, onto a new instance:

1. In RDS, restore a snapshot to a new instance. Do not replace the live instance.
2. Point a spare .env at that instance and run the API on a host that is not serving physicians.
3. Open one known application and confirm the name, PAN, and a survey answer.
4. Delete the restored instance when the check is done.

Adding an admin

On the server:

cd /var/www/ppaimt-crest
ADMIN_EMAIL="name@crest.example" ADMIN_PASSWORD="a-long-password" npm run seed:admin

The password must be at least 8 characters. An existing email is left unchanged.

Resending a survey link

In the admin application page, use Resend survey link. That revokes unused survey tokens and emails a new one. A completed survey is not reminded and cannot be opened again.

Rotating secrets

JWT_SECRET signs the admin cookie. Changing it signs everyone out. Generate a new value of at least 32 characters, put it in Secrets Manager, update backend/.env, and pm2 reload ecosystem.config.cjs.

TOKEN_PEPPER hashes resume and survey tokens. Changing it invalidates links that have not been used. Tell physicians to use the latest email, or resend the survey link from the admin page.

DATA_ENCRYPTION_KEY encrypts PAN and bank account numbers. Replacing it does not rewrite old rows by itself. To rotate:

1. Put the current key in DATA_ENCRYPTION_KEY_PREVIOUS and the new key in DATA_ENCRYPTION_KEY.
2. From backend/, run npm run reencrypt.
3. Remove DATA_ENCRYPTION_KEY_PREVIOUS, reload PM2, and confirm an admin detail page still shows a full PAN.
4. Store only the new key in Secrets Manager.

Do not run that command against the live database until a snapshot exists.

Deploy

GitHub Actions workflow Deploy SSHs to the instance and runs scripts/deploy.sh, which pulls, installs, builds, migrates, and reloads PM2. You can run the same script on the server.

Load test

npm run load --workspace backend sends a few seconds of traffic to http://127.0.0.1:4000. It refuses any other host. Do not point it at production.

Playwright

npm run test:e2e starts the API against the test database and runs the physician journey locally. Pointing that suite at staging is a separate run with PLAYWRIGHT base URL changes in a staging copy of the config, after the staging database and signature are ready. This repository does not sign off staging.