89 lines
4.1 KiB
Markdown
89 lines
4.1 KiB
Markdown
# Runbook
|
|
|
|
The API runs under PM2 as `ppaimt-crest-api` on port 4040. Nginx serves the frontend and proxies `/api`. Do not open port 4040 to the internet.
|
|
|
|
## Production checklist
|
|
|
|
- [ ] `NODE_ENV=production`, `TRUST_PROXY=true`, and `CORS_ORIGIN` is the public site origin.
|
|
- [ ] `JWT_SECRET`, `TOKEN_PEPPER`, and `DATA_ENCRYPTION_KEY` are set from Secrets Manager and are not in git.
|
|
- [ ] `S3_BUCKET` is private, encrypted, and versioning is enabled.
|
|
- [ ] RDS automated backups are on, with a retention of 7 to 35 days. The instance is not publicly accessible.
|
|
- [ ] `GET /api/health` returns `status: ok` through Nginx.
|
|
- [ ] Certbot renewal is installed (`sudo certbot renew --dry-run`).
|
|
- [ ] CloudWatch alarms below exist, and someone is subscribed.
|
|
- [ ] A restore drill has been run on a copy, not on the live database.
|
|
- [ ] `VITE_SUPPORT_EMAIL` was set before the frontend build.
|
|
- [ ] The CREST team has signed [UAT.md](UAT.md).
|
|
|
|
## Health
|
|
|
|
`GET /api/health` runs `SELECT 1`. When `S3_BUCKET` is set it also calls HeadBucket. A failure returns 503. An empty bucket means local disk storage, and storage is `skipped`.
|
|
|
|
The response includes `X-Request-Id`. PM2 logs the same id. Point the CloudWatch agent at the PM2 log files.
|
|
|
|
## Alarms
|
|
|
|
Create these in CloudWatch. There is no Sentry project.
|
|
|
|
- API 5xx count above 5 in 5 minutes, from the Nginx or application log metric filter.
|
|
- Health check failed, from a Route 53 or load balancer check of `/api/health`.
|
|
- RDS CPU above 80 percent for 10 minutes.
|
|
- RDS free storage below 5 GB.
|
|
|
|
## Backups and restore
|
|
|
|
Turn on RDS automated backups and keep them for 7 to 35 days. Enable S3 versioning:
|
|
|
|
```bash
|
|
aws s3api put-bucket-versioning --bucket YOUR_BUCKET_NAME --versioning-configuration Status=Enabled
|
|
```
|
|
|
|
Restore drill, onto a new instance:
|
|
|
|
1. In RDS, restore a snapshot to a new instance. Do not replace the live instance.
|
|
2. Point a spare `.env` at that instance and run the API on a host that is not serving physicians.
|
|
3. Open one known application and confirm the name, PAN, and a survey answer.
|
|
4. Delete the restored instance when the check is done.
|
|
|
|
## Adding an admin
|
|
|
|
On the server:
|
|
|
|
```bash
|
|
cd /var/www/ppaimt-crest
|
|
ADMIN_EMAIL="name@crest.example" ADMIN_PASSWORD="a-long-password" npm run seed:admin
|
|
```
|
|
|
|
The password must be at least 8 characters. An existing email is left unchanged.
|
|
|
|
## Resending a survey link
|
|
|
|
In the admin application page, use Resend survey link. That revokes unused survey tokens and emails a new one. A completed survey is not reminded and cannot be opened again.
|
|
|
|
## Rotating secrets
|
|
|
|
`JWT_SECRET` signs the admin cookie. Changing it signs everyone out. Generate a new value of at least 32 characters, put it in Secrets Manager, update `backend/.env`, and `pm2 reload ecosystem.config.cjs`.
|
|
|
|
`TOKEN_PEPPER` hashes resume and survey tokens. Changing it invalidates links that have not been used. Tell physicians to use the latest email, or resend the survey link from the admin page.
|
|
|
|
`DATA_ENCRYPTION_KEY` encrypts PAN and bank account numbers. Replacing it does not rewrite old rows by itself. To rotate:
|
|
|
|
1. Put the current key in `DATA_ENCRYPTION_KEY_PREVIOUS` and the new key in `DATA_ENCRYPTION_KEY`.
|
|
2. From `backend/`, run `npm run reencrypt`.
|
|
3. Remove `DATA_ENCRYPTION_KEY_PREVIOUS`, reload PM2, and confirm an admin detail page still shows a full PAN.
|
|
4. Store only the new key in Secrets Manager.
|
|
|
|
Do not run that command against the live database until a snapshot exists.
|
|
|
|
## Deploy
|
|
|
|
GitHub Actions workflow Deploy SSHs to the instance and runs `scripts/deploy.sh`, which pulls, installs, builds, migrates, and reloads PM2. You can run the same script on the server.
|
|
|
|
## Load test
|
|
|
|
`npm run load --workspace backend` sends a few seconds of traffic to `http://127.0.0.1:4000`. It refuses any other host. Do not point it at production.
|
|
|
|
## Playwright
|
|
|
|
`npm run test:e2e` starts the API against the test database and runs the physician journey locally. Pointing that suite at staging is a separate run with `PLAYWRIGHT` base URL changes in a staging copy of the config, after the staging database and signature are ready. This repository does not sign off staging.
|