Production hardening and CI/CD for a US tax-services SaaS platform

• Production releases were run by hand, with no audit trail, no CI gate and no rollback through git.
• Seven long-running environments, only three of which had been deployed to in the previous twelve months. Most infrastructure was built by hand in the console.
• Only a temporary environment and the monitoring layer were in Terraform at first.
• The observability platform held no monitors, dashboards or SLOs. Alerts went to a single inbox.
• A shared non-production database had no backup retention, and static credentials were shared between environments. These went into a ranked register of 60 findings (7 critical, 14 high) for the client.

1. Audit and documentation

• Mapped every environment: web, API and cron services, serverless functions, queues, databases, CDN, DNS, certificates, identity and third-party dependencies, using read-only access.
• Wrote a production system runbook, an architecture deep dive, an analytics and AI-pipeline deep dive, and an emergency escalation runbook built around reversing the last change.
• Documented platform quirks that cause real incidents: two ORMs with different SSL handling against one database, and frontend configuration baked in at image build.

2. CI/CD and release engineering

• Replaced deploys-on-push to seven per-environment branches with one development-to-staging flow: concurrency groups, least-privilege permissions, timeouts and environment-scoped secrets.
• Production deploys only on a manual trigger, taking the git ref and an optional rollback tag. It builds, pushes tagged images and starts the App Runner deployment for the API and cron services.
• Rollback re-tags an earlier image in ECR, so no rebuild is needed; the service then redeploys the earlier image. No real rollback has been needed so far, so I do not quote a rollback time.
• Found that the production SHA tag came from the workflow's own commit and that tags are mutable, so an old rollback tag could end up on the new image. For the 8 October release I used unique dated tags and recorded the image digests.
• Created a least-privilege production deploy identity: ECR push on two repositories, plus App Runner start-deployment on three services.
• Fixed the frontend container for the upgrade line: Node 20 base, the image-optimiser dependency, removal of a tracked .env that overrode runtime values, and host binding.

3. Production releases

• 25 August: an AI chat-history feature, ownership-check security fixes and an additive migration (one table, one column, indexes). Steps: schema diff, two database snapshots, a dry run that produced the first rollback images, the manual migration, then backend and frontend. The runbook covers four rollback scenarios.
• 8 October: the framework upgrade (71 frontend commits across 216 files, 18 backend commits across 53 files). Before deploying I verified the rollback tags, took an encrypted snapshot and captured a 60-minute traffic baseline. The order was two idempotent SQL scripts, the backend, a settings change, then the frontend. Twelve checks over 60 minutes showed 0 API and 0 site 5xx errors, and no rollback was needed.
• When six frontend commits landed after the tested commit, one depending on a function missing in production, I paused for a decision and the missing function was built the same day.

4. Observability and alerting

• Audit on 18 August: 0 monitors, 0 dashboards, 0 SLOs. CloudWatch had 25 alarms and two Synthetics canaries.
• Built in Terraform: 41 monitor blocks that expand to 70 production monitors, 2 dashboards and 4 SLOs (web availability 99.9%, API availability 99.5%, API p95 under 500 ms at 99%, database connection availability 99.99%). Monitors are tiered P1 to P3, with composite alerts for a full outage and a database root cause.
• Started in observation mode, with notifications blanked, and tuned from data: a 3-day baseline moved 4xx alerting from a percentage to an absolute count, and a 7-day audit retuned 15 monitors.
• Wrote two root-cause analyses: an alert storm on production 4xx (mostly bots, one scraper made about 54% of requests that day) and a server-side rendering alarm with duplicated alert emails.

5. Cost and clean-up

• Cut non-production snapshot storage by 79% (15 of 19 snapshots deleted, 1 kept, 3 held back) after inventory and backup checks, and account-wide manual-snapshot storage by 20.5%.
• Audited four vector-database clusters and found two orphaned ones with no traffic or API keys. I exported all 20 collections (1,338,146 objects) before recommending any deletion.
• Listed orphaned resources: 19 empty ECS clusters, a redirect-only load balancer, 151 log groups that never expire and Lambdas pointing at deleted clusters.
• Modelled environment consolidation at about 35% of monthly AWS spend (about 45% with a second phase), as a recommendation.

6. Changing a shared live system safely

• Tore down the temporary upgrade environment with Terraform: 31 resources, confirmed gone, with 17 other services and 2 database instances untouched. When the first destroy command was reported as aborted, I checked the live account instead of trusting the status, and added that check to the team's checklist.
• Tagged 16 staging resources for ownership and cost tracking. Before touching anything I scanned 197 IAM policy documents for tag-based conditions (none), piloted on one resource, drilled the rollback and diffed state after each group.
• Moved 9 staging secrets into Secrets Manager, compared SHA-256 of source and stored values (9 of 9 matched) and drilled the restore.
• Rotated the production database credential on 9 October. I scanned 85 Lambdas, 23 ECS task families and every App Runner service to find the 4 consumers, took a snapshot and a checksummed dump, dry-ran twice and did a rolling update. The old password is rejected, the new one works, with 0 authentication errors.
• Stopped a database named 'rollback' being deleted: live connection data showed about 23 a day for 30 days, traced to a QA environment.
• Declined to grant staging secrets to the shared application user, because it is used by nine services including production. I proposed staging-only task roles instead.

7. Tax-calculation service

• Applied the Terraform for a new service (21 resources in two stages: ECS Fargate or Fargate Spot, an ALB with HTTPS, ECR, Secrets Manager and logging), with App Runner state unchanged before and after.
• Ran a 46-call sweep (two tax years by 23 scenarios). First-hit latency locally averaged 7.33 s with a 91.7 s maximum; on staging the average was 1.57 s with a 2.6 s maximum and identical outputs. It is a functional and latency check, not a concurrency test, and the backend does not call the service yet.

8. Staging migration: App Runner to ECS Fargate (in progress)

• A 16-working-day plan with fixed rules: never touch production, leave existing CI/CD unchanged, keep every step reversible and drill each rollback before using it.
• Days 1 and 2 are done (the tagging and secrets work above). Days 3 to 16 cover access roles, a NAT, the ECS module, a private tax engine, frontend and API cut-overs with a 72-hour soak, cron, backups and a private database.
• The reasons for moving: App Runner no longer onboards new customers, and the move gives VPC networking, task roles and load-balancer health checks.

9. Production hardening plan (planned)

• A six-week, 23-task plan under a fixed cap on new monthly spend (about 16% of baseline AWS spend). Items include network privatisation, production Multi-AZ, an AWS Backup vault, a timed restore drill, GuardDuty, root MFA, key rotation, Secrets Manager for production, a WAF rate rule and OIDC for CI.
• Full scope would exceed the cap, so some items were deferred and costed.

10. Also

• Traced an AI web-search outage (503 errors from a retired model still configured in production) and fixed it with one setting change after publishing a backup version.
• Ran the work in Jira (more than 75 items) and delivered a client documentation library, about 13 runbooks and playbooks, two RCAs and around 18 slide decks.

• Read-only first: inventory, prove the blast radius, back up, get approval, then execute and check the live state.
• Every change has an undo: keep the state before the change, drill the rollback before running it, and write it down.
• Verify, don't trust the tool: I check the live account when a tool reports an aborted run.
• Least privilege: a state-only identity for Terraform, a scoped deploy user and staging-only secrets.
• Cost-aware: every recommendation carries a cost, and plans stay inside an agreed monthly cap.
• Production changes need written approval from the client's decision owner, and production deploys run only on a manual trigger.

Each gap is tracked against a target standard and sits in the hardening plan or the backlog.

• Compute AWS App Runner, ECS Fargate, Lambda, ALB
• API and messaging API Gateway, SQS FIFO, SNS
• Data RDS PostgreSQL, MongoDB, Weaviate, S3
• Edge and identity CloudFront, Route 53, ACM, Cognito
• Security and audit Secrets Manager, IAM, CloudTrail, AWS Config, CloudWatch with Synthetics
• Observability Datadog monitors, dashboards and SLOs
• IaC and delivery Terraform, GitHub Actions, Docker, ECR, Serverless Framework, AWS SAM
• Application NestJS, Next.js, Python, Bash