Architecture audit and multi-region feasibility for a US freight carrier's AWS platform

Architecture highlights

• Python FastAPI and .NET 8 APIs on Lambda behind API Gateway REST, HTTP and WebSocket endpoints, sharing Aurora PostgreSQL behind RDS Proxy with a read-only endpoint.
• A shared Lambda authorizer that verifies JWTs against two Cognito pools and reads role permissions from PostgreSQL.
• A React host shell that registers five micro-frontends at runtime through Vite Module Federation.
• SNS FIFO to SQS fan-out filtered by event type, and a WebSocket path that lands driver locations in DynamoDB.
• A strangler-pattern ingestion path: DMS replicates DB2, sixteen database triggers invoke a publisher Lambda, five entity queues feed processors, and the processors write the modern schema and back to DB2.
• Three Cognito user pools: staff through SAML federation, drivers through a custom challenge with Rekognition face matching, customers with just-in-time password migration.
• AWS CDK with self-mutating pipelines in a central tooling account, GitHub Actions for CI and over-the-air patching for the Android app.

Independent audit

• Ran a read-only review of all 15 repositories, tracing each claim to a file and line instead of trusting existing documents.
• Corrected documentation that was wrong. One service was documented as 42 Lambdas on SAM and is 54 on CDK. A database documented as a global cluster for disaster recovery had no secondary region attached, so nothing was replicating. The disaster-recovery pipeline in the data platform is dormant.
• Mapped production approval across the repositories: four different models, and three repositories with no pipeline approval step that rely on a manual tag-dispatch workflow.
• Found a half-finished SAM to CDK migration in three .NET repositories, where both stack sets deploy on every merge.
• Produced a platform-wide risk register and a takeover roadmap covering DevOps practice, FinOps, security hygiene and disaster recovery.

Multi-region and active/active feasibility study

Leadership asked for very high availability against a full regional outage. I owned the analysis and the recommendation, and ranked 13 blockers. Findings I could confirm in the code:

• Identity is a hard constraint. Cognito user pools are regional and hold local passwords, so they cannot simply be replicated.
• The region is fixed in every repository's configuration, and there is no failover DNS or traffic layer.
• Scheduled jobs change business data without locking or idempotency, so they would run twice in two regions.
• Aurora has no secondary today: the global-cluster wrapper exists but no second region is attached, and Aurora backup retention is the default.
• The study also flags that most tables use sequence-generated integer keys, which would collide under multi-region writes. That figure comes from the study's own analysis.

Recommendation: warm standby with automated, drill-tested failover rather than true multi-master. Proposed targets are RPO about 1 second and RTO 1–15 minutes, against an estimated analytics recovery time today of about 10 hours. These are design targets, not measurements.

• Phase 0 is hygiene: deletion protection, explicit backup retention, rotating committed secrets, reconsidering full SQL statement logging and giving alarms a notification target.
• Phases 1 to 5 then make failure detectable, decide authentication per user type, build the second region, add a traffic-failover layer and prove it with recurring drills.
• The study proposes Global Accelerator for traffic failover; the API entry points would need a suitable endpoint type behind it, which is one of the open design questions.

I delivered it in three formats: a full feasibility report, a CTO briefing pack with the decisions needed, and a plain-English guide for non-technical stakeholders. I listed eight business decisions to settle before scoping could finish.

FinOps

• Analysed six months of non-production spend with read-only Cost Explorer, Cost Optimization Hub and Cost Anomaly Detection.
• Identified 38 recommendations worth about 30% of the account's forecast month-end cost. Reservations were about 73% of the savings (eight items, 37–62% each), deleting idle resources about 26% (eleven items), and Graviton and gp3 moves the rest.
• Surfaced the drivers: relational database spend up about 22% over six months, and 91 cost anomalies in 90 days.
• Wrote a 24-item optimisation backlog for the back-office service: 6 high, 15 medium and 3 low priority, all not started.
• These are recommendations. I do not have confirmation that any were applied.

Incident analysis

• IAM role quota: the shared non-production account sat at 999 of 1,000 IAM roles. The first merge of a new service triggered parallel deploys to dev, qa and tst, which failed on role creation and rolled back, while GitHub CI on the same commit passed in 35 seconds. My RCA names the root cause; the proposed fix is a quota increase or removing unused roles. A read-only audit found 142 roles never used and 254 unused for over a year, about 396 candidates in all. The RCA records the root cause as identified and not yet resolved, and both actions are still open.
• Driver position drifting 5–50 m off the road: traced to raw phone GPS being stored and broadcast without road-snapping. I designed a fix that uses the navigation SDK's map-matched position and the existing WebSocket payload, with no backend change, estimated at 2–4 days of mobile work plus QA. It is designed, not implemented or measured.

Architecture documentation (docs-as-code)

• Wrote the architecture documentation, with 20 Python scripts that generate the Word files and diagrams so they can be regenerated as the system changes.
• A 15-section low-level design template applied to eight core services, a platform master infrastructure guide, a DevOps/SRE handbook, deployment and setup runbooks, a networking deep dive, a beginner's guide and a 100-question onboarding Q&A.
• Each document records the real state of its repository, gaps included, such as missing CI gates, wildcard IAM and alarms with no notification target.

CI standard for the web app (proposed)

• Five parallel jobs (lint, type-check, unit test, code quality and Playwright), ESLint caching, path filters so documentation-only changes skip CI, concurrency groups, and no duplicate Storybook build.
• My design review of it: keep the production build step in CI, and key the cache on the lockfile hash instead of the commit SHA. This is on a branch and not yet merged, so the speed-up is not measured.

I track every gap against a target standard. Each row is on the roadmap, ordered by risk and cost.

• Compute and API Lambda, API Gateway, FastAPI, .NET 8, GraphQL
• Data Aurora PostgreSQL, RDS Proxy, DMS, Redshift, Glue, DynamoDB, S3
• Messaging SNS, SQS, EventBridge
• Edge and identity CloudFront, Route 53, ACM, Cognito, Rekognition
• Security and operations KMS, Secrets Manager, CloudWatch, X-Ray, cost tools
• IaC and delivery AWS CDK (Python and TypeScript), CDK Pipelines, GitHub Actions with OIDC
• Application React 18, TypeScript, Vite Module Federation, Playwright, Flutter