Blog
Notes from production
Specific problems, specific fixes, and the commands that produced them. Every post maps to an engagement where we hit the thing it describes.
Last updated
- 9 min readAWS, Cost Optimisation, EKS
Hunting Zombie AWS Resources: The Spend Nobody Owns
Orphaned snapshots, unattached volumes and idle NAT Gateways compound quietly. How to find them all in the Cost and Usage Report and then kill them safely.
- 11 min readTerraform, Infrastructure as Code, AWS
Splitting a Monolithic Terraform State Without Downtime
One state file for the whole estate makes every change risky. Splitting it with terraform state mv and moved blocks, without recreating live resources.
- 10 min readAWS, ECS, Containers
Debugging ECS Fargate Deployments That Never Reach Steady State
A Fargate service stuck in a restart loop tells you nothing by default. Where the useful signal lives, and what order to check it in when a deploy hangs.
- 12 min readAWS, Security, IAM
IAM Least Privilege in Practice, Not in Theory
Least privilege fails when it blocks the team. Building IAM policies from CloudTrail evidence, using permission boundaries, and keeping them tight over time.
- 8 min readAWS, CI/CD, Security, GitHub Actions
GitHub Actions to AWS with OIDC: Deleting Your Last Static Key
Long-lived AWS keys in CI are the commonest credential leak there is. Setting up OIDC federation, scoping the trust policy, and the mistakes that leave it open.
- 11 min readKubernetes, EKS, Networking, AWS
EKS Networking: Understanding the VPC CNI Before It Bites
The VPC CNI gives every pod a real VPC IP — elegant until a subnet runs dry mid-scale-up. How ENI limits, prefix delegation and subnet sizing interact on EKS.
- 9 min readAWS, Lambda, Serverless, Performance
Lambda Cold Starts: Measuring Before Optimising
Most cold-start advice optimises the wrong thing. What the init phase really spends time on, when provisioned concurrency pays off, and how to measure it.
- 8 min readAWS, Observability, CloudWatch
Tuning CloudWatch Alarms So They Mean Something
An alarm that fires nightly is noise with a pager attached. Choosing statistics over averages, setting honest evaluation periods, and deleting dead alarms.
- 12 min readAWS, Resilience, Chaos Engineering, DR
Chaos Engineering: Actually Testing an Availability Zone Failure
Multi-AZ on the diagram is not multi-AZ in production. Running a controlled AZ failure test with AWS FIS, what breaks first, and how to fail over safely.
- 10 min readTerraform, Infrastructure as Code, DR, AWS
Detecting Infrastructure Drift Before Your DR Region Needs It
Drift stays silent until the day it matters, usually a failover. Continuous terraform plan in CI, drift as a build failure, and alerting on console changes.