On the All Things Distributed blog, AWS engineer Zak van der Merwe describes his career building control planes — the reconciliation layer that records what should exist in a cloud service and continuously corrects the gap between that and reality. After nearly fourteen years at AWS, he built control planes for EC2 and, about a decade later, for Aurora DSQL.
He contrasts his pre-Amazon job at a Cape Town telecoms company — ten named servers you could SSH into and physically walk over to — with his first EC2 assignment: health-checking a fleet where hardware failed constantly. The lesson was that at scale, failure is statistical background noise, and the control plane is what keeps humans out of the response loop.
Conceptually, he likens a control plane to a thermostat: it watches actual state, compares it to desired state, and nudges. One EC2 mantra was static stability — running VMs must keep running even if the control plane itself is down, since an outage where customers cannot launch new resources is bad, but one where everything stops is catastrophically worse.
At the bottom of EC2's control plane sat a MySQL database: the critical act of a RunInstances call is writing one row. Failure modes included the primary dying (mitigated by a hot standby cutover that still meant 3am pager duty) and steady growth overwhelming a single writer. The team added read replicas — the reason the EC2 API is eventually consistent — and then sharded, first into availability zones with independent control planes, then internally into cells. Each sharding project took years, because every database touchpoint must know its shard, and there is no universally right sharding key. He also recalls the pre-automation era when fleet-wide security patching meant dividing hosts among the whole team in shifts.
His wishlist for an ideal database — scales without heroics, highly available, no servers to babysit, fully relational — led him around 2021 to the project that launched in GA in 2025 as Amazon Aurora DSQL. DSQL spins up a Firecracker micro-VM per connection, so a failure affects one connection, not the application. Read replicas are added automatically by the control plane with strongly consistent reads, and partitioning is automatic, removing the dilemma of sharding early versus shipping fast. He positions DSQL as DynamoDB-like scalability with the relational programming model, noting many AWS control planes were built on DynamoDB for exactly this reason.
The DSQL control plane itself runs on DSQL, circular dependency notwithstanding. Benefits: the bookkeeping database scales with customer adoption automatically, and the control plane's database stays available during availability-zone outages, letting reconciliation work continue — a sharp contrast to EC2-era zone failures.
He concedes gaps: some features are missing, notably foreign key constraints, which the team is approaching cautiously because they can be dangerous at scale. Latency is good and predictable, though not single-node-Postgres cached-read fast — and he argues predictable latency at scale matters more. Migrating EC2's control plane to DSQL would take years, which he considers normal for work that matters.
He closes by connecting control planes to the agentic-coding era: as writing code becomes nearly free, the bottleneck shifts to judgment — deciding what to build and shipping it safely — and infrastructure that removes invisible bookkeeping work gives builders time to, as Vogels puts it, go build.
Comments
No comments yet — be the first.
Open the discussion
No account or password needed — just enter your e-mail and we’ll send you a one-time sign-in link. First time here? You’re set up automatically.
Your rating will be applied automatically after you sign in.
Check your inbox
We’ve sent a sign-in link to …. Open it on this device — this tab will sign you in automatically.
Nothing arrived? Check your spam folder — and mark the mail as "Not spam" so it lands in your inbox next time.