Scaling Systems: The Cost of Eventual Consistency


When systems grow fast, our biggest challenge isn’t writing more code — it’s keeping promises at scale. A sequential system works fine when traffic is predictable. Each step — fulfillment, payment, royalty, reporting — waits for the one before it. Everything stays consistent. But as traffic grows 10×, that order becomes a bottleneck. Every service waits for the other, and soon the whole system crawls. The moment we hit that point, the question appears: Do we stay consistent, or do we move fast?

Eventual consistency feels like the obvious answer. Each process runs in its own lane — fulfillment can close without waiting for ledger writes, settlements can retry independently, and queues keep the system breathing. Suddenly, the same infrastructure can handle far more traffic. We trade immediate consistency for availability and throughput. Users see faster responses, the system keeps running even when one service blips, and cost per transaction drops. The flow looks messy for a few minutes, but the system “catches up.” That’s the beauty of asynchronous design — it buys time.

But that time has a cost. Data mismatches appear. Reports show unsettled payments. Debugging across queues and retries becomes detective work. Regulatory SLAs and reconciliation jobs start to dominate our weekends. The engineering shifts from coding to watching — tracing, compensating, and auditing. Eventual consistency gives scale, but it also gives responsibility. The balance lies in designing for both — choreography for non-critical flows like royalties, orchestration for payments, and strong monitoring for convergence. Tenet #5 — Think 10×, Build 1× at a Time — reminds me that scale doesn’t come from breaking rules; it comes from learning when to bend them.

Comments

Post a Comment

Popular posts from this blog

Breaking Systems, One Mistake at a Time

Image
I’ve broken more things than I’ve built — and that’s exactly why this blog exists. Over the years, I’ve learned that the most valuable lessons rarely come from perfectly executed plans. They come from the quiet moments after things break — when you realize what you missed, what you assumed, or what you could have communicated better. Most of those insights seem obvious in hindsight. This space, Obvious in Hindsight , is my running log of those realizations — from building reliable systems at scale to learning how people, teams, and code all fail (and recover) in similar ways. I’m writing to understand — out loud. Why I’m Doing This Every engineer I respect shares one thing in common: they remember what they got wrong. Not because they’re proud of the mistake, but because they learned something from it that documentation never could. This blog is where I collect my own version of those reminders — along with some movie quotes. Each post will tie back to one of my core tenets ...
Read more

Why Now Matters More Than the Idea

Image
Years ago, I proposed building a large data warehouse that captured how our creators structured their content. Everything. The semantics. The syntax. The patterns. Which HTML tags appeared together. Which musical octaves worked in combination. Which ad-tags influenced consumption. It was a massive idea. It was also far ahead of its time. The cost was high. The ROI was unclear. We didn’t know what we would do with the data. The idea made sense, but the timing didn’t. Five years later, things changed. GenAI arrived. Personalization matured. Analytics became deeper and cheaper. Suddenly, the same idea became useful. Now we had models that could learn from the patterns. Now we had creators who wanted insights. Now we had consumers who expected smarter experiences. The idea didn’t change. The world around it did. And with that, the timing flipped from “not now” to “why aren’t we doing this already?” It reminded me that ideas don’t win on its merit alone. They win when why , why now , and ...
Read more

Optimize for Being Less Wrong

Image
The brain learns in uncomfortable ways. When you repeat something you are already good at, very little changes inside. The wiring stays the same. But the moment you make a mistake, something shifts. The brain is agitated. You feel annoyed. Embarrassed. Restless. That discomfort is not failure. That is learning starting. No errors. No rewiring. The fastest learners are not the ones who avoid mistakes. They are the ones who make many small, high-quality mistakes and correct them quickly. LLM tools understand this instinctively. Tools like Claude, Cursor, or Cline don’t try to look good. They don’t protect their ego. They optimize for one thing only — being less wrong next time. They reduce their loss function again and again, billions of times. They don’t feel shame when they fail. They don’t slow down to preserve confidence. Humans do the opposite. We avoid mistakes. We repeat what feels safe. We do what makes us look competent. And that is how talented people plateau after yea...
Read more