Author Archives: Sérgio Giraldo

How Uber Protects Against Retry Storms

Posted on by
Reply

I have spent enough time around distributed systems to be suspicious of any resilience mechanism that operates without understanding where a failure originated. I remind why that matters with Uber’s write-up on retry storms.

In a deep service graph, retries that look reasonable at the level of one caller can multiply as an error propagates upstream, increasing load on the component that is already failing. Uber addresses this by introducing error ownership into shared infrastructure. A service distinguishes an error it originated from one it is propagating, and callers use that information to decide whether another attempt can plausibly help. In one production incident, Uber reports that this mechanism prevented roughly 9.5 million spurious requests, while preserving at-least-once retry behavior where appropriate.

The architectural point extends beyond retries. Reliability controls become more effective when they operate on causal context rather than local symptoms; rate limits, circuit breakers, backpressure, and retries all make better decisions when the system can preserve enough provenance across service boundaries to identify where intervention belongs. For large service meshes, resilience is then partly a context-propagation and attribution problem, not only a configuration problem.

https://www.uber.com/us/en/blog/protecting-against-retry-storms/

The Cuckoo’s Egg

Posted on by
Reply

I just finished “The Cuckoo’s Egg” by Clifford Stoll. This true story is from the 80’s and reading it as Gen Xer gave it an entirely different weight. Stoll’s obsessive tracking of a hacker through a maze of dial-up connections and computer networks brought back a flood of memories from my own teenage years, when I spent countless evenings dialing into BBS systems and waiting for that familiar screech of a modem handshake to connect me to a small, text-based corner of the world.

Because I belong to that first generation that experienced the early online world firsthand in its earliest, clunkiest, most improvised form, the technical descriptions in the book never feel like historical curiosities to me the way they might to younger readers. The idea of tracing a security breach across a patchwork of telephone exchanges and rented lines, of treating bandwidth and connection time as genuinely scarce resources worth fighting for, mirrors exactly what it felt like to be online during that era, when connection time and computing resources often carried a very tangible cost and every connection was a small negotiation with the physical infrastructure underneath it.

What makes the book more than just a nostalgia trip for people like me is how well it captures the tension between the physical world of telephone lines, modems, and telecommunications infrastructure and the emerging digital world that would eventually swallow it whole. Stoll wrote a technical thriller that also happens to be a time capsule of a specific and fleeting moment in computing history, and having lived through even a small piece of that moment makes his obsessive hunt for a hacker feel less like fiction and more like a shared memory.

If you came of age anywhere near computers and modems during that decade, I think this book will hit you the same way it hit me, somewhere between recognition and disbelief that we ever tolerated systems that slow and still found them thrilling.

Rust Is Tier-1 Language at Microsoft

Posted on by
Reply

I have been following Microsoft’s Rust investment for years, and this week’s update from the Rust Foundation stood out.

Microsoft has classified Rust as a Tier-1 engineering language, placing it alongside C++, C#, and TypeScript for internal development, which means Rust now gets a paved path from local development to production, including secure toolchain builds, developer tooling, and compliance with Microsoft’s Security Development Lifecycle requirements. The centerpiece of this effort is rustc_codegen_utc, a code generation backend for rustc that connects the compiler to MSVC, the native platform compiler for Windows; this places it in the same architectural family as rustc_codegen_llvm and rustc_codegen_cranelift, but it targets the toolchain that Windows and C++ teams already depend on for binary hardening, post-link compliance, hotpatching, and crash-dump analysis.

For engineers who work across hybrid Rust and C++ codebases, this matters because interoperability has historically meant duplicating platform-specific investments for each language separately. With a shared backend, new MSVC features, security capabilities, and diagnostics land on a common foundation, so Rust participates directly in the same engineering lifecycle that C++ has occupied at Microsoft for decades. More than 100 internal repositories already build with rustc_codegen_utc, and the rollout continues weekly, which suggests this is an operational shift in how Windows native software gets built, not a pilot.

The broader signal here is that language interoperability at this scale requires infrastructure investment measured in years, not a single migration project. Teams evaluating Rust adoption on Windows platforms should watch how this backend matures, since it changes the calculus for what a hybrid codebase can achieve without maintaining a parallel toolchain for every platform capability.

Guest Post: Rust Is Tier-1 Language at Microsoft

Europe’s Pitch Book

Posted on by
Reply

Follow this series by @Marieke Blom : Europe’s Pitch Book; it offers a glass-half-full perspective on Europe’s economic reality.

These are the first 3 articles:
•⁠ ⁠The case against European pessimism
https://think.ing.com/opinions/europe-pitchbook-introduction/
•⁠ ⁠Doomed by demographics?
https://think.ing.com/opinions/europes-pitch-book-doomed-by-demographics/
•⁠ ⁠Rethinking the productivity gap
https://think.ing.com/opinions/europes-pitch-book-doomed-by-demographics/
​

Extensible Software in the age of LLMs

Posted on by
Reply

Most software products are built for the top of a demand curve, where a limited set of features serves the largest group of users, while a long tail of individual needs is left unaddressed because supporting it does not scale in the traditional sense Most software products are built for the top of a demand curve, where a limited set of features serves the largest group of users, while a long tail of individual needs is left unaddressed because supporting it does not scale in the traditional sense (and attempting to do so often leads to scope creep)..

Jeremy Morrell argues that large language models change this calculation, since they lower the cost of authoring an extension to the point where serving a single user’s edge case becomes economically viable, provided the surrounding platform can deploy and secure that extension safely.

The interesting part of the argument is not that LLMs can generate code, which is now a familiar claim, but that the harder problem has always been the deployment and trust boundary around that code, not its authorship. Webhooks require the user to operate a separate service and handle delivery failures themselves, and most local extension models, such as IDE plugins or Obsidian’s ecosystem, ask the user to trust every author whose extension they install, which is a workable tradeoff for a notes app but not for a system holding financial transactions or private messages. Salesforce solved a version of this problem two decades ago by building its own compiler, runtime, and standard library so that customer logic could run safely inside a shared multi-tenant platform, well before serverless made that pattern common.

What makes the current moment diffferent is that the primitives Salesforce had to build from scratch now exist off the shelf, whether as V8 isolates, WebAssembly with WASI, or microVMs, and each comes with a different tradeoff between cold start latency, isolation strength, and language flexibility. Morrell’s argument for an object capability model over a traditional API proxy is worth sitting with regardless of which runtime a team chooses, since restricting untrusted code to explicit, narrow references such as getApprovedEmail() rather than a scoped API token removes an entire category of exfiltration risk by construction, instead of relying on a proxy layer to catch every malicious pattern after the fact.

For teams building internal tooling or customer facing platforms, the practical takeaway is that extensibility is no longer purely a UX or roadmap decision, it is also an infrastructure decision, and the primitive chosen early on will determine how much custom logic a platform can safely absorb once users start asking an LLM to write it for them.

https://jeremymorrell.dev/blog/extensible-software-in-the-age-of-llms/

A Reality Check on Rewriting in Rust

Posted on by

JetBrains published a guest post by the cot.rs maintainers reviewing what actually happened when open source and commercial teams rewrote existing software in Rust, rather than relying on the hype. The findings are more nuanced than the “rewrite it in Rust” slogan suggests, since some performance gains come from Rust specific advantages such as auto vectorization or a concurrency model that makes correct parallel merge sort easier to implement, while other gains come simply from a team rewriting a project from scratch with decades of hindsight and no legacy production constraints. Separating those two causes matters, because a team that attributes a speedup to the language alone may be signing up for a multi year rewrite when a more modest refactor in the existing codebase would have delivered most of the same benefit. (btw, I have ranted here how hard it is to switch to a Rust style of programming)

The failures documented in the post are just as instructive as the wins. Cloudflare shipped an unwrap that took down a request scoring component, sudo-rs echoed a partially typed password back to the terminal after a timeout, and the first CVE in the kernel’s Rust code came from a race condition inside an unsafe block, which is a reminder that memory safety guarantees hold only within the boundaries a team actually enforces. Prisma abandoned its Rust query engine due to skill set gaps and deployment complexity, and Loglog Games left Rust after three years because the language rewarded refactoring more than the fast iteration a game studio needed; both cases show that a rewrite can be technically sound and still fail the organization that undertook it.

This same discipline shows up in Raphael Bauer’s older but still circulating argument for defaulting to PostgreSQL over a specialized database, message queue, or search engine for as long as the workload allows it. The underlying principle in both pieces is the same, even though the technologies are unrelated: reducing the number of moving parts in a system, or the number of languages and services a team must operate, tends to matter more for long term velocity than the theoretical ceiling of any single component. Rust’s own authors reach the same conclusion from the opposite direction, recommending incremental expansion into a codebase rather than a full rewrite, since the projects that succeeded at large scale, such as the Linux kernel and Windows, expanded gradually and kept the surrounding system running throughout.

The broader point for engineering leadership is that technology selection benefits from being evaluated against the operational cost of running it, not against the best case demo of a new tool, and that a decision made for good reasons in 2015 deserves the same scrutiny when reapplied to a system in 2026.

Sources:

  1. https://blog.jetbrains.com/rust/2026/08/10/rewriting-in-rust/
  2. https://www.raphaelbauer.com/posts/postgresql-everything/

DuckDB and Spark – Parquet

Posted on by

After this post from Matt Martin, I was curious if his findings would hold for Parquet files. I ran two reproducible, Docker-based experiments, using identical resources and the same 1,000-file Parquet dataset, with correctness checks to ensure both engines produced matching results. In the analytical query test, DuckDB’s median processing time was 3.1 seconds versus Spark’s 17.1 seconds, making DuckDB 5.52× faster. In the compaction test, which read, sorted, and merged the source files into a new compressed Parquet file, DuckDB finished in a median of 9.5 seconds versus Spark’s 60.4 seconds, a 6.35× advantage.

https://www.linkedin.com/feed/update/urn:li:activity:7494050617556987905/

https://github.com/sergiorgiraldo/tst1/tree/master/duck

Watermark

Posted on by

It took only days to have an easy way to remove the watermarks. There are already ~70 repos:

https://github.com/search?q=watermark+remover+created%3A%3E2026-08-04&type=repositories

Posted in AI

Shopify: Scaling inventory reservations

Posted on by

Many architecture discussions start with choosing the right database, and we recently went through a similar decision in our own project, where our understanding of the data model made a strong case for a distributed database and a master-node architecture; the expected access and partitioning patterns did not materialize as the system evolved, so we eventually moved to a non-distributed database to simplify the landscape and remove distribution costs that were no longer giving us enough value. Shopify’s inventory reservation migration is a reminder that a useful question would be if the whole transaction path is understood well enough to identify the actual constraint.

Shopify moved inventory reservations from Redis into MySQL so reservations and the inventory ledger could participate in the same ACID transaction, which removed a class of consistency problems created by coordinating state across two systems.

The interesting part is what happened after the database design worked; throughput still hit a ceiling, query latency remained acceptable, and CPU was not saturated, so the team instrumented connection usage by business process and found that other checkout work was holding connections longer than expected. After reducing reads and transactions on the primary database, and revisiting an old InnoDB concurrency setting, the system moved past the previous limit, which reinforces a point that applies well beyond MySQL: bottlenecks often appear at the boundary between components, not inside the component receiving the most attention.

I will remember this in my future architecture reviews. When the numbers do not agree with the current diagnosis, expand the observability boundary before settle on a conclusion.

https://shopify.engineering/scaling-inventory-reservations