There are two kinds of engineers.

Those who can’t stop raving about how much of a superpower LLMs are for coding, how it’s made them 100x more productive, and is unlocking things they could’ve never done before.

And those who, like you, find it to be an extremely finicky process that requires extreme amount of coddling to get average results at best.

The only thing I don’t understand is why people from the former group aren’t all utterly dominating the market and obliterating their competitors with their revolutionary products and blazing fast iteration speed.

Every time someone makes a simple solution for a problem on this website someone else points out that a much more complicated option already exists.

This is a cool series of posts, thanks for writing it!

We've released a bit about how the AWS Lambda scheduler works (a distributed, but stateful, sticky load balancer). There are a couple of reasons why Lambda doesn't use this broadcast approach to solve a similar problem to the one these posts are solving.

One is that this 'broadcast' approach introduces a tricky tradeoff decision about how long to wait for somebody to take the work before you create more capacity for that resource. The longer you wait, the higher your latency variance is. The shorter you wait, the more likely you are to 'strand' good capacity that just hasn't had a chance to respond yet. That's a tunable tradeoff, but the truly tough problem is that it creates a kind of metastable behavior under load: excess load delays responses, which makes 'stranding' more frequent, which reduces resource usage efficiency, which makes load problems worse. Again, that's a solvable problem, but solving it adds significant complexity to what was a rather simple protocol.

Another issue is dealing with failures of capacity (say a few racks lose power). The central system doesn't know what resources it lost (because that knowledge is only distributed in the workers), and so needs to discover that information from the flow of user requests. That can be OK, but again means modal latency behavior in the face of failures.

Third, the broadcast behavior requires O(N^2) messages for N requests processed (on the assumption that the fleet size is O(N) too). This truly isn't a big deal at smaller scales (packets are cheap) but can become expensive at larger scales (N^2 gets steep). The related problem is that the protocol also introduces another round-trip for discovery, increasing latency. That could be as low as a few hundred microseconds, but it's not nothing (and, again, the need to optimize for happy-case latency against bad-case efficiency makes tuning awkward).

Fourth, the dynamic behavior under load is tricky to reason about because of the race between "I can do this" and getting the work. You can be optimistic (not reserving capacity), at the cost of having to re-run the protocol (potentially an unbounded number of times!) if you lose the race to another source of work. Or, you can be pessimistic (reserving capacity and explicitly releasing what you don't need), at the cost of making the failure cases tricky (see the classic problem with 2PC coordinator failure), and reducing efficiency for popular resources (in proportion to the latency and popularity of the resource you're looking for). Slow coordinators can also cause significant resource wastage, so you're back to tuning timeouts and inventing heuristics. It's a game you can win, but a tough one.

This needle-in-a-haystack placement problem really is an interesting one, and it's super cool to see people writing about it and approaching the trade-offs in designs in a different way.

First I don't think EVs will ever fully replace ICE. There's use cases for gas.

But when I hear about the grid load, I have no pity. Or fear.

Imagine going back to 1800 and describing to an engineer the vast scale of gasoline production, distribution and consumption. Billions of ICEs distributed around the globe. The scale of fuel distribution! The network of trucks. The massive holding tanks. The inshore and offshore refineries. The pipelines and wells. The geopolitics. The strategic reserves, and prodigious tanker fleets. All because we need something everywhere that is only available in a few places.

We can send a rover to Mars for less cost than a prospecting well in the ocean, and there are lots of oil wells in the ocean. Trillions have been spent.

It seems unfathomable to go to such lengths from scratch. To invent and maintain so much, and for what? And we did.

Now imagine all that exists and you just want more. That's not unfathomable. It's just market dynamics. It'll double just fine thanks. It'll more than double.