What an amazingly well researched and interesting post. I’m very grateful to the author for having done the legwork to research all of this.
I loved how they were able to peel back the Todd Howard reality distortion field to really understand how Bethesda went from that famous E3 2005 demo to what we got in the end.
My favorite memory of playing Doom on a not-PC was when I jailbroke my iPod video and installed Rockbox on it, which could run Doom.
I used it to kill some time on a long-haul school trip from New England to Disney in Florida, a ~24 hour bus ride. Not for very long though, Rockbox and especially Doom wasn’t very kind to the battery.
As an ex-Microsoft SDET who worked on Windows, we used to test for those things as well. In 2014.
Then Microsoft made the brave decision that testers were simply unnecessary. So they laid off all SDETs, then decided that SDE’s should be in charge of the tests themselves.
Which effectively made it so there was no test coverage of windows at all, as the majority of SDE’s had not interacted with the test system prior to that point. Many/most of them did not know how to run even a single test, let alone interpret its results.
This is what Microsoft management wanted, so this is what they got. I would not expect improvement, only slow degradation as Windows becomes Bing Desktop, featuring Office and Copilot (Powered By Azure™).
Makes perfect sense. It recently became clear to me (e.g. [2]) that it's not a cohesive concept but to me personally this is the meaning of POSIWID [1].
Basically making Windows a good desktop OS is not in any meaningful way the "purpose" of that part of MS. The "purpose" of any team of 20+ SWEs _is_ the set of objectives they measure and act upon. That's the only way you can actually predict the outcomes of its work.
And the corrolary is that you can usually quite clearly look at the output of such an org and infer what its "purpose" is, when defined as such.
I have a DS1515+ which has an SSD cache function that uses a whitelisted set of known good drives that function well.
If you plug in a non whitelisted ssd and try to use it as a cache, it pops up a stern warning about potential data loss due to unsupported drives with a checkbox to acknowledge that you’re okay with the risk.
So…there’s really no excuse why they couldn’t have done this for regular drives.
That assumes that the person setting up the NAS is the same person using it, which is not going to be the case for non-tech-savvy users.
Everyone will understand it costing more, fewer people will understand why the NAS ate their data without the warning it was supposed to provide, because cheap drives that didn’t support certain metrics were used.
If Synology wants to have there be only one way that the device behaves, they have to put constraints on the hardware.
> Name literally any other technology that works this way.
The internet for one.
Not the internet itself (although it certainly can be unreliable), but rather the information on it.
Which I think is more relevant to the argument anyway, as LLM’s do in fact reliably function exactly the way they were built to.
Information on the internet is inherently unreliable. It’s only when you consider externalities (like reputation of source) that its information can then be made “reliable”.
Information that comes out of LLM’s is inherently unreliable. It’s only through externalities (such as online research) that its information can be made reliable.
Unless you can invent a truth machine that somehow can tell truth from fiction, I don’t see either of these things becoming reliable, stand-alone sources of information.
In your case, you're the only employee of your business. And if you're not there the business will literally go under. And you also get directly rewarded for being there. I would guess that being 'on call' in this manner is possibly less draining on a person's soul (depending on how well they tolerate the risk of owning a business).
Contrast that with being 'on call' for your megacorporation, who isn't giving you anything extra for your on call time because they 'already pay you enough'. And where the only negative consequence for the company if you fail to immediately respond within 15 minutes is that some executive in the company is kept waiting longer than 15 minutes, or some ads aren't being shown for 15 minutes.
But if you aren't there, your boss is going to get a phone call and that's definitely not going to look good on you. And there's no bonus for fixing the problem, that was already your job in the first place. Sucks that you had to do it outside of scheduled hours, oh well.
I'm with the author of this article. Take your on-call rotation and shove it (if you're a large corporation). I'm fortunate enough to be able to take a firm stance on this point, and do so happily.
> And you also get directly rewarded for being there. ... Contrast that with being 'on call' for your megacorporation, who isn't giving you anything extra for your on call time because they 'already pay you enough'.
I'm really not seeing the distinction here. If a company offers a salary and includes on-call as part of the deal (and communicates that up front so it's not a bait and switch), how is that different than running your own business and getting compensated for your on-call time as part of a package that you sell to a client? In both cases you agree up front that you will be part of an on-call plan. In neither case are you getting a bonus for doing a good job at on-call, because either way you're just doing your job that you committed to ahead of time!
I'm totally sympathetic to people who don't want to be part of an on-call. Jobs that have on-call aren't for you, that's fine. But I don't get this idea that it's uncompensated labor, unless there are tons of people out there who somehow ended up in jobs that sprung on-call on them without warning.
Let's say I'm a business owner and I'm frustrated with the current state of the on-call system. I have options.
I can try negotiating with my clients to lessen the load in some way. Obviously this isn't always possible but it often is. I once had a freelance project that required 24 hours of on-call after a release. I negotiated release days that were convenient for me (never Fri/Sat/Sun). One time the client pushed back, I pushed back harder, and I won. In order for my push back to work I ensured that I had enough negotiating strength to do that which I planned for ahead of time.
I can upgrade my systems. For example if my current paging system is insufficient I can choose to pay $10/month more for another system that makes my life easier. I can set aside time to refactor my alerts code to make my life easier and I don't need to justify it to anyone but myself.
I can straight up refuse to do on-call and deal with the consequences to my business. Freelancer developers do this all of the time. We choose which client work to do and not to do. We can make these choices arbitrarily. Sometimes it's seasonal. Sometimes it's just based on vibes. Doesn't matter; it's our company.
Meanwhile the average on-call engineer at a large company has none of these freedoms. The underlying systems are chosen for them and they just have to deal with it.
This story is from an indian company, the on-call "expectations" might be different in your country. We were responsible for a 400k RPM service. It handled ads, so it was fairly important to the business. Whenever I had to go out for a night, or go out for a family event, or whatever, I was always able to hand over on-call to a team member for that duration. Of course, this also happened during other's on call, where I would take over. In fact, this happened daily! From ~7pm to ~9pm every day I would play football or whatever. I would always hand over on-call to another team member during this time. I usually wake up earlier than others, so I used to respond to alerts during those hours regardless of who was on call. The nights where I was staying up to watch champions league football matches or some other reason, I would take on-call as well. We just set up pagerduty's escalation order appropriately. Probably helped that there were just 5 of us in the team - easy coordination. Of course, this was my first job, and I messed up quite a bit, but I noticed the others following a similar system without me as well.
It also depends on the nature of the alerts I suppose. For us, the majority of the alerts could be checked and resolved from a mobile phone (they are alerts that could strictly speaking be resolved in an automated fashion, but the automation would get complex enough with dependencies on other service's metrics that we wouldn't be sure of not having bugs in _that_ code). About once a week or two weeks we would get an alert that needs us to look at the logs and so on.
> Meanwhile the average on-call engineer at a large company has none of these freedoms. The underlying systems are chosen for them and they just have to deal with it.
In most cases they have all of those freedoms, and the only barrier is one that's shared with the self-employed person: not liking the consequences of choosing those options.
They could negotiate with their manager to lessen the load. They could upgrade the systems. They could straight-up refuse on call.
They don't because they don't like the consequences of taking these options—and neither does the self-employed person!
> not liking the consequences of choosing those options
Correct. "Shove it" is usually preceded by not liking something.
> They could negotiate with their manager to lessen the load.
Most of the time the manager will simply refuse. As a business owner it's my decision.
> They could upgrade the systems.
At big companies this is usually outside the scope of an on-call engineer. The on-call engineer often doesn't even have commit rights to that repository.
The specific example I gave was paying $10/month more. That can be a very hard sell at a large company because their service contracts are much more complicated/expensive.
> They could straight-up refuse on call.
A business owner has much more negotiating power than an employee does.
> They don't because they don't like the consequences of taking these options—and neither does the self-employed person!
In the vast majority of cases making changes to the on-call infrastructure has very little (if any) measurable impact on the business. Like spending a week making the systems better. Or changing deploy/release dates to be more convenient.
As a business owner I can take advantage of this and make my life easier.
As an employee I have layers of bureaucracy to wade through and will probably be refused. Not because it affects the business but for other reasons.
Do others not generally get extra pay for the time on call?
I have the enviable situation where I am on call for half of every month, I get paid significantly extra for this, and there's maybe 1 emergency call per year.
The big difference is as an owner you are fully in control of allocating your time, and so if out of hours workload is becoming too much, you can choose to not work on other things in favour of fixing that. In the corporate world, there's some manager who weighs up spending two weeks to properly fix an issue or automate a process vs just making their workers unhappy and doing something that will make the manager look good in the internal politics, and often will insist on the latter.
But that's the problem with being in a bad company no matter what. If it weren't on call it would be something else that that manager would be making you suffer through. That doesn't mean on-call as a concept is terrible or that it's uncompensated labor, it just means that bad managers are capable of screwing up your life with any tools that you give them.
Bad managers are hard to predict. Even if you like your manager now, there's nothing to say they can't be promoted/reassigned/quit and you get a new manager that sucks. On-call being a thing is easier to get an answer on ahead of time
But it's meaningless a signal for bad managers because basically everyone does on call in some form. That's what this whole discussion is about: how ubiquitous it is.
The only job I worked that didn't have a formal on-call rotation ended up with me unofficially on on-call, with the same expectations as though that had been set up up front: boss calls whenever he calls and expecting an answer, and I'm left deciding how badly I want this job. Turns out management there sucked and I ended up on on-call after all.
If you find a company that actually has a good story for why they're able to get away with no on call, that might be a good signal. But if they're out there I'd love to hear from them, because most people here are just speculating about better alternatives, not speaking from experience with ones that actually worked.
At Amazon it’s common to have terrible, week-long oncall shifts with many repetitive pages. At Google they have shifts that follow the sun and they get PTO to compensate for being oncall during the weekend. Both jobs pay similarly. And I think most people joining Amazon don’t know about the oncall they are in for.
This ultra-libertarian take is consistent but not realistic. Realistically, the group decides that some amount of work or sacrifice is not compatible with having a good life, so laws are passed that either disallow such extremes or mandate extreme compensation.
The text preview underneath the search result is one thing I remember being contentious (some news websites in France took Google to court and won if I recall correctly)
For mostly the same reasons people are against AI. If you read that text, sometimes there’s no reason to visit the website, which ‘deprives’ website owners and the content creators of ad revenue they would have gotten if Google hadn’t copied the text from their website.
After all, the news is the same regardless of if it’s written on the Google result preview or on the news website itself.
I wrote a simple scraper for a 'steam game semantic search' app I built a while ago.
It definitely won't fetch all the data that this person does though. It only fetches the current list of games on Steam, their store page information and some reviews for the game.
The code quality probably isn't amazing, but it might give you an idea of how to get started with your own scraper.
I wonder if we could combine ‘thinking’ models (which write thoughts out before replying) with a mechanism they can use to check their own entropy as they’re writing output.
Maybe it could eventually learn when it needs to have a low entropy token (to produce a more-likely-to-be-factual statement) and then we can finally have models that actually definitely know when to say “Sorry, I don’t seem to have a good answer for you.”
There's a paper that probed how strongly a model would focus on prompt-supplied tokens when generating a response as a signal that it was trying to use the prompt as the source of information as opposed to knowledge it had been trained on. Ie, how much it was trying to lie based on it assuming that the information in the prompt was true, as opposed to having a rich internal model of the thing that is being verified. It looks like it works, sort of, sometimes, when you have access to the actual labels. The results from this work, in the more real-world unsupervised setting, are better than random, sure, but not good enough to really be exciting or reliable.
Entropix will get it's time in the sun, but for now, the LLM academic community is still 2 years behind the open source community. Min_p sampling is going to end up getting an oral about it at ICLR with the scores it's getting...
> the LLM academic community is still 2 years behind the open source community
Huh, isn't it the other way around? Thanks to the academic (and open) research about LLMs, we have any open source community around LLMs in the first place.
I loved how they were able to peel back the Todd Howard reality distortion field to really understand how Bethesda went from that famous E3 2005 demo to what we got in the end.
reply