But what's the difference between one user making 900k hits and 900k different users making one hit? In both cases you have made a resource available and people are requesting it, some more than others.
If serving traffic for free is a problem, don't. If you are only able to serve N requests per second/minute/day/etc, do that. But don't complain if you give out something for free and people take it.
(also, a lot of the numbers people quote during these AI scraper "attacks" are very tame and the fact they are branded as problematic makes me suspect there's substantial incompetence in the solutions deployed to serve them)
> But what's the difference between one user making 900k hits and 900k different users making one hit?
What’s the difference between giving 900K meals to one person and feeding 900K people? The former is being abusive, wasteful, and depriving almost 900K other people of food. They are also being deceitful by pretending to be 900K different people.
Resources are finite. Web requests aren’t food, but you still pay for them. A spike in traffic may mean your service being down for the rest of the month, which is more acceptable if you helped a bunch of people who have now learned about and can talk about and share what you provided, versus having wasted all your traffic on a single bad actor who didn’t even care because they were just a robot.
> makes me suspect there's substantial incompetence in the solutions deployed to serve them
So you see bots scraping the Wikipedia webpages instead of downloading their organised dump, or scraping every git service webpage instead of cloning a repo, and think the incompetence is with the website instead of the scraper wasting time and resources to do a worse job?
There were never 900k users interested in each commit. Never was, never will be. So that's a false comparison.
These scrapers have upped both the server load (requests per second) and bandwidth requirements, without me consenting to it. If they were actual human users OR bots that were appropriately designed to minimize their impact on the target sites, that's perfectly OK.
Maybe if this was truly the only way to get to our god-like LLM to work in a god-like way (*), it would also be acceptable. But it isn't.
And on top of that, they are incompetently designed and they are causing real issues that a huge number of sites need to address.
(*) put differently, if all this current scraping activity delivered some notable benefit to humanity
Intention plays a part. (D)DoS is intentionally done to make a website unavailable to legitimate users. Scraping may do this as a side-effect (if you are incompetent and/or use the "cloud"), but isn't the intention.
Docker is effectively just a packaging format on Linux - it's not a VM (unlike on Windows/Mac where running Docker involves running a hidden Linux VM on which containers are scheduled). So I don't see why you wouldn't use it if it makes things easier (like not having to worry about distro specifics, since there is generally one canonical Docker image for major pieces of software)?
If you need to go beyond what a single bare-metal server can offer, then consider it.
But don’t discount bare-metal first! I see a lot of K8s or other cluster managers being used to manage underpowered cloud VMs, and while I understand the need for an orchestrator if you’re managing dozens of VMs, I wonder - why do you need multiple VMs in the first place if their total performance can be achieved by a handful of bare-metal machines?
The role of a software engineer is to condense the (often unclear) requirements, business domain knowledge, existing code (if any) and their skills/experience into a representation of the solution in a very concise language: a programming language.
Having to instead express all that (including the business-related part, since the agent has no context of that) in a verbose language (English) feels counter-productive, and is counter-productive in my experience.
I've successfully one-shotted easy self-contained, throwaway tasks ("make me a program that fills Redis with random keys and values" - Claude will one-shot that) but when it comes to working with complex existing codebases I've never seen the benefits - having to explain all the context to the agent and correcting its mistakes takes longer than just doing it myself (worse, it's unpredictable - I know roughly how long something will take, but it's impossible to tell in advance whether an agent will one-shot it successfully or require longer babysitting than just doing it manually from the beginning).
This is downvoted for some reason and yet it’s absolutely correct. There’s a reason telcos specifically are notorious for the shittiest customer service ever - because major telcos typically have a monopoly and/or long-term contracts the customer can’t easily leave. They could literally bill you and not provide any service, and most people would still pay at least a couple months while fighting to cancel or get it resolved to avoid getting their credit report dinged.
This doesn’t work for technical people, but it works for the other 90% of people who’d rather call and waste not only support’s time, but their own time even if their problem can be resolved by common sense or the first page of the knowledge base.
Companies need to offer an “advanced support” option - a separate number with big scary warnings where every call whose resolution is due to your own fault or something findable on the docs ends up charging the user a fee. In exchange, it directly bypasses all chatbots and first-line support.
I’ve never heard an argument why companies don’t do that, it seems like it would be a win-win for everyone to get the customers to self-triage before reaching out.
Because "customer support" is supposed to be, at least on its face, friendly and supportive. It's not good business to actively antagonize or challenge your customers.
1 US worker = About 3-8 outsourced workers. Somewhere in there the logic of hiring many US Workers to manage said advanced customer service (they'd have to native speak English) is not worth the cost of the department. Even with the fee.
But I assume they have US-based workers anyway to handle escalations. So the proposal is to simply allow users to self-escalate and pay a fee if they waste time.
I had an experience like this in a real-life non technical situation.
Walking down the street I receive a text to say my glasses were ready to be picked up. I had not purchased any glasses, and the store that I was to collect them from was not in the city I live in. By coincidence I was approximately 30 meters from a branch of the same store in my town. I popped in to tell them that someone had entered a phone number incorrectly and someone might need to told by other means that their glasses were ready.
The response? "Certainly sir, can I have your name, and address". Explaining how this information was not relevant was not fruitful. I was reluctant to provide this information because about the only thing they could have done with it was to add it to the account that matched the phone number. I wasn't in the mood to engage in identity theft for a free pair of glassees, but the conversation was going in circles. Eventually another staff member observed the rising tension and offered to take care of this difficult situation. She took my phone number, and the address of the branch that had sent the text, said thank you for the notification and she would sort it out with the other branch. I was out of the store within 30 seconds of her taking over.
reply