AI crawlers have lead to a big surge in scraping/crawling activity on the web, and many don't use proper user agents and don't stick to any scraping best practices that the industry has developed over the past two decades (robots.txt, rate limits). This comes with negative side effects for website owners (costs, downtime, etc.), as repeatedly reported on HN (and experienced myself).
Do you have any built-in features that address these issues?
I work in the adtech ad verification space and this is very true. the surge in content scraping has made things very very hard in some instances. I can’t really fault the website owners either.
Do you have any built-in features that address these issues?