> yet they allow it because it's their SV buddies.
At one time, Google used to ban sites for "cloaking": offering one version of the page to the crawler, and another to the user.
But over time they got into trouble from these sites, getting sued left and right. Being accused of putting up a wall, and abusing their monopoly. Eventually, these sites won out and Google dropped this requirement.
Similar story with Image search. "Blah blah something something copyright something". And voila - now you can't get direct access to images on the search results page.
This might be the case for news sites which show limited content until you sign into a paywall, but I don't recall the biggest offenders which show NO relevant content at all (LinkedIn and Facebook) saying anything.
At one time, Google used to ban sites for "cloaking": offering one version of the page to the crawler, and another to the user.
But over time they got into trouble from these sites, getting sued left and right. Being accused of putting up a wall, and abusing their monopoly. Eventually, these sites won out and Google dropped this requirement.