Could you give a ballpark figure for what you mean by large scale scraping? I've... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

		monkeybutton on Feb 10, 2021 \| parent \| context \| favorite \| on: Web Scraping 101 with Python Could you give a ballpark figure for what you mean by large scale scraping? I've only worked on a couple projects, one was a broad (100K to 500K domains) and shallow (root + 1 level of page depth, also with a low cap on the number of children pages). The other just a single domain but scraping around 50K pages from it.

tluyben2 on Feb 10, 2021 | [–]

I would say millions of domains regularly. That's where the pricing of most 'scraping services' falls down too compared to just doing it yourself.

RhodesianHunter on Feb 10, 2021 | [–]

My experience was with e-commerce scraping. Not many domains, but a massive catalogue.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact