The article says they got datasets from Anna's Archive. It was most likely the scihub/libgen torrent which is 96.0 TB right now and contains 92,872,581 files. That's about 1 megabyte per file.

https://annas-archive.org/datasets

Where does one find these torrent datasets? Did they download the books in bits and pieces or as a single huge multi-TB file?

I’ve got 70-80mb pirated books, I think because of the illustrations. Guess it depends on the book.

I don’t think they’re using picture heavy book for LLM training, no?

Just because the LLMs are trained on text doesn't mean that images we're a part of what they downloaded.

You clean up the data after you acquire it, not before.

Even if they didn't use the illustration(which isn't clear given multimodal models), they'd still make use the text in the books.

Presumably they didn't create the torrent

Whoever created it has a lot of spare hard disk space.

100TB is like 6 hard drives...

> 100TB is like 6 hard drives...

Discounted Seagates ? /s

You can get recertified 18TB drives, but still it's a lot of disk space. I simply don't have enough data.

Yes they do, there's multimodal models.

I don't think they need to be selective. It's not like Meta can run out of storage.

For multi-modal models, why not? They would be probably some of the best data.

Sometimes the PDF of a book is big because the book's packed with important illustrations and charts - like a textbook or journal paper.

Other times a PDF of a book is big because someone scanned it and didn't have trustworthy OCR, so they figured distributing images of text at 1.5 MB per page was better than risking OCR errors.

Why not ? Do you think that AI doesn't enjoy porn ? /s

It could be anywhere from a few million to a hundred million

https://annas-archive.org/datasets