Yes, every token is expanded into a vector that can be many thousand of dimensions. The vectors are stored for every token and every layer.

You absolutely can not fill even a single research paper in 2 GB much less an entire book.