You don't, for example, expect some ordinary farmer or tramp in the game world to know a lot about the (in-game) world or be capable of doing deep conversations about complicated topics.

So I don't think the necessary amount of text that you need to train the AI on is as insanely large as you imagine (but of course nevertheless a lot of texts have to be written - this is the price of having "much more dynamic" AI characters in the game).