Solving this problem will require us to stop using the entire web as a source of information. Anyone can write anything and put it up on the web, and LLMs have no way to distinguish truth from fantasy.
Limiting responses to curated information sources is the way forward. Encyclopedias, news outlets, research journals, and so on.
No, they're not infallible. But they're infinitely better than anonymous web sites.
You are quite right, and not only can anyone write anything, but you have a double whammy from the LLM which can further hallucinate from said information.
We had this long time before, it is called "books". It is also not very usable for niche topics, or for recent events (because curators need time to react).
I wish there were some way to get something like Polis (https://pol.is/home) in place for web results. As screwed us as X/Twitter is, Community Notes (Polis-based) has real potential. It is much harder to game. There are so many times I wish I could "downvote" garbage Google results, and it seems to me with a good trust algorithm that, those who did this responsibly would be considered pretty good gatekeepers. Absent that, chaos might be better than a committee choosing for us. Right now, we have the worst of all worlds - a committee (Google) vs. a gagillion spammers.
Limiting responses to curated information sources is the way forward. Encyclopedias, news outlets, research journals, and so on.
No, they're not infallible. But they're infinitely better than anonymous web sites.