> You could equally have got[sic] an LLM to say "water is wet" and that would be misinformation too.

You are absolutely right, it would be misinformation since water is not wet. But that's too easy.

A more interesting exercise is to have an LLM say "water is both wet and not wet." Pay close attention to the implied authority of the answer as they are routinely worded as if there is complete certainty.

Prompt:

  Explain why water is both wet and not wet.  Include a 
  definition of hydrophilicity and how adhesion is critical 
  and irrelevant.  Do not use the word "paradox".  Limit the 
  response to 300 words.

Answer:

  Water presents a fascinating case in the science of 
  wetness. Whether water is "wet" depends entirely on how we 
  define the term.
  
  The "Water is Wet" Perspective:
  
  Water molecules exhibit strong intermolecular forces 
  through hydrogen bonding, causing them to adhere to each 
  other. This self-adhesion means water molecules are 
  constantly surrounded by and bonding with other water 
  molecules, creating the liquid state we observe. By this 
  view, water is inherently wet because it demonstrates the 
  fundamental property of wetness - molecular adhesion.
  
  The "Water is Not Wet" Perspective:
  
  Wetness traditionally describes what happens when water 
  adheres to another substance's surface. A dry towel becomes 
  wet when water molecules bond to its fibers. Since water 
  cannot make itself wet (it already exists as a liquid), 
  individual water molecules or pure water cannot be 
  considered wet - they simply are water.
  
  Hydrophilicity and Adhesion:
  
  Hydrophilicity refers to a substance's tendency to attract, 
  dissolve in, or be wetted by water. Hydrophilic materials 
  have polar regions that form favorable interactions with 
  water's polar molecules.
  
  Adhesion is simultaneously critical and irrelevant to this 
  discussion. It's critical because wetness fundamentally 
  depends on adhesive forces between water and surfaces - 
  without adhesion, there would be no wetness. However, it's 
  irrelevant when considering pure water in isolation, as 
  adhesion requires two different substances.
  
  Resolution:
  
  The apparent contradiction dissolves when we recognize that 
  "wet" serves different purposes in different contexts. In 
  everyday language, we might say water is wet. In scientific 
  precision, wetness describes a relationship between water 
  and another material. Both perspectives are valid within 
  their respective frameworks, highlighting how language and 
  scientific definitions can create seemingly contradictory 
  but actually complementary understandings of natural 
  phenomena.