The failure mode here (Claude trying to satisfy rather than saying 'this is impossible with the constraints') shows up everywhere. We use it for security research - it'll keep trying to find exploits even when none exist rather than admit defeat. The key is building external validation (does the POC actually work?) rather than trusting the LLM's confidence.
Ah! I see the problem now! AI can't see shit, it's a statistical model not some form of human. It uses words, so like humans, it can say every shit it wants and it's true until you find out.
The number one rule of the internet is don't believe anything you read. This rule was lost in history unfortunately.
When reasoning about sufficiently complex mechanisms, you benefit from adopting the Intentional Stance regardless of whether the thing on the other side is "some form of human". For example, when I'm planning a competitive strategy, I'm reasoning about how $OTHER_FIRM might respond to my pricing changes, without caring whether there's a particular mental process on the other side
