That screams "moral hazard"[1] to me. See also the incident with curl and AI confabulated bug reports[2].

[1]: Maybe not in the strict original sense of the phrase. More like, an incentive to misbehave and cause downstream harm to others. [2]: https://daniel.haxx.se/blog/2024/01/02/the-i-in-llm-stands-f...