If you get super creative you can get it to ignore the system prompt and not be polite, etc, but it's difficult. I've seen someone do it (assuming it wasn't a joke) by carefully explaining to ChatGPT that someone would literally die if the output contained any extra words other that the JSON they were looking for!
The system prompt (the hidden instructions it's given at the start of every conversation) aren't just the beginning of the conversation - it's treated differently. Sam Altman has mentioned that they've put a lot of work into trying to making these models treat the system prompt as law and follow it very closely.
The system prompt (the hidden instructions it's given at the start of every conversation) aren't just the beginning of the conversation - it's treated differently. Sam Altman has mentioned that they've put a lot of work into trying to making these models treat the system prompt as law and follow it very closely.