you should try just to give an instruction like, when you're inquired about non-dietary related questions, you might entertain chit-chat and barter but try to steer the conversation back to dietary / healthy lifestyle, at the end of the day the context is king, if something is not in context the llm can infer by the lack of it, that its not -programmed- to do anything else.

these are funny systems to work with indeed