But it does not offer me a clickable "Play" button for exclusively Windows games, unfortunately. Or is this something else? Or perhaps I have to do some configuration of some sort? I am really not sure. It works for games that have Windows + Proton (icon / logo), but the games I want have only windows.
It does, you'll probably have to enable proton in Steam settings. Also, in the store pages, you can see Steam Deck compatibility rating (and details about that) which means linux in practice. Some warnings regarding small text in some games don't apply to bigger displays, of course.
It works out of the box here? Steam supports running most "Windows-only" games on Linux without trouble; you may just have to select the Proton version once.
I don't remember when I last encountered a game that didn't run. I'm sure those exist, mind. Perhaps I've just been lucky.