Yes, theorizing a future possibility that has no president is, by definition, fantasy. I mean, you can do fantasy, it makes good science fiction, but until we have evidence that von Neumann probes exist and are capable of colonizing the entire galaxy, it remains fantasy. Theoretically we could build a high speed rail between Seattle and LA, we could build a permanent settlement on Antarctica, and we could replace our jet-planes with hydrogen powered flying wings, but until we do, a world with those things are just fantasy.

The only argument for von Neumann probes that I can think of is as a specific answer to the Fermi paradox. The universe should be filled with these probes, but since it obviously isn’t, we can infer that no civilization has reach the interstellar age.

I reject this framing, the Fermi paradox is only a paradox if you assume that space colonization is a thing that is not just possible, but inevitable. My solution to the Fermi paradox simply rejects this assumption. Civilization will not colonize definitely, they will do their space explorations with telescopes and targeted probes, and they won‘t build any von Neumann probes (at least not ones continue to replicate for dozens millennia).

Aside: I am aware of the irony that my description of civilizations outside of our solar system is also a fantasy.