Have you considered a DisplayPort hub? MST support has been a thing for a long time, it should work pretty well (particularly since this is basically how ~every thunderbolt & usbc dock works)
I have MST on my displays and it is a disaster. Also, considering the usecase where you use the VM to drive one of the displays which breaks the MST chain.
For this reason and how displayport/gfx-vendors handle unplugged cables I do prefer HDMI/DVI. Wish it wasn't the case.
Maybe a MST-hub could work well but I've heard of some of the difficulties and pecularities a simple KVM-switch can suffer from so I don't have high hopes for it.
