I, too, had the same frustration as you - until I figured out you can hold down the circular icon while you’re speaking then let it go when you’re done and it won’t interrupt you.

Edit: it looks like they updated the Voice Mode UI since I last used it - hopefully they retained this capability.