It's been awhile since I used ESXi but for awhile I was running an ESXi6 server and I'd connect to a VM on the server from my linux desktop with VMware Player (I believe you officially needed workstation to do this but there was a command line backdoor). Then in the client there was a redirect USB device option.
I do the same thing now with qemu/kvm server. I just fire up virt-manager, open the VM I want to use and pick redirect USB device from the menu. Then I can select a local USB device and send it through. I haven't used it for much besides flash drives though. It requires a couple tweaks to the VM settings and I think it needs spice tools but that's expected, VMware needed VMware tools for this as well.
For what it's worth, you could also use USB over IP (http://usbip.sourceforge.net/) even if your virrualisation host doesn't support it. Watch out for authentication though, because the protocol doesn't care a whole lot about security. Might be worth the effort of setting up a wireguard/ipsec tunnel to secure the traffic.
Just running VMWare Fusion Pro on my M1 MBP. Choose connect to server -> enter IP address /login/password for your ESXi host. Launch virtual machine. Then you can simply tell Fusion to connect any USB devices to the remote VM, the same way you would if it were a local VM.
