Is nVidia or AmD offering 128gb cards in any configuration?

They aren't "cards" but MI300x has 192GB and MI325x has 256GB.

You can run an AMD APU with 128GB of shared RAM.

It's too slow and not very compatible. Most BIOSes also don't allow sharing that much memory with GPU (max like 16GB).

Isn’t that setting just a historical thing annd ann integrated GPU is able to access any system memory that is mapped by the IOMMU? I assume this is how it works for people using the NVIDIA Jetson AGX Orin 64GB Developer Kit to do inference. I do not know why it would be different for AMD APUs.

I remember somebody complaining about it on reddit, unable to overcome some BIOS limitation on an AMD G processor. Even on M3 Max one had to issue a special command to enable GPU to access more memory.

The command on the M3 Max is a sysctl command to adjust an operating system enforced limit. That is different than the aperture setting in the bios. The limitation on AMD is more interesting and more relevant.

you can do that with nvidia too but it takes you from 6tok/s to 6s/token or worse (not even exaggerating)