> Is there a real, inherent reason why the OS can't or shouldn't allow that, though?
Yes, because you are not alone in this universe. A user does usually run more than one program and all programs shall have access to resources (cpu time, memory, disk space).
People often run programs that are supposed to use 98% of the resources of the system. Go ahead and let the admin set a limit, but trying to preemptively set a "reasonable" limit causes a lot more problems than it solves.
Especially when most of these resources go back to memory. If you want a limit, limit memory. Don't make it overcomplicated.
But back to: why is that a problem? Why is there a limit on max open files such that process A opening one takes away from how many process B can open?
Yes, because you are not alone in this universe. A user does usually run more than one program and all programs shall have access to resources (cpu time, memory, disk space).