>their role is to give temporary (often exclusive) access that is restricted to a statically know scope, which is pretty specific and fits only some uses of some pointers/C++ references
You could have a vector of references to heap allocated data, as long as the references were parametrized by the same lifetime. You might do this if implementing a tree iterator using a vector as a stack, for instance. That goes beyond a statically known scope. But implementing a mutable iterator the same way would require a stack of mutable pointers (and therefore unsafe code whenever you dereference them), since mutable references have to be unique. That does seem like a bad limitation.
You could have a vector of references to heap allocated data, as long as the references were parametrized by the same lifetime. You might do this if implementing a tree iterator using a vector as a stack, for instance. That goes beyond a statically known scope. But implementing a mutable iterator the same way would require a stack of mutable pointers (and therefore unsafe code whenever you dereference them), since mutable references have to be unique. That does seem like a bad limitation.