When you say “replacing mmap()”, could you elaborate a bit on it? The way you write it sounds like you’re describing a reimplementation of mmap() with the same API, while I believe the actual goal would be to completely rewrite the persistence and caching layer to be like a “real” database.
The implementation is essentially a complete replacement for the kernel page cache and I/O scheduler, much of the behavior of which is hidden behind the mmap() functions. It is never a drop-in replacement and you wouldn't want it to be but it is functionally quite similar.
For example, while the storage will usually have a linear address space, the "pointer" to that address space won't be a literal pointer even though it may behave much like one. There may be stricter invariants around write back and page fault behavior, and madvise()-like calls have deterministic effects. You often have cheap visibility into details of page/buffer state that you don't get with mmap() that can be used in program logic. And so on. Different but similar.
You have a file and a bunch of memory and you need to make sure data is being moved from file to memory when needed and from memory to file when needed.
mmap() is one algorithm to do it, and the idea is that it is not necessarily the best one.
Knowing more about your data and application and needs should theoretically enable you to design an algorithm that will be more efficient at moving data back and forth.