Unless the system is under memory pressure, most shell initialization will read from in-memory OS file caches and not be noticeable as you note.

Where significant delays are often seen is when a seemingly innocuous extension uses network-based or some other heavy file system I/O commands (such as a "find $HOME -type f" type of thing).