If you had three years of paid vacation where you had to use the language exclusively for everything, your skills would probably improve to toddler-level or beyond. Though admittedly you already know many things that they have to learn at the same time.
Having to communicate with your family (and probably close friends/significant other/etc.) exclusively in the language probably makes a big difference as well.
I think a lot of fluency is having someone carry you around 24/7 and talk to you like a baby for a few years. Much of the rest is just having patient people around you for another 3-4 years who will listen to you babble like an idiot and correct you when you're wrong.
I agree that if I could do nothing but spend 3 years in total immersion, I'd probably accumulate an adult-level vocabulary (again, the ramp-up is steep), but I still wouldn't sound like a native at that point, whereas a toddler would sound like a native...but have a toddler vocabulary.
Having to communicate with your family (and probably close friends/significant other/etc.) exclusively in the language probably makes a big difference as well.