That's because empires don't work. In order to make them work what's needed is to have the center of the empire maintain infrastructure on the borders of the empire. The center grows when you get an empire, but ... it's an absurdly small growth compared to the border growth. Hence empires exhaust themselves attempting to guard borders and you start seeing absurdities like military fortresses manned by 5 unarmed (because too expensive) soldiers. Both the English and Roman empires did that. And then they abandon their borders to save some more money, and it all just ... fades away.

And this is a cursed choice because empires need resources (as they will find themselves in a war with just about everyone else at some point, so imports don't work). Those resources are only available in far away mines. So you need to have the huge area and borders, and infrastructure everywhere..

But you can't have the huge area and borders, and infrastructure because you can't defend it, you can't build, you can't pay for it.

So ... no empires. Or at least, no permanent ones. People keep trying though.