Think more Roman empire with Cyrodiil in charge and the rest of Tamriel being the provinces, and various provinces like Summerset and Morrowind and eventually Skyrim really not appreciating that status.

you should check out the bard song "Red diamond" it explain why imperial are imperial.. orgianlly the wild elf rule the land, they keep other race as slave.. until the dragon god support the slave queen to overthrow their elven master, and seal the barrier of oblivion.

You could also check some in-game books on the subject, if you are interested in the lore.

You had like the Nedic humans which were probably like an earlier migration of Atmorans/proto-Nords who settled most of Cyrodiil/Hammerfell/Daggerfall. The Cyrodill Nedes were subjugated by Daedra worshipping High-Elves, the Ayleids, and made into their slaves for centuries.

The Ayleids worshipped all kinds of Daedra like Molag Bal and Meridia and the Aedric Gods of course didn't like that and thus sided with the enslaved Nedic humans to get them to rebel with Akatosh making the Nedic slave Alessia a Dragonborn, Kynareth sending the Nords under Morihaus(her son?) and Shor/Lorkhan(who really hates elves and likes humans) manifesting as Pelinal Whitestreak all to help the rebellion.

In the ensuing rebellion the Nedes succesfully overthrew their Ayleid overlords, probably killing most of them with the surviving Ayleids consequently serving under the Nedes who then created the Alessian empire ruled by Alessia and the heirs she had with Morihaus. These Nedes would henceforth be named Imperials, with the Imperials/Nedes to the west of the Imperial City mixing alot with their Nordic allies to become the Colovians and the Eastern/Southern Imperials who became Nibeneans mostly remaining racially the same but with a low-degree of mixing with Ayleids and later the Akaviri.

The Colovian branch of Imperials distinguishes itself by being highly religious and militaristic as well as having lighter complexions while the Nibeneans are more magic and trade oriented and have swarthier compexions.

Also the Ayleids would later be mostly exterminated by the Imperials when their empire was usurped by the theorcratic Alessian Empire whose doctrines included violently murdering every Elf in sight with only some Ayleids escaping and integrating into Wood Elf society.