I'm opening this discussion here as well since me and several of my guild mates have this same exact issue:
When I try to login it takes long time to get in because I get 'Connection to server timed out'. Sometimes it takes between 15-30 minutes especially later in the evening and this has been only getting worse for past 6 months.
I have the new Xbox one X and I've tried all the steps with both consoles: resetting everything (including secondary mac address), tried re-installing the game (tried both internal and external harddrives), opened appropriate ports on my router, cleared console cache by cycling power off and removing the cable for 30 minutes, deleting game save file 'Delete everywhere'. I've done everything and more that Zenimax has told me to do several times, yet none of these have any effect to the issue.
I've checked that there is no ESO server or Xbox Live outages either.
So what me and several of my mates noticed due to new Xbox One X -consoles: we usually have created another Xbox Live account (e.g. brother/sister, gf's etcetera) to the old Xbox One's - so we've had the opportunity to create a new ESO accounts/characters as well. So we tried login from these Xbox Live accounts (some new, some old) and to our complete dismay we were able to login just fine from 'new' ESO accounts - without connection timing out on login.
This is clearly an issue for anyone with older account where you probably have several characters/items in inventories&bank/houses&furniture and plenty of ingame achievements etcetera.
We all have tried several times to login at same time on both consoles and the 'new' account logs in without a fail, every single time when old accounts keep timing out.