ESO NA & EU Megaservers Offline - Dec 12

ZOS_GinaBruno · December 12

Hi everyone,

We are currently running into an issue where all ESO NA and EU megaservers, plus the ESO account page, are offline due to a datacenter power failure. We are working as quickly as possible to restore power and will let you know when we have an ETA.

Thanks for your patience!

ZOS_GinaBruno · December 12

While we continue to investigate, we want to let you know this outage is going to take a significant amount of time to recover from. At this time, we are estimating it could be at least 12 hours until we're up and running, however there's a chance it could take longer. The extent of this outage is unprecedented for ESO, and we will need to take time to ensure all realms come back online safely and efficiently. We’ll continue to provide updates in this thread as we have information to share and appreciate your patience.

ZOS_GinaBruno · December 12

Just to give everyone an update, some power has been restored to a number of critical pieces in the datacenter, but there is still a lot of work to be done. We'll plan to provide another update in a couple hours.

ZOS_Kevin · December 12

Hi all, just providing an update. We are still hard at work getting systems back online. Based on what we know right now, we believe the Megaservers will most likely be offline longer than the original 12 hour estimation. We hope to provide more clarity on timeframe once we have a little more time to complete more work.

Regarding the scope of work, this issue we ran into today was an edge-case emergency power outage at the data center that did not trigger standard backup failsafes for multiple tenants affected by the outage. (This type of outage is designed to cut ALL power in the event of a fire/flood scenario.) The outage now requires us to do a full reboot of our hardware while recovering from a full loss of power. Rebuilding piece by piece involves a methodical and lengthy process, including additional verification and testing as we bring the hardware online.

Hopefully this provides some clarity on the work happening right now. Thanks again for the continued patience.

ZOS_Kevin · December 13

Hi All, minor update as we are hearing back from some teams working on getting our hardware back online. Work is progressing smoothly right now. However, it's still too early to clock timing for when megaservers will be back online. We're hoping to get a bit more work done and checks verified before giving estimated times. We should know more in a few hours and provide an update then.

ZOS_GinaBruno · December 13

Update: We are still hard at work trying to get all live realms back up and running as soon as possible, and are making good progress on PC NA. The current plan is to open realms by platform (NA and EU together) as they come back online; the current order is not known but we will keep you posted.

Thanks for your continued patience and we'll check back in with everyone in a couple hours.

ZOS_GinaBruno · December 13

A bit more of a substantial update this time! We are making solid progress, though there is still a great deal of work to be done. We are anticipating work continuing through the night to get all Live realms tested and ready to be brought back online.

With that said, we are currently targeting opening PC NA and EU servers around 10am EST/3pm GMT on Friday, December 13 and will update on the status of console realms shortly after. The PTS will remain offline through the weekend with plans to bring it back online early next week.

We'll provide the latest status of everything in the morning (EST). The ongoing patience from everyone has been greatly appreciated.

ZOS_Kevin · December 13

Hi All. We have opened the PC NA and PC EU megaservers. We have been monitoring stability and keeping a close eye on things before issuing this wider all-clear on the realms.

The team is now working on the other platforms and anticipate, barring any unforeseen issues, opening these later today. Thank you again for your patience as we resolve this.

ZOS_GinaBruno · December 13

Thanks again for everyone's continued patience! As mentioned above, PC NA and EU are back online; please submit a ticket with Customer Support if you notice any issues with your character. Additionally, the ESO account page is also back online.

We are currently focusing on getting console realms tested and expect both Xbox and PlayStation servers to be back online later today. We'll keep you posted on status and will provide an ETA when we can.

ZOS_Amy · December 13

We have opened all of the Xbox and PlayStation realms. We will continue to monitor all realms for stability. Thank you for your patience as we have worked to bring the servers back online.