Maintenance for the week of May 18:
· [COMPLETE] NA megaservers for maintenance – May 18, 4:00AM EDT (8:00 UTC) - 9:00AM EDT (13:00 UTC)
· [COMPLETE] EU megaservers for maintenance – May 18, 8:00 UTC (4:00AM EDT) - 13:00 UTC (9:00AM EDT)
· [COMPLETE] NA megaservers for maintenance – May 18, 4:00AM EDT (8:00 UTC) - 9:00AM EDT (13:00 UTC)
· [COMPLETE] EU megaservers for maintenance – May 18, 8:00 UTC (4:00AM EDT) - 13:00 UTC (9:00AM EDT)
Is PC/NA down?
MreeBiPolar
✭✭✭✭✭
We've been in a delve, and all 3 people in a group lost connection simultaneously, can can't get back (three of us are all over the globe). Getting a variety of errors, and the ticker on login screen is "Could not retrieve announcements" for everyone.
Don't see any announcements other than the "store locked" though.
Edited by ZOS_GregoryV on December 13, 2024 2:58AM Don't see any announcements other than the "store locked" though.
1
-
allochthons✭✭✭✭✭
✭I can't log in to PS NA or EU.
I was on EU and got an endless loadscreen. Shut down my PS5 and rebuilt database.
Still can't log in anywhere.She/They
PS5/NA (CP3100+)0 -
BloodWolfe✭✭✭✭I posted in PC Tech Support forum, same issue here. Got booted, error 200 constantly. Account management page won't load. I was in Cyrodiil. Login screen also says "Could not retrieve announcements" as well.0 -
KiltMaster✭✭✭✭✭they're all down probably due to the crown store thingPC/NA
GM of "Kilts for Sale"
twitch.tv/thekiltmaster
He/Him1 -
ZOS_KevinCommunity ManagerUpdate on the server being down: https://forums.elderscrollsonline.com/en/discussion/670288/eso-na-eu-megaservers-offline-dec-12Community Manager for ZeniMax Online Studio and Elder Scrolls OnlineDev Tracker | Service Alerts | ESO Twitter1 -
barney2525✭✭✭✭✭
✭✭✭✭That's what happens when they don't buy the batteries for the toys at Christmas.
2 -
allochthons✭✭✭✭✭
✭But both NA and EU are down?
@ZOS_Kevin Can you put up the flair, so the server detectors etc update?
https://esoserverstatus.net/She/They
PS5/NA (CP3100+)1 -
Ardaghion✭✭✭✭Sounds like someone has a single point of failure problem.
I worked for Dish Network as an engineer before I retired; even as cheap as that company was, every critical system had a backup and all power had two rails. We had battery backup that could run all critical power for 30 minutes although the generators could switch in under 30 seconds.
We had the generators running for 3 days during a blizzard and customers never lost any satellite service. Well, the snow on the customer's dish probably cut service but they just needed a broom.5 -
sarahthes✭✭✭✭✭
✭✭✭Sounds like someone has a single point of failure problem.
I worked for Dish Network as an engineer before I retired; even as cheap as that company was, every critical system had a backup and all power had two rails. We had battery backup that could run all critical power for 30 minutes although the generators could switch in under 30 seconds.
We had the generators running for 3 days during a blizzard and customers never lost any satellite service. Well, the snow on the customer's dish probably cut service but they just needed a broom.
They almost certainly have generator/battery backups for the servers themselves.
Doesn't help if the entire neighbourhood has no power though.1 -
MreeBiPolar✭✭✭✭✭Update on the server being down: https://forums.elderscrollsonline.com/en/discussion/670288/eso-na-eu-megaservers-offline-dec-12ZOS_GinaBruno wrote:We are currently running into an issue where all ESO NA and EU megaservers, plus the ESO account page, are offline due to a datacenter power failure. We are working as quickly as possible to restore power and will let you know when we have an ETA.Sounds like someone has a single point of failure problem.
I worked for Dish Network as an engineer before I retired; even as cheap as that company was, every critical system had a backup and all power had two rails. We had battery backup that could run all critical power for 30 minutes although the generators could switch in under 30 seconds.
We had the generators running for 3 days during a blizzard and customers never lost any satellite service. Well, the snow on the customer's dish probably cut service but they just needed a broom.
They almost certainly have generator/battery backups for the servers themselves.
Doesn't help if the entire neighbourhood has no power though.
... That's one HUGE neighborhood to get both EU and NA out of power tho...3 -
allochthons✭✭✭✭✭
✭The UPS should run long enough to let the servers be shut down gracefully, complete with announcements in game.
And a single point of failure in one data center should not take out users in the other. I've been out of tech for a decade, and I know this.
I guess if it's the account server? But still, single point of failures should not exist in AAA game network/game hardware.She/They
PS5/NA (CP3100+)3 -
sarahthes✭✭✭✭✭
✭✭✭MreeBiPolar wrote: »Update on the server being down: https://forums.elderscrollsonline.com/en/discussion/670288/eso-na-eu-megaservers-offline-dec-12ZOS_GinaBruno wrote:We are currently running into an issue where all ESO NA and EU megaservers, plus the ESO account page, are offline due to a datacenter power failure. We are working as quickly as possible to restore power and will let you know when we have an ETA.Sounds like someone has a single point of failure problem.
I worked for Dish Network as an engineer before I retired; even as cheap as that company was, every critical system had a backup and all power had two rails. We had battery backup that could run all critical power for 30 minutes although the generators could switch in under 30 seconds.
We had the generators running for 3 days during a blizzard and customers never lost any satellite service. Well, the snow on the customer's dish probably cut service but they just needed a broom.
They almost certainly have generator/battery backups for the servers themselves.
Doesn't help if the entire neighbourhood has no power though.
... That's one HUGE neighborhood to get both EU and NA out of power tho...
It's likely the login server.0 -
noblecron✭✭✭✭✭They probably are running generators but probably to keep stuff running, working so stuff wont get lost?0 -
code65536✭✭✭✭✭
✭✭✭✭✭MreeBiPolar wrote: »... That's one HUGE neighborhood to get both EU and NA out of power tho...
EU is still online, and I know people who are still online on EU.
The problem is that the login server is in NA. So you can't log into EU, even though EU is up. But if you were already logged in, you will remain logged in. Just don't log out or crash, because you won't be able to get back in if you do.Nightfighters ― PC/NA and PC/EU
Dungeons and Trials:Personal best scores:- Dragonstar Arena: 46817 (NA)
- Maelstrom Arena: 600526 (NA)
- Blackrose Prison: 99274 (EU) (Unchained)
- Maw of Lorkhaj: 165227 (EU)
- Halls of Fabrication: 220708 (NA) (Tick-Tock Tormentor #1, #2, #3, #4, #5, #6, #7, #8, #9, #10, #11, #12)
- Asylum Sanctorium: 114957 (NA) (Immortal Redeemer #1, #2, #3, #4, #5, #6, #7, #8, #9, #10)
- Cloudrest: 131256 (NA) (Gryphon Heart #1, #2, #3, #4, #5, #6, #7)
- Sunspire: 250866 (EU) (Godslayer #1, #2, #3, #4, #5, #6, #7)
- Kyne's Aegis: 235841 (EU) (Dawnbringer #1, #2, #3, #4, #5, #6, #7, #8)
Dungeon trifectas:- Mountain God and Leave No Bone Unbroken
- Apex Predator and Pure Lunacy
- Depths Defier and Relentless Raider
- Defanged the Devourer and Nature's Wrath
- In Defiance of Death and No Rest for the Wicked
- Bane of Thorns and True Genius
- Ardent Bibliophile and Subterranean Smasher
- Battlespire's Best and Bastion Breaker
- Zero Regrets and Land, Air, and Sea Supremacy
- Fist of Tava and Invader's Bane
- Curator's Champion (Valinna) and Temporal Tempest
- Unshakeable Fervor and Lighting the Embers
Extended Journal Add-Ons: Item Set Browser ― Loot Log ― Character Knowledge ― Collectibles Tracker ― RaidificatorPC/Console Add-Ons: Combat Alerts ― Group Buff PanelsOther Add-Ons: Deconstruction Junk Marker ― Purge Tracker ― Asylum Status Panel ― Smart Looter ― Improved Companion Rapport Information5 -
Ardaghion✭✭✭✭Sounds like someone has a single point of failure problem.
I worked for Dish Network as an engineer before I retired; even as cheap as that company was, every critical system had a backup and all power had two rails. We had battery backup that could run all critical power for 30 minutes although the generators could switch in under 30 seconds.
We had the generators running for 3 days during a blizzard and customers never lost any satellite service. Well, the snow on the customer's dish probably cut service but they just needed a broom.
They almost certainly have generator/battery backups for the servers themselves.
Doesn't help if the entire neighbourhood has no power though.
We had an entire battery room in the basement to power the entire site, the lights and other non-critical systems would go down until the generators kicked in and were stable. That means being completely independent of the neighborhood. We had millions of customers and they get really irate when they lose their television channels.1 -
ZOS_KevinCommunity Manager12 hours to turn the power back on!
Yikes.
It's not really to turn the power back on. It's more like to make sure we do all of the checks that need to happen as we fire things back up in the order they need to be turned on in. So not as simple as flipping a switch to turn things back on.Community Manager for ZeniMax Online Studio and Elder Scrolls OnlineDev Tracker | Service Alerts | ESO Twitter18 -
code65536✭✭✭✭✭
✭✭✭✭✭12 hours to turn the power back on!
Yikes.
If the power went out suddenly and the servers were shut down unsafely, there could be the potential for data corruption and I imagine they would likely want to verify data integrity and that sort of thing.Nightfighters ― PC/NA and PC/EU
Dungeons and Trials:Personal best scores:- Dragonstar Arena: 46817 (NA)
- Maelstrom Arena: 600526 (NA)
- Blackrose Prison: 99274 (EU) (Unchained)
- Maw of Lorkhaj: 165227 (EU)
- Halls of Fabrication: 220708 (NA) (Tick-Tock Tormentor #1, #2, #3, #4, #5, #6, #7, #8, #9, #10, #11, #12)
- Asylum Sanctorium: 114957 (NA) (Immortal Redeemer #1, #2, #3, #4, #5, #6, #7, #8, #9, #10)
- Cloudrest: 131256 (NA) (Gryphon Heart #1, #2, #3, #4, #5, #6, #7)
- Sunspire: 250866 (EU) (Godslayer #1, #2, #3, #4, #5, #6, #7)
- Kyne's Aegis: 235841 (EU) (Dawnbringer #1, #2, #3, #4, #5, #6, #7, #8)
Dungeon trifectas:- Mountain God and Leave No Bone Unbroken
- Apex Predator and Pure Lunacy
- Depths Defier and Relentless Raider
- Defanged the Devourer and Nature's Wrath
- In Defiance of Death and No Rest for the Wicked
- Bane of Thorns and True Genius
- Ardent Bibliophile and Subterranean Smasher
- Battlespire's Best and Bastion Breaker
- Zero Regrets and Land, Air, and Sea Supremacy
- Fist of Tava and Invader's Bane
- Curator's Champion (Valinna) and Temporal Tempest
- Unshakeable Fervor and Lighting the Embers
Extended Journal Add-Ons: Item Set Browser ― Loot Log ― Character Knowledge ― Collectibles Tracker ― RaidificatorPC/Console Add-Ons: Combat Alerts ― Group Buff PanelsOther Add-Ons: Deconstruction Junk Marker ― Purge Tracker ― Asylum Status Panel ― Smart Looter ― Improved Companion Rapport Information5 -
Jaraal✭✭✭✭✭
✭✭✭✭✭It's not really to turn the power back on. It's more like to make sure we do all of the checks that need to happen as we fire things back up in the order they need to be turned on in. So not as simple as flipping a switch to turn things back on.If the power went out suddenly and the servers were shut down unsafely, there could be the potential for data corruption and I imagine they would likely want to verify data integrity and that sort of thing.
So no backup batteries or generators were available for a controlled shutdown? And if so, then possible rollbacks?
Kevin, how often do you guys make data backup points? Or is that classified information?
0 -
Gerwulf67✭✭✭I knew I shouldn't have skipped buying the Datacenter Power Core in the Crown Store. My fault, all.8 -
Chufu✭✭✭✭✭
✭✭12 hours.... euuuuuuurgh- @OrcTalk on Twitter
- ESO-Skillfactory.com[EN/DE/FR] Skillplanner/Skillplaner/Planificateur de compétences
0 -
Shelgra✭✭they were supposedly taken by "deprived" lol, this kind of disaster can be prevented in advance especially with data centers0
-
Malkosha✭✭✭12 hours to turn the power back on!
Yikes.
It's not really to turn the power back on. It's more like to make sure we do all of the checks that need to happen as we fire things back up in the order they need to be turned on in. So not as simple as flipping a switch to turn things back on.
This makes sense. I'm a retired IT Geek and depending on the systems, they have to be brought up in a certain order and even then, each step had to be analyzed before the next system can come online. Shortcut the process and you could have errors which would be hard to trace ... which of course, would force them to start all over.The key point is to not lose any data and if they manage that I'd be impressed.
Saying that, ESO has a great up-time overall. I guess it time to walk on grass. For those that are new to this, don't be afraid of that big yellow thing in the sky. It's just our friend, Mr Sun.4 -
code65536✭✭✭✭✭
✭✭✭✭✭@ZOS_Kevin Would it be possible to the login servers back online sooner than the 12 hour mark, so that EU can log in (since the actual EU megaservers weren't directly affected by this) before the daily resets?Nightfighters ― PC/NA and PC/EU
Dungeons and Trials:Personal best scores:- Dragonstar Arena: 46817 (NA)
- Maelstrom Arena: 600526 (NA)
- Blackrose Prison: 99274 (EU) (Unchained)
- Maw of Lorkhaj: 165227 (EU)
- Halls of Fabrication: 220708 (NA) (Tick-Tock Tormentor #1, #2, #3, #4, #5, #6, #7, #8, #9, #10, #11, #12)
- Asylum Sanctorium: 114957 (NA) (Immortal Redeemer #1, #2, #3, #4, #5, #6, #7, #8, #9, #10)
- Cloudrest: 131256 (NA) (Gryphon Heart #1, #2, #3, #4, #5, #6, #7)
- Sunspire: 250866 (EU) (Godslayer #1, #2, #3, #4, #5, #6, #7)
- Kyne's Aegis: 235841 (EU) (Dawnbringer #1, #2, #3, #4, #5, #6, #7, #8)
Dungeon trifectas:- Mountain God and Leave No Bone Unbroken
- Apex Predator and Pure Lunacy
- Depths Defier and Relentless Raider
- Defanged the Devourer and Nature's Wrath
- In Defiance of Death and No Rest for the Wicked
- Bane of Thorns and True Genius
- Ardent Bibliophile and Subterranean Smasher
- Battlespire's Best and Bastion Breaker
- Zero Regrets and Land, Air, and Sea Supremacy
- Fist of Tava and Invader's Bane
- Curator's Champion (Valinna) and Temporal Tempest
- Unshakeable Fervor and Lighting the Embers
Extended Journal Add-Ons: Item Set Browser ― Loot Log ― Character Knowledge ― Collectibles Tracker ― RaidificatorPC/Console Add-Ons: Combat Alerts ― Group Buff PanelsOther Add-Ons: Deconstruction Junk Marker ― Purge Tracker ― Asylum Status Panel ― Smart Looter ― Improved Companion Rapport Information0 -
Firey_Hellhound✭✭✭Well shoot, both EU and NA for a single power outage? I totally understand it will take a long time to recheck everything and restore it safely, been there done that before myself so if you an do it in 12 hours will be excellent! So no daily login then today, for anyone not having claimed it yet, can we at least get the daily login please? Not much but still...0
-
ZOS_KevinCommunity Manager@ZOS_Kevin Would it be possible to the login servers back online sooner than the 12 hour mark, so that EU can log in (since the actual EU megaservers weren't directly affected by this) before the daily resets?
All of that depends on how checks go as things come back online. But to be frank, we noted in our update "that at least 12hrs" because there is the possibility it takes longer. So can't make any promises on timeline.Community Manager for ZeniMax Online Studio and Elder Scrolls OnlineDev Tracker | Service Alerts | ESO Twitter5
This discussion has been closed.