Emergency Server Downtime Hangout Thread 12/12/2024

rothan117 · December 13

The amount of time this is taking to remedy and the fact that it took down data centers on two continents suggests to me that whether the initial cause was a power failure or not, that there is much more extensive damage that is not easily fixable.

Kazren · December 13

December 13th, day two of outage. Is there any indication that subscribers will be given extensions to current subscription days due to our inability to play?

Kazren · December 13

Stormshaper wrote: »

Weird coincidence maybe but my Amazon Music service was a complete fail for a hour or more

I still have Amazon music, so maybe you need to reinstall your app.

Sakiri · December 13

Xinihp wrote: »

Sakiri wrote: »

In the case of fire and/or flood there *isn't* time to do a graceful shutdown. It goes off NOW to prevent data loss among other things, even though it doesn't always work.

In the case of fire, spraying ANYTHING onto a burning server with the power on is risking electrocution, particularly water which they'd likely be using because that's what they use on building fires.

Like I said, not really my field. Although how long would it take from issuing a shutdown command to it being off? Also even at the relatively small corporations where I have managed servers, the standard go-to was always Halon. Halon suppression systems became widely popular because Halon is a low-toxicity, chemically stable compound that does not damage sensitive equipment, documents, and valuable assets.

I have NEVER heard of a server room using water as a fire suppression system. That just seems insane lol! xD

Fire trucks on a burning building. Not a server room sprinkler system.

DragonRacer · December 13

Honestly, all of this just sounds really bad. Especially the longer it takes. BUT no sense in rushing a fix only for servers to come up corrupted/missing stuff and having to come right back down again.

I hope our digital buddies will be okay/restored.

Feel really bad for the ZOS folks having to try and recover all this, through no fault of their own but rather someone else’s crisis having ripple effects.

EIGHTS · December 13

Oh my god, the server has been down for almost a whole day. Will the data be lost? After all, it is the database hard drive that is broken. Twenty hours ago, I guessed it wouldn't be fixed so quickly, and I was right.

BelmontDrakul · December 13

Kazren wrote: »

December 13th, day two of outage. Is there any indication that subscribers will be given extensions to current subscription days due to our inability to play?

Yeah, Hollowjack hacked the servers.

Xinihp · December 13

BelmontDrakul wrote: »

Eventhough, I know nothing, only logical solution seems cutting power and try to transfer air to the outside to create a vacuum environment to extinguish fire. Is this how it works in server rooms?

I'm sure things are changing. Back in the early 2000's Halon was the standard for fire. It worked opposite how you described. The room was sealed, oxygen was removed, and replaced with Halon gas, which smothered the fire, now starved of fuel.

However I believe Halon is identified as harmful to the ozone and even though fire events are rare enough that it is likely a minor contributor to climate change in the grand scheme, it is probably being phased out in a lot of places.

However, I can't imagine that it would be replaced with water. Even if the power is completely off, hosing down computer equipment is not really a viable option. Getting liquid deep into these components will be extremely difficult and costly to dismantle, dry, and reassemble.

I also am having a difficult time imagining a datacenter that doesn't have a separate response for fire and flooding. There should be two separate sensors and emergency protocols connected to either. In the case of actual flooding it could make sense to do the hard shut down instantly as water is already present. But for fire, especially if some form of gas is used as in most cases to smother the flames, there should be time to do a safe shutdown since water is not present.

BelmontDrakul · December 13

SERVERS ARE ONLINE, GO GO GO!!!

Xinihp · December 13

BelmontDrakul wrote: »

SERVERS ARE ONLINE, GO GO GO!!!

I vote they should do a crown sale immediately to thank us for our understanding.

Frogwilliams · December 13

BelmontDrakul wrote: »

SERVERS ARE ONLINE, GO GO GO!!!

Servers are definitely not online.

Correction: Despite the launcher and account signon pages showing messages the servers are still down you can actually log in.

arena25 · December 13

THEY ARE UP. THE GREEN FLAG DROPS.

May we all log in to find our characters were left unscathed.

Alinhbo_Tyaka · December 13

BelmontDrakul wrote: »

Xinihp wrote: »

Sakiri wrote: »

In the case of fire and/or flood there *isn't* time to do a graceful shutdown. It goes off NOW to prevent data loss among other things, even though it doesn't always work.

In the case of fire, spraying ANYTHING onto a burning server with the power on is risking electrocution, particularly water which they'd likely be using because that's what they use on building fires.

Like I said, not really my field. Although how long would it take from issuing a shutdown command to it being off? Also even at the relatively small corporations where I have managed servers, the standard go-to was always Halon. Halon suppression systems became widely properly because Halon is a low-toxicity, chemically stable compound that does not damage sensitive equipment, documents, and valuable assets.
I have NEVER heard of a server room using water as a fire suppression system. That just seems insane lol! xD

Eventhough, I know nothing, only logical solution seems cutting power and try to transfer air to the outside to create a vacuum environment to extinguish fire. Is this how it works in server rooms?

They usually have some type of a chemical flooding system such as CO2 or whatever the current Halon replacement is. They are very effective at putting out fires with minimal damage to equipment. However when the klaxon goes off you run as not only do the gasses replace all of the breathable air they come out with such a force that they blow off floor tiles and most anything not strapped down goes flying.

SteveCampsOut · December 13

BelmontDrakul wrote: »

SERVERS ARE ONLINE, GO GO GO!!!

No they are not up. Not from where I'm sitting anyway.

ssewallb14_ESO · December 13

They're definitely up. Seems some can't login though.

Calastir · December 13

ssewallb14_ESO wrote: »

They're definitely up. Seems some can't login though.

ssewallb14_ESO · December 13

Calastir wrote: »

ssewallb14_ESO wrote: »

They're definitely up. Seems some can't login though.

I mean I just cycled through 6 alts and there are lots of players running around the major hubs.

Maybe you drew the short straw?

LootAllTheStuff · December 13

rothan117 wrote: »

The amount of time this is taking to remedy and the fact that it took down data centers on two continents suggests to me that whether the initial cause was a power failure or not, that there is much more extensive damage that is not easily fixable.

The EU server was taken down because it uses the same account login server as the NA one, and is housed in the affected data centre.

ZOS_Kevin · December 13

Hi All, just updated here: https://forums.elderscrollsonline.com/en/discussion/comment/8236703/#Comment_8236703

Gingaroth · December 13

Reginald_leBlem wrote: »

redlink1979 wrote: »

I'll go with 7. I'm getting a lot done around the house LOL

I'm going with 1, for similar reasons. I just did some tidying and inventory chores I've been putting of for months, so to start, I have the satisfaction of the job finally being done.

On top of that I found a stack of canned cod livers I forgot I had. They all have to be eaten this month. By me, because I'm the only one around who likes them. And not a little either, I love them.

So maybe I'm even more of a : Pacrooty who just found a sweetroll.

TomCat · December 13

@ZOS_GinaBruno , @ZOS_Kevin
Thank you to everyone involved for their efforts. It was certainly not an easy job under difficult conditions.
I wish you a relaxing weekend, which is truly earned.

Iriidius · December 13

BelmontDrakul wrote: »

Proxithe wrote: »

[snip]

The thing I cannot understand is if there is not a backup server besides the one in Texas? Trying to operate an MMORPG without backup server is huge risk. AFAIK even some private server operators have backup servers. I don't get it.

A backup server would also need to have same save files as main server the moment it crashed requiring constant actualization.
A backup server having save files from a day ago letting players play without last day progress would create more problems than it solves as players would loose progress or you would have to merge progress from both somehow which seems impossible and even if it is possible many players would have used that time to repeat lost progress or otherwise get confudes/distracted wasting time if they do not know it and what gets restored.
I wonder what happened to old servers they replaced last year, maybe they would be potential backup server. Does ZOS still own them?
The long time ZOS needed to upgrade PC EU and console servers after they upgraded PC NA shows that getting servers is not quick and easy for ZOS.
(was about half a year as far as I remember)

[edited to remove quote]

Alinhbo_Tyaka · December 13

Iriidius wrote: »

BelmontDrakul wrote: »

Proxithe wrote: »

[snip]

The thing I cannot understand is if there is not a backup server besides the one in Texas? Trying to operate an MMORPG without backup server is huge risk. AFAIK even some private server operators have backup servers. I don't get it.

A backup server would also need to have same save files as main server the moment it crashed requiring constant actualization.
A backup server having save files from a day ago letting players play without last day progress would create more problems than it solves as players would loose progress or you would have to merge progress from both somehow which seems impossible and even if it is possible many players would have used that time to repeat lost progress or otherwise get confudes/distracted wasting time if they do not know it and what gets restored.
I wonder what happened to old servers they replaced last year, maybe they would be potential backup server. Does ZOS still own them?
The long time ZOS needed to upgrade PC EU and console servers after they upgraded PC NA shows that getting servers is not quick and easy for ZOS.
(was about half a year as far as I remember)

I don't think they have been referring to a backup server but are thinking of a failover server. It is something fairly common in the Enterprise space but I'm not sure how feasible it is for something like ESO which is highly customized and would require mirroring a completely separate system/s in real time. The game has enough performance issues and I'd hate to think how much worse this would be made by real time synchronization.

[edited to remove quote]

Cin_Vhetin · December 13

merevie wrote: »

Thanks to those staying up all night.

arena25 wrote: »

Yes, my appreciation to all of the ZeniMax employees working double/triple OT to fix this - you've earned this weekend. And we owe you a round of drinks.

Hey devs,
Here's a coffee and sweetroll for all your hard work

ZOS_Icy · December 13

Greetings,

As we've removed a few comments that were bashing, trolling and conspiracy theories, this is a friendly reminder that comments need to adhere to our Community Rules to avoid thread derailment.

Bashing and Slanderous Comments: We do not permit the bashing of individuals (including ZeniMax employees), groups, or other companies on our forums. We believe that doing so is neither constructive nor in spirit of our game and community.

Conspiracy Theories and Misinformation: We do not tolerate the deliberate and malicious spreading of false info or conspiracy theories about ZeniMax Online Studios, its game(s), or its employees. This does not include honest mistakes; rather, this rule pertains to those who go out of their way to spread harmful or malicious rumors about ZeniMax Online Studios and its employees, or The Elder Scrolls Online.

Trolling or Baiting: The act of trolling is defined as something that is created for the intent to provoke conflict, shock others, or to elicit a strong negative or emotional reaction. It’s okay and very normal to disagree with others, and even to debate, but provoking conflict, baiting, inciting, mocking, etc. is never acceptable in the official The Elder Scrolls Online community. If you do not have something constructive or meaningful to add to a discussion, we strongly recommend you refrain from posting in that thread, and find another discussion to participate in instead. It is also not constructive or helpful to publicly call out others and accuse them of trolling, or call them a troll—please refrain from doing so. If you genuinely believe someone is trolling, please report the post or thread to the ESO Team, and leave it at that.

Thank you for your understanding.

Kazren · December 13

If you're reading this maybe you don't know the server is up.

Ingenon · December 13

4 console servers are down. 2 PC servers are up. All ESO console players are still waiting.

PC EU Megaserver Online
PC NA Megaserver Online
PC PTS Megaserver Online
XBOX EU Megaserver Offline
XBOX NA Megaserver Offline
PlayStation EU Megaserver Offline
PlayStation NA Megaserver Offline
Current time UTC:
December 13, 2024 6:52 PM
Your time (GMT-5):
December 13, 2024 1:52 PM

spartaxoxo · December 13

Ingenon wrote: »

4 console servers are down. 2 PC servers are up. All ESO console players are still waiting.

PC EU Megaserver Online
PC NA Megaserver Online
PC PTS Megaserver Online
XBOX EU Megaserver Offline
XBOX NA Megaserver Offline
PlayStation EU Megaserver Offline
PlayStation NA Megaserver Offline
Current time UTC:
December 13, 2024 6:52 PM
Your time (GMT-5):
December 13, 2024 1:52 PM

Why is the PTS us up before console????

Kram8ion · December 13

Wait where’d everybody go?:[

Iriidius · December 13

SeaGtGruff wrote: »

Iriidius wrote: »

ZoS should extend the time of endeavours and login reward by 24 hours and give tomorrows login reward and seals of endeavours extra for free.

It is unavoidable that players are unable to play when the server are down but not OK to still expect players to login to get their daily reward and endeavour seals when the servers are down.

In germany/frankfurt where EU server is located it was 6pm when the server went down and they will not be up until after reset at 4am. Most Europeans are not playing before that time.

Honestly, I don't think they should, as it would just set an expectation for something similar whenever there's an issue.

Mind you, I didn't play on EU last night after reset, and didn't play on NA this morning after reset, so I'm probably going to lose at least a day of daily login rewards, seals from daily endeavors, rewards from Tales of Tribute, gold from crafting, and having fun as its own reward, on both servers.

But I've been through local power outages, local internet outages, and game server downtimes before, and survived missing one or more days of game time.

I just hope the people working to get things fixed ASAP know how much they're appreciated!

The server was down for 22 hours so the server was only available for 26 hours yesterday and today together which is only 2 hours more than 1 full day with only the time between 4pm and 6pm playable on both days, so it seems logical to combine then into 1 day for daily reward and endeavours.
Majority of players on both EU and NA were unable to get their daily reward and seals of endeavour but a minority was able to get not only those but also apex mounts for 800 insteat of 8000 seals.
This was the longest and worst timed maintenance/ outage I have seen in ESO as daily player since 2017(althought only login for daily reward and endeavours as PvP has turned into an Xv1 hunting ground). Even when they changed to new servers the maintenance was shorter. Not compensating to not „set an expectation for something similar whenever there's an issue„ is nonsense as such a maintenance has never occured before except maybe in the first years and when it occurs again compensating would be absolutely justified again.

If you have local outage that is unfortunate but an individual problem and usually not ZOS fault.
But if they exclude almost every player on all servers from getting their daily reward and seals of endeavour because their server is down it is their responsibility.