Emergency Server Downtime Hangout Thread 12/12/2024

ZOS_Lunar · December 12

Rkindaleft wrote: »

This is what I ate for dinner

Did you bring enough to share?

TyrTrolas · December 12

Sometimes i dream about cheese

Destai · December 12

I've been leveling up my RL provisioning skills. Just made some sweet potato/carrot soup! Baby's in daycare, wife's at work, I'm on PTO - come on servers, come back!

ZOS_JessicaFolsom · December 12

TyrTrolas wrote: »

Sometimes i dream about cheese

Honestly, same. Bastian probably really dislikes us.

Skjaldbjorn · December 12

I'm bored. Was watching old Twitch clips. Enjoy two Mikes and a Bjorn.

LarsS · December 12

Any chance that the login server can come earlier, so EU players can log in?

Diundriel · December 12

https://youtube.com/watch?v=i-O_geHXfoA&t=713s

push

BelmontDrakul · December 12

LarsS wrote: »

Any chance that the login server can come earlier, so EU players can log in?

YES

DreadKnight · December 12

2 Billion USD in profit and there's no backup servers? I'm not bashing but aren't there contingencies in place for such a world ending event?

ArchangelIsraphel · December 12

BelmontDrakul wrote: »

LarsS wrote: »

Any chance that the login server can come earlier, so EU players can log in?

YES

WolfCombatPet · December 12

For anyone wondering:

"At this time, we are estimating it could be at least 12 hours until we're up and running" That was posted at 9:45 AM Pacific.

Check back at 9:45 PM Pacific.

Wolfkeks · December 12

ZOS_JessicaFolsom wrote: »

TyrTrolas wrote: »

Sometimes i dream about cheese

Honestly, same. Bastian probably really dislikes us.

Bastian will remember that

LiteEmUp · December 12

wilykcat wrote: »

A bunch of futuristic khajiits.

looks like my next khajiit overhaul mod in skyrim

arena25 · December 12

WolfCombatPet wrote: »

For anyone wondering:

"At this time, we are estimating it could be at least 12 hours until we're up and running" That was posted at 9:45 AM Pacific.

Check back at 9:45 PM Pacific.

They also posted another update at about 245pm Eastern (1145am Pacific) saying they have some(TM) power restored to critical pieces of the datacenter, but lots of work to do still before they can call this resolved. They also aimed to provide another update in a couple hours, so we'll see how things look at 5pm or so, but barring a technological miracle, double or triple overtime, all-nighter shifts, and the utilization of the emergency supply of caffeinated beverages appear to be in ZeniMax's immediate future. RIP the sleep schedules of Gina, Jessica, and all the other ZeniMax employees.

Giraffon wrote: »

I wonder what REALLY happened. Not really sold on the lost power thing. Big operations are supposed to have fault tolerances built in. I bet someone pulled the wrong lever. I bet it was red.

I asked earlier whether tornadoes/natural disasters might be at fault - both here and in my guild's Discord - and one of the other Discord members replied with the following:

There are DC's (datacenters) everywhere in the US.
My bet is a power upgrade/work went sideways at the location.
The company I work for had this happen once which took out an entire leg of the DC and the DC staff had cross run a bunch of the power runs to the racks so that stuff that looked like the PDUs were all correctly split between circuits we actually all on the one that was knocked out.
Was almost as fun as the Crowdstrike outage.

Though I am sure someone will have some explaining to do at some point - last time I saw an outage this bad, it was back when Blizzard tried to launch The War Within pre-patch - a mere 5 months ago.

sPark101 · December 12

You can't even cancel your payments now.
weird.

HatchetHaro · December 12

Ah the good old days.

https://www.youtube.com/watch?v=qxfuDngKwPg

o_Primate_o · December 12

Quoted post has been removed.

They locked down the crown store some time prior to the failure, from what I can determine.

arena25 · December 12

o_Primate_o wrote: »

Giraffon wrote: »

I wonder what REALLY happened. Not really sold on the lost power thing. Big operations are supposed to have fault tolerances built in. I bet someone pulled the wrong lever. I bet it was red.

They locked down the crown store some time prior to the failure, from what I can determine.

Yeah, from what I'm hearing, that was related to some faulty code with some free ESO+ trial that players were abusing/exploiting, but pretty sure that exploit and this incident have nothing to do with each other (despite the timing being close to one another) - would be curious to see if Zeni opts to fix that before unlocking the servers OR if they just straight unlock the servers but keep Crown Store locked down.

moderatelyfatman · December 12

This time I want a pink guar!

hamgatan · December 12

sorry but i dont buy it guys. every enterprise grade DC ive worked in from a power delivery perspective has;

- at least two redundant inbound mains grid connections on redundant grid circuits
- an ent class UPS (such as an APC schneider Galaxy series).
- diesel emergency genset for failover.

come on what's the real story?

sarahthes · December 12

hamgatan wrote: »

sorry but i dont buy it guys. every enterprise grade DC ive worked in from a power delivery perspective has;

- at least two redundant inbound mains grid connections on redundant grid circuits
- an ent class UPS (such as an APC schneider Galaxy series).
- diesel emergency genset for failover.

come on what's the real story?

I've worked in a business where the failover failed. While not a live service gaming company, it is one with a significant global presence. For 2 full weeks we were running an entire national division's infrastructure on a 10 year old backup-to-the-backup server and providing most of our deliverables to clients via hand typed excel spreadsheets because all the one server could manage was to make existing data internally accessible, nothing new could be written to it and it couldn't export or interface with our online client service applications.

We now have much more robust systems, but until that catastrophic failure happened, we thought our previous configuration was fine.

LatentBuzzard · December 12

sarahthes wrote: »

We now have much more robust systems, but until that catastrophic failure happened, we thought our previous configuration was fine.

That's why responsible companies run regular BCP tests, so that they don't have to wait for a catastrophic failure before they find out that they can't recover.

Jaimeh · December 12

Players right now:

MasterSpatula · December 12

So I hear there's this stuff called "grass." I'm headed out to investigate right now.

arena25 · December 12

HatchetHaro wrote: »

Ah the good old days.

snip rad video

Ok, that made me laugh harder than I should have - take an Awesome from me.

sarahthes wrote: »

I've worked in a business where the failover failed. While not a live service gaming company, it is one with a significant global presence. For 2 full weeks we were running an entire national division's infrastructure on a 10 year old backup-to-the-backup server and providing most of our deliverables to clients via hand typed excel spreadsheets because all the one server could manage was to make existing data internally accessible, nothing new could be written to it and it couldn't export or interface with our online client service applications.

We now have much more robust systems, but until that catastrophic failure happened, we thought our previous configuration was fine.

And I posted earlier this gem from one of my guildies:

There are DC's everywhere in the US.
My bet is a power upgrade/work went sideways at the location.
The company I work for had this happen once which took out an entire leg of the DC and the DC staff had cross run a bunch of the power runs to the racks so that stuff that looked like the PDUs were all correctly split between circuits we actually all on the one that was knocked out.
Was almost as fun as the crowdstrike outage.

So, yeah, stuff can happen - not every system is 100% foolproof.

DreadKnight wrote: »

2 Billion USD in profit and there's no backup servers? I'm not bashing but aren't there contingencies in place for such a world ending event?

The Titanic had contingencies in case it hit an iceberg. We all know how THAT panned out...

Desiato · December 12

MasterSpatula wrote: »

So I hear there's this stuff called "grass." I'm headed out to investigate right now.

A friend of mine said he was going to touch it and I never saw them again, I would be careful if I was you.

hamgatan · December 12

LatentBuzzard wrote: »

sarahthes wrote: »

We now have much more robust systems, but until that catastrophic failure happened, we thought our previous configuration was fine.

That's why responsible companies run regular BCP tests, so that they don't have to wait for a catastrophic failure before they find out that they can't recover.

Exactly. Thats why I have two DC's on a 100gbps trunk with vSphere HA and SAN Live Volumes as another level of failover in case the power goes to hell.

Unfortunately can't always trust third parties when they come in the mix. I had to cold start an entire DC last year because facilities conveniently left checking that the diesel genset had fuel in it off their checklists.. Least the other DC was alive though.

LalMirchi · December 12

Red banners at night, sailors cower in fright.

Green banners at daybreak, sailors delight.

carthalis · December 12

MasterSpatula wrote: »

So I hear there's this stuff called "grass." I'm headed out to investigate right now.

be careful I've heard it can be quite dangerous stuff

CrushDepth · December 12

Well, this seems like to watch RED ONE on Prime.