What caused this outage?

ldzlcs065

I'm reading the maintenance post and this paragraph is intriguing. What caused this unusual outage? Is there really a fire/flood in the datacenter?

sarahthes

ldzlcs065 wrote: »

I'm reading the maintenance post and this paragraph is intriguing. What caused this unusual outage? Is there really a fire/flood in the datacenter?

Something happened that triggered the fire/flood failsafe. In that situation is better for all power to be cut, even the emergency backup power, so that is what happened.

It seems like it was a false alarm or self contained, however, or we wouldn't be getting updates that they are in the process of bringing the data center back online.

BelmontDrakul

The answer was already given. They were practicing for Cozy Cooking Stream. Suddenly,

BelmontDrakul wrote: »

This happened,



Then this happened,



Well, it seems preliminary preparation for cooking stream went pretty bad.

I blame Gordon Ramsay.

arena25

ldzlcs065 wrote: »

I'm reading the maintenance post and this paragraph is intriguing. What caused this unusual outage? Is there really a fire/flood in the datacenter?

No, but a system designed to protect/limit damage in the event of a fire/flood accidentally got triggered, and now they have to carefully make sure nothing got corrupted as a result of the sudden loss of power/abrupt shutdown. On a scale of how fun it is to deal with as an engineer, it ranks somewhere between the Crowdstrike outage and the WannaCry worm.

ldzlcs065

Thanks for all the kind replies. I'm still kind of curious about what,triggered this false alarm? Can't they figure it out? Hope it won't happen again in the future…

Techwolf_Lupindo

Someone forgot to install the molly shield. ;-)

Atama

If the false alarm was caused by a malicious security breach, for multiple reasons it would be in their best interests to not reveal any details. And in that case we'd never know.

OldStygian

Technology is complicated and sometimes s--t just happens.

Live and learn.

DukeCybran

Let's hope the data are still intact.

Wiseau

I think we all know who is responsible...

ApoAlaia

Atama wrote: »

If the false alarm was caused by a malicious security breach, for multiple reasons it would be in their best interests to not reveal any details. And in that case we'd never know.

I had to use a similar system to take our sites offline when our communications supplier experienced a supply chain attack, unknowingly distributed a windows client 'laced' with Cobalt Strike and the wretched thing started moving laterally throughout the network.

Not saying that this is the case here though, someone (or something) might have triggered it accidentally.

In our case it took us a lot longer than 18 hours to 'get back on our feet'. It wasn't an unmitigated disaster, but it was costly and unpleasant.

Unless new information comes to light I am inclined to believe them (that this wasn't a deliberate action but an unforeseen event).

Xinihp

sarahthes wrote: »

Something happened that triggered the fire/flood failsafe. In that situation is better for all power to be cut, even the emergency backup power, so that is what happened.

For anyone who plays Starfield, this is further proof for Sam Coe that using a blowtorch to do "maintenance" on a server rack in the computer core is NOT a recommended procedure after all. :P

TwiceBornStar

https://i.pinimg.com/originals/1a/43/ba/1a43baed9584ebcb4316df332d358424.jpg

Just saying!

daim

That's gonna be one costly false firealarm button push

JimFord047

Not going to name Country or Organization... Pretty Sure I would be jailed if I did... But , During a Modernisation of the Server Area (not called A Farm, but it was), it was decided in the building to change from WATER Sprinklers , to a Halon Gas System for Fire suppression.

the new system was installed, then the old Water system Decommissioned, everything tested , Job Done!!!

then the TWIST!!!

we had left quite a mess, between the drilling , the moving of walls etc, so it HAD TO BE ALL KLEENED UP, floors brushed , lick of paint, and the last part was to seal the floors. With a Product called KLEEN, you simply mop it onto the floor and let it dry.

As that was being done , the whole room SHUT DOWN, lights / Servers / Disc Arrays EVERYTHING. The main Safety Breaker had thrown ( as in the water one ), so we had to reset and restore Everything took a few hours, and we found that yes, we had changed the fire suppression system, but not disconnected the water detector ! (did not even know it existed).


The Old Safety was simply 2 bare wires , in the event of a FIRE the sprinklers would go off, as the water hit the ground the wires completed the circuit and the breakers would throw , simple but effective!

The simple act of Mopping over it had triggered it. There was hell to pay for the oops, but it did prove that even after a couple of Decades , simple was still a working system.

the fix was to cover the wire ends with caps and Electrical tape... that lasted until the whole area was moved to a different location a few years later with the upgrade of the servers and storage.

So proof these sort of things can happen , with No malice at all involved

code65536

ldzlcs065 wrote: »

Thanks for all the kind replies. I'm still kind of curious about what,triggered this false alarm? Can't they figure it out? Hope it won't happen again in the future…

They don't own the datacenter. They, along with other companies, are tenants at the datacenter, and it sounds like this outage affected everyone at this datacenter, not just ZOS.

As for the investigation into the root cause and how much of that information gets public, my guess is that this is up to the datacenter.

JimFord047

LMAO , OK here is one that still has me laughing....

we had a call in that the server room at a location was shutting down every day, and simply rebooting ITSELF.

I went out AFTER the electrical Engineers had checked everything overnight and had found NO PROBLEMS...

I checked the servers, and found NO Problems , with the exception in the Logs the "Shutdown / Reboot " happened every day Monday to Friday at between 09:55 and 10:07. then Monday to Thursday between 14:55 and 15:10... Nothing ON a Friday after 15:00 until the Monday......

So I parked my backside in the server room with all the test gear attached, sure enough BANG it all goes down, and then comes straight back up!! Nothing in the server room was touched. All of the software was as per expected , nothing, another day in, checking everything. NOTHING!!!!

Next day I am in early , checks it all again, and I am sure there is nothing wrong in the server room, So I have the door open, and I am Standing looking around, hoping to see someone at that time fiddle with the power panel or the servers , there a re a few people moving around the corridor, but nothing untoward!

then I hear a Squeak Squeak Squeak, tinkle *** tinkle.... there is an old Woman pushing a little wooden trolley, she is handing into the rooms Tea / Coffee / Rolls / Biscuits , well I cannot be her... Surely???

As she draws Level with me, a big fat Face appears in the doorway facing me, "Tea and My Usual 3 Rolls Please!" ...

BANG.. the whole server room shuts down..... then comes back up

WTF or a lot more words to those effects , the little old lady and her trolley have now moved on to the next room, so I shouted into the room facing . " Do you get the same thing EVERY DAY?" , the big Fat face appears in the doorway , "YES, WHY?" as the power all shuts down again.....

So I go into the room and there is this HUGE FAT Guy perched on a chair, and I ask him, "can you do that again?" - " WHAT?" - ask for your Breakfast, the same way as you always do?" , the guy humours me and leans back on his seat as if to ask for his Food... ALL THE POWER TO THE SERVER ROOM GOES OFF, he leans forward and it all comes back..... REALLY!!!!!!!

So since this Guy is the size of a baby Elephant , the electrician and I took the room apart, Including lifting the floor... We then found the "Fault" , the mains Power cable came in under the floor to the Panel for the computer room , this Fat Git was putting ALL of his abundant weight onto a Single back leg of the seat, over the years the pressure had split the cable inside , weight down cable splits power off, weight off cable re-joins power back on....

The Fix, run a new power cable avoiding his fat ass!!!

JimFord047

Another FUNNY one.... Since we cannot play ESO right now!

I am sent out to a HOSPITAL, the FAULT REPORTED " every now and again, loads of the Terminals / computer screens , fold down to a point one side , and the Picture fly's off the screen!"

I get to the hospital and into the room , its a Lab attached to the Mortuary, I instantly get kicked out and sent to another room for 6 injections , once i get them I can get into the room again.. I am standing there and nothing is going wrong, so I ask when "does this happen?" answer "It varies, as does the time its off!", right then on cue as it were 5 of the screens all pinch down on the right hand side and the picture fires off like an arrow leaving a bow....

I take one of the screens and lug it in on the other side of the room, and it jumps to life,,, the screen from there (which was working) I put in to replace the one I moved, the Screen comes up, makes an arrow and fires of like the one previously... All of a sudden All of the screens come back, same way as they had left ???????

I checked the serial connections, the screens, the base units... WTF is going on?????

after about an hour it all happens again, I switch the screens and the only thing obvious is that is down one wall only !!!

OK check the termination of the cables , check the data pathways of the cables , ALL come back correct, but still no pictures, then as I am about to change another screen it ALL comes back.....


Puzzled is an understatement, so I decide to head out for a cigarette and have a think , as I am standing there a trolley with a patient comes along as well as a Wheelchair with another patient , they go in through a door, and it closes... Nothing to see here as it were , So I am about to finish off the ciggy and go back in, when I hear a BUZZING SOUND coming from the building I am Standing outside, through the lab window I can See the Screens start to flicker, the sound Now changes to a CHUNK CHUNK CHUNK, and through the windows I can see the screens all collapsing at one side, to the shape of and Arrowhead, and they all fire off then screen at that side .

Have another Ciggy.... the CHUNK CHUNK CHUNK sound stops, the Buzzing Comes back, as do the pictures on the screens the buzzing stops , and the pictures are Perfect again.....

So IN stuck my head in the door, BIG Warning Sign's,,, NO METAL!!! after this Point, , WARNING High Magnetic Field , , and the words above the inner Entry Door .... MRI

So back into the lab got one of the screens on an extension cable, as the MRI Started up the picture vanished , I walked backwards until the Picture came back...

Put the screen on the floor, went into my tool case and pulled out some electrical tape , and put a line on it across the floor....

Explained to the Boos in there, NO SCREENS OVER THAT LINE, and they will all work, otherwise your going to have to move the MRI SUITE the same distance to the wall over there away....


Who said fault finding was easy????

Xinihp

JimFord047 wrote: »

Another FUNNY one.... Since we cannot play ESO right now!

I find it incredibly disturbing there is no building code enforced requiring walls to MRI and X-Ray rooms be properly shielded.

This is just... Humanity why? I just can't.

JimFord047

LOL this was the 1980's , MRI's were just really coming into the Hospitals , the X-Rays had simply been RUN, twas a differnt time lol

Coo_PnT

All we can do is wait. Let us wait for good news.

moderatelyfatman

Coo_PnT wrote: »

All we can do is wait. Let us wait for good news.

See ya in January 2025!

BelmontDrakul

I am suffering from Withdrawal Syndrome.

Rowjoh

ldzlcs065 wrote: »

...I'm still kind of curious about what,triggered this false alarm? Can't they figure it out? Hope it won't happen again in the future…

I'm kinda more curious as to when the game will be up and running...

and even more curious to find out if everything will be exactly as it was before the shut down...

Tommy_The_Gun

I play this game since 2014 and it seems it is (so far) the most serious server malfunction they ever had. I just do hope there will be no "double maintenance", like some critical bug detected too late like item duplication or some progress being moved from pts server to live etc.

Speaking of which, I remember that PTS server was up shortly before this whole thing happened, although nothing new was being tested. Wierd.

fizl101

Nothing unusual with pts being up, it stays up after the testing period

BelmontDrakul

Tommy_The_Gun wrote: »

I play this game since 2014 and it seems it is (so far) the most serious server malfunction they ever had. I just do hope there will be no "double maintenance", like some critical bug detected too late like item duplication or some progress being moved from pts server to live etc.

Speaking of which, I remember that PTS server was up shortly before this whole thing happened, although nothing new was being tested. Wierd.

Nilandia

Tommy_The_Gun wrote: »

Speaking of which, I remember that PTS server was up shortly before this whole thing happened, although nothing new was being tested. Wierd.

The PTS is always up, unless it's specifically taken down for updates or other maintenance. Nothing weird there.

bellanca6561n

Well...can't say I'm happy and I hope folks don't lose anything. But, fact is, I so hate the holiday season that I'd been playing waaaay too much ESO lately 🥵

This extended outage snapped me out of my trance. A good thing in my case.

Yes, the digital and virtual worlds have disasters. But nothing like the world outside I was playing the game to avoid, with its earthquakes, hurricanes, tornados, and actual destructive events.

WhiteCoatSyndrome

Xinihp wrote: »

JimFord047 wrote: »

Another FUNNY one.... Since we cannot play ESO right now!

I find it incredibly disturbing there is no building code enforced requiring walls to MRI and X-Ray rooms be properly shielded.

This is just... Humanity why? I just can't.

Yeah I don’t know if it’s from an MRI specifically, but there’s a hospital that makes my radio go staticky every time I drive past the building…I feel like that’s got to be violating some kind of safety standard.

purple-magicb16_ESO