SERVER DOWN?

kyle.wilson

I would hope that the live server is being prioritized over the pts to bring back online, and not being run simultaneous.

WhiteCoatSyndrome

ZOS_GinaBruno wrote: »

kyle.wilson wrote: »

ZOS_GinaBruno wrote: »

Hi all, no update as of yet. We'll continue to check back in periodically.

Can the mods keep the dev thread about the server being brought down updated for status, and not just dev comments buried inside other threads.
@ZOS_Kevin

Yes it's right here: https://forums.elderscrollsonline.com/en/discussion/656543/temporarily-taking-down-the-pts-the-pc-na-live-server#latest

Thank you Gina!

Durnik

I'm getting close to not being able to get in tonight. That means no daily endeavors, hireling mails, missed event tickets.

fireshine6623

jcaceresw wrote: »

Someone wrote on another thread (now closed):

This may be a mess-up of potentially massive proportions, but whatever you do, I beg you, do not take disciplinary action against the employee(s) responsible for the mistake.

What if the screw up were the complete deletion of the whole PC/NA database and the last backup (if any) dated from a year ago (over exaggerating but it can happen)?

I wonder, what the affected player base can do if such screw up were to happen. Will the affected players be able to file a lawsuit or current terms and conditions protect the company of any liability?

Not intending to speak ill against the company but something like that even happen on small busineses.

I own a business, and I agree 100%. No matter how much complaining people are doing (and debating how to set up a database, which I think is hilarious since ESO is the size of Neptune), mistakes happen. Humans are humans, we make mistakes. Take care of your people ZOS, they likely will be pulling long hours to make fixes on this. And honestly, we should be taking care of each other as the ESO community, not bickering because we can't play our favorite game.

fireshine6623

Ariordin wrote: »

I came here for the Pts patch notes but oh geez this is not what I expected. Has this ever happened before in ESO or is it a new low?

I've been playing for over 7 years, and it's rare for the servers to go down (but there is planned maintenance once a week where they take it offline for a number of hours). So no, it's not normal to have a server emergency like this at all. Last time this happened was... see the hamster on the wheel comments, they missed their feeding time.

WolfStar07

Is the event going to get extended by a day? So many of us haven't even had a chance to login for the day to do the event, and between the maintenance and expected roll back, it sounds like a ton of players are going to miss out participating in jubilee today.

Ariordin

ragnarok6644b14_ESO wrote: »

Finding new mistakes is far, far better than repeating old mistakes.

It means they're capable of learning....

As long as they learn from this and make up for the lost time in game I’m ok with this.

They should also make light of the whole plane meld joke by doing something free and funny when servers come back up

HatchetHaro

jcaceresw wrote: »

Someone wrote on another thread (now closed):

This may be a mess-up of potentially massive proportions, but whatever you do, I beg you, do not take disciplinary action against the employee(s) responsible for the mistake.

What if the screw up were the complete deletion of the whole PC/NA database and the last backup (if any) dated from a year ago (over exaggerating but it can happen)?

I wonder, what the affected player base can do if such screw up were to happen. Will the affected players be able to file a lawsuit or current terms and conditions protect the company of any liability?

Not intending to speak ill against the company but something like that even happen on small busineses.

I made that thread (it got an acknowledgement from ZOS, yay!), and to take your hypothetical situation into account, if the last backup of characters is from a year ago, it's just indicative of more pipeline issues, not human error issues. Anyone in software development know the importance of keeping frequent backups everywhere. It is never the fault of one person.

Also keep in mind, the human error portion of this (admittedly massive) problem is likely someone simply forgetting to set a variable to point the server towards the correct database (either when the characters are being copied over or when the PTS build is made; depends on how they do that), an error anyone can make, especially seasoned veterans in software and game development. Of course, if it causes enough loss in revenue due to exceedingly high game downtime and loss of players, shareholders would want to see some heads roll, a sentiment that I entirely disagree with.

Wildberryjack

It's a bit annoying but oopsie's happen. They'll fix it. This kind of thing, in addition to averting burnout, is why I play more than one MMO.

Fata1moose

Sure the Staff of Worms motif wasn't dropping for me in dolmens but shutting down the entire server is a little extreme.

Jimbru

Ariordin wrote: »

I came here for the Pts patch notes but oh geez this is not what I expected. Has this ever happened before in ESO or is it a new low?

I've been a gamer my whole life and am an ESO beta tester, and to my knowledge, this kind of "DOWN THE GAME NOW" situation is an "Achievement Unlocked" for ESO. Out of all the technical problems ESO has had over the years, this definitely is one of the biggest. I've only seen something like it once in another game, which I already described earlier in this thread.

fireshine6623

WolfStar07 wrote: »

Is the event going to get extended by a day? So many of us haven't even had a chance to login for the day to do the event, and between the maintenance and expected roll back, it sounds like a ton of players are going to miss out participating in jubilee today.

They've done that in the past, did a small extension. Trust in the Jubilee force Wolf....

totzummrdave_ESO

RenTheCat wrote: »

Wonder if Zenimax will refund all these hours of server down time to ESO+ players. Don't remember paying suscription for a dead server.

They never have in all these years, and sadly they never will.

fireshine6623

Ariordin wrote: »

ragnarok6644b14_ESO wrote: »

Finding new mistakes is far, far better than repeating old mistakes.

It means they're capable of learning....

As long as they learn from this and make up for the lost time in game I’m ok with this.

They should also make light of the whole plane meld joke by doing something free and funny when servers come back up

It's great and interesting that you say that. I've wanted them for a long time to "shake things up" by doing crossover stuff. When Elswyer came out there were dragons flying over all the major cities to announce that, why not make unpredictable incursions by creatures from other zones? So a planemeld "Morlag Bal yoinked our PTS" joke with some special incursions would be welcome especially if it's like new and random bosses.

Bluepetunia

SilverBride wrote: »

Bluepetunia wrote: »

No need to panic, relax, get up out of your chair........

Take a shower
Eat something besides microwaved food
Go brush your teeth
Walk the dog
Spend time with spouse, kids, Mom, Dad (at least let them know you are still among the living)

Maybe put on some sunglasses and venture outside. I promise no dragons, werewolves, harpy's (well maybe the wife........j/k), vampires will get you'

That is assuming that we spend every second of our day at our computers and never interact with other people, or take care of our hygeine, or get fresh air, or eat healthy. That is not a fair assessment.

It was meant to be in fun. After all we are gamers.

Maythor

Well so much for my 200 endeavours too!

Bluepetunia

ArchangelIsraphel wrote: »

Bluepetunia wrote: »

No need to panic, relax, get up out of your chair........

Take a shower
Eat something besides microwaved food
Go brush your teeth
Walk the dog
Spend time with spouse, kids, Mom, Dad (at least let them know you are still among the living)

Maybe put on some sunglasses and venture outside. I promise no dragons, werewolves, harpy's (well maybe the wife........j/k), vampires will get you'

Thanks for posting your personal to-do list on the forums so that we could read it while the servers were down. I love seeing people who become self-motivated during times like these. Don't worry, you can do it-we're all cheering for you!

I did all that before the server went down. As a matter of fact I do this daily.

Ariordin

I’d imagine a good way to say sorry without requiring a ton of dev time to create would be to add extra anniversary rewards or add another XP event. Or free tickets, or a free Molag bal imp pet for people.

See devs! Options!

bossonova420

WELL CYA LATER AS I SEE THE MODS GOTTA CALM DOWN A BIT BEFORE THEY CAN HANDLE ME APPARENTLY, CYA.

Arizona_Steve

Hopefully I'll get my missed endeavors, glorious anniversary box and last day of month reward.

fireshine6623

Ariordin wrote: »

I’d imagine a good way to say sorry without requiring a ton of dev time to create would be to add extra anniversary rewards or add another XP event. Or free tickets, or a free Molag bal imp pet for people.

See devs! Options!

I'm of a different mindset, I don't think Zos, the dev staff, @ZOS_GinaBruno or anyone else on their end has anything to apologize for. It's a great game with a lot of a features, and it's a HUGE game, so things will go wrong. It happened int the past, and will again. But it does not happen that often, that's the critical part, and I mean rarely. Problems will show up, and they'll get fixed. That's how it is with online games.

Remember how under covid people were supporting the hospital staff? Sending free meals, showing up with signs of support. Frankly, I'd show up and hold vigil and bring any support I was able to the devs and staff if I could. And I am 100% serious about that, and any of you are welcome to join me. I love ESO that much, it has been a great part of my life, no question. They support us and we take them for granted, but there's a time for us to step up, even if it means being patient and not expected them to apologize for the enormous endeavor it must be to keep or favorite world up and running.

hiyde

totzummrdave_ESO wrote: »

RenTheCat wrote: »

Wonder if Zenimax will refund all these hours of server down time to ESO+ players. Don't remember paying suscription for a dead server.

They never have in all these years, and sadly they never will.

For 12 hours of downtime, that works out to... 19-25 cents.

Gspawn

In the middle of another event, test servers are just for show u guess.

M1SHAAN

ragnarok6644b14_ESO wrote: »

But it's a database, not a server.

If you have two independent circuits that never touch each other, but each have the same shaped data bus, the only thing preventing one person from plugging one circuit into the other circuit is... well, that person and their capacity to make errors*

*yes, the analogy is strained because you could put the physical circuits several meters apart... but it's software.

A better analogy is this:
You have a LAN with FILE SYSTEM A on it, and another LAN with FILE SYSTEM B on it. The only way to prevent similar-looking files from crossing the streams is to have them totally air-gapped....

... and yet, if the person working with the material is allowed to thumb-drive between the two systems, you will STILL have moments where some of the files on LAN A are identical to the files on LAN B by mistake. Almost exactly that scenario is why you can't plug thumb drives into most private organization's proprietary LANs. Because people literally make this mistake all the time. And once you ban plugging in a thumb drive, people find ever-more creative ways to make this kind of mistake.

You're totally right, but I don't think that's the full picture. It's true that human errors are inevitable, but good systems have processes to minimize the impact of said errors. Maybe there's a checklist that you have to run through before pushing something to production, or maybe someone has to double check and approve anything you do with the live server. As an extreme example, the commercial aviation industry has tons of checklists and redundancies to minimize the impact of human error because the results of serious mistakes are catastrophic.

I echo people's sentiments that I hope that the employee(s) responsible for the screwup aren't punished for it. That'd be punishing them for being human. Instead, I hope that ZOS management sees this and some of the other shockingly impactful bugs that have found their way to the live servers in recent months, and decides to allocate more funds and personnel to the ESO QA team. Nobody wants to let bugs slip through and then need to scramble to correct them, but there's only so much one person can do. However large the QA team currently is, in my opinion it needs to be larger.

Arizona_Steve

I'm honestly surprised this can even happen. The live -> PTS transfer should be a job that is run on demand, so the live and PTS connection strings should be configured somewhere and should never change.

Ideally the live DB from which they are copying should be a read replica, not the actual writable database.

(Software engineer for > 30 years)

ragnarok6644b14_ESO

M1SHAAN wrote: »

ragnarok6644b14_ESO wrote: »

But it's a database, not a server.

If you have two independent circuits that never touch each other, but each have the same shaped data bus, the only thing preventing one person from plugging one circuit into the other circuit is... well, that person and their capacity to make errors*

*yes, the analogy is strained because you could put the physical circuits several meters apart... but it's software.

A better analogy is this:
You have a LAN with FILE SYSTEM A on it, and another LAN with FILE SYSTEM B on it. The only way to prevent similar-looking files from crossing the streams is to have them totally air-gapped....

... and yet, if the person working with the material is allowed to thumb-drive between the two systems, you will STILL have moments where some of the files on LAN A are identical to the files on LAN B by mistake. Almost exactly that scenario is why you can't plug thumb drives into most private organization's proprietary LANs. Because people literally make this mistake all the time. And once you ban plugging in a thumb drive, people find ever-more creative ways to make this kind of mistake.

You're totally right, but I don't think that's the full picture. It's true that human errors are inevitable, but good systems have processes to minimize the impact of said errors. Maybe there's a checklist that you have to run through before pushing something to production, or maybe someone has to double check and approve anything you do with the live server. As an extreme example, the commercial aviation industry has tons of checklists and redundancies to minimize the impact of human error because the results of serious mistakes are catastrophic.

I echo people's sentiments that I hope that the employee(s) responsible for the screwup aren't punished for it. That'd be punishing them for being human. Instead, I hope that ZOS management sees this and some of the other shockingly impactful bugs that have found their way to the live servers in recent months, and decides to allocate more funds and personnel to the ESO QA team. Nobody wants to let bugs slip through and then need to scramble to correct them, but there's only so much one person can do. However large the QA team currently is, in my opinion it needs to be larger.

It's curious you'd use the aircraft industry as an example, because it's actually /my/ example of humans being human no matter how good the processes are.

It is infinitely more regulated and taken infinitely more seriously while having infinitely more money than ESO's server handling procedures, and yet you have stuff like doors falling off of planes, and in certain cases, tragic accidents that still occur.

They do an INSANE amount of checking, re-checking, quadruple checking, QA checking, checking the checkers, checking the boxes, checking the checklist... and yet human errors (and errors caused by human error) still happen on the regular.

biminirwb17_ESO

Khajiit didn't mean to press the "Do not Press" button - Sorry.

ragnarok6644b14_ESO

biminirwb17_ESO wrote: »

Khajiit didn't mean to press the "Do not Press" button - Sorry.

.... or perhaps Khajiiti errors.

DewiMorgan

I've a very vague memory of them adding an additional day or two to ESO Plus in the past, to cover an extended period of flakiness, though I can't remember why, and may be misremembering. I don't think it matters, but could be a nice goodwill move if technically possible.

Personally, I'm hoping they extend the event by a day: they do that quite often, too.

And yes: ZoS aren't the people I'd be angry at over this.

At least in my experience, fixing the issue isn't usually what takes the time. Even cleaning up the first-order problem doesn't take too much time.

What takes time is cleaning up after all the jerks who exploited the bugs.

Not just the time it takes to detect and ban them, that's easy: but tracing down the items they transferred to other people. Hard enough with things like attunable stations, but for stackable stuff like gold, it's a *nightmare*! Alice uses the exploit to get 2 million gold, and gives Bob 1M, but he already had some money, so now he's at 1.5M. Bob trades Claire 0.5M for Kuta, and Dave 0.5M for an undetectable trade. Who, now, do you remove the gold from?

Rinse, repeat, for every single person exploiting, every trade they made, every trade that was made on from that... some of the attunable stations will have got attuned and merged with the all-in-one stations, so that needs to be backed out... other stuff bought or traded was consumable and has been consumed... all just a gigantic mess that some poor engineers are having to work several hours of unpaid overtime to manually trace through and pick out all the threads and fix.

And they have to do it before the server comes up. Because if they bring it up and then work on unpicking those threads while the server's up and people are able to log in, then people will work frantically to launder their ill-gotten gains, or scam people with their imaginary gold, so the job will become a hundred times harder and make a hundred times more players get ripped off.

At least, that was my experience as an MMO dev.

If we see an exploitable bug, and we exploit it, we're screwing all of us by extending the downtime. We're not going to get to keep the stuff anyway, and we might get a permanent ban for doing it, losing everything. It's not worth it.

So we should all try not to be that dude: instead, /bug and report it.

And grouch at the damn exploiters for causing this downtime. Bugs happen, but I'm betting it'd all be back up by now if not for their greedy selfishness.

Sakiri

Even if no one exploited it, they'd still have to take the server down to fix it, because none of that behavior was intended.