Thanks devs and team for working late today

harvspecial

I work in software development. Launches aren’t always perfect.

Is it unfortunate PCNA can’t login? Yes. It unfortunate for players because we are undoubtedly upset but it’s also unfortunate for ZOS in lost revenue.

Thanks devs, community, engineers and other folx working late today and throughout the entire day to get this issue resolved. WE APPRECIATE YOU!

Frikgrim

It's refreshing to read something positive, as opposed to "entitled rants."

Tannus15

Also a software dev here and every time I see the downtime extend i'm thinking "oh, they are having a really bad day"

TaSheen

harvspecial wrote: »

I work in software development. Launches aren’t always perfect.

Is it unfortunate PCNA can’t login? Yes. It unfortunate for players because we are undoubtedly upset but it’s also unfortunate for ZOS in lost revenue.

Thanks devs, community, engineers and other folx working late today and throughout the entire day to get this issue resolved. WE APPRECIATE YOU!

Agree completely. I'm lucky though, I have characters on EU as well as NA, so I played on EU today.

ZOS_Kevin

Thank all of you for your patience, we know this can be frustrating. We're working on getting PCNA live for all of you. Hang tight.

CyberDiva

As a coder myself, I 100% agree. THANK YOU for all of your hard work.

Cadbury

The forums always keep me entertained during patch/maintenance, so even during downtimes like this I can look to the forums to bring a smile to my face.

SilverBride

I'm sure they aren't enjoying this any more than we are. I appreciate that they are doing their best to get things up and running again.

carly

I know how stressful these long days spent trying to resolve issues can be, thank you from me too.

FeedbackOnly

I hope at least they got pizza

Tornaad

harvspecial wrote: »

I work in software development. Launches aren’t always perfect.

Is it unfortunate PCNA can’t login? Yes. It unfortunate for players because we are undoubtedly upset but it’s also unfortunate for ZOS in lost revenue.

Thanks devs, community, engineers and other folx working late today and throughout the entire day to get this issue resolved. WE APPRECIATE YOU!

I was just thinking about the same thing, because you know there is going to be at least one super devoted dev who has been there since 2 am ET and has been working their but off the whole time, making for an incredibly long day.

KitsuneShoujo

sPark101 wrote: »

[snip]

Right, the devs screwed up the servers just so they could irritate forum-trolls.

SainguinKrist

Even if I'm frustrated, I do understand how this kind of thing can be a complete mess for the team and isn't any easier for them.

It's actually quite worse for them.

We can't play a video game.

ZOS is having their live service interrupted and have been working over 12 hours to fix it.

I think we can all be a bit more understanding of the work day they're going through.

BergisMacBride

I truly feel for the devs and engineers working on this today. They’ve all had and are still having a very bad day.

That said, these major extended downtimes are becoming an increasingly frequent occurrence, and really does cause me to question whether they have the necessary resources in terms of personnel, etc… to handle the game as it currently exists. If this is an issue, that falls on the producers. When problems like these become commonplace, obviously something needs to change and the folks calling the shots need to address it post haste.

Mr_Stach

KitsuneShoujo wrote: »

sPark101 wrote: »

Surely this is a joke.

Right, the devs screwed up the servers just so they could irritate forum-trolls.

That's what I would do lol

Spiral__Architect

I hope Kevin and Gina are getting karma hazard pay. Most people are throwing humor at the situation, but to many are really angry over much about nothing.

Back in my day, you had gone to a store, bought a game, got to disk 3 of 4 only to come across a fatal error that would not let you progress. You called the game developer during business hours hoping to get ahold of THE customer service rep to get a replacement disk. There was a small charge + shipping and handling and you could expect the new disk in the mail in about six to eight weeks. /shakescane

Yes, my version "walking to school in the snow uphill both ways during the summer".

Bouldercleave

SilverBride wrote: »

sPark101 wrote: »

Surely this is a joke.

Why would this be a joke? I'm sure they aren't enjoying this any more than we are. I appreciate that they are doing their best to get things up and running again.

While I do appreciate that they seem to be working diligently to fix the problem (that THEY created), There is really no legitimate reason that a properly tested and vetted patch should crash an entire server for 12+ hours. It means that it wasn't properly tested and somewhere down the lines some shortcuts were made.

Every patch is entitled to a few glitches and bugs here and there, but 10+ hours of not knowing WTF is going on before the last blurb of "great news, we identified the problem" is worrisome to say the least.

Especially from a company with a LONG history of issues like this.

Again I appreciate the hard work that they do and I KNOW there are a bunch of people that are having a really bad day, but if they had done their due diligence this could have been greatly avoided.


Just my 2 GP (which is of no value on PC/NA anymore)

Cadbury

FeedbackOnly wrote: »

I hope at least they got pizza

ArchMikem

I can imagine a job at a place like ZOS isn't a structured schedule. When something like this happens, you don't just drop it and make it wait as you go home from your shift. You stay there and keep working until the problem is resolved. A LOT if overtime I bet. As a guy that already works 12hr days, mad respect. I can't imagine staying sane dealing with it all.

PDarkBHood

Thank you!!!! We really appreciate all the work you are doing!!!

ectoplasmicninja

SainguinKrist wrote: »

Even if I'm frustrated, I do understand how this kind of thing can be a complete mess for the team and isn't any easier for them.

For sure. You can't plan for everything and sometimes errors slip through and the consequences can really snowball unexpectedly. Of course I wish I were playing High Isle - and collecting my daily login reward and getting my endeavours done, all of which are now not happening today - but I'm pretty sure the ZOS folks are also wishing they were at home relaxing and not trying to fix a borked server on chapter release day while angry hordes stalk the forums weeping and gnashing their teeth. Definitely not a good time.

WorldsGr8stBoss

harvspecial wrote: »

I work in software development. Launches aren’t always perfect.... b

Could this have been avoided with pre-testing?

harvspecial wrote: »

but it’s also unfortunate for ZOS in lost revenue.

TRIGGERED!!!!!

SilverBride

I can understand disappointment and frustration, but not anger.

WorldsGr8stBoss

BergisMacBride wrote: »

That said, these major extended downtimes are becoming an increasingly frequent occurrence, and really does cause me to question whether they have the necessary resources in terms of personnel, etc… to handle the game as it currently exists. If this is an issue, that falls on the producers. When problems like these become commonplace, obviously something needs to change and the folks calling the shots need to address it post haste.

Exactly, if you feel bad for the coders, start hounding on the point above. These down times are "an increasingly frequent occurrence" and it's up to the folks at the top to figure out how to fix this, whether it's more testing prior to launch, more resources (tools AND people) or something else. I'm sure they know how this could have been avoided in retrospect but now they need to fix this recurring downtime problem.

jcaceresw

Well.... I am a software developer and before we make a pass to the live server we make a test with a QA server containing the last live version and apply the last patch that will go live. We do several performance and stress tests and if some problem arises we check if the problem is software or hardware related and fix it with the maximum priority. This is done about a week before the deadline to make our last changes go to the live server.

ZOS should embrace such practice too.

pleximus

Might be a good idea for people to create characters on both servers if they need to have their ESO fix scratched?

And like others have said: thank you so much for your time fixing the game despite the negative outcry. People need to give you guys a break from the constant complaining (me included).

Vonkarolinas

Yes, thanks for all you do. It is appreciated.

Amottica

Frikgrim wrote: »

It's refreshing to read something positive, as opposed to "entitled rants."

In agreement here.

As someone who has been part of numerous deployments of systems much more critical than a game, there is always a good chance something will go wrong no matter how much Q&A is done. Anyone who has been part of major deployments will know this.

MreeBiPolar

jcaceresw wrote: »

Well.... I am a software developer and before we make a pass to the live server we make a test with a QA server containing the last live version and apply the last patch that will go live. We do several performance and stress tests and if some problem arises we check if the problem is software or hardware related and fix it with the maximum priority. This is done about a week before the deadline to make our last changes go to the live server.

ZOS should embrace such practice too.

... Does the term "PTS" ring any bells?..

That said, the communication part has been botched really bad. These extensions 2h at a time, with people hoping they MIGHT get in next time (then hoping they MIGHT get in in time to get daily rewards) were totally one of the worst things ever.

I can only QFT https://forums.elderscrollsonline.com/en/discussion/comment/7615435/#Comment_7615435

Vylaera

sPark101 wrote: »

[snip]

God forbid someone have empathy for a (likely already overworked) team having to deal with unexpected complications with full pressure from management and a bunch of manchildren on the forums tell thim they're incompetent and pathetic.

purple-magicb16_ESO

Thanks to devs and mods for hanging in there and getting through this. It can't have been easy. Go home, get some well-earned rest.