Update 33 PC Launch Postmortem

Darkmage1337

@ZOS_MattFiror

Thank you so much for your thorough, detailed, and complete explanation.
This is what great leadership and communication is all about.

ArchangelIsraphel

Kesstryl wrote: »

@ZOS_MattFiror Detailed communication like this is most welcome and exactly the kind of response your players need. I would never be upset because of hardware failures and server problems, and it was obvious your team was ready working on the issue.

What some of us now need is communication about incorrect dialogue, out of order NPC chatter, map problems, and lost quests that are all plaguing our alts with Account Wide Achievements. When will we see fixes for those, or even acknowledgement that these issues exist and are being looked into? Many of us who did not want to lose individual character tracking would be fine moving on if the game world still treated our alts individually like we had before Update 33. Losing individual character identity AND losing the game world treating our alts as individuals is a double whammy and was not necessary. AwA does not need to intrude into gamely so much. Please look at the feedback on both PTS and in official live feedback threads to see how the game world interacts with alts post update. We just want our game and immersion back for our alts.

I agree with this completely.

In addition, I would like to see a transparent breakdown of the reason why it was concluded that AWA would improve performance. I am guessing that it has something to do with the new system that was implemented, but as it stands it felt like an unnessicary change even knowing this might be the case. The q&a regarding the reasons for this change was far too generalized. I would like to see an explanation of how this change actually functions to improve performance broken down clearly as you have done for us here.

I have not seen any performance improvements at all even after the changes. The server now seems no different than it was. My load screens are still actually longer than they were before u33. This could be because the hardware issues you describe still aren't entirely stable yet. But I'd really like to know what's going on under the hood that made the company think sacrificing the happiness of so many players was worth performance improvements that feel negligible at best.

I'd also like to know what your feelings are concerning the importance of individual character stories and if the company plans to do anything at all to help those who would like a more robust means to build character identity, since we can no longer do it through independent character achievements.

I am not saying this to be snarky or rude- but the company must realize that for some players who have entirely abandoned the game because of this update, the term "postmortem" is entirely too accurate. It feels as though part of what gave the game and their characters life has been sapped away.

If you could at least fix the problems with world continuity and quests, have the world treat characters as individuals, fix the zone guide to display individual progress...it would make many of us who are still playing much happier. (I'd personally appreciate an individual fishing tracker too for all of a zones fish. I'd make a mod, but I'm afraid I don't know how yet.)

wolfie1.0.

While this is appreciated, it does make me wonder why this issue gets such a post while other issues that are just as jarring do not?

Also, curious as to if we would've gotten such a response if the issue was software related and not hardware. A hardware failure is convenient for sure for Zos

Panachudo

Brilliant. Well done ZOS. This open and straightforward explanation of critical incidents is honourable. Thank you.

Bitter_Apple21

@ZOS_MattFiror

Thank you very much. The explanation is very easily understood.

As a person whom spends their life solving problems, from my perspective, the team got hit with what I call the "distraction" effect.

A problem occurred, and the focus was immediately on "what was changed", and hence got distracted from something as mundane as a "broken" device.

From long experience, I have gotten burnt by being caught by the "distraction" principle in troubleshooting, and it is a very hard thing to break out of.

Well done and thanks again.

Ariont

Tenthirty2 wrote: »

@ZOS_MattFiror

Thank you.
THIS is the kind of communication I was looking for.
A clear and concise explanation and a no-excuses ownership of the issue and steps to recover reputation.
It also explains the complexity of the compounding challenges and why they took the shape, and time, they did.

@ZOS_GinaBruno (cc @ZOS_Kevin )
You have asked us awhile back what better communication looked like to the player base.
This thread from Matt, this is it.
Compensation is nice and all, but to be totally honest I would have been happy with just Matt's explanation.
It makes me feel like my time and money as a customer is respected, and well spent.

I have been here trying to figure out how to say what I feel about this whole situation and HERE is someone who has said it for me. Thank You to the team.

Hannerbal

Arunei wrote: »

I appreciate the honesty here, but I do have a few questions.

First, why are beta accounts and characters still being held? We don't have access to those accounts or characters, there should be no reason to keep them, at least none I can think of. Unless those are kept on a completely separate, inactive server, why waste database space on them, especially when your answer to database issues was to implement AwA in the way you did?

As for the 150% Exp Scrolls, that's 10 hours of increased experience. Which is a fraction of time compared to the five days a lot of people lost. You really should be giving people 20 or so of these things, it's not like it would cost you anything, and that would still only be 40 hours of increased experience, not even a full two days. Daily Seals should also be increased for five days, not just the weekly ones.

@ZOS_MattFiror

Edit because I was wrong, a friend informed me that the 150% scrolls are only one hour long, not two as I had thought. That's even less time people are being compensated for. We really should be getting a lot more of these if this is how you want to handle the compensation.

I mean, I get it, but they don't owe us anything as compensation.

Tandor

Quoted post has been removed.

People are using this discussion which is about communicating the week after a performance issue to request communication two months after PTS feedback commenced, all people are saying is "Thanks for the communication on performance, can we please have similar communication on account-wide achievements?" - that doesn't seem unreasonable or counter-productive to me.

[edited to remove quote]

superryan94

Being a mechanic myself, I know things break, and usually at the most untimely moments. I also know that fixing things is sometimes harder than expected because some broken part gives interference in something else making it give weird/ incorrect symptoms for the broken part.

Seeing what went wrong and how you fixed it(or tried to in the beginning here) makes me really happy as it shows things are actually being done about it and why we had to wait.

Things break, and us being human causes errors. That's normal.
But not hearing anything makes it seem like there is some kind of distance between the devs and players, however, seeing this makes it obvious you are actually trying to fix the problem.

Can we please get more updates on problems, bug and errors, this not only shows you do things wrong, but also you do your best to make them right again, which is even more important.

Lonestryder

Tandor wrote: »

Thanks @ZOS_MattFiror for the clear explanation of the performance issues, and for engaging with us pretty promptly over this aspect of the concerns many have expressed in the last week or two.

Could we now please have a similarly clear explanation and engagement to address the other concerns many have expressed in the last month or two over the way acount-wide achievements have been implemented? Specifically, what issues that were highlighted on the PTS and Live forums do you acknowledge and to what extent are you working on them or simply dismissing them as "working as intended"?

I'm thinking of things like broken NPC interactions and quests, routine NPC dialogue, and broken zone maps and guides. Your thoughts on how these things are working would be very much appreciated.

Yes, please!

Matt, I appreciate the thorough explanation related to this week's issues. Good comms! Keep 'em coming!

Pyr0xyrecuprotite

I greatly appreciate the in-depth report on how this issue happened, and why resolution took so long.

I'm still unclear as to why some of us had issues (incredibly bad network performance, it was like being on dial-up) on both PC/EU and PC/NA though. Was the affected network port on some sort of common system?

ForeverJenn

I appreciate the feedback. I personally felt it was important that this was explained as we were under the guise that Cold Storage and Sharding was supposed to make things better, and it seemed, at the time, to have made things a lot worse.

Thank you.

Ultionis

Remember when Gina was asking what sort of communication we were asking for?

This is it.

Thanks Matt. Cheers.

wenchmore420b14_ESO

'So we – and this was our second large error – decided to move ahead with enabling DB Sharding on the PC NA megaserver without addressing the Activity Finder issues.'

Thank you @ZOS_MattFiror for a wonderful and transparent response. And thank you for admitting "errors" were made.
This is the type of honest communication that we have been starved for for a while. So after my "many" angry threads, I say a heartfelt "Thank You".

Now, can we please get a response of the thought process and justification of AwA and the effects of it that took away many players playstyle and deleted 8 years of enjoyable re-playability?
TBH, I have always had pretty good game performance, even when others didn't, but still happy that ZoS is fixing it, but great performance doesn't matter when playing is no longer enjoyable and character arch development is no longer achievable.

Again, thank you for a very well written article. Now can we get one about our other concerns?

FeedbackOnly

Please do test dungeon finder before dlcs for dungeons next time.

CasgarTheSomnolent

I love this post. I can't even imagine the complexity of finding that needle the haystack, especially with it being a hardware issue, which no one ever expects. I can imagine the faces (and words coming out of them) when you guys figured it out, and I'm sure it was a relief but not pleasant in the slightest. Thanks for doing what you do, guys.

In general, I love these peeks into the back end of the ESO machine. It would be so cool and humanizing to get little frequent updates like this about the stuff you guys struggle with and solve that we never know about. Doesn't even have to be big stuff like this. There will always be something, and I think you guys deserve and ought to tell your stories about how you're keeping things going.

h8rt3rmin8r

BRIGHT MOONS!

Thank you much for the very detailed information about the recent issues. This kind of transparency and communication is what has really caused me to deeply appreciate all of the hard work you guys do. I know we all talk trash and complain perhaps too much and you guys have been such a good sport through it all XD

Keep up the amazing work

MageCatF4F

Thanks for the explanation.

But - ever since update 33 the game is almost unplayable on our oldest computer due to choppy graphics.
Yes it is still happening. I don't plan to run out and buy a new computer just for ESO.

WhiteCoatSyndrome

Thank you for the transparency! I appreciate it, and it looks like I’m not the only one.

NeeScrolls

ZOS_MattFiror wrote: »

Since Update 33 launched, I think the PC North American megaserver performance problems deserve some explanation. This post outlines what has been going on the last week or so for our North American PC players.

The DB Sharding process separates our character databases into a "live" DB and a "cold" DB; all accounts who have logged in over the past year are in the Live DB and older ones are in the Cold DB. With that background information, you can now start to understand what happened since Update 33 launched last Monday.

So, Update 33 launched last Monday and the plan was to wait until the dust settled, then actually enable sharding on PC NA. Our first big failure was we chalked these reports up to normal server startup issues after a big update.

. So we – and this was our second large error – decided to move ahead with enabling DB Sharding on the PC NA megaserver without addressing the Activity Finder issues.But the system ran slowly again Monday night so we knew it was something else.

– the issue was actually caused by a bad (as in failing) network port that was unable to process as much bandwidth as it was configured for. It wasn't a software problem at all; it was a hardware failure that, in essence, slowed down the entire megaserver.

The TL;DR is that it wasn't related to Update 33, Account Wide Achievements or DB Sharding at all, even though they all happened around the same time and we spent too much time investigating a red herring because of it.

. Given the circumstances, I think full disclosure was warranted.

Thank YOU @ZOS_MattFiror for the extensive behind-the-code-curtain explanation and level of contrition/transparency.
As a former 'Dev' on another MMO, i for one truly enjoyed (and appreciated) your post today very much.

I only wish u guys would post intel & info like this more often....and not necessarily only when things go haywire.

Anyways, KUDOS to you & the Team! Seems like it was probably just as much of a frustrating coincidence for @ZOS_ crew as it was for all of us playerbase. Onward & upward though yep. -- /tip-hat

---

Arunei wrote: »

First, why are beta accounts and characters still being held? We don't have access to those accounts or characters, there should be no reason to keep them, at least none I can think of..

uhh excuse me @Arunei but i started in BETA (pre-beta really) and , as sad as it sounds , i'm still here playing since 2013!
.

KMarble

ZOS_MattFiror wrote: »

Since Update 33 launched, I think the PC North American megaserver performance problems deserve some explanation. This post outlines what has been going on the last week or so for our North American PC players.

First, last year (which seems like decades ago) we announced a plan to increase ESO’s stability and performance, and we have been diligently performing tasks behind the scenes with every update to implement them. One of the larger items on this list was "Database Sharding" which is a simple concept: take our giant player database (DB) and separate it into two sections for "current characters" and "older characters" so the entire DB doesn't have to be queried when a player logs in. Over time, our character DB (one per Megaserver) has been growing and about two years ago, its sheer size became a bottleneck. This is why the "requesting character load" part of the login process sometimes takes a lot longer than it should.

The DB Sharding process separates our character databases into a "live" DB and a "cold" DB; all accounts who have logged in over the past year are in the Live DB and older ones are in the Cold DB. The plan, once everything is complete, is that active accounts will pull their characters from the smaller Live DB on login, greatly decreasing login time. Older characters will pull from the Cold DB on login, which will take longer, but once an account logs in their characters are moved over to the Live DB for faster access after the initial login. This character record separation happens the first time an account logs in after sharding has been enabled for that megaserver. The first login may be longer than normal as the copying happens, but after that every login afterwards should be much faster.

The good news here is that we have already done this for most of the live megaservers over the last couple of months; all console megaservers have been upgraded already and login times have greatly decreased.

You use character and account interchangeably and this makes it very confusing. I have characters on my account that I don't play often, but my account is (for now) still active.

From prior communication I was lead to believe inactive accounts were being put into cold storage. Not characters. Am I wrong in my interpretation?

As an aside, login time, at least in my experience wasn't decreased. Yesterday during the day I had to force quit the game when I tried to change characters due to how long it was taking.

ZOS_MattFiror wrote: »

...

So, Update 33 launched last Monday and the plan was to wait until the dust settled, then actually enable sharding on PC NA. On launch day, we tracked the usual in-game bugs and issues that tend to crop up and began work to address them. And there were indeed some problems. There were reports of in-game loading screen timeouts and that the Activity Finder was bogged down. Our first big failure was we chalked these reports up to normal server startup issues after a big update. We later increased our real-time monitoring which showed the Activity Finder and other processes were running a bit "hot" – they would spike a bit, then return to normal. We made adjustments both outside of and during primetime hours to try to alleviate queue issues, but this made it difficult to pinpoint if our adjustments were working or if primetime population on the server was easing.

The parts I emphasized. You - the devs - took our reports for granted and did nothing(?)/very little(?) while the game deteriorated to a point that many people couldn't even log in?

ZOS_MattFiror wrote: »

So we – and this was our second large error – decided to move ahead with enabling DB Sharding on the PC NA megaserver without addressing the Activity Finder issues.

And all of you who play on the PC NA megaserver know what happened once we flipped the DB Sharding switch: the entire server slowed down even more during primetime. The DB processes got backed up, which meant that all transfers between processes (i.e. zoning) were even slower, as well as logouts (where your character's DB record is updated) and the Activity Finder (which accesses your character records) became so bogged down it essentially ceased to function at all.

I understand working with deadlines all too well, but it's unwise to move ahead when things are falling apart around you. And make no mistake, things were already extremely bad BEFORE the maintenance for sharding happened.

You guys should be able and capable of changing course when something goes wrong, because as we all know Muphy's Law is the only one with no exceptions. Yet you plowed ahead. Why?

ZOS_MattFiror wrote: »

...
On Tuesday, with the understanding that the problem was probably not connected to DB Sharding at all, we traced every log we could find to figure out where the bottleneck was and we finally found it – the issue was actually caused by a bad (as in failing) network port that was unable to process as much bandwidth as it was configured for. It wasn't a software problem at all; it was a hardware failure that, in essence, slowed down the entire megaserver. Tuesday’s maintenance was to take that device out of service and reconfigure a replacement, and once that was up, everything returned to normal and the DB Sharding process ran as intended: behind the scenes and with no player impact.

Obviously, there are no guarantees, but we do believe we have gotten to the root of this issue. The TL;DR is that it wasn't related to Update 33, Account Wide Achievements or DB Sharding at all, even though they all happened around the same time and we spent too much time investigating a red herring because of it.

I have little to say about this. @tmbrinks has already said it better than I could, but when someone like me, who learnt to code as a hobby, can call out that the issue was probably related to hardware...

ZOS_MattFiror wrote: »

I know this hasn't been an awesome time for any of you on PC. Many of you were unable to login to play and take advantage of the Explorer's Celebration as you otherwise might have. You may have lost time and progress, and to acknowledge that, we are going to be giving out five 150% Experience Scrolls on the first day of April through the Daily Login Rewards calendar and will be tripling the number of Weekly Endeavor Seals the week of 4/4 for players on all ESO platforms.

We have so much to look forward to in April with Jester's Festival, the Anniversary Jubilee, and even more we can't wait to share with you. We hope you'll use these Experience Scrolls during the upcoming 100% bonus XP events and catch up to where you might have been, had the game been running as intended.

Thanks so much for bearing with us and for reading this long explanation. Given the circumstances, I think full disclosure was warranted.

Given the circumstances, I think better, faster communication and actions were warranted.

Many people are very upset about how AWA was implemented, but none of you deemed it important enough to talk to us and truly* explain why this was being done or why it was done so sloppily. Yet, last week extra character slots were on sale in the crown store.

For 9 days we had a game that was unreliable at best and unplayable at worse. You mentioned the group finder, but we also had problems with pretty much everything associated with guilds, companions are still having issues (non-reactive, being dismissed by the game itself), mail, the armory system etc..

Due to your lack of action I saw people blame everyone else for the problems the game was having. One of the worse ones was someone demanding all servers aside from the one they play be shut down, and deeming people who don't play on machines up to their standards to not be worthy of playing ESO.



* for the sake of performance doesn't work anymore.

FinneganFroth

This is the transparency we as a player base are looking for ZOS. We all understand problems happen, and posts like this are a great step in the right direction. Well done!

deleted220717-003304

Thanks a lot for this clear and detailed explanation; much appreciated!

cepheus74

This does not help people who are away on 4/4.

What about double Harvesting?

Stamicka

Really appreciate ZOS being more open about things. However, I'm now left with more questions about some of the other choices you've made.

ZOS_MattFiror wrote: »

And all of you who play on the PC NA megaserver know what happened once we flipped the DB Sharding switch: the entire server slowed down even more during primetime. The DB processes got backed up, which meant that all transfers between processes (i.e. zoning) were even slower, as well as logouts (where your character's DB record is updated) and the Activity Finder (which accesses your character records) became so bogged down it essentially ceased to function at all.

Activity Finder accesses our character records? I wonder why this is. Do you try to pair people who have completed content with people who haven't? Is it something completely different?

Also how is AwA fitting into your big picture performance plans. How does it change things?

Rex-Umbra

Thanks for giving us the details appreciate it.

gariondavey

sarahthes wrote: »

gariondavey wrote: »

Lastly PLEASE separate the bg queues into dm + objective (backfills into dm after 10-15 minutes if necessary) for next patch. PLEASE...this is so desperately needed.

@ZOS_MattFiror @ZOS_Kevin @ZOS_GinaBruno

I'm pretty sure the backfill timer was around that before when there were separate queues. It resulted in everything being a deathmatch because 1 person picking dm would force the 11 who queued random into their dm queue.

@sarahthes I think you misunderstand. I am saying to not backfill until after 10 or 15 minutes (or at all). That is not what was done before. Backfilling was done immediately from my understanding.

S0Z0H

Thank you very much for this deeper dive into what's going on with the server. This kind of in depth message is what is needed. This is part of the kind of communication many of us have been asking for. So going forward, we need more of this.

When players are left in the dark, they begin to go to other games, and they spread the word. So again, going forward , getting performance up is the ultimate goal but on that path, communicating in real terms like this is so very appreciated.

I hope this isn't the last time we get such a more detailed update on things. And hopefully this becomes the norm in terms of breaking things down what exactly is going on.

Not that these explainations won't be critiqued , but it shows you are clearly trying to communicate on a tech level that many actually can comprehend.

People invest A LOT in this game, it's their hobby, for some, for some it's even a major part of their life , believe it or not.
So better communication like this is owed to these players.
Thanks again. Keep up the communication please

kestrik

ZOS_MattFiror wrote: »

Thanks so much for bearing with us and for reading this long explanation. Given the circumstances, I think full disclosure was warranted.

I honestly wasn't expecting an explanation but it sure is appreciated. This post is just awesome.

Ulvich

Matt,
I can't begin to express how grateful I am for you reaching out like this. I have been posting in these forums for someone to just talk with us and let us know what happened and what to expect. Communication has been a bit slow lately. And sometimes what we see is very vague, at best.
- Thanks for the information. This really helps.