Update 33 PC Launch Postmortem

MafiaCat115 · March 2022

Thank you for being so detailed in the explanation for what occurred, it really helps us players to better know what goes on behind the scenes.

warabi · March 2022

What used to frustrate me so much when I played ESO regularly was not that things went wrong, but the lack of communication as to why things went wrong. I know you guys have to very carefully pick what you share, but I (and definitely many others) genuinely appreciate this in depth and informative post. This pleases me so much that I'm definitely resubbing. Thank you very much!

VaranisArano · March 2022

FeedbackOnly wrote: »

Please do test dungeon finder before dlcs for dungeons next time.

They can't. There's practically nobody on the PTS.

It sound elementary to test Groupfinder first, but you can't replicate the Primetime load that launch day players put on the whole system beforehand. We've seen this during past times when the software itself was the problem, and ZOS has to play whack-a-mole as they identify the worst bottleneck, fix it, and then find the next bottleneck, and so on. They can test their Groupfinder as much as they like ahead of time, but the tests simply don't match the stress on the whole Live system once Primetime hits.

So it's understandable that ZOS followed the same process as they did during the rework, by checking on different things during the high load primetime when the worst problems show up.

It's just that this time, it was a hardware bottleneck.

FluffyReachWitch · March 2022

Reminds me of a problem I had years back, where I was getting kicked from the game at regular intervals. I started tracing connectivity on my own side, discovered that it was affecting the whole house, and got a technician from the ISP involved. Turned out that the building had a faulty cable.

The communication here is appreciated, and I'm looking forward to a more stable game once I have some time to get in. That being said, I'm also curious as to whether the feedback to Update 33 itself will receive further response, and if any adjustments are being considered to alleviate some of the zone guide and map issues.

peacenote · March 2022

@ZOS_MattFiror

Thank you for the explanation on the performance issues after U33 dropped. It is much appreciated, is the type of communication I'd like to see more often, and makes sense.

However I will say that, as someone who has been here since launch, I don't think this was your biggest issue with U33. I am used to hiccups with patches in this game and all other MMOs I've played. It's a thing. I didn't expect any type of compensation and, while I'm glad that some folks are happy with it, the cries of "do I get a discount on my subscription" or "can we extend event xyz" any time there is even a small blip of unavailability, irk me more than anything else. Stuff happens; I'd rather you spend the time fixing the issues as opposed to giving us tokens to make it up to us.

I truly thought and hoped this "postmortem" would be about how you missed the boat, for many, with AwA. So, as well written as your message was, I'm still extremely disappointed. I'm even a little angry that we got such a detailed communication about something that was only an issue for about a week. There has been significantly more feedback, questions, and comments posed to you about AwA, spanning months of community angst and impacting up to eight years of game play for a portion of your player base.

Please give us some detailed responses about the current status of all the bugs and open questions, and where these are on your priority list. Please tell us why dungeon and trial achievements did not stay character-specific, whether you understood that you were decimating and alienating multiple play styles with these changes, and how you will or will not be supporting "alts" in your game going forward. I could go on and on, but I won't. I will link to a post which has a list of questions I made a while back, HERE.

Whether individuals are happy, sad, or indifferent about the current AwA implementation, I think as a whole the community is owed as much honesty and transparency as we were given here, in this thread, about a short-lived (compared to a lifetime of ESO history or months of angst and silence) performance issue. And now we can clearly see how capable you in particular and ZOS as whole is of such communication when it is decided to be necessary.

Gaeliannas · March 2022

I am wondering, or maybe I just completely misread the explanation... But what was going on for the entire week between the 1st (DLC) and 2nd (sharding) patches? I read activity finder, but how did that affect logins, porting to houses, getting booted constantly and the incredible lag during that time? Or did I miss something in the explanation?

sarahthes · March 2022

Gaeliannas wrote: »

I am wondering, or maybe I just completely misread the explanation... But what was going on for the entire week between the 1st (DLC) and 2nd (sharding) patches? I read activity finder, but how did that affect logins, porting to houses, getting booted constantly and the incredible lag during that time? Or did I miss something in the explanation?

A broken piece of hardware bottlenecked every aspect of the server.

Abigail · March 2022

sarahthes wrote: »

A broken piece of hardware bottlenecked every aspect of the server.

Matt's dog ate the database.

TitanEidolon · March 2022

I don't think it would be possible to overstate how much this goes toward easing frustration. A little transparency goes a long way

Hymzir · March 2022

When I spotted this thread I thought: "Holy... jumping water beetles! They actually finally responded."

When I noticed that the post opened with the line:

ZOS_MattFiror wrote: »

Since Update 33 launched, I think the PC North American megaserver performance problems deserve some explanation. This post outlines what has been going on the last week or so for our North American PC players.

I thought: "Err... What? I know that was kinda big deal, but it was probably just some technical thing that blind-sided them, and something they eventually fixed. What about an actual response and evaluation of all the issues people have had with AWA? Is that just gonna get swept under a rug"

Having read the post, it seems both thoughts proved to be true.

Though credit should be given, when credit is due, and this is an excellent post. It explains clearly what went wrong and what ZOS did about it, and came across as truly professional. It's a prime example of what sort of communication ZOS should be doing. On all points.

Now if only ZOS would approach the issues regarding AWA with equal candour.

p.s. -As a side note: XP scrolls are... underwhelming. Didn't Explorer Thingie have double mat drops too? Double mats is a lot more useful than even more XP scrolls. I got a stack of well over a hundred of them gathering dust by now. There was also double gold during the event, and though thanks to the gold hyper-inflation going on, it too is kinda pointless, but at least it has some uses, where as more XP is rather meaningless. Besides, there are double XP event every other week it seems anyway, and you get a steady stream of XP scrolls from log in junk. So... Yeah, offering even more of the same, comes across as rather hollow.

Though to be fair, I don't really see this thing to be the kinda thing that really requires any particular compensation. You know, stuff happens. Sometimes things go wrong and you miss out on something. It's not a biggie, and it's not the end of the world. There are far worse fates out there, than missing out on some bonus XP, mats and gold in a game.

Still, any compensation is a nice gesture, so kudos for that. I just feel that almost anything but XP scrolls would've been preferable. But that's just my though, I'm sure there are some who find a use for even more XP scrolls. I just don't think they mean a whole lot to the majority of players at this point.

Celephantsylvius_Bornasfinmo · March 2022

Good call on the memo *nods*

mfstevens · March 2022

Thank you so much for explaining what felt like a black hole of disaster. Very much appreciate the behind the scenes peek.

Arunei · March 2022

FeedbackOnly wrote: »

Arunei wrote: »

I appreciate the honesty here, but I do have a few questions.

First, why are beta accounts and characters still being held? We don't have access to those accounts or characters, there should be no reason to keep them, at least none I can think of. Unless those are kept on a completely separate, inactive server, why waste database space on them, especially when your answer to database issues was to implement AwA in the way you did?

As for the 150% Exp Scrolls, that's 10 hours of increased experience. Which is a fraction of time compared to the five days a lot of people lost. You really should be giving people 20 or so of these things, it's not like it would cost you anything, and that would still only be 40 hours of increased experience, not even a full two days. Daily Seals should also be increased for five days, not just the weekly ones.

@ZOS_MattFiror

Edit because I was wrong, a friend informed me that the 150% scrolls are only one hour long, not two as I had thought. That's even less time people are being compensated for. We really should be getting a lot more of these if this is how you want to handle the compensation.

I actually joined ESO because I had a beta account over another game. I logined into such an account too.

These accounts are now in cold storage though. So don't worry.

Yes, but those beta accounts are literally just taking up space because we can't access them. I also have a beta account, but it does me no good because I can't access that it or the characters on it. I had thought once the game went live that all the old beta accounts and characters had been deleted, because there is literally no way for us to use them and the game is so different now than it was back during beta that none of that data can possibly be useful now. It's stuff that's taking up space that could be used for actual live accounts, unless the beta accounts are stored in an entirely separate database.

Hannerbal wrote: »

Arunei wrote: »

I appreciate the honesty here, but I do have a few questions.

First, why are beta accounts and characters still being held? We don't have access to those accounts or characters, there should be no reason to keep them, at least none I can think of. Unless those are kept on a completely separate, inactive server, why waste database space on them, especially when your answer to database issues was to implement AwA in the way you did?

As for the 150% Exp Scrolls, that's 10 hours of increased experience. Which is a fraction of time compared to the five days a lot of people lost. You really should be giving people 20 or so of these things, it's not like it would cost you anything, and that would still only be 40 hours of increased experience, not even a full two days. Daily Seals should also be increased for five days, not just the weekly ones.

@ZOS_MattFiror

Edit because I was wrong, a friend informed me that the 150% scrolls are only one hour long, not two as I had thought. That's even less time people are being compensated for. We really should be getting a lot more of these if this is how you want to handle the compensation.

I mean, I get it, but they don't owe us anything as compensation.

Whether they owe us or not isn't the question. They're already offering compensation, but it feels like the bare minimum. It's 5 hours of 150% compared to five entire days (which is 120 hours), and tripled Weekly Endeavors as opposed to also tripling the Daily Seals too. Yes, they could have chosen to give us nothing, but eh, what they're giving us feels only a step up from nothing.

NeeScrolls wrote: »

ZOS_MattFiror wrote: »

Since Update 33 launched, I think the PC North American megaserver performance problems deserve some explanation. This post outlines what has been going on the last week or so for our North American PC players.

The DB Sharding process separates our character databases into a "live" DB and a "cold" DB; all accounts who have logged in over the past year are in the Live DB and older ones are in the Cold DB. With that background information, you can now start to understand what happened since Update 33 launched last Monday.

So, Update 33 launched last Monday and the plan was to wait until the dust settled, then actually enable sharding on PC NA. Our first big failure was we chalked these reports up to normal server startup issues after a big update.

. So we – and this was our second large error – decided to move ahead with enabling DB Sharding on the PC NA megaserver without addressing the Activity Finder issues.But the system ran slowly again Monday night so we knew it was something else.

– the issue was actually caused by a bad (as in failing) network port that was unable to process as much bandwidth as it was configured for. It wasn't a software problem at all; it was a hardware failure that, in essence, slowed down the entire megaserver.

The TL;DR is that it wasn't related to Update 33, Account Wide Achievements or DB Sharding at all, even though they all happened around the same time and we spent too much time investigating a red herring because of it.

. Given the circumstances, I think full disclosure was warranted.

Thank YOU @ZOS_MattFiror for the extensive behind-the-code-curtain explanation and level of contrition/transparency.
As a former 'Dev' on another MMO, i for one truly enjoyed (and appreciated) your post today very much.

I only wish u guys would post intel & info like this more often....and not necessarily only when things go haywire.

Anyways, KUDOS to you & the Team! Seems like it was probably just as much of a frustrating coincidence for @ZOS_ crew as it was for all of us playerbase. Onward & upward though yep. -- /tip-hat

Arunei wrote: »

First, why are beta accounts and characters still being held? We don't have access to those accounts or characters, there should be no reason to keep them, at least none I can think of..

uhh excuse me @Arunei but i started in BETA (pre-beta really) and , as sad as it sounds , i'm still here playing since 2013!
.

I too was a beta player, but that doesn't really matter because our live accounts are not the same as our beta accounts. As I mentioned in my other post, we can't do anything with them, we can't access them or play those characters. We won't be losing anything if they're deleted, whereas ZOS would be gaining database space for actual live accounts.

Arunei · March 2022

Gaeliannas wrote: »

I am wondering, or maybe I just completely misread the explanation... But what was going on for the entire week between the 1st (DLC) and 2nd (sharding) patches? I read activity finder, but how did that affect logins, porting to houses, getting booted constantly and the incredible lag during that time? Or did I miss something in the explanation?

Apparently it was one of the ports in one of the servers that wasn't able to handle the amount of bandwidth it was supposed to, so it was causing a lot of bottlenecks. They replaced it during Tuesday's maintenance.

TechMaybeHic · March 2022

Hmm my in game Cyrodiil performance was better though, on Monday and Tuesday night. Can you break it again?

But seriously. If it's a matter of total users logged into the game, is it possible to have it where our characters are transported into separate servers for PvP?

MidniteOwl1913 · March 2022

Thank you for this. It can't have been easy, and I appreciate the communication very much. Communication really is key to a good relationship with your player base.

xaraan · March 2022

I just want to say: I appreciate the in-depth explanation. Thank you.

FrancisCrawford · March 2022

Onomog wrote: »

Thank you for your openness on this. The transparency is refreshing.

I would love to see the same effort given to explaining why the AwA process was pushed through with out seeming to take into consideration everything that it broke.

That would be a totally different kind of post.

The story in this post was that they made simple, retroactively clear engineering mistakes (looking in the wrong place for a root cause) before getting to the unarguably 100% correct answer (identifying the true root cause and fixing it). Anything that was a matter of opinion (what makes Activity Finder so vulnerable to slowdowns, and is it OK that it hasn't been fixed for so long?) was ignored in the post and, even so, we think it's an excellent and greatly appreciated post.

What you're asking for would have a lot of debatable opinion in it.

Vaoh · March 2022

Arunei wrote: »

I appreciate the honesty here, but I do have a few questions.

First, why are beta accounts and characters still being held? We don't have access to those accounts or characters, there should be no reason to keep them, at least none I can think of. Unless those are kept on a completely separate, inactive server, why waste database space on them, especially when your answer to database issues was to implement AwA in the way you did?

As for the 150% Exp Scrolls, that's 10 hours of increased experience. Which is a fraction of time compared to the five days a lot of people lost. You really should be giving people 20 or so of these things, it's not like it would cost you anything, and that would still only be 40 hours of increased experience, not even a full two days. Daily Seals should also be increased for five days, not just the weekly ones.

Edit because I was wrong, a friend informed me that the 150% scrolls are only one hour long, not two as I had thought. That's even less time people are being compensated for. We really should be getting a lot more of these if this is how you want to handle the compensation.

Well now that those beta test accounts are kept in a separate database, it should be fine to keep them around right?

If a Beta test player comes back and asks for their beta pet, I’d assume this would be their way to get it.

SirAxen · March 2022

You love to see it.

Tornaad · March 2022

ZOS_MattFiror wrote: »

Since Update 33 launched, I think the PC North American megaserver performance problems deserve some explanation. This post outlines what has been going on the last week or so for our North American PC players.

First, last year (which seems like decades ago) we announced a plan to increase ESO’s stability and performance, and we have been diligently performing tasks behind the scenes with every update to implement them. One of the larger items on this list was "Database Sharding" which is a simple concept: take our giant player database (DB) and separate it into two sections for "current characters" and "older characters" so the entire DB doesn't have to be queried when a player logs in. Over time, our character DB (one per Megaserver) has been growing and about two years ago, its sheer size became a bottleneck. This is why the "requesting character load" part of the login process sometimes takes a lot longer than it should.

The DB Sharding process separates our character databases into a "live" DB and a "cold" DB; all accounts who have logged in over the past year are in the Live DB and older ones are in the Cold DB. The plan, once everything is complete, is that active accounts will pull their characters from the smaller Live DB on login, greatly decreasing login time. Older characters will pull from the Cold DB on login, which will take longer, but once an account logs in their characters are moved over to the Live DB for faster access after the initial login. This character record separation happens the first time an account logs in after sharding has been enabled for that megaserver. The first login may be longer than normal as the copying happens, but after that every login afterwards should be much faster.

The good news here is that we have already done this for most of the live megaservers over the last couple of months; all console megaservers have been upgraded already and login times have greatly decreased.

With that background information, you can now start to understand what happened since Update 33 launched last Monday. The PC character database (especially the North American megaserver) is far, far larger than console as ESO had a big launch year in 2014 (pre-console launch) and all those accounts are still there. In addition, all the Beta accounts (and characters) are still there as well.

So, Update 33 launched last Monday and the plan was to wait until the dust settled, then actually enable sharding on PC NA. On launch day, we tracked the usual in-game bugs and issues that tend to crop up and began work to address them. And there were indeed some problems. There were reports of in-game loading screen timeouts and that the Activity Finder was bogged down. Our first big failure was we chalked these reports up to normal server startup issues after a big update. We later increased our real-time monitoring which showed the Activity Finder and other processes were running a bit "hot" – they would spike a bit, then return to normal. We made adjustments both outside of and during primetime hours to try to alleviate queue issues, but this made it difficult to pinpoint if our adjustments were working or if primetime population on the server was easing. So we – and this was our second large error – decided to move ahead with enabling DB Sharding on the PC NA megaserver without addressing the Activity Finder issues.

And all of you who play on the PC NA megaserver know what happened once we flipped the DB Sharding switch: the entire server slowed down even more during primetime. The DB processes got backed up, which meant that all transfers between processes (i.e. zoning) were even slower, as well as logouts (where your character's DB record is updated) and the Activity Finder (which accesses your character records) became so bogged down it essentially ceased to function at all.

We had done the math and designed the DB Sharding system to work within normal server performance guidelines, so when we started addressing the slowdown issues, we naturally assumed that we had some bad calculations and started there. We made some changes (hence the downtime on Monday earlier this week) but they didn't help at all; performance was still terrible Monday night. Adding to the situation was that we could only troubleshoot on the live server, and only during primetime, because these problems cropped up mostly when the server was under moderate load. But the system ran slowly again Monday night so we knew it was something else.

On Tuesday, with the understanding that the problem was probably not connected to DB Sharding at all, we traced every log we could find to figure out where the bottleneck was and we finally found it – the issue was actually caused by a bad (as in failing) network port that was unable to process as much bandwidth as it was configured for. It wasn't a software problem at all; it was a hardware failure that, in essence, slowed down the entire megaserver. Tuesday’s maintenance was to take that device out of service and reconfigure a replacement, and once that was up, everything returned to normal and the DB Sharding process ran as intended: behind the scenes and with no player impact.

Obviously, there are no guarantees, but we do believe we have gotten to the root of this issue. The TL;DR is that it wasn't related to Update 33, Account Wide Achievements or DB Sharding at all, even though they all happened around the same time and we spent too much time investigating a red herring because of it.

I know this hasn't been an awesome time for any of you on PC. Many of you were unable to login to play and take advantage of the Explorer's Celebration as you otherwise might have. You may have lost time and progress, and to acknowledge that, we are going to be giving out five 150% Experience Scrolls on the first day of April through the Daily Login Rewards calendar and will be tripling the number of Weekly Endeavor Seals the week of 4/4 for players on all ESO platforms.

We have so much to look forward to in April with Jester's Festival, the Anniversary Jubilee, and even more we can't wait to share with you. We hope you'll use these Experience Scrolls during the upcoming 100% bonus XP events and catch up to where you might have been, had the game been running as intended.

Thanks so much for bearing with us and for reading this long explanation. Given the circumstances, I think full disclosure was warranted.

AWESOME! Thank you @ZOS_MattFiror
This is exactly the kind of thing I was hoping for when I posted my letter a few months ago.
https://forums.elderscrollsonline.com/en/discussion/593931/a-letter-to-zos-please-improve-your-communication/p1
I have a lot of hope for where it seems ESO is going and this post is only adding to that.
Thank you.

etchedpixels · March 2022

Thanks for the explanation. It really does help communicating things like this (even in less detail) because it shows you to players as a bunch of folks who care, working hard to fix the problems rather than some of the less friendly assumptions people otherwise make.

runa_gate · March 2022

Didn't read anyone's comments after reading your message. This is the communication we want and it's absolutely wonderful to see. Thank you.

Ek1 · March 2022

ZOS_MattFiror wrote: »

Thanks so much for bearing with us and for reading this long explanation. Given the circumstances, I think full disclosure was warranted.

Estimated reading time: 5 minutes, 31 seconds. Contains 1105 words
Hardly a long one :-)

Merrak · March 2022

Thank you Matt for the transparency in what took place! It's very much appreciated.

Abigail · March 2022

runa_gate wrote: »

Didn't read anyone's comments after reading your message. This is the communication we want and it's absolutely wonderful to see. Thank you.

I do not wish to be contentious, but please be less cavalier in applying 'we' to the equation. While I'm somewhat mollified by Matt's explanation, I'm convinced there's a five-hundred-pound gorilla in the corner about which he's failed to address.

Splitting the database is the proverbial thumb in the dike. While this solution may suffice so long as inactive players can be retired at a pace exceeding addition of new ones, things will be fine. AWA is another intended solution, but what happens when extant and new players add to their respective collections? What happens each time ZOS introduces a new mount, trophy, furnishing, or other piece of paraphernalia? While some of these pieces of detritus are account wide, they will affect data transfer capacity so far as Matt suggests. It's my strong suspicion that the philosophy going forward is to continue poking thumbs in dikes until lights are turned off at ESO. In short, Furor has avoided discussion of the essential flaw upon which the game exists.

So please, unless you've got a rock in your pocket, don't presume Matt Furor's memo sufficed for all.

StorybookTerror · March 2022

It's nice to hear that the performance issues during prime-time should be better for folks.

@ZOS_MattFiror, we would still very much like a post-mortem for the other U33 issues, like quests and maps and zone guides being broken. At the very least...hearing "We recognize that X, Y, and Z are in fact problems and we do intend to do something" vs. "consequences A and B are intentional changes and it's how we want the game to work going forward".

I don't think we've even really received acknowledgment that most of the problems we reported were...well, even considered real problems. And...that communication would be much more valued than any item reward.

James-Wayne · March 2022

Totally appreciate being honest and upfront about it.... it was the right thing to do!

OnThaLoose · March 2022

@ZOS_MattFiror Thank you for the transparency. You have no idea how much that actually means to us.

silvereyes · March 2022

ZOS_MattFiror wrote: »

Thanks so much for bearing with us and for reading this long explanation. Given the circumstances, I think full disclosure was warranted.

Thank you for the explanation. This is the sort of communication we've been asking for for a long time, and I do appreciate it. I especially appreciate the technical details, the explanation of why decisions were made the way they were, and the willingness to own up to mistakes. These sorts of insights really help to humanize what is going on. I would love for them to be more forthcoming in the future.