Maintenance for the week of March 27:
• [COMPLETE] PC/Mac: NA and EU megaservers for patch maintenance – March 27, 4:00AM EDT (8:00 UTC) - 9:00AM EDT (13:00 UTC)
• Xbox: NA and EU megaservers for patch maintenance – March 28, 6:00AM EDT (10:00 UTC) - 12:00PM EDT (16:00 UTC)
• PlayStation®: NA and EU megaservers for patch maintenance – March 28, 6:00AM EDT (10:00 UTC) - 12:00PM EDT (16:00 UTC)

Update 33 PC Launch Postmortem

  • Darkmage1337

    Thank you so much for your thorough, detailed, and complete explanation.
    This is what great leadership and communication is all about.
    GM of Absolute Virtue. Co-GM of Absolute Vice. 6-time Former Emperor, out of 12 characters. 3 Templars, 3 Sorcerers, 2 Nightblades, 2 Dragonknights, 1 Warden. 1 Necromancer. The Ebonheart Pact: The Dark-Mage (Former Emperor), The Undying Nightshade, The Moonlit-Knight, The Killionaire (Former Emperor), Swims-Among-Slaughterfish (Former Emperor), The Undead Mage The Aldmeri Dominion: The Dawn-Bringer (Former Empress), The Ironwood Kid (Former Emperor), The Storm-Sword The Daggerfall Covenant: The Storm-Shield (Former Empress), The Savage-Beast, The Burning-Crusader CP: 1,800 ID: @Darkmage1337 ESO Platform/Region: PC/NA.
  • Panachudo
    Brilliant. Well done ZOS. This open and straightforward explanation of critical incidents is honourable. Thank you.
  • Bitter_Apple21

    Thank you very much. The explanation is very easily understood.

    As a person whom spends their life solving problems, from my perspective, the team got hit with what I call the "distraction" effect.

    A problem occurred, and the focus was immediately on "what was changed", and hence got distracted from something as mundane as a "broken" device.

    From long experience, I have gotten burnt by being caught by the "distraction" principle in troubleshooting, and it is a very hard thing to break out of.

    Well done and thanks again.

    Edited by Bitter_Apple21 on March 25, 2022 4:49PM
  • Ariont
    Tenthirty2 wrote: »

    Thank you.
    THIS is the kind of communication I was looking for.
    A clear and concise explanation and a no-excuses ownership of the issue and steps to recover reputation.
    It also explains the complexity of the compounding challenges and why they took the shape, and time, they did.

    @ZOS_GinaBruno (cc @ZOS_Kevin )
    You have asked us awhile back what better communication looked like to the player base.
    This thread from Matt, this is it.
    Compensation is nice and all, but to be totally honest I would have been happy with just Matt's explanation.
    It makes me feel like my time and money as a customer is respected, and well spent.

    I have been here trying to figure out how to say what I feel about this whole situation and HERE is someone who has said it for me. Thank You to the team.
  • Hannerbal
    Soul Shriven
    Arunei wrote: »
    I appreciate the honesty here, but I do have a few questions.

    First, why are beta accounts and characters still being held? We don't have access to those accounts or characters, there should be no reason to keep them, at least none I can think of. Unless those are kept on a completely separate, inactive server, why waste database space on them, especially when your answer to database issues was to implement AwA in the way you did?

    As for the 150% Exp Scrolls, that's 10 hours of increased experience. Which is a fraction of time compared to the five days a lot of people lost. You really should be giving people 20 or so of these things, it's not like it would cost you anything, and that would still only be 40 hours of increased experience, not even a full two days. Daily Seals should also be increased for five days, not just the weekly ones.


    Edit because I was wrong, a friend informed me that the 150% scrolls are only one hour long, not two as I had thought. That's even less time people are being compensated for. We really should be getting a lot more of these if this is how you want to handle the compensation.

    I mean, I get it, but they don't owe us anything as compensation.
    PC | NA
    Guildmaster - Phoenix Emporium

    Ngalla - Orc Dragonknight - Main
    Aurelia Val - Breton Templar
    Aesarie - High Elf Sorcerer
    Afeared-of-Deer Khajiit Dragonknight Tank
    Chases-the-Dot - Khajiit Templar
    Eats-Bread-on-Floor - Khajiit Templar
    Eight-Lives-Left - Khajiit Sorcerer
    Encounters-Grave-Problems - Khajiit Necromancer
    Hannerbal Lecter - Nord Warden
    Hates-Mudcrabs - Khajiit Dragonknight
    Interprets-Sarcasm-Poorly - Khajiit Sorcerer
    Midnight-Zoomies - Khajiit Nightblade
    Rigor Morticia - Dark Elf Necromancer
    Sefohna - High Elf Warden
    Snack-That-Smiles-Back - Argonian Templar
    Yatul Blind-Eye - Orc Dragonknight
    Yeet Stick - High Elf Templar
  • superryan94
    Being a mechanic myself, I know things break, and usually at the most untimely moments. I also know that fixing things is sometimes harder than expected because some broken part gives interference in something else making it give weird/ incorrect symptoms for the broken part.

    Seeing what went wrong and how you fixed it(or tried to in the beginning here) makes me really happy as it shows things are actually being done about it and why we had to wait.

    Things break, and us being human causes errors. That's normal.
    But not hearing anything makes it seem like there is some kind of distance between the devs and players, however, seeing this makes it obvious you are actually trying to fix the problem.

    Can we please get more updates on problems, bug and errors, this not only shows you do things wrong, but also you do your best to make them right again, which is even more important.
  • Lonestryder
    Tandor wrote: »
    Thanks @ZOS_MattFiror for the clear explanation of the performance issues, and for engaging with us pretty promptly over this aspect of the concerns many have expressed in the last week or two.

    Could we now please have a similarly clear explanation and engagement to address the other concerns many have expressed in the last month or two over the way acount-wide achievements have been implemented? Specifically, what issues that were highlighted on the PTS and Live forums do you acknowledge and to what extent are you working on them or simply dismissing them as "working as intended"?

    I'm thinking of things like broken NPC interactions and quests, routine NPC dialogue, and broken zone maps and guides. Your thoughts on how these things are working would be very much appreciated.

    Yes, please!

    Matt, I appreciate the thorough explanation related to this week's issues. Good comms! Keep 'em coming!
    Edited by Lonestryder on March 25, 2022 5:00PM
  • Pyr0xyrecuprotite
    I greatly appreciate the in-depth report on how this issue happened, and why resolution took so long.

    I'm still unclear as to why some of us had issues (incredibly bad network performance, it was like being on dial-up) on both PC/EU and PC/NA though. Was the affected network port on some sort of common system?
  • ForeverJenn
    I appreciate the feedback. I personally felt it was important that this was explained as we were under the guise that Cold Storage and Sharding was supposed to make things better, and it seemed, at the time, to have made things a lot worse.

    Thank you.
  • FeedbackOnly
    Please do test dungeon finder before dlcs for dungeons next time.
  • CasgarTheSomnolent
    I love this post. I can't even imagine the complexity of finding that needle the haystack, especially with it being a hardware issue, which no one ever expects. I can imagine the faces (and words coming out of them) when you guys figured it out, and I'm sure it was a relief but not pleasant in the slightest. Thanks for doing what you do, guys.

    In general, I love these peeks into the back end of the ESO machine. It would be so cool and humanizing to get little frequent updates like this about the stuff you guys struggle with and solve that we never know about. Doesn't even have to be big stuff like this. There will always be something, and I think you guys deserve and ought to tell your stories about how you're keeping things going.
  • h8rt3rmin8r
    Soul Shriven

    Thank you much for the very detailed information about the recent issues. This kind of transparency and communication is what has really caused me to deeply appreciate all of the hard work you guys do. I know we all talk trash and complain perhaps too much and you guys have been such a good sport through it all XD

    Keep up the amazing work <3
  • MageCatF4F
    Thanks for the explanation.

    But - ever since update 33 the game is almost unplayable on our oldest computer due to choppy graphics.
    Yes it is still happening. I don't plan to run out and buy a new computer just for ESO.
  • WhiteCoatSyndrome
    Thank you for the transparency! :smiley: I appreciate it, and it looks like I’m not the only one.
    #proud2BAStarObsessedLoonyPAWS (Positively Against Wrip-off Stuff) - Say No to Crown Crates!A useful explanation for how RNG works
  • NeeScrolls
    Since Update 33 launched, I think the PC North American megaserver performance problems deserve some explanation. This post outlines what has been going on the last week or so for our North American PC players.

    The DB Sharding process separates our character databases into a "live" DB and a "cold" DB; all accounts who have logged in over the past year are in the Live DB and older ones are in the Cold DB. With that background information, you can now start to understand what happened since Update 33 launched last Monday.

    So, Update 33 launched last Monday and the plan was to wait until the dust settled, then actually enable sharding on PC NA. Our first big failure was we chalked these reports up to normal server startup issues after a big update.

    . So we – and this was our second large error – decided to move ahead with enabling DB Sharding on the PC NA megaserver without addressing the Activity Finder issues.But the system ran slowly again Monday night so we knew it was something else.

    – the issue was actually caused by a bad (as in failing) network port that was unable to process as much bandwidth as it was configured for. It wasn't a software problem at all; it was a hardware failure that, in essence, slowed down the entire megaserver.

    The TL;DR is that it wasn't related to Update 33, Account Wide Achievements or DB Sharding at all, even though they all happened around the same time and we spent too much time investigating a red herring because of it.

    . Given the circumstances, I think full disclosure was warranted.
    Thank YOU @ZOS_MattFiror for the extensive behind-the-code-curtain explanation and level of contrition/transparency.
    As a former 'Dev' on another MMO, i for one truly enjoyed (and appreciated) your post today very much.

    I only wish u guys would post intel & info like this more often....and not necessarily only when things go haywire. B)

    Anyways, KUDOS to you & the Team! Seems like it was probably just as much of a frustrating coincidence for @ZOS_ crew as it was for all of us playerbase. Onward & upward though yep. -- /tip-hat

    Arunei wrote: »
    First, why are beta accounts and characters still being held? We don't have access to those accounts or characters, there should be no reason to keep them, at least none I can think of..
    uhh excuse me @Arunei but i started in BETA (pre-beta really) and , as sad as it sounds , i'm still here playing since 2013! ;)
    Edited by NeeScrolls on March 25, 2022 5:34PM
  • FinneganFroth
    This is the transparency we as a player base are looking for ZOS. We all understand problems happen, and posts like this are a great step in the right direction. Well done!
  • deleted220717-003304
    Thanks a lot for this clear and detailed explanation; much appreciated!
  • cepheus74
    This does not help people who are away on 4/4.

    What about double Harvesting?
  • Rex-Umbra
    Thanks for giving us the details appreciate it.
    Xbox GT: Rex Umbrah
    GM of IMPERIUM since 2015.
  • gariondavey
    sarahthes wrote: »

    Lastly PLEASE separate the bg queues into dm + objective (backfills into dm after 10-15 minutes if necessary) for next patch. PLEASE...this is so desperately needed.

    @ZOS_MattFiror @ZOS_Kevin @ZOS_GinaBruno

    I'm pretty sure the backfill timer was around that before when there were separate queues. It resulted in everything being a deathmatch because 1 person picking dm would force the 11 who queued random into their dm queue.

    @sarahthes I think you misunderstand. I am saying to not backfill until after 10 or 15 minutes (or at all). That is not what was done before. Backfilling was done immediately from my understanding.
    PC NA @gariondavey, BG + Cyrodiil Focused Since October 2017
    Stamplar (main), Magplar, Magsorc, StamDK, MagDK, Stamblade, Magden, Stamden
  • S0Z0H
    Thank you very much for this deeper dive into what's going on with the server. This kind of in depth message is what is needed. This is part of the kind of communication many of us have been asking for. So going forward, we need more of this.

    When players are left in the dark, they begin to go to other games, and they spread the word. So again, going forward , getting performance up is the ultimate goal but on that path, communicating in real terms like this is so very appreciated.

    I hope this isn't the last time we get such a more detailed update on things. And hopefully this becomes the norm in terms of breaking things down what exactly is going on.

    Not that these explainations won't be critiqued , but it shows you are clearly trying to communicate on a tech level that many actually can comprehend.

    People invest A LOT in this game, it's their hobby, for some, for some it's even a major part of their life , believe it or not.
    So better communication like this is owed to these players.
    Thanks again. Keep up the communication please
  • kestrik
    Soul Shriven
    Thanks so much for bearing with us and for reading this long explanation. Given the circumstances, I think full disclosure was warranted.

    I honestly wasn't expecting an explanation but it sure is appreciated. This post is just awesome.
  • Ulvich
    I can't begin to express how grateful I am for you reaching out like this. I have been posting in these forums for someone to just talk with us and let us know what happened and what to expect. Communication has been a bit slow lately. And sometimes what we see is very vague, at best.
    - Thanks for the information. This really helps.
    Hit Hard. Hit Fast. Hit OftenBETA Group: 8.5b 9 Nov. 22, 2013 - March 14, 2014ESO Member Since: April 03, 2014- Savior of Nirn- Monster Hunter
Sign In or Register to comment.