Greymoor & Update 26 Launch – Game Director Update

agegarton

Maybe it's just me, but I find this a really disappointing update. I feel like I'm being treated like an idiot.

No-one, myself included, expects perfection. There are going to be mistakes, bugs, issues, all sorts. I do expect a bit of quality control though. I expect a bit of effort to into quality control, and into communicating with the player (=payer) base when stuff goes wrong.

Greymoor wasn't "could do better" territory. It was an unmitigated disaster, stuck on repeat. Servers were taken down many multiple times. Multiple issues were thought fixed and then..... oops..... they weren't. A number of the issues are straight up repeats of problem we have had before, like the guild trader problem.

And I simply do not buy the "we just have sooooo many players" excuse any more. You rolled that one out two years ago. How long does it take to properly prepare?

And finally...... do you guys remember that the centre of the map is a place called "Cyrodiil"? It's not a bit broken, a little bit annoying - it's totally unplayable at prime time.

[snip]

[Edited to remove Bashing]

Lab3360

ZOS_MattFiror wrote: »

Now that the dust is settling from the launch of Update 26, I'd like to take a moment give everyone an update on what's been happening in the last week. Obviously it was one of the more bumpy releases we've had, and it's important that you all know the issues we found, what we did to fix them, and in general how things are going.

First and foremost, we remain extremely grateful for the support we get from all of you. As I've said more than a few times, ESO as we know it would not exist without the community that plays it every day. We have been experiencing extremely high player populations, especially PC EU and PC NA, for a few months now. ESO is as popular as it has ever been, and we could not have done it without your participation. Thank you so much, and we'll continue to work diligently every day to make Tamriel your go-to virtual world.

We are still working from home and have been since mid-March. We are taking the pandemic very seriously and are doing what we need to do to ensure that everyone at ZOS and our families are safe. Working from home has presented some challenges, especially considering our large user base, and especially taking into account we launched Update 26 while working remotely.

The first issue we encountered on Tuesday morning was about two minutes after we opened the servers: we had the PTS flag set for live accounts, so everyone had access to Greymoor, whether they purchased it or not. So we had to take servers down and set everything straight – we had some lingering effects all week as some users were caught in limbo having purchased it but not being able to travel there. We have addressed almost all of these cases, but please contact Customer Support if you are still having issues.

The second problem we encountered was much more serious. Update 26 includes many new achievements, especially with a new Chapter and the new Antiquities system. This then combined with an issue where character records saved far more often than intended. These two factors together meant that save times grew over time, backing up all character loads and saves, which led to infinite load screens, timeouts while zoning from one place to another, etc. It took us a while to find the root cause and fix this one, but by Wednesday afternoon (after another maintenance that morning), it was fixed.

Next up was the "zone lag" issue that lasted until Friday where each instance of a zone became less and less responsive the more people entered that instance. This was a bug with combat Synergies – the more often you were in large combat situations (PvP, Dungeons, Trials, Harrowstorms, etc.) the more it bogged down the entire zone, becoming "laggy". This bug was especially disheartening to us, because we included a bunch of performance improvement fixes in U26, but this Synergy bug masked all of them, rendering them moot. After much investigation and analysis, we found and fixed the problem. Friday morning's maintenance contained the fix and finally, by the end of Friday, things were in a better place.

Now, as of Monday afternoon, we're down to smaller and more edge case issues (Guild Traders didn't process correctly on PC EU Sunday evening for example). We will fix these as they arise. It’s important to note that all of the fixes listed so far will be included in the Console and Stadia versions when they launch on June 9 and June 16 respectively, and we will continue to monitor for new problems and address them as they appear.

Now that we’ve fixed the Synergy bug, we are starting to run our metrics analysis on how performance has improved in Update 26. Overall server performance should be improved in all zones, and the controller/keyboard "input lag" issue first seen in Update 25 should be much alleviated. I am not saying that everything is fixed, but it should now be better, smoother, and more responsive overall. We are continuing to work on performance improvements, and we will keep giving you regular updates, although as you saw from our recent post, we are looking to adjust how we communicate our performance improvements to ensure you are getting the information you need.

Thanks everyone. Seriously. We are doing the best we can under the circumstances, and we are very proud that we got Update 26 to launch only one week later than scheduled. Launch could have been smoother, but we will work hard to ensure that ESO is stable and fun for years to come.

Stay safe and healthy, and wash your damn hands.

Matt

My goodness. Thank you for the update about performance fix. We have been asking since last performance update and heard nothing.

I am happy to hear you guys have been working on it. Just wish you guys would have communicated it long ago rather than letting your player base stew or leave.

runa_gate

Thank you.

For 6 years this is the type of communication we've been looking for, and this is much better. I have always been baffled by the wall of silence erected against the player community. Even the poor community managers don't look quite so terrified of accidentally letting some proscribed phrase slip. The cringe of the early days when the look of terror in every on-camera employee's eyes is still hard to bear.

From not mentioning in-game moderators in revolving shifts were stationed in an early quest instance to teleport questers past a certain door in the early days when so many quests had game-breaking bugs to not acknowledging PvP performance has never been acceptable since that awful month of 5 FPS at almost all times when a whole guild learned how to spam and bring the servers to it's knees. You'll know about that due to my sending the entire day's chat transcript directly to the TOS team. It was conscious and deliberate and went on for a very long time, albeit long before most who play now even started playing ESO.

So to see this post genuinely surprises me and warms my heart. Was it really so hard? Look at the reaction. You could have had our trust from the very start by not making the presumption that secrecy is paramount.

Firstmep

Rianai wrote: »

Cinbri wrote: »

And how about problem that people who didn't bought Chapter but ported there due to bug, gained access to new lines and farmed new Mythic sets before zos fully closed access, so now those people playing with Mythic sets without having Chapter. How bout fix that too.

Yes, we can't have players who don't keep shoveling money at a broken game on even terms. Gotta enforce P2W.

Eso has to be the first MMO where buying an expansion is considered p2w.
Wow increases the level cap and adds tons of new, more powerful items into expansions, yet no one calls it p2w.
And yes, as someone who spent the 30 euros or whatever buying Greymoor, I'm not happy that people who didn't, not only got access to it, but also got to keep their I'll gotten goods.
It pretty much makes my 30Euros look like a donation, since I could've just not spent that money and still gotten what I have wanted from the expac.
Also the people who did exploit the bug, should at the very least receive a warning, I don't belive anyone who did this wasn't fully aware that they didn't purchase the expac and thus shouldn't have access to it.

hiyde

Hey @ZOS_MattFiror,

Great update and it was really interesting to get a peek behind the curtain to see what was causing different issues.

You would have been completely justified in delaying this launch, yet your team perservered and gave us some really fun new content when many of us are stuck at home. ZOS peeps kept getting their paychecks. We got new content. Win/Win.

"...we're down to smaller and more edge case issues (Guild Traders didn't process correctly on PC EU Sunday evening for example). We will fix these as they arise..."

This wording just felt like an unforced error. You were doing so well, Matt!

Karivaa

So have you also fixed these for the Xbox launch or will we have these same issues?

VaranisArano

Karivaa wrote: »

So have you also fixed these for the Xbox launch or will we have these same issues?

He does say: "It’s important to note that all of the fixes listed so far will be included in the Console and Stadia versions when they launch on June 9 and June 16 respectively, and we will continue to monitor for new problems and address them as they appear."

So hopefully he's right about that, and any new problems you get are easily and quickly set with.

Vaoh

@ZOS_MattFiror I’ve been critical of many things that have happened to ESO over the years, particularly console game performance.

I gotta give props where it’s due though.

Telling us why the launch was bumpy (...to put it mildly lol) is definitely the communication we are looking for. Ofc we want things to improve too. But in the meantime, having communication like this goes a long way with your players.

DarcyMardin

Your update is much appreciated. Thanks to the entire team for all your hard work.

DUTCH_REAPER

I came back to eso after Greymoor dropped. Got to first harrowstorm and it lagged badly. I logged off. Came back last night and went into cyrodil. Very poor gameplay die to lag. Please post when its fixed and me and my friends will come running back.

technohic

technohic wrote: »

ZOS_MattFiror wrote: »

Now that we’ve fixed the Synergy bug, we are starting to run our metrics analysis on how performance has improved in Update 26. Overall server performance should be improved in all zones, and the controller/keyboard "input lag" issue first seen in Update 25 should be much alleviated. I am not saying that everything is fixed, but it should now be better, smoother, and more responsive overall. We are continuing to work on performance improvements, and we will keep giving you regular updates, although as you saw from our recent post, we are looking to adjust how we communicate our performance improvements to ensure you are getting the information you need.

I'm not sure it is THAT much better. I don't know how recently you looked before going through this write up but as more people return to their regular activities rather than just antiquities and the new chapter; it's starting to feel just as bad.

Quoting myself here because when its pop locked again; it really is no better @ZOS_MattFiror

furiouslog

I wanted to note that the input lag and misfiring issue is definitely better, but not completely fixed. So thanks for the progress that's been made, but there is still more to do, I think.

One thing that still happens a lot on my magplar is that when I use beam (almost always during execute), I often get stuck in a heavy attack animation.

GlorphNoldorin

THis

furiouslog wrote: »

I wanted to note that the input lag and misfiring issue is definitely better, but not completely fixed. So thanks for the progress that's been made, but there is still more to do, I think.

One thing that still happens a lot on my magplar is that when I use beam (almost always during execute), I often get stuck in a heavy attack animation.

This is no where near better, in any heavy load area, vma, pvp, trials, I find intolerable skill delay.

JayKwellen

furiouslog wrote: »

I wanted to note that the input lag and misfiring issue is definitely better, but not completely fixed. So thanks for the progress that's been made, but there is still more to do, I think.

One thing that still happens a lot on my magplar is that when I use beam (almost always during execute), I often get stuck in a heavy attack animation.

For me the input lag, in Cyrodiil at least, is still terrible but "different" than it was before.

Everyone knows how it was previously, but now it seems like the server is sometimes just completely failing to register ability activations.

Since Greymoor launched on console I've noticed that now I'll press a button, I'll see it on my bar as being pressed, and then nothing. No delay, no lag, just literally nothing. It's as if I pressed the key, it registered, and then the server simply forgot to make it happen.

Unlike previously I don't get locked in animations, don't lose a GCD (or two), the ability doesn't suddenly go off 2-3 seconds later, everything is running more "smoothly", but my abilities just simply aren't activating at all. It's weird and feels strange.

It honestly feels like the game is gaslighting me at this point as I often end up going in circles with myself and doubting my own sanity "Did I press X? I think I did? Did I miss it? No, I know i hit or right. Look, I just saw the ability button go off but nothing happened! Is that what happened last time or did I make a mistake? I know I pressed it. But maybe I didn't though?"

It's almost more aggravating than previously because at least with the old iteration of input lag you had enough feedback (ie everything not working properly) to know what was going on. This is almost more insidious, and absolutely more irritating.

smee_z

@ZOS_MattFiror

Thanks Matt.
Much appreciated.
Hope we get this regularly not just on post ESO update launches.

DaveMoeDee

Is this really fixed:

ZOS_MattFiror wrote: »

...

The second problem we encountered was much more serious. Update 26 includes many new achievements, especially with a new Chapter and the new Antiquities system. This then combined with an issue where character records saved far more often than intended. These two factors together meant that save times grew over time, backing up all character loads and saves, which led to infinite load screens, timeouts while zoning from one place to another, etc. It took us a while to find the root cause and fix this one, but by Wednesday afternoon (after another maintenance that morning), it was fixed.

...

Matt

I feel like this still happens a few times every play session when changing zones. Too the point when I'm really starting to get frustrated.

Elsonso

DaveMoeDee wrote: »

Is this really fixed:

ZOS_MattFiror wrote: »

...

The second problem we encountered was much more serious. Update 26 includes many new achievements, especially with a new Chapter and the new Antiquities system. This then combined with an issue where character records saved far more often than intended. These two factors together meant that save times grew over time, backing up all character loads and saves, which led to infinite load screens, timeouts while zoning from one place to another, etc. It took us a while to find the root cause and fix this one, but by Wednesday afternoon (after another maintenance that morning), it was fixed.

...

Matt

I feel like this still happens a few times every play session when changing zones. Too the point when I'm really starting to get frustrated.

I have not seen this on PC since the patch.

Gythral

And only 1 day after a reboot
Graymoor is broken again on PC-EU

[snip]

[Edited to remove Bashing]

Gythral

And buried so that the on going issues never happened/are not in fact continuing...

Destai

@ZOS_MattFiror Are you going to continue the dialogue with us or just let it drop? Asking for a friend.

Blumog

Thank you for this very informative and detailed update, Matt! @ZOS_MattFiror
I really appreciate all the time, work and love you guys put into the game.

My first The Elder Scrolls game was Arena back in 1994 and I played near all the following titles.
It's such a wonderful and fun ride so far and I can't wait for what's coming next. ^^

To get such detailed informations on a more regular basis would be awesome.
Thank you again for your time and letting us people know what is going on right now.

Stay save and healthy guys... ESO forever!
Blumog

Mayrael

GlorphNoldorin wrote: »

THis

furiouslog wrote: »

I wanted to note that the input lag and misfiring issue is definitely better, but not completely fixed. So thanks for the progress that's been made, but there is still more to do, I think.

One thing that still happens a lot on my magplar is that when I use beam (almost always during execute), I often get stuck in a heavy attack animation.

This is no where near better, in any heavy load area, vma, pvp, trials, I find intolerable skill delay.

This. Cyrodiil on PC EU is one big lagfest during prime time, even NPCs seem to not handle the situation and they are behaving wierd to.

Inaya

@ZOS_MattFiror Thank you for the update. I cannot even imagine trying to push a huge update while everyone is working from home. While it was bumpy I think under the circumstances you all did a great job, barring the everyone get's access slip.

There are still many issues that need to be corrected. I understand they are probably more complicated to fix than we realize but communication like this is key to maintaining a content player base.

Stay safe and thanks for keeping everyone employed and working thru this pandemic.