Greymoor & Update 26 Launch – Game Director Update

ZOS_MattFiror

Now that the dust is settling from the launch of Update 26, I'd like to take a moment give everyone an update on what's been happening in the last week. Obviously it was one of the more bumpy releases we've had, and it's important that you all know the issues we found, what we did to fix them, and in general how things are going.

First and foremost, we remain extremely grateful for the support we get from all of you. As I've said more than a few times, ESO as we know it would not exist without the community that plays it every day. We have been experiencing extremely high player populations, especially PC EU and PC NA, for a few months now. ESO is as popular as it has ever been, and we could not have done it without your participation. Thank you so much, and we'll continue to work diligently every day to make Tamriel your go-to virtual world.

We are still working from home and have been since mid-March. We are taking the pandemic very seriously and are doing what we need to do to ensure that everyone at ZOS and our families are safe. Working from home has presented some challenges, especially considering our large user base, and especially taking into account we launched Update 26 while working remotely.

The first issue we encountered on Tuesday morning was about two minutes after we opened the servers: we had the PTS flag set for live accounts, so everyone had access to Greymoor, whether they purchased it or not. So we had to take servers down and set everything straight – we had some lingering effects all week as some users were caught in limbo having purchased it but not being able to travel there. We have addressed almost all of these cases, but please contact Customer Support if you are still having issues.

The second problem we encountered was much more serious. Update 26 includes many new achievements, especially with a new Chapter and the new Antiquities system. This then combined with an issue where character records saved far more often than intended. These two factors together meant that save times grew over time, backing up all character loads and saves, which led to infinite load screens, timeouts while zoning from one place to another, etc. It took us a while to find the root cause and fix this one, but by Wednesday afternoon (after another maintenance that morning), it was fixed.

Next up was the "zone lag" issue that lasted until Friday where each instance of a zone became less and less responsive the more people entered that instance. This was a bug with combat Synergies – the more often you were in large combat situations (PvP, Dungeons, Trials, Harrowstorms, etc.) the more it bogged down the entire zone, becoming "laggy". This bug was especially disheartening to us, because we included a bunch of performance improvement fixes in U26, but this Synergy bug masked all of them, rendering them moot. After much investigation and analysis, we found and fixed the problem. Friday morning's maintenance contained the fix and finally, by the end of Friday, things were in a better place.

Now, as of Monday afternoon, we're down to smaller and more edge case issues (Guild Traders didn't process correctly on PC EU Sunday evening for example). We will fix these as they arise. It’s important to note that all of the fixes listed so far will be included in the Console and Stadia versions when they launch on June 9 and June 16 respectively, and we will continue to monitor for new problems and address them as they appear.

Now that we’ve fixed the Synergy bug, we are starting to run our metrics analysis on how performance has improved in Update 26. Overall server performance should be improved in all zones, and the controller/keyboard "input lag" issue first seen in Update 25 should be much alleviated. I am not saying that everything is fixed, but it should now be better, smoother, and more responsive overall. We are continuing to work on performance improvements, and we will keep giving you regular updates, although as you saw from our recent post, we are looking to adjust how we communicate our performance improvements to ensure you are getting the information you need.

Thanks everyone. Seriously. We are doing the best we can under the circumstances, and we are very proud that we got Update 26 to launch only one week later than scheduled. Launch could have been smoother, but we will work hard to ensure that ESO is stable and fun for years to come.

Stay safe and healthy, and wash your damn hands.

Matt

Gelmir

Now this is what I call "nerd feedback" Thanks @ZOS_MattFiror !

BansheeVT

ZOS_MattFiror wrote: »

(...) Now, as of Monday afternoon, we're down to **smaller and more edge case issues (Guild Traders didn't process correctly on PC EU Sunday evening for example). We will fix these as they arise.*** It’s important to note that all of the fixes listed so far will be included in the Console and Stadia versions when they launch on June 9 and June 16 respectively, and we will continue to monitor for new problems and address them as they appear.
(...)

As much as I appreciate the update on the issues and the work that got put in to fix these issues I'm really disappointed that a whole system which is mandatory for trading (which is a huge part of ESO) has crashed AGAIN and it's only seen as "smaller and more edge case issues" and put in braces like it wasn't that important and nobody noticed..

Princess_Ciri

Although i can't see a single mention of it here, I did notice (and so have many end game PvE players) that you seem to have fixed the issues where bosses and adds were spinning around in circles and lagging terribly.

Don't know why it wasn't mentioned, because it's been a terrible issue plaguing the game since the 'performance patch' a few months ago.

Still, thanks for fixing. The game is fun to play again.

ESO_Nightingale

I hope there is room for balance updates next patch!

sirinsidiator

Nice to see a post like this which actually goes somewhat into details about what is happening behind the scenes. Definitely something you should do more often!

Just one thing that somewhat irks me as a long time player and GM of a trading guild.

ZOS_MattFiror wrote: »

As I've said more than a few times, ESO as we know it would not exist without the community that plays it every day.

You say so, but in the same post you dismiss a pretty major problem that had quite the negative impact on the whole EU community as being an edge case.

ZOS_MattFiror wrote: »

smaller and more edge case issues (Guild Traders didn't process correctly on PC EU Sunday evening for example).

It's true that it only directly affects ~230 players, but they represent guilds with up to 500 members each and the countless people who buy stuff from the guild traders. Not something I would call a small edge case.

It's also not that trader change just suddenly stopped working correctly this week. It's been busted and taking more than 20 minutes for almost two months now and it was pretty clear that at some point something like this would happen (for me at least). Yet there is no fix or even an acknowledgement of that specific problem in sight, like for so many other things that by this point seem to have just become "part of the game" that we have to live with. (mushy ground textures for example - I report that for years already with easy steps how to reproduce it and there is still no progress).

In general I feel you should show guilds more love. I believe they are the glue that keeps the community you have in such high regard together and alive, but ever since launch there have barely been any improvements to the fundamentals of that system (guild finder aside). Quite the opposite actually (guild history nerf).

Maybe it's time to do a guild update which focuses on bringing this essential part of the game into the year 2020?

VaranisArano

Thank you for the update! While its easy for us to get discouraged while in the middle of problems and feel like there's a dearth of communication, it is good to have these after-the-fact update posts as well.

One follow-up question: With such high player populations now and ESO continuing to grow in popularity, what's the plan for scaling up the current game in preparation for both continued player growth and population spikes for popular events like Midyear Mayhem, New Life, or free trials?

MissBizz

Princess_Ciri wrote: »

Although i can't see a single mention of it here, I did notice (and so have many end game PvE players) that you seem to have fixed the issues where bosses and adds were spinning around in circles and lagging terribly.

Don't know why it wasn't mentioned, because it's been a terrible issue plaguing the game since the 'performance patch' a few months ago.

Still, thanks for fixing. The game is fun to play again.

That seemed to be random as per the instance so I'm not convinced it's for sure fixed. I only really noticed it in trials and have only been to a handful since the update

xaraan

Thank you for the detailed information on what happened. It does seem like a lot of the issues have cleared up.

I have to say though, I'm still getting random loading screens for no reason just walking around. Happens across country in cyrodiil (especially up north), in Craglorn, in Rimmen, and even in Solitude (has happened three times in last few days). Which is even more worrisome considering the city is doored off and not exactly a crafting hub (ie it wasn't that crowded when this happened compared to cities that are hubs, even for a new zone).

Also the new trial is still pretty buggy. Players consistently crashing when trying to rez up after a wipe (we had a different player DC every time when rezzing). Players stuck in loading screens going from door to door. It took us 10+ minutes all sitting around waiting for one player to log in after one of those DCs and then having to walk through several loading doors to get back to boss fight (can't try pull in method in case they get stuck in a loading screen coming into fight like has happened in vSS many times). And on top of that, the trial also loses sound, the whole game almost, and this has happened every time we've done the trial to at least 1/3 of the team. Very hard to play when you are used to using sounds as combat cues.

GlassHalfFull

@ZOS_MattFiror
I find it really nice to see a post like this with lots of useful information. I encourage you to do this more often. As a software engineer, I can relate to these problems, solutions and working from home.

code65536

Princess_Ciri wrote: »

Although i can't see a single mention of it here, I did notice (and so have many end game PvE players) that you seem to have fixed the issues where bosses and adds were spinning around in circles and lagging terribly.

Don't know why it wasn't mentioned, because it's been a terrible issue plaguing the game since the 'performance patch' a few months ago.

Still, thanks for fixing. The game is fun to play again.

@Princess_Ciri There was an additional backend adjustment that was made some time Friday evening, a number of hours after that morning's hotfix (oddly, ZOS hasn't said anything about this publicly; I know about it only because some trials guilds were asked to send feedback about trials performance after this adjustment), which seems to have mostly resolved the issue.

It's not perfect, and there can still be "bad" instances. But it's markedly better than how things were for the entirety of Update 25. Whatever it is that they changed Friday evening, it seems to be responsible for the improved trials performance.

Sarousse

@ZOS_MattFiror

Performance in cyrodiil as soon as there is 3 bars or organized groups is awful, and skill delay is unbrearable.

No need to monitor. Just go to cyrodiil on eu server during prime time, and play.

Ode2Order

I'm happy to read some updates from you, communication like this is really appreciated! Thank you for all your hard (remote) work!

That being said, any hope Craglorn's "loading screens every second" issue will be addressed soon?

gatekeeper13

Cyro performance is as bad as it was prior Update 26. Zero performance improvement.

Also lag has skyrocketed in almost all areas of the game.

LLawlietW

Posts like this go a long way for communication about issues with the game and the community. These types of posts should happen way more frequently than they do. most players have an understanding that things are not going to be perfect when they are going live, especially when WFH and quarantine happens. But just let us know what is happening with the game and what you are doing to try to fix it. communicate with the community more on these issues and it will help alleviate the frustration.

technohic

ZOS_MattFiror wrote: »

Now that we’ve fixed the Synergy bug, we are starting to run our metrics analysis on how performance has improved in Update 26. Overall server performance should be improved in all zones, and the controller/keyboard "input lag" issue first seen in Update 25 should be much alleviated. I am not saying that everything is fixed, but it should now be better, smoother, and more responsive overall. We are continuing to work on performance improvements, and we will keep giving you regular updates, although as you saw from our recent post, we are looking to adjust how we communicate our performance improvements to ensure you are getting the information you need.

I'm not sure it is THAT much better. I don't know how recently you looked before going through this write up but as more people return to their regular activities rather than just antiquities and the new chapter; it's starting to feel just as bad.

Nomad1098

I think someone forgot about pvp. Its still not performing well. Congrats on fixing one bug though.

KageNin

Input and animations still hit and miss mostly miss, cya in few months.

React

ZOS_MattFiror wrote: »

Now that the dust is settling from the launch of Update 26, I'd like to take a moment give everyone an update on what's been happening in the last week. Obviously it was one of the more bumpy releases we've had, and it's important that you all know the issues we found, what we did to fix them, and in general how things are going.

First and foremost, we remain extremely grateful for the support we get from all of you. As I've said more than a few times, ESO as we know it would not exist without the community that plays it every day. We have been experiencing extremely high player populations, especially PC EU and PC NA, for a few months now. ESO is as popular as it has ever been, and we could not have done it without your participation. Thank you so much, and we'll continue to work diligently every day to make Tamriel your go-to virtual world.

We are still working from home and have been since mid-March. We are taking the pandemic very seriously and are doing what we need to do to ensure that everyone at ZOS and our families are safe. Working from home has presented some challenges, especially considering our large user base, and especially taking into account we launched Update 26 while working remotely.

The first issue we encountered on Tuesday morning was about two minutes after we opened the servers: we had the PTS flag set for live accounts, so everyone had access to Greymoor, whether they purchased it or not. So we had to take servers down and set everything straight – we had some lingering effects all week as some users were caught in limbo having purchased it but not being able to travel there. We have addressed almost all of these cases, but please contact Customer Support if you are still having issues.

The second problem we encountered was much more serious. Update 26 includes many new achievements, especially with a new Chapter and the new Antiquities system. This then combined with an issue where character records saved far more often than intended. These two factors together meant that save times grew over time, backing up all character loads and saves, which led to infinite load screens, timeouts while zoning from one place to another, etc. It took us a while to find the root cause and fix this one, but by Wednesday afternoon (after another maintenance that morning), it was fixed.

Next up was the "zone lag" issue that lasted until Friday where each instance of a zone became less and less responsive the more people entered that instance. This was a bug with combat Synergies – the more often you were in large combat situations (PvP, Dungeons, Trials, Harrowstorms, etc.) the more it bogged down the entire zone, becoming "laggy". This bug was especially disheartening to us, because we included a bunch of performance improvement fixes in U26, but this Synergy bug masked all of them, rendering them moot. After much investigation and analysis, we found and fixed the problem. Friday morning's maintenance contained the fix and finally, by the end of Friday, things were in a better place.

Now, as of Monday afternoon, we're down to smaller and more edge case issues (Guild Traders didn't process correctly on PC EU Sunday evening for example). We will fix these as they arise. It’s important to note that all of the fixes listed so far will be included in the Console and Stadia versions when they launch on June 9 and June 16 respectively, and we will continue to monitor for new problems and address them as they appear.

Now that we’ve fixed the Synergy bug, we are starting to run our metrics analysis on how performance has improved in Update 26. Overall server performance should be improved in all zones, and the controller/keyboard "input lag" issue first seen in Update 25 should be much alleviated. I am not saying that everything is fixed, but it should now be better, smoother, and more responsive overall. We are continuing to work on performance improvements, and we will keep giving you regular updates, although as you saw from our recent post, we are looking to adjust how we communicate our performance improvements to ensure you are getting the information you need.

Thanks everyone. Seriously. We are doing the best we can under the circumstances, and we are very proud that we got Update 26 to launch only one week later than scheduled. Launch could have been smoother, but we will work hard to ensure that ESO is stable and fun for years to come.

Stay safe and healthy, and wash your damn hands.

Matt

Saying that performance has improved in all zones and that the input lag has been "Much alleviated" is a flat out lie. It has improved from the LAUNCH of greymoor to two weeks after the launch, but from one year ago today it has continually degraded every patch.

The state of cyrodiil is mind numbing. As soon as there are more than 3 TOTAL bars of population in the main campaign, the input delay is horrendous. For PVP players, having the main PVP zone be unplayable more than 80% of the time is reason enough not to play the game at all. Additionally we are now seeing terrible input delay while dueling in major zones, even after the "fix" that alleviated the 1000+ ping issue after the launch of greymoor. The delay in major zones was NOT present at all last patch, so this is something that you clearly need to re visit.

When the AP event happens this month, the outrage you're going to experience will be overwhelming. The whole game lagged as a result of the AP event years ago, and the state of the game is FAR worse now than it was then. I really hope that when this disaster of an event occurs and a large portion of the playerbase is exposed to what PVP performance really is, you'll be forced to acknowledge finally that nothing has improved on that front in the past two years.

mav1234

Thank you for the updates, I appreciate the detailed descriptions of what was fixed. Seriously, it is VERY welcome!

I wanted to add that I have not noticed an improvement to input lag in intense - PvP and PvE - situations this patch relative to last patch. I still notice *extremely* frequently that e.g. my armor buff requires multiple key presses. Maybe it is less frequent and I just don't realize how frequent it was last patch, but it is still very noticeable this patch. This is frustrating as a tank in PvE and downright deadly in PvP. On top of that, the servers continue togrind to a halt in large scale Cyro encounters. Since I play this game primarily to PvP with my friends - and since BG queues were made solo only - Cyro is my endgame. Its inability to perform is crippling to my enjoyment of the game.

None the less, I appreciate the transparency. I hope it continues. I just want to see things get better for the game I love. And finally, thank you and the ZOS team for working during the pandemic as best you can.

Tarrocan

Sorry but how will u fix this game when this combat system is broken as ***? i cant even play with skills anymore lol and when i hit someone with a light attack between skills then i feel like i won in lotto

Czekoludek

Wow, example of proper communication from ZOS in 2020. More of that please, it is nice to understand last issues.

Derra

ZOS_MattFiror wrote: »

the controller/keyboard "input lag" issue first seen in Update 25 should be much alleviated. I am not saying that everything is fixed, but it should now be better, smoother, and more responsive overall.

It is not.

Thoragaal

Sarousse wrote: »

@ZOS_MattFiror

Performance in cyrodiil as soon as there is 3 bars or organized groups is awful, and skill delay is unbrearable.

No need to monitor. Just go to cyrodiil on eu server during prime time, and play.

To monitor includes collecting data. They are well aware of the issues in Cyrodiil already.

Thoragaal

ZOS_MattFiror wrote: »

Now that the dust is settling from the launch of Update 26, I'd like to take a moment give everyone an update on what's been happening in the last week [...]

Matt

Thank you for the update Matt. More of this, please!

Gythral

More results - less waffle , please!

Fluffy_Fr0zR

Derra wrote: »

ZOS_MattFiror wrote: »

the controller/keyboard "input lag" issue first seen in Update 25 should be much alleviated. I am not saying that everything is fixed, but it should now be better, smoother, and more responsive overall.

It is not.

It is currently at it's very worst. But worry not, I'm sure there's going to be a new "worst" not too long from now!

karliahquinn

Also really disappointed that the Guild trader mess up is described as an edge issue, when it has a huge impact on the game for all who buy and sell as well as GMs and trade guilds. It's also happened about 4 times so a preventative fix would be ideal.

UntilValhalla13

Is this update going to fix the stamina bar bug, and the problem where skills don't fire/looks like your character is twitching? Trials have been unplayable on PS4 for quite a while.

thedavidventer

Thank you @ZOS_MattFiror

These updates are much appreciated!

As a player on the macOS client, I can say that things have definitely greatly improved and I’m having much fun exploring the new areas. Thanks for all the work you guys are putting into this. I’m obsessed with this game!

That said, there are two seriously frustrating and debilitating macOS game client specific issues that still need to be fixed in order to make this game run “as intended“ on macOS with full macOS feature compatibility. Please see the following threads detailing these issues incl. Apple developer documentation links on how to fix them:

1) Fullscreen/Multitasking/Spaces integration on the macOS Game client:

https://forums.elderscrollsonline.com/en/discussion/520106/broken-windowed-fullscreen-mode-and-limited-resolution-on-macos-catalina#latest

2) Controller input on the macOS game client:

https://forums.elderscrollsonline.com/en/discussion/520104/gamepad-input-is-broken-in-eso-on-macos-catalina#latest


Please also take a look at this thread for more details:

https://forums.elderscrollsonline.com/en/discussion/508005/state-of-eso-on-the-mac-2020#latest

Thanks again for all you’ve done so far, I’m looking forward to further improvements on the macOS game client.

volkeswagon

Just goes to show how difficult your job can be and a little patience, respect and understanding from the community rather than name calling rants would be appreciated. The comments i see about how Zos doesn't care or listen always makes me shake my head.