Game Performance Improvements Preview – Update 27

ZOS_JessicaFolsom

Hi everyone,

Below is the first of our quarterly game performance improvement previews. This was written by our Engineering team and explains the main performance improvements planned for Update 27. If you haven’t yet, we encourage you to also check out the Update 27 Combat Preview here.

--

Hi friends!

We're here to bring you a preview of some of the performance improvements planned for Update 27. While it is impossible to capture everything that the team has done, we aim to highlight some of the areas we are focusing on and what we expect to gain as a result.

Persistent AoE Improvements: Area of effect abilities that last for long durations have the potential to send a large number of messages down to the client over time. This is exacerbated in zones like Cyrodiil where many players are routinely casting and being hit by these types of abilities. To mitigate this, we have implemented new AoE tech on the server that will keep the same functionality, but significantly reduce the number of hit result messages the client has to process. Internal testing has been positive, showing a reduction in latency and FPS drops on the client. We are taking a cautious approach to rolling out this new tech, choosing to focus solely on Dragonknight Standard (and morphs) for U27. We will evaluate the improvements on the PTS and live servers and continue to update additional player abilities in future updates.

Note: In addition to the persistent AoE Improvement work, the team will also be conducting a series of tests on the live servers in the next few weeks. These tests will center around AoE’s in Cyrodiil and will allow us to gather even more metrics of real-world scenarios. Expect more info about what exactly the tests will entail in the next couple weeks.

Item Set Abilities: The team has been hard at work auditing and streamlining each and every item set ability in the game, with a focus on looking for ways to improve performance. This has resulted in reduced calculations on the server and, similarly to the AoE improvements, a reduction in the number of messages sent down for the client to process.

Critical Memory Situations: We know how frustrating it is to teleport into a busy city and encounter a crash. While we have significantly reduced the number of out-of-memory related crashes in recent updates, it is still something that can happen to our players. To prevent these crashes, the console versions will now detect when the game is about to run out of memory and react by temporarily unloading vanity pets and personalities from players outside of your group. We were able to build upon the work completed in U26 that moved vanity pets to the client to help make this possible. Once memory is back within a safe threshold, these assets will begin to load back in as players move through the world.

Physics Implementation: Our Havok physics engine implementation, specifically dealing with Keep pieces in Cyrodiil, has had some issues over the years. We took some time to clean up the code surrounding this, reducing unnecessary complexity and making Keep pieces behave more like normal fixtures. Essentially, they will still look and behave as they do now. While working on this, we tracked down an elusive bug where some physics objects were not being cleaned up properly. This issue was not specific to any zone and would result in the client crashing when it occurred. The end result of the physics hardening should be fewer bugs and crashes.

Trial Performance: Our metrics have shown that Trial performance on the server has not been a consistent experience. We were able to track down the root cause of this–instance launchers running multiple Trials simultaneously would routinely perform worse overall. We've taken a pass at the algorithm which determined how Trial instances are distributed and now factor this into account. This will decrease the chances of our instance launchers being overloaded in high usage situations and impacting Trial performance as a result.

Database Improvements: We've made some backend changes to incorporate an in-memory database cache in our server processes, which will reduce the amount of contention on our production databases when under stress. Initially for U27, we are setting the foundation and focusing on a small piece of the Activity Finder tool which should result in smoother processing overall. Once we’ve evaluated this on the live servers, it will create opportunities in future updates to move over additional functionality, allowing us to perform more work simultaneously without performance degradation.

Intermittent Load Screens: It's a source of frustration for everyone when you're running around Tamriel and you're intermittently hit with short load screens. We've identified two areas where this behavior could occur. The first issue is a bug with streaming of scaled assets that are switching between LOD models and main models. This should go to the live servers with an upcoming U26 incremental. The second issue is regarding a bug that could cause the game to have much bigger physics streaming radius when a high view distance is selected, resulting in more loads than necessary. This fix will release with U27. We will be actively monitoring the live server metrics after U27 releases to determine if additional investigation and work is needed.

We know this won't solve every issue our players are experiencing, and there is still much work to be done. We will continue working diligently to provide you the best experience possible. Be on the lookout for some experimental Cyrodiil changes over the course of U27. Details for that will come in a future update. Thank you!

Cinbri

I hope any of those will fix new bug twhere using gap-closer ability during high latency, i.e. primetime cyro will guarantee result in your charater to be desynced.

Niaver

Sounds promising. I kind of like how they are at rolling changes in steps to test them on live servers.

asalemi

Sounds great but It’s words. Let’s see action.

Aynethol

Fair play on the transparency so far and keeping us in the loop with the performance updates!

Rianai

What about the frequent unresponsiveness of skills and other actions such as break free, the reduced reliability of block and dodge and desyncs of all kind (position, animation, resource bars, ...) that got introduced or worsened with the harrowstorm update?

technohic

Note: In addition to the persistent AoE Improvement work, the team will also be conducting a series of tests on the live servers in the next few weeks. These tests will center around AoE’s in Cyrodiil and will allow us to gather even more metrics of real-world scenarios. Expect more info about what exactly the tests will entail in the next couple weeks.

I'm a bit frustrated it's taken this long to come up with this but glad it's finally being done. Many of us have noticed higher latency when certain types of groups using mass AOE are around so I think this is a good place to look. I wouldn't expect DK Standard though; as the first place to start. Maybe grave robber or something more spammable but at least its something.

The other thing that happens in this is speed from building to be fast, to gap closers; almost guarantee position desync where no ability will go off and often you rubberband back to where the server list you.

Nairinhe

As a dev I can say that good code cleanup always sparks joy!

ZOS_JessicaFolsom

Cinbri wrote: »

I hope any of those will fix new bug twhere using gap-closer ability during high latency, i.e. primetime cyro will guarantee result in your charater to be desynced.

Just checked in with the team about this and they are still investigating the root cause of this one.

ESO_Nightingale

ZOS_JessicaFolsom wrote: »

Cinbri wrote: »

I hope any of those will fix new bug twhere using gap-closer ability during high latency, i.e. primetime cyro will guarantee result in your charater to be desynced.

Just checked in with the team about this and they are still investigating the root cause of this one.

does this also mean they're looking at nature's grasp's rubber banding issues? that skill is a gap closer but a bit different since you aim it at an ally.

carlos424

technohic wrote: »

Note: In addition to the persistent AoE Improvement work, the team will also be conducting a series of tests on the live servers in the next few weeks. These tests will center around AoE’s in Cyrodiil and will allow us to gather even more metrics of real-world scenarios. Expect more info about what exactly the tests will entail in the next couple weeks.

I'm a bit frustrated it's taken this long to come up with this but glad it's finally being done. Many of us have noticed higher latency when certain types of groups using mass AOE are around so I think this is a good place to look. I wouldn't expect DK Standard though; as the first place to start. Maybe grave robber or something more spammable but at least its something.

The other thing that happens in this is speed from building to be fast, to gap closers; almost guarantee position desync where no ability will go off and often you rubberband back to where the server list you.

I’m glad I’m not the only one scratching my head about the DK standard. You hardly even see standards in Cyrodil, it’s all leaps. Maybe just a sneaky way to nerf standard? Maybe tick every 2 seconds instead of every second? Lol. I guess we’ll see.

ItsJustHashtag

technohic wrote: »

Note: In addition to the persistent AoE Improvement work, the team will also be conducting a series of tests on the live servers in the next few weeks. These tests will center around AoE’s in Cyrodiil and will allow us to gather even more metrics of real-world scenarios. Expect more info about what exactly the tests will entail in the next couple weeks.

I'm a bit frustrated it's taken this long to come up with this but glad it's finally being done. Many of us have noticed higher latency when certain types of groups using mass AOE are around so I think this is a good place to look. I wouldn't expect DK Standard though; as the first place to start. Maybe grave robber or something more spammable but at least its something.

The other thing that happens in this is speed from building to be fast, to gap closers; almost guarantee position desync where no ability will go off and often you rubberband back to where the server list you.

I don’t think it’s necessarily the damage aoe effects but more so the healing/buff effects

relentless_turnip

I really appreciate the level of detail here.
My concern is the talk about optimising performance with these changes.
Which is fantastic and I really hope they help, but we never hear from you guys regarding what is actually broken.

So in harrowstorm a change was made or something went wrong that caused massive skill delays and desyncs.
On pc EU it is still very apparent I have posting clips here for you guys:
https://forums.elderscrollsonline.com/en/discussion/535282/performance-issues-submit-your-clips-here#latest

What I would really like to hear(and I fear I am not alone) is where we stand in fixing this?

Quelios

Always happy to see good news about game improvement...
As usual, I think we will still face some bugs and issues, but, at the end of the day, it's important to see your team involved and dedicated.
After so many years of good and bad things we are still here with ZOS and ZOS is still here with us, being together a nice community...

Destai

ZOS_JessicaFolsom wrote: »

Hi everyone,

Below is the first of our quarterly game performance improvement previews. This was written by our Engineering team and explains the main performance improvements planned for Update 27. If you haven’t yet, we encourage you to also check out the Update 27 Combat Preview here.

--

Hi friends!

We're here to bring you a preview of some of the performance improvements planned for Update 27. While it is impossible to capture everything that the team has done, we aim to highlight some of the areas we are focusing on and what we expect to gain as a result.

Persistent AoE Improvements: Area of effect abilities that last for long durations have the potential to send a large number of messages down to the client over time. This is exacerbated in zones like Cyrodiil where many players are routinely casting and being hit by these types of abilities. To mitigate this, we have implemented new AoE tech on the server that will keep the same functionality, but significantly reduce the number of hit result messages the client has to process. Internal testing has been positive, showing a reduction in latency and FPS drops on the client. We are taking a cautious approach to rolling out this new tech, choosing to focus solely on Dragonknight Standard (and morphs) for U27. We will evaluate the improvements on the PTS and live servers and continue to update additional player abilities in future updates.

Note: In addition to the persistent AoE Improvement work, the team will also be conducting a series of tests on the live servers in the next few weeks. These tests will center around AoE’s in Cyrodiil and will allow us to gather even more metrics of real-world scenarios. Expect more info about what exactly the tests will entail in the next couple weeks.

Item Set Abilities: The team has been hard at work auditing and streamlining each and every item set ability in the game, with a focus on looking for ways to improve performance. This has resulted in reduced calculations on the server and, similarly to the AoE improvements, a reduction in the number of messages sent down for the client to process.

Critical Memory Situations: We know how frustrating it is to teleport into a busy city and encounter a crash. While we have significantly reduced the number of out-of-memory related crashes in recent updates, it is still something that can happen to our players. To prevent these crashes, the console versions will now detect when the game is about to run out of memory and react by temporarily unloading vanity pets and personalities from players outside of your group. We were able to build upon the work completed in U26 that moved vanity pets to the client to help make this possible. Once memory is back within a safe threshold, these assets will begin to load back in as players move through the world.

Physics Implementation: Our Havok physics engine implementation, specifically dealing with Keep pieces in Cyrodiil, has had some issues over the years. We took some time to clean up the code surrounding this, reducing unnecessary complexity and making Keep pieces behave more like normal fixtures. Essentially, they will still look and behave as they do now. While working on this, we tracked down an elusive bug where some physics objects were not being cleaned up properly. This issue was not specific to any zone and would result in the client crashing when it occurred. The end result of the physics hardening should be fewer bugs and crashes.

Trial Performance: Our metrics have shown that Trial performance on the server has not been a consistent experience. We were able to track down the root cause of this–instance launchers running multiple Trials simultaneously would routinely perform worse overall. We've taken a pass at the algorithm which determined how Trial instances are distributed and now factor this into account. This will decrease the chances of our instance launchers being overloaded in high usage situations and impacting Trial performance as a result.

Database Improvements: We've made some backend changes to incorporate an in-memory database cache in our server processes, which will reduce the amount of contention on our production databases when under stress. Initially for U27, we are setting the foundation and focusing on a small piece of the Activity Finder tool which should result in smoother processing overall. Once we’ve evaluated this on the live servers, it will create opportunities in future updates to move over additional functionality, allowing us to perform more work simultaneously without performance degradation.

Intermittent Load Screens: It's a source of frustration for everyone when you're running around Tamriel and you're intermittently hit with short load screens. We've identified two areas where this behavior could occur. The first issue is a bug with streaming of scaled assets that are switching between LOD models and main models. This should go to the live servers with an upcoming U26 incremental. The second issue is regarding a bug that could cause the game to have much bigger physics streaming radius when a high view distance is selected, resulting in more loads than necessary. This fix will release with U27. We will be actively monitoring the live server metrics after U27 releases to determine if additional investigation and work is needed.

We know this won't solve every issue our players are experiencing, and there is still much work to be done. We will continue working diligently to provide you the best experience possible. Be on the lookout for some experimental Cyrodiil changes over the course of U27. Details for that will come in a future update. Thank you!

@ZOS_JessicaFolsom

Thanks for the update notes!

There's been a long standing issue with skill bars and slotted items not responding when stamina's been exhausted. So if I run through a zone and try to attack the enemies, nothing works. This bug has existed since U25.

When is this being addressed?

Izanagi.Xiiib16_ESO

Critical Memory Situations: We know how frustrating it is to teleport into a busy city and encounter a crash. While we have significantly reduced the number of out-of-memory related crashes in recent updates, it is still something that can happen to our players. To prevent these crashes, the console versions will now detect when the game is about to run out of memory and react by temporarily unloading vanity pets and personalities from players outside of your group. We were able to build upon the work completed in U26 that moved vanity pets to the client to help make this possible. Once memory is back within a safe threshold, these assets will begin to load back in as players move through the world.

I was hoping this point might reference the constant crash in cyrodiil when approaching busy keeps. In this situation turning mini-pets off isn't going to make any difference as they aren't on in cyro. Will personalities be disabled in cyro too due to this?
What about costumes, assuming they have the same effect?

@ZOS_JessicaFolsom

Also the biggest performance issue in cyro right now is the infinite debuff bug. This causes 90% of the desync and slow performance and happens all the time. I've had 3 infinite streaks on my char screen and it results in not being able to use any skills, constantly break freeing when bashing & break free delay, strafing when trying to run normally, synergy delay.

Can we not just make it so that debuffs like this 1) have a maximum duration and 2) are not applied outside of combat in cyro / any debuffs on the player drop when out of combat.

technohic

ZOS_JessicaFolsom wrote: »

Cinbri wrote: »

I hope any of those will fix new bug twhere using gap-closer ability during high latency, i.e. primetime cyro will guarantee result in your charater to be desynced.

Just checked in with the team about this and they are still investigating the root cause of this one.

I feel just giving you an awesome here is not enough to express how much this simple comment is appreciated. Just because it says this specific item is being looked at rather than just a generic " we're still working on performance issues "

Tinolyn

Intermittent Load Screens:

I can point to one absolutely specific spot (it happens EVERY. SINGLE. TIME) I go there in Reaper's March where I get a tiny loadscreen. Just southwest of the Dune wayshrine.

Message if you want a screenshot.

Every other one I've found is random, but not this one. And my PC should not have a problem with this at all.

WrathOfInnos

On the topic of Dragonknight Standard, would it be possible to hide the visuals from group members when ally effects are turned off? When you get 5-6 of these going in trials the banners can be a significant view obstruction and they seem to make targeting worse for abilities like Mystic Siphon. The rocks that appear on the ground around a Standard also hide AoE telegraphs (things like Flame Spit on Lokkestiiz HM).

Another thing that really needs to be hidden is an ally’s Maw of the Infernal. I feel bad for the trial tanks that have to see what’s going on through 8 of these friendly daedroths, and rendering that many additional pets can’t be good for performance.

x48rph

Kudos for giving us a detailed explanation on what was being done and why. It's very much appreciated.

gp1680

I like the detail, but seriously this only pertains to a portion of the issues in Cyrodiil. Disconnection when getting near a keep, 3-5 second delays with skills, and health de-syncs is still a major issue that needs to be addressed.

ForeverJenn

Thank you for finally addressing the load screens and inability to ground cast.

Hotdog_23

carlos424 wrote: »

technohic wrote: »

Note: In addition to the persistent AoE Improvement work, the team will also be conducting a series of tests on the live servers in the next few weeks. These tests will center around AoE’s in Cyrodiil and will allow us to gather even more metrics of real-world scenarios. Expect more info about what exactly the tests will entail in the next couple weeks.

I'm a bit frustrated it's taken this long to come up with this but glad it's finally being done. Many of us have noticed higher latency when certain types of groups using mass AOE are around so I think this is a good place to look. I wouldn't expect DK Standard though; as the first place to start. Maybe grave robber or something more spammable but at least its something.

The other thing that happens in this is speed from building to be fast, to gap closers; almost guarantee position desync where no ability will go off and often you rubberband back to where the server list you.

I’m glad I’m not the only one scratching my head about the DK standard. You hardly even see standards in Cyrodil, it’s all leaps. Maybe just a sneaky way to nerf standard? Maybe tick every 2 seconds instead of every second? Lol. I guess we’ll see.

I guess it's because you don't see it a lot and if the changes bugs out then they will have something to build and work on. This way it would not be to great of a problem if it flat out fails and causes more problem then if fixes. Because if it did you know the forums would be in a uproar quickly.

Be safe and have fun

Nomad1098

As always, hoping for the best, expecting nothing.

MachineGod

@ZOS_JessicaFolsom

Does the AoE work include siege weapons for cyrodiil too? I believe this could also be a contributing factor especially at large keep fights.

Nordo

Great, but until I see actions taken to actually improve the performance in every aspect of the game, I'll remain doubtful.

xaraan

Sounds like some good minor fixes, but not sure what to expect. Cyro performance is pretty bad when pop is high, so seems like something bigger is needed there. But anything helps at this point I guess.

As for trial performance, yes, more is needed there. Obviously even frames can get rough in fights no matter what, but the longer you are in the trial (esp some trials) the buggier they get. Things desync, etc.

We also have random bugs, like in KA, taunt just drops or doesn't work on mobs sometimes. Our team for example splits one pull where I will tank a couple mobs like 2-hander and sword and board mobs to side away from group while the group and other tank deals with everything else, but quite often I will taunt one of them, the skill and buff shows, but the mob still runs to other group. I've retaunted them and they still remain, this is not a problem with the other tank taunting by accident, we've been watching for it and making sure b/c it happens so often. Sometimes I'll fire inner fire off on first of the two mobs on countdown and it doesn't even apply even though it fires off. Definitely something weird going on here, but when you are progging something and having it fail b/c the game doesn't work right, it makes people a lot more frustrated.

Galarthor

ZOS_JessicaFolsom wrote: »

Cinbri wrote: »

I hope any of those will fix new bug twhere using gap-closer ability during high latency, i.e. primetime cyro will guarantee result in your charater to be desynced.

Just checked in with the team about this and they are still investigating the root cause of this one.

While you are at it, could you also ask if streaking will become reliably possible during primetime in cyro?

c363b

The devs mention 2 possible causes for the "Intermittent load screens." Could either of those account for the wide variety of circumstances where players have experienced the problem, as reported in the long "Load screen flashes" thread on the Bug Reports forum? In particular, some have occurred when the players' characters were stationary and not viewing distant scenes.

As my characters moved between Bruma and Capstone Cave, they typically had at least 5-6 flashes, including 3 that occurred in rapid succession within about 30 seconds.

WrathOfInnos

xaraan wrote: »

Sounds like some good minor fixes, but not sure what to expect. Cyro performance is pretty bad when pop is high, so seems like something bigger is needed there. But anything helps at this point I guess.

As for trial performance, yes, more is needed there. Obviously even frames can get rough in fights no matter what, but the longer you are in the trial (esp some trials) the buggier they get. Things desync, etc.

We also have random bugs, like in KA, taunt just drops or doesn't work on mobs sometimes. Our team for example splits one pull where I will tank a couple mobs like 2-hander and sword and board mobs to side away from group while the group and other tank deals with everything else, but quite often I will taunt one of them, the skill and buff shows, but the mob still runs to other group. I've retaunted them and they still remain, this is not a problem with the other tank taunting by accident, we've been watching for it and making sure b/c it happens so often. Sometimes I'll fire inner fire off on first of the two mobs on countdown and it doesn't even apply even though it fires off. Definitely something weird going on here, but when you are progging something and having it fail b/c the game doesn't work right, it makes people a lot more frustrated.

@xaraan Yep, we experienced this same issue many times. I don’t even think it’s limited to vKA, it can happen in vSS and maybe other trials. ESOlogs helped find out what was causing the problem, it’s not that the taunt doesn’t cast or apply, check out this series of events:



Unclear if this “remove everything” is intended or a bug (it has been reported) but it tends to happen a few milliseconds after a battle starts. The best thing you can do for now is make sure a light attack lands on something to trigger the “remove everything” and then start taunting.

Elsonso

c363b wrote: »

The devs mention 2 possible causes for the "Intermittent load screens." Could either of those account for the wide variety of circumstances where players have experienced the problem, as reported in the long "Load screen flashes" thread on the Bug Reports forum? In particular, some have occurred when the players' characters were stationary and not viewing distant scenes.

My thinking is that one of the two reasons can account for all of the "intermittent loading screens" that I have seen in Cyrodiil, Northern Elsweyr, Craglorn, etc. I am hoping that the one they are fixing in an upcoming patch fixes most of them. Otherwise, I may turn down my view distance to see if that makes a difference, if I start seeing those a lot. Of course, staying out of Cyrodiil usually fixes most them, even without a fix.