Update 28 Game Performance Preview

ZOS_GinaBruno · October 2020

Hi everyone,

Below is our next quarterly game performance preview. This time, we are starting with a recap of Update 27 performance improvement effectiveness and next steps, followed by a preview of what’s coming for Update 28 in November. As with our last game performance preview, this includes details from our Engineering team.

Thank you, and if you haven’t yet had a chance, we also encourage you to check out the Update 28 Combat Preview here.

Update 27 Retro

Cyrodiil AoE Ability Testing: The AoE ability performance tests in Cyrodiil are wrapping up, with the final round of tests concluding on October 19. Out of the first three weeks of tests, the global cooldown test resulted in the greatest performance gains when compared to the others (roughly 25% increase in server performance). However, there is still work to be done as the performance gains, though substantial, are not enough to make the overall experience as smooth as we would like it to be. We will be running more combinations of tests right after this last round completes, and providing a more comprehensive summary of this round of tests. Stay tuned for more info soon!

Here are a couple general notes about the AoE ability testing:

Our ideal server FPS is less than 30ms. Anything over 50ms and players will see noticeable delays in abilities firing. The global cooldown test brought server FPS in high-intensity situations down almost 25%, but it still spikes above 50 in large battles.
Population remained consistent during peak hours for these tests. (i.e. – On PC EU, Ravenwatch was still pop-capped for all three Alliances)

Trial & Dungeon Performance on the Server: We had some issues with configuration settings when we first launched these initial changes that are now resolved. In the past few weeks, things have been stable and running smoothly. We identified a few more adjustments that we need to make, and those should be in for Update 29.

Database Improvements – Activity Finder: We are happy with how well these changes have gone thus far. Getting into dungeon groups is a much quicker and more reliable experience than it has been in the past. There are still a few edge-case bugs we’re tracking, but overall things have vastly improved.

Intermittent Load Screens: The fixes we implemented to address intermittent load screen issues have performed remarkably well. There has been a significant drop in intermittent load screens across the board, in all zones. For example – in Cyrodiil (which was one of the worst offenders) we have seen a 95+% decrease in intermittent load screens randomly popping up in the zone. (We went from approx. 50k a day down to under 3k a day).

Overall Client Stability (critical memory system & Havok physics engine hardening): We have seen an improvement in overall stability due to these initiatives. We have not fixed all the crashes and still have work to do, but our crash rate on consoles has decreased dramatically since U27 launch.

Update 28 Preview
For Update 28, we have the following initiatives coming:

Large-Scale Combat Performance Improvements:
- These include optimizations for how the client updates visual effects and should result in a higher framerate when participating in mass-scale combat with lots of abilities being cast (client FPS improvements).
Frustum Culling & Render Thread Optimizations:
- Improved framerate in situations where there are many individual objects visible, such as crowded areas with many players or within player housing that has a high volume of placed furnishings.
- This was done by reducing the workload on the rendering system by caching data and eliminating unnecessary commands.
- Note: These optimizations are console ONLY.

The Update 28 game performance work noted above is live on the PTS right now and will go into the game in November.

Thank you all for your patience waiting for this quarter’s game performance preview, and we hope this helped illustrate how the Update 27 work went and what we’re working on for Update 28.

HeroOfNone · October 2020

So improvements but not significant ones. Were there any particular abilities that were used more often than others that saw the greatest impact to server performance? I assume we'll see a restructure of some of those abilities with potential cooldowns in the future.

NirnStorm · October 2020

The activity finder still suffers from MANY issues, especially in battlegrounds queue.
Players can sit for over an hour before finding a battlegrounds match at times, this happened to me and many guild members, friends and stream viewers multiple times this patch. Can happen anywhere from top MMR to lowbie BGs.
-
The dungeon finder is better, but not enough either. You can sometimes queue up with a group of four and still have to sit and wait for literally 10 minutes to get an instance for your group, and many times you can't even cancel your active queue.

The AoE Test have taught us a lot as well.
It's clear to see that in none of these tests we've reached the performance we want to see in the game (and so do you), even though some of these tests were incredibly extreme in terms of punishment for AoE usage.
Will you consider testing Cyrodiil's performance without cross-healing and aoe effects between groups at all while continuing to reduce the max group size until we notice the difference in performance?

Great job overall, the loadscreens have improved MASSIVELY, and we haven't had too many crash and FPS drops related issues. I really hope to see the performance in PvP improves as well and hopefully the right server side changes will be done as well as the client side fixes

P.S. can we get the door in Blackheart Haven fixed so I don't have to wait 10 minutes to get out of combat and go through it?
potc5_dtlr4_2k_r709_marketing_stills_022317_86412_r_8a775940.0.jpeg

potc5_dtlr4_2k_r709_marketing_stills_022317_86412_r_8a775940.0.jpeg

VaranisArano · October 2020

Cyrodiil
Global AOE cooldowns did the best?
Not looking forward to the class reworks necessary if this is the solution.

I am very interested to read more about the results.

Activity Finder
Is one of the "edge cases" you are working on the sub-50 players ending up in vet dungeons?

CaffeinatedMayhem · October 2020

Any comments on the noticeable difference many people commented on with reduced group size in Cyro?

forthwinds · October 2020

On PC/NA there were never 2 factions poplocked at a time during the first week of testing. I highly doubt, even if there was one campaign that reached pop lock across, that there were huge faction fights like in the other weeks of testing. This is because Test 1 was during the Imperial City Event. The devs also have to realize that putting cooldowns in this game will ruin it and completely change the core combat.

A lot of us play this game because the combat is amazing compared to other MMOs. If you go through with putting cooldowns in this game, there's no reason why people would stay here vs. another MMO that has better performance. It's pretty sad that this change to core combat is even being discussed over 5 years after release.

MCBIZZLE300 · October 2020

Week one's gameplay felt awful. I'd rather have the performance we have right now to be fair. Its not that bad.

ZOS_GinaBruno · October 2020

FYI: Just corrected a typo in the post. Originally one of the bullet points stated the global cooldown test brought server FPS in high-intensity situations down almost 45% - this should've said 25%, and has been corrected in the original post.

mav1234 · October 2020

The group changes seemed to have the biggest improvement in performance while having the least impact on my play. I think it is clear that groups still cause lag even with only 6 people, as long a enough of them are in an area. And, it sucks being unable to heal people outside your group.

I wonder if directional heal spells could help solve a lot of the lag we see while still allowing people to be healed out of groups.

Regardless of the impact on performance, a global 3s CD is simply a deal breaker for many of us in terms of ESO without a complete re-design on healing and many class skills.

Kingslayer513 · October 2020

ZOS_GinaBruno wrote: »

FYI: Just corrected a typo in the post. Originally one of the bullet points stated the global cooldown test brought server FPS in high-intensity situations down almost 45% - this should've said 25%, and has been corrected in the original post.

Pretty sure that should say "ping" or "latency" instead of FPS. FPS makes no sense in that context and you don't measure FPS in milliseconds.

CaffeinatedMayhem · October 2020

Kingslayer513 wrote: »

ZOS_GinaBruno wrote: »

FYI: Just corrected a typo in the post. Originally one of the bullet points stated the global cooldown test brought server FPS in high-intensity situations down almost 45% - this should've said 25%, and has been corrected in the original post.

Pretty sure that should say "ping" or "latency" instead of FPS. FPS makes no sense in that context and you don't measure FPS in milliseconds.

I'm guessing ms in that context is time to calculate a frame.

Flamingfunk · October 2020

Could the widespread usage of proc-sets have a negative effect on the server?
In my logic, proc-sets are handled by the server and therefor put a substantial load on the server.
These tests have felt like testing medications against bacteria infections in a patient after you've introduced a viral infection.

How about a test where proc-sets are turned off, group size is max 12 and no cross healing between groups?

TPishek · October 2020

Most of the tests caused the worst performance I have ever seen in Cyrodiil. I don't know what your metrics have been showing in terms of the background calculations, but the actual on the ground player experience outside of one or two nights has never been more unpleasant.

Crash427 · October 2020

Spikes over 50? We can be at twice that (on the less than reliable built in meter) just sitting at the gates in Cyro. Do the devs even go in to Cyro at primetime? 30ms is a pipe dream for this game.

Gariele · October 2020

Does this mean more double AP Cyro with future test?

Kingslayer513 · October 2020

CaffeinatedMayhem wrote: »

Kingslayer513 wrote: »

ZOS_GinaBruno wrote: »

FYI: Just corrected a typo in the post. Originally one of the bullet points stated the global cooldown test brought server FPS in high-intensity situations down almost 45% - this should've said 25%, and has been corrected in the original post.

Pretty sure that should say "ping" or "latency" instead of FPS. FPS makes no sense in that context and you don't measure FPS in milliseconds.

I'm guessing ms in that context is time to calculate a frame.

Why is the server doing anything with your frames? Frame rendering should be entirely client side.

ExoY · October 2020

Kingslayer513 wrote: »

ZOS_GinaBruno wrote: »

FYI: Just corrected a typo in the post. Originally one of the bullet points stated the global cooldown test brought server FPS in high-intensity situations down almost 45% - this should've said 25%, and has been corrected in the original post.

Pretty sure that should say "ping" or "latency" instead of FPS. FPS makes no sense in that context and you don't measure FPS in milliseconds.

I am no computer engenieer but a quick google search for "fps server" concluded, that when talking about server fps has a different meaning than "frames per second". It is something like "how often the server looks for new information".
So I am asuming it was not a mistake calling it FPS and it is not to be confused with our client fps or latency.

Kingslayer513 · October 2020

Crash427 wrote: »

Spikes over 50? We can be at twice that (on the less than reliable built in meter) just sitting at the gates in Cyro. Do the devs even go in to Cyro at primetime? 30ms is a pipe dream for this game.

I'm guessing that server latency is probably only looking at the time for the server to process actions after receiving that data. So the client side latency that you see would be server latency + the time for your data to travel from client to server. Your client latency is variable depending on where you live so it makes sense that they aren't including it.

ZOS_GinaBruno · October 2020

Kingslayer513 wrote: »

ZOS_GinaBruno wrote: »

FYI: Just corrected a typo in the post. Originally one of the bullet points stated the global cooldown test brought server FPS in high-intensity situations down almost 45% - this should've said 25%, and has been corrected in the original post.

Pretty sure that should say "ping" or "latency" instead of FPS. FPS makes no sense in that context and you don't measure FPS in milliseconds.

In this case, we're using FPS to refer to server frames.

Crash427 · October 2020

Kingslayer513 wrote: »

Crash427 wrote: »

Spikes over 50? We can be at twice that (on the less than reliable built in meter) just sitting at the gates in Cyro. Do the devs even go in to Cyro at primetime? 30ms is a pipe dream for this game.

I'm guessing that server latency is probably only looking at the time for the server to process actions after receiving that data. So the client side latency that you see would be server latency + the time for your data to travel from client to server. Your client latency is variable depending on where you live so it makes sense that they aren't including it.

Makes sense. Thanks for clearing that up.

marius_buys · October 2020

Yes lets just all become lethal arrow and dizzy spamming single target nightblades. OMW

MachineGod · October 2020

@ZOS_GinaBruno

When can we expect to see the results of last weeks and this weeks test? They seem to have had the most beneficial performance increases so far.

Orpheaus · October 2020

This was a predictable but unfortunate result. I'm personally not surprised that global AOE CD had the most effect, almost by necessity it literally means less skills firing. What's so frustrating is that this specific test was the one that felt the worst to me by far. The game didn't feel like ESO during that week and I hope you explore other solutions because I'm not sure I'm interested in playing in Cyrodiil with the condition of AOE global CDs.

Additionally this question is directed at someone who has knowledge of this industry that I don't, is there a specific reason that ZOS can't upgrade their servers to create better performance? Or is this a situation where the hardware literally won't help and that's a pointless request?

marius_buys · October 2020

forthwinds wrote: »

On PC/NA there were never 2 factions poplocked at a time during the first week of testing. I highly doubt, even if there was one campaign that reached pop lock across, that there were huge faction fights like in the other weeks of testing. This is because Test 1 was during the Imperial City Event. The devs also have to realize that putting cooldowns in this game will ruin it and completely change the core combat.

A lot of us play this game because the combat is amazing compared to other MMOs. If you go through with putting cooldowns in this game, there's no reason why people would stay here vs. another MMO that has better performance. It's pretty sad that this change to core combat is even being discussed over 5 years after release.

My guild runs every single night always at the same time on PC EU Blackreach and not ONCE in 6 weeks were we pop locked, so I will second that comment.

Kingslayer513 · October 2020

ExoY wrote: »

Kingslayer513 wrote: »

ZOS_GinaBruno wrote: »

FYI: Just corrected a typo in the post. Originally one of the bullet points stated the global cooldown test brought server FPS in high-intensity situations down almost 45% - this should've said 25%, and has been corrected in the original post.

Pretty sure that should say "ping" or "latency" instead of FPS. FPS makes no sense in that context and you don't measure FPS in milliseconds.

I am no computer engenieer but a quick google search for "fps server" concluded, that when talking about server fps has a different meaning than "frames per second". It is something like "how often the server looks for new information".
So I am asuming it was not a mistake calling it FPS and it is not to be confused with our client fps or latency.

You might be right. I am not a server engineer, but "server FPS" strikes me as odd terminology when it could be called something more intuitive like server latency.

Kingslayer513 · October 2020

ZOS_GinaBruno wrote: »

Kingslayer513 wrote: »

ZOS_GinaBruno wrote: »

FYI: Just corrected a typo in the post. Originally one of the bullet points stated the global cooldown test brought server FPS in high-intensity situations down almost 45% - this should've said 25%, and has been corrected in the original post.

Pretty sure that should say "ping" or "latency" instead of FPS. FPS makes no sense in that context and you don't measure FPS in milliseconds.

In this case, we're using FPS to refer to server frames.

Thanks for clarifying

AgentZenish · October 2020

25%? I’ll take the lag over any kind of permanent global aoe cool down

Ragnaroek93 · October 2020

The last two weeks were kinda fine for me for most of the time but prime time was still unplayable. I'd probably play Cyro a lot more if the performance would actually be good.

xaraan · October 2020

forthwinds wrote: »

On PC/NA there were never 2 factions poplocked at a time during the first week of testing. I highly doubt, even if there was one campaign that reached pop lock across, that there were huge faction fights like in the other weeks of testing. This is because Test 1 was during the Imperial City Event. The devs also have to realize that putting cooldowns in this game will ruin it and completely change the core combat.

A lot of us play this game because the combat is amazing compared to other MMOs. If you go through with putting cooldowns in this game, there's no reason why people would stay here vs. another MMO that has better performance. It's pretty sad that this change to core combat is even being discussed over 5 years after release.

Couldn't agree with you more. Global cooldowns will be a no-go for me to ever play any real pvp again. Hopefully they can come up with a better solution than changing what makes the combat interesting. Otherwise they might as well start giving us 20 skills on each bar to use like other MMOs.

Kingslayer513 · October 2020

TPishek wrote: »

Most of the tests caused the worst performance I have ever seen in Cyrodiil. I don't know what your metrics have been showing in terms of the background calculations, but the actual on the ground player experience outside of one or two nights has never been more unpleasant.

I think it was pretty obvious to myself and many others that the week 5 tests (limited group size and heals only in group) felt the best both in terms of performance and gameplay health. However, whenever 1 or more ball groups logged on and began layering 10x radiating regens on every player, there was an immediate and profound negative effect on performance. I think that many of the people saying there was no difference with these tests just happened to experience them while the ball groups were running and performance was at its lowest.

All in all, I think that the week 5 tests could be a step in the right direction without bludgeoning the core gameplay experience that keeps people playing eso. However, much more work needs to be done specifically targeting the elephant in the room that is ball groups.

Niaver · October 2020

Well, even if these harsh restrictions can't fix Cyro...