Maintenance for the week of May 29:
• PC/Mac: No maintenance – May 29

Update on Cyrodiil Performance & Upcoming AOE Tests

  • The Uninvited
    The Uninvited
    ✭✭✭✭✭
    As part of our ongoing performance optimizations, we have been spending a lot of time analyzing and addressing problems in Cyrodiil. I'm going to give a summary of what has happened over the years, what we have been doing to alleviate the problems, and our future plans.

    When talking about ESO server performance, it is important to understand that all abilities in the game (with some exceptions) have a "soft" limit on the number of times they can be executed. An example of a "hard" limit on an ability is an ability that can be executed one time every two seconds, enforced by the server. ESO doesn't have many of those. Instead, we designed the game to have soft limits, which mean you can execute as many abilities as you want as long as you have the resources - Stamina or Magicka - to cast them. There is a global "cool down" timer on all abilities, which is set to 1000 milliseconds.

    The design goal of soft limits in any game is to allow players to create builds that let them execute abilities more often depending on build choices made, and not to have hard ceilings on damage or healing per second. This gives a lot of control to the player, which makes for a really fun and interesting system, but it can lead to situations where players cast too many abilities too quickly (and continuously) if strict limits on resources are not enforced.

    Over the years, player power has grown considerably. With the addition of the Champion System, various armor/weapons sets, and changes to abilities, we have reached the stage where players - with the right build - can cast near-infinite numbers of abilities. If you add in a properly managed group, with some focused on damage and some focused on healing and regen boosts, you have a perpetually running never-ending stream of abilities.

    With that in mind, consider how Area of Effect (AOE) abilities work: when cast, they look at a specific target area - almost always the area directly around the caster - find targets, and perform the ability. Each of those steps requires server calculations.

    At launch, Cyrodiil’s processes were able to keep up with the number of AOEs cast, because most players couldn’t cast that many of them: they ran out of Magicka or Stamina, so they had to use AOEs judiciously. Over time, as player knowledge grew and regen builds grew in power, more players could cast more and more AOEs before running out of resources.

    During these years, we found and fixed many performance issues in Cyrodiil, and we continue to do so. We've fixed significant issues, uncovered more, and continue to find and fix with every Update.

    However, one foundational issue remains. At some point, we crossed a threshold where most players in PvP were able to cast endless AOE abilities, without ever running out of resources. This is done through player knowledge, builds and group mechanics – resulting in a constant stream of AOEs with many players never using any other type of ability.

    This is not what we intended, but part of the fun of Elder Scrolls games is designing a build that has unexpected and powerful results, and we allowed it. However, as this behavior grew more prevalent, we reached a point where casting so many continuous AOE abilities in such a small area started to overwhelm the server process for that area, leading to situations where the "lag meter" spikes and the server becomes unresponsive for a period of time.

    Our initial response to this problem – starting with Update 22 – was to find (and fix) more than a few problems with AOE ability calculations and make them more performant - and we stayed (mostly) ahead of the problem. But as more players reached maximum Champion Level and more players started utilizing this particular method of "AOE spamming", we have reached the point where we cannot fix this issue around the edges: we need to address the core problem, which we will be starting with Update 27.

    To do this, we will need to first do some analysis, and we can only do this on the live servers. As much as we try to avoid running tests on live servers, they are the only place where the combination of player behavior, specific builds, and mass battles happen. So, starting the week of August 24, we will be running a series of tests on the live PC servers – both NA & EU, only in Cyrodiil. Please note that we will not be running these tests on any Console servers. Each test should take about a week, but if needed, we will extend the testing time.

    The first round of tests we are planning will focus on Area of Effect (AOE) abilities in Cyrodiil and will make it more difficult for AOE abilities to be the only abilities used, adjusting cooldown, cost and regen values of all AOEs (damage and healing).

    Specific details on the tests we will be running in Cyrodiil are as follows:
    • Test 1 – Shared global AOE cooldown - 3 second timer. This test adds a global 3 second shared cooldown to any AOE ability. This means that when you cast an AOE, you will not be able to cast another for 3 seconds. For example, as a Templar, if I cast Ritual of Retribution, I wouldn’t be able to cast Puncturing Sweep for 3 seconds.
    • Test 2 – Individual AOE cooldown - 3 second timer. This test adds an individual AOE cooldown to each AOE ability. This means that when you cast an AOE, you will not be able to cast that same AOE ability for 3 seconds. For example – as a Templar, if I cast Puncturing Sweep, I wouldn’t be able to cast Puncturing Sweep again for 3 seconds.
    • Test 3 – No cooldown, global ramping AOE cost. This test adds a global ramping AOE cost for each AOE cast. Similar to how streak or roll dodge works, where when you cast an AOE you receive a debuff for 5 seconds, each stack of the debuff increases the cost of any AOE cast.
    • Test 4 – Individual AOE cooldown – 3 second timer, global ramping AOE cost. This test adds an individual AOE cooldown to each ability as in test 2, but also combines that with a global ramping cost from test 3.
    During the times that any of these tests are active, we will be awarding double Alliance Points for anyone active in Cyrodiil.

    After we complete the above tests, we may try other combinations of cooldown, cost, and regen values on AOE abilities. However, we need to run these tests first and then assess the data. We will then let everyone know what we found and how we will move forward. We will be very upfront, but please be aware that if these tests confirm our hypothesis, then chain-casting AOE abilities will no longer form the core of the ESO PvP experience in the way it has for the last few years. We would then go through each class and ensure that there are viable builds for each and make adjustments as necessary.

    The code for these changes will be going into today’s PTS patch, and we will be running some basic tests throughout the day on Tuesday in Cyrodiil to ensure we’re able to make the above changes without requiring any maintenance or downtime. Once we launch Update 27 for PC on August 24, we will announce when one of the tests are beginning through an in-game announcement and will have a forum thread detailing which test is currently being performed. Remember, these tests will be limited to Cyrodiil, so AOE abilities in other PVP spaces (Imperial City and Battlegrounds) and will remain unchanged for now. We will evaluate how these tests go, and let everyone know next steps.

    Thanks for being patient and understanding with these tests. They are extremely important for the team and will help us work towards improving Cyrodiil performance.

    It seems to me you know exactly where the problem is, namely the CP system. So why not address that?
    Pandora's Promise (rip) | LND | Pactriotic | IKnowWhatUDidLastWinter's | The Uninvited |

    Ride the paranoia | All life is pain | Only the grave is real
  • ks888
    ks888
    ✭✭✭✭
    Moral of the story: ZOS thought Cyro would just be a bunch of individuals pew pewing in the large battles they advertised in the concept artwork but didn't have the foresight to see that the organized groups or blobs of players their system allowed for would be utilized. Cyro performance went ignored or they scrambled for solutions. 6 years later, Cyro performance suffers for everyone - competitive organized groups, unorganized balls of pugs and solo players. GG ZOS, as always, GG.
    DC NA - Norri - Khole
    DC NA - Ginger Gaspode (sorc stuff)
    AD NA - Mahdis (sometimes sorc stuff)
    AD NA - Dee Tix (heals for hire)
    EP NA - Norr'i (heals for hire) - Vanguard/Get the Brick/subs in places/solo potato
    EP NA - Nor'ree (does warden things)
    AD NA - Here for the Tick (fail bomber)
    DC NA - Boot Lizard (just here to meatbag plebs)
    EP NA - Sorc in a Bottle (negates stuff)
    EP NA - someothervariation of Norri (other fail or templar bomber)
    AD NA- Windowless Van (reasons)

    ** I have too many toons **
    RIP every alt I deleted - where am I? what year is it?
  • raegun
    raegun
    ✭✭
    If this applys to all aoe abilities does that include every two hander ability? sure sounds like it if sweeps would be considered an aoe. that's a complete weapon line that would be unplayable not to mention templar as a class for the test 1. and are heals going to be included in this cause there goes most of the heals in the game.
  • Firstmep
    Firstmep
    ✭✭✭✭✭
    ✭✭✭
    Sanguinor2 wrote: »
    Firstmep wrote: »
    I guess ill just play a ranged plar during the test spamming dark flare from keep walls, looks like all other options for the class are removed :D

    You can use javelin as spammable :), procs burning light and ignores resistances :)

    No it doesnt proc burning light on pts anymore, unless you ONLY spam javelin at someone 4 times in a row.

    Also its a cc ability, and the cost is tuned for that.

    I actually played ranged magplar right now, but not sure how it will work after burning light changes, will probably have to move to procsets, and it looks like zenimax wants to remove melee magplar from the picture.
  • valeriiya
    valeriiya
    ✭✭✭✭✭
    At least they're communicating with us and trying something. That something might not work but it's an improvement to communication which had the same lag as Cyrodiil.
    My skills don't work half the time in PvP anyway, hopefully this will help or we're all just screwed, who knows.
  • Kronuxx
    Kronuxx
    ✭✭✭✭
    Firstmep wrote: »
    Remove smart healing, especially from aoe heals and thing like radiating regen.
    Every time you cast it the server has to make a list of players in range, then choose the lowest hp target(s) and then run thru all the usual calculations like cp, healing bonuses etc.
    Multiply this by 20,30 times in a ballgroup every second, and you start to see the problem.
    It's really not hard to figure this out pls don't destroy combat.
    @ZOS_RichLambert

    This is actually a very valid point. You can stay see this evident because the moment the Volendrug hammer appears, even if populations aren't high, when players amass together in one spot spamming AOE heals, you can tell the server performance dips. The moment the hammer disappears, the server performance peaks back up.
    Remove smart healing and instead replace with targeted healing. This will also allow for more skilled group play. With this vein of thought, why not change how AOE damage skill/healing calculations work rather than add cast times. When three or more of the same AOE damage or healing skill is used in a group, the lower the damage or healing that stacked AOE does. It could have ramping values the larger the group. So, for example, the larger the group you are in, the lower your stacked AOE damage skill, like wise the less healing your stacked AOE skill does as well. This will allow, so that at some point you reach maximum efficiency at some particular group size and beyond that group size would mean your AOE skill will not do much for the group. This will promote smaller group play in Cyrodiil, along with less stacking of players in one spot. This I feel, would also help with performance, as less players overall casting stacked AOE's. @ZOS_RichLambert
    Edited by Kronuxx on July 27, 2020 4:51PM
  • PrimusNephilim
    PrimusNephilim
    ✭✭✭✭✭
    I_am_Groot wrote: »
    Whats next, turn based combat?

    not sure if you experienced the original Fallout game but that's the new combat model for next year
    Perfect, i hope they'll add random encounters and dots around global map during movement from zone to zone, i want those alien blaster

    But wait it gets better, by visiting the crown store, you can purchase extra moves at the low cost of 5000 crowns each, if you buy more than 10 extra moves, we'll send you a free non-combat pet.
  • Sugaroverdose
    Sugaroverdose
    ✭✭✭✭✭
    I_am_Groot wrote: »
    I_am_Groot wrote: »
    Whats next, turn based combat?

    not sure if you experienced the original Fallout game but that's the new combat model for next year
    Perfect, i hope they'll add random encounters and dots around global map during movement from zone to zone, i want those alien blaster

    But wait it gets better, by visiting the crown store, you can purchase extra moves at the low cost of 5000 crowns each, if you buy more than 10 extra moves, we'll send you a free non-combat pet.
    Can't wait
    giphy.gif


  • TX12001rwb17_ESO
    TX12001rwb17_ESO
    ✭✭✭✭✭
    ✭✭✭
    You know if they just removed animation canceling it would probably achieve the desired performance boost without crippling classes who rely on certain abilities, this has no effect on me though as my mageblade is all about single target abilities.
    Edited by TX12001rwb17_ESO on July 27, 2020 4:52PM
  • mattiellosa
    mattiellosa
    ✭✭✭
    If I wanted to play a game with cooldowns I would go and play world of warcraft. I play ESO for its ACTION COMBAT. There is no action in having to wait for cast your skills.
  • Tommy_The_Gun
    Tommy_The_Gun
    ✭✭✭✭✭
    ✭✭✭✭✭
    As part of our ongoing performance optimizations, we have been spending a lot of time analyzing and addressing problems in Cyrodiil. I'm going to give a summary of what has happened over the years, what we have been doing to alleviate the problems, and our future plans.

    When talking about ESO server performance, it is important to understand that all abilities in the game (with some exceptions) have a "soft" limit on the number of times they can be executed. An example of a "hard" limit on an ability is an ability that can be executed one time every two seconds, enforced by the server. ESO doesn't have many of those. Instead, we designed the game to have soft limits, which mean you can execute as many abilities as you want as long as you have the resources - Stamina or Magicka - to cast them. There is a global "cool down" timer on all abilities, which is set to 1000 milliseconds.

    The design goal of soft limits in any game is to allow players to create builds that let them execute abilities more often depending on build choices made, and not to have hard ceilings on damage or healing per second. This gives a lot of control to the player, which makes for a really fun and interesting system, but it can lead to situations where players cast too many abilities too quickly (and continuously) if strict limits on resources are not enforced.

    Over the years, player power has grown considerably. With the addition of the Champion System, various armor/weapons sets, and changes to abilities, we have reached the stage where players - with the right build - can cast near-infinite numbers of abilities. If you add in a properly managed group, with some focused on damage and some focused on healing and regen boosts, you have a perpetually running never-ending stream of abilities.

    With that in mind, consider how Area of Effect (AOE) abilities work: when cast, they look at a specific target area - almost always the area directly around the caster - find targets, and perform the ability. Each of those steps requires server calculations.

    At launch, Cyrodiil’s processes were able to keep up with the number of AOEs cast, because most players couldn’t cast that many of them: they ran out of Magicka or Stamina, so they had to use AOEs judiciously. Over time, as player knowledge grew and regen builds grew in power, more players could cast more and more AOEs before running out of resources.

    During these years, we found and fixed many performance issues in Cyrodiil, and we continue to do so. We've fixed significant issues, uncovered more, and continue to find and fix with every Update.

    However, one foundational issue remains. At some point, we crossed a threshold where most players in PvP were able to cast endless AOE abilities, without ever running out of resources. This is done through player knowledge, builds and group mechanics – resulting in a constant stream of AOEs with many players never using any other type of ability.

    This is not what we intended, but part of the fun of Elder Scrolls games is designing a build that has unexpected and powerful results, and we allowed it. However, as this behavior grew more prevalent, we reached a point where casting so many continuous AOE abilities in such a small area started to overwhelm the server process for that area, leading to situations where the "lag meter" spikes and the server becomes unresponsive for a period of time.

    Our initial response to this problem – starting with Update 22 – was to find (and fix) more than a few problems with AOE ability calculations and make them more performant - and we stayed (mostly) ahead of the problem. But as more players reached maximum Champion Level and more players started utilizing this particular method of "AOE spamming", we have reached the point where we cannot fix this issue around the edges: we need to address the core problem, which we will be starting with Update 27.

    To do this, we will need to first do some analysis, and we can only do this on the live servers. As much as we try to avoid running tests on live servers, they are the only place where the combination of player behavior, specific builds, and mass battles happen. So, starting the week of August 24, we will be running a series of tests on the live PC servers – both NA & EU, only in Cyrodiil. Please note that we will not be running these tests on any Console servers. Each test should take about a week, but if needed, we will extend the testing time.

    The first round of tests we are planning will focus on Area of Effect (AOE) abilities in Cyrodiil and will make it more difficult for AOE abilities to be the only abilities used, adjusting cooldown, cost and regen values of all AOEs (damage and healing).

    Specific details on the tests we will be running in Cyrodiil are as follows:
    • Test 1 – Shared global AOE cooldown - 3 second timer. This test adds a global 3 second shared cooldown to any AOE ability. This means that when you cast an AOE, you will not be able to cast another for 3 seconds. For example, as a Templar, if I cast Ritual of Retribution, I wouldn’t be able to cast Puncturing Sweep for 3 seconds.
    • Test 2 – Individual AOE cooldown - 3 second timer. This test adds an individual AOE cooldown to each AOE ability. This means that when you cast an AOE, you will not be able to cast that same AOE ability for 3 seconds. For example – as a Templar, if I cast Puncturing Sweep, I wouldn’t be able to cast Puncturing Sweep again for 3 seconds.
    • Test 3 – No cooldown, global ramping AOE cost. This test adds a global ramping AOE cost for each AOE cast. Similar to how streak or roll dodge works, where when you cast an AOE you receive a debuff for 5 seconds, each stack of the debuff increases the cost of any AOE cast.
    • Test 4 – Individual AOE cooldown – 3 second timer, global ramping AOE cost. This test adds an individual AOE cooldown to each ability as in test 2, but also combines that with a global ramping cost from test 3.
    During the times that any of these tests are active, we will be awarding double Alliance Points for anyone active in Cyrodiil.

    After we complete the above tests, we may try other combinations of cooldown, cost, and regen values on AOE abilities. However, we need to run these tests first and then assess the data. We will then let everyone know what we found and how we will move forward. We will be very upfront, but please be aware that if these tests confirm our hypothesis, then chain-casting AOE abilities will no longer form the core of the ESO PvP experience in the way it has for the last few years. We would then go through each class and ensure that there are viable builds for each and make adjustments as necessary.

    The code for these changes will be going into today’s PTS patch, and we will be running some basic tests throughout the day on Tuesday in Cyrodiil to ensure we’re able to make the above changes without requiring any maintenance or downtime. Once we launch Update 27 for PC on August 24, we will announce when one of the tests are beginning through an in-game announcement and will have a forum thread detailing which test is currently being performed. Remember, these tests will be limited to Cyrodiil, so AOE abilities in other PVP spaces (Imperial City and Battlegrounds) and will remain unchanged for now. We will evaluate how these tests go, and let everyone know next steps.

    Thanks for being patient and understanding with these tests. They are extremely important for the team and will help us work towards improving Cyrodiil performance.

    It seems to me you know exactly where the problem is, namely the CP system. So why not address that?
    It is not exactly true. CP system modifies stuff so in theory server has to do more calculations, but no-CP lags as well... In some cases even more than CP.

    It all boils down to low APM - high APM players. Trials for example lag significantly more and have way more de-syncs if you have good group. "High APM" players that animation cancel well, can cast way more abilities per minute vs "low APM" players. As a result they make more DPS - but they also cause more stress on the server.

    Same in Cyro. If a ball group shows up in No-CP it will have same lag & de-syncs as CP. CP seems to have "almost" no impact on performance (and if it does it is marginal).
    Edited by Tommy_The_Gun on July 27, 2020 4:59PM
  • DUTCH_REAPER
    DUTCH_REAPER
    ✭✭✭✭✭
    NB just got a buff...maybe now cloak will work properly and allow players to get away from people AOE spamming?
  • ealdwin
    ealdwin
    ✭✭✭✭✭
    West93 wrote: »
    Templar especially stamina will become completely unplayable and useless class in pvp.

    If such changes go live, than remake jabs into single target ability and rework skill completely, if not well I guess it will be time to take a long break from ESO till they make templar viable.

    Seriously. If any of those changes make it through testing, Templar will need a serious look over. Any part of the "house" play-style remaining would be completely demolished.
  • technohic
    technohic
    ✭✭✭✭✭
    ✭✭✭✭✭
    NB just got a buff...maybe now cloak will work properly and allow players to get away from people AOE spamming?

    lol Can hardly do that now in lag if the NB is mobile. To be fair though; cant even hit an enemy that is not hidden if they are mobilie
  • Feric51
    Feric51
    ✭✭✭✭✭
    Surely the answer to jabs is to just include ground AOEs/durational AOEs in any cooldown.

    Why not make ground AOEs cool down on duration? ie I can't recast blockade until it's run out, or a couple of seconds before?

    If it's the number of calculations I don't see why not being able to recast a ground AOE which lasts 10 seconds for 3 seconds will make any difference.

    If you cast a ground AoE with a 12-second duration and the target(s) die or move out of the area in two seconds, you don't want to wait 9-12 seconds to be able to recast it. Even basic PvE isn't a static rotation.

    Edited by Feric51 on July 27, 2020 5:05PM
    Feric51
    Xbox NA

    Darkness Falls: The Crusade survivor (you young kids will never know the struggle of text-based games)


  • Major_Lag
    Major_Lag
    ✭✭✭✭✭
    So... ESO is officially on track to become the next WOW then, with separate per-ability cooldowns instead of a common "global cooldown"?

    It's odd that the developers didn't think about axing LA weaving first, as that change alone would already significantly reduce the amount of calculations the server has to process. /s
  • PureEnvelope35
    PureEnvelope35
    ✭✭✭✭
    Also just out of curiosity, why not address AoE cross healing first given that that is the most likely to have multiple performance related issues given that AoE damage is far less potent and sustainable compared AoE healing, at least that's just my assumption after being in Cyrodiil for far too bloody long...
    Fashion is the true endgame.
    PC EU
    DC Bonneville the Pure Magicka Dragonknight (PvE)
    DC Lyudmila the Omen Stamina Necromancer (PvE)
    DC Breña the Pure Stamina Nightblade (PvP)
    DC Cirise the Pure Magicka Warden (PvP)
    DC Malina the Pure Stamina Warden (PvP)
    DC Melinda the Omen Stamina Necromancer (PvP)
    DC Adeliane the Pure Magicka Templar (PvP)
    AD Valencia the Pure Magicka Dragonknight (PvE)
    AD Do'Nhadir Stamina Nightblade (PvP)
    AD Xiphias Sword-Like-Comet Stamgicka Warden (PvP)
    AD Eleanora the Pure Stamina Sorcerer (PvE)
    EP Athena the Pure Stamina Dragonknight (PvP)
    EP Elizabeth the Pure Stamina Sorcerer (PvP)
  • mattiellosa
    mattiellosa
    ✭✭✭
    The server lag is a core issue that needs to be fixed server side. ESO will become a top tier game once they start to improve the servers rather than removing game mechanics to band aid the problem. Just give it 2 more patches and there will be another problem where they will remove yet another core game mechanic. This behavior needs to stop. If the team carries on down this path the game will eventually just become a walking simulator. (and soon after that they will remove walking to improve server performance.) To anyone from ZOS reading this, please stop now before its too late and all the players are gone. (Saying that as if all the vet players haven't already left xd)
This discussion has been closed.