Update on Cyrodiil Performance & Upcoming AOE Tests

ZOS_RichLambert

As part of our ongoing performance optimizations, we have been spending a lot of time analyzing and addressing problems in Cyrodiil. I'm going to give a summary of what has happened over the years, what we have been doing to alleviate the problems, and our future plans.

When talking about ESO server performance, it is important to understand that all abilities in the game (with some exceptions) have a "soft" limit on the number of times they can be executed. An example of a "hard" limit on an ability is an ability that can be executed one time every two seconds, enforced by the server. ESO doesn't have many of those. Instead, we designed the game to have soft limits, which mean you can execute as many abilities as you want as long as you have the resources - Stamina or Magicka - to cast them. There is a global "cool down" timer on all abilities, which is set to 1000 milliseconds.

The design goal of soft limits in any game is to allow players to create builds that let them execute abilities more often depending on build choices made, and not to have hard ceilings on damage or healing per second. This gives a lot of control to the player, which makes for a really fun and interesting system, but it can lead to situations where players cast too many abilities too quickly (and continuously) if strict limits on resources are not enforced.

Over the years, player power has grown considerably. With the addition of the Champion System, various armor/weapons sets, and changes to abilities, we have reached the stage where players - with the right build - can cast near-infinite numbers of abilities. If you add in a properly managed group, with some focused on damage and some focused on healing and regen boosts, you have a perpetually running never-ending stream of abilities.

With that in mind, consider how Area of Effect (AOE) abilities work: when cast, they look at a specific target area - almost always the area directly around the caster - find targets, and perform the ability. Each of those steps requires server calculations.

At launch, Cyrodiil’s processes were able to keep up with the number of AOEs cast, because most players couldn’t cast that many of them: they ran out of Magicka or Stamina, so they had to use AOEs judiciously. Over time, as player knowledge grew and regen builds grew in power, more players could cast more and more AOEs before running out of resources.

During these years, we found and fixed many performance issues in Cyrodiil, and we continue to do so. We've fixed significant issues, uncovered more, and continue to find and fix with every Update.

However, one foundational issue remains. At some point, we crossed a threshold where most players in PvP were able to cast endless AOE abilities, without ever running out of resources. This is done through player knowledge, builds and group mechanics – resulting in a constant stream of AOEs with many players never using any other type of ability.

This is not what we intended, but part of the fun of Elder Scrolls games is designing a build that has unexpected and powerful results, and we allowed it. However, as this behavior grew more prevalent, we reached a point where casting so many continuous AOE abilities in such a small area started to overwhelm the server process for that area, leading to situations where the "lag meter" spikes and the server becomes unresponsive for a period of time.

Our initial response to this problem – starting with Update 22 – was to find (and fix) more than a few problems with AOE ability calculations and make them more performant - and we stayed (mostly) ahead of the problem. But as more players reached maximum Champion Level and more players started utilizing this particular method of "AOE spamming", we have reached the point where we cannot fix this issue around the edges: we need to address the core problem, which we will be starting with Update 27.

To do this, we will need to first do some analysis, and we can only do this on the live servers. As much as we try to avoid running tests on live servers, they are the only place where the combination of player behavior, specific builds, and mass battles happen. So, starting the week of August 24, we will be running a series of tests on the live PC servers – both NA & EU, only in Cyrodiil. Please note that we will not be running these tests on any Console servers. Each test should take about a week, but if needed, we will extend the testing time.

The first round of tests we are planning will focus on Area of Effect (AOE) abilities in Cyrodiil and will make it more difficult for AOE abilities to be the only abilities used, adjusting cooldown, cost and regen values of all AOEs (damage and healing).

Specific details on the tests we will be running in Cyrodiil are as follows:

• Test 1 – Shared global AOE cooldown - 3 second timer. This test adds a global 3 second shared cooldown to any AOE ability. This means that when you cast an AOE, you will not be able to cast another for 3 seconds. For example, as a Templar, if I cast Ritual of Retribution, I wouldn’t be able to cast Puncturing Sweep for 3 seconds.
• Test 2 – Individual AOE cooldown - 3 second timer. This test adds an individual AOE cooldown to each AOE ability. This means that when you cast an AOE, you will not be able to cast that same AOE ability for 3 seconds. For example – as a Templar, if I cast Puncturing Sweep, I wouldn’t be able to cast Puncturing Sweep again for 3 seconds.
• Test 3 – No cooldown, global ramping AOE cost. This test adds a global ramping AOE cost for each AOE cast. Similar to how streak or roll dodge works, where when you cast an AOE you receive a debuff for 5 seconds, each stack of the debuff increases the cost of any AOE cast.
• Test 4 – Individual AOE cooldown – 3 second timer, global ramping AOE cost. This test adds an individual AOE cooldown to each ability as in test 2, but also combines that with a global ramping cost from test 3.

During the times that any of these tests are active, we will be awarding double Alliance Points for anyone active in Cyrodiil.

After we complete the above tests, we may try other combinations of cooldown, cost, and regen values on AOE abilities. However, we need to run these tests first and then assess the data. We will then let everyone know what we found and how we will move forward. We will be very upfront, but please be aware that if these tests confirm our hypothesis, then chain-casting AOE abilities will no longer form the core of the ESO PvP experience in the way it has for the last few years. We would then go through each class and ensure that there are viable builds for each and make adjustments as necessary.

The code for these changes will be going into today’s PTS patch, and we will be running some basic tests throughout the day on Tuesday in Cyrodiil to ensure we’re able to make the above changes without requiring any maintenance or downtime. Once we launch Update 27 for PC on August 24, we will announce when one of the tests are beginning through an in-game announcement and will have a forum thread detailing which test is currently being performed. Remember, these tests will be limited to Cyrodiil, so AOE abilities in other PVP spaces (Imperial City and Battlegrounds) and will remain unchanged for now. We will evaluate how these tests go, and let everyone know next steps.

Thanks for being patient and understanding with these tests. They are extremely important for the team and will help us work towards improving Cyrodiil performance.

Dottzgaming

This unfortunately will probably get a "positive" performance result. Combat is going to slow down to a grinding hault, so performance will improve since people will be playing the game at a significantly slower speed.

ESO's hallmark feature is it's combat speed and fluidity, especially for PvP. Cooldowns shouldn't be introduced into a system that was fundamentally designed to not have them. Period.

NirnStorm

IMO this test is gonna work out great, will definitely fix performance issues.
On another note, if we disabled combat altogether in PvP, that will also fix performance issues.

This test is basically "lets see if making our dynamic combat system less dynamic, will fix performance".
Of course it will. But at what cost?


Edit: I created a new video discussing this post and my personal take on it.
https://www.youtube.com/watch?v=OD67tq4rDNQ
P.S. thanks for 100 agrees!

colossalvoids

Thanks for the insight of sorts but make sure to actually discuss the results with class representatives at least to minimise some possible negative impacts to areas where it might be not desirable. Like introduction of one of those fixes through the Battle Spirit and the other one (or none) for a Trial encounter etc. Surely it's just a test but it might lead to some pretty wild decisions that wouldn't benefit game's health at all.

Edit: this video should be on a first page and i really hope @ZOS_RichLambert would take some insights from that, that's very important before any decision making.

https://youtube.com/watch?v=nO-NLBjSO1U

Rezdayn

Please dont put this into pve....

FatherDelve

All I can say is, that i have way less Performance Problems with 40 random zerglings around me, then having 8 Ballgroup Members spamming every 1 second a *** of abilities at 5 meters.

hollywood

So , the AoE spamming blobs cause much of the lag issues.. WHO WOULD HAVE TOUGHT.

DCZergNoob

Rezdayn wrote: »

Please dont put this into pve....

PVE has it paradise compared to PVPers, dont even try and pull that card

lrs8855b14_ESO

Test 3 seems the obvious answer. Why would ZoS even suggest taking away spam-Jabs from Templar?

Addressing the symptom, and scapegoating the true cause - hardware insufficiencies and ZoS's inability to address their responsibility to support a functioning product.

Princess_Ciri

yikes

Ode2Order

Sounds like you found the root of the problem! Well done! I'm happy to see you communicate clearly about how to move forward from this point, and I hope it will bring us to a solution (:

H o w e v e r
The constant loading screen issue also exists in Craglorn, where it is arguably even worse (sometimes can't even catch a breather to teleport OUT of there). There are no issues with skill casting, as I doubt there are that many AOEs being spammed there, but obviously I don't have the numbers. The issue sometimes even persists in Craglorn trials, mostly in Aetherian Archive (from anecdotal experience). What I would like to ask is if you could look into that zone as well please?

Thank you for trying to make this a better game with good performance!

Rezdayn

DCZergNoob wrote: »

Rezdayn wrote: »

Please dont put this into pve....

PVE has it paradise compared to PVPers, dont even try and pull that card

This is not a fight between pvp and pve... stop.

This will hurt people with bad ping in pve. The pvp side can be discussed in the thread.

GandTheImpaler

Regardless of what happens here, I pray none of these become solutions. I'm not a fan of WoW.

ESO_Nightingale

This would *** destroy this game's popularity. What on earth?

DCZergNoob

Rezdayn wrote: »

DCZergNoob wrote: »

Rezdayn wrote: »

Please dont put this into pve....

PVE has it paradise compared to PVPers, dont even try and pull that card

This is not a fight between pvp and pve... stop.

This will just be bad in pve. People here can fight about the pvp side below.

Your right it isnt. But still dont pull it

CaffeinatedMayhem

Interesting, but maybe look at limiting casting in groups? How about 3 Eye of the Storm/Magicka Detonation per group? Maybe stop Hiti’s Hearth from stacking at all? Killing solo play isn’t going to fix the real issue: ball groups.

VixxVexx

Oh no

erio

Hey streak is an aoe right? give it double ramp please

Honestly id rather play WITH LAG than with these changes

Tattooo

There we go....

Meme of the month Love it, had a good laugh.. almost as good as the changes they wanted to implement with heavy/light attacks.

ZOS finally decided to remove the only good thing going for them(their dynamic and intuitive combat) .. and made their game turn based, to "fix" performance .. what a joke.

I guess their PUBLIC TEST SERVER can't hadle these kind of TESTS, so they have to put it on live.

And they dare to say that they fixed many performance issues in cyrodiil, while that may be true ... they might've fixed 2problems, but added another 10..

Removed animation canceling to fix dsyncs, which didn't happen .. just made performance worse .. but no instead of reverting these changes they keep the trainwreck going, making combat even slower..

Say bye bye to your endgame community, dead game.. let's move on.

Actual 2Heads working at ZOS.

After reviewing what I said ..
I actually hope that ZOS will get some good data from the testing, actually fixing some of the performance issues..

Also these changes will surely not go trough like this in the future, it's just a matter of gathering data on what specifically is causing the lag when using aoe abilities.

Jarrods32

Are these aoe changes going to effect pve gameplay. Aoe are a big part of pvp would be greatly hindering on trial situations or with the aoe work differently in pvp vs pve? Why not just split the server or way server reacts to aoe when in cyro

Eur0ed

NirnStorm wrote: »

IMO this test is gonna work out great, will definitely fix performance issues.
On another note, if we disabled combat altogether in PvP, that will also fix performance issues.

This test is basically "lets see if making our dynamic combat system less dynamic, will fix performance".
Of course it will. But at what cost?

I was going to write my own reply..but this is perfect.

Gutting the flow and feel of combat to fix performance will 100% work while also eliminating the one key thing that makes PVP in this game stand out.

Rezdayn

DCZergNoob wrote: »

Rezdayn wrote: »

DCZergNoob wrote: »

Rezdayn wrote: »

Please dont put this into pve....

PVE has it paradise compared to PVPers, dont even try and pull that card

This is not a fight between pvp and pve... stop.

This will just be bad in pve. People here can fight about the pvp side below.

Your right it isnt. But still dont pull it

Yawn. People can chew gum and walk at the same time... Dont need to hurt one side to help the other...
yawnnnnnnnnnnnnnnn

MasterPleb

this would be great to get data, but i pray none of these will actually go live permanently because it would absolutely kill the game for anyone who enjoys any type of smallscale pvp

Alienoutlaw

a 3s cool down for AOE will kill PvP for most players. 3s in PvP is an eternity, the main issue for me at least is groups exploiting the lag and building to specifically produce that lag, an increase of AOE cost will have no effect in ball groups as they run multiple buffs to ensure they never run out of resources, solo play will also take a massive hit and with a 3s cool down either global or player specific will mean less ppl will want to play that way. cyrodiil needs its own dedicated server and has done from the start, alot of the item set and skill balancing has been a direct result of the server performance in PVP and has had a knock on effect to PVE a separate server will allow a more refined balancing approach

Trandaner

The only good thing left in this game, the combat system without insanely long cooldowns, and they want to remove it. I just wonder how you think it's going to be possible to play something like templar.

redlink1979

Thanks for the update @ZOS_RichLambert

Dagoth_Rac

Do you intend to make these changes global, eventually? That is, will this apply to Trials?

It is one thing to shake up Cyrodiil where your enemies will be affected by the same limitations. A new meta and equilibrium will be reached. Maybe players won't like it, but you will still be able to PvP.

But Trials are dependent on certain DPS thresholds being met, and "cleave" damage can be critical to meeting those thresholds. If players have to single target every trash mob, every boss add, etc. That is going to get very tedious, very quickly, and really mess with the intended fight designs.

Heck, even "easy" content like Darkshade II will be massively impacted. I cannot imagine fighting Grobull or the waves of Dwemer automatons up the hill after Grobull, without tons of AoE. And then there is Vet Black Rose Prison. That is one of the hardest pieces of content in the game and it is all about throwing huge numbers of enemies at you and I do not see clearing that without throwing huge numbers at AoEs at those enemies.

code65536

So the elephant in the room is... if this test confirms your hypothesis, will you be limiting your adjustments to just PvP?

WombatNipples65

Sorry but do you play your own game? These changes are absurd, half of the classes won't be bale to work.

Templar - Jabs THEIR SPAMMABLE is AoE so you can't jab for 3 seconds or a ramping cost on your spammable. I don't need to explain how stupid this is.

Warden - Sub assault used once every 3 seconds - ramping cost for 5 seconds you might as well give it a 5 second time before it goes off.

Smallscale PvP dies if these go live. There is no skill in running around in big groups, I could run around in a 24 man group with no gear if these changes go through and not have a single issue.

Why do you hate Small Scalers?

Itzmichi

I mean interesting post and all, but how will this turn out to be reasonable?

maxjapank

What a nightmare for a Templar’s during these tests as our main spammable, Sweeps, is aoe.