Update on Cyrodiil Performance & Upcoming AOE Tests

Altyrann

If we assume the big issue is AoE DoTs rather than instant / channeled skills, can't you just reduce the frequency of damage ticks and buff damage of each tick when in Cyro?

FangOfTheTwoMoons

With all the people who quit of course your performance will go up lol.

The game prides itself on its combat and you wanna neuter it.

Isnt there a better way to improve performance? Is there a type of upgrade that could be done server side? Like upgrading hardware or something? This is a legitimate question because I dont know squat about things like this.

likecats

Ishtharo wrote: »

CaffeinatedMayhem wrote: »

Interesting, but maybe look at limiting casting in groups? How about 3 Eye of the Storm/Magicka Detonation per group? Maybe stop Hiti’s Hearth from stacking at all? Killing solo play isn’t going to fix the real issue: ball groups.

The issue isn't ball groups so much as ZOS not able to fix their own spaghetti code. "Our server's aren't working, must be the players."

That's bad game design AND bad programming all wrapped into one.

Even at the most optimum level its not a code issue, its a scaling issue inherent with AOE.

In single target abilities you only need to consider the first target, so the run time cost is a constant time.
In AOE you need to consider all the targets in the AOE and apply the operations on all of them which means the run time cost is linear time.

There is simply no logical way where an AOE can be as performant as a single target ability. Even if their code is designed by the best developers in this world, AOEs will still be less performant than single target abilities.

ZOS's mistake was to design the game around AOE spam and smart healing spam to start with. It worked out for them initially because fewer number of players would play, and the average player did not spam AOEs as much.

They likely cannot out scale this problem without bankrupting themselves. Even if they can then it still might be a smarter business decision for them to revamp AOE's as they would be able to maintain the same profit while having 1/5 (or whatever the average number of targets an AOE hits) of the player base.

Aegonnn

caeliusstarbreaker wrote: »

ZOS_RichLambert wrote: »

As part of our ongoing performance optimizations, we have been spending a lot of time analyzing and addressing problems in Cyrodiil. I'm going to give a summary of what has happened over the years, what we have been doing to alleviate the problems, and our future plans.

When talking about ESO server performance, it is important to understand that all abilities in the game (with some exceptions) have a "soft" limit on the number of times they can be executed. An example of a "hard" limit on an ability is an ability that can be executed one time every two seconds, enforced by the server. ESO doesn't have many of those. Instead, we designed the game to have soft limits, which mean you can execute as many abilities as you want as long as you have the resources - Stamina or Magicka - to cast them. There is a global "cool down" timer on all abilities, which is set to 1000 milliseconds.

The design goal of soft limits in any game is to allow players to create builds that let them execute abilities more often depending on build choices made, and not to have hard ceilings on damage or healing per second. This gives a lot of control to the player, which makes for a really fun and interesting system, but it can lead to situations where players cast too many abilities too quickly (and continuously) if strict limits on resources are not enforced.

Over the years, player power has grown considerably. With the addition of the Champion System, various armor/weapons sets, and changes to abilities, we have reached the stage where players - with the right build - can cast near-infinite numbers of abilities. If you add in a properly managed group, with some focused on damage and some focused on healing and regen boosts, you have a perpetually running never-ending stream of abilities.

With that in mind, consider how Area of Effect (AOE) abilities work: when cast, they look at a specific target area - almost always the area directly around the caster - find targets, and perform the ability. Each of those steps requires server calculations.

At launch, Cyrodiil’s processes were able to keep up with the number of AOEs cast, because most players couldn’t cast that many of them: they ran out of Magicka or Stamina, so they had to use AOEs judiciously. Over time, as player knowledge grew and regen builds grew in power, more players could cast more and more AOEs before running out of resources.

During these years, we found and fixed many performance issues in Cyrodiil, and we continue to do so. We've fixed significant issues, uncovered more, and continue to find and fix with every Update.

However, one foundational issue remains. At some point, we crossed a threshold where most players in PvP were able to cast endless AOE abilities, without ever running out of resources. This is done through player knowledge, builds and group mechanics – resulting in a constant stream of AOEs with many players never using any other type of ability.

This is not what we intended, but part of the fun of Elder Scrolls games is designing a build that has unexpected and powerful results, and we allowed it. However, as this behavior grew more prevalent, we reached a point where casting so many continuous AOE abilities in such a small area started to overwhelm the server process for that area, leading to situations where the "lag meter" spikes and the server becomes unresponsive for a period of time.

Our initial response to this problem – starting with Update 22 – was to find (and fix) more than a few problems with AOE ability calculations and make them more performant - and we stayed (mostly) ahead of the problem. But as more players reached maximum Champion Level and more players started utilizing this particular method of "AOE spamming", we have reached the point where we cannot fix this issue around the edges: we need to address the core problem, which we will be starting with Update 27.

To do this, we will need to first do some analysis, and we can only do this on the live servers. As much as we try to avoid running tests on live servers, they are the only place where the combination of player behavior, specific builds, and mass battles happen. So, starting the week of August 24, we will be running a series of tests on the live PC servers – both NA & EU, only in Cyrodiil. Please note that we will not be running these tests on any Console servers. Each test should take about a week, but if needed, we will extend the testing time.

The first round of tests we are planning will focus on Area of Effect (AOE) abilities in Cyrodiil and will make it more difficult for AOE abilities to be the only abilities used, adjusting cooldown, cost and regen values of all AOEs (damage and healing).

Specific details on the tests we will be running in Cyrodiil are as follows:

• Test 1 – Shared global AOE cooldown - 3 second timer. This test adds a global 3 second shared cooldown to any AOE ability. This means that when you cast an AOE, you will not be able to cast another for 3 seconds. For example, as a Templar, if I cast Ritual of Retribution, I wouldn’t be able to cast Puncturing Sweep for 3 seconds.
• Test 2 – Individual AOE cooldown - 3 second timer. This test adds an individual AOE cooldown to each AOE ability. This means that when you cast an AOE, you will not be able to cast that same AOE ability for 3 seconds. For example – as a Templar, if I cast Puncturing Sweep, I wouldn’t be able to cast Puncturing Sweep again for 3 seconds.
• Test 3 – No cooldown, global ramping AOE cost. This test adds a global ramping AOE cost for each AOE cast. Similar to how streak or roll dodge works, where when you cast an AOE you receive a debuff for 5 seconds, each stack of the debuff increases the cost of any AOE cast.
• Test 4 – Individual AOE cooldown – 3 second timer, global ramping AOE cost. This test adds an individual AOE cooldown to each ability as in test 2, but also combines that with a global ramping cost from test 3.

During the times that any of these tests are active, we will be awarding double Alliance Points for anyone active in Cyrodiil.

After we complete the above tests, we may try other combinations of cooldown, cost, and regen values on AOE abilities. However, we need to run these tests first and then assess the data. We will then let everyone know what we found and how we will move forward. We will be very upfront, but please be aware that if these tests confirm our hypothesis, then chain-casting AOE abilities will no longer form the core of the ESO PvP experience in the way it has for the last few years. We would then go through each class and ensure that there are viable builds for each and make adjustments as necessary.

The code for these changes will be going into today’s PTS patch, and we will be running some basic tests throughout the day on Tuesday in Cyrodiil to ensure we’re able to make the above changes without requiring any maintenance or downtime. Once we launch Update 27 for PC on August 24, we will announce when one of the tests are beginning through an in-game announcement and will have a forum thread detailing which test is currently being performed. Remember, these tests will be limited to Cyrodiil, so AOE abilities in other PVP spaces (Imperial City and Battlegrounds) and will remain unchanged for now. We will evaluate how these tests go, and let everyone know next steps.

Thanks for being patient and understanding with these tests. They are extremely important for the team and will help us work towards improving Cyrodiil performance.

You’re telling us how your game works, when I know for sure on numerous occasions in meetings devs as well as class rep meetings where things were brought up that you had no clue about.

Let’s be honest, you have NEVER fixed anything regarding performance in cyrodiil. It’s still the same abhorrent mess it’s always been. Off hours where the population/crippling lag isn’t high/bad enough to show the glaring mismanagement of your own game is basically the last bastion for many players, who unfortunately cling to your game like itchy addicts because when the game works... there is simply nothing else like it. You can cater all you want to the brainless slot machine players that blob around a circle in the middle of a zone, and provided the game still works... people play.

Problem is, it hasn’t worked, to some form or another, for the games entire life.
And your “test” is of course going to provide positive results, because there will be functionally no fighting. Look at all your abilities that have secondary and tertiary effects that hit more than enemy... it vastly outweighs the number of remaining available skills that fit the criteria for your test.

Good post

exeeter702

Elsonso wrote: »

MipMip wrote: »

Dottzgaming wrote: »

This unfortunately will probably get a "positive" performance result. Combat is going to slow down to a grinding hault, so performance will improve since people will be playing the game at a significantly slower speed.

ESO's hallmark feature is it's combat speed and fluidity, especially for PvP. Cooldowns shouldn't be introduced into a system that was fundamentally designed to not have them. Period.

Thanks @Dottzgaming for this excellent comment, and @NirnStorm and many others who point out how such changes will make the game slower and less interesting.

I also agree with those who say that these issues should be addressed not by changing fundamental game mechanics but by increasing server performance (many of us spend a lot of money on subs and in the crown store...)

If these changes go through I think I will recommend the game to my grandparents!

I think that the whole point is that there is no where left to increase server performance.

Where I think that @Dottzgaming is missing the point is that he fails to mention that something has to give. If the cooldowns are unacceptable, then remove the spammable AoE so players are not able to even do it, and if that is not acceptable, do whatever they did for MYM that is obviously something so unspeakable that they won't even suggest it.

It certainly appears that ESO has arrived at a zero-sum situation. More was added than the game can handle. Something needs to be removed, since it is apparently not possible to have a bigger box to put it all in.

Precisely this. We have reached a point where we cannot have our cake and eat it. What rich said made 100 percent sense. The issue is that eso is not a sophisticated game in terms of targeting. Had this been a game with some kind of lock on focus targeting system, you may have seen a different situation in pvp zones where large mobs of players congregate and Aoe spam along side smart multi target healing becomes the only reasonable approach. The sheer hubris to proclaim that cooldowns of any kind shouldnt exist "period" is just grandstanding as far as I'm concerned. Duration based aoe abilites with a price hike would possibly dissuade spamming them and making their placement more meaningful, likewise, instant effect aoe abilities can be tuned to have power but be expensive as to not be spammable.

Either way, eso has reached a point where something needs to give. To assume zos doesnt want the game to be a good as it can and that the issues herein are purely stemmed from some kind of degree of ineptitude is nonsense. The main issue is perhaps a lack of foresight into the effects of power creep and sustain leniency along side designing a combat system that mandates aoe use when you are dealing with more than 4 targets a time among other things like pets and whatnot.

relentless_turnip

This also put duel wield further behind 2h with an execute already double the cost, that now can only be cast every 3 secs or increases in cost... Or both 😂

runningtings

Sugaroverdose wrote: »

runningtings wrote: »

Excellent news, only ones complaining are PVE players who seem to have missed that this is for PVP and ball group sad cases.

Rofl, i don't want to make you feel bad, but this "test" will harm solo and small scale players hell a lot harder than even semi-organised zergs

At this point as long as they fix the lag and general performance I am fine with that.

I will adapt, that will be within my control, at the moment I can do nothing when these idiots are running about a keep spamming AoEs.

Gorreck

ZOS_RichLambert wrote: »

. For example – as a Templar, if I cast Puncturing Sweep, I wouldn’t be able to cast Puncturing Sweep again for 3 seconds.

Whilst I largely agree with the idea of AOE cds in PvP (to reign in ball-groups), if you do this to Templars sweeps (mag and stam) you're going to have to give them another class spamable.

TheAlphaRaider

FatherDelve wrote: »

All I can say is, that i have way less Performance Problems with 40 random zerglings around me, then having 8 Ballgroup Members spamming every 1 second a *** of abilities at 5 meters.

THANK YOU - Add penalties to resistance on groups of more than 4 or maybe 6. ADD THIS WHEN IN A ZERG: No cooldown, global ramping AOE cost. This test adds a global ramping AOE cost for each AOE cast. Similar to how streak or roll dodge works, where when you cast an AOE you receive a debuff for 5 seconds, each stack of the debuff increases the cost of any AOE cast.

INCREASE DMG FOR PROXY DET by 5x solo bombing.

DOWN WITH ZERGS

MurderMostFoul

relentless_turnip wrote: »

This also put duel wield further behind 2h with an execute already double the cost, that now can only be cast every 3 secs or increases in cost... Or both 😂

Please folks, stop spamming the the thread with skill specific complaints. If any of these AOE changes are adopted, we can expect nearly every AOE skill to see major changes, as well as numerous other skills being changed for balancing.

ClawOfTheTwoMoons

Everyone should just get together in cyro from all factions and just cast as many ground aoes and ground hots as possible just to show them its not going to help. Instead of doing the obvious thing and upgrading the server hardware, they decide the players are the problem. Honk honk

TheAlphaRaider

maxjapank wrote: »

What a nightmare for a Templar’s during these tests as our main spammable, Sweeps, is aoe.

its a nightmare when all you do is spam jabs and sweeps its effective. Pretty boring to watch a templar 1 v x

MiraG

How about instead of these disastrous proposed changes we improve the game performance and enjoyment by making the following 2 changes:

1) Limit Cyrodiil group sizes to 12.

2.) Healing and buffs can only be bestowed by group members. You can still help your fellow alliance members by attacking the common enemy and staying alive yourself. And , better yet, by grouping up. Eg, you and 3 others get wiped individually and respawn at the nearest keep, quickly group up and go back out and seek your vengeance. This will provide an incentive for people to group in lieu of only zerg surfing. This will promote sociality AND small scale pvp as more and more small groups would be running around and encountering each other.

This would leave damage AOE and other mechanics untouched and I believe greatly reduce stress on the server.

Sugaroverdose

SmokeHappiness wrote: »

I hope you will someday give up your policy of guessing problem solving.

keep your pocket open, hah. Unfortunately whole ZoS are strong as wrobel - they do crap without even trying to think a little beforehand

runningtings wrote: »

Sugaroverdose wrote: »

runningtings wrote: »

Excellent news, only ones complaining are PVE players who seem to have missed that this is for PVP and ball group sad cases.

Rofl, i don't want to make you feel bad, but this "test" will harm solo and small scale players hell a lot harder than even semi-organised zergs

At this point as long as they fix the lag and general performance I am fine with that.

I will adapt, that will be within my control, at the moment I can do nothing when these idiots are running about a keep spamming AoEs.

Problem is, zergs aint going nowhere, and you'll get spammed by radiant destruction instead)

likecats

ClawOfTheTwoMoons wrote: »

Everyone should just get together in cyro from all factions and just cast as many ground aoes and ground hots as possible. Instead of doing the obvious thing and upgrading the server hardware, they decide the players are the problem. Honk honk

It's a scaling problem. There is no way to make AOE's as performant as single target abilities. Hypothetically they might be able to add servers and outscale this problem. But they will eventually have to address what they just wrote about.

Fur_like_snow

@ZOS_RichLambert

End the three banner war bring the troops home we want a new PvP DLC tell the story of the post war. Give us a smaller map with a much smaller population cap the game functions to a certain degree when it’s not pop locked. Cyrodiil is old content that you don’t monetize that creates so many problems in other parts of the game. Zeni you were not interested in fixing Cyrodiil years ago it’s time to move on.

SillyGT

There is one simple fix to cyrodiil lag. To put it simply just need the amount of ap you earn for capturing a castle. There will be no reason for 150+ people spamming abilities and putting stress on the server. If people are there to play objectively and earn points for score eval just make keeps and resources worth more like scrolls. I honestly think this would encourage people to spread out and not cause huge server stress.

runningtings

Sugaroverdose wrote: »

SmokeHappiness wrote: »

I hope you will someday give up your policy of guessing problem solving.

keep your pocket open, hah. Unfortunately whole ZoS are strong as wrobel - they do crap without even trying to think a little beforehand

runningtings wrote: »

Sugaroverdose wrote: »

runningtings wrote: »

Excellent news, only ones complaining are PVE players who seem to have missed that this is for PVP and ball group sad cases.

Rofl, i don't want to make you feel bad, but this "test" will harm solo and small scale players hell a lot harder than even semi-organised zergs

At this point as long as they fix the lag and general performance I am fine with that.

I will adapt, that will be within my control, at the moment I can do nothing when these idiots are running about a keep spamming AoEs.

Problem is, zergs aint going nowhere, and you'll get spammed by radiant destruction instead)

I don't think Zergs are the issue, there was plenty of that when the grouping mysteriously broke a few weeks back and the lag was nowhere near as bad as when it got fixed and the ballgroups were back.

Sugaroverdose

runningtings wrote: »

Sugaroverdose wrote: »

SmokeHappiness wrote: »

I hope you will someday give up your policy of guessing problem solving.

keep your pocket open, hah. Unfortunately whole ZoS are strong as wrobel - they do crap without even trying to think a little beforehand

runningtings wrote: »

Sugaroverdose wrote: »

runningtings wrote: »

Excellent news, only ones complaining are PVE players who seem to have missed that this is for PVP and ball group sad cases.

Rofl, i don't want to make you feel bad, but this "test" will harm solo and small scale players hell a lot harder than even semi-organised zergs

At this point as long as they fix the lag and general performance I am fine with that.

I will adapt, that will be within my control, at the moment I can do nothing when these idiots are running about a keep spamming AoEs.

Problem is, zergs aint going nowhere, and you'll get spammed by radiant destruction instead)

I don't think Zergs are the issue, there was plenty of that when the grouping mysteriously broke a few weeks back and the lag was nowhere near as bad as when it got fixed and the ballgroups were back.

Grouping and zerging are not equal, you don't need to be in group to be in zerg, zerg is just a massive number of pugs who moves in similar direction.

Gorreck

TheAlphaRaider wrote: »

maxjapank wrote: »

What a nightmare for a Templar’s during these tests as our main spammable, Sweeps, is aoe.

its a nightmare when all you do is spam jabs and sweeps its effective. Pretty boring to watch a templar 1 v x

No main spamable in the game would survive a 3 second cooldown as a viable option (unless EVERY main spamable had one - which would reduce DPS massively across the board).

Given they've specifically mentioned sweeps as an example, I'm not entirely sure they'd remotely thought this through.

Sorbin

To put it bluntly: saving Cyrodiil isn't worth gutting the core game mechanics. If performance can't be improved in Cyrodiil, then use it as a means to effectively end the conflict there and introduce new, smaller scale PvP instances that players can engage in.

JBNimble

That's some really... creative... workaround...

I am so angry right now I don't know what to say without being obscene. I'll better just leave...

AScaryDinosaur

I appreciate everyone at ZOS for continually working to improve Cyrodiil performance. Thank you!

This stated, I hope these changes never come to fruition. Specifically, tests 1,2, and 4 (the ones containing ability cooldowns). I understand these are simply tests and are in no way certain. However, I would be surprised if these tests were NOT successful in reducing lag and they would then be potentially considered as viable performance options. I, as well as many other players, wish to see PvP performance improved significantly. However not at the cost of ESO's identity and Distinction from other MMOs on the market. Then again if this made the game playable... I am torn there. Just remember there are classes which rely on AOE skills (like templars and their spammable, Puncturing Strikes) which will be disproportionally hindered by these changes. I fell in love with this game for it's combat system and the freedom to do whatever you want with it and I would be devastated if it was partially taken away.

To end positively, Thank you again to everyone at ZOS for continuing to work on improving PvP performance and I hope everyone has a great, productive, and safe week!

Crow_IX

What does changing AOE's have to do with fixing these issues?

1: Positioning desync (targeting issues)
2: Health desync
3: Skills that take resources but don't activate (dud skills)
4: Skill delay

TequilaFire

TheAlphaRaider wrote: »

FatherDelve wrote: »

All I can say is, that i have way less Performance Problems with 40 random zerglings around me, then having 8 Ballgroup Members spamming every 1 second a *** of abilities at 5 meters.

THANK YOU - Add penalties to resistance on groups of more than 4 or maybe 6. ADD THIS WHEN IN A ZERG: No cooldown, global ramping AOE cost. This test adds a global ramping AOE cost for each AOE cast. Similar to how streak or roll dodge works, where when you cast an AOE you receive a debuff for 5 seconds, each stack of the debuff increases the cost of any AOE cast.

INCREASE DMG FOR PROXY DET by 5x solo bombing.

DOWN WITH ZERGS

How about remove solo players from Cyro by requiring groups only?
Just as fair as what you suggest.

Elsonso

MurderMostFoul wrote: »

Is a hardware solution on ZOS's end impossible?

StShoot wrote: »

there is always a way to increase server performance, they could pay for an own mega server for pvp, raids and so on. But that would cost them alot. So the question is not if there IS a way to increase server performance. The question is if zos wants to pay more money for increasing server performance.

Based on what they have said on the long road that leads here, I think they hit the maximum performance that they could get by buying better hardware back in 2014. At least, for the hardware platform they are running on. I doubt that other commercial platforms would be all that much faster. Everything since then has just been incremental improvement as technology changes, and I think they have basically been looking for ways to tune the game itself to improve performance. That is basically what Lambert said today.

likecats

Lannharr wrote: »

With all due respect, isn't hardware upgrade a solution but not cost effective?

Infinite hardware resources is a solution to all inefficient code problems. But it's usually smarter to fix software inefficiencies. Transitioning from AOE spam to single target spam with intermittent AOE's will not destroy the game, but it will make performance a lot better.

Ofcourse a lot of skills and classes will need to be redesigned before this takes place.

KageNin

Or you can upgrade servers like you did for MyM but hey gotta save the buck and downgrade only advantage the game has (fast paced combat). Also you wanna reduce impact of AoE skills in Cyro ? start with group healing only, like Fengrush is advising for more than a year now.
Good luck.

likecats

KageNin wrote: »

Or you can upgrade servers like you did for MyM but hey gotta save the buck and downgrade only advantage the game has (fast paced combat). Well good luck.

I only played Cyrodil once during MyM, but precisely because the performance was so terrible.
We need a better standard.

Sugaroverdose

Well after observing this topic for some time, i believe that game just need to reverse all of battle mechanics changes to 1.5 patch state, with soft caps and everyone will be happy