Performance test: Is there something we are not being told?

React · February 2021

The balance has improved dramatically, and combat feels better in general when you boil the game down to base stats. I think this is the correct route for zenimax to take as far as PVP balancing goes, and I hope they continue to follow it.

That said, the performance might have gotten worse these past few days, and I cannot understand why. This test should be massively reducing the checks and calculations that we've been told are likely the main cause of Cyrodiil lag. The fact that things have been made worse makes me think that either cross healing/cross buffing is heavily taxing on the servers (this was re-enabled for the test), or that zenimax is using the test to experiment with server capacity.

I'd like to believe that it is the former, and I think that zenimax should use at least one week during test to try disabling cross healing/buffing again with the set limitations in place.

If that fails to improve performance at all, I'd be curious to know if they've made adjustments to their server capacity or virtual machines during this test. We had a login queue two nights in a row since the test began, which seems odd given that the PVP population during the queue was not signifcantly higher than it normally is. Additionally within the campaigns themselves, it almost seems like the population caps have been brought down - there are consistently very few conflicts happening on the map for the number of players the campaigns supposedly have.

It's a shame because this test truly seemed like the first thing major enough to make a difference, and the balance in pvp is better than it has been in years.

VaranisArano · February 2021

Correlation is not necessarily causation. You've seen this from Wheeler, right?

"When this test begins on February 15, we will also be reverting a prior test where abilities cast on allies were restricted to group targets only. This adjustment gave no appreciable gains towards performance improvements, so this restriction will be removed until further notice and abilities which target/can be triggered by players outside of your group will once again function as they do outside of Cyrodiil."

There are some players who think ZOS is lying through their teeth about that. Is that your theory, too?

VaranisArano · February 2021

To address the title question, yes there ARE several things we aren't being told.

1. What does ZOS do during Midyear Mayhem to improve performance during peak population and then revert once MYM is over?

2. What has ZOS done to alleviate the stress caused by moving combat calculations server-side, both in response to the old cheating problem and for Stadia?

3. Before testing, ZOS said their major problem was massed AOE spam, yet none of the lasting changes they've made have substantially changed the ability of 12-man ball groups to produce massed AOE spam. Is this still something ZOS identifies as a problem? If so, what's their plan?

danthemann5 · February 2021

Of course there are things we're not being told.

On the other hand, let's take a look at what we were told, namely that ZOS is acquiring new servers.

Let's say I'm buying new servers for my business. I want to maximize profits and minimize expenses. How much server capacity do I buy? If I care about the quality and longevity of my product, I buy enough to maintain or improve the quality of my product and allow for projected growth. If I only care about short-term profit, I would buy the absolute minimum to operate while maintaining my revenue.

Assuming I choose the latter option, I have to figure out how little server I can get away with and not hurt my sales. To do that, a reasonable approach would be to run a test.

Of course, that's just what I would do if I were running a business. Purely hypothetical. There's no way ZOS would do that as they care immensely about the quality and longevity of their product.

Joy_Division · February 2021

I don;t think ZOs is secretly testing something else. I think this is a case of expectations not meeting reality and being prisoner of the moment.

For some reason, many people got their hopes up that disabling proc sets would make a difference. And many of those same people also happen to prefer non-proc gameplay. So what we have here are two levels of disappointment, which is going to make the regular old annoying lag, which I doubt has changed despite all the reports of pros and cons, to seem all the more frustrating.

You clearly wanted this to work. That's going to impact perceptions.

Every single patch, people say "the lag is worse." People are prisoners of the moment have have basically zero sense of accurately comparing past and present without access to very good primary (and ideally objective) sources of data. Way back in the Summer of 2020 before ZOS came up with the idea of testing, my tolerance for playing on PC-NA main CP campaign was about 15 minutes on a weekend. During ZOS's testing it was 15 minutes. After the testing with the removal of cross heal it was 15 minutes. From what I experienced so far, this weekend if going to be 15 minutes (although, perhaps longer because at least I dont have to deal with crimson, venomous smite, Thews, etc.).

For the past 4-5 years, ZOS has probably removed more calculations than any of there tests are ever going to do. The CP system was super bloated, so many skills were "standardized" such that their multi-functionality was removed and impact of gameplay severely reduced (see for example, the history of Blazing Spear), every patch I read in the notes about this change/update that "removes excess calculations," and we're not even considering like 3 or 4 years ago, they totally removed CP from every campaign and that still did not affect performance, something they admitted on an episode of ESO Live. ZOS keeps insisting that it's the volume of calculations that is the source of lag, yet every test, update, and change they have made in the past 6 years, none of which had a perceptible effect, strongly suggests that answer is at best, superficial and incomplete. Since removing calculations does little, to me that means the issue is the amount of calculations the server can handle in the first place is way too small; the proverbial cup is overflowing as it is, such that reducing the amount of water incoming isn;t going to make a difference.

I will say I play on no CP and as much as I dislike the snobbishness that sometimes accompanies attitudes from players that it's the "skillful" server, when I play on a weeknight primetime, I would say 90% of the time when I press an ability, it goes off. I would, however, attribute this mostly to the general routine of how PVP is played there than the removal of CP. On most nights there aren't as many of the organized tightly pact groups as on the CP server. There usually aren't the sort of epic keep sieges that I've experienced on the default CP campaign. But on Friday night, when the patterns of gameplay resemble the default CP campaign, so does the performance. This is probably what ZOS was getting at when it mentioned "player behavior" and cross healing, but ZOS clearly does not know what motivates players in the first place to change that behavior. Forcing people to group and changing the rules to favor those players outputting a disproportionally high number of calculations to begin with was a premise that was flawed from the start and no surprise it didn't work

I guess what I am trying to say is play your weeknights on no CP rather than putting your hopes that ZOS's next test, update, or set of balance changes is going to have a discernible impact on the main CP campaign. Friday nights though are still going to be frustrating.

React · February 2021

VaranisArano wrote: »

You've seen this from Wheeler, right?

"When this test begins on February 15, we will also be reverting a prior test where abilities cast on allies were restricted to group targets only. This adjustment gave no appreciable gains towards performance improvements, so this restriction will be removed until further notice and abilities which target/can be triggered by players outside of your group will once again function as they do outside of Cyrodiil."

There are some players who think ZOS is lying through their teeth about that. Is that your theory, too?

I mean, I'm honestley not sure. From and outside perspective, limiting cross healing/buffing seems like it MUST have some level of positive performance impact. Not only is healing being calculated across a ton of different people, but AOE applied passives like minor sorcery on templar, minor brutality on DK, minor prophecy on sorcerer, etc are being accounted for as well.

They said that there was no noticeable effect on performance with the cross healing change, but yet they go straight into this test which should reduce calculations signifcantly more than any previous test, and we see a negative impact on performance from just a few weeks ago. It just doesn't make any sense, and logically there only seems like there could be a few explanations.

1. The cross healing did actually have an impact, accounting for the lack of performance increase with this test due to them re enabling it.

2. They have literally no idea what they're doing, they've been feeding us nonsense by identifying these "calculations" as the reason the server is lagging.

3. They're messing around with their servers in some capacity without telling us. Turning off virtual machines, reducing max population caps, etc.

VaranisArano wrote: »

To address the title question, yes there ARE several things we aren't being told.

1. What does ZOS do during Midyear Mayhem to improve performance during peak population and then revert once MYM is over?

2. What has ZOS done to alleviate the stress caused by moving combat calculations server-side, both in response to the old cheating problem and for Stadia?

3. Before testing, ZOS said their major problem was massed AOE spam, yet none of the lasting changes they've made have substantially changed the ability of 12-man ball groups to produce massed AOE spam. Is this still something ZOS identifies as a problem? If so, what's their plan?

To address your second comment:

1. They DEFINITELY do something during MYM, and its shady that they won't tell us what it is. There is no denying it at this point; MYM functioned better with more players online and every set in the game enabled than this test is functioning right now.

2. This is a long debated topic. I can't say much with my limited understanding of how these things work, but there were several points when certain calculations were moved server side that caused direct negative impacts on performance and functionality. The biggest one in recent memory was the change to the range checks for abilities, that suddenly caused almost every close range instant cast ability to become unreliable at best.

3. Ball groups are definitely taxing on the server. Personally I think that the changes they made during the AOE tests were mostly unacceptable - eso is a game that cannot incorporate ability cooldowns like the ones proposed. The only changes in these tests that seemed appropriate were the ones that only forced these limitations on to groups larger than 6.

React · February 2021

Joy_Division wrote: »

I don;t think ZOs is secretly testing something else. I think this is a case of expectations not meeting reality and being prisoner of the moment.

For some reason, many people got their hopes up that disabling proc sets would make a difference. And many of those same people also happen to prefer non-proc gameplay. So what we have here are two levels of disappointment, which is going to make the regular old annoying lag, which I doubt has changed despite all the reports of pros and cons, to seem all the more frustrating.

You clearly wanted this to work. That's going to impact perceptions.

Every single patch, people say "the lag is worse." People are prisoners of the moment have have basically zero sense of accurately comparing past and present without access to very good primary (and ideally objective) sources of data. Way back in the Summer of 2020 before ZOS came up with the idea of testing, my tolerance for playing on PC-NA main CP campaign was about 15 minutes on a weekend. During ZOS's testing it was 15 minutes. After the testing with the removal of cross heal it was 15 minutes. From what I experienced so far, this weekend if going to be 15 minutes (although, perhaps longer because at least I dont have to deal with crimson, venomous smite, Thews, etc.).

For the past 4-5 years, ZOS has probably removed more calculations than any of there tests are ever going to do. The CP system was super bloated, so many skills were "standardized" such that their multi-functionality was removed and impact of gameplay severely reduced (see for example, the history of Blazing Spear), every patch I read in the notes about this change/update that "removes excess calculations," and we're not even considering like 3 or 4 years ago, they totally removed CP from every campaign and that still did not affect performance, something they admitted on an episode of ESO Live. ZOS keeps insisting that it's the volume of calculations that is the source of lag, yet every test, update, and change they have made in the past 6 years, none of which had a perceptible effect, strongly suggests that answer is at best, superficial and incomplete. Since removing calculations does little, to me that means the issue is the amount of calculations the server can handle in the first place is way too small; the proverbial cup is overflowing as it is, such that reducing the amount of water incoming isn;t going to make a difference.

I will say I play on no CP and as much as I dislike the format and the snobbishness that usually accompanies attitudes from players that it's the "skillful" server, when I play on a weeknight primetime, I would say 90% of the time when I press an ability, it goes off. I would, however, attribute this mostly to the general routine of how PVP is played there than the removal of CP. On most nights there aren't as many of the organized tightly pact groups as on the CP server. There usually aren't the sort of epic keep sieges that I've experienced on the default CP campaign. But on Friday night, when the patterns of gameplay resemble the default CP campaign, so does the performance. This is probably what ZOS was getting at when it mentioned "player behavior" and cross healing, but ZOS clearly does not know what motivates players in the first place to change that behavior. Forcing people to group and changing the rules to favor those players outputting a disproportionally high number of calculations to begin with was a premise that was flawed from the start and no surprise it didn't work

I guess what I am trying to say is play your weeknights on no CP rather than putting your hopes that ZOS's next test, update, or set of balance changes is going to have a discernible impact on the main CP campaign. Friday nights though are still going to be frustrating.

I'm definitely guilty of being a player that prefers pure, stat based gameplay as well as one that was genuinely hoping for good results from this test - even so, a negative impact on performance by this test is just not something that seems possible at any level. I do genuinely think that there was a negative impact as well, and that the opinion isn't skewed by disappointment - I tested different campaigns at different levels of population, and got the same or worse results that I've seen in previous patches as far as ability usage goes.

I wish it was just limited to CP, but the no cp campaign is nearly as bad. Just yesterday at 7-8PM eastern, with 2 bar 2 bar 3 bar in no-cp while fighting in outlying areas with no major conflicts visible on the map, the delay was at the point I'd deem it "unplayable", with abilities taking 2-3 seconds to fire and damage numbers taking seconds to register after an ability actually goes off.

I guess I agree with your sentiment that they probably have no idea what the issue is, and they're pushing the "calculations" thing as a way to stall time while the continue to release DLCs and work on the more profitable areas of the game. I just wish that they would be more transparent about what is actually happening behind the scenes. Unfortunately there just isnt enough pressure on them in any public forum for them to actually provide this insight, so here I am making the desperation forum post before departing from the game again for the foreseeable future.

ne.ga.kurai_ESO · February 2021

I was fighting solo vs a group at Bleakers last night. Hit a Radiating Regen to prebuff and instead healed the full HP guards instead of me. I already had to switch from Rapid Regen to the other morph.

I can tell you my behavior has immediately changed to spamming heals, b/c I can't rely on Rapid to heal me anymore.

Faded · February 2021

Yeah, there are probably things we aren't being told.

Consider this: the devs really like proc sets. They don't magically appear in the game, somebody designs them and and somebody else says "My god that's a great idea! Holding block and doing damage at the same time!" and they drop them into a game that is already crammed with proc sets. Look how short the approved list of sets is for this test. PVP players are less enthused about proc sets. So a lot are showing up for this test, even people who left years ago.

It's possible they only want to see what effect stripping all those extra calculations has on Cyrodiil. But while they have a consistent high population in their live-server testing environment... good a time as any to quietly check other things. Data is useful.

That seems likely. Why bother to mention it to us.

It's also informative to see how many players will talk themselves and try to talk others into outcomes that aren't happening so they can push a "solution" they like. Look how hard people are clinging to cross healing being the Great Performance Problem. How much feedback can we trust and how much is always confirmation bias noise. If they're going to keep running tests on the live servers, a feel for that will be useful too.

Joy_Division · February 2021

Liam12548 wrote: »

Joy_Division wrote: »

I don;t think ZOs is secretly testing something else. I think this is a case of expectations not meeting reality and being prisoner of the moment.

For some reason, many people got their hopes up that disabling proc sets would make a difference. And many of those same people also happen to prefer non-proc gameplay. So what we have here are two levels of disappointment, which is going to make the regular old annoying lag, which I doubt has changed despite all the reports of pros and cons, to seem all the more frustrating.

You clearly wanted this to work. That's going to impact perceptions.

Every single patch, people say "the lag is worse." People are prisoners of the moment have have basically zero sense of accurately comparing past and present without access to very good primary (and ideally objective) sources of data. Way back in the Summer of 2020 before ZOS came up with the idea of testing, my tolerance for playing on PC-NA main CP campaign was about 15 minutes on a weekend. During ZOS's testing it was 15 minutes. After the testing with the removal of cross heal it was 15 minutes. From what I experienced so far, this weekend if going to be 15 minutes (although, perhaps longer because at least I dont have to deal with crimson, venomous smite, Thews, etc.).

For the past 4-5 years, ZOS has probably removed more calculations than any of there tests are ever going to do. The CP system was super bloated, so many skills were "standardized" such that their multi-functionality was removed and impact of gameplay severely reduced (see for example, the history of Blazing Spear), every patch I read in the notes about this change/update that "removes excess calculations," and we're not even considering like 3 or 4 years ago, they totally removed CP from every campaign and that still did not affect performance, something they admitted on an episode of ESO Live. ZOS keeps insisting that it's the volume of calculations that is the source of lag, yet every test, update, and change they have made in the past 6 years, none of which had a perceptible effect, strongly suggests that answer is at best, superficial and incomplete. Since removing calculations does little, to me that means the issue is the amount of calculations the server can handle in the first place is way too small; the proverbial cup is overflowing as it is, such that reducing the amount of water incoming isn;t going to make a difference.

I will say I play on no CP and as much as I dislike the format and the snobbishness that usually accompanies attitudes from players that it's the "skillful" server, when I play on a weeknight primetime, I would say 90% of the time when I press an ability, it goes off. I would, however, attribute this mostly to the general routine of how PVP is played there than the removal of CP. On most nights there aren't as many of the organized tightly pact groups as on the CP server. There usually aren't the sort of epic keep sieges that I've experienced on the default CP campaign. But on Friday night, when the patterns of gameplay resemble the default CP campaign, so does the performance. This is probably what ZOS was getting at when it mentioned "player behavior" and cross healing, but ZOS clearly does not know what motivates players in the first place to change that behavior. Forcing people to group and changing the rules to favor those players outputting a disproportionally high number of calculations to begin with was a premise that was flawed from the start and no surprise it didn't work

I guess what I am trying to say is play your weeknights on no CP rather than putting your hopes that ZOS's next test, update, or set of balance changes is going to have a discernible impact on the main CP campaign. Friday nights though are still going to be frustrating.

I'm definitely guilty of being a player that prefers pure, stat based gameplay as well as one that was genuinely hoping for good results from this test - even so, a negative impact on performance by this test is just not something that seems possible at any level. I do genuinely think that there was a negative impact as well, and that the opinion isn't skewed by disappointment - I tested different campaigns at different levels of population, and got the same or worse results that I've seen in previous patches as far as ability usage goes.

I wish it was just limited to CP, but the no cp campaign is nearly as bad. Just yesterday at 7-8PM eastern, with 2 bar 2 bar 3 bar in no-cp while fighting in outlying areas with no major conflicts visible on the map, the delay was at the point I'd deem it "unplayable", with abilities taking 2-3 seconds to fire and damage numbers taking seconds to register after an ability actually goes off.

I guess I agree with your sentiment that they probably have no idea what the issue is, and they're pushing the "calculations" thing as a way to stall time while the continue to release DLCs and work on the more profitable areas of the game. I just wish that they would be more transparent about what is actually happening behind the scenes. Unfortunately there just isnt enough pressure on them in any public forum for them to actually provide this insight, so here I am making the desperation forum post before departing from the game again for the foreseeable future.

I don't doubt you believe the preformance impact was genuinely negative, but humans are lousy at making these sorts of accurate determinations based purely on their mental faculties alone, sample size is way too short and small given the high variance of PvP performance, and we are dealing with expectations, wants, and other sorts of unwanted subjective factors. How exactly does one compare last night to August 13, 2020 on Greyhost PC-NA? Memory is inherently unreliable; it is not stored in your brain ala a harddrive and thus accessed when the time calls. The memory is reconstructed and re-remembered every time and thus is altered and adulterated given current circumstances.

I wouldn't say ZOS is stalling as a sort of duplicitous measure. I think they honestly believe that removing server calculations will have the desired effect. They have invested much time and manpower in doing so and on the surface, it's not "wrong": removing say 80% of the calculations would make the game's performance pretty good.

MYM is often mentioned as a time when ZOS does *something* to the servers that suddenly makes the performance better. Generally I'm keen to avoid putting on a tinfoil hat and do think that some of the perceived improvement is due to gameplay patterns: the organized groups are generally spread out across multiple servers and we have a large number of inexperienced players who do not output very many server calculations. So I don't totally buy this.

That being said, there are two things that do make me wonder. Understanding everything I said above about memory, I don't think it's debatable the very first MYM ZOS had was a disaster regarding performance. Indeed, a search of forum posts at that time make it clear that lag and especially disconnects were rampant and far more prevalent than recent MYM events. And the PTS server is definitely set on lower juice/capacity/whatever the proper term is, so that does suggest they can be "upgraded." I don't know if you were around in FEB 2015 when ZOS went to the 1.6 and CP format, but for one night the devs actually set up a PvP event on the PTS in which they were going to participate and stream. Well, of course we all wanted to nuke down a dev and have that on video for prosperity, so we all logged into PTS Cyrodiil that night. Even though it had the same pop caps, oh boy, spent the majority of the 45 minutes or so in loading screens and disconnecting, in-between very sporadic lag dominated gameplay that was akin to a PowerPoint slide that lasted at most 2-3 minutes before being kicked to a loading screen. But then again, anyone who just tries to test the parse on a target dummy knows what the PTS can handle is a fraction of what the Live can handle. Logically, improvements in other fields of technology and especially computers/information do strongly suggest what ESO is capable of in 2021 should be much greater than 2015.
Heck, in 2014 Youtube limited me to 10 minute videos and it took hours to upload them. Yet somehow when it comes to PvP in Cyrodiil, time, technology, and progress are somehow locked, static, and stuck circa 2014, incapable of advancement.

kollege14a5 · February 2021

Cyro performance is definitely worse than before the test.

I wish they disable cross healing for a week so we can actually see the difference.

Stamicka · February 2021

I completely agree with this post. It makes absolutely no sense that limiting sets would somehow worsen performance. I have 2 ideas about why this may be the case:

1. Whatever they did to disable the 5 pieces of so many sets may be straining the server more than actually having those sets equipped.

2. Number of server calculations were never the problem. We are told that AOEs, CP, sets, and heals cause server checks, but we have seen Cyrodiil with each one of those check types disabled and it never seems to help. This leads me to believe that the problem has nothing to do with checks and has some other origin.

I don't have much knowledge on how coding works but I am leaning towards the first idea. Whatever is preventing certain sets from giving you their benefits is stressing the server.

I have a hard time believing that the reintroduction to cross healing is the culprit. This is mainly because in previous years (pre One Tam) there were very few "server check" sets. During this pre One Tam era, people were able to have 24 man groups as well as cross heals. On top of that, Cyrodiil used to have a higher population cap, yet the lag was still MUCH better than it is during this test. CP is also not the culprit because no-cp cyro has this lag as well.

karekiz · February 2021

I will laugh when in the end the lag is how Cyrodiil was designed years and years ago vs today. That in the end they will need to 100% revamp and re-code Cyrodiil down to siege functions to make it work properly.

Spaghetti code be code.

Lord_Hev · February 2021

Yea there is something not being told, and it's the unpopular but hard unbiased truth. Proc meta lessened the lag because people actually died. In this current test, there's only like a dozen sets to choose from. So even bad players have thick heals and good stats out of sheer rng of just mix matching the only available sets. Everyone is built with solid good stats, everyone is on relatively equal statistical playing ground. And because power-creep, in zerg v zerg there is less deaths and more and more stacking of groups on all sides in a conflict.

Pure stat balanced game-play, is no longer balanced like it was years and years ago. The power-creep of the game has now revolved around proc sets. The game cannot support pure stat based meta anymore.

lol

HanStolo · February 2021

Their failure to respond to any inquiries and address issues with their customer's satisfaction is now resulting in conspiracy theories. LOL

ZOS_ConnorG · February 2021

Greetings all,

After review we have closed this thread as it has begun to devolve into conspiracy theories. The original post also lends itself to generating conspiracy theories as well. The spreading of misinformation and conspiracy theories is against our Community Rules.

You are welcome to review the Community Rules here.