AOE CD Result - Objective Analysis, Opinions, & Suggestions

Kungfu

This is coming from one who's been almost 100% Magplar over at least the past 4-5 years and easily 80%+ of my time spent in Cyro.
I have spent my years zerg surfing as a solo, leading healers in large "ball group", healing in "small man" (3-12 person), and have been 100% duo for this test.

Current test: 3-second shared cooldowns on all AOE abilities

Objective Analysis:
AOE CD appears to have an effect on current issues. **BUT** it may be anecdotal.
Last night, I did not encounter any of the desyncs that I have in recent past. At least four times, I found LARGE battles and used Templar's charge ability to dive into battle. Not once did I get stuck like I have been for the past year or so. Lag seemed MUCH better as well: There was a time or two that I *felt* like Crushing Shock did not go off when I clicked. But the problem here is: I don't run Crushing Shock normally... I have almost always run jabs. So... I cannot say whether this was game performance or me with a "new" ability.

Why it may be anecdotal:
This being the first day of this test, I changed my bars slightly in an attempt to survive. So... not only do you now have players not using their normal loadouts & abilities, you also _technically_ cannot mathematically calculate whether this improvement is due to fewer executions of the abilities in question or fewer executions of abilities in general.

Conclusion:
A 3-second global CD is definitely not a viable solution. Though there may be other options in the gray area in between, still.


Suggestions:
Note: mostly Templar specific because that is my expertise. I will not speak to other classes as I am not as proficient & experienced as others with them.

1. Recommend considering a quick patch IMMEDIATELY to lower that AOE CD from three seconds to TWO. If you can find the "break point" where this CD truly affects performance, you will have useful information.
2. Though I do not believe a global CD like this is the solution, Templars could live again with the following changes if this went live:
3. • Change jabs to single-target so they are not on the CD
   • Remove true single-target abilities from this CD. Examples off the top of my head are Honor the Dead, Rapid Regeneration (5s HOT from Resto line)
   • Rework Templar's Rune Focus ability and its morphs. This ability has no reason to be a ground effect anymore. Change them all to be self-only "holy armor" buff akin to stuff like DK armor. Keep the current effects & morphs... just remove it from CD list and make it a buff rather than a ground effect.
4. Suggest separating "damage" AOEs and "healing" AOEs

I believe item #1 is of utmost import. You have a unique opportunity here to test and it seems wise to understand where game performance drops off. Is it at a 2-second CD or 1.5-second CD or 1-second CD? Again, I do not believe a fully global CD is the answer to your problems. But surely there is a happier medium here where the CD is shorter AND the desyncs & "lag" are mitigated.


And now for my rant. Because I deserve it and, even though I know this will fall on deaf ears, it still feels good to get it out.
Opinions:
Templar now has a global CD that hits (quite literally) 7 out of 10 abilities across both of my "normal" bars. I may be self-indulgent here, but it sure feels like Magicka Templars took the biggest hit (as many predicted). It really is unplayable... you built a class with the intention that they would depend on their ability to heal and you ignorantly put this CD on abilities that aren't even AOEs (see: Rapid Regen, Honor the Dead, and even Channeled Focus). Y'all really didn't think this through. Isn't it about time you start listening to your playerbase? You were warned in here about this but you still act like we're idiots. The truth is: you're not actually good at development. You act like a small company of cowboy devs with no large project experience.

When are you going to learn that you have a MASSIVE pool of real-life skills to pull from with us? I am an applications developer with 25 years under my belt in IT. I have made posts in here offering help that have been locked by mods who have no idea what they're reading because they're barely tier 1 support level in their careers.
I have met others in-game who are better than I am at the job (and I'm an egomaniac!). WE CAN HELP if you would ...just... let us. But you have to be capable of putting aside your pride and be willing to listen. Your code is crap. It's been crap for years. You NEED to spin up a separate set of server environments and assign a team to go back to the drawing board & optimize from the base up.

I can tell you just from playing that your multi-threading is done at too high a level - possibly even middle tier. I'm betting you also have overloaded functions that are no longer necessary but still fire. And probably overloaded functions where the original isn't even necessary anymore.

In the game's current state, I will not be playing Templar to test any further unless you make changes to current test. I do not pay you every month so that I can log into the game I love and get frustrated because you've removed the Templar's ability to heal. Cannot even self heal with single-target abilities.

Seriously, ZOS. Implement a true SDLC. Start acting like development professionals. QA/QC your crap and think more thoroughly before you implement - yes, even "tests" like this. I mean, c'mon... y'all are TESTING IN PROD! How the mighty have fallen.

manny254

On PC NA, I could still feel a very noticeable delay on all my skills when fighting in prime time. I tried walking near keep battles twice and crashed twice.

Still feels like this solves almost nothing.

Kungfu

@manny254 Definitely appreciate the input!
My experience could EASILY just have been "good luck". I'm sure we've all seen over the past few years how a whole group of people can seemingly run fine while one or two just seem to get picked on by these crashes & disconnects.

hollywood

It's was only the first day , but it yesterday was a big improvement for me. My ping stayed stable pretty much all day, going as low as 170 during prime time which is insane for me. I could actually play during large battles, it was fun.

But it was fun because I was using a meatbag, since I decided to play my magicka templar and the class is gutted. The second you go on the defensive you're screwed, and that is pretty much the same for all magicka classes as most magicka heals are aoe.

This 3 second global cooldown really isn't the solution. Slows down the game and without reworking several skills there are classes that just aren't fun to play.

Pyvos

I was on PC EU, Standard No CP Ravenwatch last night and the performance was not at all better. The server was performing just as bad as it always does, except now it actually felt worse because the game was literally preventing me from reacting in a timely manner.

Satiar

Performance felt..... same? We were still getting a ton of roll-backs, infinite status effects, long break-frees, etc. It felt slightly better but there weren't that many people on so that could very well be it too.

If this particular test went live I'd be out, almost 100%. Ruins the fluid combat that keeps us coming back to this game despite all its problems.

React

When the solution to the performance being so bad that combat doesn't function is to destroy combat entirely, why even bother?

If you don't see a significant performance increase throughout this entire test, I'd say none of the other ones will increase performance either. The only one that may have an effect should this one fail would be the "Heals/buffs only go to grouped players" one combined with the one that imposes cooldowns on groups larger than 6.

Keep you eyes peeled for me, I'll be on a nightblade harvesting all the poor noobs trying to play the game with a fundamentally gamebreaking change affecting their ability to fight back or defend themselves.

Cinbri

manny254 wrote: »

On PC NA, I could still feel a very noticeable delay on all my skills when fighting in prime time. I tried walking near keep battles twice and crashed twice.

Still feels like this solves almost nothing.

Thats my experience also. Feels like lag decreased only abit.

katorga

I have to assume ZOS has a way to actually measure this. It "felt a little better" is not a valid measurement.

The first test will either fix the issue or not.

If the first test does not fix the issue, then the premise that aoe causes server lag is false then none the other tests will work because they are all different mechanisms to attack the same premise.

AbraXuSeXile

Dunno about other 'ball group' guilds but we cant be bothered playing during tests its not worth the effort to kick water uphill and change everything every week.

Im interested to know if other groups are gonna run. We think we could succesfully run during this but the effort and to test a game we pay for 6 years into its life cycle is just pathetic.

Izanagi.Xiiib16_ESO

AbraXuSeXile wrote: »

Dunno about other 'ball group' guilds but we cant be bothered playing during tests its not worth the effort to kick water uphill and change everything every week.

Im interested to know if other groups are gonna run. We think we could succesfully run during this but the effort and to test a game we pay for 6 years into its life cycle is just pathetic.

Personally the tests offer a chance for actual build and class changes so its kinda refreshing to play during the tests rather than 'more of the same' so far its been fun, takes some getting used to though.

Darkblade_305

The only information that I can provide is that at the same time (NO CP EU) was always full of players, Since the beginning of the testing period it is much emptier, so that may affect performance.

PS: As magplar only can say that... I´ll no come back Cyrodiil maybe never more.

Soul_Demon

Kungfu wrote: »

This is coming from one who's been almost 100% Magplar over at least the past 4-5 years and easily 80%+ of my time spent in Cyro.
I have spent my years zerg surfing as a solo, leading healers in large "ball group", healing in "small man" (3-12 person), and have been 100% duo for this test.

Current test: 3-second shared cooldowns on all AOE abilities

Objective Analysis:
AOE CD appears to have an effect on current issues. **BUT** it may be anecdotal.
Last night, I did not encounter any of the desyncs that I have in recent past. At least four times, I found LARGE battles and used Templar's charge ability to dive into battle. Not once did I get stuck like I have been for the past year or so. Lag seemed MUCH better as well: There was a time or two that I *felt* like Crushing Shock did not go off when I clicked. But the problem here is: I don't run Crushing Shock normally... I have almost always run jabs. So... I cannot say whether this was game performance or me with a "new" ability.

Why it may be anecdotal:
This being the first day of this test, I changed my bars slightly in an attempt to survive. So... not only do you now have players not using their normal loadouts & abilities, you also _technically_ cannot mathematically calculate whether this improvement is due to fewer executions of the abilities in question or fewer executions of abilities in general.

Conclusion:
A 3-second global CD is definitely not a viable solution. Though there may be other options in the gray area in between, still.


Suggestions:
Note: mostly Templar specific because that is my expertise. I will not speak to other classes as I am not as proficient & experienced as others with them.

1. Recommend considering a quick patch IMMEDIATELY to lower that AOE CD from three seconds to TWO. If you can find the "break point" where this CD truly affects performance, you will have useful information.
2. Though I do not believe a global CD like this is the solution, Templars could live again with the following changes if this went live:
3. • Change jabs to single-target so they are not on the CD
   • Remove true single-target abilities from this CD. Examples off the top of my head are Honor the Dead, Rapid Regeneration (5s HOT from Resto line)
   • Rework Templar's Rune Focus ability and its morphs. This ability has no reason to be a ground effect anymore. Change them all to be self-only "holy armor" buff akin to stuff like DK armor. Keep the current effects & morphs... just remove it from CD list and make it a buff rather than a ground effect.
4. Suggest separating "damage" AOEs and "healing" AOEs

I believe item #1 is of utmost import. You have a unique opportunity here to test and it seems wise to understand where game performance drops off. Is it at a 2-second CD or 1.5-second CD or 1-second CD? Again, I do not believe a fully global CD is the answer to your problems. But surely there is a happier medium here where the CD is shorter AND the desyncs & "lag" are mitigated.


And now for my rant. Because I deserve it and, even though I know this will fall on deaf ears, it still feels good to get it out.
Opinions:
Templar now has a global CD that hits (quite literally) 7 out of 10 abilities across both of my "normal" bars. I may be self-indulgent here, but it sure feels like Magicka Templars took the biggest hit (as many predicted). It really is unplayable... you built a class with the intention that they would depend on their ability to heal and you ignorantly put this CD on abilities that aren't even AOEs (see: Rapid Regen, Honor the Dead, and even Channeled Focus). Y'all really didn't think this through. Isn't it about time you start listening to your playerbase? You were warned in here about this but you still act like we're idiots. The truth is: you're not actually good at development. You act like a small company of cowboy devs with no large project experience.

When are you going to learn that you have a MASSIVE pool of real-life skills to pull from with us? I am an applications developer with 25 years under my belt in IT. I have made posts in here offering help that have been locked by mods who have no idea what they're reading because they're barely tier 1 support level in their careers.
I have met others in-game who are better than I am at the job (and I'm an egomaniac!). WE CAN HELP if you would ...just... let us. But you have to be capable of putting aside your pride and be willing to listen. Your code is crap. It's been crap for years. You NEED to spin up a separate set of server environments and assign a team to go back to the drawing board & optimize from the base up.

I can tell you just from playing that your multi-threading is done at too high a level - possibly even middle tier. I'm betting you also have overloaded functions that are no longer necessary but still fire. And probably overloaded functions where the original isn't even necessary anymore.

In the game's current state, I will not be playing Templar to test any further unless you make changes to current test. I do not pay you every month so that I can log into the game I love and get frustrated because you've removed the Templar's ability to heal. Cannot even self heal with single-target abilities.

Seriously, ZOS. Implement a true SDLC. Start acting like development professionals. QA/QC your crap and think more thoroughly before you implement - yes, even "tests" like this. I mean, c'mon... y'all are TESTING IN PROD! How the mighty have fallen.

Well thought out and on point.....just a couple of things to add, the validity of a test that is a response to long standing issues where players say "I cant cast skills that will win the fight or save me in the middle of one due to lag delays in skills" doesn't seem to be logical at the onset. Placing a cool-down forcing a delay in skills that will win or lose the fight to be imposed by them on purpose- is not addressing the problem players complained of.

It becomes 'we delayed this by design' and not we addressed the issues and are going to try a "fix" for them. To make it more clear, the results of this test I would submit could be 100% reproduced one week from now with the exact same results by simply blocking 70% of the players from logging in for a week. Its the same concept- they are not here to use skills, so few can use skills the lag will be better. Its not something that solves the problem----

Furthermore the staff at ZOS apparently sold the game to players and use a PTS server for testing....however over time the players have stopped offering input to them from those tests and even stopped testing due to the response (ignored) of all the hard work and effort the players put in.......so now they force compliance by introducing testing to the live environment, against the will of the players all to compensate for YEARS of ignoring them when they offered input over issues on PTS.

How very dictatorial that is of an environment----you will do what we want or we will simply beta test the game in the live area----because we can force you to comply. If the history was not so bad with choices they made to ignore the players for years this would not be done....but it wasn't the players that forced this with their choices, it was ZOS by ignoring and again they try to 'fix' it by 'forcing' us to participate in it. Either that or do not play the game at all.

Last thing I would add is the blanket banning, warnings and deleting of posts that don't 100% bandwagon or support decisions made is tactic used in certain countries to silence dissent- its never used to encourage problem solving and fixing things, its used to bury them. I fully expect one if not several mods to either warn, delete or ban for this post with some of the newest over used 'bashing' labels......I would also suggest this is done to condition us to accept what they do without question, much like our current testing being forced on paying customers who over time have been ignored by ZOS on the PTS to the point they wont test now----they simply force it all under the guise of solving a problem---that NONE of the players wanted.

Dont ask questions or not support faulty reasoning in the testing----or else.

IAmIcehouse

This is a weird perspective for an applications developer.

Let me preface this with saying,I exclusively play Cyrodil and I am having a horrible time in cyrodil the last two days. I am playing an alternative class because my current class and build just does not work during these tests.

But we know these aren't solutions, right? From what I see:

Week One is testing how much AoE actually impacts the server.

Week Two is testing, is it caused by single ability spam or various.

The last two (IDGAF about 5/6) weeks help them see if sustain is a potential lever to pull.

These tests aren't solutions but to see where they need to focus.

Also, let's stop pretending that "ZOS is going to think the tests were a success because the servers were empty". Do you seriously think population going to be overlooked? This isn't some guy in his basement making changes randomly, obviously they will compare their data at similar populations. I love how this community think they are the ultimate game developers and data scientists.

---

Now for my two cents: even if we are poplocked, survivability is destroyed during this first week of tests. This means there are fewer abilities you can cast. Both AOEs and single target--because you are dead. You heal less, shield less, kite less.

From lasting minutes on minutes of kiting and playing around with zerg backlines, during siege and dipping back to inner when the wall drops, changed getting instagibbed because we can't sustain and kite as effectively.

Now, I'm still somewhere on the server contribution to the poplocked server, but I am not fighting. I'm respawned at a different keep. Yes, during these tests, I have changed my playstyle to NOT go into backlines and kite, but to sit in the keep and wait to try to ult dump... But again, you have fewer abilities being casted, because there is less fighting going on during siege.

From a playability standpoint these tests are unbearable. I'm obviously in cyro less than normally, though I will still try to play my part.

Kungfu

It saddens me, but I have to amend my OP.

Kungfu wrote: »

Current test: 3-second shared cooldowns on all AOE abilities

Objective Analysis:
AOE CD appears to have an effect on current issues. **BUT** it may be anecdotal.

Played tonight for multiple hours during & after primetime. This time, I've been on my stamblade because I ... just ... cannot ... stomach playing my magplar. That said, I have to report that I *did* experience abilities failing to go off and, though I did not desync myself (I have no charge or teleport on my bar with which to do so), I did see an enemy player desync himself once when he charged me.

All of the negative effects were experienced during large-scale battles or at sites where a large-scale battle had JUST died down.

@ZOS_RichLambert I realize you all have all of the information but, again: I would rather say something you know than fail to say something you needed to hear. It really *feels* like the game performs best immediately after patch or a restart or whatever and then degrades over time.

It really just feels like a trash collection issue, even a single bit value being left behind one at a time every billionth execution or something... or fragmented files or files still "in use". I dunno... spitballing here. But that what it "feels" like from this hardcore player's end.

If that is correct, I would assume every player will be experiencing the same old negative effects even in small-scale encounters within a day or two.

Kungfu

Edited to remove a little off-the-cuff frustration pointed at ZOS and to better format these thoughts.

IAmIcehouse wrote: »

This is a weird perspective for an applications developer.

Yeah, not really. But thanks for your input.
I mean, there's a reason I made it clear that the opinions I wrote were just that. The rest was as objective as I can get.

If you so choose to read my rebuttal, please understand (due to lack of voice inflection) it is *not* written in a negative tone.

We've degraded to the point of testing in PROD. That alone is enough to really... really degrade confidence in the team's abilities / expertise level and cause apprehension/frustration for any developer. And, unfortunately, once that customer confidence is lost, it's been my experience that you never REALLY get it back to original levels.

So, there are plenty of tools out there that can be leveraged to automate impersonation of accounts and load test servers up with activity they could have simply collected from prod over the years. Not saying it would be easy.

But, since this has gone on for years, let's extrapolate that out a bit. Even if they only have 3 developers on this team, that's 6,000+ hours per year. So... (and I think I'm being generous here) at 3 years of this performance degradation, one could suggest that a portion of that 18,000 man-hours could easily have been applied to building an environment in which these things could have been tested through data mining and automated testing. Or... ORRrrr that project could be taken on now. Like immediately.

If I were planning this with my team: one of my server admins and three of my developers would have the new VMs & environments built in a month. All code updated & tested in the following month to make sure it matched PROD. Three months to implement automation tools. Six to populate them with all the data and enough different versions of character types & builds to replicate, using data that I would expect another team to gather in the meantime... In one year with only 3 real devs involved, we would have automated testing running 24/7 while we built reporting tools and had server data collecting to tell us EXACTLY where the point(s) of failure is/are.

Furthermore, these tests really won't tell them enough about what they need to focus on. It will be six weeks of "are AOE effects related". They are REALLY hyper-focused on this being the root cause. That's great if it is! But with so many variables in this size of a system, they need to remove their emotional attachment to their list of "favorite" causes and get pragmatic about it again. You have to let the systems tell you what's wrong but you can't hear it if you're not listening to all aspects. A "can't see the forest for the trees" kinda thing.

The group size tests won't supply any meaningful value because ball groups are simply going to run multiple 12-man groups, for example. But two 12-man groups running side by side only rules out a VERY small portion of set effects and AOEs in the game that are "group member only" type effects. It would be a simpler solution to just list out all of those available effects, compare that list with what is currently / commonly run in Cyro, and focus in on those very particular abilities / effects / etc.


So...
No, I do not feel responsible to test anything for these people. My time is worth money... and not a small amount of it. I pay them for the opportunity to play this. And I will continue to help in whatever way I choose. But I'm certainly not gonna frustrate myself over a game because [editing out a little frustration]. I am *still* the customer in this situation; not the consultant.

RDMyers65b14_ESO

Well, from my perspective, the cool downs have not changed the lag at all. In fact, the performance was worse. (Mind you, our group of 12 was running into a zerg of 60.) This is not the answer to the lag problem. I man a stamden support build and the healers couldn't do their job because of the cool downs. The biggest problem was purges not being able to go off.

Joy_Division

I dont think the AoE test is to see if lag is reduced (i.e., it's not a potential solution), rather for Zos to ascertain if the server load functions differently if players were forced (or upcoming tests, incentivized) to mostly used single target abilities.

hollywood

Second round of testing didn't start off very well for me. Noticeable increase in latency from last week , skills not working. The same pop as on the first day of testing, PC NA Grey Host.

I did notice more guilds running around since now you can actually heal/purge stuff. Let's see how it is tomorrow when the IC event ends and everyone is back at Cyrodill.

Kaysha

PC EU NOCP, pop locked: the lag is back

guilds are back, ball groups are back, skills take 5 seconds to fire. the fun is gone.

katorga

So what is consensus from first week of tests....did it solve performance issues or no?

hollywood

First week for PC NA only the first day had enough pop for testing purposes , and the lag was better. The other days DC was mostly MIA for Grey Host so there wasn't much lag.