Why is improving the latency issue in GH not possible, and what is root cause of the issue?

Lagzee · January 6

Is it the server technology is too old? Is it a change you guys made that caused it and cannot be reversed? What is it? I have so many unanswered questions regarding the performance issues in this game. Its crazy to think, roughly 5 years into the game we got the year of performance, and then 5 years later we get "this is not possible". But here we are.

We recognize that some of our players would prefer there be no changes to their characters, effectively how they are in the Gray Host Cyrodiil campaign, while enjoying the higher population and reduced latency/game performance issues of the Vengeance campaign. This is not something that will be possible. Based on what we have learned from the tests so far, we can offer one or the other, not both, and we want to be transparent about that.

Im a bit confused on how we got here, how this is acceptable to the good people at zos to just throw up your hands and admit defeat, why the claim is that nothing can be done to improve things, or at least try to combat the issue a bit, and what the root cause of the issue even is.

I remember telling many of my friends, back when vengeance was announced, that my biggest concern was that this would be zos' excuse to give up on performance. Not that they would replace regular cyrodiil with it, because that would lose 90% of pvpers, but that they would simply say "well if you dont want to lag, go to vengeance". And here we are.

Everyone knows gray host is unplayable for like 3-6 hours a night. Or basically any time it has a que. So im not going to harp on it, other than to say its the worst ive ever seen it in 10 years, and its not even close. Just utterly unplayable during that time. And its consistently the worst ive seen it, meaning its consistently unplayable for hours, until enough players leave.

So again, how did we get here? Why wasnt this issue taken more seriously before it got to this point? And why have you given up on improving the latency issue? And yes, to say "this is not possible" sounds like you've given up. Clearly something happened to get us to this point, and clearly measures could be taken to get us back to at least where we were a year or two ago. Even if you wont/cant fix the core of the issue.

The most important question here is, what is the root cause of this issue? Did you guys ever get to the bottom of it? Clearly whatever the root cause is, the price tag, or work, to fix it has been determined to be too much. Or at least thats what i have to assume. Even though a well performing, rewarding, game would only bring in more players, but thats another topic. But afaik, we still havent gotten a clear answer on what the root of the issue is, that causes latency issues like this, and why it has been deemed unfixable.

And finally my last questions are, first, why will you guys not address the ball group issue? and by the same token, the heal stacking issue? Yet another issue that has only been getting worse over the years, and added to the performance problems. Which leads me to my second question, why not try and make smaller adjustments to combat the performance problems? Not everything has to be massive sweeping changes. Adjust heal stacking. Remove/adjust cross healing out of groups. Adjust certain* problem sets, or disable them. Maybe even some kind of weekly server refresh idk. These things could all make small impacts that turn into a bigger impact.

Yes, some things have been done before, but its like you just gave up on the small adjustments that were tried in the past. A problem like this, that only gets worse year after year, needs multiple small changes to add up and make some kind of difference. But when you say changing/removing something didnt have enough of an impact on performance, to keep the change, then it will never get better. 10 small changes that alone didnt make a big difference could, together, turn into a big difference.

SolarRune · January 6

I would agree its really important we get clear official written answers that can be referred to in the future to stop cycling around the same arguments/discussions. Maybe with the new management committed to more openness and transparency we will get this level of clarity.

My interpretarion of events goes back to what we saw from the vengeance 1 feedback, in my opinion what this clearly showed was that the computation load was the underlying cause of the issues, this would also make sense with the original game design having more client based calculations in the old days that ended up being moved to server side. The only time vengeance 1 was remotely laggy was the massive stalemate battles and as was shown in the feedback this was the only time the computation load got near the levels of GH. (That if I recall was with population 4 or 5 times higher than GH). Computational load being the underlying issue, also explains why as features were turned on more people experienced lag on vengeance at lower population levels.

So if my interpretation of what has been presented so far is correct, and the problem is computational load, then that's a pretty big issue, how do you limit healing stacking or add/remove functionality through battle spirit without increasing computational load?

You mention disabling certain sets, that would be potential way to clearly reduce calculations, but given the response from people with vengeance not having sets I can see why ZOS may not want another backlash in that space (I know removing a handful of sets is different to no sets at all, but just saying ZOS are in a no win situation on this element imo).

xylena · January 6

Lagzee wrote: »

but its like you just gave up on the small adjustments that were tried in the past

They tried this for over a decade without success. Sunk cost fallacy to think there's a silver bullet right around the corner. Vengeance is their solution to performance and mostly fixes it.

MincMincMinc · January 6

Sadly its the result of power creep. People are too happy having their milk and cookies too. Take any example of bad changes and half the population will revolt if you mention removing it. The top brass at zos are going to want a simple lightswitch solution, which clearly didnt exist in the current vengeance testing. Which indicates the entirety of the game design combat wise has just ballooned out of control. They are going to see it as having to reverse 10 years of un-moderated releases. Thus they are going with the cop out solution of cutting cyrodil pop again, except now its small enough they need to finally make a new map.

The real question is whether performance is going down linearly or exponential. For instance right now we have two datapoints. Veng at 900 players and GH at 300 players.......... If we picked a game design half way between, how many players would it support? a linear 600? or perhaps its more exponential and we may be able to get 700 or 800?

Lagzee · January 6

xylena wrote: »

Lagzee wrote: »

but its like you just gave up on the small adjustments that were tried in the past

They tried this for over a decade without success. Sunk cost fallacy to think there's a silver bullet right around the corner. Vengeance is their solution to performance and mostly fixes it.

I disagree. And thats exactly the point im making. They tried many things, individually, but they never tried multiple things at once. They would try something and say it didnt make enough of an impact. Which is a backwards way of thinking when it comes to an issue like this.

You need to do make multiple small changes at the same time, not just try one thing, deem it not impactful enough, revert it, then try something else a year later, and repeat. Idk why you're bringing up a silver bullet, why would anyone think that would happen after 10 years of this. Im just saying they need to make some small changes to combat how bad its become. It will never be perfect, even vengeance lags.

And vengeance is not a solution to the problems in gray host. its just something completely different.

Lagzee · January 7

SolarRune wrote: »

I would agree its really important we get clear official written answers that can be referred to in the future to stop cycling around the same arguments/discussions. Maybe with the new management committed to more openness and transparency we will get this level of clarity.

My interpretarion of events goes back to what we saw from the vengeance 1 feedback, in my opinion what this clearly showed was that the computation load was the underlying cause of the issues, this would also make sense with the original game design having more client based calculations in the old days that ended up being moved to server side. The only time vengeance 1 was remotely laggy was the massive stalemate battles and as was shown in the feedback this was the only time the computation load got near the levels of GH. (That if I recall was with population 4 or 5 times higher than GH). Computational load being the underlying issue, also explains why as features were turned on more people experienced lag on vengeance at lower population levels.

So if my interpretation of what has been presented so far is correct, and the problem is computational load, then that's a pretty big issue, how do you limit healing stacking or add/remove functionality through battle spirit without increasing computational load?

You mention disabling certain sets, that would be potential way to clearly reduce calculations, but given the response from people with vengeance not having sets I can see why ZOS may not want another backlash in that space (I know removing a handful of sets is different to no sets at all, but just saying ZOS are in a no win situation on this element imo).

Ya maybe this is the reason. But then the question is why isnt it reversable, or something that can be improved or changed. And ya some people will always complain but if changes helped performance more people would be happy. Removing sets would probably be the biggest issue for people, but there are other things that could be done. Like disabling cross healing out of groups, sure the people that randomly go around healing zergs, that dont want to group, wouldnt like it, but how many people actually fall into that category?

And the heal stacking issue, and how many people can actually get healed by certain skills, is deff a problem. But as for how they can remove/adjust these things through battle spirit without increasing computational load, im not sure but i would imagine the benefit would out weigh the cost. And thats if it even works that way. Like maybe the load of battle spirit, spread out through the entirety of gray host, works way less than the load of 2 ball groups fighting at a keep spamming a million heals for a half hour.

turning off addons is another test they could do. Like i wonder how much impact addons have on performance.

You can literally feel the complete degradation of performance across the entire map when there is a massive fight, with multiple ball groups. Performance goes from bad to unplayable. Like at this point i would take bad performance over unplayable performance, which is what were getting every night in gray host. And honestly its mind blowing to say that about a game like this, made by a company like zenimax, backed by microsoft, with millions of dollars behind it, but this is where we are at this point.

Ive said it time and time again over the years, i refuse to believe a team like this cannot get to the bottom of this issue and fix it. Or at the very least give us the exact issue, and reason why its impossible to fix. I think at the very least we deserve that much after all this time, and so many broken promises.

Lagzee · January 7

MincMincMinc wrote: »

Sadly its the result of power creep. People are too happy having their milk and cookies too. Take any example of bad changes and half the population will revolt if you mention removing it. The top brass at zos are going to want a simple lightswitch solution, which clearly didnt exist in the current vengeance testing. Which indicates the entirety of the game design combat wise has just ballooned out of control. They are going to see it as having to reverse 10 years of un-moderated releases. Thus they are going with the cop out solution of cutting cyrodil pop again, except now its small enough they need to finally make a new map.

The real question is whether performance is going down linearly or exponential. For instance right now we have two datapoints. Veng at 900 players and GH at 300 players.......... If we picked a game design half way between, how many players would it support? a linear 600? or perhaps its more exponential and we may be able to get 700 or 800?

It is indeed the result of power creep but performance has been steadily getting worse since like 2016. It lagged in haderus, it lagged vivec, and it lags in gray host. The main campaign has always lagged, and now they have chased away so many pvpers that there is no other campaigns with population. You're lucky to even see it on the weekend at prime time. And how did they chase those players away? With bad rewards, terrible changes, and abysmal performance. For a decade.

And if they cut cyrodiil pop again it will only chase away more people. Its just a disaster. They just need to make some small adjustments to the critical things impacting performance. But ofc these are all band aids, these things will never fix the issue. A year ago i would have said they need to get to the root of the issue and fix it, but now they have said they wont do that and have basically given up. So now all we can hope for is band aid fixes to improve things by even a small amount.

Im sorry but to just let it get worse and worse forever, without trying to do anything, is just unforgivable. And thats what they have done here for the last decade. And thats also why ive always said i will never play another title from zos, because of the way they have handled cyrodiil. Not that we need to worry about that anymore.

My only hope for cyrodiil at this point is crossplay. At least then we can get some more campaigns, and actually have population in them. Without having it completely full, with a half hr que, like gray host. And then performance will be a bit better. But thats got a lot of IFs behind it. If crossplay actually comes. If pvp is included. And it doesnt cause more issues.

Muizer · January 7

MincMincMinc wrote: »

Veng at 900 players and GH at 300 players.......... If we picked a game design half way between, how many players would it support? a linear 600? or perhaps its more exponential and we may be able to get 700 or 800?

Now, what I'm going to say has to come with a huge disclaimer. I don't know the game's architecture any better than anyone else here. But I don't think this is the 'best guess'.

In terms of computational load, the best guess is that population is most likely to have an exponential impact (range checks) whereas the impact of ability/set complexity will be more linear. Doubling population will require more than halving the ability/set complexity to compensate. I'd say the population at a 'halfway' scenario in terms of ability/set complexity will accommodate less, not more, than double the population. Basically, closer toward the 300 end than toward the 900 end.

xylena · January 7

Lagzee wrote: »

You need to do make multiple small changes at the same time

Yeah they've been doing plenty of that for over a decade too, and it's extra frustrating because it's like, the opposite of science. Now nobody can tell what actually changed what.

Any given problem with GH is really dozens of other problems in a trench coat. There is no single root cause to heal stacking, one shotting, etc.

SaffronCitrusflower · January 7

Lagzee wrote: »

SolarRune wrote: »

I would agree its really important we get clear official written answers that can be referred to in the future to stop cycling around the same arguments/discussions. Maybe with the new management committed to more openness and transparency we will get this level of clarity.

My interpretarion of events goes back to what we saw from the vengeance 1 feedback, in my opinion what this clearly showed was that the computation load was the underlying cause of the issues, this would also make sense with the original game design having more client based calculations in the old days that ended up being moved to server side. The only time vengeance 1 was remotely laggy was the massive stalemate battles and as was shown in the feedback this was the only time the computation load got near the levels of GH. (That if I recall was with population 4 or 5 times higher than GH). Computational load being the underlying issue, also explains why as features were turned on more people experienced lag on vengeance at lower population levels.

So if my interpretation of what has been presented so far is correct, and the problem is computational load, then that's a pretty big issue, how do you limit healing stacking or add/remove functionality through battle spirit without increasing computational load?

You mention disabling certain sets, that would be potential way to clearly reduce calculations, but given the response from people with vengeance not having sets I can see why ZOS may not want another backlash in that space (I know removing a handful of sets is different to no sets at all, but just saying ZOS are in a no win situation on this element imo).

Ya maybe this is the reason. But then the question is why isnt it reversable, or something that can be improved or changed. And ya some people will always complain but if changes helped performance more people would be happy. Removing sets would probably be the biggest issue for people, but there are other things that could be done. Like disabling cross healing out of groups, sure the people that randomly go around healing zergs, that dont want to group, wouldnt like it, but how many people actually fall into that category?

And the heal stacking issue, and how many people can actually get healed by certain skills, is deff a problem. But as for how they can remove/adjust these things through battle spirit without increasing computational load, im not sure but i would imagine the benefit would out weigh the cost. And thats if it even works that way. Like maybe the load of battle spirit, spread out through the entirety of gray host, works way less than the load of 2 ball groups fighting at a keep spamming a million heals for a half hour.

turning off addons is another test they could do. Like i wonder how much impact addons have on performance.

You can literally feel the complete degradation of performance across the entire map when there is a massive fight, with multiple ball groups. Performance goes from bad to unplayable. Like at this point i would take bad performance over unplayable performance, which is what were getting every night in gray host. And honestly its mind blowing to say that about a game like this, made by a company like zenimax, backed by microsoft, with millions of dollars behind it, but this is where we are at this point.

Ive said it time and time again over the years, i refuse to believe a team like this cannot get to the bottom of this issue and fix it. Or at the very least give us the exact issue, and reason why its impossible to fix. I think at the very least we deserve that much after all this time, and so many broken promises.

If they can't fix GH then they don't have what it takes to build a whole new system up from scratch either. That's how we know vengeance will fail at least as bad or worse than GH is failing now.

Dalinar4 · January 7

1000% agree with the person who mentioned turning add-ons off. As far as I can tell it is the one thing they haven't tested. At the very least: run a test. Three days with no add-ons in Cyro, or even across the game. Get some hard data points.

imPDA · January 7

Lagzee wrote: »

Is it the server technology is too old? Is it a change you guys made that caused it and cannot be reversed? What is it? I have so many unanswered questions regarding the performance issues in this game. Its crazy to think, roughly 5 years into the game we got the year of performance, and then 5 years later we get "this is not possible". But here we are.

Long long ago a lot of computations were moved to server from client to prevent cheating. Since then, number of calculations drastically increased. It was compensated by reducing number of players per campaign (but never announced), but now it is impossible to make it even lower because it would be very obvious and sad looking.

Lagzee wrote: »

The most important question here is, what is the root cause of this issue? Did you guys ever get to the bottom of it? Clearly whatever the root cause is, the price tag, or work, to fix it has been determined to be too much. Or at least thats what i have to assume. Even though a well performing, rewarding, game would only bring in more players, but thats another topic. But afaik, we still havent gotten a clear answer on what the root of the issue is, that causes latency issues like this, and why it has been deemed unfixable.

I think that increased number of buffs made it this bad. I remember myself theorycrafting about one buffs vs others, how can I make a build with some unique buff and what will I lose if I will play with it. Nowadays it is just a game to collect all buffs, minor, major resolve, evasion, prophecy, berserk, sorcery, brutality, expedition, etc. And every class has access (especially after subclassing) to (almost) all buffs, so you can have them all. Each buff should be used in calculations each time you cast something or receive damage. Also, there are more damage sources now, which can land in one GCD, so number of calculations per second is huge and is not even which leads to stutters, freezes (on server side) and then to lag spikes on client side. It is my POV, not proven, but I think the fact ZOs made SO simple skills for vengeance and limited number of targets to 3 speaks by itself.

They can do nothing with it because in current paradigm each class should have even and balanced skills, so HUGE amount of buffs were added to all skills to make them somewhat equal and now you can just swap one damage or healing skill with another, you will not probably feel big difference.

MincMincMinc · January 8

Muizer wrote: »

MincMincMinc wrote: »

Veng at 900 players and GH at 300 players.......... If we picked a game design half way between, how many players would it support? a linear 600? or perhaps its more exponential and we may be able to get 700 or 800?

Now, what I'm going to say has to come with a huge disclaimer. I don't know the game's architecture any better than anyone else here. But I don't think this is the 'best guess'.

In terms of computational load, the best guess is that population is most likely to have an exponential impact (range checks) whereas the impact of ability/set complexity will be more linear. Doubling population will require more than halving the ability/set complexity to compensate. I'd say the population at a 'halfway' scenario in terms of ability/set complexity will accommodate less, not more, than double the population. Basically, closer toward the 300 end than toward the 900 end.

I think its more of just cascading multiplicative factors. Players are the obvious largest factor in the equation, but not really once you see how many things can interact and proc each other.

100x players x 4 proc sets x 10 status effects x 10cp effects x 4 proc passives x racial procs x armor procs x A skill that does 4 effects x timers.

We did no proc pvp and it didnt solve the problem......well sure you remove the 4x proc sets and surprise the number of calcs is still insane with all the other systems. Too many procs proccing procs kinda situation. One pebble is thrown and then a whole mountain of rocks starts rolling down the hill.

A good expedited way to understand what is wrong with the way ESO has power creeped is to look at a game like Risk of Rain. Initially you start off with no items and the game is flawless. However an hour in with 80 item effects proccing and any computer starts to lag. The only counter to this is to design systems in a way to avoid unnecessary calcs or end them when not necessary or simply never make them in the first place. Maybe we dont need 4 paragraph long skill tooltips?

Sluggy · January 9

MincMincMinc wrote: »

Muizer wrote: »

MincMincMinc wrote: »

Veng at 900 players and GH at 300 players.......... If we picked a game design half way between, how many players would it support? a linear 600? or perhaps its more exponential and we may be able to get 700 or 800?

Now, what I'm going to say has to come with a huge disclaimer. I don't know the game's architecture any better than anyone else here. But I don't think this is the 'best guess'.

In terms of computational load, the best guess is that population is most likely to have an exponential impact (range checks) whereas the impact of ability/set complexity will be more linear. Doubling population will require more than halving the ability/set complexity to compensate. I'd say the population at a 'halfway' scenario in terms of ability/set complexity will accommodate less, not more, than double the population. Basically, closer toward the 300 end than toward the 900 end.

I think its more of just cascading multiplicative factors. Players are the obvious largest factor in the equation, but not really once you see how many things can interact and proc each other.

100x players x 4 proc sets x 10 status effects x 10cp effects x 4 proc passives x racial procs x armor procs x A skill that does 4 effects x timers.

We did no proc pvp and it didnt solve the problem......well sure you remove the 4x proc sets and surprise the number of calcs is still insane with all the other systems. Too many procs proccing procs kinda situation. One pebble is thrown and then a whole mountain of rocks starts rolling down the hill.

A good expedited way to understand what is wrong with the way ESO has power creeped is to look at a game like Risk of Rain. Initially you start off with no items and the game is flawless. However an hour in with 80 item effects proccing and any computer starts to lag. The only counter to this is to design systems in a way to avoid unnecessary calcs or end them when not necessary or simply never make them in the first place. Maybe we dont need 4 paragraph long skill tooltips?

At the end of the day it's an issue of architecture and technology (likely scripting languages rather than native code) that is causing performance issues. Computers today are insanely, stupidly, inconceivably, powerful devices. If designed in a way that focuses purely on making use of that, there is almost no practical limit to the number of players they could support on the server side as the bottleneck would become one purely of how to actually network all of that data.

But many programs are designed in a way to make the development process easier for the devs and provide more flexibility for extension and expansion of ideas. The problem is that this usually involves architecturing things in a way that is significantly less friendly for the computer to process.

Taarente · January 9

The core issue with Grey Host isn’t one single system or “too many procs” by itself — it’s how combat in Cyrodiil is resolved when lots of players are close together.

Every skill use in ESO isn’t just “do damage”. It’s a chain of conditional checks:

If this hits, then check resistances.
If it crits, check mitigation.
If it applies a status effect, check immunities.
If that effect triggers something else, check cooldowns.
If the target has protections, shields, CP passives, or Battle Spirit modifiers, factor those in too.

None of those checks are especially expensive on their own. The problem is that many of them depend on the current state of other players, not just the person pressing the button.

When player density is low, the server can resolve these chains quickly.
When you have dozens of players in the same space — all changing state at the same time — the server has to wait until it knows the outcome of multiple “if-then” paths before it can commit results.

That’s why the symptoms aren’t “low FPS” but things like:
• Skills firing late
• Bar swaps not registering
• Damage and heals landing after a delay
• Players appearing to die after they’ve already moved

Removing proc sets (or running Vengeance) helps because it reduces the number of conditional branches, not because procs are “bad” by themselves. But it doesn’t eliminate the problem, because the underlying combat system is still highly state-dependent.

This is also why lots of small changes can be hard to notice. Each one may reduce server work slightly, but unless enough conditional paths are removed at once, players won’t feel a clear improvement — only the loss of whatever was changed.

In short: Grey Host struggles at peak times because too many interdependent combat decisions have to be resolved at once, not because the servers are weak or because one system is broken.

MincMincMinc · January 9

Sluggy wrote: »

MincMincMinc wrote: »

Muizer wrote: »

MincMincMinc wrote: »

Veng at 900 players and GH at 300 players.......... If we picked a game design half way between, how many players would it support? a linear 600? or perhaps its more exponential and we may be able to get 700 or 800?

Now, what I'm going to say has to come with a huge disclaimer. I don't know the game's architecture any better than anyone else here. But I don't think this is the 'best guess'.

In terms of computational load, the best guess is that population is most likely to have an exponential impact (range checks) whereas the impact of ability/set complexity will be more linear. Doubling population will require more than halving the ability/set complexity to compensate. I'd say the population at a 'halfway' scenario in terms of ability/set complexity will accommodate less, not more, than double the population. Basically, closer toward the 300 end than toward the 900 end.

I think its more of just cascading multiplicative factors. Players are the obvious largest factor in the equation, but not really once you see how many things can interact and proc each other.

100x players x 4 proc sets x 10 status effects x 10cp effects x 4 proc passives x racial procs x armor procs x A skill that does 4 effects x timers.

We did no proc pvp and it didnt solve the problem......well sure you remove the 4x proc sets and surprise the number of calcs is still insane with all the other systems. Too many procs proccing procs kinda situation. One pebble is thrown and then a whole mountain of rocks starts rolling down the hill.

A good expedited way to understand what is wrong with the way ESO has power creeped is to look at a game like Risk of Rain. Initially you start off with no items and the game is flawless. However an hour in with 80 item effects proccing and any computer starts to lag. The only counter to this is to design systems in a way to avoid unnecessary calcs or end them when not necessary or simply never make them in the first place. Maybe we dont need 4 paragraph long skill tooltips?

At the end of the day it's an issue of architecture and technology (likely scripting languages rather than native code) that is causing performance issues. Computers today are insanely, stupidly, inconceivably, powerful devices. If designed in a way that focuses purely on making use of that, there is almost no practical limit to the number of players they could support on the server side as the bottleneck would become one purely of how to actually network all of that data.

But many programs are designed in a way to make the development process easier for the devs and provide more flexibility for extension and expansion of ideas. The problem is that this usually involves architecturing things in a way that is significantly less friendly for the computer to process.

Yeah, but that doesnt necessarily excuse whats happening. We know that we can do 900(3x live) players no issue and even going up to 1200(4x) on the first day of testing was possible where we just began to see server issues. To match lives server issues we would probably have to push the server to 1500 or 1800 like zos originally quoted back in 2014.

The only thing standing between the current 300 and 1800 is the game rules and design. Which is why I think it would have been smart to go halfway between vengeance and live's bloated game with the bare essentials in performative ways. Instead did we really need to test if the achievement system worked in cyrodil? Do we really need momentos? These were all tested and could have been completely ignored. They could be completely disabled on live and I doubt anyone would really care.

Sluggy · January 9

MincMincMinc wrote: »

Sluggy wrote: »

MincMincMinc wrote: »

Muizer wrote: »

MincMincMinc wrote: »

Veng at 900 players and GH at 300 players.......... If we picked a game design half way between, how many players would it support? a linear 600? or perhaps its more exponential and we may be able to get 700 or 800?

Now, what I'm going to say has to come with a huge disclaimer. I don't know the game's architecture any better than anyone else here. But I don't think this is the 'best guess'.

In terms of computational load, the best guess is that population is most likely to have an exponential impact (range checks) whereas the impact of ability/set complexity will be more linear. Doubling population will require more than halving the ability/set complexity to compensate. I'd say the population at a 'halfway' scenario in terms of ability/set complexity will accommodate less, not more, than double the population. Basically, closer toward the 300 end than toward the 900 end.

I think its more of just cascading multiplicative factors. Players are the obvious largest factor in the equation, but not really once you see how many things can interact and proc each other.

100x players x 4 proc sets x 10 status effects x 10cp effects x 4 proc passives x racial procs x armor procs x A skill that does 4 effects x timers.

We did no proc pvp and it didnt solve the problem......well sure you remove the 4x proc sets and surprise the number of calcs is still insane with all the other systems. Too many procs proccing procs kinda situation. One pebble is thrown and then a whole mountain of rocks starts rolling down the hill.

A good expedited way to understand what is wrong with the way ESO has power creeped is to look at a game like Risk of Rain. Initially you start off with no items and the game is flawless. However an hour in with 80 item effects proccing and any computer starts to lag. The only counter to this is to design systems in a way to avoid unnecessary calcs or end them when not necessary or simply never make them in the first place. Maybe we dont need 4 paragraph long skill tooltips?

At the end of the day it's an issue of architecture and technology (likely scripting languages rather than native code) that is causing performance issues. Computers today are insanely, stupidly, inconceivably, powerful devices. If designed in a way that focuses purely on making use of that, there is almost no practical limit to the number of players they could support on the server side as the bottleneck would become one purely of how to actually network all of that data.

But many programs are designed in a way to make the development process easier for the devs and provide more flexibility for extension and expansion of ideas. The problem is that this usually involves architecturing things in a way that is significantly less friendly for the computer to process.

Yeah, but that doesnt necessarily excuse whats happening. We know that we can do 900(3x live) players no issue and even going up to 1200(4x) on the first day of testing was possible where we just began to see server issues. To match lives server issues we would probably have to push the server to 1500 or 1800 like zos originally quoted back in 2014.

The only thing standing between the current 300 and 1800 is the game rules and design. Which is why I think it would have been smart to go halfway between vengeance and live's bloated game with the bare essentials in performative ways. Instead did we really need to test if the achievement system worked in cyrodil? Do we really need momentos? These were all tested and could have been completely ignored. They could be completely disabled on live and I doubt anyone would really care.

The bloated skills and the long chain of things needed to be checked to arrive at the final number is certainly the cause of the performance problem and I'm not excusing the performance in any way. What I'm trying to say is that some people arrive at the conclusion that it's just "too many calculations" when what it really boils down to is "the system isn't designed correctly to handle those calculations in mass" or maybe it's just "too many calculations for the current system design". Years ago they stated that it wasn't a hardware issue and you know what, I believed it then and I still believe that now. I'm like, 99% certain that's what the push for the re-architecturing was about a couple years ago but at some point they probably realized it was a massive task and concluded that the sunk cost fallacy was at play.

The fact that a hardware upgrade helped was nice but I also think that it was always only a matter of time before it wasn't good enough. Vegeance has also proven that reducing the raw amount of data that needs to be accessed in combat helps massively and it appears, indeed, that building combat from the ground up using their existing technology is a path forward that minimized costs and time while maximizing the performance gains. At least when compared to trying to redesign the guts of their engine to accommodate the old system.

At the end of the day it is a solvable problem. The question is if it's worth the cost. The plan they've arrived at seems to be threading the needle.

MincMincMinc · January 9

Sluggy wrote: »

MincMincMinc wrote: »

Sluggy wrote: »

MincMincMinc wrote: »

Muizer wrote: »

MincMincMinc wrote: »

Veng at 900 players and GH at 300 players.......... If we picked a game design half way between, how many players would it support? a linear 600? or perhaps its more exponential and we may be able to get 700 or 800?

Now, what I'm going to say has to come with a huge disclaimer. I don't know the game's architecture any better than anyone else here. But I don't think this is the 'best guess'.

In terms of computational load, the best guess is that population is most likely to have an exponential impact (range checks) whereas the impact of ability/set complexity will be more linear. Doubling population will require more than halving the ability/set complexity to compensate. I'd say the population at a 'halfway' scenario in terms of ability/set complexity will accommodate less, not more, than double the population. Basically, closer toward the 300 end than toward the 900 end.

I think its more of just cascading multiplicative factors. Players are the obvious largest factor in the equation, but not really once you see how many things can interact and proc each other.

100x players x 4 proc sets x 10 status effects x 10cp effects x 4 proc passives x racial procs x armor procs x A skill that does 4 effects x timers.

We did no proc pvp and it didnt solve the problem......well sure you remove the 4x proc sets and surprise the number of calcs is still insane with all the other systems. Too many procs proccing procs kinda situation. One pebble is thrown and then a whole mountain of rocks starts rolling down the hill.

A good expedited way to understand what is wrong with the way ESO has power creeped is to look at a game like Risk of Rain. Initially you start off with no items and the game is flawless. However an hour in with 80 item effects proccing and any computer starts to lag. The only counter to this is to design systems in a way to avoid unnecessary calcs or end them when not necessary or simply never make them in the first place. Maybe we dont need 4 paragraph long skill tooltips?

At the end of the day it's an issue of architecture and technology (likely scripting languages rather than native code) that is causing performance issues. Computers today are insanely, stupidly, inconceivably, powerful devices. If designed in a way that focuses purely on making use of that, there is almost no practical limit to the number of players they could support on the server side as the bottleneck would become one purely of how to actually network all of that data.

But many programs are designed in a way to make the development process easier for the devs and provide more flexibility for extension and expansion of ideas. The problem is that this usually involves architecturing things in a way that is significantly less friendly for the computer to process.

Yeah, but that doesnt necessarily excuse whats happening. We know that we can do 900(3x live) players no issue and even going up to 1200(4x) on the first day of testing was possible where we just began to see server issues. To match lives server issues we would probably have to push the server to 1500 or 1800 like zos originally quoted back in 2014.

The only thing standing between the current 300 and 1800 is the game rules and design. Which is why I think it would have been smart to go halfway between vengeance and live's bloated game with the bare essentials in performative ways. Instead did we really need to test if the achievement system worked in cyrodil? Do we really need momentos? These were all tested and could have been completely ignored. They could be completely disabled on live and I doubt anyone would really care.

The bloated skills and the long chain of things needed to be checked to arrive at the final number is certainly the cause of the performance problem and I'm not excusing the performance in any way. What I'm trying to say is that some people arrive at the conclusion that it's just "too many calculations" when what it really boils down to is "the system isn't designed correctly to handle those calculations in mass" or maybe it's just "too many calculations for the current system design". Years ago they stated that it wasn't a hardware issue and you know what, I believed it then and I still believe that now. I'm like, 99% certain that's what the push for the re-architecturing was about a couple years ago but at some point they probably realized it was a massive task and concluded that the sunk cost fallacy was at play.

The fact that a hardware upgrade helped was nice but I also think that it was always only a matter of time before it wasn't good enough. Vegeance has also proven that reducing the raw amount of data that needs to be accessed in combat helps massively and it appears, indeed, that building combat from the ground up using their existing technology is a path forward that minimized costs and time while maximizing the performance gains. At least when compared to trying to redesign the guts of their engine to accommodate the old system.

At the end of the day it is a solvable problem. The question is if it's worth the cost. The plan they've arrived at seems to be threading the needle.

Yeah thats why when the did the hardware "upgrade" everyone thought it was going to be a silver bullet but made next to no change. However they ignore that matt had said there would be no performance increase and it was just to make maintenance easier. The old game engine that is 15+ years old is certainly not helping.

MeIina · January 9

Zenimax don't want to pay the licensing fee's for a a big name anti-cheat, so everything is calculated server side. Practically nothing is done client side which means all the stress is put on the servers.

Sluggy · January 10

MeIina wrote: »

Zenimax don't want to pay the licensing fee's for a a big name anti-cheat, so everything is calculated server side. Practically nothing is done client side which means all the stress is put on the servers.

That is the opposite of the truth. Client-side Anti-cheat is actually a budget solution to avoid having to running lots of calculations on the server or indeed, needing a server at all, in many cases. The electricity to run the servers ain't free and the more they calculate the more it costs.

xylena · January 10

MeIina wrote: »

Zenimax don't want to pay the licensing fee's for a a big name anti-cheat, so everything is calculated server side. Practically nothing is done client side which means all the stress is put on the servers.

A lot of stuff was client side at launch, enabling rampant Cheat Engine abuse. They moved the stuff server side a couple years in, which put an end to Cheat Engine, but made the lag far worse.

MeIina · January 10

xylena wrote: »

MeIina wrote: »

Zenimax don't want to pay the licensing fee's for a a big name anti-cheat, so everything is calculated server side. Practically nothing is done client side which means all the stress is put on the servers.

A lot of stuff was client side at launch, enabling rampant Cheat Engine abuse. They moved the stuff server side a couple years in, which put an end to Cheat Engine, but made the lag far worse.

If instead of moving everything server side, they just instead used a good client side kernel level anti-cheat then we wouldn't be here complaining about the performance and cheat engine wouldn't be a worry because of them using a good anti - cheat... lol

LootAllTheStuff · January 10

MeIina wrote: »

xylena wrote: »

MeIina wrote: »

Zenimax don't want to pay the licensing fee's for a a big name anti-cheat, so everything is calculated server side. Practically nothing is done client side which means all the stress is put on the servers.

A lot of stuff was client side at launch, enabling rampant Cheat Engine abuse. They moved the stuff server side a couple years in, which put an end to Cheat Engine, but made the lag far worse.

If instead of moving everything server side, they just instead used a good client side kernel level anti-cheat then we wouldn't be here complaining about the performance and cheat engine wouldn't be a worry because of them using a good anti - cheat... lol

Are there any good kernel level anti-cheat tools around? Every time anti-cheat comes up in the context of other games, I see nothing but complaints about how it degrades performance.

Taarente · January 10

MeIina wrote: »

xylena wrote: »

MeIina wrote: »

Zenimax don't want to pay the licensing fee's for a a big name anti-cheat, so everything is calculated server side. Practically nothing is done client side which means all the stress is put on the servers.

A lot of stuff was client side at launch, enabling rampant Cheat Engine abuse. They moved the stuff server side a couple years in, which put an end to Cheat Engine, but made the lag far worse.

If instead of moving everything server side, they just instead used a good client side kernel level anti-cheat then we wouldn't be here complaining about the performance and cheat engine wouldn't be a worry because of them using a good anti - cheat... lol

If what you’re saying is correct are you able to explain

Which parts of the current server workload do you think would be reduced by kernel anti-cheat?
What specific logic would move back to the client? (hit detection? buff/debuff application? proc evaluation? position authority?)
How would the server verify outcomes without re-doing the same computation (which puts you back where you started)?
What’s the failure mode when anti-cheat misses something or a new bypass appears?

thanks.

Tigor · January 10

People playing Ballgroups are the issue of all problems

soelslaev · January 10

Lagzee wrote: »

... made by a company like zenimax, backed by microsoft, with millions of dollars behind it...

Please stop perpetuating this falsehood.

When company A buys company B, the purpose of the purchase is to extract the wealth from company B and transmit that wealth to the owners of company A. That is why whatever brand name product that your dad swore by is now garbage. From Sears tractors to Gibson Les Paul guitars.

It used to be, a small group of dedicated people made a good product, and it got a good reputation; that takes 20 years. Then the original folks are tired and hand it over to the next gen. Then next gen gets an offer they would be fools to refuse. The new owners gamble on lowering cost-of-goods-sold in any and all ways possible and making their money back before the good reputation is ruined by the cheap shoddy products now being delivered. The only difference is that nowadays, technology companies can, in some cases, accelerate this to only 5 years before being bought out.

MS buying Bethesda was not good news. It was the beginning of the end.

Darethran · January 12

Do any of you even read the Vengeance threads? Zenimax said themselves exactly what the problem was in the last Q&A:

Have you considered that not the Skills, but other things lead to the poor performance? For example certain server side updates, which lead to big performance regressions. Even if you run around alone in Cyrodiil, running can be slowed down, or riding on your mount can be sluggish and get slow downs, without ANYONE being near. - RedJohn_COF

A. Yes, we've considered and tested what contributes to game performance impact and have looked at the various things including physics processes (the things that control destructible states on keeps and such). We’ve looked into collections, outfit stations, quests, inventory, etc. Everything pales in comparison to ability usage combined with passives (this includes item sets, CP, and skill line passives), which is why they were the focus of our Vengeance tests.

JustLovely · January 12

Dalinar4 wrote: »

1000% agree with the person who mentioned turning add-ons off. As far as I can tell it is the one thing they haven't tested. At the very least: run a test. Three days with no add-ons in Cyro, or even across the game. Get some hard data points.

I don't think I can play without add ons. I rely very heavily on my custom UI, just for starters. And I wouldn't do writs without Dulgubons.

JustLovely · January 12

SaffronCitrusflower wrote: »

Lagzee wrote: »

SolarRune wrote: »

I would agree its really important we get clear official written answers that can be referred to in the future to stop cycling around the same arguments/discussions. Maybe with the new management committed to more openness and transparency we will get this level of clarity.

My interpretarion of events goes back to what we saw from the vengeance 1 feedback, in my opinion what this clearly showed was that the computation load was the underlying cause of the issues, this would also make sense with the original game design having more client based calculations in the old days that ended up being moved to server side. The only time vengeance 1 was remotely laggy was the massive stalemate battles and as was shown in the feedback this was the only time the computation load got near the levels of GH. (That if I recall was with population 4 or 5 times higher than GH). Computational load being the underlying issue, also explains why as features were turned on more people experienced lag on vengeance at lower population levels.

So if my interpretation of what has been presented so far is correct, and the problem is computational load, then that's a pretty big issue, how do you limit healing stacking or add/remove functionality through battle spirit without increasing computational load?

You mention disabling certain sets, that would be potential way to clearly reduce calculations, but given the response from people with vengeance not having sets I can see why ZOS may not want another backlash in that space (I know removing a handful of sets is different to no sets at all, but just saying ZOS are in a no win situation on this element imo).

Ya maybe this is the reason. But then the question is why isnt it reversable, or something that can be improved or changed. And ya some people will always complain but if changes helped performance more people would be happy. Removing sets would probably be the biggest issue for people, but there are other things that could be done. Like disabling cross healing out of groups, sure the people that randomly go around healing zergs, that dont want to group, wouldnt like it, but how many people actually fall into that category?

And the heal stacking issue, and how many people can actually get healed by certain skills, is deff a problem. But as for how they can remove/adjust these things through battle spirit without increasing computational load, im not sure but i would imagine the benefit would out weigh the cost. And thats if it even works that way. Like maybe the load of battle spirit, spread out through the entirety of gray host, works way less than the load of 2 ball groups fighting at a keep spamming a million heals for a half hour.

turning off addons is another test they could do. Like i wonder how much impact addons have on performance.

You can literally feel the complete degradation of performance across the entire map when there is a massive fight, with multiple ball groups. Performance goes from bad to unplayable. Like at this point i would take bad performance over unplayable performance, which is what were getting every night in gray host. And honestly its mind blowing to say that about a game like this, made by a company like zenimax, backed by microsoft, with millions of dollars behind it, but this is where we are at this point.

Ive said it time and time again over the years, i refuse to believe a team like this cannot get to the bottom of this issue and fix it. Or at the very least give us the exact issue, and reason why its impossible to fix. I think at the very least we deserve that much after all this time, and so many broken promises.

If they can't fix GH then they don't have what it takes to build a whole new system up from scratch either. That's how we know vengeance will fail at least as bad or worse than GH is failing now.

This is what it all boils down to.

If ZOS can't identify and fix issues with GH, then they don't have the ability to create a whole new system from scratch that won't also suffer the exact same issues as GH. So may as well focus on fixing issues in GH.

Taarente · January 12

Darethran wrote: »

Do any of you even read the Vengeance threads? Zenimax said themselves exactly what the problem was in the last Q&A:

Have you considered that not the Skills, but other things lead to the poor performance? For example certain server side updates, which lead to big performance regressions. Even if you run around alone in Cyrodiil, running can be slowed down, or riding on your mount can be sluggish and get slow downs, without ANYONE being near. - RedJohn_COF

A. Yes, we've considered and tested what contributes to game performance impact and have looked at the various things including physics processes (the things that control destructible states on keeps and such). We’ve looked into collections, outfit stations, quests, inventory, etc. Everything pales in comparison to ability usage combined with passives (this includes item sets, CP, and skill line passives), which is why they were the focus of our Vengeance tests.

If you look at what’s happening and what they’ve been doing then the problems are fairly obvious. However the web of interactions make it incredibly difficult to diagnose one specific thing. So the best thing is to start with simpler setups. it’s a legacy of 10 years of things being added to the game. Vengeance shows that making it simpler to track changes in state for the pvp system does seem to help.

Lagzee · January 13

soelslaev wrote: »

Lagzee wrote: »

... made by a company like zenimax, backed by microsoft, with millions of dollars behind it...

Please stop perpetuating this falsehood.

When company A buys company B, the purpose of the purchase is to extract the wealth from company B and transmit that wealth to the owners of company A. That is why whatever brand name product that your dad swore by is now garbage. From Sears tractors to Gibson Les Paul guitars.

It used to be, a small group of dedicated people made a good product, and it got a good reputation; that takes 20 years. Then the original folks are tired and hand it over to the next gen. Then next gen gets an offer they would be fools to refuse. The new owners gamble on lowering cost-of-goods-sold in any and all ways possible and making their money back before the good reputation is ruined by the cheap shoddy products now being delivered. The only difference is that nowadays, technology companies can, in some cases, accelerate this to only 5 years before being bought out.

MS buying Bethesda was not good news. It was the beginning of the end.

this is a weird point to be all up in arms about. It is not a falsehood. I used to say the same thing in my old posts about performance 5 years ago BEFORE msft bought zenimax. This is a company worth millions of dollars, it is strange to me that they cannot get to the root of an issue like this, that has been plaguing the game for nearly its entire existence. IF they truly wanted to.

Trust me, i know what microsoft buying them means. I was literally playing new world the day amazon announced that it was laying off the entire studio, among many others, getting out of online gaming, and that new world would recieve no more updates after the one we had just gotten 2 weeks prior. In one fell swoop they killed the game, 2 weeks into the best update it ever got. A game a spent thousands of hours in and loved. It was a really weird experience, and very sad.

So trust me, i know these companies are nothing good. Whether its a big company like amazon, with their own studio like ags, or a company like microsoft buying a company like zenimax. It just means more steeper demand to make more money.

I just always felt that fixing the root of the performance issue would only make them more money in the long run. Just like better rewards would make them more money in the long run. The same way not chasing away half of the player base with bad changes like update 35 and subclassing, would make them more money in the long run. More players = more money, and better performance means more players. Almost every player i ever spoke to that quit did it over bad performance, a lack of in game rewards, or bad changes to the core of the game. That is my philosophy in saying that. I know these companies only care about the bottom line, that is obvious.

My only hope with the microsoft thing is, now that they have shut down project blackbird, they will leave zos alone to make money off of eso. I mean if they get rid of zos what did they even by zenimax for? Betheseda? Elder scrolls 6 in 5 years and then another game in 10? I dont think so, and i hope not. But yes, if zos wasnt making enough money for msft of course they would axe it.