We know how to END lagg

Amottica · February 2022

I would suggest if the solution was this simple Zenimax would have figured that out long ago.

I have been told performance was much better years ago. That alone disproves that the NPCs spawning in is the culprit or the solution since that happened long ago as well.

The link to a notice about what Zenimax is planning to do this year makes much more sense than anything I have read in the past year. Ofc, nothing is a guarantee.

BazOfWar · February 2022

Amottica wrote: »

I would suggest if the solution was this simple Zenimax would have figured that out long ago.

I have been told performance was much better years ago. That alone disproves that the NPCs spawning in is the culprit or the solution since that happened long ago as well.

The link to a notice about what Zenimax is planning to do this year makes much more sense than anything I have read in the past year. Ofc, nothing is a guarantee.

Agreed, I really do hope they fix the performance for the new players if nothing else. I am not holding out much hope though because they can't even fix the out of combat bug so what chance do we have of them fixing and rewriting something as complex as the game code.

vesselwiththepestle · February 2022

There is still something sketchy about non player characters (npc). I give two examples.

The npc on a ressource, when it uses that big heal to get to full health when almost down... After the npc did that, you won't be able to fossilize that guard ever again until it is dead. It got full immunity to fossilize, maybe other stuns, too. I don't know why, it behaves similar to the exploit where you use mist form to get immunity against fossilize (which is one of the main reasons dragonknights complain their fossilize doesn't work in cyrodiil correctly). I've never seen behavior like that in pve, my assumption is, that there is something wrong with the npc scripting, although ZOS tried to fix some stuff around the guards a while back. (I don't remember which patch).

The other thing I noticed, sometimes you can't interrupt the guard on top of the ressource tower, it has also very strange targeting issues, like it's hit box is changing, I am standing right in front of it and can't touch it. Then other times I interrupt this guard and I am getting a second channeled attack/stun directly after the interrupt.

Also I'd like to point out... When you've got a big fight at Glade, you might notice insane lag at Warden, too. It's connected. Then there is something strange stuff going around... Yesterday I was at Ales farm, big blue zerg, big ep zerg and yes, it was laggy, but nothing out of the extra-ordinary. Then I had a look at Nikel, there were maybe 10 DC, 15 AD, but it was all smallscale stuff, biggest group was some 4 player organized smallscale group. 3 dudes fighting there, 4 over there, another two at the big rock... and the lag was insane, way more than the amount of players should have caused. I tried to fight, but abilities weren't cast, and even if it worked it would have several seconds delay... Absolutely useless, so I went back to the zerg fights. And I can assure there weren't that many guards at NIKEL.

_adhyffbjjjf12 · February 2022

server performance is not predicated on the existence of NPCs/scripts which has a very linear and predictable impact on performance. Its root cause is poor code and the ability to handle the level of message traffic caused by players interacting. The devs have already acknowledged (after 8 years?!) that they need to significant rewrite the engine. This has been well known since day, people are grasping at straws for a quick silver bullet, but it is what it is.

The key question now is will they genuinely resolve the root cause issues or will they be compromised by budget and management and patch tactical fixes.

Einstein_ · February 2022

_adhyffbjjjf12 wrote: »

server performance is not predicated on the existence of NPCs/scripts which has a very linear and predictable impact on performance. Its root cause is poor code and the ability to handle the level of message traffic caused by players interacting. The devs have already acknowledged (after 8 years?!) that they need to significant rewrite the engine. This has been well known since day, people are grasping at straws for a quick silver bullet, but it is what it is.

The key question now is will they genuinely resolve the root cause issues or will they be compromised by budget and management and patch tactical fixes.

I never said and i dont think that it will jsut fix it completly, but IT will for sure reduce the stress onto the server. NPC can get DMG/Buffed/Healed so the for sure produce calculations the same way players do.

Even if it helps slightly its worth to do because they are anyway useless !!

and ofc they need to rewrite parts of there code, but that will take ages and this is a short fix which helps to improve it slightly untill its fixed.....

AJones43865 · February 2022

Silversmith wrote: »

Einstein_ wrote: »

Not really completly i just wanted to make a clickbait title, but we acually observed a big source of lagg and it might be possible to reduce lagg by a good margin in cyro. I know that ZoS works on it (hopefully), but that will take alot time till finished and we can't/don't want to wait so long.

Introduction:
With "we" i mean our Guild Project Nova, but that thread should not be about ballgroups (no they dont "produce" lagg, its the sheer amount of "targets" in a area). But we play regulary in big fights with heavy lagg, so we have alot experiance when it comes to performance issues and bugs in cyro. We have good feeling what inceases lagg and how to self fix some bugs, but i will tell that maybe in a spoiler since that should be the main topic.

Yes, ball groups produce lag. They produce more lag than other groups. NPCs create very little lag, newer games put unrendered assets/npcs in "stasis" when a player isn't in render distance or in the camera's field of view. Some games have two layers of stasis to improve latency.

As someone with knowledge of networking, and coding, and gridlocking, and processing in game development; i want to explain this to people in super simple terms without getting turbo nerd about it.

Every player is an entity and any action they perform or interaction they have must be processed. The more players and actions you have creates more of a work load for the game to process. Every heal/dot/proc/buff/debuff has to be processed. When you have a bunch of players spamming abilities, refreshing buffs and putting hots on each other and ground effects the game must constantly process these interactions. If a player is wearing a set that procs then the game has even more of a work load to process. If ESO has any kind of network logs, they know that the CPU-PER-USER for a ball group player is higher than a non ball group player.

The following are real ways to lower latency in ESO. You and others may not like it but it doesn't change the fact that the following will help latency in PVP.

Reduce Player Population
Less people, Less actions to process. It sucks to do this when you promise a big faction war but if your engine cannot process it then you just *** people off.

Longer Buffs
ESO has short buffs that need to constantly be reapplied. It's tedious and process heavy. Other MMOs get around this by having long lasting buffs (normally 30mins - 2 hours) and only need to be processed when first cast, expires, or if a player leaves the group with the player who applied the buff. The fix would be longer lasting buffs for things like Major Savagery to 10 minutes.

Remove the majority of Procs and Proc Sets
Procs cause latency. A lot of MMOs don't have procs in their sets because they know this. Sets giving flat bonuses or altering an ability is kinda boring but you make up for it with less processes and better latency.

Dont bottleneck interactions (Mostly damage) in ticks and give priority to healing/defense processes
Simply put, this is why several of your GCDs can be countered by a roll dodge/shield/heal even if the player didn't roll/cast at the correct time to counter the attack but they rolled/cast before the tick processed so the game counts it. This is a tricky one to fix and requires the engine to be optimized for PvP interactions and have things processed in real time and not gated.

Slower Players
The faster a player entity moves, the more the game has to process it's location and update it back and forth between both clients and the server to see if an attack is able to connect or not. This also affects LOS. This also causes render lag and desync. Everyone reading this has at one point had their skill light up like a player is in range, push the button, and have nothing happen because by the time the game processed the distance the targe is no longer valid and within range. Most MMOs cap all classes at a base speed with a few minor speed increasing skills to a few classes. ESO lets all players zoom up to 200% speed constantly.

Change States on how Procs Proc.
Simply put, Procs should only check when they are allowed to Proc. For a random example: Random ability procs off damage 10% of the time and can only proc every 30 seconds. The ability procs and cannot proc again for 30 seconds but the game keeps checking to see if it could proc every time you deal damage. This adds more processes and is wasteful. What should happen is that when the ability procs the game puts a state change and doesn't even allow a check for a chance to proc for 30 seconds. With big AOE skills you would cut down on all the checks and improve latency.

Remove Doodads and Terrain Clutter.
Less things to render. Game won't look as pretty but you get better performance.

This is an excellent post! Thank you.

However, you did not mention one thing that is relevant to ESO. A company also has to lease servers that are capable of supporting their game. There has to be enough processing power on the other end of the game to keep up, especially if most of the calculations are being done on the server end vs. the client end.

Also, ZOS has reduced the population caps in cyrodiil to approximately 75 players/faction. It used to be 500/faction. So they've already reduced populations to a number that is too low for healthy competition.

xylena_lazarow · February 2022

Amottica wrote: »

I have been told performance was much better years ago.

Rose colored glasses, grass is greener, etc... this game has lagged badly since launch, there are at least fewer bugs and crashes now, and people often forget that one of the biggest hits to performance was itself a fix to the massive Cheat Engine epidemic seen by the first couple years of this game's life.

I'd still argue poor balance drives players away just as much (or more than) the poor performance. Unstoppable high pop PvDoor zerging, unkillable troll groups, obnoxious sets like Dark Convergence... you'd be surprised how many players would put up with bad performance if the game was actually balanced and competitive.

_adhyffbjjjf12 · February 2022

Einstein_ wrote: »

_adhyffbjjjf12 wrote: »

server performance is not predicated on the existence of NPCs/scripts which has a very linear and predictable impact on performance. Its root cause is poor code and the ability to handle the level of message traffic caused by players interacting. The devs have already acknowledged (after 8 years?!) that they need to significant rewrite the engine. This has been well known since day, people are grasping at straws for a quick silver bullet, but it is what it is.

The key question now is will they genuinely resolve the root cause issues or will they be compromised by budget and management and patch tactical fixes.

I never said and i dont think that it will jsut fix it completly, but IT will for sure reduce the stress onto the server. NPC can get DMG/Buffed/Healed so the for sure produce calculations the same way players do.

Even if it helps slightly its worth to do because they are anyway useless !!

and ofc they need to rewrite parts of there code, but that will take ages and this is a short fix which helps to improve it slightly untill its fixed.....

no it wont because its not on the compute critical path i.e when there are issues removing a guard is going to have zero impact. Aside from this, if it did have an impact they would have done it already. There is no silver bullets for this kind of issue, in fact wasting time on this kind of fluff obfuscates the issue and actually takes resource away from what needs to happen.

AJones43865 · February 2022

xylena_lazarow wrote: »

Amottica wrote: »

I have been told performance was much better years ago.

Rose colored glasses, grass is greener, etc... this game has lagged badly since launch, there are at least fewer bugs and crashes now, and people often forget that one of the biggest hits to performance was itself a fix to the massive Cheat Engine epidemic seen by the first couple years of this game's life.

I'd still argue poor balance drives players away just as much (or more than) the poor performance. Unstoppable high pop PvDoor zerging, unkillable troll groups, obnoxious sets like Dark Convergence... you'd be surprised how many players would put up with bad performance if the game was actually balanced and competitive.

I agree. I was one of the people screaming about fixing all the cheating. It was a huge problem. It seems obvious that we need far fewer proc sets to reduce server side calculations. Then they created Dark Convergence and two other proc sets to the PvP rewards. *shrug*

Amottica · February 2022

xylena_lazarow wrote: »

Amottica wrote: »

I have been told performance was much better years ago.

Rose colored glasses, grass is greener, etc... this game has lagged badly since launch, there are at least fewer bugs and crashes now, and people often forget that one of the biggest hits to performance was itself a fix to the massive Cheat Engine epidemic seen by the first couple years of this game's life.

I'd still argue poor balance drives players away just as much (or more than) the poor performance. Unstoppable high pop PvDoor zerging, unkillable troll groups, obnoxious sets like Dark Convergence... you'd be surprised how many players would put up with bad performance if the game was actually balanced and competitive.

Maybe time has tempered their memories but the problem is not as simple as presented in the OP here.

Also, poor balance is not the topic of the thread here unless I missed something.

_adhyffbjjjf12 · February 2022

Amottica wrote: »

xylena_lazarow wrote: »

Amottica wrote: »

I have been told performance was much better years ago.

Rose colored glasses, grass is greener, etc... this game has lagged badly since launch, there are at least fewer bugs and crashes now, and people often forget that one of the biggest hits to performance was itself a fix to the massive Cheat Engine epidemic seen by the first couple years of this game's life.

I'd still argue poor balance drives players away just as much (or more than) the poor performance. Unstoppable high pop PvDoor zerging, unkillable troll groups, obnoxious sets like Dark Convergence... you'd be surprised how many players would put up with bad performance if the game was actually balanced and competitive.

Maybe time has tempered their memories but the problem is not as simple as presented in the OP here.

Also, poor balance is not the topic of the thread here unless I missed something.

yeah its a different discussion and issue. These performance issues haver exist for 8 years, that's before proc sets were a thing, or DC or [insert here], The devs have now acknowledged the engine needs to be fundamentally written, which has also been called out from day one.

xylena_lazarow · February 2022

AJones43865 wrote: »

I agree. I was one of the people screaming about fixing all the cheating. It was a huge problem. It seems obvious that we need far fewer proc sets to reduce server side calculations. Then they created Dark Convergence and two other proc sets to the PvP rewards. *shrug*

They tried that, there was a whole 3mo where procs were disabled across campaigns, only the 18 or so pure passive stat sets were allowed, and performance was still awful (some said worse than with procs enabled). Balance was terrible too, there was basically two builds and 3-4 viable sets, which really amplified class imbalance, this was before the past year's class tweaks so it was "oops all Wardens."

Something like Dcon doesn't cause lag, but it sure makes it extra miserable to play during lag, pulling you from outside the visual radius and the lag not allowing you to respond in time to avoid the explosion.

Einstein_ · February 2022

_adhyffbjjjf12 wrote: »

Einstein_ wrote: »

_adhyffbjjjf12 wrote: »

server performance is not predicated on the existence of NPCs/scripts which has a very linear and predictable impact on performance. Its root cause is poor code and the ability to handle the level of message traffic caused by players interacting. The devs have already acknowledged (after 8 years?!) that they need to significant rewrite the engine. This has been well known since day, people are grasping at straws for a quick silver bullet, but it is what it is.

The key question now is will they genuinely resolve the root cause issues or will they be compromised by budget and management and patch tactical fixes.

I never said and i dont think that it will jsut fix it completly, but IT will for sure reduce the stress onto the server. NPC can get DMG/Buffed/Healed so the for sure produce calculations the same way players do.

Even if it helps slightly its worth to do because they are anyway useless !!

and ofc they need to rewrite parts of there code, but that will take ages and this is a short fix which helps to improve it slightly untill its fixed.....

no it wont because its not on the compute critical path i.e when there are issues removing a guard is going to have zero impact. Aside from this, if it did have an impact they would have done it already. There is no silver bullets for this kind of issue, in fact wasting time on this kind of fluff obfuscates the issue and actually takes resource away from what needs to happen.

Because you work for ZoS and know what the bottleneck is ? i guess not.

I also guess you think the Devs at zos just did the AOE testing for fun and the proplem is not the interactions between targets. Alone based on the fact that all this tests where made for AOEs only shows that the proplems is somewhere between the interactions of Targets.

i am also not claiming that this is any kind of silver bullet, but i claim based on multiple OBSERVASIONS that the spawn of NPCs in a keep fight creats a incease of lagg. And ofc this will not fix lagg but even if its only getting slightly better its worthy. Since nobody needs wolfs/monsters/guards anyway.

_adhyffbjjjf12 · February 2022

Einstein_ wrote: »

_adhyffbjjjf12 wrote: »

Einstein_ wrote: »

_adhyffbjjjf12 wrote: »

server performance is not predicated on the existence of NPCs/scripts which has a very linear and predictable impact on performance. Its root cause is poor code and the ability to handle the level of message traffic caused by players interacting. The devs have already acknowledged (after 8 years?!) that they need to significant rewrite the engine. This has been well known since day, people are grasping at straws for a quick silver bullet, but it is what it is.

The key question now is will they genuinely resolve the root cause issues or will they be compromised by budget and management and patch tactical fixes.

I never said and i dont think that it will jsut fix it completly, but IT will for sure reduce the stress onto the server. NPC can get DMG/Buffed/Healed so the for sure produce calculations the same way players do.

Even if it helps slightly its worth to do because they are anyway useless !!

and ofc they need to rewrite parts of there code, but that will take ages and this is a short fix which helps to improve it slightly untill its fixed.....

no it wont because its not on the compute critical path i.e when there are issues removing a guard is going to have zero impact. Aside from this, if it did have an impact they would have done it already. There is no silver bullets for this kind of issue, in fact wasting time on this kind of fluff obfuscates the issue and actually takes resource away from what needs to happen.

Because you work for ZoS and know what the bottleneck is ? i guess not.

I also guess you think the Devs at zos just did the AOE testing for fun and the proplem is not the interactions between targets. Alone based on the fact that all this tests where made for AOEs only shows that the proplems is somewhere between the interactions of Targets.

i am also not claiming that this is any kind of silver bullet, but i claim based on multiple OBSERVASIONS that the spawn of NPCs in a keep fight creats a incease of lagg. And ofc this will not fix lagg but even if its only getting slightly better its worthy. Since nobody needs wolfs/monsters/guards anyway.

well i've been a software engineer for over 20 years on large enterprise solutions so i can recognise well worn patterns but that's actually irrelevant as the developers for ZOS have already called out in a lengthy post what the issues are. I suspect you have not read or understood but obviously 'observations' in game by a player is hardly metrics (its confirmation bias)

BazOfWar · February 2022

The blame can't even be placed solely on procs because (ravenhost) "no proc, no cp and no volendrung" is also laggy as hell when near to capacity.

vgastel · February 2022

Either way, the PvP letter mentions special rulesets or weekend events. Strongly reducing or removing npcs at keeps resources would an easily implementable weekend event and imo besides lag would be interesting to see the changed gameplay.

Present gameplay, certain coordinated groups excepted, anyways seems to levitate towards massively sitting in keeps, having a few scouts report about incoming groups and then faction zerg them down.

...and new servers, optimizing net code will of course all help to reduce lag. Ultimately, the latter will likely have the largest impact if it ever happens. But throwing out most npcs for a weekend and seeing what happens imo can't be a bad thing.

Kwoung · February 2022

I am somewhat curious to see if they even do a MYM event this year, it is supposed to be happening soon no? I question it, because in all previous years the lag magically disappears for the duration of the event, even though there are numerous more instances of Cyro spun up and they are most all pop locked to boot.

I mean that in itself says the issue is solvable right now, if they can manage to solve it a couple times a year at will. They simply choose not to the rest of the time, most likely because it cuts into their profits more than losing some players or complaints on the forums do.

Zelos · February 2022

You know I have to say my friend and I did indeed find the source of lag in cyrodiil and fortunately it was not in your post, so I'll break it down for you. Using some mod tools to capture packets my friend and I found a "Status"(not the actual full name but to keep it simple) packet that is sent to and from players in and out of render distance. Once when a player comes in and once when a player goes out of distance. This packet is really, "spammed" for lack of a better word between all players. So when you put 100 players in proximity of each other, the amount of these packets that go out per second are probably in the 10,000s between all players. Since computers and servers can really only process 1 packet at a time(albeit incredibly quickly) when you have to process 10,000 packets first before your damage packet or skill packet, sh*t tends to slow down. Like a queue of people in line ahead of you. It looks something like this. Your computer sends the status packet to the server, server sends that packet to EVERY player in your render distance and they all send theirs to you which also pass through the server. In-between all that you have actual packets like damage packets and skills and animation packets sent between the clients and server so when someone light attacks you can see the animation and everyone else can too. As far as I can tell this status packet is pretty much empty and has nothing really in it. The best metaphor i can think of to break it down more goes like this. A printer printing a blank page in-between each page of a 1,000 page document which starts slowing the process down considerably. My friend and I think this status packet accounts for 80% of the lag in cyrodiil at the minimum with our current theory if it's correct. Hope this shed some light on the problem:)

*Edit*
The status packet is also attached to every packet you send as well. Its ALWAYS weaved in whenever your character does ANYTHING. So really with 100 players in proximity always doing something the number per second is probably significantly higher then 10,000. So when you have that 1 packet clogging up the server and client it slows down and the playing time also slows down. Despite how fast computers are when you have that many packets processed in such a short amount of time, it starts adding up to full seconds before actual packets get through. Remove the status packet and you would see a large increase in performance if the theory is correct. However the packet looks useless and empty mostly from our end, I don't know how important it is and I don't really care, but it is indeed the problem:)

Packet = A small file limited in size that transfers key information from your client to others through a server. It's very small in bytes so it can be encrypted within fractions of a millisecond, uploaded to the server and downloaded to clients in fractions of milliseconds. This is the basis for online play. Everything you do in game is sent to the server in "packets of information" hence the name, packet.

_adhyffbjjjf12 · February 2022

Zelos wrote: »

You know I have to say my friend and I did indeed find the source of lag in cyrodiil and fortunately it was not in your post, so I'll break it down for you. Using some mod tools to capture packets my friend and I found a "Status"(not the actual full name but to keep it simple) packet that is sent to and from players in and out of render distance. Once when a player comes in and once when a player goes out of distance. This packet is really, "spammed" for lack of a better word between all players. So when you put 100 players in proximity of each other, the amount of these packets that go out per second are probably in the 10,000s between all players. Since computers and servers can really only process 1 packet at a time(albeit incredibly quickly) when you have to process 10,000 packets first before your damage packet or skill packet, sh*t tends to slow down. Like a queue of people in line ahead of you. It looks something like this. Your computer sends the status packet to the server, server sends that packet to EVERY player in your render distance and they all send theirs to you which also pass through the server. In-between all that you have actual packets like damage packets and skills and animation packets sent between the clients and server so when someone light attacks you can see the animation and everyone else can too. As far as I can tell this status packet is pretty much empty and has nothing really in it. The best metaphor i can think of to break it down more goes like this. A printer printing a blank page in-between each page of a 1,000 page document which starts slowing the process down considerably. My friend and I think this status packet accounts for 80% of the lag in cyrodiil at the minimum with our current theory if it's correct. Hope this shed some light on the problem:)

*Edit*
The status packet is also attached to every packet you send as well. Its ALWAYS weaved in whenever your character does ANYTHING. So really with 100 players in proximity always doing something the number per second is probably significantly higher then 10,000. So when you have that 1 packet clogging up the server and client it slows down and the playing time also slows down. Despite how fast computers are when you have that many packets processed in such a short amount of time, it starts adding up to full seconds before actual packets get through. Remove the status packet and you would see a large increase in performance if the theory is correct. However the packet looks useless and empty mostly from our end, I don't know how important it is and I don't really care, but it is indeed the problem:)

Packet = A small file limited in size that transfers key information from your client to others through a server. It's very small in bytes so it can be encrypted within fractions of a millisecond, uploaded to the server and downloaded to clients in fractions of milliseconds. This is the basis for online play. Everything you do in game is sent to the server in "packets of information" hence the name, packet.

Stick animation cancelling into that equation where people are spamming 2 skills a second over and above everything else they are doing, then AOE, then procs, then a core engine that has had issues from the start and it's Meltdown time. NPCs and little deer don't do this as far as I can tell