Maintenance for the week of December 29:
• PC/Mac: No maintenance – December 29

Findings related to Cyrodiil lag

NoodleESO
NoodleESO
✭✭✭✭✭
Here im going to talk about Cyrodiil lag and some interesting findings on what could be causing it. Theres many different kinds of lag in Cyrodiil but here when I say lag I am referring to server latency during prime time PC NA. In my experience my fps will be fine and my abilities will cast but none of them will land on an enemy standing right in front of me and the same for them even if it's far away from any fight.

(there will be tldr at the end)

Cyrodiil wasn't always like this and most players can agree its been getting worse each update, so what has changes since day one? I decided to do some digging and having looked through all the old patch notes I can tell you its a lot. They added new keep passives, underdog bonuses, how guards work, how resources work, DLC sized overhauls over and over again.The more they added, the more the performance slipped.

Zos is trying to improve performance in pvp by adjusting things we all use in pvp (sprint, block, race passives) to reduce server stress. A few years ago Zos even tried disabling Champion points for a weekend and the performance was better but the lag never really went away during primetime. With performance getting progressively worse this would suggest that there is no one thing causing problems but instead years of caked on underlying issues surrounding pvp, since this kind of lag is not present at all in pve/overworld.

- One of the things that all pvp players have in common is there is an artificial rule stating that players will not receive armor degeneration from pvp kills/exp/deaths, you will also not receive armor degen from killing or dying from an alliance guard, taking/defending keeps, or turning in alliance daily quest. Armor degenerating is not turned off in Cyrodiil though and you can still get it from killing/ dying to non alliance npcs. This means there's some kind of checking system in place and I believe after doing some digging in older patch notes this might be one of the causes for Cyrodiil lag.
  • Update 1 - changes to Armor degeneration and armor repair cost were made and in the next hot patch (1.1.3) we got the first mentions of Cyrodiil latency in the patch notes. This would get overshadowed by patch 1.2 (the lighting patch) when everyone's fps hit 5-10.
  • Update 6 - changes to the impen trait, aoe caps removed, champion points and crown store introduced. very important for later.
  • Update 10 - More changes to impen trait, now reduces the items durability damage by 40%. This would be overshadowed by the events surrounding 2016 Memorial Day weekend.

Another thing pvp players all have in common is wearing the impen trait on all 7 pieces of armor (8 including shield). Back when these trait changes were made its unlikely they were thought about on a large scale or how it would work with the artificial rule on a large scale. Patch 1.1.3 should have been the first sign of trouble and for them to add the 40% degeneration reduction on impen could mean none of this was looked at as a link to server latency.

Armor degeneration applies to all 7 pieces (8 if shield is on the bar) all at the same time when a player dies. The only other way is when you gain exp from a kill or healing the slayer, and when you gain the exp according to my test you will receive armor degeneration to a random piece of armor every time you gain exp from a kill. With there being no aoe caps on damage or healing you can get exp from many kills just by pressing one button. In a group setting you can get exp from other players in the group getting the kills and when it comes to pvp there's a lot of maxed out groups running around. And every time a player gains exp from a kill, or dies the server has to apply the checking system to all of their gear.

Depending on how its coded it could look like this
  • armor degen > - 40% > is pvp death y/n. (per piece of gear(x7) from dying)
  • armor degen > random 1/7 > -40% > is pvp kill y/n. (for a kill)
If this is the case it would result in a lot of sever calculations for the main campaign come prime time. In pve this wouldn't be an issue since most instances don't have 3 whole factions of people dying and killing vs Cyrodiil is one whole instance. In Battlegrounds there is only 12 people max plus players don't receive any exp until then end of the match so the checking is halved with less people, and champion points are disabled. Dueling has no exp gained and this could be why 30 people dueling in one place causes no issues. Even dolmen trains have less latency and I suspect its because people rarely die and there's only a hand full of mobs active at a time.

Changes to how keeps and resources give ap/exp not long ago, with the old way a player would have to wait at a keep/resource to get credit for the defense. With the new system anyone who shows up and does a light attack will get credit (ap/exp) in 10 mins. This means more players are getting exp more often and if the checking system for armor degen is linked to that exp it could add to the problem

From my testing is the armor degeneration from pve kills and deaths is inconsistent which could mean there's another thing in the checking order that isn't making sense. All of my test were done with no champion points applied and in all training trait gear with SnB equipped at all times in the standard campaign with winters peak giants. The map there is stale so there were no keep bonuses being applied. In the test killing a giant gave me a repair bill of 4 gold but killing a tundra deadeye gave me a 2g repair, meaning there's some kind of extra check for different enemies in the game. Dying from both of them would give me a repair bill of 45g, but sometimes dying to a giant would give me randomly a 38g repair with snb still equip, another inconsistency. Dying without snb is a 43g repair. And finally sometimes killing a tundra deadeye would give me a 1g repair instead of 2g, killing a giant would randomly give me a 2g repair instead of 4g. The degen isn't based on exp percentage like intended, I've tested with a 50% exp boost scroll and got the same results just to be sure.

A good way for Zos to test this is for a whole weekend remove/disable all exp gained in Cyrodiil, remove the 40% reduced durability from impenetrable, and remove/disable the merchant favored champion point passive from the shadow tree. This might not fix the issue completely since Armor degen applies to deaths too but it could reduce the lag if its the root of the problem. If it is an issue then the whole armor degen/ armor repair system needs an overhaul but I suggest removing it completely, an overhaul would cost time and resources and new systems could cause more problems.

(TLDR: The artificial rule of no armor degeneration from pvp could be causing issues and the impen trait may be adding to it.)
  • Alienoutlaw
    Alienoutlaw
    ✭✭✭✭✭
    ✭✭
    interesting theory

  • Thevampirenight
    Thevampirenight
    ✭✭✭✭✭
    ✭✭
    Though those issues could contribute to this. I think the biggest issues would be with the servers doing the clients job. The client use to do a lot more from what I understand. That is why the game was able to handle so much. But then there were two issues. The bots biggest reason and then cheat engine. Back in the day bots were able to fly around fly under the map and do all kinds of things. Until they moved a lot of the code to the server. Which might have fixed a lot of the issue with bots but it was a gag fix at best finally they got the idea of the delete limit and the bot spam problem in zone chat went away as their most effective means to farm for gold were cut off from them.

    But even then they didn't have everything be calculated by the Servers. Attributes being the biggest one.So the client for a while still calculated Attribute values. Then someone had to have a bright idea to find a way to modify those values. Thus they created the cheat engine. However Zenimax got on top of it pretty quickly. Which I think was changing the attributes to be calculated by the server removing that functionality from the Client.

    So a lot of what the client use to do is being done by the servers. When pvping the Server calculates for each person, Attributes, passives, abilties, Cp Points, Armor sets. Its doing it all and when players battle one another in huge zerg groups and ball groups. Spamming ability after ability of server degrading abilties like healing springs and many kinds of aoe effects. The calculations plus all that going on causes lag sinks, Sinks that prevents the server from disturbing out the calculations because so many are happening in one spot.

    What they should do is this the majority of all CP Passives need to be converted to flat based calculations like they did with most of the racial passives. The entire Cp system needs to be reworked and be made to use only flat based calculations.
    Armor sets need to be made to work off flat based values and almost every passive should be reworked to be flat based. Maybe not all armor sets need to be made flat but a majority of them should be converted to flat based values. Reducing as many calculations as possible the server has to process the better it will perform.

    Here is what it would do when many players are stacked around in ball groups the calculations going all around wouldn't be extensive as it is right now. There is other things they could do. They could just remove the armor degrading system. Its not even visual all it does is weaken your gear or make it useless. If its part of the problem then removing it could help with that.
    Even after removing most of the percent based values. There might still be some issues with lag but it would be possibly so much better and the stuff they are doing such as rewriting the combat systems and cold storage will also help a lot also.

    So one of the biggest fixes would be reducing those % based values and converting most of them to a flat based values system. This would help a lot with making calculations on the server far less extensive and help with performance. As clearly the servers were not meant to do the clients job. Going flat based would help a lot when it comes to the servers. Players might hate it but I think its needed.
    Edited by Thevampirenight on March 5, 2020 2:29AM
    PC NA
    Please add Fangs to Vampires.
  • DelosTheta
    DelosTheta
    ✭✭✭
    Are we sure that there is not just an extra GOTO 10 line in the code somewhere?
  • Ranger209
    Ranger209
    ✭✭✭✭✭
    What exactly did bots do in Cyrodiil before the anti-bot program was implemented? Curious, I never played in there before that sadly.
  • Thevampirenight
    Thevampirenight
    ✭✭✭✭✭
    ✭✭
    Ranger209 wrote: »
    What exactly did bots do in Cyrodiil before the anti-bot program was implemented? Curious, I never played in there before that sadly.

    They were overrunning pve zones. Killing world bosses, killing mobs and mass collecting crafting nodes by flying under the map. It was so bad that it really hurt the games reputation at launch. It was so very bad they ended up making the changes they made to stop it from happening. Since the majority of the game is pve. It really hurt the pve portions of the game play. It was bad enough to cost them money that is how it was back then.

    I think the majority of the games overall direction that ended up where the game is today was in part because of what the bot programmers did. Its the main reason Performance is so bad right now. It was a sacrifice they were willing to make. I think there were some players that ran bots in pvp but I think the majority of them were infesting the pve zones back when the game launched.
    Edited by Thevampirenight on March 5, 2020 2:41AM
    PC NA
    Please add Fangs to Vampires.
  • NinchiTV
    NinchiTV
    ✭✭✭✭✭
    You know that glitch that says your burning for 49897days because your fighting a magdk in lag? The server probably calculates that huh? And console needs a "pvp setting" that being turning off shadows, grass, the sun glare. Im talking straight 2004 graphics. Bet that would help.
  • Iskiab
    Iskiab
    ✭✭✭✭✭
    ✭✭✭
    What I don’t get is why during mid year mayhem cyrodiil was running better but overland was a bit worse for lag.

    That’s what makes me think it’s a server capacity issue with other things layered on top of it. Cyrodiil also has been getting progressively worse since murkmire and it was playable in murkmire, that to me means there’s probably something recent that’s contributing to poor performance.
    Looking for any guildies I used to play with:
    Havoc Warhammer - Alair
    LoC EQ2 - Mayi and Iskiab
    PRX and Tabula Rasa - Rift - Iskiab
    Or anyone else I used to play games with in guilds I’ve forgotten
  • vgabor
    vgabor
    ✭✭✭✭✭
    NoodleESO wrote: »
    ...there is no one thing causing problems but instead years of caked on underlying issues...

    ^ This.

    In software development it's called "technical debt", when you ignore / don't care / delay fixing bugs and issues when they occur saying they aren't that urgent or important then pushing out new features. So over time they add up, and there comes a point when everything start crumbling down and you realise you have now fundamental problems in your system which are very expensive to fix and you have to rewrite large part of your system because you kept building on defected codes instead fixing them in time.

    It is called technical debt because the longer you ignore the more work and effort it will require to fix, just like real debt there's an interest you have to pay the longer you delay fixing these technical issues. And as ZOS did nothing the last few years but ignored the issues, the result of years of neglect now getting apparent...
  • Cavedog
    Cavedog
    ✭✭✭✭✭
    vgabor wrote: »
    NoodleESO wrote: »
    ...there is no one thing causing problems but instead years of caked on underlying issues...

    ^ This.

    In software development it's called "technical debt", when you ignore / don't care / delay fixing bugs and issues when they occur saying they aren't that urgent or important then pushing out new features. So over time they add up, and there comes a point when everything start crumbling down and you realise you have now fundamental problems in your system which are very expensive to fix and you have to rewrite large part of your system because you kept building on defected codes instead fixing them in time.

    It is called technical debt because the longer you ignore the more work and effort it will require to fix, just like real debt there's an interest you have to pay the longer you delay fixing these technical issues. And as ZOS did nothing the last few years but ignored the issues, the result of years of neglect now getting apparent...

    ...this is why I think we are very near needing the community to demand a complete rebuild from the ground up.
  • Nairinhe
    Nairinhe
    ✭✭✭✭✭
    ✭✭✭
    Cavedog wrote: »
    vgabor wrote: »
    NoodleESO wrote: »
    ...there is no one thing causing problems but instead years of caked on underlying issues...

    ^ This.

    In software development it's called "technical debt", when you ignore / don't care / delay fixing bugs and issues when they occur saying they aren't that urgent or important then pushing out new features. So over time they add up, and there comes a point when everything start crumbling down and you realise you have now fundamental problems in your system which are very expensive to fix and you have to rewrite large part of your system because you kept building on defected codes instead fixing them in time.

    It is called technical debt because the longer you ignore the more work and effort it will require to fix, just like real debt there's an interest you have to pay the longer you delay fixing these technical issues. And as ZOS did nothing the last few years but ignored the issues, the result of years of neglect now getting apparent...

    ...this is why I think we are very near needing the community to demand a complete rebuild from the ground up.

    Do you really think it's possible? I mean rebuild, demands are common stuff here.
  • marius_buys
    marius_buys
    ✭✭✭✭✭
    NoodleESO wrote: »
    Here im going to talk about Cyrodiil lag and some interesting findings on what could be causing it. Theres many different kinds of lag in Cyrodiil but here when I say lag I am referring to server latency during prime time PC NA. In my experience my fps will be fine and my abilities will cast but none of them will land on an enemy standing right in front of me and the same for them even if it's far away from any fight.

    (there will be tldr at the end)

    Cyrodiil wasn't always like this and most players can agree its been getting worse each update, so what has changes since day one? I decided to do some digging and having looked through all the old patch notes I can tell you its a lot. They added new keep passives, underdog bonuses, how guards work, how resources work, DLC sized overhauls over and over again.The more they added, the more the performance slipped.

    Zos is trying to improve performance in pvp by adjusting things we all use in pvp (sprint, block, race passives) to reduce server stress. A few years ago Zos even tried disabling Champion points for a weekend and the performance was better but the lag never really went away during primetime. With performance getting progressively worse this would suggest that there is no one thing causing problems but instead years of caked on underlying issues surrounding pvp, since this kind of lag is not present at all in pve/overworld.

    - One of the things that all pvp players have in common is there is an artificial rule stating that players will not receive armor degeneration from pvp kills/exp/deaths, you will also not receive armor degen from killing or dying from an alliance guard, taking/defending keeps, or turning in alliance daily quest. Armor degenerating is not turned off in Cyrodiil though and you can still get it from killing/ dying to non alliance npcs. This means there's some kind of checking system in place and I believe after doing some digging in older patch notes this might be one of the causes for Cyrodiil lag.
    • Update 1 - changes to Armor degeneration and armor repair cost were made and in the next hot patch (1.1.3) we got the first mentions of Cyrodiil latency in the patch notes. This would get overshadowed by patch 1.2 (the lighting patch) when everyone's fps hit 5-10.
    • Update 6 - changes to the impen trait, aoe caps removed, champion points and crown store introduced. very important for later.
    • Update 10 - More changes to impen trait, now reduces the items durability damage by 40%. This would be overshadowed by the events surrounding 2016 Memorial Day weekend.

    Another thing pvp players all have in common is wearing the impen trait on all 7 pieces of armor (8 including shield). Back when these trait changes were made its unlikely they were thought about on a large scale or how it would work with the artificial rule on a large scale. Patch 1.1.3 should have been the first sign of trouble and for them to add the 40% degeneration reduction on impen could mean none of this was looked at as a link to server latency.

    Armor degeneration applies to all 7 pieces (8 if shield is on the bar) all at the same time when a player dies. The only other way is when you gain exp from a kill or healing the slayer, and when you gain the exp according to my test you will receive armor degeneration to a random piece of armor every time you gain exp from a kill. With there being no aoe caps on damage or healing you can get exp from many kills just by pressing one button. In a group setting you can get exp from other players in the group getting the kills and when it comes to pvp there's a lot of maxed out groups running around. And every time a player gains exp from a kill, or dies the server has to apply the checking system to all of their gear.

    Depending on how its coded it could look like this
    • armor degen > - 40% > is pvp death y/n. (per piece of gear(x7) from dying)
    • armor degen > random 1/7 > -40% > is pvp kill y/n. (for a kill)
    If this is the case it would result in a lot of sever calculations for the main campaign come prime time. In pve this wouldn't be an issue since most instances don't have 3 whole factions of people dying and killing vs Cyrodiil is one whole instance. In Battlegrounds there is only 12 people max plus players don't receive any exp until then end of the match so the checking is halved with less people, and champion points are disabled. Dueling has no exp gained and this could be why 30 people dueling in one place causes no issues. Even dolmen trains have less latency and I suspect its because people rarely die and there's only a hand full of mobs active at a time.

    Changes to how keeps and resources give ap/exp not long ago, with the old way a player would have to wait at a keep/resource to get credit for the defense. With the new system anyone who shows up and does a light attack will get credit (ap/exp) in 10 mins. This means more players are getting exp more often and if the checking system for armor degen is linked to that exp it could add to the problem

    From my testing is the armor degeneration from pve kills and deaths is inconsistent which could mean there's another thing in the checking order that isn't making sense. All of my test were done with no champion points applied and in all training trait gear with SnB equipped at all times in the standard campaign with winters peak giants. The map there is stale so there were no keep bonuses being applied. In the test killing a giant gave me a repair bill of 4 gold but killing a tundra deadeye gave me a 2g repair, meaning there's some kind of extra check for different enemies in the game. Dying from both of them would give me a repair bill of 45g, but sometimes dying to a giant would give me randomly a 38g repair with snb still equip, another inconsistency. Dying without snb is a 43g repair. And finally sometimes killing a tundra deadeye would give me a 1g repair instead of 2g, killing a giant would randomly give me a 2g repair instead of 4g. The degen isn't based on exp percentage like intended, I've tested with a 50% exp boost scroll and got the same results just to be sure.

    A good way for Zos to test this is for a whole weekend remove/disable all exp gained in Cyrodiil, remove the 40% reduced durability from impenetrable, and remove/disable the merchant favored champion point passive from the shadow tree. This might not fix the issue completely since Armor degen applies to deaths too but it could reduce the lag if its the root of the problem. If it is an issue then the whole armor degen/ armor repair system needs an overhaul but I suggest removing it completely, an overhaul would cost time and resources and new systems could cause more problems.

    (TLDR: The artificial rule of no armor degeneration from pvp could be causing issues and the impen trait may be adding to it.)

    Valid sounding theory!
    Golden Clover AD PvP on PC EU (since 2017) Guildex https://eso.guildex.org/view-guild/17669 Facebook https://www.facebook.com/groups/131211320795196
  • marius_buys
    marius_buys
    ✭✭✭✭✭
    Backend server calculus is deffo the issue, individual gear and movement, too many cross checks into armor, mitigation, damage multipliers and then there is the cosmetics omw the poor server :)
    Golden Clover AD PvP on PC EU (since 2017) Guildex https://eso.guildex.org/view-guild/17669 Facebook https://www.facebook.com/groups/131211320795196
  • kojou
    kojou
    ✭✭✭✭✭
    ✭✭
    Backend server calculus is deffo the issue, individual gear and movement, too many cross checks into armor, mitigation, damage multipliers and then there is the cosmetics omw the poor server :)

    I suppose they could make Cyrodiil "No CP" only, and give everyone the same PVP issued gear that only gives a specific set of flat stats to save on server calculations.
    Playing since beta...
  • Kessra
    Kessra
    ✭✭✭
    As far as I understood Rich Lambert in his stream they performed several tests using bots and had problems to reproduce the current problems. According to him the latency issue is definitely not caused by the number of combat-based calculations, i.e. light-attacking vs. heavy-attacking or stuff like that, but is probably based around characters being to big, in the sense of the data model each player carries around, i.e. which quests were done and similar stuff, which grew over the years and thus is probably related to a loading/storage problem. Three years ago we already had the issue with very long loading times which affected players that did plenty of questing more than players that hardly quested. Though this should be easily testable by simply grinding to level 10 and then joining the <50 campaign. But reading this thread here (https://forums.elderscrollsonline.com/en/discussion/591970/insane-lag-on-one-account-only) fosters that assumption somehow.

    But the theory on gear degredation and XP gain does sound plausible as well, though over the years less and less players are joining the battles in Cyrodiil and thus the performance should have gotten either better or at least stayed the same, which it hasn't. And I also think that ZOS has some sharding technology in Cyrodiil active as if a battle for a keep lags out, the lag doesn't affect the whole server but only a limited area near the keep. Therefore the degredation calculation theory somehow may just be a part of the puzzle but not the sole reason.
    Edited by Kessra on November 23, 2021 3:02PM
  • mocap
    mocap
    ✭✭✭✭✭
    i only see lags when some crazy ball group login. Other than that, it's playable most of the time.

    So here for you another stuff to investigate: reducing group size from 24 to 12 in Cyro.
  • ZOS_Volpe
    ZOS_Volpe
    admin
    Greetings,

    We have gone ahead and closed this thread down since it is rather old. Please keep in mind that sometimes it is better to create a new topic, instead of replying in one that was posted a while ago.

    Thank you for your understanding.
    The Elder Scrolls Online: Tamriel Unlimited - ZeniMax Online Studios
    Forum Rules | Code of Conduct | Terms of Service | Home Page | Help Site
    Staff Post
This discussion has been closed.