Maintenance for the week of March 25:
• [COMPLETE] ESO Store and Account System for maintenance – March 28, 9:00AM EDT (13:00 UTC) - 12:00PM EDT (16:00 UTC)

Cyrodiil lag solution proposal.... Maybe

Lovndaddy
Lovndaddy
Now I’m no expert at coding by any means but how hard would it be for Zos to lower the population of Cyrodiil even if it’s just a little at a time till the game runs better until they find a fix I mean even if they have to open up new campaigns so be it
  • Giljabrar
    Giljabrar
    ✭✭✭✭
    They've done this in the past.

    You used to see a thousand people in a single campaign server after launch.

    Let's stop compromising, and demand the server upgrades that we've basically paid for.
  • technohic
    technohic
    ✭✭✭✭✭
    ✭✭✭✭✭
    This was earlier today. 3 bars for AD and EP, 2 DC. I posted it to show it's not network or video lag, look at my ping and frames; but it applies here. The ability you can see blink from press never went off. Maybe it's still too much for the server to handle. Should we take it down to 2 bar pop limit? I have a feeling that will not help as I hear battle grounds and PVE raids also have the issues at 12-24 players.


    https://www.youtube.com/watch?v=rXYMDukroKc

    I think the bottom line is it's either code or; server for the entire game, not just our instances; are overloaded and they need to invest resources
  • xshatox
    xshatox
    ✭✭✭✭
    I don’t think its just capacity problem, if ESO only has capacity problem lowering population might helped but if its code it will be difficult. From what i read they are still investigating on that end and might take a while (up to 2021 looking at their timeline) since they are mostly focus on content at the moment.
  • MincVinyl
    MincVinyl
    ✭✭✭✭✭
    Lowering the capacity of the server is just a blanket fix....much like how the blanket reduction in healing "fixes" the tank meta. There is no simple solution to something that has multiple variables causing the problem.
    1. Server hardware
    2. Anti-cheat system
    3. How dots/hots and aoe calculations work
    4. Over complicated sets/abilities/etc
    5. Cyrodil group meta (more players = more win)

    Lets say we have a current 30 player group hitting another 30 player group with aoes.....thats 900 calculations happening. Add in each player having to check 10 procs each....thats 9000. Now lets say that aoe has a dot attached and they all stack over 10sec.....9000+30players*30ticks*10sec*10procs = 99k calculations.

    Lets now say group zerging has been spread out by some change and those 30v30 can roughly hit 8 people with that aoe.....so that is 240 calculations. Now lets say procs are cut down in the game to each player having 3 procs...... 720 calculations. Now lets say that dots/hots do not stack(like how it used to be) and now there can only ever be 1 tick per player per second.....720+30players*1tick*10sec*3procs= 1620 calculations
    If only one of these gets solved we likely wont see a massive difference, but a chunk of lag removed. All of these issues cause the other issues to become exponentially worse. Inevitably these calculations will also get passed through the anti-cheat and the server has to deal with them. Zos has tried to fix the aoe issue before by doing an aoe cap. Which i suppose artificially blanket fixed one part of one of the issues(at the price of player enjoyment), but not the root of the issue. The root in the aoe case being how spread out players were.
  • Sheuib
    Sheuib
    ✭✭✭✭
    I would like it if they made one campaign that had a lower population limit. Some of us like playing on low population campaigns.

    We have gone through the cycles of 7 day vs 30 day and CP vs no CP. How about an iteration of low pop lock vs high pop lock. That way we all get to play on the type of campaign we like and it might be a good performance test as well.
  • gepe87
    gepe87
    ✭✭✭✭✭
    Lower pop cap and open more campaigns (cp and nocp) with both campaign lock and unlocked .

    At this point is better to have depopulated servers than non functional.
    Gepe, Dunmer MagSorc Pact Grand Overlord | Gaepe, Bosmer MagSorc Dominion General

    If you see edits on my replies: typos. English isn't my main language
  • idk
    idk
    ✭✭✭✭✭
    ✭✭✭✭✭
    As already noted, Zos has reduced the pop cap multiple times. While reducing the pop cap, in theory, reduces the server load but the real impact is large numbers of players at the same keep, not how many are actually in Cryodiil.

    In reality, Cyrodiill worked better before they reduced the pop cap than it does today. Zos has added more aspects to the game that have added to the server load as well as moving checks from the client to the server.

    In other words, it is not a good solution.
  • Alchimiste1
    Alchimiste1
    ✭✭✭✭✭
    Vinyl is completely right about the calculations and lag.
  • usmcjdking
    usmcjdking
    ✭✭✭✭✭
    ✭✭✭
    The only solution is as @MincVinyl pointed out.

    0331
    0602
  • MincVinyl
    MincVinyl
    ✭✭✭✭✭
    gepe87 wrote: »
    Lower pop cap and open more campaigns (cp and nocp) with both campaign lock and unlocked .

    At this point is better to have depopulated servers than non functional.

    You are proposing another blanket concept much like how zos fixed the "tank" meta that does not fix the blatant problem at hand.
    idk wrote: »
    In reality, Cyrodiill worked better before they reduced the pop cap than it does today. Zos has added more aspects to the game that have added to the server load as well as moving checks from the client to the server.

    From what I have been able to test a lot of the code behind the scenes is being introduced this patch, which we will have to see. They messed around with how damage calculations are done during the last pts (seen from a bug in cast time abilities missing a factor of 2 in part of the calculation).

    What concerned me was that they changed around how some proc sets work, removing the chance possibility. This might indicate that zos is looking to move things back client side, people could cheat those chance values but not change a time based cooldown value without eventually being caught.
  • Durham
    Durham
    ✭✭✭✭✭
    Get rid of ball group play it's the cancer that almost destroyed pvp in this game
    PVP DEADWAIT
    PVP The Unguildables
  • Crash427
    Crash427
    ✭✭✭✭✭
    Durham wrote: »
    Get rid of ball group play it's the cancer that almost destroyed pvp in this game

    You cant get rid of ball group play since a ball group is just a group that min/maxes together. You're always going to have that in any mmo
  • maddiniiLuna
    maddiniiLuna
    ✭✭✭✭
    MincVinyl wrote: »
    Lowering the capacity of the server is just a blanket fix....much like how the blanket reduction in healing "fixes" the tank meta. There is no simple solution to something that has multiple variables causing the problem.
    1. Server hardware
    2. Anti-cheat system
    3. How dots/hots and aoe calculations work
    4. Over complicated sets/abilities/etc
    5. Cyrodil group meta (more players = more win)

    Lets say we have a current 30 player group hitting another 30 player group with aoes.....thats 900 calculations happening. Add in each player having to check 10 procs each....thats 9000. Now lets say that aoe has a dot attached and they all stack over 10sec.....9000+30players*30ticks*10sec*10procs = 99k calculations.

    Lets now say group zerging has been spread out by some change and those 30v30 can roughly hit 8 people with that aoe.....so that is 240 calculations. Now lets say procs are cut down in the game to each player having 3 procs...... 720 calculations. Now lets say that dots/hots do not stack(like how it used to be) and now there can only ever be 1 tick per player per second.....720+30players*1tick*10sec*3procs= 1620 calculations
    If only one of these gets solved we likely wont see a massive difference, but a chunk of lag removed. All of these issues cause the other issues to become exponentially worse. Inevitably these calculations will also get passed through the anti-cheat and the server has to deal with them. Zos has tried to fix the aoe issue before by doing an aoe cap. Which i suppose artificially blanket fixed one part of one of the issues(at the price of player enjoyment), but not the root of the issue. The root in the aoe case being how spread out players were.


    Thank you for sharing this, it was interesting, but unfortunately completely missed. The game currently has an aoe cap - meaning you can only hit 6 people with an area of effect ability, regardless of if it's a heal or damaging ability. Of coarse you could argue, that from the 30 players (Let's call them group A) everybody has a different target in group B. This is highly unrealistic tho, as first of all not everybody will be in reach and furthermore people tend to target whatever is closest or low on health. Good players are able to pick out single targets on the flanks, but still it's not hitting all 30 players from group B.

    There is more then this factor that adds to the calculations and i do appreciate your share. The real value of these calculations is much much lower.

    Another thing i want to say is, that we have flying cars. 21 century and we literally have flying cars. I mean we can make cars fly, but we can't make a server that could do these calculations? Sounds absurd to me. A processor with 1.8 (Note that 1.8 GHz processors are already "antique", as we are slowly approaching 6Ghz) can handle 1,800,000,000 operations PER SECOND. Try to type something in Google. Anything. On the top you will see how many results were found and in which time. For example if you search for "google" - About 10,380,000,000 results (0.51 seconds).


    So what does all this mean? It means the server architecture can most likely handle the calculations thrown out by a stand-up in Cyrodiil. One of the reasons for those laggs is the update rate and this is practically impossible to adjust for everybody. So the update rate is how fast the server sends and receives updates. You see updates are a bit like your FPS. The more you have, the better it gets. Even when you don't know what FPS (Frames per seconds) is, you probably have experienced stuttering, slideshows, rapid movements - things like this to help understand what i mean. Basically if you had 30 updates per second there is a delay of 33 ms between you and the server. I couldn't explain why, it is just how it is. However with 60 updates per seconds you are decreasing the time it takes and it's now only 16.66 ms.
    So you have your ping + the update rate.



    So here is, where it gets tricky. You press a button, the action will be send to the server and from there to the client you have targeted. I think it's understandable so far? But you have more then one person hitting the same enemy. So every action from let's say 10 players go to the server and are send to the target. But the server can only send so much information at once, so it will que up. The abilites of person 1 - 3 are already processed, 4 - 6 are in the making and 7 - 10 are currently qued up. What happens? The game will send as much data as it can to the target in one packet, but don't forget the update rate. It will do this every 16.66 ms (this is not an actual number, as i don't know the update rate of the servers). Before summing this up, just a side note: 1 second is 1000 milliseconds. So from the moment you press the button you have your ping + update rate going to the server, then from the server update rate + target ping going to the target. Then you have a target answer like did your ability miss or did you stun him whatever i don't know. So to answer to you, the target sends back the information (ping + update rate), which goes to the server and from there to you.

    Pretend like you have a ping of 80 and your target has a ping of let's say 20, because there is always somebody.

    80 + 16 = 96 ms
    96 * 2 = 192 ms

    20 + 16 = 36 ms
    36 * 2 = 72 ms

    Total = 264 ms or (almost) one third of a second.

    I hope this makes sense so far, i'm trying my best. See as mentioned before you are not the only one hitting the target, right? Let's say others have a better ping then you do, so their actions get to the server earlier then yours. Your actions are qued up and send when ready. It still happens very very quickly, but what if you are the last person in the que? So you press a button literally the same time as 10 others, but they have better speed and internet can't do magic, so you have to wait 10 times 264 ms - that's 2640 ms and as we noted before 1 second has 1000 milliseconds. So it takes 2 full seconds from your button press to get a result, due to the amount of people pressing the button and that's where your lagg is.

    People that understood this will go AHA! Others will think hmm 2 seconds is not much. Just keep in mind, this is purely fictional numbers, okay?

    So you have 2 second delay from pressing your button. But what if... What if others can simply press faster then you? Or use macros? Animation cancel? Weaving? All this adds up. If somebody was using macros you'd have the ability + a light attack coming in at basically the same time, resulting in a fully packed package that contains the ability, light attack, maybe an execute from you and others and the result is, that you run in to a battle full health and in 2 seconds you loose your entire life bar and you think what the heck just happened?



    How to improve this situation? Well there is certain things you can do like targeting animation cancel (not saying it's a bad thing, just saying it doesn't help performance), aoe caps. Matters to reduce the que basically. You could also improve the update rate of the server, but again i don't know which one the server has - it might already be at it's peak performance. One really good action to take would be to drastically increase the measures against macro users, because humans are usually not as quick as a computer generated action. Reducing the amount of macros will significantly reduce the que. Other things could be something a fixed light attack damage (which we currently don't have, as it's based on weapon damage). All in all there is room for improvements, but only so much can actually be done. My best bet is to implement some sort of anti cheating system that scans your memory so you can have more actions on your local client and less on the server. There is reason why everything goes over the server and that is, because a client sided health bar for example is a feast for cheaters that will just cheat themselves infinite health.
  • Izanagi.Xiiib16_ESO
    Izanagi.Xiiib16_ESO
    ✭✭✭✭✭
    ✭✭✭
    MincVinyl wrote: »
    Lowering the capacity of the server is just a blanket fix....much like how the blanket reduction in healing "fixes" the tank meta. There is no simple solution to something that has multiple variables causing the problem.
    1. Server hardware
    2. Anti-cheat system
    3. How dots/hots and aoe calculations work
    4. Over complicated sets/abilities/etc
    5. Cyrodil group meta (more players = more win)

    Lets say we have a current 30 player group hitting another 30 player group with aoes.....thats 900 calculations happening. Add in each player having to check 10 procs each....thats 9000. Now lets say that aoe has a dot attached and they all stack over 10sec.....9000+30players*30ticks*10sec*10procs = 99k calculations.

    Lets now say group zerging has been spread out by some change and those 30v30 can roughly hit 8 people with that aoe.....so that is 240 calculations. Now lets say procs are cut down in the game to each player having 3 procs...... 720 calculations. Now lets say that dots/hots do not stack(like how it used to be) and now there can only ever be 1 tick per player per second.....720+30players*1tick*10sec*3procs= 1620 calculations
    If only one of these gets solved we likely wont see a massive difference, but a chunk of lag removed. All of these issues cause the other issues to become exponentially worse. Inevitably these calculations will also get passed through the anti-cheat and the server has to deal with them. Zos has tried to fix the aoe issue before by doing an aoe cap. Which i suppose artificially blanket fixed one part of one of the issues(at the price of player enjoyment), but not the root of the issue. The root in the aoe case being how spread out players were.


    Thank you for sharing this, it was interesting, but unfortunately completely missed. The game currently has an aoe cap - meaning you can only hit 6 people with an area of effect ability, regardless of if it's a heal or damaging ability.

    Sorry I didn't read your full post if you posted something different below but actually this first statement is false.
    AOE Healing abilities currently have a target cap for their healing. This is normally 6 although for some skills such as Barrier / Siege shield it is 12. (im counting SS damage mit as healing).

    AOE Damage abilities have a cap of 64. Some skills which have secondary effects still only affect 6 players but the damage from them goes to all - for example talons will root 6 players but damage up to 64. It was changed in 2017 iirc, this help article clearly hasn't been updated.
    @Solar_Breeze
    NA ~ Izanerys: Dracarys (Videos | Dracast Podcast)
    EU ~ Izanagi: Roleplay Circle (AOE Rats/ Zerg Squad / Banana Squad)
  • technohic
    technohic
    ✭✭✭✭✭
    ✭✭✭✭✭
    That was a lot of effort in a long post only to start off wrong.
  • NekoN3ko
    NekoN3ko
    ✭✭✭
    The lag spikes are insane! Almost like a mini DDoS. There is some dirty/inefficient code seeping through your product, whew....
  • idk
    idk
    ✭✭✭✭✭
    ✭✭✭✭✭
    MincVinyl wrote: »
    Lowering the capacity of the server is just a blanket fix....much like how the blanket reduction in healing "fixes" the tank meta. There is no simple solution to something that has multiple variables causing the problem.
    1. Server hardware
    2. Anti-cheat system
    3. How dots/hots and aoe calculations work
    4. Over complicated sets/abilities/etc
    5. Cyrodil group meta (more players = more win)

    Lets say we have a current 30 player group hitting another 30 player group with aoes.....thats 900 calculations happening. Add in each player having to check 10 procs each....thats 9000. Now lets say that aoe has a dot attached and they all stack over 10sec.....9000+30players*30ticks*10sec*10procs = 99k calculations.

    Lets now say group zerging has been spread out by some change and those 30v30 can roughly hit 8 people with that aoe.....so that is 240 calculations. Now lets say procs are cut down in the game to each player having 3 procs...... 720 calculations. Now lets say that dots/hots do not stack(like how it used to be) and now there can only ever be 1 tick per player per second.....720+30players*1tick*10sec*3procs= 1620 calculations
    If only one of these gets solved we likely wont see a massive difference, but a chunk of lag removed. All of these issues cause the other issues to become exponentially worse. Inevitably these calculations will also get passed through the anti-cheat and the server has to deal with them. Zos has tried to fix the aoe issue before by doing an aoe cap. Which i suppose artificially blanket fixed one part of one of the issues(at the price of player enjoyment), but not the root of the issue. The root in the aoe case being how spread out players were.


    Thank you for sharing this, it was interesting, but unfortunately completely missed.
    The game currently has an aoe cap - meaning you can only hit 6 people with an area of effect ability, regardless of if it's a heal or damaging ability. Of coarse you could argue, that from the 30 players (Let's call them group A) everybody has a different target in group B. This is highly unrealistic tho, as first of all not everybody will be in reach and furthermore people tend to target whatever is closest or low on health. Good players are able to pick out single targets on the flanks, but still it's not hitting all 30 players from group B.

    There is more then this factor that adds to the calculations and i do appreciate your share. The real value of these calculations is much much lower.

    Another thing i want to say is, that we have flying cars. 21 century and we literally have flying cars. I mean we can make cars fly, but we can't make a server that could do these calculations? Sounds absurd to me. A processor with 1.8 (Note that 1.8 GHz processors are already "antique", as we are slowly approaching 6Ghz) can handle 1,800,000,000 operations PER SECOND. Try to type something in Google. Anything. On the top you will see how many results were found and in which time. For example if you search for "google" - About 10,380,000,000 results (0.51 seconds).


    So what does all this mean? It means the server architecture can most likely handle the calculations thrown out by a stand-up in Cyrodiil. One of the reasons for those laggs is the update rate and this is practically impossible to adjust for everybody. So the update rate is how fast the server sends and receives updates. You see updates are a bit like your FPS. The more you have, the better it gets. Even when you don't know what FPS (Frames per seconds) is, you probably have experienced stuttering, slideshows, rapid movements - things like this to help understand what i mean. Basically if you had 30 updates per second there is a delay of 33 ms between you and the server. I couldn't explain why, it is just how it is. However with 60 updates per seconds you are decreasing the time it takes and it's now only 16.66 ms.
    So you have your ping + the update rate.



    So here is, where it gets tricky. You press a button, the action will be send to the server and from there to the client you have targeted. I think it's understandable so far? But you have more then one person hitting the same enemy. So every action from let's say 10 players go to the server and are send to the target. But the server can only send so much information at once, so it will que up. The abilites of person 1 - 3 are already processed, 4 - 6 are in the making and 7 - 10 are currently qued up. What happens? The game will send as much data as it can to the target in one packet, but don't forget the update rate. It will do this every 16.66 ms (this is not an actual number, as i don't know the update rate of the servers). Before summing this up, just a side note: 1 second is 1000 milliseconds. So from the moment you press the button you have your ping + update rate going to the server, then from the server update rate + target ping going to the target. Then you have a target answer like did your ability miss or did you stun him whatever i don't know. So to answer to you, the target sends back the information (ping + update rate), which goes to the server and from there to you.

    Pretend like you have a ping of 80 and your target has a ping of let's say 20, because there is always somebody.

    80 + 16 = 96 ms
    96 * 2 = 192 ms

    20 + 16 = 36 ms
    36 * 2 = 72 ms

    Total = 264 ms or (almost) one third of a second.

    I hope this makes sense so far, i'm trying my best. See as mentioned before you are not the only one hitting the target, right? Let's say others have a better ping then you do, so their actions get to the server earlier then yours. Your actions are qued up and send when ready. It still happens very very quickly, but what if you are the last person in the que? So you press a button literally the same time as 10 others, but they have better speed and internet can't do magic, so you have to wait 10 times 264 ms - that's 2640 ms and as we noted before 1 second has 1000 milliseconds. So it takes 2 full seconds from your button press to get a result, due to the amount of people pressing the button and that's where your lagg is.

    People that understood this will go AHA! Others will think hmm 2 seconds is not much. Just keep in mind, this is purely fictional numbers, okay?

    So you have 2 second delay from pressing your button. But what if... What if others can simply press faster then you? Or use macros? Animation cancel? Weaving? All this adds up. If somebody was using macros you'd have the ability + a light attack coming in at basically the same time, resulting in a fully packed package that contains the ability, light attack, maybe an execute from you and others and the result is, that you run in to a battle full health and in 2 seconds you loose your entire life bar and you think what the heck just happened?



    How to improve this situation? Well there is certain things you can do like targeting animation cancel (not saying it's a bad thing, just saying it doesn't help performance), aoe caps. Matters to reduce the que basically. You could also improve the update rate of the server, but again i don't know which one the server has - it might already be at it's peak performance. One really good action to take would be to drastically increase the measures against macro users, because humans are usually not as quick as a computer generated action. Reducing the amount of macros will significantly reduce the que. Other things could be something a fixed light attack damage (which we currently don't have, as it's based on weapon damage). All in all there is room for improvements, but only so much can actually be done. My best bet is to implement some sort of anti cheating system that scans your memory so you can have more actions on your local client and less on the server. There is reason why everything goes over the server and that is, because a client sided health bar for example is a feast for cheaters that will just cheat themselves infinite health.

    I would suggest their comment was far from a complete miss. Even if their calculations were slightly off that is far from a complete miss.

    What they point out very well, and seems to have been overlooked, is that Zos has made the came overly complex in some areas that have greatly impacted performance in Cyrodiil as it is now worse than it was when we have much larger pop caps. As such, there is not a simple solution.
  • DjinnAeternam
    DjinnAeternam
    ✭✭✭
    Lowering population doesn't fix it, because the servers can't hold a good 20v20 fight, even if you consider 2 groups GvG, like 12v12 or 10v10, server doesn't catch up with skills/damage and all sort of calculations.

    So having a Cyro instance with 20 player cap per faction wouldn't even solve it if that 20 were mostly ballgroups.
Sign In or Register to comment.