[Server Lag Explained] Server calculations(skill actions) are not the main reason for lag.

RouDeR · September 2019

First i would like to introduce you to the Server ticks terminology.
What are server ticks?
This is the amount of time per second that the server is sending data to the client (players game).
Supposedly ESO servers are working on 60 or 64 ticks, how does this transfer to gameplay?

Let's say that you are alone in an area and you are doing nothing but standing still > disregarding that you are not doing ANYTHING you are still receiving 64 data rows per second containing information about your location/state/visuals.

Now let's say that another player comes by to you > here is what happens:

You will receive your 64 rows of data per second + another 64 rows of data per second because of the second player, so this is 128 rows of data per second for you AND another 128 rows of data per second for the other player which is total 256 rows of data per second sent by the server:

Here is detailed explanation: https://www.nbnco.com.au/blog/entertainment/what-is-tick-rate-and-what-does-it-do

Here is a small showcase in image:

Since Cyrodiil is the largest Zone in the game and have the largest player capacity (600) per instance if i`m not mistaken this results in huge number of player interactions and for some reason there appears to be a bottleneck which i conclude might be one of the following 2 reasons:

Reason 1: network limitation bottleneck by the server side (Every PvP server instance might have dedicated network bandwith that it exceeds in huge fights)

Reason 2: client code(game) is not optimized to receive large amount of data sent by the server side, therefore there is a queue happening( lag ) when receiving information

Possible solution (dirty fix)
Reduce the server ticks to 30/32 ticks per second - since this is not a Shooter game we (the players) won't feel significant (if any) impact to our gameplay, but this will literally be reducing the lag in half.

systema555 · September 2019

It would be so comical that the player base would come up with a better solution than ZOS dev team on issues plaguing TESO

or not...

Inzebaba
EJ Lord commander

RouDeR · September 2019

Ye but, unfortunately everyone prefer to spam meaningless and non technical accurate stuff like "Cut groups in half" add/remove AoE caps, remove CP, remove Animation canceling - as a reasons for the server lag ..

Aznarb · September 2019

RouDeR wrote: »

Ye but, unfortunately everyone prefer to spam meaningless and non technical accurate stuff like "Cut groups in half" add/remove AoE caps, remove CP, remove Animation canceling - as a reasons for the server lag ..

You forget the more beautiful one : "nerf healing"

lpw · September 2019

This is very informative and I'm sure the issue would have to fixed in a few different ways.

The purpose of the thread I started was more so about smart heals and people spamming mutagen than 24 man groups.

Are you seriously saying that processing a skill that has to heal two people out of a pool of 12 would be no quicker for any part of the whole system than a skill that has to heal 2 out of 400? This will take no more work/code and is not a less complicated problem to solve?

redlink1979 · September 2019

The problem has always been how servers handle the data. Your suggestion is very plausible since we saw the problem escalate with the increasing of player amount.

RouDeR · September 2019

lpw wrote: »

The purpose of the thread I started was more so about smart heals and people spamming mutagen than 24 man groups.

Are you seriously saying that processing a skill that has to heal two people out of a pool of 12 would be no quicker for any part of the whole system than a skill that has to heal 2 out of 400? This will take no more work/code and is not a less complicated problem to solve?

This thread is about network bandwith (sent by server and received by client) which has nothingning in common with server calculations.
A regular 500 CPU can perform Milions of calculations every minute, just thinking that a server machine will not be able to do a simple math is a pure nonsense.

TequilaFire · September 2019

ESO also uses TCP requiring client to acknowledge each packet has been received instead of UDP which uses checksums instead. TCP may be less prone to corruption but UDP is faster and is more common in online gaming.

lpw · September 2019

I understand they're two different issues and while your OP is informative and warranted, you pretty much just rubbished everyone elses suggestions/input.

I'll stay out of this thread but you didn't really answer my question.

mairwen85 · September 2019

There would be risk of a greater semblance of lag at fixed lower tick as client side prediction would easily desynchronise (essentially, in your model this is your suggested reason for lag, net bottleneck causing client prediction to over compensate, or allow only tick truthiness to display)—false prediction and placement could already be part of the trouble. Intelligent tick rates that compensate for connection latency, ie higher or lower depending on user connection have been proven to work better, but provide a much more obvious advantage for those in the higher end group. I think the problem is multipart, tcp is a primary culprit for the overhead in unpacking, verifying and progressing data packages in double handshakes, server tick rate non adaptive (possibly by design) for the packet size, overly vigilant client side prediction for desync, and badly optimised calculations buried in anti cheat interception code for delay in server side processing.

zaria · September 2019

RouDeR wrote: »

Since Cyrodiil is the largest Zone in the game and have the largest player capacity (600) per instance if i`m not mistaken this results in huge number of player interactions and for some reason there appears to be a bottleneck which i conclude might be one of the following 2 reasons:

Reason 1: network limitation bottleneck by the server side (Every PvP server instance might have dedicated network bandwith that it exceeds in huge fights)

Reason 2: client code(game) is not optimized to receive large amount of data sent by the server side, therefore there is a queue happening( lag ) when receiving information

Possible solution (dirty fix)
Reduce the server ticks to 30/32 ticks per second - since this is not a Shooter game we (the players) won't feel significant (if any) impact to our gameplay, but this will literally be reducing the lag in half.

Overwatch drops ticks for users with low bandwidth to avoid dropped packages, this is worse than fewer if the last packages piles up and get lost.
I assume server simply group up status of players into blocks the size of visual range in the game check board style, then send send the needed blocks to players. You send extra blocks as this is ready assembled data and you just has to send it.
Client use this to place players, npc, lootable stuff and effects in the world then drop stuff it don't need to render like enemies behind an wall.
Are you sure ESO uses 64 ticks?
And no this is not an major issue for the server network. if it is add an 10Gb/s network card.

This is pretty easy to test, go to Alkir desert then you have lots of dolmen zergs running.
If you are well away from the zergs farming or questing is performance normal?
If not its an server issue. Server has problems handling the actions from all the players, this will affect all players in zone.

If performance is normal follow an large zerg group. Do an dolmen with them, performance drops obviously.
Now move a bit away while still in visual range and turn away from the action. How is performance?
If back to standard the issue is your graphic card, if not its the cpu and bad programming.
Yes ZoS could start dropping effect if they all bluer together.Simply reducing quality in Cyrodil and trials would help a lot.

zaria · September 2019

RouDeR wrote: »

lpw wrote: »

The purpose of the thread I started was more so about smart heals and people spamming mutagen than 24 man groups.

Are you seriously saying that processing a skill that has to heal two people out of a pool of 12 would be no quicker for any part of the whole system than a skill that has to heal 2 out of 400? This will take no more work/code and is not a less complicated problem to solve?

This thread is about network bandwith (sent by server and received by client) which has nothingning in common with server calculations.
A regular 500 CPU can perform Milions of calculations every minute, just thinking that a server machine will not be able to do a simple math is a pure nonsense.

However its not simple math, you might have over 100 effects on you at once many are circles you move between.
Assume the reason to not let you change bars if group is in combat is probably an performance issue, you can then bake in all the buffs from cp, gear, bars and other into the database and skip lots of calculations.

Jeremy · September 2019

RouDeR wrote: »

First i would like to introduce you to the Server ticks terminology.
What are server ticks?
This is the amount of time per second that the server is sending data to the client (players game).
Supposedly ESO servers are working on 60 or 64 ticks, how does this transfer to gameplay?

Let's say that you are alone in an area and you are doing nothing but standing still > disregarding that you are not doing ANYTHING you are still receiving 64 data rows per second containing information about your location/state/visuals.

Now let's say that another player comes by to you > here is what happens:

You will receive your 64 rows of data per second + another 64 rows of data per second because of the second player, so this is 128 rows of data per second for you AND another 128 rows of data per second for the other player which is total 256 rows of data per second sent by the server:

Here is detailed explanation: https://www.nbnco.com.au/blog/entertainment/what-is-tick-rate-and-what-does-it-do

Here is a small showcase in image:

Since Cyrodiil is the largest Zone in the game and have the largest player capacity (600) per instance if i`m not mistaken this results in huge number of player interactions and for some reason there appears to be a bottleneck which i conclude might be one of the following 2 reasons:

Reason 1: network limitation bottleneck by the server side (Every PvP server instance might have dedicated network bandwith that it exceeds in huge fights)

Reason 2: client code(game) is not optimized to receive large amount of data sent by the server side, therefore there is a queue happening( lag ) when receiving information

Possible solution (dirty fix)
Reduce the server ticks to 30/32 ticks per second - since this is not a Shooter game we (the players) won't feel significant (if any) impact to our gameplay, but this will literally be reducing the lag in half.

Yeah I seriously doubt "calculations" is the cause either. If the system struggles to do basic math formulas then we're all in trouble and I wonder how this game even functions.

But it's probably a mixture of both - bottlenecking and a lot of corruption occurring in the code. I'm unfamiliar with your proposed solution - but it sounds interesting.

Facefister · September 2019

It's 2019, nearly 2020. Server issues shouldn't be excused. Other MMOs handle loads fine, why not ZoS?

SirAndy · September 2019

RouDeR wrote: »

... Let's say that you are alone in an area and you are doing nothing but standing still > disregarding that you are not doing ANYTHING you are still receiving 64 data rows per second containing information about your location/state/visuals ...

That's not how proper high traffic client/server (game) servers function ...

InvictusApollo · September 2019

SirAndy wrote: »

RouDeR wrote: »

... Let's say that you are alone in an area and you are doing nothing but standing still > disregarding that you are not doing ANYTHING you are still receiving 64 data rows per second containing information about your location/state/visuals ...

That's not how proper high traffic client/server (game) servers function ...

Please elaborate.

SirAndy · September 2019

InvictusApollo wrote: »

SirAndy wrote: »

RouDeR wrote: »

... Let's say that you are alone in an area and you are doing nothing but standing still > disregarding that you are not doing ANYTHING you are still receiving 64 data rows per second containing information about your location/state/visuals ...

That's not how proper high traffic client/server (game) servers function ...

Please elaborate.

@InvictusApollo

Disclaimer:
I've worked in the gaming industry for many years (As a programmer, system architect and development manager) and i have built several (successful) high client count / data volume systems.

A good client/server system only sends data when a state changes. When you are sitting idle, there is no need to send anything from you to the server for processing which also means there's no need for the server to send anything about you to anyone else around you.
The server knows your last state and only needs updating when that state changes.

Furthermore, data is usually sent via UDP without ACK to speed up processing times. That means that the server needs some sort of prediction to account for connection lag and possible data loss. That also means that the clients also need some sort of local prediction for the same reasons.

When implemented correctly, only a small amount of data needs to be transmitted from/to each client and the server maintains a "ground truth" state of the game world. Ground truth here means that any client side prediction can be invalidated and overwritten by the server if needed (usually happens when one or more clients have a laggy connection or high ping).

If done correctly, that also means that a high ping/laggy client can *never* slow down other players in the world since the server does *not* wait for client data to arrive to compute state changes, make predictions or broadcast updates to clients.

TequilaFire · September 2019

Your keywords were "proper" and "UDP".

usmguy1234 · September 2019

SirAndy wrote: »

InvictusApollo wrote: »

SirAndy wrote: »

RouDeR wrote: »

... Let's say that you are alone in an area and you are doing nothing but standing still > disregarding that you are not doing ANYTHING you are still receiving 64 data rows per second containing information about your location/state/visuals ...

That's not how proper high traffic client/server (game) servers function ...

Please elaborate.

@InvictusApollo

Disclaimer:
I've worked in the gaming industry for many years (As a programmer, system architect and development manager) and i have built several (successful) high client count / data volume systems.

A good client/server system only sends data when a state changes. When you are sitting idle, there is no need to send anything from you to the server for processing which also means there's no need for the server to send anything about you to anyone else around you.
The server knows your last state and only needs updating when that state changes.

Furthermore, data is usually sent via UDP without ACK to speed up processing times. That means that the server needs some sort of prediction to account for connection lag and possible data loss. That also means that the clients also need some sort of local prediction for the same reasons.

When implemented correctly, only a small amount of data needs to be transmitted from/to each client and the server maintains a "ground truth" state of the game world. Ground truth here means that any client side prediction can be invalidated and overwritten by the server if needed (usually happens when one or more clients have a laggy connection or high ping).

If done correctly, that also means that a high ping/laggy client can *never* slow down other players in the world since the server does *not* wait for client data to arrive to compute state changes, make predictions or broadcast updates to clients.

TequilaFire wrote: »

Your keywords were "proper" and "UDP".

BINGO!

Gilvoth · September 2019

if this thread were created by a developer with the Zenimax loggo and developer title on the writers name i would have more belief in what is presented.
it should be noted that this thread was created with a detailed agenda in mind.

wavingblue · September 2019

TequilaFire wrote: »

ESO also uses TCP requiring client to acknowledge each packet has been received instead of UDP which uses checksums instead. TCP may be less prone to corruption but UDP is faster and is more common in online gaming.

Not only does it have to acknowledge every packet but if a packet is missed for whatever reason it has to request re-transmission, halt everything done with that thread, get the packet and then process the backlog. This of course creates a spiral of processing which is fine at 60 to 80% utilization but start reaching heights of 90..95...near 100% utilization on a processor and data link.. and it becomes a death spiral.

Would love if they switched to UDP... that would free up significant resources, which probably means they won't do it.

Also not sure size of TCP packets in game but they are typically 20 to 60 bytes vs 8 bytes for UDP, that's a fair amount of extra processing overhead to get to the actual game message that needs to be acted on.

InvictusApollo · September 2019

SirAndy wrote: »

InvictusApollo wrote: »

SirAndy wrote: »

RouDeR wrote: »

... Let's say that you are alone in an area and you are doing nothing but standing still > disregarding that you are not doing ANYTHING you are still receiving 64 data rows per second containing information about your location/state/visuals ...

That's not how proper high traffic client/server (game) servers function ...

Please elaborate.

@InvictusApollo

Disclaimer:
I've worked in the gaming industry for many years (As a programmer, system architect and development manager) and i have built several (successful) high client count / data volume systems.

A good client/server system only sends data when a state changes. When you are sitting idle, there is no need to send anything from you to the server for processing which also means there's no need for the server to send anything about you to anyone else around you.
The server knows your last state and only needs updating when that state changes.

Furthermore, data is usually sent via UDP without ACK to speed up processing times. That means that the server needs some sort of prediction to account for connection lag and possible data loss. That also means that the clients also need some sort of local prediction for the same reasons.

When implemented correctly, only a small amount of data needs to be transmitted from/to each client and the server maintains a "ground truth" state of the game world. Ground truth here means that any client side prediction can be invalidated and overwritten by the server if needed (usually happens when one or more clients have a laggy connection or high ping).

If done correctly, that also means that a high ping/laggy client can *never* slow down other players in the world since the server does *not* wait for client data to arrive to compute state changes, make predictions or broadcast updates to clients.

Thank you.

One thing caught my interest:

SirAndy wrote: »

If done correctly, that also means that a high ping/laggy client can *never* slow down other players in the world since the server does *not* wait for client data to arrive to compute state changes, make predictions or broadcast updates to clients.

You are saying that incorectly implemented client-server system may allow one player to give high ping to other players.
I often felt that I experienced this on some ocassions. I struggled to adapt to new lag that enemies introduced, while they were in their element.
Have you experienced something similar in ESO?

mairwen85 · September 2019

InvictusApollo wrote: »

SirAndy wrote: »

InvictusApollo wrote: »

SirAndy wrote: »

RouDeR wrote: »

... Let's say that you are alone in an area and you are doing nothing but standing still > disregarding that you are not doing ANYTHING you are still receiving 64 data rows per second containing information about your location/state/visuals ...

That's not how proper high traffic client/server (game) servers function ...

Please elaborate.

@InvictusApollo

Disclaimer:
I've worked in the gaming industry for many years (As a programmer, system architect and development manager) and i have built several (successful) high client count / data volume systems.

A good client/server system only sends data when a state changes. When you are sitting idle, there is no need to send anything from you to the server for processing which also means there's no need for the server to send anything about you to anyone else around you.
The server knows your last state and only needs updating when that state changes.

Furthermore, data is usually sent via UDP without ACK to speed up processing times. That means that the server needs some sort of prediction to account for connection lag and possible data loss. That also means that the clients also need some sort of local prediction for the same reasons.

When implemented correctly, only a small amount of data needs to be transmitted from/to each client and the server maintains a "ground truth" state of the game world. Ground truth here means that any client side prediction can be invalidated and overwritten by the server if needed (usually happens when one or more clients have a laggy connection or high ping).

If done correctly, that also means that a high ping/laggy client can *never* slow down other players in the world since the server does *not* wait for client data to arrive to compute state changes, make predictions or broadcast updates to clients.

Thank you.

One thing caught my interest:

SirAndy wrote: »

If done correctly, that also means that a high ping/laggy client can *never* slow down other players in the world since the server does *not* wait for client data to arrive to compute state changes, make predictions or broadcast updates to clients.

You are saying that incorectly implemented client-server system may allow one player to give high ping to other players.
I often felt that I experienced this on some ocassions. I struggled to adapt to new lag that enemies introduced, while they were in their element.
Have you experienced something similar in ESO?

I have. Recently in Selene's Web, I swear that for the whole run fps was jittery and ping was higher than normal (usually @ 200ms, but this time @ 300ms) with spikes to 500 every time I went near a certain player. At the end of the run, I tested this by going as far as possible and then close to this player and was amazed at the coincidence that it shot through the roof every time I got near him. After he left the group and ported out, fps shot up to 60 and ping settled back @ 200ms; I was willing to brush it off as pure coincidence that I was having a bad network patch for the exact duration up to that moment and apparently in the exact moments I was in proximity of the other player -- but something at the back of my mind has been playing on me since that it would be one hell of a coincidence

There have been similar instances too -- and oddly enough, every time I get this impression, the other player happens to be a necromancer... not sure if it means anything at all.

Vapirko · September 2019

But then how do you explain really bad lag in small fights or even 1v1 and sometimes with only 1/2 bars of population?

InvictusApollo · September 2019

Vapirko wrote: »

But then how do you explain really bad lag in small fights or even 1v1 and sometimes with only 1/2 bars of population?

Maybe lag isn't a derivative of only the number of players involved but also the quality of their connection to game server and the number of their actions per second? Maybe bad implementation of client-server system lets laggy players, who are far away from the server and have poor connection, increase lag to other players?

TequilaFire · September 2019

InvictusApollo wrote: »

Vapirko wrote: »

But then how do you explain really bad lag in small fights or even 1v1 and sometimes with only 1/2 bars of population?

Maybe lag isn't a derivative of only the number of players involved but also the quality of their connection to game server and the number of their actions per second? Maybe bad implementation of client-server system lets laggy players, who are far away from the server and have poor connection, increase lag to other players?

I have noticed this particularly when in a group with certain members causing more lag and loading screens, even more so when that member appears invisible to the group only showing status bar.

randomkeyhits · September 2019

wavingblue wrote: »

TequilaFire wrote: »

ESO also uses TCP requiring client to acknowledge each packet has been received instead of UDP which uses checksums instead. TCP may be less prone to corruption but UDP is faster and is more common in online gaming.

Not only does it have to acknowledge every packet but if a packet is missed for whatever reason it has to request re-transmission, halt everything done with that thread, get the packet and then process the backlog. This of course creates a spiral of processing which is fine at 60 to 80% utilization but start reaching heights of 90..95...near 100% utilization on a processor and data link.. and it becomes a death spiral.

Would love if they switched to UDP... that would free up significant resources, which probably means they won't do it.

Also not sure size of TCP packets in game but they are typically 20 to 60 bytes vs 8 bytes for UDP, that's a fair amount of extra processing overhead to get to the actual game message that needs to be acted on.

With TCP transmission is synchronous but delivery is not, packets can arrive out of order and stored in the receive buffer until whatever the missing packet is arrives. This happens all the time. Lost packet does not cause connection flooding as its a controlled function with increasing delay between retries and a limit on retry attempts.

I'm assuming the utilisation you mention is that of the ethernet connection? In which case sure, getting to 95% increases the number of collisions and hence retries but if your connection is that saturated, boy.... you've got a whole bunch of problems anyway.

The TCP stacks in most OS are pretty refined these days and very tunable so you can usually get almost every last drop of performance from a server. Most issues are around how servers/clients are coded to handle the exception situations as quite often they simply dont.

Last thing, while I generally am in favour of UDP for MMO connections over TCP it does come with a whole bunch of different challenges. Would ZoS get it right? maybe. Not sure I'd want to risk the game over another maybe.

SirAndy · September 2019

InvictusApollo wrote: »

One thing caught my interest:

SirAndy wrote: »

If done correctly, that also means that a high ping/laggy client can *never* slow down other players in the world since the server does *not* wait for client data to arrive to compute state changes, make predictions or broadcast updates to clients.

You are saying that incorectly implemented client-server system may allow one player to give high ping to other players.
I often felt that I experienced this on some ocassions. I struggled to adapt to new lag that enemies introduced, while they were in their element.
Have you experienced something similar in ESO?

Yes, unfortunately the way the client/server interaction is implemented in ESO it *DOES* allow for a laggy client to slow down server processing and thus introducing lag for other players in the immediate area. And yes, this can be done intentionally.

SirAndy · September 2019

randomkeyhits wrote: »

Last thing, while I generally am in favour of UDP for MMO connections over TCP it does come with a whole bunch of different challenges. Would ZoS get it right? maybe. Not sure I'd want to risk the game over another maybe.

But all the "challenges" that come with UDP have already been solved ad nauseum by developers going back decades.
Heck, just download the original quake source code and have a look at the base principles of using client/server prediction with UDP transfer. Most of those mechanics still apply and have been refined by many, many other publishers.

There's no shortage of code examples about how to do this correctly ...

SirAndy · September 2019

SirAndy wrote: »

... Heck, just download the original quake source code and have a look at the base principles of using client/server prediction with UDP transfer ...

Actually, one of the fastest and best optimized "netcode" examples can be downloaded for free and that is the source code for "Quake 3 Team Arena". It also includes very good client/server prediction code.

brandonv516 · September 2019

SirAndy wrote: »

InvictusApollo wrote: »

One thing caught my interest:

SirAndy wrote: »

If done correctly, that also means that a high ping/laggy client can *never* slow down other players in the world since the server does *not* wait for client data to arrive to compute state changes, make predictions or broadcast updates to clients.

You are saying that incorectly implemented client-server system may allow one player to give high ping to other players.
I often felt that I experienced this on some ocassions. I struggled to adapt to new lag that enemies introduced, while they were in their element.
Have you experienced something similar in ESO?

Yes, unfortunately the way the client/server interaction is implemented in ESO it *DOES* allow for a laggy client to slow down server processing and thus introducing lag for other players in the immediate area. And yes, this can be done intentionally.

This explains a lot as to why certain players are capable of easily exploiting the desync issues with Snipe, while with others it can actually be countered.

There are only 2 guys that I seem to have reoccurring issues with, and I have a sneaking suspicion they are not located anywhere near the NA servers and their connection is junk.