Maintenance for the week of December 15:
• PC/Mac: NA and EU megaservers for maintenance – December 15, 4:00AM EST (9:00 UTC) - 12:00PM EST (17:00 UTC)
• Xbox: NA and EU megaservers for maintenance – December 15, 4:00AM EST (9:00 UTC) - 12:00PM EST (17:00 UTC)
• PlayStation®: NA and EU megaservers for maintenance – December 15, 4:00AM EST (9:00 UTC) - 12:00PM EST (17:00 UTC)
• PC/Mac: NA and EU megaservers for maintenance – December 15, 4:00AM EST (9:00 UTC) - 12:00PM EST (17:00 UTC)
• Xbox: NA and EU megaservers for maintenance – December 15, 4:00AM EST (9:00 UTC) - 12:00PM EST (17:00 UTC)
• PlayStation®: NA and EU megaservers for maintenance – December 15, 4:00AM EST (9:00 UTC) - 12:00PM EST (17:00 UTC)
Lag is a bandwidth issue, not a server issue
vgastel
✭✭✭
One thing about lag in this game, in particular in PvP, is that it is roughly constant at about 3 seconds. This rather strongly points to a bandwidth issue in the client-server communication, rather than too weak servers. If the server itself can't keep up, the lag would greatly vary depending whether a big siege or fight is going on or not.
The bandwidth is a measure for a maximum amount of data traffic between clients and server per unit of time. When the amount of data traffic exceeds the bandwidth, it gets queued, like in a real traffic jam, resulting in lag of roughly constant duration.
The lag in Cyrodiil only reduces when the campaign is empty, i.e., in the morning when only a few are playing and the bandwidth limit is not reached.
New servers may thus not help if not enough bandwidth is available and the data can't reach the server (and the clients) fast enough.
One way to reduce bandwidth problems would be to scrutinize the data packets between clients and server, eliminate redundancy, minimize packet size, check that noone found a way to prioritize their packet over those of others (resulting in some players skills going off and others' skill don't).
My 5 cents worth
.
Oh, and of course, Happy Newyear!
Edited by vgastel on December 30, 2021 9:16PM The bandwidth is a measure for a maximum amount of data traffic between clients and server per unit of time. When the amount of data traffic exceeds the bandwidth, it gets queued, like in a real traffic jam, resulting in lag of roughly constant duration.
The lag in Cyrodiil only reduces when the campaign is empty, i.e., in the morning when only a few are playing and the bandwidth limit is not reached.
New servers may thus not help if not enough bandwidth is available and the data can't reach the server (and the clients) fast enough.
One way to reduce bandwidth problems would be to scrutinize the data packets between clients and server, eliminate redundancy, minimize packet size, check that noone found a way to prioritize their packet over those of others (resulting in some players skills going off and others' skill don't).
My 5 cents worth
.Oh, and of course, Happy Newyear!
7
-
fred4✭✭✭✭✭
✭✭✭I will grant you it could be a bandwidth issue within ZOS' internal network, but my bet is still that Cyrodiil gets bottlenecked on a single server thread. Bit of a moot point, though. We could speculate all day, but only ZOS can do actual performance measurements.2 -
AuraStorm43✭✭✭✭✭Its a server issue judging by the improvement we get during midyear when they rent better servers19
-
Elsonso✭✭✭✭✭
✭✭✭✭✭I will grant you it could be a bandwidth issue within ZOS' internal network, but my bet is still that Cyrodiil gets bottlenecked on a single server thread. Bit of a moot point, though. We could speculate all day, but only ZOS can do actual performance measurements.
Internet lag is always going to be one of the reasons that there is lag in the game. The variety of ISPs around the world almost guarantees this.
I am thinking, based on something Lambert said, that Cyrodiil performance is about the database and how characters are stored and referenced. Everything about this game is database, though. On some level, an MMO is basically a fancy way of sharing database information between large numbers of players. XBox EU/NA:@ElsonsoJannus
PC NA/EU: @Elsonso
PSN NA/EU: @ElsonsoJannus
Total in-game hours: 11321
X/Twitter: ElsonsoJannus0 -
Reverb✭✭✭✭✭
✭✭✭✭✭It’s a net code issue.Battle not with monsters, lest ye become a monster, and if you gaze into the abyss, the abyss gazes also into you. ~Friedrich Nietzsche4 -
Merlin13KAGL✭✭✭✭✭
✭✭✭✭Bandwidth: You're doing it wrong.
Queued traffic would continue to add up and eventually get dropped entirely. The delay would get progressively worse.
In your traffic jam analogy, the line of cars waiting to get on the interstate continues to get longer and longer. They don't pause for a set amount of time and then suddenly resume travel at top speed.Just because you don't like the way something is doesn't necessarily make it wrong...
Earn it.
IRL'ing for a while for assorted reasons, in forum, and in game.I am neither warm, nor fuzzy...
Probably has checkbox on Customer Service profile that say High Aggro, 99% immunity to BS2 -
vgastel✭✭✭Merlin13KAGL wrote: »Bandwidth: You're doing it wrong.
Queued traffic would continue to add up and eventually get dropped entirely. The delay would get progressively worse.
In your traffic jam analogy, the line of cars waiting to get on the interstate continues to get longer and longer. They don't pause for a set amount of time and then suddenly resume travel at top speed.
Isn't getting dropped what is happening when one loses lobby connection? In this way, the number of packets (cars) cannot go uncontrollably high?0 -
Thrasher91604✭✭✭✭✭And lag does go up appreciably in large battles. Anyone who plays much in Cyrodiil notices it. Heck you can detect a big Zerg coming just by the increase in lag.9 -
Idinuse✭✭✭✭✭
✭✭I will grant you it could be a bandwidth issue within ZOS' internal network, but my bet is still that Cyrodiil gets bottlenecked on a single server thread. Bit of a moot point, though. We could speculate all day, but only ZOS can do actual performance measurements.
Internet lag is always going to be one of the reasons that there is lag in the game. The variety of ISPs around the world almost guarantees this.
I am thinking, based on something Lambert said, that Cyrodiil performance is about the database and how characters are stored and referenced. Everything about this game is database, though. On some level, an MMO is basically a fancy way of sharing database information between large numbers of players.
The thing is that I play other MMO games with mostly zero lag. On the same hardware and on the same internet connection. Also, looking at the traffic going back and forth during gameplay, it's in the hundreds of bytes, not megabytes, maybe some Kb at times. To me it suggests on server calculation and net code issues.Sed ut perspiciatis unde omnis iste natus error sit voluptatem accusantium dolorem que laudantium, totam rem aperiam, eaque ipsa quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt explicabo. Nemo enim ipsam voluptatem quia voluptas sit aspernatur aut odit aut fugit, sed quia consequuntur magni dolores eos qui ratione voluptatem sequi nesciunt. Neque porro quisquam est, qui dolorem ipsum quia dolor sit amet, consectetur, adipisci velit, sed quia non numquam eius modi tempora incidunt ut labore et dolore magnam aliquam quaerat voluptatem. Ut enim ad minima veniam, quis nostrum exercitationem ullam corporis suscipit laboriosam, nisi ut aliquid ex ea commodi consequatur? Quis autem vel eum iure reprehenderit qui in ea voluptate velit esse quam nihil molestiae consequatur, vel illum qui dolorem eum fugiat quo voluptas nulla pariatur?6 -
notyuu✭✭✭✭✭
✭✭Given that during the PvP event cyro becomes smooth as butter and 120% lag free (at the cost of making trials basically unplayable) I don't think that's entirely the cause....10 -
sajackson✭✭✭I believe ZOS have said previously that they know where the issue is, but that it's not an easy one to fix.
Network latency could account for a small degree of lag, but my understanding is that players are experiencing abilities not firing at all at times in PvP zones which would point to sync issues in the client & server code.6 -
DinoZavr✭✭✭✭✭
✭One thing about lag in this game, in particular in PvP, is that it is roughly constant at about 3 seconds. This rather strongly points to a bandwidth issue in the client-server communication, rather than too weak servers.
Happy New Year to you too, Vgastel
As for lags root cause, i am afraid, you might be wrong.
Try connecting at the most desert time (i play at PC EU so like 5am..6AM GMT is the right time).
My pingplotter shows 45..50 mS round trip (e.g. like 23-25 mS one side), while ingame ping shows 120 .. 150 mS at the very same time. This means, time to process packets by server is greater than network delays. Same story with connecting at the peak time: ping still shows 45 .. 50 mS, which indicate network is not the issue (as with severe queuing we'd have noticeable ping delays as well, because echo packets would also have to wait in the queue along with other packets).
Load balancers show some serious packet losses, though i believe the effect is causes by the fact they are not using session persistence.
Also game-makers experimenting with proc sets and group sizes shows the queues are at CPU cores, rather than at Network Interface cards.
Normally, NICs are hardly the most utilized components in modern computing systems, hard disks and CPU are bottlenecks, provided we have enough RAM to cache everything relevant from disks and avoid swapping at all.PC EU4 -
vgastel✭✭✭One thing about lag in this game, in particular in PvP, is that it is roughly constant at about 3 seconds. This rather strongly points to a bandwidth issue in the client-server communication, rather than too weak servers.
Happy New Year to you too, Vgastel
As for lags root cause, i am afraid, you might be wrong.
Try connecting at the most desert time (i play at PC EU so like 5am..6AM GMT is the right time).
My pingplotter shows 45..50 mS round trip (e.g. like 23-25 mS one side), while ingame ping shows 120 .. 150 mS at the very same time. This means, time to process packets by server is greater than network delays. Same story with connecting at the peak time: ping still shows 45 .. 50 mS, which indicate network is not the issue (as with severe queuing we'd have noticeable ping delays as well, because echo packets would also have to wait in the queue along with other packets).
Load balancers show some serious packet losses, though i believe the effect is causes by the fact they are not using session persistence.
Also game-makers experimenting with proc sets and group sizes shows the queues are at CPU cores, rather than at Network Interface cards.
Normally, NICs are hardly the most utilized components in modern computing systems, hard disks and CPU are bottlenecks, provided we have enough RAM to cache everything relevant from disks and avoid swapping at all.
Yes, that makes sense. Ok, point taken . I'll gladly be wrong if it helps getting closer to a solution. It has been too quiet for too long about this after all. 1 -
Elsonso✭✭✭✭✭
✭✭✭✭✭The thing is that I play other MMO games with mostly zero lag. On the same hardware and on the same internet connection. Also, looking at the traffic going back and forth during gameplay, it's in the hundreds of bytes, not megabytes, maybe some Kb at times. To me it suggests on server calculation and net code issues.
It is not the same internet connection. There are different distances, and probably even different service providers involved.
I like to describe it as driving to the grocery store. We don't drive right out of our driveway and into the store parking lot. There are a lot of things between the two. The only thing we control is our driveway, where we start and end our trip. The only thing the grocery store controls is their parking lot. The in-between has traffic, which can vary by time of day, road construction, stoplights, and even poor quality roads.
The internet connection isn't exactly like driving to the grocery store, but it shares many similarities. ZOS can control how much traffic is transferred and how their servers handle that traffic. They can control ESO bandwidth limits in and out of the data center they use. They can control what they use for DDoS protection. Outside of that, they are pretty much at the mercy of the global internet infrastructure, which is shared by every other person using the internet. There are a host of things that can happen to traffic as it travels back and forth.
There will always be lag and latency associated with that. It is a given. The only variables are "how much" and "how bad".
As for what the solution to Cyrodiil is, we will find that out when they finally start talking about what they are doing outside of Lambert's stream. Until then, it is a puzzle being built from various pieces dropped during the stream. Right now, the puzzle is starting to look like "database design" issues, but Lambert is not technical on the level where this is being fixed and may be using words "wrong", or even crossing subjects by accident.
The only thing we do know, at this point, is that it is not fixed by hardware (CPU).
Edited by Elsonso on December 31, 2021 2:56PMXBox EU/NA:@ElsonsoJannus
PC NA/EU: @Elsonso
PSN NA/EU: @ElsonsoJannus
Total in-game hours: 11321
X/Twitter: ElsonsoJannus3 -
drsalvation✭✭✭✭✭So is this why I need to be pinpoint accurate to have the enemy players standing right in front of me so that I can hit them with 1 single jab, but they can hit me with executioner while I'm 3 meters away behind a pillar?3 -
EdmondDontes✭✭✭✭✭
✭One thing about lag in this game, in particular in PvP, is that it is roughly constant at about 3 seconds. This rather strongly points to a bandwidth issue in the client-server communication, rather than too weak servers. If the server itself can't keep up, the lag would greatly vary depending whether a big siege or fight is going on or not.
The bandwidth is a measure for a maximum amount of data traffic between clients and server per unit of time. When the amount of data traffic exceeds the bandwidth, it gets queued, like in a real traffic jam, resulting in lag of roughly constant duration.
The lag in Cyrodiil only reduces when the campaign is empty, i.e., in the morning when only a few are playing and the bandwidth limit is not reached.
New servers may thus not help if not enough bandwidth is available and the data can't reach the server (and the clients) fast enough.
One way to reduce bandwidth problems would be to scrutinize the data packets between clients and server, eliminate redundancy, minimize packet size, check that noone found a way to prioritize their packet over those of others (resulting in some players skills going off and others' skill don't).
My 5 cents worth.
Oh, and of course, Happy Newyear!
You are wrong. Seven years plus of dealing with these issues, and we can say unequivocally that the vast majority of the latency and disconnecting issues originate from ZOS servers. Other games do not have these issues is how we know it's a ZOS thing.
By the way, I'm not in game right now because I got another crash to desktop while in a big fight, and it's only 10am EST.Edited by EdmondDontes on December 31, 2021 3:12PM2 -
Vevvev✭✭✭✭✭
✭✭✭✭✭You must not have been playing PvP for more than an hour at a time to see performance morph and change as the server becomes stressed from large groups fighting each other. It's very much a server issue than a bandwidth one because if it was a bandwidth one you wouldn't see the issue so widely felt.PC NA - Ceyanna Ashton - Breton Vampire MagDK2 -
sajackson✭✭✭It's worth pointing out that this issue isn't limited to just PvP, although I appreciate the pain is probably felt more in PvP content.
I was running dungeons this afternoon and was having trouble getting target specific abilities to fire - Purifying Light in particular was being very tempramental as to whether or not it would actually fire when I pressed the key. Pretty sure it wasn't a latency issue because there was no juddering or players teleporting around and tbh if I can't get decent latency on a 350MBps fibre connection then I don't blame the internet for that anyway, that has to be an issue at the host end.
Not trying to steal any limelight from the PvP crowd as I totally get that in serious PvP this kind of problem is a turn-off. On the PvE side people tend to ignore it if they're still able to down bosses and pick up loot.2 -
TechMaybeHic✭✭✭✭✭
✭✭They pretty much confirmed server issues themselves when their attempts to fix have revolved around reducing calculations.2 -
Rohamad_Ali✭✭✭✭✭
✭✭✭✭✭The one will come an they will be recognized as the most important figure in modern times. The one that fixed Lag. I've already begun preparing a Monument.1 -
zaria✭✭✭✭✭
✭✭✭✭✭
They does not rent better servers but the huge ball groups fighting each others get broken up by PvDoor players flooding in and filling the servers. Simply buying 3-6 beefy servers would be much cheaper than all the work they done trying to fix the Cyrodil issues.AuraStorm43 wrote: »Its a server issue judging by the improvement we get during midyear when they rent better servers
Its an mix of issues, server load in huge fight even if your not in it, network load and client performance, has to load all the outfit styles and mounts people paid for
Honestly I have more issues with trials than Cyrodil for unknown reasons.Grinding just make you go in circles.
Asking ZoS for nerfs is as stupid as asking for close air support from the death star.0 -
TequilaFire✭✭✭✭✭
✭✭✭✭✭Always the "huge" ball groups. lolEdited by TequilaFire on December 31, 2021 6:12PM0 -
xPoisin✭✭✭It is definitely a server issue, but :
- client side running at 30 FPS need 3 times less calculations as client running 90 FPS (pure math)
- cyro issue is the same as trial issue and is caused by calculations per second. There is a hardcap somewhere ( fix GCD, static content, etc..) but you have to realize that HW is geting higher and higher, but the game architecture was not designed for that (and also amount of players is going higher)
- procsets that are carried by servers are the heaviest calculations - in PvP itis calculation per player and per source (what is in ballgroups pure hell), in PvE it is quite the same issue with trials what is comparable to fight between two bigger ballgroups (too many buffs, debuffs, procsets, synergies, healing, etc. and that all based on source/target mechanics).
- in no cp/no proc PvP it is caused by server performance ( I suggest that servers for no proc are much more “slower” than for proc instances) that is dynamically alocated…
The solution for trials and small group dedicated content (dungeons, BGs) might be in redesign of the game on the HW platform, pushing more servers to be cappable running more instances (but that will bring population cap and issues with synchronization between instances for PvE and PvP if realized for bigger size content).
For the bigger content it is a question of money 😅😅😅 more servers or architecture change, but still must be able to earn some money to the owner 😉😉😉0 -
xPoisin✭✭✭BTW the FPS differences on client side are causing the majority of lag and desync issues, we have done some testing with friends and have seen much less issues when dueling in closed location (housing ) with the fast same FPS on both sides than with higher gap (difference more than 50 FPS)…Edited by xPoisin on December 31, 2021 9:52PM0
-
jedtb16_ESO✭✭✭✭✭
✭✭if i got money for every time i saw the phrase... i only get lag on this game..... i'd have a lot of money.
if i got money for every time i saw the phrase... i don't get lag on any other game..... i'd have a lot more money.2 -
AuraStorm43✭✭✭✭✭
They does not rent better servers but the huge ball groups fighting each others get broken up by PvDoor players flooding in and filling the servers. Simply buying 3-6 beefy servers would be much cheaper than all the work they done trying to fix the Cyrodil issues.AuraStorm43 wrote: »Its a server issue judging by the improvement we get during midyear when they rent better servers
Its an mix of issues, server load in huge fight even if your not in it, network load and client performance, has to load all the outfit styles and mounts people paid for
Honestly I have more issues with trials than Cyrodil for unknown reasons.
They do rent better servers, why do you think the ping improves during the event?2 -
sajackson✭✭✭jedtb16_ESO wrote: »if i got money for every time i saw the phrase... i only get lag on this game..... i'd have a lot of money.
if i got money for every time i saw the phrase... i don't get lag on any other game..... i'd have a lot more money.
Thanks for that insight.
Just a shame that the issue we're talking about has nothing to do with lag.1 -
Kaelthorn_Nightbloom✭✭✭✭✭
✭✭✭The server is likely experiencing high CPU usage when lots of players are in the same area.
Hypothetically, let's say 100 players are near a keep and each player sends 10 position updates per second.
10 * 100 * 100 = 100,000 updates per second that the server must process
100k is actually very low for a CPU, however there's a lot going on with abilities. Lots of logic checks. Range, line of sight, player resources, cooldowns, etc. This all adds up. A single, unoptimized part of the code can cause massive delays (hundreds of milliseconds).
High CPU usage = delayed processing = "lag"
Edited by Kaelthorn_Nightbloom on January 2, 2022 5:02AMPC NA3 -
twev✭✭✭✭✭
✭✭SkaraMinoc wrote: »The server is likely experiencing high CPU usage when lots of players are in the same area.
Hypothetically, let's say 100 players are near a keep and each player sends 10 position updates per second.
10 * 100 * 100 = 100,000 updates per second that the server must process
100k is actually very low for a CPU, however there's a lot going on with abilities. Lots of logic checks. Collision, light of sight, player resources, cooldowns, etc. This all adds up. A single, unoptimized part of the code can cause massive delays (hundreds of milliseconds).
High CPU usage = delayed processing = "lag"
And they still can't fix the bug that occurs when I'm running solo across the map killing random NPCs and I have to rolldodge or swap weapons half the time to be able to loot a body.The problem with society these days is that no one drinks from the skulls of their enemies anymore.2 -
etchedpixels✭✭✭✭✭
✭One thing about lag in this game, in particular in PvP, is that it is roughly constant at about 3 seconds. This rather strongly points to a bandwidth issue in the client-server communication, rather than too weak servers. If the server itself can't keep up, the lag would greatly vary depending whether a big siege or fight is going on or not.
It's server (or client). The data exists as you can measure it.
Collect the TCP session statistics for the game run with tcpdump, throw them through network analyzer tools and it'll tell you four things (at least when I did this for EU last) with a fibre connection at 900Mbit.
- EU network ping time is really good (it's about a 23ms RTT for me from the UK to their German site)
- Server response time when it's good is about 70ms (as reported by the game), so about 50ms is server and or client overhead. That's not good by modern MMO standards.
- Traffic bandwidth isn't enough to overwhelm anything but a slow link - even around dolmens although it does spike up sharply during things like entering areas with a lot of players, then drops. I guess it's caching lots of player data.
- Server response time regularly spikes up to far bigger numbers, and most of the spikes are not network packet drops.
I've not specifically done captures and measured Cyrodiil to see how it looks.
Looking at some of the other people's runs there also seems to be some bufferbloat/tcp capture problems on some of the paths to their server, or their servers so if you have a relatively slow connection you suffer far worse performance than you should.
Edited by etchedpixels on January 1, 2022 5:31PMToo many toons not enough time3 -
EdmondDontes✭✭✭✭✭
✭etchedpixels wrote: »One thing about lag in this game, in particular in PvP, is that it is roughly constant at about 3 seconds. This rather strongly points to a bandwidth issue in the client-server communication, rather than too weak servers. If the server itself can't keep up, the lag would greatly vary depending whether a big siege or fight is going on or not.
It's server (or client). The data exists as you can measure it.
Collect the TCP session statistics for the game run with tcpdump, throw them through network analyzer tools and it'll tell you four things (at least when I did this for EU last) with a fibre connection at 900Mbit.
- EU network ping time is really good (it's about a 23ms RTT for me from the UK to their German site)
- Server response time when it's good is about 70ms (as reported by the game), so about 50ms is server and or client overhead. That's not good by modern MMO standards.
- Traffic bandwidth isn't enough to overwhelm anything but a slow link - even around dolmens although it does spike up sharply during things like entering areas with a lot of players, then drops. I guess it's caching lots of player data.
- Server response time regularly spikes up to far bigger numbers, and most of the spikes are not network packet drops.
I've not specifically done captures and measured Cyrodiil to see how it looks.
Looking at some of the other people's runs there also seems to be some bufferbloat/tcp capture problems on some of the paths to their server, or their servers so if you have a relatively slow connection you suffer far worse performance than you should.
Nice explanation. Thank you for taking the time.
Please go to cyrodiil during prime time in the main campaign, find a big fight, and run your analysis again, and please report back with the results.1
