ZOS, Massive Spike in Ping/LAG in Recent Days - What Gives?

DreadKnight · December 2024

Massive lag on xbox na tonight, dc'd twice then our entire group kicked. Not going to bother trying again

Ugrak · December 2024

thinkaboutit wrote: »

Are we sure its CPU horsepower related and something not on the networking end? Doesn't seem like everyone lags at the same time, which is odd.

With the caveat that we don't know what the hardware in the servers even is, the intel 13th and 14th gen issue has a lot broader ranging effects than CPU "horsepower" so to speak.

The TL;DR is that these CPUs are unstable causing all kinds of errors and crashes, and degrading physically with use. The cause seems to be a combination of Intel having achieved their competitive performance through basically overclocking past safe limits as their default setting, combined with some kind of manufacturing issue causing corrosion of the CPU circuitry. They've released some microcode updates but as Matthew Cassells of Alderon Games says, the general consensus seems to be that they're just bandaids.

This is a good summary.

https://radgametools.com/oodleintel.htm

There is a confirmed hardware problem affecting 13th and 14th generation Intel Core processors (13xxx and 14xxx series) that manifests as general system instability and causes, among other things, Oodle Data decompression failures and crashes in games built with Unreal.

When starting an Unreal Engine-based game, the most common failure is of this type:

DecompressShader(): Could not decompress shader (GetShaderCompressionFormat=Oodle)

However, this problem does not only affect Oodle, and machines that suffer from this instability will also exhibit failures in standard benchmark and stress test programs. Any programs which heavily use the processor on many threads may cause crashes or unpredictable behavior. There have been crashes seen in Adobe Premiere, RealBench, CineBench, Prime95, Handbrake, Visual Studio, and more. This problem can also show up as a GPU error message, such as spurious "out of video memory" errors, even though it is caused by the CPU.

The root cause is now confirmed by Intel to be rapid aging of clock signal circuitry in the chip due to a confluence of several other issues. This is actual physical wear of the chip itself, and once this deterioration has occurred, it is sadly irreversible.

Alderon Games are perhaps the least diplomatic about it.

https://alderongames.com/intel-crashes

Intel is selling defective 13-14th Gen CPUs
Update October 2024

Intel has released some possible fixes as Microcode and BIOS updates. Please make sure your BIOS is updated to the latest version. Please note that Intel's fix is a bandaid and doesn't address the root cause and your CPUs are likely already defective, You will need to send them back for a replacement / RMA.

Intel should be replacing or refunding your CPU without costing any money, if intel is trying to scam you please reach out we try help you to put some pressure on Intel to make sure they actually RMA your CPU and don't scam you.

My team at Alderon Games, working on the multiplayer dinosaur survival game Path of Titans, has been encountering significant problems with Intel CPU stability. These issues, including crashes, instability, and memory corruption, are confined to the 13th and 14th generation processors. Despite all released microcode, BIOS, and firmware updates, the problem remains unresolved.

We have identified failures in five main areas:
End Customers: Thousands of crashes on Intel CPUs on 13th and 14th Gen CPUs in our crash reporting tools.

Official Dedicated Game Servers: Experiencing constant crashes, taking entire servers down.

Development Team: Developers using these CPUs face frequent instability while building and working on the game. It can also cause SSD and memory corruption.

Game Server Providers: Hosting community servers with persistent crashing issues.

Benchmarking Tools: Decompression and memory tests unrelated to Path of Titans also fail.

Over the last 3–4 months, we have observed that CPUs initially working well deteriorate over time, eventually failing. The failure rate we have observed from our own testing is nearly 100%, indicating it's only a matter of time before affected CPUs fail. This issue is gaining attention from news outlets and has been noted by Fortnite and RAD Game Tools, which powers decompression behind Unreal Engine.

Users are also receiving misleading error messages about running out of video driver memory, despite having sufficient memory.

But again, do the ESO servers use Intel 13th or 14th gen CPUs? Maybe they don't.

Arunei · December 2024

Seravi wrote: »

I am running new characters with the XP boost in Auridon. I see chat blowing up with lag issues and large amounts of players being kicked. I went to Skywatch since most are there for the nudballs and I could see players having issues, floating in the air or taking off like warp speed but I never had a frame latency issue through their issues.

I don't group. Not sure if that has any bearing on this problem but it seems to me that a lot of players having issues are grouped, even if it is with only one other person.

90% of the time when I crash or have bad lag, I'm not grouped with anyone.

Quackery · December 2024

Got a 72 hour ban, and a final warning because I was angry at ZOS and the state of the server. This will be my last post, at least for quite a while (demoralizing being here).

I found a workaround that worked for me: disconnect all your items from WiFi and remove ethernet cables from the router. Then connect an ethernet cable straight into your gaming PC (without connecting anything else to WiFi), and you should notice a drastic difference. My game has worked perfectly for days now, despite the occasional spike in ping around ball groups.

You can now connect your other devices to your WiFi (only use WiFi for the rest, not cable).

Merry Christmas and a Happy New Year!

Edit: My Internet speed is 1GB, so the speed is high enough to offset the "bumps".

Seravi · December 2024

Arunei wrote: »

Seravi wrote: »

I am running new characters with the XP boost in Auridon. I see chat blowing up with lag issues and large amounts of players being kicked. I went to Skywatch since most are there for the nudballs and I could see players having issues, floating in the air or taking off like warp speed but I never had a frame latency issue through their issues.

I don't group. Not sure if that has any bearing on this problem but it seems to me that a lot of players having issues are grouped, even if it is with only one other person.

90% of the time when I crash or have bad lag, I'm not grouped with anyone.

Ah thanks. I see chat in game and it seems the crash outs are groups. This event is really taking a toll on stability for sure.

YandereGirlfriend · December 2024

Ugrak wrote: »

Aurielle wrote: »

Tinfoil hat time: the power outage was just a cover for ZOS to migrate ESO over to older, cheaper servers.

Moderation disclaimer: the above statement is a joke, and is not an actual conspiracy theory.

That's what they WANT you to believe! The truth is there are no servers! They're using local AI to create the illusion that we're playing online.

Joke aside, going back to the old might not be as wildly unthinkable a scenario as one might imagine. Because the top end Intel 13th and 14th gen CPU models turned out to have manufacturing or even design defects. Some other game companies had tons of problems with their new servers due to this. If the ESO server upgrade used these, the issues would make a lot of sense.

I have no idea what hardware is in those servers though so it's just a stray thought.

I have a 14900K owner in my PvP guild who fell victim to the 14th Gen degradation issue. And another with a 14700K who is likely at the beginning stages of it. It is super real.

It's not unheard of to have games run off of the higher-end Intel desktop SKUs so IF they are using them then I would hope that it is a possibility that they are taking seriously.

Rkindaleft · December 2024

Rkindaleft wrote: »

Rkindaleft wrote: »

PlayStation NA.

Performance has been really inconsistent since the servers went down a week or so ago. Tonight was the worst I've seen in a while.

Below is a clip of our Unstoppable group running approximately 1-2 hours ago from posting. The first half of the clip is an example of how bad the rubberbanding was (freezing for several seconds and speeding up) and the second part of the clip was literally 4 players getting kicked at the exact same time. I also disconnected tonight but I didn't save the clip. I will state these players are from different continents, so the fact they all got kicked at the same time has nothing to do with their own connections.

Edit - it keeps giving me an error saying the file size is too large so I'm going to post the link to the YouTube video below.
https://youtu.be/Md7ZrI7o_nU

Back again a day later.

This is our Unstoppable group again. As you can see in the top left towards the end of the clip, we literally had 5 people disconnect at the same time for no reason. Something has to give. PS/NA has been so bad since the servers went down a week ago. I'm in several trial discords and it's all the same complaints. Lots of groups I know have been cancelling their runs the past few days because it's almost impossible to get any meaningful progression in when you have multiple disconnections every trial.

https://youtu.be/Ky4ssMSS-SA

Back again a 3rd time.

Dreadsail last night. We had 3 people get disconnected and literally the whole group froze for multiple seconds. Neither of my progression groups are running this week due the holidays (but also because the game is unplayable.) Hopefully after Christmas we get some acknowledgement that ever since the servers went down a week and a half ago the problem has been significantly worse. I'm going to keep posting these as they come up because if it continues like this soon enough there isn't going to be an endgame community at all.

https://youtu.be/AdxZsPtXDfY

AvalonRanger · December 2024

PC-NA Far-East zone

2024/12/23
Yesterday, I got massive lag in the group dungeon.

oldbobdude · December 2024

Good afternoon for me. Only one DC.

cptscotty · December 2024

DC's and Lag Spikes again. Trials and Infinite Archive have been rough lately. Not sure how it got worst but it has.

Arcanasx · December 2024

Frequent and sudden lag spikes without disconnecting tonight.

Sluggy · December 2024

Ugrak wrote: »

But again, do the ESO servers use Intel 13th or 14th gen CPUs? Maybe they don't.

Considering they upgraded the hardware two years before either of those came out it seems highly... unlikely.

Sluggy · December 2024

Quackery wrote: »

Got a 72 hour ban, and a final warning because I was angry at ZOS and the state of the server. This will be my last post, at least for quite a while (demoralizing being here).

I found a workaround that worked for me: disconnect all your items from WiFi and remove ethernet cables from the router. Then connect an ethernet cable straight into your gaming PC (without connecting anything else to WiFi), and you should notice a drastic difference. My game has worked perfectly for days now, despite the occasional spike in ping around ball groups.

You can now connect your other devices to your WiFi (only use WiFi for the rest, not cable).

Merry Christmas and a Happy New Year!

Edit: My Internet speed is 1GB, so the speed is high enough to offset the "bumps".

If that worked then I wouldn't have been complaining about the lag for the last five years...

LadyGP · December 2024

Quackery wrote: »

Got a 72 hour ban, and a final warning because I was angry at ZOS and the state of the server. This will be my last post, at least for quite a while (demoralizing being here).

I found a workaround that worked for me: disconnect all your items from WiFi and remove ethernet cables from the router. Then connect an ethernet cable straight into your gaming PC (without connecting anything else to WiFi), and you should notice a drastic difference. My game has worked perfectly for days now, despite the occasional spike in ping around ball groups.

You can now connect your other devices to your WiFi (only use WiFi for the rest, not cable).

Merry Christmas and a Happy New Year!

Edit: My Internet speed is 1GB, so the speed is high enough to offset the "bumps".

1) I feel you for the ban... not going to talk about specifics because I don't want to get banned but... lets just say.. many of us are right there with you.

2) I'll give this a shot.

LadyGP · December 2024

Sluggy wrote: »

Quackery wrote: »

Got a 72 hour ban, and a final warning because I was angry at ZOS and the state of the server. This will be my last post, at least for quite a while (demoralizing being here).

I found a workaround that worked for me: disconnect all your items from WiFi and remove ethernet cables from the router. Then connect an ethernet cable straight into your gaming PC (without connecting anything else to WiFi), and you should notice a drastic difference. My game has worked perfectly for days now, despite the occasional spike in ping around ball groups.

You can now connect your other devices to your WiFi (only use WiFi for the rest, not cable).

Merry Christmas and a Happy New Year!

Edit: My Internet speed is 1GB, so the speed is high enough to offset the "bumps".

If that worked then I wouldn't have been complaining about the lag for the last five years...

NVM... lol

Sakiri · December 2024

I get spikes but luckily no dcs in a while.

edward_frigidhands · December 2024

YandereGirlfriend wrote: »

Ugrak wrote: »

Aurielle wrote: »

Tinfoil hat time: the power outage was just a cover for ZOS to migrate ESO over to older, cheaper servers.

Moderation disclaimer: the above statement is a joke, and is not an actual conspiracy theory.

That's what they WANT you to believe! The truth is there are no servers! They're using local AI to create the illusion that we're playing online.

Joke aside, going back to the old might not be as wildly unthinkable a scenario as one might imagine. Because the top end Intel 13th and 14th gen CPU models turned out to have manufacturing or even design defects. Some other game companies had tons of problems with their new servers due to this. If the ESO server upgrade used these, the issues would make a lot of sense.

I have no idea what hardware is in those servers though so it's just a stray thought.

I have a 14900K owner in my PvP guild who fell victim to the 14th Gen degradation issue. And another with a 14700K who is likely at the beginning stages of it. It is super real.

It's not unheard of to have games run off of the higher-end Intel desktop SKUs so IF they are using them then I would hope that it is a possibility that they are taking seriously.

Ugrak wrote: »

thinkaboutit wrote: »

Are we sure its CPU horsepower related and something not on the networking end? Doesn't seem like everyone lags at the same time, which is odd.

With the caveat that we don't know what the hardware in the servers even is, the intel 13th and 14th gen issue has a lot broader ranging effects than CPU "horsepower" so to speak.

The TL;DR is that these CPUs are unstable causing all kinds of errors and crashes, and degrading physically with use. The cause seems to be a combination of Intel having achieved their competitive performance through basically overclocking past safe limits as their default setting, combined with some kind of manufacturing issue causing corrosion of the CPU circuitry. They've released some microcode updates but as Matthew Cassells of Alderon Games says, the general consensus seems to be that they're just bandaids.

This is a good summary.

https://radgametools.com/oodleintel.htm

There is a confirmed hardware problem affecting 13th and 14th generation Intel Core processors (13xxx and 14xxx series) that manifests as general system instability and causes, among other things, Oodle Data decompression failures and crashes in games built with Unreal.

When starting an Unreal Engine-based game, the most common failure is of this type:

DecompressShader(): Could not decompress shader (GetShaderCompressionFormat=Oodle)

However, this problem does not only affect Oodle, and machines that suffer from this instability will also exhibit failures in standard benchmark and stress test programs. Any programs which heavily use the processor on many threads may cause crashes or unpredictable behavior. There have been crashes seen in Adobe Premiere, RealBench, CineBench, Prime95, Handbrake, Visual Studio, and more. This problem can also show up as a GPU error message, such as spurious "out of video memory" errors, even though it is caused by the CPU.

The root cause is now confirmed by Intel to be rapid aging of clock signal circuitry in the chip due to a confluence of several other issues. This is actual physical wear of the chip itself, and once this deterioration has occurred, it is sadly irreversible.

Alderon Games are perhaps the least diplomatic about it.

https://alderongames.com/intel-crashes

Intel is selling defective 13-14th Gen CPUs
Update October 2024

Intel has released some possible fixes as Microcode and BIOS updates. Please make sure your BIOS is updated to the latest version. Please note that Intel's fix is a bandaid and doesn't address the root cause and your CPUs are likely already defective, You will need to send them back for a replacement / RMA.

Intel should be replacing or refunding your CPU without costing any money, if intel is trying to scam you please reach out we try help you to put some pressure on Intel to make sure they actually RMA your CPU and don't scam you.

My team at Alderon Games, working on the multiplayer dinosaur survival game Path of Titans, has been encountering significant problems with Intel CPU stability. These issues, including crashes, instability, and memory corruption, are confined to the 13th and 14th generation processors. Despite all released microcode, BIOS, and firmware updates, the problem remains unresolved.

We have identified failures in five main areas:
End Customers: Thousands of crashes on Intel CPUs on 13th and 14th Gen CPUs in our crash reporting tools.

Official Dedicated Game Servers: Experiencing constant crashes, taking entire servers down.

Development Team: Developers using these CPUs face frequent instability while building and working on the game. It can also cause SSD and memory corruption.

Game Server Providers: Hosting community servers with persistent crashing issues.

Benchmarking Tools: Decompression and memory tests unrelated to Path of Titans also fail.

Over the last 3–4 months, we have observed that CPUs initially working well deteriorate over time, eventually failing. The failure rate we have observed from our own testing is nearly 100%, indicating it's only a matter of time before affected CPUs fail. This issue is gaining attention from news outlets and has been noted by Fortnite and RAD Game Tools, which powers decompression behind Unreal Engine.

Users are also receiving misleading error messages about running out of video driver memory, despite having sufficient memory.

But again, do the ESO servers use Intel 13th or 14th gen CPUs? Maybe they don't.

I am curious about this as well.

Given the timeline when the server upgrade happened and the fact that Intel hardware has been the typical choice for server machines, I am wondering if this is an issue affecting ESO server hardware right now.

I am a concerned owner of a 14900 KF myself who has yet to experience these issues, however I have already updated my computer BIOS. Not sure if their server maintenance team did the same. Even if they did there's always the possibility the possibility that the software issue had already caused permanent hardware damage to their server hardware.

It's good practice generally to replace server hardware every 36-42 months so I am hoping a hardware update for their servers is either around the corner or can be expedited due to the reasoning mentioned above.

Regardless of what direction ZOS is going with for game content, server performance, stability and connectivity are going to be the primary determinants of whether I stay with the game and I suspect most players are in the same boat as me right now.

I hope we can get some response regarding this particular aspect of these concerns soon here.

Ugrak · December 2024

edward_frigidhands wrote: »

Given the timeline when the server upgrade happened and the fact that Intel hardware has been the typical choice for server machines, I am wondering if this is an issue affecting ESO server hardware right now.

Hopefully it is not, and looking closer at the timeline makes it seem unlikely.

13th gen released in October 2022, and I think the first server to be upgraded was the PTS server back in maybe March 2022, followed by the PCNA server in May 2022. PCEU upgraded in April 2023 I think, so could theoretically have had 13th gen if that component was purchased/supplied incredibly late, but I don't think PCEU have been more plagued by issues than others. In fact I'm of the impression that PCNA have had it worse.

edward_frigidhands · December 2024

Ugrak wrote: »

edward_frigidhands wrote: »

Given the timeline when the server upgrade happened and the fact that Intel hardware has been the typical choice for server machines, I am wondering if this is an issue affecting ESO server hardware right now.

In fact I'm of the impression that PCNA have had it worse.

PC NA is housed on the same machine as the PTS and there is continuous lag on Mondays when the PTS goes live or is being updated.

There have been other stability, connectivity and performance issues in addition to what is usually observed whenever the PTS is active and up.

BasP · December 2024

edward_frigidhands wrote: »

PC NA is housed on the same machine as the PTS and there is continuous lag on Mondays when the PTS goes live or is being updated.

There have been other stability, connectivity and performance issues in addition to what is usually observed whenever the PTS is active and up.

The PTS is active and up the entire year, though.

sleepy_worm · December 2024

sleepy_worm wrote: »

I have a VPN that shows me my connection to 1 hop away from the game servers. Got severe rubberbanding and even a kick to the login screen (during combat, so -1 thread) by myself in an instance of IA today. VPN showed a stable connection the entire time.

It feels bad to lose 3 hours of progress to something like this.

Time for some anecdotes! My experience the very next day was the polar opposite. Silky smooth IA run with no rubber-banding and ping never went above 300 (my usual ceiling). I had a blast. My VPN reported the exact same latency values over time.

I try to remember the good times, but the frustration of the previous day stinks louder, if that makes sense. It's just human nature to fixate on what doesn't go right, and even if that happens less than half the time, it's still way too often.

AngryPenguin · December 2024

Quackery wrote: »

Got a 72 hour ban, and a final warning because I was angry at ZOS and the state of the server. This will be my last post, at least for quite a while (demoralizing being here).

I found a workaround that worked for me: disconnect all your items from WiFi and remove ethernet cables from the router. Then connect an ethernet cable straight into your gaming PC (without connecting anything else to WiFi), and you should notice a drastic difference. My game has worked perfectly for days now, despite the occasional spike in ping around ball groups.

You can now connect your other devices to your WiFi (only use WiFi for the rest, not cable).

Merry Christmas and a Happy New Year!

Edit: My Internet speed is 1GB, so the speed is high enough to offset the "bumps".

That's not the source of the problem. Most people I know, myself included, only use wired hardware for gaming. Wired mouse, wired Ethernet etc. Everything is wired, and we still have the same issues with lag/ping.

blktauna · December 2024

BasP wrote: »

edward_frigidhands wrote: »

PC NA is housed on the same machine as the PTS and there is continuous lag on Mondays when the PTS goes live or is being updated.

There have been other stability, connectivity and performance issues in addition to what is usually observed whenever the PTS is active and up.

The PTS is active and up the entire year, though.

I don't believe that's the case

BasP · December 2024

blktauna wrote: »

BasP wrote: »

edward_frigidhands wrote: »

PC NA is housed on the same machine as the PTS and there is continuous lag on Mondays when the PTS goes live or is being updated.

There have been other stability, connectivity and performance issues in addition to what is usually observed whenever the PTS is active and up.

The PTS is active and up the entire year, though.

I don't believe that's the case

It actually is. I'm logged into it right now:

I use it to test upcoming changes in the weeks the PTS is used to gather feedback for a new patch, but I also test builds on the PTS all year long before replicating those on the live server (to see which sets I want to farm or craft, and to not waste countless of potions while parsing).

silky_soft · December 2024

I play at high ping ~250ms. When I verse certain players my game will stop responding. I can still run around them, but I cannot skill at all. Cast time or not cast time.

It's specifically certain players, this can be reproduced on another pc in my house with another account. Playing on another class, using completely different sets and add-ons.

Got to be something to do with account priority. The game doesn't stall and I'm not in a null field. Using stam or mag or ulti it doesn't matter.

James-Wayne · December 2024

ZOS_Kevin wrote: »

LadyGP wrote: »

Meant to tag @ZOS_Kevin @ZOS_JessicaFolsom in the above post.

I know we just got the big update from Rich - can we expect additional updates, hopefully before Jan 1, on how the investigation is going and if anything has been discovered? Doesn't have to be from Rich directly.. but just... something to help give some confidence to people about the issue?

First, thank you for gathering the screenshots above.

Regarding updates, we are currently working on a survey to better track information from players who experience performance and lag issues. This is in an effort to better track the information from more places (like Steam, Reddit, Discord Servers, etc,) like what you provided in the screenshots above, just in a more organized way and breaks it down the info that our engineers need. The survey when finished, it will send information directly to our engineers working on this and is intended to better organize the kind of information you took screenshots of above. We are hoping to have this situated in the New Year.

Beyond what we provided above and Rich's last update, we won't have an update until after the New Year. Most of the team is leaving today for the holidays while the rest are gone after Monday, so no updates will be provided until a few weeks after we are back in the New Year and have a little more time to work.

Can we make sure the survey is available longer than a few days unlike other surveys. Most times when I see these surveys they have already been closed off but surely it would be better off to gather as much data as you can on these situations... playing from Australia I have tons of feedback to provide.

blktauna · December 2024

BasP wrote: »

blktauna wrote: »

BasP wrote: »

edward_frigidhands wrote: »

PC NA is housed on the same machine as the PTS and there is continuous lag on Mondays when the PTS goes live or is being updated.

There have been other stability, connectivity and performance issues in addition to what is usually observed whenever the PTS is active and up.

The PTS is active and up the entire year, though.

I don't believe that's the case

It actually is. I'm logged into it right now:

I use it to test upcoming changes in the weeks the PTS is used to gather feedback for a new patch, but I also test builds on the PTS all year long before replicating those on the live server (to see which sets I want to farm or craft, and to not waste countless of potions while parsing).

Yes because its available for testing of the upcoming stuff now. They used to close it off several times a year because I go to test out some builds at times, but the server was unavailable on multiple occasions.

BasP · December 2024

blktauna wrote: »

BasP wrote: »

blktauna wrote: »

BasP wrote: »

edward_frigidhands wrote: »

PC NA is housed on the same machine as the PTS and there is continuous lag on Mondays when the PTS goes live or is being updated.

There have been other stability, connectivity and performance issues in addition to what is usually observed whenever the PTS is active and up.

The PTS is active and up the entire year, though.

I don't believe that's the case

It actually is. I'm logged into it right now:

I use it to test upcoming changes in the weeks the PTS is used to gather feedback for a new patch, but I also test builds on the PTS all year long before replicating those on the live server (to see which sets I want to farm or craft, and to not waste countless of potions while parsing).

Yes because its available for testing of the upcoming stuff now. They used to close it off several times a year because I go to test out some builds at times, but the server was unavailable on multiple occasions.

Hmm, in that case I must've imagined all those hours I've spent on the PTS outside of the weeks in which an upcoming patch is tested then. My bad.

There isn't any new stuff on the PTS at the moment by the way; it's the same update as the Live Server.

blktauna · December 2024

Maybe that's changed then. I stopped doing pts last year so it might be different now. But one of the reasons I stopped was it was offline so much.

edward_frigidhands · December 2024

LadyGP wrote: »

I'm gonna be that person. I put a ton of responses in here, Richs feedback, other dev comments, took information from Reddit and told ChatGPT to analyze all of the available information. I'm sure someone at ZoS has already done this and their team knows more about this issue than ChatGPT but none the less I'm going to put it here in the 0.5% chance it sparks something to help solve the problem.

The issue appears to be tied to a recent client or server-side update that introduced a new network handling regression. This regression is causing the client to block while waiting for server responses, leading to frequent "stalls" (high ping, freezing) and then sudden bursts of updates once the server data arrives.

Detailed Analysis:
Based on the additional user report and eliminating the Bandits UI add-on as a factor, we need to refocus on what could be causing network stalls that affect all players, regardless of their hardware, ISP, or installed add-ons.

Key points from the user’s comments:

1. Wait Chain Analysis Shows Network I/O Blocking:
The user notes that Windows’ built-in wait chain analysis shows the game’s threads waiting on network I/O. This suggests that the game client’s thread(s) responsible for networking are blocked, waiting for a response from the server. The client may be making synchronous network calls or is otherwise unable to proceed with normal rendering and logic updates until it receives needed data packets. As a result, the entire game "feels" frozen because the main loop or critical update thread is stalled.

2. Cross-Environment Consistency:
The user reinstalled Windows, refreshed drivers, and still experiences these issues exclusively with this game. Other AAA titles run smoothly on the same system, with no spikes in latency or frame drops. This eliminates local hardware, drivers, and ISP as the cause, reinforcing that the problem is external.

3. Coincidence with the Most Recent Client Update:
The issues began following a recent client patch or update. Such timing strongly suggests a code change—possibly a new feature or optimization gone wrong—that introduced a more synchronous or less fault-tolerant network path.

4. Global, Not Add-On Dependent:
Initial suspicions pointed towards add-ons like Bandits UI exploiting map pings and causing a network message flood. However, reports from users without any add-ons and across various hardware configurations indicate that the problem persists. This points to a more fundamental network or server-side code issue.

5. Similar Symptoms, Multiple Users:
Players report skill delays, high ping, and periods of "no movement" followed by rapid catch-up. This pattern indicates that the server is not delivering updates steadily. Instead, the client starves for data, then receives a sudden burst of packets once the server "catches up." The root cause could be an overloaded server process, a network queue backlogging, or a newly introduced logic path that locks the server’s response temporarily.

Likely Root Cause:
The most probable cause is a server or client-side network handling regression introduced by the recent update. On the server side, there could be a message queue buildup or a bottleneck in message processing, causing delayed responses. On the client side, there could be a newly introduced synchronous wait for server data—perhaps a previously asynchronous call is now blocking the main thread. When the server finally sends the data, the client rushes to process everything at once, causing the "speed-up" effect.

Conclusion:
Even if add-ons were initially suspected, current evidence points more clearly toward a systemic issue introduced by recent code changes. The client is being forced into synchronous waits for server responses, and when combined with server-side delays, this results in the freezing and catch-up behavior observed. The root cause is most likely a network processing or code regression on the server or client side (or both) introduced during the recent patch, rather than user hardware, ISPs, or add-ons.

I am curious how this has persisted since April 2024 without some progress.