ZOS, Massive Spike in Ping/LAG in Recent Days - What Gives?

DreadKnight

Massive lag on xbox na tonight, dc'd twice then our entire group kicked. Not going to bother trying again

Ugrak

thinkaboutit wrote: »

Are we sure its CPU horsepower related and something not on the networking end? Doesn't seem like everyone lags at the same time, which is odd.

With the caveat that we don't know what the hardware in the servers even is, the intel 13th and 14th gen issue has a lot broader ranging effects than CPU "horsepower" so to speak.

The TL;DR is that these CPUs are unstable causing all kinds of errors and crashes, and degrading physically with use. The cause seems to be a combination of Intel having achieved their competitive performance through basically overclocking past safe limits as their default setting, combined with some kind of manufacturing issue causing corrosion of the CPU circuitry. They've released some microcode updates but as Matthew Cassells of Alderon Games says, the general consensus seems to be that they're just bandaids.

This is a good summary.

https://radgametools.com/oodleintel.htm

There is a confirmed hardware problem affecting 13th and 14th generation Intel Core processors (13xxx and 14xxx series) that manifests as general system instability and causes, among other things, Oodle Data decompression failures and crashes in games built with Unreal.

When starting an Unreal Engine-based game, the most common failure is of this type:

DecompressShader(): Could not decompress shader (GetShaderCompressionFormat=Oodle)

However, this problem does not only affect Oodle, and machines that suffer from this instability will also exhibit failures in standard benchmark and stress test programs. Any programs which heavily use the processor on many threads may cause crashes or unpredictable behavior. There have been crashes seen in Adobe Premiere, RealBench, CineBench, Prime95, Handbrake, Visual Studio, and more. This problem can also show up as a GPU error message, such as spurious "out of video memory" errors, even though it is caused by the CPU.

The root cause is now confirmed by Intel to be rapid aging of clock signal circuitry in the chip due to a confluence of several other issues. This is actual physical wear of the chip itself, and once this deterioration has occurred, it is sadly irreversible.

Alderon Games are perhaps the least diplomatic about it.

https://alderongames.com/intel-crashes

Intel is selling defective 13-14th Gen CPUs
Update October 2024

Intel has released some possible fixes as Microcode and BIOS updates. Please make sure your BIOS is updated to the latest version. Please note that Intel's fix is a bandaid and doesn't address the root cause and your CPUs are likely already defective, You will need to send them back for a replacement / RMA.

Intel should be replacing or refunding your CPU without costing any money, if intel is trying to scam you please reach out we try help you to put some pressure on Intel to make sure they actually RMA your CPU and don't scam you.

My team at Alderon Games, working on the multiplayer dinosaur survival game Path of Titans, has been encountering significant problems with Intel CPU stability. These issues, including crashes, instability, and memory corruption, are confined to the 13th and 14th generation processors. Despite all released microcode, BIOS, and firmware updates, the problem remains unresolved.

We have identified failures in five main areas:

• End Customers: Thousands of crashes on Intel CPUs on 13th and 14th Gen CPUs in our crash reporting tools.
• Official Dedicated Game Servers: Experiencing constant crashes, taking entire servers down.
• Development Team: Developers using these CPUs face frequent instability while building and working on the game. It can also cause SSD and memory corruption.
• Game Server Providers: Hosting community servers with persistent crashing issues.
• Benchmarking Tools: Decompression and memory tests unrelated to Path of Titans also fail.

Over the last 3–4 months, we have observed that CPUs initially working well deteriorate over time, eventually failing. The failure rate we have observed from our own testing is nearly 100%, indicating it's only a matter of time before affected CPUs fail. This issue is gaining attention from news outlets and has been noted by Fortnite and RAD Game Tools, which powers decompression behind Unreal Engine.

Users are also receiving misleading error messages about running out of video driver memory, despite having sufficient memory.

But again, do the ESO servers use Intel 13th or 14th gen CPUs? Maybe they don't.

Arunei

Seravi wrote: »

I am running new characters with the XP boost in Auridon. I see chat blowing up with lag issues and large amounts of players being kicked. I went to Skywatch since most are there for the nudballs and I could see players having issues, floating in the air or taking off like warp speed but I never had a frame latency issue through their issues.

I don't group. Not sure if that has any bearing on this problem but it seems to me that a lot of players having issues are grouped, even if it is with only one other person.

90% of the time when I crash or have bad lag, I'm not grouped with anyone.

Quackery

Got a 72 hour ban, and a final warning because I was angry at ZOS and the state of the server. This will be my last post, at least for quite a while (demoralizing being here).

I found a workaround that worked for me: disconnect all your items from WiFi and remove ethernet cables from the router. Then connect an ethernet cable straight into your gaming PC (without connecting anything else to WiFi), and you should notice a drastic difference. My game has worked perfectly for days now, despite the occasional spike in ping around ball groups.

You can now connect your other devices to your WiFi (only use WiFi for the rest, not cable).

Merry Christmas and a Happy New Year!

Edit: My Internet speed is 1GB, so the speed is high enough to offset the "bumps".

Seravi

Arunei wrote: »

Seravi wrote: »

I am running new characters with the XP boost in Auridon. I see chat blowing up with lag issues and large amounts of players being kicked. I went to Skywatch since most are there for the nudballs and I could see players having issues, floating in the air or taking off like warp speed but I never had a frame latency issue through their issues.

I don't group. Not sure if that has any bearing on this problem but it seems to me that a lot of players having issues are grouped, even if it is with only one other person.

90% of the time when I crash or have bad lag, I'm not grouped with anyone.

Ah thanks. I see chat in game and it seems the crash outs are groups. This event is really taking a toll on stability for sure.

YandereGirlfriend

Ugrak wrote: »

Aurielle wrote: »

Tinfoil hat time: the power outage was just a cover for ZOS to migrate ESO over to older, cheaper servers.

Moderation disclaimer: the above statement is a joke, and is not an actual conspiracy theory.

That's what they WANT you to believe! The truth is there are no servers! They're using local AI to create the illusion that we're playing online.

Joke aside, going back to the old might not be as wildly unthinkable a scenario as one might imagine. Because the top end Intel 13th and 14th gen CPU models turned out to have manufacturing or even design defects. Some other game companies had tons of problems with their new servers due to this. If the ESO server upgrade used these, the issues would make a lot of sense.

I have no idea what hardware is in those servers though so it's just a stray thought.

I have a 14900K owner in my PvP guild who fell victim to the 14th Gen degradation issue. And another with a 14700K who is likely at the beginning stages of it. It is super real.

It's not unheard of to have games run off of the higher-end Intel desktop SKUs so IF they are using them then I would hope that it is a possibility that they are taking seriously.

Rkindaleft

Rkindaleft wrote: »

Rkindaleft wrote: »

PlayStation NA.

Performance has been really inconsistent since the servers went down a week or so ago. Tonight was the worst I've seen in a while.

Below is a clip of our Unstoppable group running approximately 1-2 hours ago from posting. The first half of the clip is an example of how bad the rubberbanding was (freezing for several seconds and speeding up) and the second part of the clip was literally 4 players getting kicked at the exact same time. I also disconnected tonight but I didn't save the clip. I will state these players are from different continents, so the fact they all got kicked at the same time has nothing to do with their own connections.

Edit - it keeps giving me an error saying the file size is too large so I'm going to post the link to the YouTube video below.
https://youtu.be/Md7ZrI7o_nU

Back again a day later.

This is our Unstoppable group again. As you can see in the top left towards the end of the clip, we literally had 5 people disconnect at the same time for no reason. Something has to give. PS/NA has been so bad since the servers went down a week ago. I'm in several trial discords and it's all the same complaints. Lots of groups I know have been cancelling their runs the past few days because it's almost impossible to get any meaningful progression in when you have multiple disconnections every trial.

https://youtu.be/Ky4ssMSS-SA

Back again a 3rd time.

Dreadsail last night. We had 3 people get disconnected and literally the whole group froze for multiple seconds. Neither of my progression groups are running this week due the holidays (but also because the game is unplayable.) Hopefully after Christmas we get some acknowledgement that ever since the servers went down a week and a half ago the problem has been significantly worse. I'm going to keep posting these as they come up because if it continues like this soon enough there isn't going to be an endgame community at all.

https://youtu.be/AdxZsPtXDfY

AvalonRanger

PC-NA Far-East zone

2024/12/23
Yesterday, I got massive lag in the group dungeon.

oldbobdude

Good afternoon for me. Only one DC.

cptscotty

DC's and Lag Spikes again. Trials and Infinite Archive have been rough lately. Not sure how it got worst but it has.

Arcanasx

Frequent and sudden lag spikes without disconnecting tonight.

Sluggy

Ugrak wrote: »

But again, do the ESO servers use Intel 13th or 14th gen CPUs? Maybe they don't.

Considering they upgraded the hardware two years before either of those came out it seems highly... unlikely.

Sluggy

Quackery wrote: »

Got a 72 hour ban, and a final warning because I was angry at ZOS and the state of the server. This will be my last post, at least for quite a while (demoralizing being here).

I found a workaround that worked for me: disconnect all your items from WiFi and remove ethernet cables from the router. Then connect an ethernet cable straight into your gaming PC (without connecting anything else to WiFi), and you should notice a drastic difference. My game has worked perfectly for days now, despite the occasional spike in ping around ball groups.

You can now connect your other devices to your WiFi (only use WiFi for the rest, not cable).

Merry Christmas and a Happy New Year!

Edit: My Internet speed is 1GB, so the speed is high enough to offset the "bumps".

If that worked then I wouldn't have been complaining about the lag for the last five years...

LadyGP

Quackery wrote: »

Got a 72 hour ban, and a final warning because I was angry at ZOS and the state of the server. This will be my last post, at least for quite a while (demoralizing being here).

I found a workaround that worked for me: disconnect all your items from WiFi and remove ethernet cables from the router. Then connect an ethernet cable straight into your gaming PC (without connecting anything else to WiFi), and you should notice a drastic difference. My game has worked perfectly for days now, despite the occasional spike in ping around ball groups.

You can now connect your other devices to your WiFi (only use WiFi for the rest, not cable).

Merry Christmas and a Happy New Year!

Edit: My Internet speed is 1GB, so the speed is high enough to offset the "bumps".

1) I feel you for the ban... not going to talk about specifics because I don't want to get banned but... lets just say.. many of us are right there with you.

2) I'll give this a shot.

LadyGP

Sluggy wrote: »

Quackery wrote: »

Got a 72 hour ban, and a final warning because I was angry at ZOS and the state of the server. This will be my last post, at least for quite a while (demoralizing being here).

I found a workaround that worked for me: disconnect all your items from WiFi and remove ethernet cables from the router. Then connect an ethernet cable straight into your gaming PC (without connecting anything else to WiFi), and you should notice a drastic difference. My game has worked perfectly for days now, despite the occasional spike in ping around ball groups.

You can now connect your other devices to your WiFi (only use WiFi for the rest, not cable).

Merry Christmas and a Happy New Year!

Edit: My Internet speed is 1GB, so the speed is high enough to offset the "bumps".

If that worked then I wouldn't have been complaining about the lag for the last five years...

NVM... lol

Sakiri

I get spikes but luckily no dcs in a while.