ZOS, Massive Spike in Ping/LAG in Recent Days - What Gives?
Maintenance for the week of March 24:
• PC/Mac: NA and EU megaservers for patch maintenance – March 24, 4:00AM EDT (8:00 UTC) - 10:00AM EDT (14:00 UTC)
• Xbox: NA and EU megaservers for patch maintenance – March 26, 6:00AM EDT (10:00 UTC) - 12:00PM EDT (16:00 UTC)
• Playstation®: NA and EU megaservers for patch maintenance – March 26, 6:00AM EDT (10:00 UTC) - 12:00PM EDT (16:00 UTC)
• PC/Mac: NA and EU megaservers for patch maintenance – March 24, 4:00AM EDT (8:00 UTC) - 10:00AM EDT (14:00 UTC)
• Xbox: NA and EU megaservers for patch maintenance – March 26, 6:00AM EDT (10:00 UTC) - 12:00PM EDT (16:00 UTC)
• Playstation®: NA and EU megaservers for patch maintenance – March 26, 6:00AM EDT (10:00 UTC) - 12:00PM EDT (16:00 UTC)
-
DreadKnight✭✭✭✭Massive lag on xbox na tonight, dc'd twice then our entire group kicked. Not going to bother trying again2 -
Ugrak✭✭✭✭thinkaboutit wrote: »Are we sure its CPU horsepower related and something not on the networking end? Doesn't seem like everyone lags at the same time, which is odd.
With the caveat that we don't know what the hardware in the servers even is, the intel 13th and 14th gen issue has a lot broader ranging effects than CPU "horsepower" so to speak.
The TL;DR is that these CPUs are unstable causing all kinds of errors and crashes, and degrading physically with use. The cause seems to be a combination of Intel having achieved their competitive performance through basically overclocking past safe limits as their default setting, combined with some kind of manufacturing issue causing corrosion of the CPU circuitry. They've released some microcode updates but as Matthew Cassells of Alderon Games says, the general consensus seems to be that they're just bandaids.
This is a good summary.
https://radgametools.com/oodleintel.htmThere is a confirmed hardware problem affecting 13th and 14th generation Intel Core processors (13xxx and 14xxx series) that manifests as general system instability and causes, among other things, Oodle Data decompression failures and crashes in games built with Unreal.
When starting an Unreal Engine-based game, the most common failure is of this type:
DecompressShader(): Could not decompress shader (GetShaderCompressionFormat=Oodle)
However, this problem does not only affect Oodle, and machines that suffer from this instability will also exhibit failures in standard benchmark and stress test programs. Any programs which heavily use the processor on many threads may cause crashes or unpredictable behavior. There have been crashes seen in Adobe Premiere, RealBench, CineBench, Prime95, Handbrake, Visual Studio, and more. This problem can also show up as a GPU error message, such as spurious "out of video memory" errors, even though it is caused by the CPU.
The root cause is now confirmed by Intel to be rapid aging of clock signal circuitry in the chip due to a confluence of several other issues. This is actual physical wear of the chip itself, and once this deterioration has occurred, it is sadly irreversible.
Alderon Games are perhaps the least diplomatic about it.
https://alderongames.com/intel-crashesIntel is selling defective 13-14th Gen CPUs
Update October 2024
Intel has released some possible fixes as Microcode and BIOS updates. Please make sure your BIOS is updated to the latest version. Please note that Intel's fix is a bandaid and doesn't address the root cause and your CPUs are likely already defective, You will need to send them back for a replacement / RMA.
Intel should be replacing or refunding your CPU without costing any money, if intel is trying to scam you please reach out we try help you to put some pressure on Intel to make sure they actually RMA your CPU and don't scam you.
My team at Alderon Games, working on the multiplayer dinosaur survival game Path of Titans, has been encountering significant problems with Intel CPU stability. These issues, including crashes, instability, and memory corruption, are confined to the 13th and 14th generation processors. Despite all released microcode, BIOS, and firmware updates, the problem remains unresolved.
We have identified failures in five main areas:- End Customers: Thousands of crashes on Intel CPUs on 13th and 14th Gen CPUs in our crash reporting tools.
- Official Dedicated Game Servers: Experiencing constant crashes, taking entire servers down.
- Development Team: Developers using these CPUs face frequent instability while building and working on the game. It can also cause SSD and memory corruption.
- Game Server Providers: Hosting community servers with persistent crashing issues.
- Benchmarking Tools: Decompression and memory tests unrelated to Path of Titans also fail.
Over the last 3–4 months, we have observed that CPUs initially working well deteriorate over time, eventually failing. The failure rate we have observed from our own testing is nearly 100%, indicating it's only a matter of time before affected CPUs fail. This issue is gaining attention from news outlets and has been noted by Fortnite and RAD Game Tools, which powers decompression behind Unreal Engine.
Users are also receiving misleading error messages about running out of video driver memory, despite having sufficient memory.
But again, do the ESO servers use Intel 13th or 14th gen CPUs? Maybe they don't.5 -
Arunei✭✭✭✭✭
✭✭✭✭
90% of the time when I crash or have bad lag, I'm not grouped with anyone.I am running new characters with the XP boost in Auridon. I see chat blowing up with lag issues and large amounts of players being kicked. I went to Skywatch since most are there for the nudballs and I could see players having issues, floating in the air or taking off like warp speed but I never had a frame latency issue through their issues.
I don't group. Not sure if that has any bearing on this problem but it seems to me that a lot of players having issues are grouped, even if it is with only one other person.Character List [RP and PvE]:SpoilerStands-Against-Death: Argonian Magplar Healer - Crafter
Krisiel: Redguard Stamsorc DPS - Literally crazy Werewolf, no like legit insane. She nuts
Kiju Veran: Khajiit Stamblade DPS - Ex-Fighters Guild Suthay who likes to punch things, nicknamed Tinykat
Niralae Elsinal: Altmer Stamsorc DPS - Young Altmer with way too much Magicka
Sarah Lacroix: Breton Magsorc DPS - Fledgling Vampire who drinks too much water
Slondor: Nord Tankblade - TESified verson of Slenderman
Marius Vastino: Imperial <insert role here> - Sarah's apathetic sire who likes to monologue
Delthor Rellenar: Dunmer Magknight DPS - Sarah's ex who's a certified psychopath
Lirawyn Calatare: Altmer Magplar Healer - Traveling performer and bard who's 101% vanilla bean
Gondryn Beldeau: Breton Tankplar - Sarah's Mages Guild mentor and certified badass old person
Gwendolyn Jenelle: Breton Magplar Healer - Friendly healer with a coffee addiction
Soliril Larethian- Altmer Magblade DPS - Blind alchemist who uses animals to see and brews plagues in his spare time
Tevril Rallenar: Dunmer Stamcro DPS - Delthor's "special" younger brother who raises small animals as friends
Celeroth Calatare: Bosmer <insert role here> - Shapeshifting Bosmer with enough sass to fill Valenwood
PC - NA - EP - CP1000+
Avid RPer. Hit me up in-game @Ras_Lei if you're interested in getting together for some arr-pee shenanigans!3 -
Quackery✭✭✭✭✭Got a 72 hour ban, and a final warning because I was angry at ZOS and the state of the server. This will be my last post, at least for quite a while (demoralizing being here).
I found a workaround that worked for me: disconnect all your items from WiFi and remove ethernet cables from the router. Then connect an ethernet cable straight into your gaming PC (without connecting anything else to WiFi), and you should notice a drastic difference. My game has worked perfectly for days now, despite the occasional spike in ping around ball groups.
You can now connect your other devices to your WiFi (only use WiFi for the rest, not cable).
Merry Christmas and a Happy New Year!
Edit: My Internet speed is 1GB, so the speed is high enough to offset the "bumps".Edited by Quackery on December 23, 2024 9:04PM3 -
Seravi✭✭✭✭✭90% of the time when I crash or have bad lag, I'm not grouped with anyone.
Ah thanks. I see chat in game and it seems the crash outs are groups. This event is really taking a toll on stability for sure.0 -
YandereGirlfriend✭✭✭✭✭
✭✭✭✭✭
That's what they WANT you to believe! The truth is there are no servers! They're using local AI to create the illusion that we're playing online.
Joke aside, going back to the old might not be as wildly unthinkable a scenario as one might imagine. Because the top end Intel 13th and 14th gen CPU models turned out to have manufacturing or even design defects. Some other game companies had tons of problems with their new servers due to this. If the ESO server upgrade used these, the issues would make a lot of sense.
I have no idea what hardware is in those servers though so it's just a stray thought.
I have a 14900K owner in my PvP guild who fell victim to the 14th Gen degradation issue. And another with a 14700K who is likely at the beginning stages of it. It is super real.
It's not unheard of to have games run off of the higher-end Intel desktop SKUs so IF they are using them then I would hope that it is a possibility that they are taking seriously.1 -
Rkindaleft✭✭✭✭✭
✭Rkindaleft wrote: »
Back again a day later.
This is our Unstoppable group again. As you can see in the top left towards the end of the clip, we literally had 5 people disconnect at the same time for no reason. Something has to give. PS/NA has been so bad since the servers went down a week ago. I'm in several trial discords and it's all the same complaints. Lots of groups I know have been cancelling their runs the past few days because it's almost impossible to get any meaningful progression in when you have multiple disconnections every trial.
https://youtu.be/Ky4ssMSS-SA
Back again a 3rd time.
Dreadsail last night. We had 3 people get disconnected and literally the whole group froze for multiple seconds. Neither of my progression groups are running this week due the holidays (but also because the game is unplayable.) Hopefully after Christmas we get some acknowledgement that ever since the servers went down a week and a half ago the problem has been significantly worse. I'm going to keep posting these as they come up because if it continues like this soon enough there isn't going to be an endgame community at all.
https://youtu.be/AdxZsPtXDfY Edited by Rkindaleft on December 23, 2024 9:45PMhttps://youtube.com/@rkindaleft PlayStation NA. I upload parses and trial POVs sometimes.SpoilerHave cleared all trial hard modes.
6/9 trial trifectas.5 -
AvalonRanger✭✭✭✭✭PC-NA Far-East zone
2024/12/23
Yesterday, I got massive lag in the group dungeon.My playing time Mon-Friday UTC13:00-16:00 [PC-NA] CP over2000 now.
I have [1Tough tank] [1StamSorc-DD] [1Necro-DD] [1Real Healer]
with [1Stam Blade].
But, I'm Tank main player. Recently I'm doing Healer.
2023/12/21
By the way...Dungeon-Meshi(One of Famous Japanese fantasy story comic book) got finale...
Good-bye "King of Monster Eater".
2024/08/23
Farewell Atsuko Tanaka...(-_-) I never forget epic acting for major Motoko Kusanagi.0 -
cptscotty✭✭✭DC's and Lag Spikes again. Trials and Infinite Archive have been rough lately. Not sure how it got worst but it has.2
-
Sluggy✭✭✭✭✭Got a 72 hour ban, and a final warning because I was angry at ZOS and the state of the server. This will be my last post, at least for quite a while (demoralizing being here).
I found a workaround that worked for me: disconnect all your items from WiFi and remove ethernet cables from the router. Then connect an ethernet cable straight into your gaming PC (without connecting anything else to WiFi), and you should notice a drastic difference. My game has worked perfectly for days now, despite the occasional spike in ping around ball groups.
You can now connect your other devices to your WiFi (only use WiFi for the rest, not cable).
Merry Christmas and a Happy New Year!
Edit: My Internet speed is 1GB, so the speed is high enough to offset the "bumps".
If that worked then I wouldn't have been complaining about the lag for the last five years...4 -
LadyGP✭✭✭✭✭
✭Got a 72 hour ban, and a final warning because I was angry at ZOS and the state of the server. This will be my last post, at least for quite a while (demoralizing being here).
I found a workaround that worked for me: disconnect all your items from WiFi and remove ethernet cables from the router. Then connect an ethernet cable straight into your gaming PC (without connecting anything else to WiFi), and you should notice a drastic difference. My game has worked perfectly for days now, despite the occasional spike in ping around ball groups.
You can now connect your other devices to your WiFi (only use WiFi for the rest, not cable).
Merry Christmas and a Happy New Year!
Edit: My Internet speed is 1GB, so the speed is high enough to offset the "bumps".
1) I feel you for the ban... not going to talk about specifics because I don't want to get banned but... lets just say.. many of us are right there with you.
2) I'll give this a shot.LadyGP/xCatGuy
PC/NA2 -
LadyGP✭✭✭✭✭
✭
If that worked then I wouldn't have been complaining about the lag for the last five years...
NVM... lolLadyGP/xCatGuy
PC/NA0 -
edward_frigidhands✭✭✭✭✭YandereGirlfriend wrote: »
I have a 14900K owner in my PvP guild who fell victim to the 14th Gen degradation issue. And another with a 14700K who is likely at the beginning stages of it. It is super real.
It's not unheard of to have games run off of the higher-end Intel desktop SKUs so IF they are using them then I would hope that it is a possibility that they are taking seriously.
With the caveat that we don't know what the hardware in the servers even is, the intel 13th and 14th gen issue has a lot broader ranging effects than CPU "horsepower" so to speak.
The TL;DR is that these CPUs are unstable causing all kinds of errors and crashes, and degrading physically with use. The cause seems to be a combination of Intel having achieved their competitive performance through basically overclocking past safe limits as their default setting, combined with some kind of manufacturing issue causing corrosion of the CPU circuitry. They've released some microcode updates but as Matthew Cassells of Alderon Games says, the general consensus seems to be that they're just bandaids.
This is a good summary.
https://radgametools.com/oodleintel.htm
Alderon Games are perhaps the least diplomatic about it.
https://alderongames.com/intel-crashes
But again, do the ESO servers use Intel 13th or 14th gen CPUs? Maybe they don't.
I am curious about this as well.
Given the timeline when the server upgrade happened and the fact that Intel hardware has been the typical choice for server machines, I am wondering if this is an issue affecting ESO server hardware right now.
I am a concerned owner of a 14900 KF myself who has yet to experience these issues, however I have already updated my computer BIOS. Not sure if their server maintenance team did the same. Even if they did there's always the possibility the possibility that the software issue had already caused permanent hardware damage to their server hardware.
It's good practice generally to replace server hardware every 36-42 months so I am hoping a hardware update for their servers is either around the corner or can be expedited due to the reasoning mentioned above.
Regardless of what direction ZOS is going with for game content, server performance, stability and connectivity are going to be the primary determinants of whether I stay with the game and I suspect most players are in the same boat as me right now.
I hope we can get some response regarding this particular aspect of these concerns soon here.
4 -
Ugrak✭✭✭✭edward_frigidhands wrote: »Given the timeline when the server upgrade happened and the fact that Intel hardware has been the typical choice for server machines, I am wondering if this is an issue affecting ESO server hardware right now.
Hopefully it is not, and looking closer at the timeline makes it seem unlikely.
13th gen released in October 2022, and I think the first server to be upgraded was the PTS server back in maybe March 2022, followed by the PCNA server in May 2022. PCEU upgraded in April 2023 I think, so could theoretically have had 13th gen if that component was purchased/supplied incredibly late, but I don't think PCEU have been more plagued by issues than others. In fact I'm of the impression that PCNA have had it worse.0 -
edward_frigidhands✭✭✭✭✭
PC NA is housed on the same machine as the PTS and there is continuous lag on Mondays when the PTS goes live or is being updated.
There have been other stability, connectivity and performance issues in addition to what is usually observed whenever the PTS is active and up.
2 -
BasP✭✭✭✭✭edward_frigidhands wrote: »PC NA is housed on the same machine as the PTS and there is continuous lag on Mondays when the PTS goes live or is being updated.
There have been other stability, connectivity and performance issues in addition to what is usually observed whenever the PTS is active and up.
The PTS is active and up the entire year, though.2 -
sleepy_worm✭✭✭✭sleepy_worm wrote: »I have a VPN that shows me my connection to 1 hop away from the game servers. Got severe rubberbanding and even a kick to the login screen (during combat, so -1 thread) by myself in an instance of IA today. VPN showed a stable connection the entire time.
It feels bad to lose 3 hours of progress to something like this.
Time for some anecdotes! My experience the very next day was the polar opposite. Silky smooth IA run with no rubber-banding and ping never went above 300 (my usual ceiling). I had a blast. My VPN reported the exact same latency values over time.
I try to remember the good times, but the frustration of the previous day stinks louder, if that makes sense. It's just human nature to fixate on what doesn't go right, and even if that happens less than half the time, it's still way too often.2 -
AngryPenguin✭✭✭✭✭Got a 72 hour ban, and a final warning because I was angry at ZOS and the state of the server. This will be my last post, at least for quite a while (demoralizing being here).
I found a workaround that worked for me: disconnect all your items from WiFi and remove ethernet cables from the router. Then connect an ethernet cable straight into your gaming PC (without connecting anything else to WiFi), and you should notice a drastic difference. My game has worked perfectly for days now, despite the occasional spike in ping around ball groups.
You can now connect your other devices to your WiFi (only use WiFi for the rest, not cable).
Merry Christmas and a Happy New Year!
Edit: My Internet speed is 1GB, so the speed is high enough to offset the "bumps".
That's not the source of the problem. Most people I know, myself included, only use wired hardware for gaming. Wired mouse, wired Ethernet etc. Everything is wired, and we still have the same issues with lag/ping.7 -
blktauna✭✭✭✭✭
✭✭
The PTS is active and up the entire year, though.
I don't believe that's the casePCNA
PCEU0 -
BasP✭✭✭✭✭
I don't believe that's the case
It actually is. I'm logged into it right now:
I use it to test upcoming changes in the weeks the PTS is used to gather feedback for a new patch, but I also test builds on the PTS all year long before replicating those on the live server (to see which sets I want to farm or craft, and to not waste countless of potions while parsing).4 -
silky_soft✭✭✭✭✭
✭I play at high ping ~250ms. When I verse certain players my game will stop responding. I can still run around them, but I cannot skill at all. Cast time or not cast time.
It's specifically certain players, this can be reproduced on another pc in my house with another account. Playing on another class, using completely different sets and add-ons.
Got to be something to do with account priority. The game doesn't stall and I'm not in a null field. Using stam or mag or ulti it doesn't matter.
I have no will left to help with lag until high action per minute devs play via a vpn from Asia or Oceania to NA and live stream thier experience of thier actions being declined by the server because they are out of frame.1 -
James-Wayne✭✭✭✭✭
✭
First, thank you for gathering the screenshots above.
Regarding updates, we are currently working on a survey to better track information from players who experience performance and lag issues. This is in an effort to better track the information from more places (like Steam, Reddit, Discord Servers, etc,) like what you provided in the screenshots above, just in a more organized way and breaks it down the info that our engineers need. The survey when finished, it will send information directly to our engineers working on this and is intended to better organize the kind of information you took screenshots of above. We are hoping to have this situated in the New Year.
Beyond what we provided above and Rich's last update, we won't have an update until after the New Year. Most of the team is leaving today for the holidays while the rest are gone after Monday, so no updates will be provided until a few weeks after we are back in the New Year and have a little more time to work.
Can we make sure the survey is available longer than a few days unlike other surveys. Most times when I see these surveys they have already been closed off but surely it would be better off to gather as much data as you can on these situations... playing from Australia I have tons of feedback to provide.Edited by James-Wayne on December 26, 2024 12:13AMPERTH, AUSTRALIA | PC | NA | @Aussie-Elders
TENTH ANNIVERSARY - Thanks for sticking with us for 10 years.
James-Wayne you earned this badge 9:56AM on 4th of February 2024.
529 people have also earned this badge.5 -
blktauna✭✭✭✭✭
✭✭
It actually is. I'm logged into it right now:
I use it to test upcoming changes in the weeks the PTS is used to gather feedback for a new patch, but I also test builds on the PTS all year long before replicating those on the live server (to see which sets I want to farm or craft, and to not waste countless of potions while parsing).
Yes because its available for testing of the upcoming stuff now. They used to close it off several times a year because I go to test out some builds at times, but the server was unavailable on multiple occasions.PCNA
PCEU0 -
BasP✭✭✭✭✭
Yes because its available for testing of the upcoming stuff now. They used to close it off several times a year because I go to test out some builds at times, but the server was unavailable on multiple occasions.
Hmm, in that case I must've imagined all those hours I've spent on the PTS outside of the weeks in which an upcoming patch is tested then. My bad.
There isn't any new stuff on the PTS at the moment by the way; it's the same update as the Live Server.1 -
blktauna✭✭✭✭✭
✭✭Maybe that's changed then. I stopped doing pts last year so it might be different now. But one of the reasons I stopped was it was offline so much.PCNA
PCEU0 -
edward_frigidhands✭✭✭✭✭I'm gonna be that person. I put a ton of responses in here, Richs feedback, other dev comments, took information from Reddit and told ChatGPT to analyze all of the available information. I'm sure someone at ZoS has already done this and their team knows more about this issue than ChatGPT but none the less I'm going to put it here in the 0.5% chance it sparks something to help solve the problem.
I am curious how this has persisted since April 2024 without some progress.5
