ZOS, Massive Spike in Ping/LAG in Recent Days - What Gives?

Destai

ZOS_Kevin wrote: »

Hi all, brief update here. Didn't get info back until late yesterday, so apologies for the delay in update here. We have increased the number of engineers investigating this and working with our networking team to go back over past server data to see if there are major discrepancies we can identify. We are still submitting your feedback to the team to make sure they have the latest info when continuing their investigation.

Also to address closing of other threads, we are rerouting those conversation points here to make sure all feedback and commentary is in one place. It's the best way that we can send info to all relevant teams investigating here and make sure we are not missing any critical piece of info from you as reports or updates come in.

Thanks for the update. There's frequent questions about the impact of Akamai and the new AI chat monitor on this issue. Can you acknowledge those questions and get some answers to them?

KaironBlackbard

Xbox NA - perfectly fine
PC NA - practically dead. 999+ ping 90% if the time, half the time spent frozen. Unplayable. Haven't even started dailies yet.

Edit: I take that back. Vvardenfell toon having insane lag, but my toon in hew's bane isn't going over 165 ping, not going under 120 though, but that's normal for mine. I question why.

Edit 2: hit 500 ping in hews bane. Didn't really stutter though, course I was just grabbing research at the time. Hope it doesn't nuke me during daily quests.

Edit 3: hew's bane guy went to deshaan, picked up fighters and mages guild quests, perfectly fine.
Went to grahtwood, lag spike, go through gate, lag spike, head to skyshard, get into combat, lag spike, pick up lorebook, booted from server.

p00tx

KaironBlackbard wrote: »

Xbox NA - perfectly fine
PC NA - practically dead. 999+ ping 90% if the time, half the time spent frozen. Unplayable. Haven't even started dailies yet.

Edit: I take that back. Vvardenfell toon having insane lag, but my toon in hew's bane isn't going over 165 ping, not going under 120 though, but that's normal for mine. I question why.

Edit 2: hit 500 ping in hews bane. Didn't really stutter though, course I was just grabbing research at the time. Hope it doesn't nuke me during daily quests.

Edit 3: hew's bane guy went to deshaan, picked up fighters and mages guild quests, perfectly fine.
Went to grahtwood, lag spike, go through gate, lag spike, head to skyshard, get into combat, lag spike, pick up lorebook, booted from server.

XBox NA is terrible in 12 person content. The disconnects are almost nonstop and the lag spikes are getting people killed.

KaironBlackbard

We need the ability to read instances, kind of like we had in Neverwinter, which is honestly a dead game since they changed everything and removed most of the content. Not the point. Point it they had the ability for players to see various instances and hop between them. They could see the ping and capacity and choose which one would be best, to give them the best experience. As is now the game randomly selects an instance, and half the time it ends up being a bad one. You (the player) can hop a dozen zones and when you return it's the same terrible instance for the zone you need to work in. And you (the player) can't change it. I wish we could. Then we could find and instance that isn't as bad and play there.

KaironBlackbard

p00tx wrote: »

KaironBlackbard wrote: »

Xbox NA - perfectly fine
PC NA - practically dead. 999+ ping 90% if the time, half the time spent frozen. Unplayable. Haven't even started dailies yet.

Edit: I take that back. Vvardenfell toon having insane lag, but my toon in hew's bane isn't going over 165 ping, not going under 120 though, but that's normal for mine. I question why.

Edit 2: hit 500 ping in hews bane. Didn't really stutter though, course I was just grabbing research at the time. Hope it doesn't nuke me during daily quests.

Edit 3: hew's bane guy went to deshaan, picked up fighters and mages guild quests, perfectly fine.
Went to grahtwood, lag spike, go through gate, lag spike, head to skyshard, get into combat, lag spike, pick up lorebook, booted from server.

XBox NA is terrible in 12 person content. The disconnects are almost nonstop and the lag spikes are getting people killed.

I ran around overland on 2 characters doing fighters and mages dailies and had no hiccups on xbox.

However, trying to do it on PC and I can't do squat.

I got a feeling it's specific servers. If we had the ability to manually choose instances, we could avoid the problem a bit and perhaps through everyone avoiding/abandoning a specific server, the devs would be able to figure out where the problem is and get it fixed.
As is now, we don't have that capability.

LadyGP

KaironBlackbard wrote: »

p00tx wrote: »

KaironBlackbard wrote: »

Xbox NA - perfectly fine
PC NA - practically dead. 999+ ping 90% if the time, half the time spent frozen. Unplayable. Haven't even started dailies yet.

Edit: I take that back. Vvardenfell toon having insane lag, but my toon in hew's bane isn't going over 165 ping, not going under 120 though, but that's normal for mine. I question why.

Edit 2: hit 500 ping in hews bane. Didn't really stutter though, course I was just grabbing research at the time. Hope it doesn't nuke me during daily quests.

Edit 3: hew's bane guy went to deshaan, picked up fighters and mages guild quests, perfectly fine.
Went to grahtwood, lag spike, go through gate, lag spike, head to skyshard, get into combat, lag spike, pick up lorebook, booted from server.

XBox NA is terrible in 12 person content. The disconnects are almost nonstop and the lag spikes are getting people killed.

I ran around overland on 2 characters doing fighters and mages dailies and had no hiccups on xbox.

However, trying to do it on PC and I can't do squat.

I got a feeling it's specific servers. If we had the ability to manually choose instances, we could avoid the problem a bit and perhaps through everyone avoiding/abandoning a specific server, the devs would be able to figure out where the problem is and get it fixed.
As is now, we don't have that capability.

I hear you but something like this would be difficult to implement and not anything I see ZoS doing given the amount of time this game has been out and overall lifespan remaining of this game.

Theist_VII

Theist_VII wrote: »

Seems like they’re keeping this thread up from May, so I’ll just drop the body of mine here:

Bad enough the game can’t be balanced respectfully, now it seems we can’t even play it.



I don’t know what’s going on behind the scenes, but a lot of us are starting to get fed up.

Now I’m not saying my opinion is more valid or important than anyone else, but when you have customers like me paying a premium price for your product…


Is it wrong to expect a premium product?

It was bad enough when Cyrodiil used to be the only place this happened, but now the servers are so horrible that it’s happening everywhere, even in solo content like my run of Infinite Archive.

Priorities need to change, if there is any expectation of this game lasting till it’s 20th Anniversary game performance needs to come first… and no I’m not talking about server maintenance, I’m talking about rearranging your budget for better servers.

This was posted from an Xbox NA experience.

This is what it looks like to be experiencing the lag we’re talking about…



@KaironBlackbard perhaps it might be worth looking into your PC’s ethernet cable resistance? When I play on my PC, it’s the exact same problem as on Xbox, one is not worse than the other.

KaironBlackbard

Theist_VII wrote: »

@KaironBlackbard perhaps it might be worth looking into your PC’s ethernet cable resistance? When I play on my PC, it’s the exact same problem as on Xbox, one is not worse than the other.

Not the point. It's the servers.
The servers it sent my Xbox play to was good.
A couple of the servers it is sending my PC play to are good, but most are bad.

Theist_VII

KaironBlackbard wrote: »

Theist_VII wrote: »

@KaironBlackbard perhaps it might be worth looking into your PC’s ethernet cable resistance? When I play on my PC, it’s the exact same problem as on Xbox, one is not worse than the other.

Not the point. It's the servers.
The servers it sent my Xbox play to was good.
A couple of the servers it is sending my PC play to are good, but most are bad.

Then what’s the point exactly?

You’re saying performance is better on Xbox servers, it is not. I got on my PC for the PTS and hopped over to live and the problems are near identical.

There are no “good servers” it’s all just one megaserver with players sharded into different instances.

You’ve got XB NA and XB EU, PC NA and PC EU, and lastly PS NA and PS EU. Those are your servers.

MedicInTheWild

Still don't see this on the DEV TRACKER, already mentioned it before. Would make this easier as a whole with the multiple forums.

@ZOS_Kevin Can we add this since it is or should be considered a MAJOR issue?

manukartofanu

MedicInTheWild wrote: »

Still don't see this on the DEV TRACKER, already mentioned it before. Would make this easier as a whole with the multiple forums.

@ZOS_Kevin Can we add this since it is or should be considered a MAJOR issue?

What is DEV TRACKER?

vsrs_au

manukartofanu wrote: »

MedicInTheWild wrote: »

Still don't see this on the DEV TRACKER, already mentioned it before. Would make this easier as a whole with the multiple forums.

@ZOS_Kevin Can we add this since it is or should be considered a MAJOR issue?

What is DEV TRACKER?

It's a forum section, there's a button at the top of the forum page labeled "DEV TRACKER".

KaironBlackbard

Theist_VII wrote: »

KaironBlackbard wrote: »

Theist_VII wrote: »

@KaironBlackbard perhaps it might be worth looking into your PC’s ethernet cable resistance? When I play on my PC, it’s the exact same problem as on Xbox, one is not worse than the other.

Not the point. It's the servers.
The servers it sent my Xbox play to was good.
A couple of the servers it is sending my PC play to are good, but most are bad.

Then what’s the point exactly?

You’re saying performance is better on Xbox servers, it is not. I got on my PC for the PTS and hopped over to live and the problems are near identical.

There are no “good servers” it’s all just one megaserver with players sharded into different instances.

You’ve got XB NA and XB EU, PC NA and PC EU, and lastly PS NA and PS EU. Those are your servers.

One server can't handle 15 millions people. I doubt one can handle 500000. Each datacenter has multiple servers. Each server has many instances. We keep getting connected to a bad server. 1% isn't 100%. Why do you think it's not fixed yet? Only a few servers are having issues, not all of them.
Every coast has a datacenter, if not multiple.
Datacenters can have hundreds to thousands of servers.
Only a few are having problems, and it's probably just old tech or old wires that either way need to be replaced, if not old OS or software that needs an update.

RaikaNA

As of 7:56 PM EST... I can't log into the server... It keeps giving me this message:

BlueViolet

At this point I don't even know why I'm bothering to pay for one, let alone two subscriptions, because nearly every single time I want to play I have to fight the game to just let me log in. Its driving me bonkers.

I tried to log in yesterday afternoon to play but immediately recieved the "Lost connection to Lobby: error. AGAIN.

RomanRex

rubber banding in PC/NA Ravenwatch for me. stopped trying.

FelisCatus

In before, thread is locked for not being constructive.

MrGarlic

PC/EU is appalling tonight, after reset. 999+ ping and taking 2 seconds for my characters to sprint after key press.

I did manage to complete endeavours, but it was a challenge with rubber-banding and skill delays.

Just awful. Not going to bother playing tonight. No wonder people are leaving the game.

Just_Attivi

Forreal though I was warping time around like Dr Who, and saw christmas past present and future too!
and then I crashed. once was enough.

skyrimfantasy

Last night, just standing idle in Riften and I was hit with an error report box again and booted out of the game. I try to log in once a day since I'm a GM and I'll play until I'm booted but that's it. Might be an hour, might be 5 min, who knows when the game is going to kick you for whatever unknown reason. I am not relogging, I'm done, I'll try again tomorrow.

LadyGP

skyrimfantasy wrote: »

Last night, just standing idle in Riften and I was hit with an error report box again and booted out of the game. I try to log in once a day since I'm a GM and I'll play until I'm booted but that's it. Might be an hour, might be 5 min, who knows when the game is going to kick you for whatever unknown reason. I am not relogging, I'm done, I'll try again tomorrow.

This is kind of how I play as well. Unless it's a raid day I don't log back in. On raid days it's a relog every 20ish minutes or so.

thinkaboutit

@ZOS_Kevin what is the likelihood we can get a networking SME on this thread to work with us on providing more in-depth details i.e. packet captures from our end etc.. ?

LadyGP

@ZOS_Kevin Assuming we get this fixed at some point would it be possible to get a write up about what caused the issue? for those of us who have been experiencing this since May I think it would be helpful to know the "how/why".

peacenote

I happened to notice this thread so will share my experiences in case it is helpful to whoever is troubleshooting. I'm PC/NA.

Tuesday night I attended a raid, 9pm EST. I had no trouble logging in but I had a really tough time playing. Often my skills would not go off, even if I stopped hitting all buttons, paused, and then just slowly hit a single one. Once, I had to hit my ultimate 5 times before it would trigger.

Wednesday night I changed NOTHING about my setup and I couldn't log in. Around 8pm EST. Got the lobby error. Got it 4 times. Was too tired to troubleshoot so gave up.

This morning, around 8:45am EST, I tried logging in on multiple characters. Still lobby error. I disabled all add-ons (which theoretically shouldn't be the problem since I was able to get in fine Tuesday night with them active and I hadn't updated them) and now my characters can log in... but it is still EXTREMELY SLOW whereas normally when I disable add-ons login is usually instantaneous. Once in, the game seems unplayable. I could barely walk a foot or two over to the horse trainer on an alt parked right next to one. It just kind of felt like my computer all of a sudden couldn't handle the game. Huge delay when walking, when moving the camera. No way I could even do writs like this.

My conclusion is that there's clearly some kind of back end performance issue, even though disabling add-ons technically got me into the game. It just changed the situation from not being able to get in at all, to not being able to do anything once I was in.

LadyGP

Passing this Discord message along that I got. This person never got a forum invite so they can't post.


Again, imagine how many people don't have forum access or even know about the forums who are having issues. I think this is a widespread issue.. much larger than what is being reported on here.

also @ZOS_Kevin is it possible to get weekly updates on where we are at with the investigation? (I know I'm being needy but... this has been personally effecting me for many...many months now).

Aurielle

Great event, guys.

Trying to do crafting writs in WW was almost impossible, because massive lag spikes were preventing me from interacting with crafting tables and writ turn in coffers. Then I tried to actually participate in the event and got kicked from the game when approaching a world boss. I've tried to log in again for the past ten minutes, and I can't.

I think I might be done.

EDIT: Wow. I actually managed to log in, and found myself ROLLED BACK from where I was when I got booted from the server.



Red x is where I actually was on the map when I got booted.

Estin

The lag this week is a lot worse than last week. I'm in my own home (Sleek Creek House) trying to learn a new class and my skills can't even fire properly. Skills will get register and fire almost a full second later or just not fire at all according to CMX. My ping never went above 110 during this time. The game also is trying to treat my light attacks as heavy attacks sometimes. I was stuck in a heavy attack animation that never went off for almost 10 seconds before it did it.

Huge Delay between skills getting registered and firing


Skill constantly being registered but never firing


Light attacks suddenly registering as heavy attacks

LadyGP

Raid tonight - lets see how it goes.

Just_Attivi

-load ESO
-Queue GH
-played 3 minutes

An error has occurred, please wait before logging in again

#Itakebackallmypraisefixthisgarbage

Nharimlur_Finor

Photosniper89 wrote: »

Hoping ZOS can chime in and give some context as to what is happening behind the scenes (without giving away possible unannounced changes).

In recent days there has been a massive spike in PC NA ping/lag. Usually when the servers go down for maintenance the ping/lag improves for a few days but the spike we have seen the last week or so is abnormal. Affected areas are both PvE and PvP - I've experienced it most in PvP, though. Based on zone chat this appears to be a widespread issue and not localized to a handful of players. Things that have been observed are ping spikes to 999+ for an extended period of time (3-5 seconds) followed by massive ruberbanding or simply the game freezing then unfreezing after a few seconds.

Overall the average ping has risen from around the 75ish (for me) to 350ish.

With the Cyro issues a few weeks back that were never explained, then the recent PTS issue, I'm making an assumption that there is a large effort behind the scenes to either refactor some code/networking or some other massive network/DB overhaul that's yet to be announced (which I don't blame ZOS for not announcing these efforts due to the last few attempts not working and the player base going full agro mode).

Anyways, haven't seen a lot of dialogue about these recent issues so wanted to get the conversation going and hoping that someone at ZOS could chime in and acknowledge they are aware and maybe give a hint at some reason behind these aggravating issues as of late.

https://en.wikipedia.org/wiki/Microsoft_Gaming

Microsoft Gaming shut down three development studios on May 7, 2024: Alpha Dog Games in Canada, Arkane Austin in the US, and Tango Gameworks in Japan. Additionally, Roundhouse Studios will merge with ZeniMax Online Studios in the US.[60] According to Microsoft Gaming, this move is a part of a larger "reprioritization of titles and resources" that Microsoft's gaming division is carrying out. The intention is to boost investment in their library of games and new intellectual assets while focusing on high-impact titles. Some teams have been reassigned to other projects inside Bethesda and ZeniMax, and production on some games has stopped as a result of these companies closing.