ZOS, Massive Spike in Ping/LAG in Recent Days - What Gives?

VouxeTheMinotaur · August 20

Photosniper89 wrote: »

Tandor wrote: »

Is Akamai only used along certain routes, e.g. from the Southern Hemisphere countries? That would explain why most players don't have these problems (which in itself strongly supports the idea of it being down to Akamai rather than the servers, as if it was the servers everyone would have the same experience).

I highly doubt ZoS would come out and say that - they are done with this thread.

At the end of the day there is no way for us to get around it we are kind of SOL.

Is there any other way to bring this to the devs attention? If this thread is muted by Kevin and other members, which if true is absolutely terribly poor behavior considering a lot of people were giving feedback as asked, what other way is there to get this through ZOS' heads?

I mean the game is in a terrible state, are they really just doing a "sweep it under the rug" move?

shifty af if so

Photosniper89 · August 20

VouxeTheMinotaur wrote: »

Photosniper89 wrote: »

Tandor wrote: »

Is Akamai only used along certain routes, e.g. from the Southern Hemisphere countries? That would explain why most players don't have these problems (which in itself strongly supports the idea of it being down to Akamai rather than the servers, as if it was the servers everyone would have the same experience).

I highly doubt ZoS would come out and say that - they are done with this thread.

At the end of the day there is no way for us to get around it we are kind of SOL.

Is there any other way to bring this to the devs attention? If this thread is muted by Kevin and other members, which if true is absolutely terribly poor behavior considering a lot of people were giving feedback as asked, what other way is there to get this through ZOS' heads?

I mean the game is in a terrible state, are they really just doing a "sweep it under the rug" move?

shifty af if so

PvP has been swept under the rug for 7 years now why would anything be different.

All I can think of is to tag other people who haven't muted this thread. I've sent this to Kevin on social media multiple times in response to some of his ESO posts so I know he has seen we are still interested in some answers in this thread.

Jaraal · August 21

VouxeTheMinotaur wrote: »

Photosniper89 wrote: »

Tandor wrote: »

Is Akamai only used along certain routes, e.g. from the Southern Hemisphere countries? That would explain why most players don't have these problems (which in itself strongly supports the idea of it being down to Akamai rather than the servers, as if it was the servers everyone would have the same experience).

I highly doubt ZoS would come out and say that - they are done with this thread.

At the end of the day there is no way for us to get around it we are kind of SOL.

Is there any other way to bring this to the devs attention?

We're not telling them anything they don't already know. In fact, they know far more about Akamai dysfunction than we do.

Photosniper89 · August 21

Jaraal wrote: »

VouxeTheMinotaur wrote: »

Photosniper89 wrote: »

Tandor wrote: »

Is Akamai only used along certain routes, e.g. from the Southern Hemisphere countries? That would explain why most players don't have these problems (which in itself strongly supports the idea of it being down to Akamai rather than the servers, as if it was the servers everyone would have the same experience).

I highly doubt ZoS would come out and say that - they are done with this thread.

At the end of the day there is no way for us to get around it we are kind of SOL.

Is there any other way to bring this to the devs attention?

We're not telling them anything they don't already know. In fact, they know far more about Akamai dysfunction than we do.

Each red line is a crash @ZOS_Kevin

Get Akamai TF out of the routing table this is embarrassing on both parties side.

Photosniper89 · August 21

@ZOS_Kevin

Auberon1983 · August 21

Not trying to beat a dead horse, but the lag/latency is appalling right now on PS/NA. As in currently, right now.

Not Cyro, not IC, not BGs. Overland. Not even West Weald, but Western Skyrim.

I know I'm preaching to the choir, but this is the worst I've seen performance in years. Granted it isn't as bad as when Markarth dropped, but it's getting close.

YandereGirlfriend · August 22

I came here to meme about the new servers from a bit ago actually being Intel 13th and 14th Gen (look it up if you're confused - and update your BIOS if you have one of them...) but the Akamai information above is fairly conclusive evidence on its own.

My only anecdote is from last night while ballgrouping in Grey Host. The entire 12-player raid was booted from the server simultaneously upon reaching the Roe farm and when we got back on we all had giga-ping and huge amounts of input lag. Suffice to say, 12 players being booted at the same time is NOT an individual player issue but rather a ZOS issue.

dcrush · August 22

I’m doing overland quests in the Deadlands. Ping shoots up randomly and even during quest conversations there’s massive delays regularly. I’ve not had it this bad until recently. It’s affecting every area of the game, not just Cyrodiil. On some days it’s near unplayable.

Destai · August 22

@ZOS_Kevin Can we get an update on this please? I can understand you have a lot on your plate, but we should be getting some help here. We got updates on teabagging, it would be nice if we could get some details here. More than just silence or “we’re still investigating”. Engage us already.

Nharimlur_Finor · August 22

Destai wrote: »

@ZOS_Kevin Can we get an update on this please? I can understand you have a lot on your plate, but we should be getting some help here. We got updates on teabagging, it would be nice if we could get some details here. More than just silence or “we’re still investigating”. Engage us already.

A public statement should be provided for legislative purposes.

Seravi · August 22

Sigh massive lag spikes so bad I cannot finish daily writs in West Weald, PC NA. Still no response from ZOS I see.

dcrush · August 22

Doing IA, out of nowhere I get 999+ ping for about a minute and then disconnected. Every single day something like this happens and it doesn’t matter what I’m doing. Absolute trash.

Photosniper89 · August 22

LaintalAy wrote: »

Destai wrote: »

@ZOS_Kevin Can we get an update on this please? I can understand you have a lot on your plate, but we should be getting some help here. We got updates on teabagging, it would be nice if we could get some details here. More than just silence or “we’re still investigating”. Engage us already.

A public statement should be provided for legislative purposes.

My daily routine.

Make coffee.
Dial into standup.
Load this thread to see if @ZOS_Kevin ever replied.

Photosniper89 · August 22

ZOS_Kevin wrote: »

Destai wrote: »

Two questions:
Why were the related threads on the official forums not treated with a higher priority? This thread has been opened since May 7 and now, 23 days later, we get an update.

What are the other issues? Surely this is an anticipated question, as you all should know by now that performance concerns are quite visible. Can we get some background on what the other issues are?

You're assuming this was not being taken seriously. As Rich mentioned, we have been dealing with this for the last few weeks. Also to note, we normally do not comment on DDoS attacks. Rich felt it appropriate to mention as it came up during the AMA, but we've been tracking this thread for awhile now.

Again as stated earlier, we are still investigating. I'm going into another meeting about this shortly to see what other information could be provided to assist in resolving this issue. Anything provided now would be a guess at best. Once we have a better idea of what else could be causing issues, we'll communicate that.

The last 10+ pages is why people make comments like we assume you're not taking it seriously. (Rich stream trolling us too is another reason why we think the current leadership in ZoS doesn't take this issue seriously but that is a whole different conversation)

ZOS_Kevin wrote: »

Hi all. After chatting with the team, we're looking to gather more information from players who have been experiencing new and/or worse latency/lag spikes/rubber-banding issues in ESO since the beginning of May 2024 (when the emergent issues in this thread were initially reported.) To help us with our ongoing investigation, could you please answer the following questions?
What server/platform do you play on?

What is the nature of the issues you started experiencing or noticed got worse after the beginning of May 2024? (ex. did characters rubberband/warp, did abilities get delayed/fail to fire, etc.)

Around what time of day have you experienced the new and/or worse issues reported in this thread?

Where in-game have you experienced these issues? Please be as specific as possible (name/area of overland zone, delve, dungeon, Trial, etc.)

Have you experienced these issues outside of combat situations (in houses, for example)

Have you noticed these issues are worse during any specific times of day? If so, what times?

Have you noticed any changes/improvements as of today?

51 days since this - last we have heard from anyone at ZoS about this (Gina never once responded here by the way).

Where are we? What has been happening with all the information we are providing you? What if anything has been determined as the root cause of all this lag? What if anything has the team done to try to pinpoint our issues? Are there any graphics or anything you can share (even if they don't show the root cause but shows the process the team has gone through to try to nail this down?.

Transparency is NOT something ZoS is good at but I challenge everyone to ZoS to be more transparent. No one is expecting this issue to be fixed overnight - nor are we expecting them to have all the answers. What we are expecting is an acknowledgment more than "we are investigating". This is a line that has been used when us PvP'er bring up the lag for 7 years now - we are mainly just ignored tbh. Now that it's hitting PvE ZoS can't hide it anymore.

It's time to make some changes on how you communicate. Gina famously said in 2019-ish that Zos communication has not been up to par and there is no excuse for that - what changes have been made to improve coms since then? I'd argue they have only gotten worse. @ZOS_Kevin

TaSheen · August 22

Gina isn't in the community team any more; she's now doing something with "content creators".

Photosniper89 · August 22

TaSheen wrote: »

Gina isn't in the community team any more; she's now doing something with "content creators".

Yup, which is right around the same time @ZOS_Kevin stopped responding.

Jaraal · August 22

Photosniper89 wrote: »

TaSheen wrote: »

Gina isn't in the community team any more; she's now doing something with "content creators".

Yup, which is right around the same time @ZOS_Kevin stopped responding.

The entire dev team has only responded to about 6 forum queries this month according to the Dev Tracker, but one of them was from Gina. So it appears she is still popping in occasionally to help out.

FoJul · August 22

The game for me and a lot of other's is going into unplayable territory.

vsrs_au · August 22

FoJul wrote: »

The game for me and a lot of other's is going into unplayable territory.

For me, it's just gone into "not playing" mode, i.e. I haven't logged in for about 5 weeks, for various reasons I won't describe here because of how this forum works.

blktauna · August 22

I'm on East Coast US and on I go through 4 akamai servers before traceroute crashes...
198.20.200.1 as per their direction for game issues.

Photosniper89 · August 22

Jaraal wrote: »

Photosniper89 wrote: »

TaSheen wrote: »

Gina isn't in the community team any more; she's now doing something with "content creators".

Yup, which is right around the same time @ZOS_Kevin stopped responding.

The entire dev team has only responded to about 6 forum queries this month according to the Dev Tracker, but one of them was from Gina. So it appears she is still popping in occasionally to help out.

Curious if they all still work from home....

ZOS_Kevin · August 22

Sorry for the long absence, all. I was away for work travel over the last several weeks and catching up on things before launch.

After checking back in with the team, there isn't much progress to report right now. There were several attempts at pushing forth possible solutions, but those attempts have not resolved the issue. The engineering teams have poured over your reports, player logs, etc. to see if there is a commonality to tackle.

We are still continuing to investigate. However, outside of reporting issues as you come across them, there isn't much else at the moment to do outside of our engineering teams continuing to test for possible solutions. We are still sharing reports as they come in with those teams for investigation.

Photosniper89 · August 22

ZOS_Kevin wrote: »

Sorry for the long absence, all. I was away for work travel over the last several weeks and catching up on things before launch.

After checking back in with the team, there isn't much progress to report right now. There were several attempts at pushing forth possible solutions, but those attempts have not resolved the issue. The engineering teams have poured over your reports, player logs, etc. to see if there is a commonality to tackle.

We are still continuing to investigate. However, outside of reporting issues as you come across them, there isn't much else at the moment to do outside of our engineering teams continuing to test for possible solutions. We are still sharing reports as they come in with those teams for investigation.

Has Akamai been discussed at all? Its starting to look like that may be the issue and packet loss.

I've literally changed all of my hardware.

Different pc's, different router/wifi, different cables, differnt ISP (starlink vs fiber) and the results are the exact same. Akamai packet loss every 20ish or so minutes which results in crashes or the massive rubber banding.

Things got slightly better and the rubber banding stopped (dc's didnt) a few weeks back when Akamai was removed from the routing table. Now that it is back it's worse.

Jaraal · August 22

Photosniper89 wrote: »

ZOS_Kevin wrote: »

Sorry for the long absence, all. I was away for work travel over the last several weeks and catching up on things before launch.

After checking back in with the team, there isn't much progress to report right now. There were several attempts at pushing forth possible solutions, but those attempts have not resolved the issue. The engineering teams have poured over your reports, player logs, etc. to see if there is a commonality to tackle.

We are still continuing to investigate. However, outside of reporting issues as you come across them, there isn't much else at the moment to do outside of our engineering teams continuing to test for possible solutions. We are still sharing reports as they come in with those teams for investigation.

Has Akamai been discussed at all? Its starting to look like that may be the issue and packet loss.

I did some rudimentary research on Akamai's responsibility for adding latency and packet loss to data transmissions, and it's definitely not an ESO exclusive problem. Many other games and services are hindered by the effect, although curiously, ESO seems to be exceptionally hard hit by the dysfunction. Some sort of synergistic compounding of lag related to the way the Zenimax servers communicate with the Akamai servers, and vice versa.

Seravi · August 23

ZOS_Kevin wrote: »

Sorry for the long absence, all. I was away for work travel over the last several weeks and catching up on things before launch.

After checking back in with the team, there isn't much progress to report right now. There were several attempts at pushing forth possible solutions, but those attempts have not resolved the issue. The engineering teams have poured over your reports, player logs, etc. to see if there is a commonality to tackle.

We are still continuing to investigate. However, outside of reporting issues as you come across them, there isn't much else at the moment to do outside of our engineering teams continuing to test for possible solutions. We are still sharing reports as they come in with those teams for investigation.

Around June 18th the Akami routes were no longer in my route to the server. This lasted for a while and it was glorious! Now they are back and so is the skill fire delay, major lag spikes, etc. @ZOS_Kevin maybe you could check with the powers that be and find out why the Akami routes were taken out and see if this can be done again? I am in Texas within cuddle range of the Data Center so there is no "another state" "another country" and such issue with my connection.

ZOS_Kevin · August 23

Akamai has been discussed and conversations with Akamai happen regularly. However, we will bring this up with the team in our next meeting and see if the observations made here can be explored further.

Photosniper89 · August 23

ZOS_Kevin wrote: »

Akamai has been discussed and conversations with Akamai happen regularly. However, we will bring this up with the team in our next meeting and see if the observations made here can be explored further.

Glad to see you back in the thread!

I'd be curious to see what the feedback is.

Unrelated - turning shadows off fixed alot of the lag when around azure blight heavy fights so there is something with blight and shadows thats going on in parallel with the akamai packet loss issues.

Jaraal · August 23

ZOS_Kevin wrote: »

Akamai has been discussed and conversations with Akamai happen regularly. However, we will bring this up with the team in our next meeting and see if the observations made here can be explored further.

Thank you for that. Ideally, ZOS should have an in-house DDoS system that functions as Akamai does, however I'm sure it wouldn't be cheap and I'm not sure the game engine/data transmission system is designed to allow for that type of functionality. Eliminating the middleman seems like it would alleviate some issues.

I'm sure you aren't allowed to go into any sort of detail, Kevin.... but could you give us a safely vague explanation for why Microsoft doesn't stay in-house with Azure for your DDoS protection? Seems logical that using your own processes rather than paying somebody else to do the dirty work for you would be more economical, not to mention having greater control of the protocols and processes themselves.

ZOS_Kevin · August 23

Jaraal wrote: »

I'm sure you aren't allowed to go into any sort of detail, Kevin.... but could you give us a safely vague explanation for why Microsoft doesn't stay in-house with Azure for your DDoS protection? Seems logical that using your own processes rather than paying somebody else to do the dirty work for you would be more economical, not to mention having greater control of the protocols and processes themselves.

That info unfortunately is far above my paygrade. Even if someone shared that info, there is no way I would be allowed to share that.

Photosniper89 wrote: »

Unrelated - turning shadows off fixed alot of the lag when around azure blight heavy fights so there is something with blight and shadows thats going on in parallel with the akamai packet loss issues.

Thanks for the feedback here. Will pass this along as well.

Photosniper89 · August 23

ZOS_Kevin wrote: »

Jaraal wrote: »

I'm sure you aren't allowed to go into any sort of detail, Kevin.... but could you give us a safely vague explanation for why Microsoft doesn't stay in-house with Azure for your DDoS protection? Seems logical that using your own processes rather than paying somebody else to do the dirty work for you would be more economical, not to mention having greater control of the protocols and processes themselves.

That info unfortunately is far above my paygrade. Even if someone shared that info, there is no way I would be allowed to share that.

Photosniper89 wrote: »

Unrelated - turning shadows off fixed alot of the lag when around azure blight heavy fights so there is something with blight and shadows thats going on in parallel with the akamai packet loss issues.

Thanks for the feedback here. Will pass this along as well.

To further this problem along - anything we can run on our machines in parallel with the game to help troubleshoot?

I've been running ping plotter when I run ESO and every time I crash or get the 5-10 second rubber band there is a 100% packet loss on the Akamai route. I know I'm just one case but curious if there is anymore info we can gather to drill down why some people have issues with Akamai and others done.

Again, I have had this issue with fiber, starlink, and DLS. All different locations, different routers, different cables, etc so it seems like it's more "location" specific then hardware/isp/firewall specific.