ZOS, Massive Spike in Ping/LAG in Recent Days - What Gives?

Photosniper89

Jaraal wrote: »

Photosniper89 wrote: »

I'm in the process of documenting everything that has been brought up in this thread, my experience with pvp going back since launch but more importantly the lack of action and significant decrease in performance these last 5+ years.

In the few times they've publicly addressed the performance issues, data bloat was the primary reason given. They eliminated data such as alternate character achievements to 'make room for new systems and achievements' (such as companions, the card game, etc.). They've pared down data that they felt was less relevant (deleting deer in Cyrodiil, both announced and stealth removal of spell effect visuals, reworking glyphs to be less data intensive, reducing mail timers and storage, slashing group sizes in half, to name a few) while allowing things that are profitable (new gear sets, flashy mounts, polymorphs, arms packs and whatnot) to continually increase in number and complexity.

And in the ever enlarging efforts (over 650 unique armor/weapon sets and growing) to come up with new 5th piece ideas, effects are becoming more and more complicated.... having to keep track of resource percentages and constantly changing skill parameters of numerous enemies and allies wearing a plethora of different possible combinations... it's easy to see how servers can become quickly overwhelmed. And when they were installing the new servers a while back, they told us not to expect performance increases, as the new hardware changes were more focused on back end processes and less forward facing effects. They also noted that the data rework/sharding efforts weren't producing the results they expected, so they walked that back as well.

So, as the game continues to get bigger and more bloated with each update and pixel-heavy crown mount/polymorph/recall/interactive furnishings, while retaining the same data limits... performance will only continue to get worse. There will come a point where there will be less and less long standing game features to delete to make room for new data. And what will happen then? Only time will tell.

All very good points.

Seravi

Game response was MUCH better when Akami was out of the loop. Removing Akami is clearly something they "tried" as I had a few weeks without Akami and it was awesome. Now I have two Akami routes and the issues are back. Is Akami the only reason for the problems, of course not. It does help though when those two bottlenecks are removed.

I am really struggling with why I continue to keep playing hours every day. I am right by the data center and my routes are 2-3ms every where EXCEPT Akami. They are local routes but typical Akami bottlenecks. I should not have to keep hitting my key as fast as I can to get one or two skill fires off against the enemy.

Photosniper89

Seravi wrote: »

Game response was MUCH better when Akami was out of the loop. Removing Akami is clearly something they "tried" as I had a few weeks without Akami and it was awesome. Now I have two Akami routes and the issues are back. Is Akami the only reason for the problems, of course not. It does help though when those two bottlenecks are removed.

I am really struggling with why I continue to keep playing hours every day. I am right by the data center and my routes are 2-3ms every where EXCEPT Akami. They are local routes but typical Akami bottlenecks. I should not have to keep hitting my key as fast as I can to get one or two skill fires off against the enemy.

To your point about why do you come back - we all ask ourself that. Luckily AoC alpha 2 keys go on sale next week. I already know of two entire guilds that are leaving ESO when phase 3 goes live. Their lack of action on these performance issues will financially impact them drastically once AoC launches.

Now onto the main part of your post

---

I really think you are on to something. I ran a path ping and everything is 2-3ms except - you guess it - Akami.

I'd be willing to bet that would explain why all the rubber banding and crashing is happening. I agree with you - there was a window there last month where it was significantly better then when this first thread was created (Azure blight was the issue with crashes - you can see it within peoples clips blight goes off and they crash) but now the rubber banding is back and if I synced my clips with pingplotters table I'd be willing to be you it's everytime time it hits Akami the rubber banding starts.

@ZOS_Kevin Surely you still see this thread even though you no longer comment on it.

Take this back to the dev/network engineers. Get Akami TF out of ZoS pathing.

Sluggy

For what it's worth, I logged on last night and had only minor issues - the usual stuff that happens when many people show up to a hammer fight. But other than that, no disconnects. No (unusual) lag spikes. If I wasn't enjoying myself so much I probably might have stopped to run a trace route but who's gonna bother with that when everything is working. Considering how on-and-off it is and how it only affects certain people this to me means either it has to do with the the path taken to the server or it has to do with the specific machine they land on once the load balancing decides who goes where in the internal system.

Maybe next time I'll try to do a path comparison between a bad night and a good night and see what pops up.

TaSheen

I wouldn't wonder if you guys haven't hit the nail on the head. One of my minor financial companies uses Akamai for whatever stupid reason, and I do have these issues with their site loading (sometimes it won't load at all, sometimes it's like molasses running uphill in January at the North Pole).

I don't have these issues with ESO though - and I think that's because my route to the server never hits Akamai:

C:\Windows\system32>tracert eso.com

Tracing route to eso.com [141.193.213.21]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.42.1
2 559 ms 616 ms 647 ms 67.143.249.65
3 575 ms 571 ms 560 ms 67.143.253.5
4 621 ms 616 ms 582 ms 67.143.253.4
5 577 ms 604 ms 616 ms 67.143.253.74
6 546 ms 571 ms 572 ms 192.168.29.1
7 594 ms 604 ms 605 ms hu0-15-0-1.sl-crs2-che.swl.cogentco.com [144.232.166.249]
8 631 ms 649 ms 627 ms be3.sl-crs2-oro.swl.cogentco.com [144.232.22.3]
9 613 ms 604 ms 648 ms be2.sl-crs2-stk.swl.cogentco.com [144.232.15.239]
10 591 ms 593 ms 638 ms be3.sl-crs2-sj.swl.cogentco.com [144.232.22.178]
11 656 ms 626 ms 603 ms ae22.0.sl-mst51-sj2.swl.cogentco.com [144.232.2.106]
12 597 ms 648 ms 615 ms be3012.ccr41.sjc03.atlas.cogentco.com [144.232.8.194]
13 625 ms 604 ms * 38.88.224.86
14 645 ms 637 ms 637 ms 172.68.188.76
15 606 ms 616 ms 583 ms 141.193.213.21

Unless of course Akamai is part of cogentco.com (the which I don't know).

Rkindaleft

Photosniper89 wrote: »

@ZOS_Kevin It's disappointing that prior to Gina leaving you were so active in this thread and since then it's been radio silent from you. I can only speculate on why that is (won't do that on here).

With that being said.... performance is back to being worse than BEFORE I MADE THIS THREAD.

Rubber banding is back in full force and it's a 5+ second full lock up followed by fast fwd at 200x speed until you catch back up to "real time".

I'm struggling to remember a time when performance was this bad - and I'm not being dramatic there.

Also - Azureblight is causing a metric (poop) ton of lag spikes.

I haven't experienced issues as regularly as some of the others here but the lack of communication and clarity has probably been the worst part of this whole situation to be honest. Lag issues and disconnects have had their ups and downs since this thread was created 3 months ago but for a lot of people the problems have gotten significantly worse in the recent weeks and nobody from ZOS has responded here in over a month.

Seravi

@ZOS_CouchTato I saw you replied on another lag thread recently. Is there any way you can get attention on this? It has been quite a while since anyone from ZOS commented on this on going issue. Thanks!!

ZOS_CouchTato

@Seravi I do not currently have any updates or known issues concerning this, but I can share the links for some troubleshooting:

For PC: https://help.elderscrollsonline.com/#en/answer/51890

For Xbox: https://help.elderscrollsonline.com/#en/home/category/8/platform/20/techissuetype/877

For PlayStation: https://help.elderscrollsonline.com/#en/home/category/8/platform/19/techissuetype/877

It would also help to send in a ticket to Support if you are still having the issue, you can do that by clicking "Continue to Submit Ticket" or, "Need More Help" at the bottom of the article.

FoJul

Maybe this can help some people with high ping, idk if it was mentioned in this thread or not.

My friend, gave me a little trick I wasn't aware of before:

Open command prompt (cmd), and right click it and run as admin.

Simply just type ipconfig /flushdns >hit enter> type exit> hit enter.

All this does is flush the network cache. It didn't 100% fix the issue, but i haven't been sitting at 600 ping anymore.

There is other command prompts you can try, feel free to google, but that one specifically helped my game quite a bit.

Hope this helps someone!

Sluggy

FoJul wrote: »

Maybe this can help some people with high ping, idk if it was mentioned in this thread or not.

My friend, gave me a little trick I wasn't aware of before:

Open command prompt (cmd), and right click it and run as admin.

Simply just type ipconfig /flushdns >hit enter> type exit> hit enter.

All this does is flush the network cache. It didn't 100% fix the issue, but i haven't been sitting at 600 ping anymore.

There is other command prompts you can try, feel free to google, but that one specifically helped my game quite a bit.

Hope this helps someone!

I do this basically every time I have issues with this game along with several other similar tools that can be used. It's never once shown to make any difference whatsoever.

thinkaboutit

Seravi wrote: »

Game response was MUCH better when Akami was out of the loop. Removing Akami is clearly something they "tried" as I had a few weeks without Akami and it was awesome. Now I have two Akami routes and the issues are back. Is Akami the only reason for the problems, of course not. It does help though when those two bottlenecks are removed.

I am really struggling with why I continue to keep playing hours every day. I am right by the data center and my routes are 2-3ms every where EXCEPT Akami. They are local routes but typical Akami bottlenecks. I should not have to keep hitting my key as fast as I can to get one or two skill fires off against the enemy.

I doubt they removed Akamai - its more likely we get load balanced on different in unpredictable ways, that's why the 'horribleness' is spread around and doesn't happen to everyone in a 12 man group at once. Just small clusters of them.

Funny how it works in their benefit though, because issues like this can easily be incorrectly blamed on the individual customer internet side.

FoJul

Sluggy wrote: »

FoJul wrote: »

Maybe this can help some people with high ping, idk if it was mentioned in this thread or not.

My friend, gave me a little trick I wasn't aware of before:

Open command prompt (cmd), and right click it and run as admin.

Simply just type ipconfig /flushdns >hit enter> type exit> hit enter.

All this does is flush the network cache. It didn't 100% fix the issue, but i haven't been sitting at 600 ping anymore.

There is other command prompts you can try, feel free to google, but that one specifically helped my game quite a bit.

Hope this helps someone!

I do this basically every time I have issues with this game along with several other similar tools that can be used. It's never once shown to make any difference whatsoever.

I'm sorry to hear, maybe yours is a different issue, but it did fix about 75% of my pinging issues. As well as helping my friend's too.

Photosniper89

ZOS_CouchTato wrote: »

@Seravi I do not currently have any updates or known issues concerning this, but I can share the links for some troubleshooting:

For PC: https://help.elderscrollsonline.com/#en/answer/51890

For Xbox: https://help.elderscrollsonline.com/#en/home/category/8/platform/20/techissuetype/877

For PlayStation: https://help.elderscrollsonline.com/#en/home/category/8/platform/19/techissuetype/877

It would also help to send in a ticket to Support if you are still having the issue, you can do that by clicking "Continue to Submit Ticket" or, "Need More Help" at the bottom of the article.

I appreciate you trying to help - I truly do.

However, all of these troubleshooting threads are complete BS relating to the lag issue. IT IS NOT AN ISSUE ON OUR SIDE THIS IS A ZOS ISSUE! Can someone please send a pigeon with a letter to @ZOS_Kevin and ask him to unmute his notis for this thread and give us an update?

Photosniper89

thinkaboutit wrote: »

Seravi wrote: »

Game response was MUCH better when Akami was out of the loop. Removing Akami is clearly something they "tried" as I had a few weeks without Akami and it was awesome. Now I have two Akami routes and the issues are back. Is Akami the only reason for the problems, of course not. It does help though when those two bottlenecks are removed.

I am really struggling with why I continue to keep playing hours every day. I am right by the data center and my routes are 2-3ms every where EXCEPT Akami. They are local routes but typical Akami bottlenecks. I should not have to keep hitting my key as fast as I can to get one or two skill fires off against the enemy.

I doubt they removed Akamai - its more likely we get load balanced on different in unpredictable ways, that's why the 'horribleness' is spread around and doesn't happen to everyone in a 12 man group at once. Just small clusters of them.

Funny how it works in their benefit though, because issues like this can easily be incorrectly blamed on the individual customer internet side.

Bingo pt.2

Seravi

ZOS_CouchTato wrote: »

@Seravi I do not currently have any updates or known issues concerning this, but I can share the links for some troubleshooting:

For PC: https://help.elderscrollsonline.com/#en/answer/51890

For Xbox: https://help.elderscrollsonline.com/#en/home/category/8/platform/20/techissuetype/877

For PlayStation: https://help.elderscrollsonline.com/#en/home/category/8/platform/19/techissuetype/877

It would also help to send in a ticket to Support if you are still having the issue, you can do that by clicking "Continue to Submit Ticket" or, "Need More Help" at the bottom of the article.

PC NA
Thank you for trying to help but I have tried everything and nothing helps. I sent tickets and I've done all of the recommended "update every driver and your bios", "uninstall and reinstall the game", "format your hard drive and reinstall doze and the game" only to end up in the same spot I am now. @ZOS_Kevin "WAS" keeping us up to date and passing on info to the devs. Now we have nothing but crickets.

Kevin is responding to other threads on the forum. Just not this thread anymore.

Thanks again for responding.

vsrs_au

Photosniper89 wrote: »

ZOS_CouchTato wrote: »

@Seravi I do not currently have any updates or known issues concerning this, but I can share the links for some troubleshooting:

For PC: https://help.elderscrollsonline.com/#en/answer/51890

For Xbox: https://help.elderscrollsonline.com/#en/home/category/8/platform/20/techissuetype/877

For PlayStation: https://help.elderscrollsonline.com/#en/home/category/8/platform/19/techissuetype/877

It would also help to send in a ticket to Support if you are still having the issue, you can do that by clicking "Continue to Submit Ticket" or, "Need More Help" at the bottom of the article.

I appreciate you trying to help - I truly do.

However, all of these troubleshooting threads are complete BS relating to the lag issue. IT IS NOT AN ISSUE ON OUR SIDE THIS IS A ZOS ISSUE! Can someone please send a pigeon with a letter to @ZOS_Kevin and ask him to unmute his notis for this thread and give us an update?

Agreed, all the replies on this thread have amply demonstrated that the problem is either with the ESO servers or the Akamai routers, and definitely not with any of the players' hardware or software.

Auberon1983

More "rubber banding" today.

Also, is anyone else having issues sprinting while not in combat? I've noticed multiple times now that I'll start sprinting, then my character starts doing the "running man", and then rapidly speeds back up. It'd be funny if it wasn't in conjunction with the rest of the lag issues.

Photosniper89

Auberon1983 wrote: »

More "rubber banding" today.

Also, is anyone else having issues sprinting while not in combat? I've noticed multiple times now that I'll start sprinting, then my character starts doing the "running man", and then rapidly speeds back up. It'd be funny if it wasn't in conjunction with the rest of the lag issues.

yup this is more of the input lag and overall server lag thats been going on.

edward_frigidhands

I have noticed that it gets worse on PC NA when they put out a new PTS.

ZOS_CouchTato wrote: »

@Seravi I do not currently have any updates or known issues concerning this, but I can share the links for some troubleshooting:

For PC: https://help.elderscrollsonline.com/#en/answer/51890

For Xbox: https://help.elderscrollsonline.com/#en/home/category/8/platform/20/techissuetype/877

For PlayStation: https://help.elderscrollsonline.com/#en/home/category/8/platform/19/techissuetype/877

It would also help to send in a ticket to Support if you are still having the issue, you can do that by clicking "Continue to Submit Ticket" or, "Need More Help" at the bottom of the article.

Hello @ZOS_CouchTato.

Are you able to provide details regarding how or if this matter is being investigated on your end?

What have you guys uncovered in your process of looking into it?

Has the staff at ZOS been able to replicate or observe any of the issues that many of these people are speaking of?

Has the information provided in this thread proven to be helpful to ZOS staff in the process of looking into the various similar issues described here?

Please let us know so that we can provide better information here.

edward_frigidhands

thinkaboutit wrote: »

Seravi wrote: »

Game response was MUCH better when Akami was out of the loop. Removing Akami is clearly something they "tried" as I had a few weeks without Akami and it was awesome. Now I have two Akami routes and the issues are back. Is Akami the only reason for the problems, of course not. It does help though when those two bottlenecks are removed.

I am really struggling with why I continue to keep playing hours every day. I am right by the data center and my routes are 2-3ms every where EXCEPT Akami. They are local routes but typical Akami bottlenecks. I should not have to keep hitting my key as fast as I can to get one or two skill fires off against the enemy.

I doubt they removed Akamai - its more likely we get load balanced on different in unpredictable ways, that's why the 'horribleness' is spread around and doesn't happen to everyone in a 12 man group at once. Just small clusters of them.

Funny how it works in their benefit though, because issues like this can easily be incorrectly blamed on the individual customer internet side.

Interesting. I have not been aware of this discussion at all.

Jaraal

edward_frigidhands wrote: »

thinkaboutit wrote: »

Seravi wrote: »

Game response was MUCH better when Akami was out of the loop. Removing Akami is clearly something they "tried" as I had a few weeks without Akami and it was awesome. Now I have two Akami routes and the issues are back. Is Akami the only reason for the problems, of course not. It does help though when those two bottlenecks are removed.

I am really struggling with why I continue to keep playing hours every day. I am right by the data center and my routes are 2-3ms every where EXCEPT Akami. They are local routes but typical Akami bottlenecks. I should not have to keep hitting my key as fast as I can to get one or two skill fires off against the enemy.

I doubt they removed Akamai - its more likely we get load balanced on different in unpredictable ways, that's why the 'horribleness' is spread around and doesn't happen to everyone in a 12 man group at once. Just small clusters of them.

Funny how it works in their benefit though, because issues like this can easily be incorrectly blamed on the individual customer internet side.

Interesting. I have not been aware of this discussion at all.

This has been a known issue for years. Ever since the lighting patch, when they moved certain calculations from the client to server side and started routing everything through an anti-DDoS service's (Akamai) servers.

Searching for "Akamai" on the forums will bring up a number of threads with tracert documentation.

https://forums.elderscrollsonline.com/en/discussion/436992/high-latency-at-akamai-com/p1

Photosniper89

Jaraal wrote: »

edward_frigidhands wrote: »

thinkaboutit wrote: »

Seravi wrote: »

Game response was MUCH better when Akami was out of the loop. Removing Akami is clearly something they "tried" as I had a few weeks without Akami and it was awesome. Now I have two Akami routes and the issues are back. Is Akami the only reason for the problems, of course not. It does help though when those two bottlenecks are removed.

I am really struggling with why I continue to keep playing hours every day. I am right by the data center and my routes are 2-3ms every where EXCEPT Akami. They are local routes but typical Akami bottlenecks. I should not have to keep hitting my key as fast as I can to get one or two skill fires off against the enemy.

I doubt they removed Akamai - its more likely we get load balanced on different in unpredictable ways, that's why the 'horribleness' is spread around and doesn't happen to everyone in a 12 man group at once. Just small clusters of them.

Funny how it works in their benefit though, because issues like this can easily be incorrectly blamed on the individual customer internet side.

Interesting. I have not been aware of this discussion at all.

This has been a known issue for years. Ever since the lighting patch, when they moved certain calculations from the client to server side and started routing everything through an anti-DDoS service's (Akamai) servers.

Searching for "Akamai" on the forums will bring up a number of threads with tracert documentation.

https://forums.elderscrollsonline.com/en/discussion/436992/high-latency-at-akamai-com/p1

Yup, now that stadia is gone I wish they would bring the calcs back to client side. Sure, there is an opportunity to cheat but you can't tell me a company of this size with this skill set can't implement a 3rd party cheat detection fairly easy to combat that.

The trade off on customer experience vs the as is system now would be night and day.

Photosniper89

Yup.... it's Akamai.



@ZOS_Kevin might wanna check this out.....

For those who are curious download ping plotter and then setup a run with ESO IPs (https://help.elderscrollsonline.com/#en/answer/30903). You can literally watch the hops and see it get to Akamai and then have a 100% packet loss every 5ish or so minutes.

Tandor

Is Akamai only used along certain routes, e.g. from the Southern Hemisphere countries? That would explain why most players don't have these problems (which in itself strongly supports the idea of it being down to Akamai rather than the servers, as if it was the servers everyone would have the same experience).

Photosniper89

Tandor wrote: »

Is Akamai only used along certain routes, e.g. from the Southern Hemisphere countries? That would explain why most players don't have these problems (which in itself strongly supports the idea of it being down to Akamai rather than the servers, as if it was the servers everyone would have the same experience).

I highly doubt ZoS would come out and say that - they are done with this thread.

Only way we can test is people start running their on route traces and posting the results here and then saying what general location they are in.

At the end of the day there is no way for us to get around it we are kind of SOL.

Photosniper89

My only packet loss and higher then 5ms travel time are on ips associated with Akamai. @ZOS_Kevin I wonder if there are some new ports that are being utilized that need to be opened (imo we should have to open ports for a game to work there are a lot of security concerns there but that is a convo for another day)

Tandor

Photosniper89 wrote: »

Tandor wrote: »

Is Akamai only used along certain routes, e.g. from the Southern Hemisphere countries? That would explain why most players don't have these problems (which in itself strongly supports the idea of it being down to Akamai rather than the servers, as if it was the servers everyone would have the same experience).

I highly doubt ZoS would come out and say that - they are done with this thread.

Only way we can test is people start running their on route traces and posting the results here and then saying what general location they are in.

At the end of the day there is no way for us to get around it we are kind of SOL.

That's what I was basing my question on.

M33tr3

Tandor wrote: »

Is Akamai only used along certain routes, e.g. from the Southern Hemisphere countries? That would explain why most players don't have these problems (which in itself strongly supports the idea of it being down to Akamai rather than the servers, as if it was the servers everyone would have the same experience).

No, I'm in the USA and go through Akamai.

malistorr

IA is a problem as well. If I play solo there is little to no performance issues. I'm in CA on half-gig fiber and am hardwired in on a Series X. If I play with another player (not sure where they're from or what kind of internet they have) performance in IA is hugely affected and goes to poop.

TaSheen

M33tr3 wrote: »

Tandor wrote: »

Is Akamai only used along certain routes, e.g. from the Southern Hemisphere countries? That would explain why most players don't have these problems (which in itself strongly supports the idea of it being down to Akamai rather than the servers, as if it was the servers everyone would have the same experience).

No, I'm in the USA and go through Akamai.

I'm in the US too, but don't go through Akamai (something to do with the way HughesNet is set up/routes traffic I guess).