ZOS, Massive Spike in Ping/LAG in Recent Days - What Gives?

Destai · June 24

ZOS_Kevin wrote: »

Kainarel wrote: »

Ping is becoming a more prominent problem, it gets higher and higher. 120s is already high for my norm 80-90, but now 130s is posing as new normal. It's frustrating.

Tried deleting ShaderCache file like of the players here suggested, but it didn't do anything about performance, only made my traveling loading screens significantly longer while before I was able to travel between wayshrines almost instantly. I wonder now whether I should put the old Cache file back in the folder to fix that new issue.

The performance and lag in this game is becoming worse every day and once again we get no maintenance tomorrow. It's sad.

@ZOS_Kevin please, please speak to us, we need any update and acknowledgement for our issues, they are breaking our game. Please, help us.

Just following up here. We have been reporting your ticket numbers, anecdotal accounts, and spikes in reports every time they come up in this thread and in other threads on the forum as well. We're providing our teams with all of the info you are providing here in the thread. So we just wanted to make it clear that we haven't forgotten about this. Currently, we do not have an update other than to continue posting your reports and issues in the thread. It's the best way for us to seem them and add everything to our ongoing report on this.

Where is this issue on their backlog though? Is it actively being worked on/diagnosed, or is the effort still being sized as information comes in? Again, from our perspective - performance is a regular issue - so it's a bit alarming that there's nothing to report after 50 days.

If you could provide weekly updates on this until it's resolved, that'd be much appreciated. Thanks for passing info along.

Wraithseer_ESO · June 24

I currently play from central Asia, here is a general traceroute

|

|

|
| gpon-infinitum.nokia.com - 0 | 31 | 31 | 1 | 5 | 80 | 1 |
| <my static IP> - 0 | 31 | 31 | 2 | 19 | 124 | 35 |
| 82.200.243.68 - 0 | 31 | 31 | 91 | 106 | 179 | 95 |
| mail-d.petrokazakhstan.com - 0 | 31 | 31 | 91 | 104 | 148 | 101 |
| hu0-0-0-16.br06.hkg12.as3491.net - 9 | 23 | 21 | 91 | 101 | 156 | 100 |
| 205-252-232-50.static.pccwglobal.net - 0 | 30 | 30 | 92 | 105 | 199 | 121 |
|a104-86-66-106.deploy.static.akamaitechnologies.com - 0 | 30 | 30 | 92 | 105 | 175 | 115 |
|a184-26-104-154.deploy.static.akamaitechnologies.com - 0 | 30 | 30 | 396 | 405 | 453 | 401 |
| 159.100.230.1 - 0 | 30 | 30 | 396 | 407 | 453 | 404 |
|________________________________________________|______|______|______|______|______|______|
WinMTR v1.00 GPLv2 (original by Appnor MSP - Fully Managed Hosting & Cloud Provider)

As you cna see it goes all the way to ZoS server and adds a 300ms spike between the nodes a104-86-66-106.deploy.static.akamaitechnologies.com and a184-26-104-154.deploy.static.akamaitechnologies.com

When I use NL VPN

|

|

|
| <my static VPN IP> - 0 | 38 | 38 | 85 | 91 | 108 | 108 |
| . - 0 | 38 | 38 | 87 | 92 | 109 | 87 |
| GW-PQHOST.rtr2.nl1.mirhosting.net - 0 | 38 | 38 | 86 | 93 | 134 | 87 |
| et315-464.rt.srv.dro.nl.retn.net - 0 | 38 | 38 | 86 | 92 | 104 | 93 |
| ae18-2.rt.tc2.ams.nl.retn.net - 0 | 38 | 38 | 89 | 95 | 129 | 96 |
| ams-ix.prolexic.com - 3 | 34 | 33 | 88 | 96 | 134 | 92 |
| po110.bs-a.sech-ams.netarch.akamai.com - 0 | 38 | 38 | 89 | 96 | 125 | 94 |
| Request timed out. - 100 | 8 | 0 | 0 | 0 | 0 | 0 |
|ae120.access-a.sech-ams.netarch.akamai.com - 0 | 38 | 38 | 89 | 94 | 115 | 93 |
| 93.191.172.246 - 0 | 38 | 38 | 88 | 94 | 111 | 89 |
|a184-26-104-154.deploy.static.akamaitechnologies.com - 0 | 38 | 38 | 95 | 101 | 118 | 95 |
| 159.100.230.1 - 0 | 38 | 38 | 96 | 102 | 127 | 99 |
|________________________________________________|______|______|______|______|______|______|
WinMTR v1.00 GPLv2 (original by Appnor MSP - Fully Managed Hosting & Cloud Provider)

Connects to the same node a184-26-104-154.deploy.static.akamaitechnologies.com but there is no spike. Is it a case of regional descrimination?

My ticket Reference:
240621-001004

Jaraal · June 25

Hit a stack of enemies on the bridge tonight with a cold stone treb. Everything froze for a second, then my screen was full of 20-28k oneshots.... and then I got kicked. "An unexpected error occurred, contact customer support." And when I finally got back in, I was where my siege had been but was now enemy territory and got zerged down in 3 seconds. Then I got kicked again from lag, logged back in, and was still stuck in combat even though I got booted!

So tired of being punished by a combination of server lag and over the top boot triggers. Last night I was trying to pick a pocket for a possible event box and get it done before the NPC was in range of the guards, but the [E] key was grayed out due to zonewide (lots of people complaining) lag. So I kept pressing it to get it to register before I got caught, and got booted from the server for "Sending too many messages."

Pretty disappointing when you get temporarily sanctioned and lose your valuable siege and risk indefensible death from players and guards through no fault of your own. It gets harder and harder to get motivated to log in, and I know I'm not the only one.

Kainarel · June 25

I'm seeing more and more red ping numbers during the gameplay now than I've ever seen before, and it's no joke.
Lag spikes are becoming new normal, and they hit any time, any place overland as well as group activity locations. Even when I travel into my home, I get red ping on arrival.
Every travel through wayshrine results in red ping upon arrival as well.

During combat some powers can just up and fail to fire even though you're pressing the button. Like, more and more it happens to me that I'm pressing to heavy attack and the attack doesn't fire, so I'm standing there like a fool being pounded by enemy mobs. That happens so frequently that it's also new norm.

It's just all getting more frustrating, along with my new normal ping being in 130s instead of 80-90.

TechMaybeHic · June 25

Interesting enough, I was in Cyrodiil for PCNA prime time and didn't have any issues out of the ordinary. Sitting at stormhave skill respec, and I crashed.

cptscotty · June 25

Why is it we are being told the best way for ZOS to know of the huge glaring problem is if we, the paying customer, take time out to write reports for ZOS?

Pretty sure the best way for you guys to know the problem is by having your own paid testers play the game.

I mean...does anyone else realize that it wasn't even mentioned whether or not if the ZOS dev team is even looking into this problem? Just that the forum moderators are handing off the threads to them?

Pelanora · June 25

cptscotty wrote: »

Why is it we are being told the best way for ZOS to know of the huge glaring problem is if we, the paying customer, take time out to write reports for ZOS?

Pretty sure the best way for you guys to know the problem is by having your own paid testers play the game.

I mean...does anyone else realize that it wasn't even mentioned whether or not if the ZOS dev team is even looking into this problem? Just that the forum moderators are handing off the threads to them?

The data should tell them. What analytics do they actually run? How can a Microsoft company be bad at servers?

fizzylu · June 25

cptscotty wrote: »

Why is it we are being told the best way for ZOS to know of the huge glaring problem is if we, the paying customer, take time out to write reports for ZOS?

Pretty sure the best way for you guys to know the problem is by having your own paid testers play the game.

I mean...does anyone else realize that it wasn't even mentioned whether or not if the ZOS dev team is even looking into this problem? Just that the forum moderators are handing off the threads to them?

I actually think they did claim that they were looking into it.... but honestly, I think that just makes things more questionable if that truly is the case because I have never seen any other so called triple A MMO have a known problem like this persist for so long while seeming to only get worse.
What extra gets me though is the comment that was made about the server issues being related to a DDoS attack and last I've checked, there's been no follow-up comment on that supposed attack at all and definitely no improvement in performance.... so uh, yeah.... many questions should probably be getting asked, but I think most people that still bother to play this game have accepted that Zenimax will not give proper answers anyway. I know I'm finding it hard to play knowing how so much doesn't seem to add up here and have once again canceled my sub, just hoping this time I manage to not make the mistake of renewing it again.

Photosniper89 · June 26

Back after a trip to time out (frustration with this game got the best of me).

This issue is getting worse and worse.

I'm now experiencing a 2-3 second 100% freeze when hitting areas where I assume there is part of the map that loads in. Not loading screens but just going across overland areas every time part of the map starts to fill in everything comes to a complete stop.

Everything in this game performance wise is getting worse and worse. It used to be isolated to PvP but now it's just as bad in PvE.

React · June 26

PC/NA. Last night between 12AM and 1AM, GH was a complete slideshow with only 1 bar, 2 bar, 3 bar population.

Every time I got close to any signifcant fight on the map, my ping would go from it's average 80-110 up to 500-700. Every cast, barswap, and core combat function was delayed by a solid 2 seconds. FPS completely tanking. Players rubberbanding.

Completely unplayable for all intents and purposes.

Jaraal · June 26

React wrote: »

PC/NA. Last night between 12AM and 1AM, GH was a complete slideshow with only 1 bar, 2 bar, 3 bar population.

Every time I got close to any signifcant fight on the map, my ping would go from it's average 80-110 up to 500-700. Every cast, barswap, and core combat function was delayed by a solid 2 seconds. FPS completely tanking. Players rubberbanding.

Completely unplayable for all intents and purposes.

I think it may have something do do with so many PvPers running the new repositioning (knockback and pull) scribes. A large number of players are now constantly being pushed and pulled around some pretty significant distances, and the server has to calculate and modify effects and processes based on the sudden shifting of positions. This ubiquitous new ping-pong effect has got to be taking it's toll on performance.

React · June 26

Jaraal wrote: »

React wrote: »

PC/NA. Last night between 12AM and 1AM, GH was a complete slideshow with only 1 bar, 2 bar, 3 bar population.

Every time I got close to any signifcant fight on the map, my ping would go from it's average 80-110 up to 500-700. Every cast, barswap, and core combat function was delayed by a solid 2 seconds. FPS completely tanking. Players rubberbanding.

Completely unplayable for all intents and purposes.

I think it may have something do do with so many PvPers running the new repositioning (knockback and pull) scribes. A large number of players are now constantly being pushed and pulled around some pretty significant distances, and the server has to calculate and modify effects and processes based on the sudden shifting of positions. This ubiquitous new ping-pong effect has got to be taking it's toll on performance.

Possibly that. But I think it's likely got more to do with the use of menders bond (constantly checking for people moving in and out of it/refreshing affects on everyone that does) and ulfsilds (constantly proccing and looking for appropriate targets to distribute buffs and shields/heals to) by coordinated 12 man groups that is really tanking it.

But who knows. All I know is that this is quite a significant step down in any case from the performance of last patch, and is getting very close to "pre server upgrade" levels of poor performance.

Photosniper89 · June 26

Jaraal wrote: »

React wrote: »

PC/NA. Last night between 12AM and 1AM, GH was a complete slideshow with only 1 bar, 2 bar, 3 bar population.

Every time I got close to any signifcant fight on the map, my ping would go from it's average 80-110 up to 500-700. Every cast, barswap, and core combat function was delayed by a solid 2 seconds. FPS completely tanking. Players rubberbanding.

Completely unplayable for all intents and purposes.

I think it may have something do do with so many PvPers running the new repositioning (knockback and pull) scribes. A large number of players are now constantly being pushed and pulled around some pretty significant distances, and the server has to calculate and modify effects and processes based on the sudden shifting of positions. This ubiquitous new ping-pong effect has got to be taking it's toll on performance.

Nope, was happening before scribing entered the chat.

BUT I will agree with you that items like that, or the the set that gives random buffs/debuffs, make it worse. Everyone has been screaming it for years but the way the game does calculations is what is causing this and they won't refactor the code to fix it.

Every single time a new item or skill or something enters the game that has to scan every player in the area things just go to hell. @ZOS_Kevin

Did the team scrap the code refactoring effort? There was a thread a while back (year+ now) that talked about it and I thought there was going to be some updates about that periodically but that thread kind of fell off the cliff (or maybe something was decided on it and I'm just unaware).

But.... the lag now is so bad in PvE as well which is odd to me. Sure, we are used to this level of performance on the PvP side.. it's been like this for so many years but now it's all over PvE and making it really difficult to even do writs.

SeaUnicorn · June 27

Pelanora wrote: »

cptscotty wrote: »

Why is it we are being told the best way for ZOS to know of the huge glaring problem is if we, the paying customer, take time out to write reports for ZOS?

Pretty sure the best way for you guys to know the problem is by having your own paid testers play the game.

I mean...does anyone else realize that it wasn't even mentioned whether or not if the ZOS dev team is even looking into this problem? Just that the forum moderators are handing off the threads to them?

The data should tell them. What analytics do they actually run? How can a Microsoft company be bad at servers?

Server upgrade took place before acquisition by MSFT. So data center config has nothing to do with Microsoft.

Kainarel · June 28

Still getting laggier overland.
Impossible to travel anywhere without a red spike of ping numbers upon arrival that hold for a while.
FPS are lowered, ping is hightened. My norm fps is 60-80s, now it's mostly 40+. My normal ping is 80-90, but now it's mostly 130+.

This is frustrating. It was all working so much better during the Festival when so many people were around hunting motifs and zerging geisers. I had zero issues then amidst those crowds. Now there are issues when you're even alone someplace secluded.

I wonder if @ZOS_Kevin and others still monitor this topic where people post less and less, seemingly giving up.
Now that launcher issue... [sigh]

Photosniper89 · June 28

999+ all night long and high 600+ just being at my house trying to swap out gear.

These servers are cooked.

What would it take to have the servers restarted multiple times a week... like... just a quick my bad I leaned on the power switch kinda blip?

When the servers reboot the lag goes away for a bit (I'm assuming there are memory leaks or just the DB is filling up and getting booged down.

React · June 28

6/7, PC NA GH, 4PM EST. Constant ping spikes as high as 750. Multiple second delay on skills. Players Rubberbanding everywhere.

Stridig · June 28

React wrote: »

6/7, PC NA GH, 4PM EST. Constant ping spikes as high as 750. Multiple second delay on skills. Players Rubberbanding everywhere.

I had to just log out for the night. It's getting old not being able to play the game and just getting the same old "we're looking into it" story.

thinkaboutit · June 28

How about an update on why we are disconnecting from the game when switching zones. @ZOS_Kevin

Photosniper89 · June 28

Stridig wrote: »

React wrote: »

6/7, PC NA GH, 4PM EST. Constant ping spikes as high as 750. Multiple second delay on skills. Players Rubberbanding everywhere.

I had to just log out for the night. It's getting old not being able to play the game and just getting the same old "we're looking into it" story.

I did the same as well. Had to log off it was that bad.

VouxeTheMinotaur · June 28

PS/NA

disconnected after maybe 30 minutes of logging in with an Error 103 popping up on my screen. first time i saw that one.

was logged into a dps running veteran bal sunnar. got to the second boss, the nix-ox, game did the whole - everyone's frozen around me, music stopped playing, enemies stopped moving and stood in their idle animation while my group mates were looping their attacks nonstop, then freeze, then Error 103.

VouxeTheMinotaur · June 28

in lightheartedness though, i'd like to imagine this is how it's going right now

(we appreciate the help kevin, we're just frustrated)

ZOS_Kevin · June 28

Hi All, just wanted to provide an update here before the weekend. Chatted with a few folks about this issue. The team is still looking into how to improve this situation and trying to line up your feedback/tickets with our graphs to see if we can spot consistent correlation in data to address this.

Additionally, we are asking the team if there is anything else other than the videos and tickets placed so far that could assist here. For now, videos and tickets seem to be the best course of action for documentation. However, we will continue to update you if that changes.

Lastly, we want to emphasize that we are frustrated with this, just as you are. We want your experience with ESO to always be a smooth one, so we are continuing to address. We are still reading this thread (along with others), updating our reports with new videos, comments and tickets to give our teams more data to work with. Thanks for your continued patience as we work through this.

ZOS_Kevin · June 28

VouxeTheMinotaur wrote: »

in lightheartedness though, i'd like to imagine this is how it's going right now

(we appreciate the help kevin, we're just frustrated)

Okay, but this picture made me laugh/cry. lol.

kyle.wilson · June 28

ZOS_Kevin wrote: »

Hi All, just wanted to provide an update here before the weekend. Chatted with a few folks about this issue. The team is still looking into how to improve this situation and trying to line up your feedback/tickets with our graphs to see if we can spot consistent correlation in data to address this.

Additionally, we are asking the team if there is anything else other than the videos and tickets placed so far that could assist here. For now, videos and tickets seem to be the best course of action for documentation. However, we will continue to update you if that changes.

Lastly, we want to emphasize that we are frustrated with this, just as you are. We want your experience with ESO to always be a smooth one, so we are continuing to address. We are still reading this thread (along with others), updating our reports with new videos, comments and tickets to give our teams more data to work with. Thanks for your continued patience as we work through this.

If the ZOS team continues to prove incapable of dealing with it, why does ZOS President Matt Firor not demand escalation and assistance from Microsoft's server team.
The pvp server performance has been a problem for almost the entire 10 years, with periods of extreme problems.

TaSheen · June 28

kyle.wilson wrote: »

ZOS_Kevin wrote: »

Hi All, just wanted to provide an update here before the weekend. Chatted with a few folks about this issue. The team is still looking into how to improve this situation and trying to line up your feedback/tickets with our graphs to see if we can spot consistent correlation in data to address this.

Additionally, we are asking the team if there is anything else other than the videos and tickets placed so far that could assist here. For now, videos and tickets seem to be the best course of action for documentation. However, we will continue to update you if that changes.

Lastly, we want to emphasize that we are frustrated with this, just as you are. We want your experience with ESO to always be a smooth one, so we are continuing to address. We are still reading this thread (along with others), updating our reports with new videos, comments and tickets to give our teams more data to work with. Thanks for your continued patience as we work through this.

If the ZOS team continues to prove incapable of dealing with it, why does ZOS President Matt Firor not demand escalation and assistance from Microsoft's server team.
The pvp server performance has been a problem for almost the entire 10 years, with periods of extreme problems.

Well.... I think it's fairly doubtful Microsoft's server team would step in to solve this situation for free.... And in fact, it might cost ZOS so much money they'd have problems putting out content normally for a while.

kyle.wilson · June 28

TaSheen wrote: »

kyle.wilson wrote: »

ZOS_Kevin wrote: »

Hi All, just wanted to provide an update here before the weekend. Chatted with a few folks about this issue. The team is still looking into how to improve this situation and trying to line up your feedback/tickets with our graphs to see if we can spot consistent correlation in data to address this.

Additionally, we are asking the team if there is anything else other than the videos and tickets placed so far that could assist here. For now, videos and tickets seem to be the best course of action for documentation. However, we will continue to update you if that changes.

Lastly, we want to emphasize that we are frustrated with this, just as you are. We want your experience with ESO to always be a smooth one, so we are continuing to address. We are still reading this thread (along with others), updating our reports with new videos, comments and tickets to give our teams more data to work with. Thanks for your continued patience as we work through this.

If the ZOS team continues to prove incapable of dealing with it, why does ZOS President Matt Firor not demand escalation and assistance from Microsoft's server team.
The pvp server performance has been a problem for almost the entire 10 years, with periods of extreme problems.

Well.... I think it's fairly doubtful Microsoft's server team would step in to solve this situation for free.... And in fact, it might cost ZOS so much money they'd have problems putting out content normally for a while.

Microsoft owns the entirety of Zenimax Media.

TaSheen · June 28

kyle.wilson wrote: »

TaSheen wrote: »

kyle.wilson wrote: »

ZOS_Kevin wrote: »

Hi All, just wanted to provide an update here before the weekend. Chatted with a few folks about this issue. The team is still looking into how to improve this situation and trying to line up your feedback/tickets with our graphs to see if we can spot consistent correlation in data to address this.

Additionally, we are asking the team if there is anything else other than the videos and tickets placed so far that could assist here. For now, videos and tickets seem to be the best course of action for documentation. However, we will continue to update you if that changes.

Lastly, we want to emphasize that we are frustrated with this, just as you are. We want your experience with ESO to always be a smooth one, so we are continuing to address. We are still reading this thread (along with others), updating our reports with new videos, comments and tickets to give our teams more data to work with. Thanks for your continued patience as we work through this.

If the ZOS team continues to prove incapable of dealing with it, why does ZOS President Matt Firor not demand escalation and assistance from Microsoft's server team.
The pvp server performance has been a problem for almost the entire 10 years, with periods of extreme problems.

Well.... I think it's fairly doubtful Microsoft's server team would step in to solve this situation for free.... And in fact, it might cost ZOS so much money they'd have problems putting out content normally for a while.

Microsoft owns the entirety of Zenimax Media.

I am fully aware of that. I still doubt they're going to do something like that for free. There would be if not actual charges, at least tradeoffs - and that might not be a good thing either.

kyle.wilson · June 28

TaSheen wrote: »

kyle.wilson wrote: »

TaSheen wrote: »

kyle.wilson wrote: »

ZOS_Kevin wrote: »

Hi All, just wanted to provide an update here before the weekend. Chatted with a few folks about this issue. The team is still looking into how to improve this situation and trying to line up your feedback/tickets with our graphs to see if we can spot consistent correlation in data to address this.

Additionally, we are asking the team if there is anything else other than the videos and tickets placed so far that could assist here. For now, videos and tickets seem to be the best course of action for documentation. However, we will continue to update you if that changes.

Lastly, we want to emphasize that we are frustrated with this, just as you are. We want your experience with ESO to always be a smooth one, so we are continuing to address. We are still reading this thread (along with others), updating our reports with new videos, comments and tickets to give our teams more data to work with. Thanks for your continued patience as we work through this.

If the ZOS team continues to prove incapable of dealing with it, why does ZOS President Matt Firor not demand escalation and assistance from Microsoft's server team.
The pvp server performance has been a problem for almost the entire 10 years, with periods of extreme problems.

Well.... I think it's fairly doubtful Microsoft's server team would step in to solve this situation for free.... And in fact, it might cost ZOS so much money they'd have problems putting out content normally for a while.

Microsoft owns the entirety of Zenimax Media.

I am fully aware of that. I still doubt they're going to do something like that for free. There would be if not actual charges, at least tradeoffs - and that might not be a good thing either.

Nobody does anything for free. We've been standing by watching this company make promises about the server performance for a decade and not able to deliver. They have the ability to have a third party with a financial interest in the results and the technical capability to conduct an audit and give their recommendations on a path forward.

I've played since beta and only created my account here when pvp server performance became a slideshow in 2016.

MrGarlic · June 28

PC/EU

Interestingly.

I had 2 accounts logged on at the same time yesterday while I was crafting items for research.

My crafting account had terrible ping, over 999+ while the ALT account had normal ping.

I've crafted this way plenty of times previously, I have more than one other alt account I'm doing research on, but never have I had such an odd disparity of pings/latency like that.

I assume if it was my internet connection causing problems, then both accts would have had similar ping. The fact that one was normal and the other was terrible might help ZOS techs.

(My wife was playing ESO on her PC, and had no issues at all at the same time.)