ZOS, Massive Spike in Ping/LAG in Recent Days - What Gives?

LPapirius · August 23

FoJul wrote: »

The game for me and a lot of other's is going into unplayable territory.

It's difficult to play a game that keeps kicking you from the server so reliably and so frequently.

Photosniper89 · August 23

LPapirius wrote: »

FoJul wrote: »

The game for me and a lot of other's is going into unplayable territory.

It's difficult to play a game that keeps kicking you from the server so reliably and so frequently.

Can you try this and post the results here? Be sure to remove your personal IP and any info that could dox you

https://help.elderscrollsonline.com/#en/answer/30903

Mainly would like to get more conversation around the Akamia section on these logs.

M33tr3 · August 23

I've been running a tracert every half hour or so and have a "request timed out" between these two hops EVERY time.

edit: just noticed yours are dfw and mine are sjc

Ardriel · August 23

Is there any logical explanation why this only affects some players and not all? I often have problems with lags that show up in such a way that my skills or actually any inputs don't work the first time. I often have to press buttons several times before I see a reaction. E.g. pressing Ulti 3 times. I often hear the right sound effect the first time, e.g. with the potion or the typical Ulti sound. But the actual skill effect doesn't happen. In raids, I have to press almost every button twice for the skill to fire. I often see after bar swap that a skill didn't fire and then have to go back to the other bar. Of course, this doesn't have a particularly good effect on dps. The strange thing is that it's not always equally bad. Some days things work almost normally and my dps is 15-25% better. The other players often say that they know this too. But it seems to be far less extreme for them. My average ping is between 70 and 110 on the EU server. I play in Hungary. What could be the reason that it only affects some players and never all of them? Is it a random problem per account?

Photosniper89 · August 23

Ardriel wrote: »

Is there any logical explanation why this only affects some players and not all? I often have problems with lags that show up in such a way that my skills or actually any inputs don't work the first time. I often have to press buttons several times before I see a reaction. E.g. pressing Ulti 3 times. I often hear the right sound effect the first time, e.g. with the potion or the typical Ulti sound. But the actual skill effect doesn't happen. In raids, I have to press almost every button twice for the skill to fire. I often see after bar swap that a skill didn't fire and then have to go back to the other bar. Of course, this doesn't have a particularly good effect on dps. The strange thing is that it's not always equally bad. Some days things work almost normally and my dps is 15-25% better. The other players often say that they know this too. But it seems to be far less extreme for them. My average ping is between 70 and 110 on the EU server. I play in Hungary. What could be the reason that it only affects some players and never all of them? Is it a random problem per account?

It's hard to say. There are a lot of factors that can go into input lag. I'd say overall server performance has taken a big hit this year. People who aren't having the same rubber banding and dc's, I believe due to Akamai, are having overall lag.

"It's really laggy right now" is super common for everyone where as "I'm rubberbanding" or "Oh, I think I'm crashing" is more of an isolated issue (again I think due to Akamai).

Destai · August 23

ZOS_Kevin wrote: »

Sorry for the long absence, all. I was away for work travel over the last several weeks and catching up on things before launch.

If you're traveling and can't be on the forums, I feel like that should be communicated ahead of time. No one wants to post things and tag you in vain if you're focused elsewhere. Again, 50+ days is a long time to go without a response on a major issue. Threads shouldn't come to the point where people are begging or looking at LinkedIn for who to message.

And really, it's a bit confusing to see this as the explanation. The last thread update was back on June 28th, IIRC. In that time, there were still posts in other topics and links posted to official announcements. So I guess, why wasn't a major technical issue on that list of posts in 50+ days? For people here, it seems like a big deal, but do you guys have the same priority?

ZOS_Kevin wrote: »

After checking back in with the team, there isn't much progress to report right now. There were several attempts at pushing forth possible solutions, but those attempts have not resolved the issue. The engineering teams have poured over your reports, player logs, etc. to see if there is a commonality to tackle.

So people are coming here, providing technical details and doing a lot of testing. In my experience, when two parties are troubleshooting, it's best practice to provide details on both sides. Looks like you need to meet us in the middle there.

You mentioned there several attempts at resolving this. What were the solutions put forth that failed? What's the scope of people impacted here?

ZOS_Kevin wrote: »

We are still continuing to investigate. However, outside of reporting issues as you come across them, there isn't much else at the moment to do outside of our engineering teams continuing to test for possible solutions. We are still sharing reports as they come in with those teams for investigation.

From your perspective, how often do you think you should be updating us? And if you're overloaded, which I'm sure you are, is there anyone else you can delegate these things to? It's fine if engineering is still working on things, but I don't think it's fair to our time for us to only get updates if and when they've discovered a solution. Can we get updates every two weeks or aligned to your sprint cycle?

Photosniper89 · August 23

Destai wrote: »

ZOS_Kevin wrote: »

Sorry for the long absence, all. I was away for work travel over the last several weeks and catching up on things before launch.

If you're traveling and can't be on the forums, I feel like that should be communicated ahead of time. No one wants to post things and tag you in vain if you're focused elsewhere. Again, 50+ days is a long time to go without a response on a major issue. Threads shouldn't come to the point where people are begging or looking at LinkedIn for who to message.

And really, it's a bit confusing to see this as the explanation. The last thread update was back on June 28th, IIRC. In that time, there were still posts in other topics and links posted to official announcements. So I guess, why wasn't a major technical issue on that list of posts in 50+ days? For people here, it seems like a big deal, but do you guys have the same priority?

ZOS_Kevin wrote: »

After checking back in with the team, there isn't much progress to report right now. There were several attempts at pushing forth possible solutions, but those attempts have not resolved the issue. The engineering teams have poured over your reports, player logs, etc. to see if there is a commonality to tackle.

So people are coming here, providing technical details and doing a lot of testing. In my experience, when two parties are troubleshooting, it's best practice to provide details on both sides. Looks like you need to meet us in the middle there.

You mentioned there several attempts at resolving this. What were the solutions put forth that failed? What's the scope of people impacted here?

ZOS_Kevin wrote: »

We are still continuing to investigate. However, outside of reporting issues as you come across them, there isn't much else at the moment to do outside of our engineering teams continuing to test for possible solutions. We are still sharing reports as they come in with those teams for investigation.

From your perspective, how often do you think you should be updating us? And if you're overloaded, which I'm sure you are, is there anyone else you can delegate these things to? It's fine if engineering is still working on things, but I don't think it's fair to our time for us to only get updates if and when they've discovered a solution. Can we get updates every two weeks or aligned to your sprint cycle?

Great points. This is why I mentioned in a previous post I would absolutely love if someone at ZoS could provide some graphics or something to help educate us on the WHAT has been going on behind the scenes to help trouble shoot. ZoS could easily control the narrative "zos doesn't care about PvP (even though its not in PvE), etc if they would just be a bit more open about what they are doing behind the scenes to troubleshoot.

Like the code refactoring... that was hyped up and then just dropped off a cliff.
Now that stadia is dead has there been any conversations about bringing the calculations back client side?

These are the kind of things we would love to see/hear about.

Photosniper89 · August 23

Not to derail the above points.

For those who are suffering from Akamai issues or crashes/lag... do you have anyone else playing ESO at the same time within the same network?

I'm almost to the point where I want to break my network into different subnets and put all the people who play in my house on their own subnet with their own routing/port fwd just to see if that makes any change. Maybe ZoS network is having issues between the different clients in the same house? I've noticed if you have about 1-2 seconds of 100% packet loss ZoS is like "nah fam you done (booted)" where as other games can handle a much longer time frame of packet loss.

Just_Attivi · August 23

Photosniper89 wrote: »

Not to derail the above points.

For those who are suffering from Akamai issues or crashes/lag... do you have anyone else playing ESO at the same time within the same network?

I'm almost to the point where I want to break my network into different subnets and put all the people who play in my house on their own subnet with their own routing/port fwd just to see if that makes any change. Maybe ZoS network is having issues between the different clients in the same house? I've noticed if you have about 1-2 seconds of 100% packet loss ZoS is like "nah fam you done (booted)" where as other games can handle a much longer time frame of packet loss.

Im well past the giving-a-hoot about troubleshooting point in my ESO career (barely logged since the event ended, just dont care to, performance is terrible), but for what its worth, prior to moving to my current state at the end of june/beginning of july, I lived alone (no one else on my hardlined net) and experienced the significant lag/skill delays/rubberbanding/teleporting etc etc etc. so... while you may discover that shared net is causing some lag, it definitely isnt the only issue.

Tandor · August 23

Photosniper89 wrote: »

Not to derail the above points.

For those who are suffering from Akamai issues or crashes/lag... do you have anyone else playing ESO at the same time within the same network?

I'm almost to the point where I want to break my network into different subnets and put all the people who play in my house on their own subnet with their own routing/port fwd just to see if that makes any change. Maybe ZoS network is having issues between the different clients in the same house? I've noticed if you have about 1-2 seconds of 100% packet loss ZoS is like "nah fam you done (booted)" where as other games can handle a much longer time frame of packet loss.

Also, have people asked their ISP to re-route so as to avoid the Akamai hops? I know my own ISP co-operates in this way whenever they can, so it's always worth asking. Have those ISPs got technical forums on which the problem can be raised and supporting traceroutes provided?

merevie · August 23

No one else is playing in my household.

On top of that, in PvP, it is incredibly obvious when the entire faction is back at the start point after 2/3 of them crashing at once, and not coincidentally, PvE guild chat has several people howling about how their trial just got nuked.

In PvP on the weekends, it is not uncommon to watch an enemy group or faction actually disappear from the keep - did they get killed? No...

Hellzgaming -a well known Ad zerg leader, went out live on stream at least three times within an hour -and it's not like he has a poor setup.

PvE for me, as an active-avoider, seems to be mostly problematic around wayshrines, with 999+ persisting for awhile after using one.

Azzras · August 23

It's getting so bad it's almost not worth it anymore.

Photosniper89 · August 23

merevie wrote: »

No one else is playing in my household.

On top of that, in PvP, it is incredibly obvious when the entire faction is back at the start point after 2/3 of them crashing at once, and not coincidentally, PvE guild chat has several people howling about how their trial just got nuked.

In PvP on the weekends, it is not uncommon to watch an enemy group or faction actually disappear from the keep - did they get killed? No...

Hellzgaming -a well known Ad zerg leader, went out live on stream at least three times within an hour -and it's not like he has a poor setup.

PvE for me, as an active-avoider, seems to be mostly problematic around wayshrines, with 999+ persisting for awhile after using one.

Yeah, I saw that too. Also watch 3/4 of homicide crash.

It's funny to walk into a keep and just see 20-30 people just standing still because they crashed.

Seravi · August 23

The DFW Akami routes are mine as I live in Texas. This part of the route reflects the location.

Thanks @ZOS_Kevin for the replies. I do hope something can be done about Akami.

Photosniper89 · August 24

Tandor wrote: »

Photosniper89 wrote: »

Not to derail the above points.

For those who are suffering from Akamai issues or crashes/lag... do you have anyone else playing ESO at the same time within the same network?

I'm almost to the point where I want to break my network into different subnets and put all the people who play in my house on their own subnet with their own routing/port fwd just to see if that makes any change. Maybe ZoS network is having issues between the different clients in the same house? I've noticed if you have about 1-2 seconds of 100% packet loss ZoS is like "nah fam you done (booted)" where as other games can handle a much longer time frame of packet loss.

Also, have people asked their ISP to re-route so as to avoid the Akamai hops? I know my own ISP co-operates in this way whenever they can, so it's always worth asking. Have those ISPs got technical forums on which the problem can be raised and supporting traceroutes provided?

Thats a thing?

TaSheen · August 24

Photosniper89 wrote: »

Tandor wrote: »

Photosniper89 wrote: »

Not to derail the above points.

For those who are suffering from Akamai issues or crashes/lag... do you have anyone else playing ESO at the same time within the same network?

I'm almost to the point where I want to break my network into different subnets and put all the people who play in my house on their own subnet with their own routing/port fwd just to see if that makes any change. Maybe ZoS network is having issues between the different clients in the same house? I've noticed if you have about 1-2 seconds of 100% packet loss ZoS is like "nah fam you done (booted)" where as other games can handle a much longer time frame of packet loss.

Also, have people asked their ISP to re-route so as to avoid the Akamai hops? I know my own ISP co-operates in this way whenever they can, so it's always worth asking. Have those ISPs got technical forums on which the problem can be raised and supporting traceroutes provided?

Thats a thing?

Yes it is. Back when I was still on wildblue satellite, for some really weird reason I found myself being routed literally all around the world from here where the dish was to almost anywhere I needed to go (mostly at the time to my host provider - since at that time I was managing a bunch of client websites - most of them in the US, but several in other countries, Canada, Great Britain, France). My host provider was headquartered in Seattle (in other words, the servers were located there though they are now "elsewhere" - he migrated to AWS, and I'm not sure where the physical location is now, nor does it matter to me). That also included ESO (which I had just started playing) and RIFT (which I was still playing a bit back then).

It was.... impossibly laggy. Far beyond "simple satellite lag". So after a few months of almost total inability to do anything with client sites (yeah, the games too, but the clients were of course FAR more important), I called the provider himself and during the conversation, sent him tracerts etc.

He had his techs (all local at the time) get with wildblue and "sanitize" the routing. I don't know exactly what they did (or why wildblue was willing to mess with it) but things got materially better in a couple of weeks. That was 7+ years ago.

Since then, I've moved to the much more robust platform provided by HughesNet, and my provider has also moved to AWS, and now provides tech support through a large corporate entity in India. I've had to have them do "route sanitizing" a couple of times in the last year or so - and I don't have any of the lag issues y'all are dealing with.

Of course, I do still have the basic sat lag. That's workable for me though. I don't really understand any of it outside of a simple tracert, but I do know that I've had very good responses from both wildblue and hughesnet when I've had issues - and they've both been willing to work with my host provider to fix problems so my clients aren't jumping down my throat.

I just don't have the lag issues so many of you have. Yes, I have sat lag - but it's nothing like you're all dealing with. I haven't been dropped from either PC NA or EU in.... uh.... probably a year.... Not really sure, it's just been so long I haven't really a clue as to time frame.

[Ah, just to note.... IF I was having issues as so many of you are, I would be quite likely MORE "vocal" (eh, you can't call posting on a forum vocal maybe) about it. I'm not exactly "quiet" when my fun is interfered with, any more than I'm quiet when client issues need me to confront the provider....)

jonenuka · August 24

Not only the ping value is unstable, but also the line will drop all the time. I am about to crash. I don't know what causes it

Azzras · August 24

The disconnects during BGs are the worst. No other MMO I've played has this kind of connectivity issue. What gives?
I DC then get to wait for 5 mins when I log back in.
I'm so close to just quitting this game for good this time.
I've stayed so quiet for a decade, but this is just ridiculous now.

Tandor · August 24

Photosniper89 wrote: »

Tandor wrote: »

Photosniper89 wrote: »

Not to derail the above points.

For those who are suffering from Akamai issues or crashes/lag... do you have anyone else playing ESO at the same time within the same network?

I'm almost to the point where I want to break my network into different subnets and put all the people who play in my house on their own subnet with their own routing/port fwd just to see if that makes any change. Maybe ZoS network is having issues between the different clients in the same house? I've noticed if you have about 1-2 seconds of 100% packet loss ZoS is like "nah fam you done (booted)" where as other games can handle a much longer time frame of packet loss.

Also, have people asked their ISP to re-route so as to avoid the Akamai hops? I know my own ISP co-operates in this way whenever they can, so it's always worth asking. Have those ISPs got technical forums on which the problem can be raised and supporting traceroutes provided?

Thats a thing?

Very much so.

Here's a link to my ISP's Broadband forum as an example:-

https://community.virginmedia.com/t5/Broadband/ct-p/Broadband

You'll see a Gaming Support section and in it problems are replied to by both fellow customers and forum staff who can provide technical advice as well as arrange engineers' visits. I've seen cases where they were able to organise re-routing to avoid bottlenecks in the network.

Photosniper89 · August 24

TaSheen wrote: »

Photosniper89 wrote: »

Tandor wrote: »

Photosniper89 wrote: »

Not to derail the above points.

For those who are suffering from Akamai issues or crashes/lag... do you have anyone else playing ESO at the same time within the same network?

I'm almost to the point where I want to break my network into different subnets and put all the people who play in my house on their own subnet with their own routing/port fwd just to see if that makes any change. Maybe ZoS network is having issues between the different clients in the same house? I've noticed if you have about 1-2 seconds of 100% packet loss ZoS is like "nah fam you done (booted)" where as other games can handle a much longer time frame of packet loss.

Also, have people asked their ISP to re-route so as to avoid the Akamai hops? I know my own ISP co-operates in this way whenever they can, so it's always worth asking. Have those ISPs got technical forums on which the problem can be raised and supporting traceroutes provided?

Thats a thing?

Yes it is. Back when I was still on wildblue satellite, for some really weird reason I found myself being routed literally all around the world from here where the dish was to almost anywhere I needed to go (mostly at the time to my host provider - since at that time I was managing a bunch of client websites - most of them in the US, but several in other countries, Canada, Great Britain, France). My host provider was headquartered in Seattle (in other words, the servers were located there though they are now "elsewhere" - he migrated to AWS, and I'm not sure where the physical location is now, nor does it matter to me). That also included ESO (which I had just started playing) and RIFT (which I was still playing a bit back then).

It was.... impossibly laggy. Far beyond "simple satellite lag". So after a few months of almost total inability to do anything with client sites (yeah, the games too, but the clients were of course FAR more important), I called the provider himself and during the conversation, sent him tracerts etc.

He had his techs (all local at the time) get with wildblue and "sanitize" the routing. I don't know exactly what they did (or why wildblue was willing to mess with it) but things got materially better in a couple of weeks. That was 7+ years ago.

Since then, I've moved to the much more robust platform provided by HughesNet, and my provider has also moved to AWS, and now provides tech support through a large corporate entity in India. I've had to have them do "route sanitizing" a couple of times in the last year or so - and I don't have any of the lag issues y'all are dealing with.

Of course, I do still have the basic sat lag. That's workable for me though. I don't really understand any of it outside of a simple tracert, but I do know that I've had very good responses from both wildblue and hughesnet when I've had issues - and they've both been willing to work with my host provider to fix problems so my clients aren't jumping down my throat.

I just don't have the lag issues so many of you have. Yes, I have sat lag - but it's nothing like you're all dealing with. I haven't been dropped from either PC NA or EU in.... uh.... probably a year.... Not really sure, it's just been so long I haven't really a clue as to time frame.

[Ah, just to note.... IF I was having issues as so many of you are, I would be quite likely MORE "vocal" (eh, you can't call posting on a forum vocal maybe) about it. I'm not exactly "quiet" when my fun is interfered with, any more than I'm quiet when client issues need me to confront the provider....)

Good info there! I have a Starlink High Performance dish as my fallback WAN and fiber as my main. I'm to the point now where I'm wanting to just get rid of fiber to see if my routing table cleans up a bit. Back when I has Starlink basic dish I had a lot of crashing but ZOS was also having massive issues with server performance (worse then now) back then so it's really hard to tell what is sat vs ZoS.

Fiber clearly isn't working with the oruting as is so time to see if Starlink HP bypasses this routing. Or - I can call my ISP and see what they can do.

Photosniper89 · August 24

Tandor wrote: »

Photosniper89 wrote: »

Tandor wrote: »

Photosniper89 wrote: »

Not to derail the above points.

For those who are suffering from Akamai issues or crashes/lag... do you have anyone else playing ESO at the same time within the same network?

I'm almost to the point where I want to break my network into different subnets and put all the people who play in my house on their own subnet with their own routing/port fwd just to see if that makes any change. Maybe ZoS network is having issues between the different clients in the same house? I've noticed if you have about 1-2 seconds of 100% packet loss ZoS is like "nah fam you done (booted)" where as other games can handle a much longer time frame of packet loss.

Also, have people asked their ISP to re-route so as to avoid the Akamai hops? I know my own ISP co-operates in this way whenever they can, so it's always worth asking. Have those ISPs got technical forums on which the problem can be raised and supporting traceroutes provided?

Thats a thing?

Very much so.

Here's a link to my ISP's Broadband forum as an example:-

https://community.virginmedia.com/t5/Broadband/ct-p/Broadband

You'll see a Gaming Support section and in it problems are replied to by both fellow customers and forum staff who can provide technical advice as well as arrange engineers' visits. I've seen cases where they were able to organise re-routing to avoid bottlenecks in the network.

My ISP is kind of a rural company so I'm not 100% what they can do - doesn't hurt to give them a call though.

Photosniper89 · August 24

Another question for the group

Of those who are having massive issues - do you have the following Ethernet controller?

Any of you who have the rubber banding and stuff play on wifi? Just trying to isolate this issue and see if there if anything in common aside from Akamai.

M33tr3 · August 24

Photosniper89 wrote: »

Another question for the group

Of those who are having massive issues - do you have the following Ethernet controller?

Any of you who have the rubber banding and stuff play on wifi? Just trying to isolate this issue and see if there if anything in common aside from Akamai.

Nope, mine is

LaintalAy · August 24

Tandor wrote: »

Also, have people asked their ISP to re-route so as to avoid the Akamai hops? I know my own ISP co-operates in this way whenever they can, so it's always worth asking. Have those ISPs got technical forums on which the problem can be raised and supporting traceroutes provided?

TaSheen wrote: »

Yes it is. Back when I was still on wildblue satellite, for some really weird reason I found myself being routed literally all around the world from here where the dish was to almost anywhere I needed to go (mostly at the time to my host provider - since at that time I was managing a bunch of client websites - most of them in the US, but several in other countries, Canada, Great Britain, France). My host provider was headquartered in Seattle (in other words, the servers were located there though they are now "elsewhere" - he migrated to AWS, and I'm not sure where the physical location is now, nor does it matter to me). That also included ESO (which I had just started playing) and RIFT (which I was still playing a bit back then).

It was.... impossibly laggy. Far beyond "simple satellite lag". So after a few months of almost total inability to do anything with client sites (yeah, the games too, but the clients were of course FAR more important), I called the provider himself and during the conversation, sent him tracerts etc.

He had his techs (all local at the time) get with wildblue and "sanitize" the routing. I don't know exactly what they did (or why wildblue was willing to mess with it) but things got materially better in a couple of weeks. That was 7+ years ago.

Since then, I've moved to the much more robust platform provided by HughesNet, and my provider has also moved to AWS, and now provides tech support through a large corporate entity in India. I've had to have them do "route sanitizing" a couple of times in the last year or so - and I don't have any of the lag issues y'all are dealing with.

Of course, I do still have the basic sat lag. That's workable for me though. I don't really understand any of it outside of a simple tracert, but I do know that I've had very good responses from both wildblue and hughesnet when I've had issues - and they've both been willing to work with my host provider to fix problems so my clients aren't jumping down my throat.

I just don't have the lag issues so many of you have. Yes, I have sat lag - but it's nothing like you're all dealing with. I haven't been dropped from either PC NA or EU in.... uh.... probably a year.... Not really sure, it's just been so long I haven't really a clue as to time frame.

[Ah, just to note.... IF I was having issues as so many of you are, I would be quite likely MORE "vocal" (eh, you can't call posting on a forum vocal maybe) about it. I'm not exactly "quiet" when my fun is interfered with, any more than I'm quiet when client issues need me to confront the provider....)

How extraordinary!

You'd think that ZOS would have asked us to try this.

Thanks for the feedback.

Photosniper89 · August 24

I don't think ZOS would encourage us to try to bypass the akamai route. There is a reason (that I can't figure out) they put it in there.

LaintalAy · August 24

I don't think my ISP will listen.

Tandor · August 24

Photosniper89 wrote: »

I don't think ZOS would encourage us to try to bypass the akamai route. There is a reason (that I can't figure out) they put it in there.

Except that not all players go through Akamai in the first place, so it's not something that is specifically required by ZOS in order to play the game.

TaSheen · August 24

Photosniper89 wrote: »

I don't think ZOS would encourage us to try to bypass the akamai route. There is a reason (that I can't figure out) they put it in there.

It's in there to protect against DDOS attacks. That's also why my one financial institution uses it. That's probably the one thing Akamai is known for....

Photosniper89 · August 24

LaintalAy wrote: »

I don't think my ISP will listen.

Sameeeeeeeee lol

Photosniper89 · August 24

TaSheen wrote: »

Photosniper89 wrote: »

I don't think ZOS would encourage us to try to bypass the akamai route. There is a reason (that I can't figure out) they put it in there.

It's in there to protect against DDOS attacks. That's also why my one financial institution uses it. That's probably the one thing Akamai is known for....

Doesnt seem to be working well.....