PC/NA just went down

AngryChelse · May 11

I just completed the Trial. The network situation is much better than it was a few days ago!

What is puzzling is that I can't figure out why this problem was resolved. I did not change any network settings, nor did I reinstall.

It has been a strange few days

Desiato · May 11

AngryChelse wrote: »

I just completed the Trial. The network situation is much better than it was a few days ago!

What is puzzling is that I can't figure out why this problem was resolved. I did not change any network settings, nor did I reinstall.

It has been a strange few days

Well, players have reported a variety of different issues. Some players say they have had issues since Tuesday. Some say since the last update. Some have experienced issues between 2AM and aprox 9AM EDT.

For me, I experienced the same issues everyone seemed to on Tuesday the 7th when this thread was created.

I had no issues on Wednesday, but shortly after I stopped playing others said issues began at around 2AM on Thursday.

On Thursday, I experienced 1-2% packet loss while pinging the IP address ZOS provides for troubleshooting NA gameplay connection issues. I had minor issues when I would experienced dropped packets: stutter, bars not swapping, etc. It was annoying but playable. Yet for others, it was unplayable.

Today has been perfect for me. No issues whatsoever. I've done a couple of dungeons and a couple of trials and everything seemed fine for my groups as well.

So it might be issues vary based on when people play and their routing to the NA server.

Wildberryjack · May 11

I was on and it did ok for a few hours then it went right back to high latency spikes and freezing. I swapped to the EU server and it was fine. Back to NA and it was so bad I just logged for the night.

davidtk · May 11

Jaraal wrote: »

StihlReign wrote: »

Jaraal wrote: »

Packet loss and high latency can be traced to the anti-DDoS servers all data is routed to before it gets to the ESO servers. This has been well documented with tracerts for years.

https://forums.elderscrollsonline.com/en/discussion/436992/high-latency-at-akamai-com/p1

Good ole Akamai anti-ddos service at it again?

Easy to test.

Windows key + R

Type cmd or cmd.exe in the box.

Type: tracert 198.20.200.1 in the command window and press Enter. Akamai filtering confirmed.

You can also run ping 198.20.200.1

May I ask what I should see? I got a timeout after Akamai.
Ping 198.20.200.1 without packet loss
My tracert:

Tracing route to 198.20.200.1 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  xxx.xxx.xxx.xxx
  2     2 ms     2 ms     2 ms  xxx.xxx.xxx.xxx
  3    12 ms    16 ms     2 ms  xxx.xxx.xxx.xxx
  4     3 ms     2 ms     3 ms  82.113.32.226
  5     2 ms     2 ms     2 ms  xe-4-0-0.prg11.ip4.tinet.net [213.200.74.93]
  6     2 ms     2 ms     2 ms  ip4.gtt.net [77.67.120.54]
  7     7 ms     7 ms     7 ms  ae23-xcr1.vie.cw.net [195.2.25.254]
  8     8 ms     8 ms     8 ms  62.208.246.59
  9     7 ms     8 ms     8 ms  a2-21-119-97.deploy.static.akamaitechnologies.com [2.21.119.97]
 10     *        *        *     Request timed out.
 11   147 ms   147 ms   143 ms  198.20.200.1

For comparison EU server 159.100.232.124

Tracing route to 159.100.232.124 over a maximum of 30 hops

  1    <1 ms    <1 ms    <1 ms  xxx.xxx.xxx.xxx
  2     3 ms     2 ms     3 ms  xxx.xxx.xxx.xxx
  3     3 ms     2 ms     2 ms  xxx.xxx.xxx.xxx
  4     3 ms     3 ms     3 ms  82.113.32.226
  5     5 ms     2 ms     4 ms  xe-4-0-0.prg11.ip4.tinet.net [213.200.74.93]
  6     2 ms     2 ms     2 ms  ip4.gtt.net [77.67.120.54]
  7     7 ms     7 ms     8 ms  ae23-xcr1.vie.cw.net [195.2.25.254]
  8     8 ms     8 ms     8 ms  62.208.246.59
  9     8 ms     8 ms     8 ms  a2-21-119-147.deploy.static.akamaitechnologies.com [2.21.119.147]
 10    17 ms    20 ms    19 ms  a184-26-104-158.deploy.static.akamaitechnologies.com [184.26.104.158]
 11    20 ms    19 ms    20 ms  195.122.154.2
 12    28 ms    26 ms    26 ms  159.100.224.33
 13    15 ms    19 ms    19 ms  159.100.232.124

Desiato · May 11

Jaraal wrote: »

Windows key + R

Type cmd or cmd.exe in the box.

Type: tracert 198.20.200.1 in the command window and press Enter. Akamai filtering confirmed.

You can also run ping 198.20.200.1

That's not what it means. Many routers are configured not to respond to pings and traceroutes. Try googling "traceroute doesn't reach destination" or "no response to traceroute". There are many other explanations and it's not a sign of a problem. Even if a traceroute fails to reach its destination, it doesn't mean that an RTBH is in use and it's not an indication of a problem.

It's a good thing that ZOS is a customer of akamai because they provide top notch ddos mitigation solutions but it doesn't mean there's any ddos mitigations currently active. It would be a bad thing if ZOS wasn't ready with ddos mitigations.

This is how myths get spread. Like back in 2014 when half the population of Cyrodiil was convinced that lighting was serverside and that's why cyrodiil performance was bad -- because someone said so in the forums. In this case, people see akamai in a traceroute and they think it's a bad sign when it's not.

Jaraal · May 11

davidtk wrote: »

May I ask what I should see? I got a timeout after Akamai.

Yes, it looks like your signal got hung up at Akami before it was redirected to ZOS. 143ms is significant lag.

Desiato wrote: »

It's a good thing that ZOS is a customer of akamai because they provide top notch ddos mitigation solutions but it doesn't mean there's any ddos mitigations currently active.

I guarantee you that every bit of data that gets to the ZOS servers is sent to the scrubber service first.

[snip]
[edited for baiting]

Desiato · May 11

Jaraal wrote: »

[snip]

I'm not whiteknighting for ZOS at all. I work with facts. When I had packet loss all day Thursday, I reported it here.

If Akamai ddos mitigations are active and that affects performance, that's a *good* thing because it would be much worse without ddos mitigations during a ddos. ddos mitigations are not active when there is no ddos and there is no performance impact.

Incidentally, here is what the hop between akamai and eso looks like for me:

8 58 ms 60 ms 64 ms 184.26.104.144
9 58 ms 58 ms 60 ms 198.20.200.1

I edited my original response to verify my assumption, but I've basically confirmed davidtk appears to be in Europe which is why his ping to NA ESO is high compared to EU. He is entering the Akamai network in Europe and exiting it when it connects to the ESO NA network. That is why there is a large latency delta between akamai and the eso NA server. Akamai has a world-wide network and he appears to be crossing the Atlantic through it.

[edited to remove quote]

Seravi · May 11

Here is my tracert:

Tracing route to 198.20.200.1 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms xxx.xxx.xxx.xxx
2 <1 ms <1 ms <1 ms xxx.xxx.xxx.xxx
3 1 ms 1 ms 1 ms xxx.xxx.xxx.xxx
4 2 ms 2 ms 1 ms xxx.xxx.xxx.xxx
5 11 ms 7 ms 7 ms xxx.xxx.xxx.xxx
6 * * * Request timed out.
7 3 ms 3 ms 3 ms 32.130.25.5
8 6 ms 6 ms 6 ms 12.89.241.34
9 5 ms 5 ms 5 ms po110.bs-a.sech-dfw.netarch.akamai.com [23.57.102.243]
10 * * * Request timed out.
11 6 ms 6 ms 6 ms ae120.access-a.sech-dfw.netarch.akamai.com [23.57.102.249]
12 6 ms 5 ms 5 ms 93.191.172.167
13 * * * Request timed out.
14 7 ms 6 ms 6 ms 198.20.200.1

So yeah the game delay..ugh.. I don't use addons either. Not worried about the time outs, it happens and some hops don't reply.

Desiato · May 11

Seravi wrote: »

Here is my tracert:
<snip>
14 7 ms 6 ms 6 ms 198.20.200.1

So yeah the game delay..ugh.. I don't use addons either. Not worried about the time outs, it happens and some hops don't reply.

This is excellent information that could help ZOS narrow down the scope of the issue by seeing the route you take to the NA server.

Your latency to the NA server is so low! I am envious. However, traceroute isn't great at showing intermittent packet loss. I suggest pinging the server continuously with: ping -t 198.20.200.1

The benefit of pinging continuously is one can switch to the shell window to see if their in-game issues correlate with packet loss. That was usually the case for me on Thursday. On Friday I had almost no packet loss and absolutely no issues while grinding a trial and doing misc overland/trading/banking tasks in between from 7 PM to 1 AM EDT.

A continuous ping can be interrupted with a ctrl-c, which will display stats.

Seravi · May 11

Desiato wrote: »

Seravi wrote: »

Here is my tracert:
<snip>
14 7 ms 6 ms 6 ms 198.20.200.1

So yeah the game delay..ugh.. I don't use addons either. Not worried about the time outs, it happens and some hops don't reply.

This is excellent information that could help ZOS narrow down the scope of the issue by seeing the route you take to the NA server.

Your latency to the NA server is so low! I am envious. However, traceroute isn't great at showing intermittent packet loss. I suggest pinging the server continuously with: ping -t 198.20.200.1

The benefit of pinging continuously is one can switch to the shell window to see if their in-game issues correlate with packet loss. That was usually the case for me on Thursday. On Friday I had almost no packet loss and absolutely no issues while grinding a trial and doing misc overland/trading/banking tasks in between from 7 PM to 1 AM EDT.

A continuous ping can be interrupted with a ctrl-c, which will display stats.

Yes I had planned on a continuous ping trace today. I hope I don't see any issues but if I do maybe some packet loss will show up.

davidtk · May 11

Jaraal wrote: »

davidtk wrote: »

May I ask what I should see? I got a timeout after Akamai.

Yes, it looks like your signal got hung up at Akami before it was redirected to ZOS. 143ms is significant lag.

I'm honestly surprised that my connection to the NA server is over 11 hops and to the EU (which is in a neighboring state) over 13.
Anyway, I've been observing that timeout on Akamai for about 2-3 years, when I was solving my internet over two ISPs and almost 10 network devices due to ESO (Crashes, disconnects, booting from server etc. Luckily after ISP change this was solved by replacing the second ISP's router with a gateway on my fiber connection [A lot of arguing with technicians about having two routers in a row when I'm still not allowed to touch their main one is nonsense lol, I hate not having control over the settings, so I got a gateway], and I also still have two more routers in backup for possible replacement).

Desiato · May 11

davidtk wrote: »

Jaraal wrote: »

davidtk wrote: »

May I ask what I should see? I got a timeout after Akamai.

Yes, it looks like your signal got hung up at Akami before it was redirected to ZOS. 143ms is significant lag.

I'm honestly surprised that my connection to the NA server is over 11 hops and to the EU (which is in a neighboring state) over 13.
Anyway, I've been observing that timeout on Akamai for about 2-3 years, when I was solving my internet over two ISPs and almost 10 network devices due to ESO (Crashes, disconnects, booting from server etc. Luckily after ISP change this was solved by replacing the second ISP's router with a gateway on my fiber connection [A lot of arguing with technicians about having two routers in a row when I'm still not allowed to touch their main one is nonsense lol, I hate not having control over the settings, so I got a gateway], and I also still have two more routers in backup for possible replacement).

According to a geoip lookup, you're connecting to the akamai router in Austria. You don't see more hops because between them and ZOS you're within Akamai's network. There is likely several hops between Austria and Texas.

Of course one should expect latency between Austria and Maryland Texas. The geo ip info for 198.20.200.1, but a ZOS help article says the NA server is hosted in Texas.

davidtk · May 11

Desiato wrote: »

According to a geoip lookup, you're connecting to the akamai router in Austria. You don't see more hops because between them and ZOS you're within Akamai's network. There is likely several hops between Austria and Texas.

Of course one should expect latency between Austria and Maryland Texas. The geo ip info for 198.20.200.1, but a ZOS help article says the NA server is hosted in Texas.

Thanks for explanation about Akamai.
I tried several geoip lookup for EU and NA servers months ago and sometimes it was Maryland and most Texas.
Ofc I can't expect that from Czechia to Texas will have low latency but as I said before there, i played on NA a bit with ping 120-160 but now its horrible 240+++ up to 999+.
And I have about 6+ hours difference from NA so when I played on NA it was usualy about 1pm in US. So no prime time presume, ESO looked empty.
And only change recently was upgrade from Win10 to Win11 but on wife's Win10 is the same situation, so nothing to do with Windows version.

Seravi · May 11

Yes the sever is in Texas. I could throw a rock at the data center and hit it since I live in Dallas.

This is why I have to wonder about the server being an issue. My connection is flawless unless my provider or the server is having issues. No provider issues but I will run the ping today when I play and see what I get.

Desiato · May 11

Seravi wrote: »

Yes the sever is in Texas. I could throw a rock at the data center and hit it since I live in Dallas. This is why I have to wonder about the server being an issue. My connection is flawless unless my provider or the server is having issues. No provider issues but I will run the ping today when I play and see what I get.

Yeah, to be clear I'm not making any assumptions about what you're experiencing. It seems to me that respondents in this thread are experiencing at least a couple of different kinds of issues.

But I think due diligence is important when troubleshooting.

Because the server is working well for so many players, I tend to think there may be a network issue such as a problematic router affecting some, but not all links to the zos server. Maybe it's something to do with their ISP or maybe it's within their data center. But there are other possibilities as well. I personally think a client issue is least likely, but it's not impossible.

Wildberryjack · May 11

Desiato wrote: »

I personally think a client issue is least likely, but it's not impossible.

Here's the thing, this all started after the May 7 update and that mess with stuff carrying over from PTS to live. I don't believe in coincidences. It's on their end. They need to sort it and fix it.

noblecron · May 11

Meanwhile I have npcs not showing up and ESO just randomly crashed on me. didn't catch the error message that popped up either but it was so weird. I'm going to try and run a repair for the heck of it

Desiato · May 11

Wildberryjack wrote: »

Desiato wrote: »

I personally think a client issue is least likely, but it's not impossible.

Here's the thing, this all started after the May 7 update and that mess with stuff carrying over from PTS to live. I don't believe in coincidences. It's on their end. They need to sort it and fix it.

Coindences are common. If troubleshooting is part of one's job, they learn coincidences are common and to not make assumptions. Correlation does not mean causation.

I would expect a client issue to affect more people. If most players were experiencing the severe issues some are reporting here, there would be 100 pages of responses.

But again, I don't think it's impossible you're experiencing a client issue. I know that I'm not, however. Even if it is a network issue, then it is probably on their end and they have to figure it out, so it's better if players provide as much detail about their experiences as possible.

Also, because so many players are experiencing no issues at all, it's possible that you're affected by a ZOS issue, but another player with the same symptoms may not be, so no one should neglect troubleshooting in their end.

ghastley · May 11

Wildberryjack wrote: »

Here's the thing, this all started after the May 7 update and that mess with stuff carrying over from PTS to live. I don't believe in coincidences. It's on their end. They need to sort it and fix it.

The logical conclusion is that any ddos attack was likely to have been provoked by the PTS incident, not that the issue is at the server.

Wildberryjack · May 12

ghastley wrote: »

Wildberryjack wrote: »

Here's the thing, this all started after the May 7 update and that mess with stuff carrying over from PTS to live. I don't believe in coincidences. It's on their end. They need to sort it and fix it.

The logical conclusion is that any ddos attack was likely to have been provoked by the PTS incident, not that the issue is at the server.

Either way it's on their end. Funny that it started for everyone on May 7, after that PTS debacle, and that the EU server is not having this issue. The fact that I'm having superior results playing on the EU server when I live in the same state as the NA server is highly suspect also. There is no way that should be the case.

I ran a repair on the game just in case. I checked my connection and reset the router just in case. I made sure everything on my computer is updated just in case. I play other games whose servers are in California and Chicago with no lag or freezing. There is something wrong with their NA server, whether it's hardware or software or whatever I don't care but they need to sort it and fix it.

davidtk · May 12

Wildberryjack wrote: »

Either way it's on their end. Funny that it started for everyone on May 7, after that PTS debacle, and that the EU server is not having this issue.

I've definitely had issues with NA for longer than six days... But yes really, it's the worst since May 7.
I have been experiencing higher ping and lags for almost a month. And this is exactly since the problems with PTS when their database got mixed up with NA. Before it was bearable. I've checked on my end what I could. I can't do anything on ISP side and Akamai network.
Edit: I forgot to mention that EU have similar issues too what I know from guildies and some others.
Even there: on other topic

TechMaybeHic · May 12

Pretty sure the issue where PTS items came across to live PCNA showed they are tied together in some way. Maybe not compute, but data at least. Their fix for that probably involved walling them off and probably impacted the way the game works. That or; I am never against blaming Akamai.

StihlReign · May 12

Jaraal wrote: »

davidtk wrote: »

May I ask what I should see? I got a timeout after Akamai.

Yes, it looks like your signal got hung up at Akami before it was redirected to ZOS. 143ms is significant lag.

Desiato wrote: »

It's a good thing that ZOS is a customer of akamai because they provide top notch ddos mitigation solutions but it doesn't mean there's any ddos mitigations currently active.

I guarantee you that every bit of data that gets to the ZOS servers is sent to the scrubber service first.

[snip]
[edited for baiting]

Yeah, still a problem. Same slowdown...adding time and lag. I'm seeing it here.

Jaimeh · May 14

Still getting disconnected with the following message:

and when I try to log back in I get the following:

I have to wait around 5 minutes in order to be able to log back in. This is on PC/NA, while doing overland questing. Any news please on why this keeps happening and whether it might be resolved soon @ZOS_GinaBruno, @ZOS_JessicaFolsom, @ZOS_Kevin?

runa_gate · May 14

There certainly doesn't seem to be an lessening of people reporting issues; I myself strangely only have issues in IA or when attempting to log in after not playing for a while and never making it to the character select screen, necessitating killing the .exe and relaunching (I never use the Launcher except to update, though the beta launcher memory leak might have already been resolved there's no reason to leave it open, or open it at all except to update).

Yamenstein · May 15

So any update on all the disconnects/crashes?

Logging in takes two attempts most of the time.
The game just crashed on me randomly while running BGs - message came up about corrupted files on the launcher. But I can then start up the game fine again. Thankfully deserter penalty is only a couple of minutes..

sarahthes · May 15

Had to cancel a raid mid run tonight as several people froze and more DC'd. Laggy bar swaps etc too.

droz8881 · May 15

same here - constantly getting DC'd.

raised a ticket a while ago. i have to assume ZOS team are investigating.

some sort of status update from ZOS team would be really appreciated.

runa_gate · May 15

https://forums.elderscrollsonline.com/en/discussion/comment/8112241#Comment_8112241

Tandor · May 15

runa_gate wrote: »

https://forums.elderscrollsonline.com/en/discussion/comment/8112241#Comment_8112241

That relates to the PTS, not to the live servers.