PC/NA just went down

furiouslog · May 15

StihlReign wrote: »

Jaraal wrote: »

Packet loss and high latency can be traced to the anti-DDoS servers all data is routed to before it gets to the ESO servers. This has been well documented with tracerts for years.

https://forums.elderscrollsonline.com/en/discussion/436992/high-latency-at-akamai-com/p1

Good ole Akamai anti-ddos service at it again?

Tracing route to 198.20.200.1 over a maximum of 30 hops

1 7 ms 14 ms <1 ms XXXXXXXXXXXXXXX
2 2 ms 2 ms 2 ms lo0-100.RCMDVA-VFTTP-309.verizon-gni.net [72.84.244.1]
3 3 ms 3 ms 3 ms B3309.RCMDVA-LCR-21.verizon-gni.net [130.81.30.208]
4 * * * Request timed out.
5 * * * Request timed out.
6 7 ms 7 ms 7 ms akamai-ic-363013.ip.twelve99-cust.net [62.115.38.17]
7 8 ms 8 ms 9 ms a104-86-71-82.deploy.static.akamaitechnologies.com [104.86.71.82]
8 * * * Request timed out.
9 39 ms 39 ms 39 ms 198.20.200.1

Trace complete.

Tracing route to 198.20.198.110 over a maximum of 30 hops

1 <1 ms 1 ms <1 ms XXXXXXXXXXXXXX
2 1 ms 1 ms 2 ms lo0-100.RCMDVA-VFTTP-309.verizon-gni.net [72.84.244.1]
3 14 ms 14 ms 14 ms B3309.RCMDVA-LCR-22.verizon-gni.net [130.81.30.210]
4 * * * Request timed out.
5 * * * Request timed out.
6 8 ms 9 ms 8 ms akamai-ic-317124.ip.twelve99-cust.net [62.115.146.141]
7 7 ms 7 ms 8 ms po111.bs-a.sech-iad.netarch.akamai.com [209.200.144.194]
8 7 ms 9 ms 7 ms a72-52-1-181.deploy.static.akamaitechnologies.com [72.52.1.181]
9 9 ms 9 ms 9 ms ae120.access-a.sech-iad.netarch.akamai.com [209.200.144.197]
10 7 ms 7 ms 8 ms a72-52-63-42.deploy.static.akamaitechnologies.com [72.52.63.42]
11 * * * Request timed out.
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.

Okay, which one of you netcode jockeys has something to say about this? I don't know much about this stuff at all. However, looking at the old support thread, it pretty much seems like whatever was happening back in 2018 is still happening now, so that's a thing, I guess.

sarahthes · May 15

I don't think it's the DDoS protection causing issues. I think it's a ddos. I also think that's why they're not saying anything (because giving it visibility is the last thing you want to do).

Desiato · May 15

furiouslog wrote: »

27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.

Okay, which one of you netcode jockeys has something to say about this? I don't know much about this stuff at all. However, looking at the old support thread, it pretty much seems like whatever was happening back in 2018 is still happening now, so that's a thing, I guess.

Here's a thread with different valid explanations about why this happens:

https://superuser.com/questions/278952/why-can-i-ping-an-ip-address-but-not-traceroute-it

The explaination I think is relevant here is:

Several ISPs are known to make their routers stealth to traceroute in two ways: they either don't decrement TTL in IP packets (making themselves IP wormholes) or they don't respond to expired TTL while still forwarding ICMP.

The reason is to keep their internal network topology private. That's all

I would guess the point here is to hide network details of the login structure from malicious actors.

You might get an easier to understand result with pathping, but it won't really tell you anything more.

ZOS provides two NA IPs for troubleshooting connection issues. One for the login server and one for gameplay connection issues. I can tracert to the gameplay IP np, but not the login server. Yet I have no problems connecting to the game. If I use pathping instead, it will recognize the last router providing the information it is looking for.

Even though there are sometimes routers we cannot see, the value of running the tracert is to see if we can identify latency in the routers that do respond. It's not a perfect tool, but can be helpful in diagnosing connection issues.

Desiato · May 15

sarahthes wrote: »

I don't think it's the DDoS protection causing issues. I think it's a ddos. I also think that's why they're not saying anything (because giving it visibility is the last thing you want to do).

It's possible it's a ddos affecting some parts of the network, but not others, but there could be other possibilities.

It's definitely not the ddos protection itself. It's crazy to me that people would blame it and think it's a problem to have it.

I don't think it's as widespread as some seem to think because if you search for lag in the eso subreddit and sort by new, it's not a recent topic of discussion.

I don't doubt that there's a problem at some level and/or at some times affecting a lot of users though.

sarahthes · May 15

Desiato wrote: »

sarahthes wrote: »

I don't think it's the DDoS protection causing issues. I think it's a ddos. I also think that's why they're not saying anything (because giving it visibility is the last thing you want to do).

It's possible it's a ddos affecting some parts of the network, but not others, but there could be other possibilities.

It's definitely not the ddos protection itself. It's crazy to me that people would blame it and think it's a problem to have it.

I don't think it's as widespread as some seem to think because if you search for lag in the eso subreddit and sort by new, it's not a recent topic of discussion.

I don't doubt that there's a problem at some level and/or at some times affecting a lot of users though.

It has been far far worse than usual. Not since the patch, but since the night PC NA crashed repeatedly. Last night wasn't as bad as that night, but several raids got cancelled as far as I know. It was intermittent throughout the night both in trials and PvP zones.

GooGa592 · May 15

Desiato wrote: »

sarahthes wrote: »

I don't think it's the DDoS protection causing issues. I think it's a ddos. I also think that's why they're not saying anything (because giving it visibility is the last thing you want to do).

It's possible it's a ddos affecting some parts of the network, but not others, but there could be other possibilities.

It's definitely not the ddos protection itself. It's crazy to me that people would blame it and think it's a problem to have it.

I don't think it's as widespread as some seem to think because if you search for lag in the eso subreddit and sort by new, it's not a recent topic of discussion.

I don't doubt that there's a problem at some level and/or at some times affecting a lot of users though.

Except the problem is widespread. There have been many new posts about it in the last couple weeks, and you've posted in most of those threads downplaying the problem.

It's not a new issue though, so most people have just given up and stopped bothering to post about it because they've realized that complaining about it is a waste of time.

Desiato · May 15

sarahthes wrote: »

It has been far far worse than usual. Not since the patch, but since the night PC NA crashed repeatedly. Last night wasn't as bad as that night, but several raids got cancelled as far as I know. It was intermittent throughout the night both in trials and PvP zones.

It's been far worse for SOME players. If it was everyone, surely there would be mention of it on reddit, no? And if it affected everyone to the degree described by some in this thread, the forums would be melting down. I would be upset too.

But it's not.

Also, the server wasn't crashing repeatedly on the 7th. When one could get on it, the server was fantastic. I connected and dropped multiple times with no rollbacks -- which would happen if it was crashing. The problem that night was connectivity issues.

And while you say your problems began that night, others say before then, so it seems to me that there's more than one issue affecting some people differently.

GooGa592 wrote: »

Except the problem is widespread. There have been many new posts about it in the last couple weeks, and you've posted in most of those threads downplaying the problem.

It's not a new issue though, so most people have just given up and stopped bothering to post about it because they've realized that complaining about it is a waste of time.

If it was widespread, there would be MUCH more discussion about it. It's funny that you want others to believe your issues are real, which I do, but some of you refuse to accept other players have no problems at all.

I searched for lag on the eso subreddit and sorted by new and saw no discussion on Reddit about this issue.

I am part of a raiding Discord with more than 1k players several HM progression groups and no one is talking about it. I've been in multiple vet trial groups in which no one has experienced issues.

Again, I'm not doubting you, so please don't doubt my point of view either. I believe there's an issue, but we're not seeing a reaction indicative of something affecting most players.

sarahthes · May 15

Desiato wrote: »

sarahthes wrote: »

It has been far far worse than usual. Not since the patch, but since the night PC NA crashed repeatedly. Last night wasn't as bad as that night, but several raids got cancelled as far as I know. It was intermittent throughout the night both in trials and PvP zones.

It's been far worse for SOME players. If it was everyone, surely there would be mention of it on reddit, no? And if it affected everyone to the degree described by some in this thread, the forums would be melting down. I would be upset too.

But it's not.

Also, the server wasn't crashing repeatedly on the 7th. When one could get on it, the server was fantastic. I connected and dropped multiple times with no rollbacks -- which would happen if it was crashing. The problem that night was connectivity issues.

And while you say your problems began that night, others say before then, so it seems to me that there's more than one issue affecting some people differently.

GooGa592 wrote: »

Except the problem is widespread. There have been many new posts about it in the last couple weeks, and you've posted in most of those threads downplaying the problem.

It's not a new issue though, so most people have just given up and stopped bothering to post about it because they've realized that complaining about it is a waste of time.

If it was widespread, there would be MUCH more discussion about it. It's funny that you want others to believe your issues are real, which I do, but some of you refuse to accept other players have no problems at all.

I searched for lag on the eso subreddit and sorted by new and saw no discussion on Reddit about this issue.

I am part of a raiding Discord with more than 1k players several HM progression groups and no one is talking about it. I've been in multiple vet trial groups in which no one has experienced issues.

Again, I'm not doubting you, so please don't doubt my point of view either. I believe there's an issue, but we're not seeing a reaction indicative of something affecting most players.

And I know the problems tend to start late, when they happen. It's connectivity issues but I don't think they are ISP related, given that one night I was grouped with a person from Ohio and another from Louisiana. I'm in western Canada. Me and the Ohio person had the exact same issues, the LA person didn't.

I do not believe it's the server directly, ISPs, or add-ons.

But when people from vastly different areas have lag spikes simultaneously in raid, then disconnect, it's almost certainly at ZOS' end.

There was one raid recently where 5 people crashed simultaneously.

The people I talk about it with generally don't post on the forums or Reddit. The discussions are mostly in discord, of which I'm in (at last count) 35 separate ones with between 200 and 20000 members. Most of the people I play with are progging trifectas or scorepushing.

Desiato · May 15

sarahthes wrote: »

And I know the problems tend to start late, when they happen. It's connectivity issues but I don't think they are ISP related, given that one night I was grouped with a person from Ohio and another from Louisiana. I'm in western Canada. Me and the Ohio person had the exact same issues, the LA person didn't.

I do not believe it's the server directly, ISPs, or add-ons.

But when people from vastly different areas have lag spikes simultaneously in raid, then disconnect, it's almost certainly at ZOS' end.

There was one raid recently where 5 people crashed simultaneously.

I agree that there's likely a connectivity issue that ZOS is responsible for identifying and solving. It could be an issue with hosting or the ISP(s) ZOS uses. It may be that one of their service providers is performing their own upgrades or maintenance during off-peak hours.

I play prime time and am always done by 2 AM EDT, but usually before 12.

I'm not blaming any users themselves or saying it's their isp. Although, in some cases it may be so everyone should perform due diligence and ensure there are no issues on their end. Learning to ping the gateway of one's isp is a valuable troubleshooting tool that has helped me identify isp issues dozens of times in my life. And when I say gateway, I don't mean the home router. My ISP had problems yesterday and I was able to identify them quickly by doing so and get the issue fixed within hours.

Also, users shouldn't just post their bad experiences here. Creating support tickets is the best way to make ZOS aware of an issue. Especially if they are submitted in-game while the issue is happening.

reazea · May 16

Desiato wrote: »

sarahthes wrote: »

And I know the problems tend to start late, when they happen. It's connectivity issues but I don't think they are ISP related, given that one night I was grouped with a person from Ohio and another from Louisiana. I'm in western Canada. Me and the Ohio person had the exact same issues, the LA person didn't.

I do not believe it's the server directly, ISPs, or add-ons.

But when people from vastly different areas have lag spikes simultaneously in raid, then disconnect, it's almost certainly at ZOS' end.

There was one raid recently where 5 people crashed simultaneously.

I agree that there's likely a connectivity issue that ZOS is responsible for identifying and solving. It could be an issue with hosting or the ISP(s) ZOS uses. It may be that one of their service providers is performing their own upgrades or maintenance during off-peak hours.

I play prime time and am always done by 2 AM EDT, but usually before 12.

I'm not blaming any users themselves or saying it's their isp. Although, in some cases it may be so everyone should perform due diligence and ensure there are no issues on their end. Learning to ping the gateway of one's isp is a valuable troubleshooting tool that has helped me identify isp issues dozens of times in my life. And when I say gateway, I don't mean the home router. My ISP had problems yesterday and I was able to identify them quickly by doing so and get the issue fixed within hours.

Also, users shouldn't just post their bad experiences here. Creating support tickets is the best way to make ZOS aware of an issue. Especially if they are submitted in-game while the issue is happening.

People absolutely SHOULD be posting about their in game issues on the official forum that exists here for that specific purpose. That's how others learn whether or not it's just them having the issue or if it's widespread, as this lag spiking and disconnecting issue clearly is. That's why there are so many posts about it even though most people have given up on the issue ever being resolved and have stopped posting about it.

Desiato · May 16

reazea wrote: »

Desiato wrote: »

Also, users shouldn't just post their bad experiences here. Creating support tickets is the best way to make ZOS aware of an issue. Especially if they are submitted in-game while the issue is happening.

People absolutely SHOULD be posting about their in game issues on the official forum that exists here for that specific purpose.

I didn't say they shouldn't. I said they should also log support tickets because that's the best way to make ZOS aware of an issue.

It also gives them relevant information such as account information, how we connect to them, PC specs, and perhaps details about the server when the issue occurred. If enough people do that, it can help them connect the dots. They're not going to look up this information based on forum posts.

TechMaybeHic · May 16

I had a lot of issues a while ago when I normally hadn't and it had to do with the beta patcher. Switched it off from Beta and no more issues. Try that if you haven't

sarahthes · May 16

TechMaybeHic wrote: »

I had a lot of issues a while ago when I normally hadn't and it had to do with the beta patcher. Switched it off from Beta and no more issues. Try that if you haven't

I don't know anybody using the beta launcher.

Desiato · May 21

I haven't had the consistent issues others in this thread have described, however tonight I am experiencing 2-4% packet loss to 198.20.200.1 that corresponds with severe in-game lag that sometimes has significant gameplay impacts.

Here are my most recent ping statistics:

Ping statistics for 198.20.200.1:
Packets: Sent = 1396, Received = 1359, Lost = 37 (2% loss),
Approximate round trip times in milli-seconds:
Minimum = 48ms, Maximum = 96ms, Average = 61ms

I pinged the gateway of my isp and there was no packet loss. I also performed traceroutes and everything up to ESO looked good. The hop before 198.20.200.1 is 184.26.104.150.

There doesn't seem to be a way to report this kind of lag on the online support site, but we can report it in-game. I highly recommend everyone who experiences this issue to take a moment to do so when it happens. Use the /bug command to bring up this window and select the options shown. Provide as much information about your experience as you can.

@ZOS_Kevin

Desiato · May 22

Another night of packet loss:

Ping statistics for 198.20.200.1:
Packets: Sent = 1012, Received = 989, Lost = 23 (2% loss),
Approximate round trip times in milli-seconds:
Minimum = 51ms, Maximum = 92ms, Average = 61ms

Edit to add that it's been consistent all night:

Ping statistics for 198.20.200.1:
Packets: Sent = 13326, Received = 13039, Lost = 287 (2% loss),
Approximate round trip times in milli-seconds:
Minimum = 49ms, Maximum = 99ms, Average = 70ms

shadyjane62 · May 22

Just went down in PCNA. three different errors. PCEU no problem.