High latency at akamai.com

Justmee

I do have a ticket open with support on this as well
for Elder Scrolls Online (http://www.elderscrollsonline.com)
Question Reference # 180920-003202

akamai.com the bottle neck appears to be at YOUR
network! Some of the information in here may not
be relevant for you akamai.com but please read
the full email this was also sent in a ticket on
Elder Scrolls Online as well.

Ok so there seems to be a constant latency to the
Elder Scrolls online North America servers.

I am getting a latency a lot of time in excess of
175 ms.

Now so you know I am a trained network admin myself
and I know the issue is NOT on my side here.

I am on a wired connection and on a machine powerful
enough to handle this game.

Intel i7-2500 3.40 Ghz 16 GB of RAM running
Windows 10 Pro (64 bit)

So I know the issue is not here on my side because
I know my network and everything is updated including
my router and please don't tell me to open up ports
I have already tried it. It does not solve the issue.

So I first used Wireshark to find out the IP for
the server which I was on it appears to be
198.20.200.155 which of course comes back to Zenimax
as the owner as expected.

https://whois.arin.net/rest/net/NET-198-20-192-0-1/pft?s=198.20.200.155


So since the latency can be caused anywhere along
a hop that may be congested so I started to do several
traceroutes.


traceroute 198.20.200.155
traceroute to 198.20.200.155 (198.20.200.155), 30 hops max, 60 byte packets
1 router.lan.darkshado.ca (192.168.0.1) 0.445 ms 0.541 ms 1.029 ms
2 rc1we-be107-1.ed.shawcable.net (64.59.186.53) 9.895 ms 9.907 ms 11.333 ms
3 rc3no-be6.cg.shawcable.net (66.163.64.69) 13.619 ms 14.279 ms 14.324 ms
4 rc2wt-be100.wa.shawcable.net (66.163.75.233) 29.925 ms 29.882 ms 29.833 ms
5 rc1wt-be18-1.wa.shawcable.net (66.163.64.81) 30.010 ms 29.878 ms rc4sj-be60.cl.shawcable.net (66.163.78.74) 50.815 ms
6 rc3sj-be60.cl.shawcable.net (66.163.75.90) 49.131 ms 47.988 ms 47.940 ms
7 akamai.prolexic.com (206.223.116.136) 53.334 ms 47.109 ms 46.701 ms
8 po110.bs-a.sech-sjc.netarch.akamai.com (209.200.184.192) 48.783 ms po110.bs-b.sech-sjc.netarch.akamai.com (209.200.184.200) 48.713 ms po110.bs-a.sech-sjc.netarch.akamai.com (209.200.184.192) 48.742 ms
9 ae120.access-a.sech-sjc.netarch.akamai.com (209.200.184.197) 266.280 ms ae121.access-a.sech-sjc.netarch.akamai.com (209.200.184.205) 265.902 ms 266.602 ms
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *
16 * * *
17 * * *
18 * * *
19 * * *
20 * * *
21 * * *
22 * * *
23 * * *
24 ae120.access-a.sech-sjc.netarch.akamai.com (209.200.184.197) 233.638 ms !X * *


well that really isn't good 265 MS just one hop inside of akamai's network
so it hits the routers on the border and the times are great at about 48 ms

Now where ever that piece of equipment is, is severely congested and that appears
to be where the latency issue is coming from.

The stars don't bother me at all because I know that you can setup a piece of
equipment to not answer ICMP requests which is more than likely what
akamai.com is doing since they are an anti-ddos type service.

But what does bother me is the huge jump in the latency time once it gets past
border switches on akamai.com it takes a huge jump from 48 ms to 265 ms
at IP 209.200.184.205 and as I said it is constant I did multiple traceroutes
and it seems like it bottlenecks in the same place every time.

Akamai is a anti-ddos service and I understand why they use them but
thier equipment should be able to handle the load the game servers
will put on it and it appears that it isn't right now.

Now this was done on September 20 2018 at about 4:00 PM MST (GMT-0700)

Come on can some one at the support please get in contact with akamai.com
and have them resolve this issue please.

Looks like I am not the only one this is just one example of many that
you can find about a high latency.

https://forums.elderscrollsonline.com/en/discussion/435522/disconnections-rubber-banding-high-latency-spikes-and-my-tech-support-ticket-180910-002528

Also I have tried disabling any security software I have which is
Kaspersky 18.0.0.405(i) and that does NOT fix the issue the issue
clearly is NOT on my end at all and this isn't a connection
issue here on my side.


I will also email this to the Network Operations Center ("NOC") at
akamai.com to see if they can take a look into the issue.


Oh so a bit of an update at the request of Akamai.com they wanted me to do a tracetcp which I did
on September 21 2018 about 11:50 AM MST (GMT-0700)

tracetcp 198.20.200.155

Tracing route to 198.20.200.155 on port 80
Over a maximum of 30 hops.
1 1 ms 1 ms 1 ms 192.168.0.1 [router.lan.darkshado.ca]
2 25 ms 11 ms 10 ms 64.59.186.53 [rc1we-be107-1.ed.shawcable.net]
3 16 ms 17 ms 14 ms 66.163.64.69 [rc3no-be6.cg.shawcable.net]
4 31 ms 28 ms 29 ms 66.163.75.233 [rc2wt-be100.wa.shawcable.net]
5 50 ms 52 ms 30 ms 66.163.64.81 [rc1wt-be18-1.wa.shawcable.net]
6 49 ms 50 ms 54 ms 66.163.67.81 [rc3sj-tge0-0-0-0.cl.shawcable.net]
7 48 ms 48 ms 49 ms 206.223.116.136 [akamai.prolexic.com]
8 49 ms 48 ms 48 ms 209.200.184.192 [po110.bs-a.sech-sjc.netarch.akamai.com]
9 259 ms * * 209.200.184.205 [ae121.access-a.sech-sjc.netarch.akamai.com]
10 * * * Request timed out.
11 * 76 ms 77 ms 198.20.192.3
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.
Trace Complete.

Now you can see at hop 8 everything is great which again is the boarder of their
network at akamai.com but once again once it hits the 2nd hop inside of akamai.com
it only answers once and is at 259 ms.

On hop 8 again it is about 48 - 49 MS which is great and then on hop 9
once again it jumps up to 259 ms.

Again this issue is definately inside of akamai.com's network but these guys are idiots
at akamai.com and claim they won't do anything unless one of their customers complain
about the issue. Which would have to be Zenimax.

These guys are complete idiots at akamai.com I have a ticket open with them
as well AKAM-NOCC #B-P-2HW3BOR

If you are getting the same problem be sure to email
nocc-tix -[at]- akamai.com, nocc-shift -[at]- akamai.com, noc -[at]- akamai.com,
support -[at]- akamai.com

Can some one from Zenimax please complain to them about this issue and get this
fixed the people at akamai.com have no clue how to properly run their network and
the bottle neck clearly is within their network but they refuse to do anything because
I am not their direct "customer".

I hate idiot companies like this I have as much or more training in the computer field
that than that idiot that answered the ticket.

gamer559

Same issue you are having & same results. The problem is with Akamai.

@ZOS_GinaBruno @ZOS_KaiSchober @ZOS_JessicaFolsom @ZOS_BillE @ZOS_GaryA This needs to be addressed!

Justmee

Did you use wireshark first to capture a few packets to ensure that
you are connecting to the same server as me and that your packets
are going to the exact same IP like I did?

Ok if you contact Akamai they will probably ask that you do the tracetcp as well.

But ya it looks like akamai.com is the issue here and the source of the latency
some piece of equipment on their network seems to be bottle necking.

If Akamai.com isn't willing or can't fix this issue then Zenimax
needs to move to another ddos protection service immediately.

I have my ticket open and I attached the log files that were requested when I created
the ticket and they responded asking me to to attach the Game Consult
logs again I told em to go back and look at the last response I made because they
were already attached and I attached them again for a 2nd time and uploaded it to
my mediafire.com dropbox as well and gave them the link to it there as well.

God the support is just horrible on this game I need to get to some one higher
with in the organization some one with at least some common sense that
I can talk to about this issue. Maybe a network admin or some one.

Justmee

So I opened this ticket on September 20 2018 and it has now been open for 9
days and NO response on this ticket.

What the heck I mean really does Zenimax just not give a damn about its
players and issues they are experiencing this is absolute BS the game
is unplayable and you lag constantly.

I sent them a response asking them what is going on with this ticket
and that the issue is still ongoing and no response yet from them.

The problem STILL exists This traceroute was done on September 27 2018
at about 5:25 PM MST (GMT-0700)

Again look at the jump on hop 9 goes from about 47 - 54 MS all the way up to
219 - 254 MS

This is just unacceptable nothing has been fixed on this issue!

tracert 198.20.200.155

Tracing route to 198.20.200.155 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms router.lan.darkshado.ca [192.168.0.1]
2 9 ms 8 ms 8 ms rc1we-be107-1.ed.shawcable.net [64.59.186.53]
3 14 ms 13 ms 13 ms rc3no-be6.cg.shawcable.net [66.163.64.69]
4 28 ms 28 ms 28 ms rc2wt-be100.wa.shawcable.net [66.163.75.233]
5 51 ms 49 ms 49 ms rc4sj-be60.cl.shawcable.net [66.163.78.74]
6 49 ms 50 ms 50 ms rc3sj-tge0-0-0-0.cl.shawcable.net [66.163.67.81]
7 47 ms 48 ms 48 ms akamai.prolexic.com [206.223.116.136]
8 47 ms 54 ms 47 ms po110.bs-a.sech-sjc.netarch.akamai.com [209.200.184.192]
9 219 ms 254 ms 230 ms ae120.access-a.sech-sjc.netarch.akamai.com [209.200.184.197]
10 * * * Request timed out.
11 75 ms 75 ms 75 ms 198.20.192.4
12 74 ms 74 ms 74 ms 198.20.200.155

Trace complete.


And here is another tracetcp as well and yet again the times go from 49 ms
on hop 8 all the way up to 252 ms on hop 9

tracetcp 198.20.200.155

Tracing route to 198.20.200.155 on port 80
Over a maximum of 30 hops.
1 1 ms 1 ms 2 ms 192.168.0.1 [router.lan.darkshado.ca]
2 13 ms 12 ms 10 ms 64.59.186.53 [rc1we-be107-1.ed.shawcable.net]
3 15 ms 14 ms 14 ms 66.163.64.69 [rc3no-be6.cg.shawcable.net]
4 30 ms 29 ms 31 ms 66.163.75.233 [rc2wt-be100.wa.shawcable.net]
5 53 ms 35 ms 29 ms 66.163.64.81 [rc1wt-be18-1.wa.shawcable.net]
6 53 ms 49 ms 48 ms 66.163.75.90 [rc3sj-be60.cl.shawcable.net]
7 49 ms 58 ms 49 ms 206.223.116.136 [akamai.prolexic.com]
8 49 ms 49 ms 49 ms 209.200.184.192 [po110.bs-a.sech-sjc.netarch.akamai.com]
9 199 ms 223 ms 252 ms 209.200.184.205 [ae121.access-a.sech-sjc.netarch.akamai.com]
10 * * * Request timed out.
11 77 ms 78 ms * 198.20.192.4
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.
Trace Complete.

The issue is still ongoing as I showed zenimax in these
traceroutes one is a tcptraceroute and the other is a regular
traceroute (ICMP)

See ticket at akamai [AKAM-NOCC #B-P-2HW3BOR] and
[AKAM-CASE #F-CS-3010727]

Elder Scrolls Support Ticket #: 180920-003202


Issue is still there and absolutely nothing is being done
by Zenimax to resolve the issue wow absolutely horrible service!

AngryFire

Please see my latest updates, same issue

https://forums.elderscrollsonline.com/en/discussion/438321/extreme-latency-999-game-unplayable

Vapirko

I did some research on this a while back after discovering the same thing. From what I understand Akamai was implemented to protect against DDOS attacks (a year or so ago? Maybe two?), that also coincided with a noticeable downgrade in connection quality especially for overseas users. I posted a couple of threads and comments about it but I don’t think you’ll see any response from ZOS, and everyone will just give you the usual responses. It’s definitely the cause of bad latency issues for some users though, and I would guess that ZOS is really not interested in dealing with it since the issue has been around for some time, is something that ZOS decided to implement and so has some measure of control over, and has as of yet made no statement about it either at inception or otherwise.

AngryFire

It is obvious to me that ZOS has no interest at all in resolving this issue or even acknowledging that it is even an issue to begin with. Unless your issue is a simple issue which can be resolved by their pre-written canned responses with total disregard to evidence provided you are SOL. I cancelled my ESO Plus and will no longer be giving this company my money.

Vapirko

AngryFire wrote: »

It is obvious to me that ZOS has no interest at all in resolving this issue or even acknowledging that it is even an issue to begin with. Unless your issue is a simple issue which can be resolved by their pre-written canned responses with total disregard to evidence provided you are SOL. I cancelled my ESO Plus and will no longer be giving this company my money.

Same just after WW patch. And looking at the disaster that is Murkmire patch notes I ain’t coming back.

Justmee

Vapirko wrote: »

I did some research on this a while back after discovering the same thing. From what I understand Akamai was implemented to protect against DDOS attacks (a year or so ago? Maybe two?), that also coincided with a noticeable downgrade in connection quality especially for overseas users. I posted a couple of threads and comments about it but I don’t think you’ll see any response from ZOS, and everyone will just give you the usual responses. It’s definitely the cause of bad latency issues for some users though, and I would guess that ZOS is really not interested in dealing with it since the issue has been around for some time, is something that ZOS decided to implement and so has some measure of control over, and has as of yet made no statement about it either at inception or otherwise.

Well it shouldn't be the source of latency if akamai.com could actually keep their
network properly updated and stop running too much traffic though that one piece
of equipment on their network.

There should be no way that this should be occuring and akamai.com needs to
look at that device and figure out what is going on. If they are running
too much traffic though it then they need to add more hardware there
to be able to handle the traffic that they are trying to run though that piece
of equipment.

It just comes down to akamai.com not wanting to fork out the money to properly
maintain their network

Stuff like a Cisco ISR 4431 router can cost you like $18K CAD to buy
at full cost.

http://www.router-switch.com/cisco-isr4431-sec-k9-p-23207.html

But at the same time a company like akamai.com makes millions a year and
should be properly maintaining their network as well.

This is totally unacceptable for this to be occuring and ZOS needs to look into
this and if Akamai.com won't fix the issue then well maybe it is time
that they find another anti-ddos service that can handle the load and
won't send the latency though the roof.

Korinth

Wonder if this is what is causing some of us launcher patch day issues (launcher freezing, not downloading updates).

Vapirko

Justmee wrote: »

Vapirko wrote: »

I did some research on this a while back after discovering the same thing. From what I understand Akamai was implemented to protect against DDOS attacks (a year or so ago? Maybe two?), that also coincided with a noticeable downgrade in connection quality especially for overseas users. I posted a couple of threads and comments about it but I don’t think you’ll see any response from ZOS, and everyone will just give you the usual responses. It’s definitely the cause of bad latency issues for some users though, and I would guess that ZOS is really not interested in dealing with it since the issue has been around for some time, is something that ZOS decided to implement and so has some measure of control over, and has as of yet made no statement about it either at inception or otherwise.

Well it shouldn't be the source of latency if akamai.com could actually keep their
network properly updated and stop running too much traffic though that one piece
of equipment on their network.

There should be no way that this should be occuring and akamai.com needs to
look at that device and figure out what is going on. If they are running
too much traffic though it then they need to add more hardware there
to be able to handle the traffic that they are trying to run though that piece
of equipment.

It just comes down to akamai.com not wanting to fork out the money to properly
maintain their network

Stuff like a Cisco ISR 4431 router can cost you like $18K CAD to buy
at full cost.

http://www.router-switch.com/cisco-isr4431-sec-k9-p-23207.html

But at the same time a company like akamai.com makes millions a year and
should be properly maintaining their network as well.

This is totally unacceptable for this to be occuring and ZOS needs to look into
this and if Akamai.com won't fix the issue then well maybe it is time
that they find another anti-ddos service that can handle the load and
won't send the latency though the roof.

Well Im pretty sure we can lay all the performance issues with ESO at the foot of money mountain. I personally don't know a whole lot about networking and all this stuff, but others who work in those fields and play this game are pretty convinced that there's no issue that simply cannot be fixed or at least improved, it just comes down to money and time (which makes sense) and its pretty clear ZOS is not interested in investing much of either. We know we have players who are spending thousands if not tens of thousands (actual forum threads admitting as much) on crown crates and items. And that's just a few people who post here, so as long as that's happening why would ZOS invest what would probably be a substantial amount of time and funding into latency issues which pretty much only seriously affects PvP which is not a money making activity. I hate to say it but the sooner you realize that ZOS, no matter what the devs say, is not in this to make the playing experience as good as it can be, the better. ZOS is a business, they're here to make money first. Plus ESO is pretty well into its life span, so I really doubt it makes any fiscal sense for them to invest in fixing some latency issues which only apply to a small segment of its player base. If they're not losing a substantial amount of money from its current state, then there's no reason to invest in it. If the issue is with Akamai entirely, then if ZOS is paying them what motivation do they have to spend more money? Personally Ive finally come to terms with this, I cut my sub off a few months ago, haven't played more than a handful of times since then and overall Im much happier and less stressed out lol. This game is not in a healthy place if you care about competitive PvP/Gameplay.

Aria_GuildDestroyer

My ping usually sits at 100 but this weekend it's been bouncing between 170 and 700... it's also not on my end. It's definitely an issue and kind of a big one. I chose not to live in Australia for a reason but it doesn't seem to matter at the moment!

SirDopey

Yeah here in Australia there's been a problem for about a month now, overall latency is about 40 - 50ms higher than it use to be. Based on my traceroute's I did a year ago compare to recent ones it's seems to be the second hop inside akamai that is getting seriously congested.

RinaldoGandolphi

I think its pretty irresponsible to blame this issue on Akamai. Akamai is one of the Internet Backbones for North American along with Level Networks 3 and Verizon who bought Genuiity.



I only get a 25ms ping inside Akamai's Network. All the request timeouts mean is that those routers filter ICMP Time Exceeded packets, which is common in many network setups. They in now way cause any issues.

Now my ISP is Comcast, and I know that Comcast has peering agreements with both Level 3 and Akamai. This could be the reason why I only see a 25 ms ping and you guys are seeing higher pings.

If you Google "Comcast Level 3 peering dispute" you will find that the big ISP often sign peering agreements where each side agrees to peer(allow traffic though the network) on contractual specified terms. If one side ends up peering more traffic then the other then that side usually ends up having to pay an agreed amount of cost via the contract. Its for these reasons that ISP's and Large CDN's will sometimes change routers on traffic if it appears they may be on the short end of the peering agreement.

Simply put, their are routes for traffic coming from networks where a peering agreement is in place, and routes for traffic that doesn't come from a source with a peering agreement. It has been like this since 1990's, these are signed contracts between companies. This is the reason it seems every year AT&T, Comcast, Time Warner, etc...the big ones raise rates, because costs go up on both sides due to these peering agreements.

If your traffic is being funneled though a node that seems slower, its probably being done because your ISP is on the short side of the agreement to peer traffic and has to send that traffic though a different route until they even out the peering arrangement.

I don't see this as Akamai's fault, even Akamai has been on the short side of this with Level 3 and Comcast before and had to dial things back a bit to get things even again. Their are no routers overloaded on Akamai's side of the fence.

It is what it is. Maybe one day this will change, but for now even perring network traffic isn't a free lunch, and no company should be forced to peer someones traffic for free.

code65536

The problem is with Akamai's DDoS protection. You're not communicating directly with ZOS's servers. Traffic goes through one of Akamai's anti-DDoS scrubbing centers. And that's been adding quite a lot of latency, esp. during peak hours.

aetherial_heavenn

Just noticed Iinet Australian ISP is being routed thru Japan. It used to hop sydney/brisbane to san francisco/LA. Now
Tracert shows Akamai node latency rockets to 500ms.

Cannot play due to disconnects. Ping spikes of 2000 ms up from 360ms avarage for last 6 months. (Which was up from 280 2 years ago.)
Reinstalling game. Download is a crawl.
So thought i would check forum to see if there was a zos server problem.

BTW This happened before when akamai had DDoS and routed traffic to their scrubber in asia (singapore iirc) rather than in sydney. Few aussies sent tickets and made forum comments. No idea if this was addressed in the akamai parch last year?

Edit. This (peer to peer agreements) maybe the ISP related issue Gina referred to in yet another high ping disconnect issue thread that i can't link atm.

Justmee

Question Reference # 180920-003202

Ok so I get an email today (October 11 2018) from the support team and
the support guy is now Brad has identified himself as being an
Esclation Agent.

So he asks me for yet more traceroutes This is now the 3rd
set of traceroutes I have sent to them and the issue
still is not resolved.

So I sent them yet another set of traceroutes
(Trace route to 198.20.200.155)

On Hop 8 it was at about 46 - 47 ms then on hop 9 again it skyrocketed
up to 252 ms - 283 ms

tracert 198.20.200.155

Tracing route to 198.20.200.155 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms router.lan.darkshado.ca [192.168.0.1]
2 9 ms 10 ms 10 ms rc1we-be107-1.ed.shawcable.net [64.59.186.53]
3 12 ms 13 ms 13 ms rc3no-be6.cg.shawcable.net [66.163.64.69]
4 28 ms 28 ms 27 ms rc2wt-be100.wa.shawcable.net [66.163.75.233]
5 50 ms 50 ms 50 ms rc4sj-be60.cl.shawcable.net [66.163.78.74]
6 50 ms 50 ms 50 ms rc3sj-tge0-0-0-0.cl.shawcable.net [66.163.67.81]
7 47 ms 47 ms 47 ms akamai.prolexic.com [206.223.116.136]
8 47 ms 47 ms 46 ms po110.bs-a.sech-sjc.netarch.akamai.com [209.200.184.192]
9 * 252 ms 283 ms ae120.access-a.sech-sjc.netarch.akamai.com [209.200.184.197]
10 * * * Request timed out.
11 74 ms 74 ms 75 ms 198.20.192.4
12 74 ms 74 ms 75 ms 198.20.200.155

And here is a tracetcp done at about 4:47 PM MST on October 11 2018 same
problem. Look at that on hop 9 257 ms, 266 ms, 263 ms

Now again on hop 8 everything was great at the border of akamai.com's network
times were 48 ms, 50 ms, 48 ms

tracetcp 198.20.200.155

Tracing route to 198.20.200.155 on port 80
Over a maximum of 30 hops.
1 1 ms 2 ms 2 ms 192.168.0.1 [router.lan.darkshado.ca]
2 22 ms 10 ms 12 ms 64.59.186.53 [rc1we-be107-1.ed.shawcable.net]
3 14 ms 15 ms 14 ms 66.163.64.69 [rc3no-be6.cg.shawcable.net]
4 30 ms 29 ms 29 ms 66.163.75.233 [rc2wt-be100.wa.shawcable.net]
5 31 ms 52 ms 50 ms 66.163.78.74 [rc4sj-be60.cl.shawcable.net]
6 51 ms 52 ms 49 ms 66.163.67.81 [rc3sj-tge0-0-0-0.cl.shawcable.net]
7 49 ms 49 ms 50 ms 206.223.116.136 [akamai.prolexic.com]
8 48 ms 50 ms 48 ms 209.200.184.200 [po110.bs-b.sech-sjc.netarch.akamai.com]
9 257 ms 266 ms 263 ms 209.200.184.205 [ae121.access-a.sech-sjc.netarch.akamai.com]
10 * * * Request timed out.
11 * 78 ms 76 ms 198.20.192.4
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.
Trace Complete.

And here we go again same problem different day akamai.com
has done absolutely nothing to resolve the issue.

in the ICMP traceroute (the first one) it hits the border
of Akamai.com's network and everything is great
47 ms, 47 ms, 46 ms (hop 8)

And then it goes one more hop into akamai.com's
network and everything goes sideways yet again

Hop 9 252 ms and 283 ms really?

The same problems also occured in the tracetcp
as well same hop and same problems. (2nd one)

The game is just unplayable load times are horrible
if you go into any kind of a dungeon or even just starting
up the game. I am basically to the point of quitting.

What Zennimax needs to do is pull any services being
provided by akamai.com and find a anti-ddos provider
that will actually work with them on issues like this.

And yes akamai.com is aware of this because I made
their NOC aware of this issue and I got told oh we can't
look into this isuse because you are not our customer.

What a useless *** I am a network admin myself and
went to school for the same thing I know how to properly
troubleshoot an issue like this.

Justmee

code65536 wrote: »

The problem is with Akamai's DDoS protection. You're not communicating directly with ZOS's servers. Traffic goes through one of Akamai's anti-DDoS scrubbing centers. And that's been adding quite a lot of latency, esp. during peak hours.

Yes but I have been in contact with Akamai.com and they are a
bunch of idiots I made their NOC aware of this issue and
their support staff and I got told that they can't look into
the issue because I am not their direct customer.

Wow I am a network admin myself I know how to spot
and troubleshoot network issues.

Wow really so akamai.com knows about the issue but buries
their head in and sand and refuses to do anything to resolve
the problem.

Akamai.com is absolutely useless what ZOS needs to do is
pull all services that akamai.com is providing them and
find an anti-ddos provider that will actually work with them
when there are issues like this.

Right now the game is basically unplayable load times
just loading into the game is absolutely horrible or loading
into a dungeon or anything really.

Justmee

RinaldoGandolphi wrote: »

I think its pretty irresponsible to blame this issue on Akamai. Akamai is one of the Internet Backbones for North American along with Level Networks 3 and Verizon who bought Genuiity.



I only get a 25ms ping inside Akamai's Network. All the request timeouts mean is that those routers filter ICMP Time Exceeded packets, which is common in many network setups. They in now way cause any issues.

Now my ISP is Comcast, and I know that Comcast has peering agreements with both Level 3 and Akamai. This could be the reason why I only see a 25 ms ping and you guys are seeing higher pings.

If you Google "Comcast Level 3 peering dispute" you will find that the big ISP often sign peering agreements where each side agrees to peer(allow traffic though the network) on contractual specified terms. If one side ends up peering more traffic then the other then that side usually ends up having to pay an agreed amount of cost via the contract. Its for these reasons that ISP's and Large CDN's will sometimes change routers on traffic if it appears they may be on the short end of the peering agreement.

Simply put, their are routes for traffic coming from networks where a peering agreement is in place, and routes for traffic that doesn't come from a source with a peering agreement. It has been like this since 1990's, these are signed contracts between companies. This is the reason it seems every year AT&T, Comcast, Time Warner, etc...the big ones raise rates, because costs go up on both sides due to these peering agreements.

If your traffic is being funneled though a node that seems slower, its probably being done because your ISP is on the short side of the agreement to peer traffic and has to send that traffic though a different route until they even out the peering arrangement.

I don't see this as Akamai's fault, even Akamai has been on the short side of this with Level 3 and Comcast before and had to dial things back a bit to get things even again. Their are no routers overloaded on Akamai's side of the fence.

It is what it is. Maybe one day this will change, but for now even perring network traffic isn't a free lunch, and no company should be forced to peer someones traffic for free.

[edited comment]

It has absolutely NOTHING to do with peering agreements what so ever
or traffic shaping.

all of the hops to the boarder of akamai.com's network is GREAT if it was
a routing issue INTO akamai's network I would see it at another hop
if it was bottle necking there.

[edited comment] it bottle necks a couple hops
inside of akamai.com's own NETWORK. The traffic is already
inside their network when the issue starts occuring.


Here is a traceroute done on October 11 2018 at about 4:42 PM MST
(GMT-0700)

tracert 198.20.200.155

Tracing route to 198.20.200.155 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms router.lan.darkshado.ca [192.168.0.1]
2 9 ms 10 ms 10 ms rc1we-be107-1.ed.shawcable.net [64.59.186.53]
3 12 ms 13 ms 13 ms rc3no-be6.cg.shawcable.net [66.163.64.69]
4 28 ms 28 ms 27 ms rc2wt-be100.wa.shawcable.net [66.163.75.233]
5 50 ms 50 ms 50 ms rc4sj-be60.cl.shawcable.net [66.163.78.74]
6 50 ms 50 ms 50 ms rc3sj-tge0-0-0-0.cl.shawcable.net [66.163.67.81]
7 47 ms 47 ms 47 ms akamai.prolexic.com [206.223.116.136]
8 47 ms 47 ms 46 ms po110.bs-a.sech-sjc.netarch.akamai.com [209.200.184.192]
9 * 252 ms 283 ms ae120.access-a.sech-sjc.netarch.akamai.com [209.200.184.197]
10 * * * Request timed out.
11 74 ms 74 ms 75 ms 198.20.192.4
12 74 ms 74 ms 75 ms 198.20.200.155

Look It goes right from my ISP directly pretty well to akamai.com and
on the outskirts of their network everything is great look at hop 7
which would be the boarder to their network 47 ms, 47 ms, 47 ms (hop 7)

Hop 8 again really good which is one more hop into their network
at akamai.com again 47 ms, 47 ms, 46 ms

Then it hits hop 9 which again is owned by akamai.com again
IP 209.200.184.197 which IS owned by akamai.com

https://whois.arin.net/rest/net/NET-209-200-128-0-1/pft?s=209.200.184.197

And that is were you will see everything go sideways
look at the times on hop 9 252 ms, 283 ms wow.

And you see the same kind of issues on a tcptrace too
This was done again October 11 2018 at about 4:47 PM MST
(GMT-0700)

tracetcp 198.20.200.155

Tracing route to 198.20.200.155 on port 80
Over a maximum of 30 hops.
1 1 ms 2 ms 2 ms 192.168.0.1 [router.lan.darkshado.ca]
2 22 ms 10 ms 12 ms 64.59.186.53 [rc1we-be107-1.ed.shawcable.net]
3 14 ms 15 ms 14 ms 66.163.64.69 [rc3no-be6.cg.shawcable.net]
4 30 ms 29 ms 29 ms 66.163.75.233 [rc2wt-be100.wa.shawcable.net]
5 31 ms 52 ms 50 ms 66.163.78.74 [rc4sj-be60.cl.shawcable.net]
6 51 ms 52 ms 49 ms 66.163.67.81 [rc3sj-tge0-0-0-0.cl.shawcable.net]
7 49 ms 49 ms 50 ms 206.223.116.136 [akamai.prolexic.com]
8 48 ms 50 ms 48 ms 209.200.184.200 [po110.bs-b.sech-sjc.netarch.akamai.com]
9 257 ms 266 ms 263 ms 209.200.184.205 [ae121.access-a.sech-sjc.netarch.akamai.com]
10 * * * Request timed out.
11 * 78 ms 76 ms 198.20.192.4
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.
Trace Complete.

The piece of equipment that is either overloaded with too much traffic
or is failing is OWNED by akamai.com how can this NOT be their issue.
The piece of equipment at 209.200.184.205 belongs to akamai.com

This has absolutely NOTHING to do with a peering dispute or a peering
issue at all. you are way off base buddy!

[edited comment] this is most definately an equipment issue
or traffic issue within Akamai.com's own network it appears to
be about 2 -3 hops inside their OWN network that the issue is occuring.

in fact as I said before it appears that my ISP has a direct route with
akamai.com because traffic goes directly from my ISP directly
to akamai.com.

There are no other hops between my ISP and akamai.com and the times
are great until I get about 2 - 3 hops inside akamai.com's own network!

SirDopey

Similar issues with Telstra having changed its routing recently. When I get home I'll post the tracert I did around this time last year compared to now.

Something else is going on through as previously using Express VPN to route directly to SF would lower ping to around the 240/250 mark but still getting pings in excess of 300 so I have a feeling this could be an NBNco congestion/throttling issue

Justmee

SirDopey wrote: »

Similar issues with Telstra having changed its routing recently. When I get home I'll post the tracert I did around this time last year compared to now.

Something else is going on through as previously using Express VPN to route directly to SF would lower ping to around the 240/250 mark but still getting pings in excess of 300 so I have a feeling this could be an NBNco congestion/throttling issue

well I know in my case that isn't the issue the issue with me is not
traffic shaping / throttling in this case. The traffic goes directly
from MY ISP to akamai.com

And even up to the edge of Akamai.com's network it is fine
when you get about 3 hops in though that is when everything
starts going sideways and the latency starts getting out of hand

The issue I am seeing is within akamai.com's own network.

SirDopey

Is the IP for tracert no longer 198.20.198.110? When did this change?

SirDopey

Two Tracerts that I've just done. Now I can't ping the servers and neither trace finishes but I believe this is part n parcel of the anti-ddos service at work and intended. The important things to note, we can see its around 190ish when it first hits akamai in the US, which from Australia isn't that bad and only about 10 - 15 ms worse than a year ago. YET, in game right now my ping is jumping between 300 - 340 so I can only interrupt that as nearly an extra 100ms occurring inside akamai's network?

traceroute to 198.20.200.155 (198.20.200.155), 64 hops max, 52 byte packets
1 router.asus.com (192.168.1.1) 2.563 ms 2.635 ms 2.060 ms
2 ip-103-85-38-4.syd.xi.com.au (103.85.38.4) 17.714 ms 18.255 ms 17.891 ms
3 ip-103-85-38-1.syd.xi.com.au (103.85.38.1) 18.139 ms 18.485 ms 17.481 ms
4 tengige0-2-0-6-3.ken-edge903.sydney.telstra.net (203.54.131.57) 19.397 ms 19.065 ms 20.043 ms
5 bundle-ether17.ken-core10.sydney.telstra.net (203.50.11.172) 21.185 ms 19.736 ms 19.017 ms
6 bundle-ether1.pad-gw11.sydney.telstra.net (203.50.6.61) 27.686 ms 21.175 ms 19.152 ms
7 bundle-ether1.sydp-core04.sydney.reach.com (203.50.13.90) 20.861 ms 19.922 ms 21.194 ms
8 i-52.tlot-core02.bx.telstraglobal.net (202.84.137.101) 196.069 ms 197.199 ms 196.446 ms
9 i-93.eqla01.telstraglobal.net (202.84.251.194) 197.992 ms 195.957 ms 198.202 ms
10 zayo-peer.eqla01.pr.telstraglobal.net (134.159.63.18) 194.716 ms 195.713 ms 196.726 ms
11 128.177.68.30 (128.177.68.30) 199.721 ms 205.850 ms 199.897 ms
12 po110.bs-b.sech-lax.netarch.akamai.com (23.57.96.245) 196.391 ms
po110.bs-a.sech-lax.netarch.akamai.com (23.57.96.243) 199.019 ms
po110.bs-b.sech-lax.netarch.akamai.com (23.57.96.245) 202.993 ms
13 ae120.access-a.sech-lax.netarch.akamai.com (23.57.96.249) 198.719 ms 201.683 ms
ae121.access-a.sech-lax.netarch.akamai.com (23.57.96.251) 214.950 ms
14 * * *

traceroute to 198.20.198.110 (198.20.198.110), 64 hops max, 52 byte packets
1 router.asus.com (192.168.1.1) 3.195 ms 2.179 ms 2.220 ms
2 ip-103-85-38-4.syd.xi.com.au (103.85.38.4) 17.874 ms 17.364 ms 17.969 ms
3 ip-103-85-38-1.syd.xi.com.au (103.85.38.1) 18.894 ms 18.747 ms 19.408 ms
4 tengige0-2-0-6-3.ken-edge903.sydney.telstra.net (203.54.131.57) 19.659 ms 19.104 ms 19.622 ms
5 bundle-ether17.ken-core10.sydney.telstra.net (203.50.11.172) 19.203 ms 20.032 ms 19.886 ms
6 bundle-ether1.pad-gw11.sydney.telstra.net (203.50.6.61) 19.166 ms 20.896 ms 19.589 ms
7 bundle-ether1.sydp-core04.sydney.reach.com (203.50.13.90) 20.756 ms 70.678 ms 21.327 ms
8 i-52.tlot-core02.bx.telstraglobal.net (202.84.137.101) 197.392 ms 198.431 ms 197.186 ms
9 i-93.eqla01.telstraglobal.net (202.84.251.194) 213.943 ms 196.055 ms 214.040 ms
10 zayo-peer.eqla01.pr.telstraglobal.net (134.159.63.18) 218.590 ms 194.465 ms 196.791 ms
11 128.177.68.30 (128.177.68.30) 203.531 ms 211.797 ms 335.790 ms
12 po110.bs-b.sech-lax.netarch.akamai.com (23.57.96.245) 195.654 ms
po110.bs-a.sech-lax.netarch.akamai.com (23.57.96.243) 198.663 ms
po110.bs-b.sech-lax.netarch.akamai.com (23.57.96.245) 196.015 ms
13 ae120.access-a.sech-lax.netarch.akamai.com (23.57.96.249) 198.553 ms 199.076 ms
ae121.access-a.sech-lax.netarch.akamai.com (23.57.96.251) 196.840 ms
14 * * *
15 * * *

Justmee

SirDopey wrote: »

Two Tracerts that I've just done. Now I can't ping the servers and neither trace finishes but I believe this is part n parcel of the anti-ddos service at work and intended. The important things to note, we can see its around 190ish when it first hits akamai in the US, which from Australia isn't that bad and only about 10 - 15 ms worse than a year ago. YET, in game right now my ping is jumping between 300 - 340 so I can only interrupt that as nearly an extra 100ms occurring inside akamai's network?

<SNIP>

It isn't the anti-ddos service working as intended though what you can do is on
the routers themselves you can set them up to NOT respond to ICMP requests
which is what a lot of companies do to harden security just a bit more.

That is done actually on the router itself so when you are seeing a * come up
instead of a number that is why.

And today I am seeing some record high latency October 13 2018
and I am getting a latency of between 500 - 600 MS today and I am
just in Canada I mean this is just absolutely stupid!

Come on ZOS get this issue resolved!

Justmee

This is being added to my already existing ticket which
is open Elder Scrolls Support Ticket #: 180920-003202

Also posted on the ESO forms as well where there
has been a long discussion about this and other people
seeing the same issue as what I am

https://forums.elderscrollsonline.com/en/discussion/comment/5474257#Comment_5474257

Ok so it is October 13 2018 and all day today it has been
just horrible lag getting a latency at time between
500 ms - to 600 ms at points.

So I opened up Wireshark again and got the IP of the
server i was connecting to.

It looks like I am connected currently to
198.20.200.32 it is October 13 2018 at about 3:47 PM MST

Traceroute was done just about 3:49 PM MST (GMT-0700)

tracert 198.20.200.32

Tracing route to 198.20.200.32 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms router.lan.darkshado.ca [192.168.0.1]
2 10 ms 14 ms 9 ms rc1we-be107-1.ed.shawcable.net [64.59.186.53]
3 14 ms 14 ms 13 ms rc3no-be6.cg.shawcable.net [66.163.64.69]
4 29 ms 28 ms 28 ms rc2wt-be100.wa.shawcable.net [66.163.75.233]
5 29 ms 29 ms 34 ms rc1wt-be18-1.wa.shawcable.net [66.163.64.81]
6 51 ms 51 ms 47 ms rc3sj-be60.cl.shawcable.net [66.163.75.90]
7 47 ms 47 ms 47 ms akamai.prolexic.com [206.223.116.136]
8 48 ms 46 ms 46 ms po110.bs-a.sech-sjc.netarch.akamai.com [209.200.184.192]
9 245 ms 227 ms 251 ms ae120.access-a.sech-sjc.netarch.akamai.com [209.200.184.197]
10 * * * Request timed out.
11 75 ms 74 ms 75 ms 198.20.192.4
12 77 ms 74 ms 77 ms 198.20.200.32

Trace complete.


And here we go again... Hop 7 and which are on the outer
edge of akamai.com's network and they are ok.. it goes one
more hop inside of their network and the times get horrible
245 ms, 227 ms, 251 ms on hop 9 again that is INSIDE of
akamai.com's own network where the latency is coming from!


And here is a tracetcp too same issue being shown there
as well.


tracetcp was done at about 3:55 PM MST (GMT-0700) on Oct 13 2018

tracetcp 198.20.200.32

Tracing route to 198.20.200.32 on port 80
Over a maximum of 30 hops.
1 2 ms 1 ms 2 ms 192.168.0.1 [router.lan.darkshado.ca]
2 13 ms 10 ms 10 ms 64.59.186.53 [rc1we-be107-1.ed.shawcable.net]
3 15 ms 17 ms 16 ms 66.163.64.69 [rc3no-be6.cg.shawcable.net]
4 31 ms 29 ms 30 ms 66.163.75.233 [rc2wt-be100.wa.shawcable.net]
5 31 ms 31 ms 30 ms 66.163.64.81 [rc1wt-be18-1.wa.shawcable.net]
6 51 ms 50 ms 50 ms 66.163.75.90 [rc3sj-be60.cl.shawcable.net]
7 50 ms 51 ms 50 ms 206.223.116.136 [akamai.prolexic.com]
8 50 ms 50 ms 48 ms 209.200.184.200 [po110.bs-b.sech-sjc.netarch.akamai.com]
9 239 ms 293 ms 230 ms 209.200.184.205 [ae121.access-a.sech-sjc.netarch.akamai.com]
10 * * * Request timed out.
11 76 ms * * 198.20.192.4
12 * * * Request timed out.
13 * * * Request timed out.
14 * * * Request timed out.
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.
Trace Complete.


Again look at hop 8 everything is great 50 ms, 50ms, 48ms
then it hits hop 9 again and you are going up to
239 ms, 293 ms, 230 ms

It isn't a routing issue or anything like it looks like shaw.ca
who is my ISP peers directly with akamai.com because it goes right
from Shaw directly to akamai.com,. prolexic.com is akamai.com
as well FYI.

Once again it seems like the issue is within akamai.com's own
network about 3 hops in that is causing the issue

I am seeing the exact same problem as to what I saw on September 20 2018
when I opened this ticket the problem has not been resolved and the high
latency still exists in game. Takes forever loading in to areas or
just getting into the game itself. The game is pretty much unplayable!

Justmee

And just to prove there is no issues with my connection I go to http://www.speedtest.net/
and I run a speed test I am getting 84.60 Mbps download and 8.13 Mbps Upload

my connetion is running perfectly that is pretty fast! No reason for me to get a
lantechy like I am in ESO. The problem isn't here on my end that is fore sure.

You can see the results at http://www.speedtest.net/result/7715359977

The issue lies within akamai.com on that 3rd hop into their network.

HybrisCross

Well got same latency issues with high peaks several times per minute.

My server is Europe and problem don't come by me since i got no lag with others games. There is well-know problems with internet provider but Zenimax not move an inch at all.

Justmee

HybrisCross wrote: »

Well got same latency issues with high peaks several times per minute.

My server is Europe and problem don't come by me since i got no lag with others games. There is well-know problems with internet provider but Zenimax not move an inch at all.

Well do some traceroutes and tcptraces as well like I did above... be intresting to see
on the Eurpoean servers as to what the route is.

I certainaly hope that they are not using akamai.com servers in the USA over on the Eurpoean
servers or at least hopefully they have a seperate anti-ddos servers setup some where over
in Eurpoe some where.

I don't know if it is with their "provider" or not but it appears to be with in my case
akamai.com who provides an anti-ddos type service for ZOS's servers.

akamai.com provides a filtering / scrubbing service basically that can
detect and block all of the unwanted traffic and that appears in my
case where the issue is.

https://stratusly.com/what-is-a-ddos-scrubbing-center/
https://security.radware.com/ddos-knowledge-center/ddospedia/scrubbing-center/

You can not have a latency jumping from like 48 - 50 ms all the way up to
200 - 300 ms and not expect the players to feel those effects.

I mean to protect a website and have that high of a latency you may not notice
other than it taking a second or two more to load but when you are playing an
interactive game like Elder Scrolls and you are constantly interacting with
other players and NPCs in the game itself and loading in and out of different
zones you definately notice it when you get a high latency.

MrGarlic

I submitted a ticket on what i assume is the same problem as I am getting but it took ZOS 6 days to reply to it, and then only after I prodded them to get back to me. All I got was a generic 'check your own system' response.

(Ticket is 181009-004670 if anyone cares to check.)

I'm not sure if it's worth continuing with this game. It's extremely difficult to play when I don't know if anything is going to work or even walk through a door in reasonable time.

Aquatorch

I'd like to get some information on this as well. I'm sitting at a resting ping of 700 at the moment and PingPlotter is telling me that it's getting stuck at akamai.

jcm2606

@ZOS_GinaBruno @ZOS_JessicaFolsom @ZOS_KaiSchober @ZOS_BillE @ZOS_GaryA Can we please get an official statement on this? In order to have the game even be playable, I've had to use Mudfish, which even that results in the occasional ping spike. I shouldn't have to pay a third party VPN service to have your game be playable.