High latency at akamai.com

bufnut

this started yesterday


Tracing route to 198.20.200.26 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms NF18ACV.Home []
2 12 ms 11 ms 10 ms loop1671791800.bng1.bne.aussiebb.net []
3 11 ms 11 ms 11 ms Bundle-Ether24.cha-edge902.brisbane.telstra.net [110.145.204.109]
4 11 ms 11 ms 11 ms bundle-ether2.woo-edge902.brisbane.telstra.net [203.50.11.147]
5 12 ms 12 ms 12 ms bundle-ether6.woo-core1.brisbane.telstra.net [203.50.11.138]
6 22 ms 23 ms 23 ms bundle-ether20.chw-core10.sydney.telstra.net [203.50.11.180]
7 35 ms 35 ms 36 ms bundle-ether8.exi-core10.melbourne.telstra.net [203.50.11.125]
8 49 ms 48 ms 57 ms bundle-ether2.way-core10.adelaide.telstra.net [203.50.6.230]
9 75 ms 75 ms 74 ms bundle-ether3.pie-core10.perth.telstra.net [203.50.6.235]
10 74 ms 75 ms 75 ms tengige0-0-1-0.pthp-core01.perth.net.reach.com [203.50.13.250]
11 124 ms 123 ms 123 ms 202.84.140.38
12 122 ms 123 ms 123 ms 202.84.140.165
13 149 ms 148 ms 147 ms 202.84.136.174
14 148 ms 148 ms 148 ms i-0-0-0-3.hkmi02.telstraglobal.net [202.84.138.137]
15 149 ms 148 ms 148 ms unknown.telstraglobal.net [202.127.78.186]
16 149 ms 149 ms 149 ms po110.bs-a.sech-hkg2.netarch.akamai.com [72.52.2.176]
17 157 ms 149 ms 149 ms ae120.access-a.sech-hkg2.netarch.akamai.com [72.52.2.181]
18 * * * Request timed out.
19 301 ms 301 ms 300 ms a72-52-29-88.deploy.static.akamaitechnologies.com [72.52.29.88]
20 301 ms 301 ms 302 ms 198.20.192.4
21 301 ms 300 ms 301 ms 198.20.200.26

Trace complete.

done trace 2 days ago it was fine went from my place to brisbane sydney then across the pacifc, talked to aussie broadband lastnight about why its going to perth then asia they told that me its not their end and they cant do anything thats just the way the game is routing me so now i got ping in the high 350+

MrGarlic

I've got this as well, but my domicile is NZ, not Aus. My ping is much higher. 500ms with spikes longer than 15 seconds.

The problem appears to be Akamai routing through Europe.

I contacted Akamai to ask why but they told me they can't discuss it with me as I am not their customer. It's up to ZOS to contact them. I upgraded my outstanding ticket to inform ZOS of this.

TumlinTheJolly

I'm in NZ. My average ping is usually stable at around 217. It is currently around 301, with regular spikes to around 500, 700 or 999+.
My brother has had a similar increase (he lives in a different city).

I'm assuming this is the same issue.

Justmee

jcm2606 wrote: »

@ZOS_GinaBruno @ZOS_JessicaFolsom @ZOS_KaiSchober @ZOS_BillE @ZOS_GaryA Can we please get an official statement on this? In order to have the game even be playable, I've had to use Mudfish, which even that results in the occasional ping spike. I shouldn't have to pay a third party VPN service to have your game be playable.

Hell I am the one that opened up the ticket and I can't even get ZOS to give me
an answer.. The ticket is still open and the last response I got from ZOS was
them asking me for more traceroutes which I gave them and other than that
ZOS won't even provide the person that opened the ticket a response.

This is absolute BS! ZOS is an absolutely horrible company!

Justmee

MrGarlic wrote: »

I submitted a ticket on what i assume is the same problem as I am getting but it took ZOS 6 days to reply to it, and then only after I prodded them to get back to me. All I got was a generic 'check your own system' response.

(Ticket is 181009-004670 if anyone cares to check.)

I'm not sure if it's worth continuing with this game. It's extremely difficult to play when I don't know if anything is going to work or even walk through a door in reasonable time.

My ticket has been open since about September 20 2018 and still NO answer
Today is October 17 2018 so we are coming up on a month with no answer to the
problem or anything done to correct the problem.

ZOS support claims that they have sent it to an "Esclation Specialist"
at this point and I have not actually gotten a response back on my
ticket ZOS is absolutely horrible!

I would never recommend any one else play this game with the
way Zenimax is running this game absolutely horrible!

I should have had a response weeks ago on this ticket and still nothing
and as I said I am coming up on my ticket being open for a month
now! Elder Scrolls Support Ticket #: 180920-003202

silfy

I am from NZ aswell with my regular ping around 200 now its at 303+ sometime it spike between 500 too 999+ aswell. looks like its time to quit eso

DRTE

Same issue for me in NZ was playing fine a few days ago now i'm getting massive ping spikes of 14000ms and a constant of 320-350ms. It is akamai as i did a test with and without a VPN everytime it hits their servers it routes the traffic to the Netherlands then back to NA.

Without VPN.


With VPN.

MrGarlic

nissan_drifter_07rwb17_ESO wrote: »

Same issue for me in NZ was playing fine a few days ago now i'm getting massive ping spikes of 14000ms and a constant of 320-350ms. It is akamai as i did a test with and without a VPN everytime it hits their servers it routes the traffic to the Netherlands then back to NA.

IMAGES SNIPPED

My own testing agrees with you.

lurkuld

Im also from New Zealand and im also getting this issue, along with the odd Netherlands ip in tracert.

Perwulf

bufnut wrote: »

this started yesterday

Also happened yesterday before maintenance, guess servers finally gave up in life.

Playing from Asia.

Seri

nissan_drifter_07rwb17_ESO wrote: »

Same issue for me in NZ was playing fine a few days ago now i'm getting massive ping spikes of 14000ms and a constant of 320-350ms. It is akamai as i did a test with and without a VPN everytime it hits their servers it routes the traffic to the Netherlands then back to NA.

The abnormal IP there must be just 'occasional' as the subsequent ping times don't show the latency (but would explain the occasional massive spikes you see in game). The bigger issue is that, for whatever reason, the akamai node in use for the 'DDOS' protection is in Hong Kong. That's a huge backwards detour when travelling from NZ to USA.

A substantial portion of Aus ISPs routed to up there as well for a year, excluding anyone that used Telstra Wholesale / Telstra Global for transit - now it seems even they're out of luck.

Justmee

October 18 2018,

Well that is it for me I am done I would not say it has been fun because it hasn't
I have quit the game after close to a month with NO response to my ticket I asked
for a full refund of my purchase price and my subscription and it was given
back to me.

I absolutely would NEVER recommend that anyone play Elder Scrolls Online (ESO)

Zenimax is doing an absolutely horrible job os answering tickets and after close
to a month of having my ticket open and still not getting a response on this issue
I am sick of it.

Zennimax did absolutely nothing to resolve the isuse it wasn't like it was a day
or two here either it was actually close to a month that my ticket
was open and I got no responses on the ticket and no actual resolution of the
issue.

What an absolutely horrible company!

DRTE

@ZOS_BillE is anyone looking into this you seem to be the only one active on the forums at this point.

ZOS_Bill

@DRTE

There are a variety of reasons for possible latency jumps in traceroutes depending on location, connection type, and the connection route being taken. To help us investigate this further please make sure to submit a ticket with customer support that includes your traceroute information. In addition, make sure to also detail any troubleshooting you've done to help improve your latency.

Justmee

ZOS_BillE wrote: »

@DRTE

There are a variety of reasons for possible latency jumps in traceroutes depending on location, connection type, and the connection route being taken. To help us investigate this further please make sure to submit a ticket with customer support that includes your traceroute information. In addition, make sure to also detail any troubleshooting you've done to help improve your latency.

Your so called answer is absolute BS First off I am a trained network admin and I am the one that started
this thread.

The latency in al lthe traceroutes I have provided Zenimax shows exactly
where the issue is on akamai.com

Now this isn't an issue with another hop outside of akamai.com but rather about
2 - 3 hops inside their network.

Your answer is nothing more than a generalized BS answer how about you
try answering my question as to why I am getting this latency from
within akamai.com who you use for your anti-ddos service to protect your servers.

Because that is where the lag is coming from my my case there would be
no reason for akamai.com to have this kind of high latency 2 - 3 hops
within their own network with the infastructure that they have.

And yes I have a ticket open about this and have had a ticket open
about this issue since September 20 2018 and it is now close to a month
(today is October 19 2019) and no one could ever address the issue
or even give me a reply to the issue.

Elder Scrolls Support Ticket #: 180920-003202

Why don't you actually try and find out what is going on there instead
of just providing the people here a generalized BS type answer that
doesn't address the issue that I had reported in that ticket above
and which other players are also seeing.

I have provided you with at least 3 different sets of traceroutes (ICMP)
as well as tcptraces (tcp/ip) and both sets show the same problem
in the same location every time.

Also there isn't much that can be done to "improve your latency" either
Zos_Bille because the issue is not within their network but rather about
2 - 3 hops within akamai.com's network where it constantly bottlenecks.

again you use them as an anti-ddos service to protect your servers
and I understand that and they scrub the packets before it gets to
your servers to that they can filter whatever unwanted traffic at
akamai.com before it hits your servers IE: DDOS attack or other
malicious packets.

But it is absolutely no good if the DDOS service which you are using is
actually going to cause a higher latency and lag in game espically not
on a live game like this where you are interacting with other players and
as well as NPCs in the game. If it was just a website you may not feel it
as much but in a live game like this you definately do.

Who is your supervisor ZOS_BillE because I would really like this issue
esclated to her / him at this point.

No one has answered my question and yes I have been through all of the
troubleshooting steps already with your support and it has been
determined already that this issue is NOT on my end and was esclated
to a "esclation speciliast" which was one of the last responses I got
and then no one has resolved the issue since then.

The problem is still going on and no one at Zenimax seems to want
to address the issue.

MrGarlic

Justmee wrote: »

No one has answered my question and yes I have been through all of the
troubleshooting steps already with your support and it has been
determined already that this issue is NOT on my end and was esclated
to a "esclation speciliast" which was one of the last responses I got
and then no one has resolved the issue since then.

The problem is still going on and no one at Zenimax seems to want
to address the issue.

My ZOS ticket has also been 'moved to a specialised team.'

I contacted Akamai but they were quite specific that they only communicated with their clients, not affected users, about issues with traffic.

Ticket 181009-004670

tuwtle

I also have been unable to do everything I have wanted to in this event because of abysmal latency from NZ. I am not sure how useful traceroutes through akamai are though. Here is one from a VPS in dallas

96.44.144.17 US, Texas
128.177.133.253 US, Texas
64.125.26.62 US
64.125.29.21 US
64.125.27.39 US
128.177.68.30 US, California
23.57.96.243 NL
23.57.96.245 NL
23.57.96.251 NL
93.191.173.113 NL
... * * * a lot ...

Seri

MrGarlic wrote: »

Justmee wrote: »

No one has answered my question and yes I have been through all of the
troubleshooting steps already with your support and it has been
determined already that this issue is NOT on my end and was esclated
to a "esclation speciliast" which was one of the last responses I got
and then no one has resolved the issue since then.

The problem is still going on and no one at Zenimax seems to want
to address the issue.

My ZOS ticket has also been 'moved to a specialised team.'

I contacted Akamai but they were quite specific that they only communicated with their clients, not affected users, about issues with traffic.

Ticket 181009-004670

I submitted a ticket in October last year and it was closed with "As it stands, we do not have the capability to alter Akamai's pathing. "

Perhaps I wasn't forceful enough, but it'd be nice it could be fixed for all the Aus/NZ people having their traffic being sent up to Asia. Incidentally, since that ticket, more ISPs (including one I listed as unaffected) are now caught by it, increasing latency.

# 171006-000565 if it matters.

Vapirko

Life’s just a lot better when you just unsub from this game and make it a once in a while thing, especially if you’re PvP focused.

Justmee

Seri wrote: »

MrGarlic wrote: »

Justmee wrote: »

No one has answered my question and yes I have been through all of the
troubleshooting steps already with your support and it has been
determined already that this issue is NOT on my end and was esclated
to a "esclation speciliast" which was one of the last responses I got
and then no one has resolved the issue since then.

The problem is still going on and no one at Zenimax seems to want
to address the issue.

My ZOS ticket has also been 'moved to a specialised team.'

I contacted Akamai but they were quite specific that they only communicated with their clients, not affected users, about issues with traffic.

Ticket 181009-004670

I submitted a ticket in October last year and it was closed with "As it stands, we do not have the capability to alter Akamai's pathing. "

Perhaps I wasn't forceful enough, but it'd be nice it could be fixed for all the Aus/NZ people having their traffic being sent up to Asia. Incidentally, since that ticket, more ISPs (including one I listed as unaffected) are now caught by it, increasing latency.

# 171006-000565 if it matters.

Reopen the ticket and tell them that they are full of crap because akamai.com
is responsible for properly maintaining their network and ZOS is their customer
Zennimax is using akamai.com as a anti-ddos service and if akamai.com
can't maintain their servers properly then maybe it is time that Zennimax find
a anti-ddos provider who can properly maintain their services.

Their "pathing" that is kinda funny because the issue is about 2 to 3 hops
inside of Akamai.com's own network where I am seeing the issue. And if
it is Akamai.com's own network then they have the responsibility
to maintain their equipment. It isn't like it ia routing issue in my case to
akamai.com but it is inside of akamai.com's own network.

The support agent you got is a complete moron and has no clue about
networking and is just a low level grunt with very little education when
it comes to computer networking. People like that fustrate the hell
out of me.

There is no way that on a game like this you can such a high latency and not
expect the players to feel the effects of it. When you are on a live game
like this where there is interaction between players and NPCs in game
of course people are going to complain.

Open up your ticket again and tell them that you want a manager immediately

Just be sure to give them all the traceroutes (ICMP) and tcptraces (tcp/ip)
to backup what you are saying otherwise ya they will just close off your
ticket.

TumlinTheJolly

The game remains unplayable from New Zealand. If this is not addressed soon you will lose your NZ customers.

StackonClown

TumlinTheJolly wrote: »

The game remains unplayable from New Zealand. If this is not addressed soon you will lose your NZ customers.

and most if not all of the australian customers..

MrGarlic

Interestingly, my route has changed overnight and now no longer heads to Sydney and onwards to Hong Kong but rather, heads straight across the Pacific.

Only downside is that I have 100ms added to my ping and still getting the occasional ping spikes (+999)

I tested this on my tethered smartphone (Vodafone) and it followed the same route as my ISP, which is BigPipe (via Spark infrastructure.)

Someone, somewhere is messing with the internet.

EDIT: My average ping to PC/NA is now ~350ms, whereas it used to be ~250ms. (Ping times on PC/EU were in the order of 500-600ms with many spikes into seconds, an increase of about 200ms. I can manage without PC/EU though.)

It's better today, but it's still faulty.

TumlinTheJolly

Definitely an Akamai issue then... Not our ISPs.

Seri

TumlinTheJolly wrote: »

I just solved the problem by downloading Mudfish. My ping is now back to a stable 217 when Mudfish is running, and over 300 when it is not. Is the problem definitely with Akamai? Or could certain ISPs (I'm with Spark) be choosing cheaper routes? I don't have good enough knowledge of how routing works to figure that out myself. Anyway, for anybody else who is trying to solve the problem (I'M LOOKING AT YOU ZOS), I hope this information is helpful.

IMO it's an Akamai issue - if it was ISP, I wouldn't expect the every single ISP in Australia to be routing up to HK, including every major ISP over here. Telstra has multiple links over to NA but Akamai are advertising to our part of the planet that we need to go to HK.

idk

MrGarlic wrote: »

Justmee wrote: »

No one has answered my question and yes I have been through all of the
troubleshooting steps already with your support and it has been
determined already that this issue is NOT on my end and was esclated
to a "esclation speciliast" which was one of the last responses I got
and then no one has resolved the issue since then.

The problem is still going on and no one at Zenimax seems to want
to address the issue.

My ZOS ticket has also been 'moved to a specialised team.'

I contacted Akamai but they were quite specific that they only communicated with their clients, not affected users, about issues with traffic.

Ticket 181009-004670

@ZOS_BillE

You say to open a ticket on the issue but if this is the response then what is the point?

At some point Zos needs to get serious about getting with Akamai to get a solution. Closing tickets, especially with this reason, does not seem to be taking the situation seriously.

Here is another person who opened a ticket about this issue and Zos closed the ticket as well after Zos blamed Akamai. Hate to say it BillE, it does not look like Zos is doing much with those tickets.

https://forums.elderscrollsonline.com/en/discussion/441599/latency-ticket-got-closed#latest

TumlinTheJolly

Are we allowed to use software to route our own connection to ZOS, or does this go against terms and conditions?

MrGarlic

TumlinTheJolly wrote: »

Are we allowed to use software to route our own connection to ZOS, or does this go against terms and conditions?

Open a command line in windows (right click on windows icon and select RUN, then type CMD in the run box. Then type TRACERT 198.100.198.20 into the command line box and hit enter.

That should give you what you need.

Should look something like this

TumlinTheJolly

DRTE

MrGarlic wrote: »

TumlinTheJolly wrote: »

Are we allowed to use software to route our own connection to ZOS, or does this go against terms and conditions?

Open a command line in windows (right click on windows icon and select RUN, then type CMD in the run box. Then type TRACERT 198.100.198.20 into the command line box and hit enter.

That should give you what you need.

Should look something like this

Wrong way around should be 198.20.198.110