High latency at akamai.com

Kiwi

I cant stay connected to the game for ore than 10 minutes lately
no way its my connection my speed is in the 100s and its only this game, never anything else
I did like this game and when it was going i got great performance on max settings
just these *** servers and the lack of acknowledgement is testament to how little the player base matters to them
theres no excuse for ignoring legitimate issues

Suddwrath

What I don't understand is how there has been zero communication from ZOS on this.

"We're looking into this"
"Thank you for bringing this to our attention. We will let you know if there are any updates"
"Unfortunately, we are not able to publicly discuss this specific issue. However, we would like to reassure you that we are aware of it"

Not a single peep. That's what makes it so strange.

MrGarlic

lordrichter wrote: »

<SNIP>
With Akamai, you will probably get a response. Not immediately, but I bet they do respond. They will probably send you an email back telling you to contact ZOS. Then again, they might collect information from you and add it to a pile of work to be looked at.
<SNIP>

Here is the reply I got from Akamai in October regarding this issue, which I might remind everyone has been ongoing for THREE months now.

We understand your concern. Could you please raise this concern with Akamai's customer whose services your customer is using, they would in turn raise a case with Akamai and have this investigated.

We are unable to accept this request due to compliance policy. As per the policy, we are to take request coming from direct customers of Akamai alone.

If you'd like to investigate more, please bring this up with our customer and they'd take this further with us.

So there you go. Akamai will only discuss this with ZOS, but only if ZOS raises the issue with them.

However as my ZOS support ticket went beyond first-level, all I got back from them was a patronising response that the problem is my ISP. Which is BS frankly as my ISP does not control where international data travels to Akamai's Prolexic scrubbers are located. Not even their DNS server can fix that as Akamai is spoofing ZOS ip addresses to collect data requests in order to direct to the nearest Akamai presence, which for me is in Hong Kong as it is 1000 klms closer than Akamai California.

Only ZOS in consultation with Akamai can fix this problem but they are clearly either unable or unwilling to do anything about it. Not even willing to admit there is a problem to those affected

ZOS is losing customers over this but clearly not enough for them to take any interest or allocate resources into fixing it.

The risk of DDoS attacks and damage that occurs as a consequence not doubt outweighs the loss of a few customers outside North America or Europe

Don't expect his to be resolved any time soon, if at all.

Korax

I think most here don't expect them to share any details... it doesn't take anything away to at least acknowledge that there is an issue here. But from my point of view, reading news posts, reading the forums, reading patch notes etc. It just seems as if ZOS is pretending as if this does not exist at all.

For me lag spikes would be an improvement. I can't get any of my characters loaded AT ALL. Loading a character would be like playing dice. And this has been going on for 3 months already!

Or better said in business terms: No play for the whole of Q4, and what does ZOS do? Go on vacation. Like this issue is not important to get sorted out for a whole of Q4!

How come is it then that ALL other online MMOs I have works just fine??
Sorry to say but this is squarely a ZOS issue from where I stand.

I couldn't play this game since September. And I haven't payed for anything with RLM since, because I'm not buying content that I know I can't access.

Kiwi

yep since i have not been able to stay connected to this game for longer than about 10 mins at a time sine early yesterday ive been playing a couple of other online mmos without any problems at all

24+ hours later, been playing another mmo all day without any problems but i still cant stay connected to eso servers for more than a few minutes

worst part is i bought the new expansion and i cant even play it at all
what a waste of money

Elsonso

SaltySudd wrote: »

What I don't understand is how there has been zero communication from ZOS on this.

"We're looking into this"
"Thank you for bringing this to our attention. We will let you know if there are any updates"
"Unfortunately, we are not able to publicly discuss this specific issue. However, we would like to reassure you that we are aware of it"

Not a single peep. That's what makes it so strange.

Sigh. Yeah. It's not like any of those responses would cause unimaginable pain to the writer.

ZOS confuses the heck out of me on this whole lack of communications thing.

Lelldorano

So I just tried out mudfish VPN to see if it would make any difference. Went from a 300+ ping with horrible spikes to 220-250 and no spikes! This is connecting from NZ on spark fibre. You get some free credit when signing up so don't have to pay anything to try it.

The fact a VPN works like this proves that it isn't our fault, right ?

Elsonso

Lelldorano wrote: »

So I just tried out mudfish VPN to see if it would make any difference. Went from a 300+ ping with horrible spikes to 220-250 and no spikes! This is connecting from NZ on spark fibre. You get some free credit when signing up so don't have to pay anything to try it.

The fact a VPN works like this proves that it isn't our fault, right ?

Where is your VPN coming out? Do you know? (trace route)

SkysOutThizeOut

Bumpity bump bump

Lelldorano

lordrichter wrote: »

Lelldorano wrote: »

So I just tried out mudfish VPN to see if it would make any difference. Went from a 300+ ping with horrible spikes to 220-250 and no spikes! This is connecting from NZ on spark fibre. You get some free credit when signing up so don't have to pay anything to try it.

The fact a VPN works like this proves that it isn't our fault, right ?

Where is your VPN coming out? Do you know? (trace route)

Using a US west node (San Jose). So basically I connect straight to the states, then to ESO, bypassing all the dodgy international akamai servers.

Tracing route to 198.20.200.155 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 10.254.106.1
2 174 ms 174 ms 174 ms 10.254.254.1
3 174 ms 174 ms 174 ms 10.255.242.46
4 * * * Request timed out.
5 196 ms 207 ms 187 ms 45.32.142.65
6 * * * Request timed out.
7 176 ms 178 ms 177 ms xe-0-0-14-2.a00.snjsca04.us.bb.gin.ntt.net [129.250.207.197]
8 175 ms 175 ms 176 ms ae-8.r11.snjsca04.us.ce.gin.ntt.net [165.254.191.154]
9 175 ms 175 ms 175 ms po110.bs-b.sech-sjc.netarch.akamai.com [209.200.184.200]
10 175 ms 175 ms 175 ms po587-10.bs-a.sech-ams.netarch.akamai.com [72.52.1.201]
11 175 ms 175 ms 176 ms ae121.access-a.sech-sjc.netarch.akamai.com [209.200.184.205]
12 * * * Request timed out.
13 214 ms 213 ms 213 ms 198.20.192.3
14 214 ms 213 ms 213 ms 198.20.200.155

Trace complete.

That reflects the ping I get in game when using the VPN.

Kiwi

8 137 ms 139 ms 141 ms akamai.prolexic.com [198.32.176.228]
9 137 ms 138 ms 140 ms po111.bs-b.sech-sjc.netarch.akamai.com [209.200.184.202]
10 * * * Request timed out.
11 138 ms 138 ms 138 ms ae121.access-a.sech-sjc.netarch.akamai.com [209.200.184.205]
12 177 ms 192 ms 175 ms a72-52-29-88.deploy.static.akamaitechnologies.com [72.52.29.88]

...

8 142 ms 139 ms 137 ms akamai.prolexic.com [198.32.176.228]
9 137 ms 139 ms 140 ms po111.bs-b.sech-sjc.netarch.akamai.com [209.200.184.202]
10 * * * Request timed out.
11 138 ms 140 ms 137 ms ae121.access-a.sech-sjc.netarch.akamai.com [209.200.184.205]
12 177 ms 174 ms 175 ms a72-52-29-88.deploy.static.akamaitechnologies.com [72.52.29.88]

hmm ok whats 10?
10 = po587-10.bs-a.sech-ams.netarch.akamai.com [72.52.1.201]

C:\WINDOWS\system32> ping 72.52.1.201

Pinging 72.52.1.201 with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 72.52.1.201:
Packets: Sent = 4, Received = 0, Lost = 4 (100% loss)

the game use to work fine, i spent some money and the very next day it refused to stay connected for more than a couple of minutes and has done so ever since
i have been playing other online games the whole time without any problems, gw2, poe, fo76 etc ZERO problems
its ONLY ESO servers, nothing else

Lelldorano

Just to compare here is a trace without the VPN.

Tracing route to 198.20.200.155 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 192.168.1.254
2 2 ms 1 ms 1 ms 125-239-247-1-adsl.sparkbb.co.nz [125.239.247.1]
3 * * * Request timed out.
4 10 ms 11 ms 10 ms ae8-10.akbr6.global-gateway.net.nz [122.56.116.5]
5 11 ms 10 ms 10 ms ae7-2.akbr7.global-gateway.net.nz [122.56.119.53]
6 32 ms 32 ms 32 ms xe5-0-3.sgbr3.global-gateway.net.nz [122.56.127.182]
7 36 ms 36 ms 35 ms ae7-10.sebr4.global-gateway.net.nz [122.56.127.214]
8 39 ms 38 ms 38 ms i-0-11-0-0.sydp01.bi.telstraglobal.net [202.84.220.210]
9 37 ms 38 ms 38 ms i-0-1-0-6.sydp-core03.bi.telstraglobal.net [202.84.220.209]
10 151 ms 151 ms 151 ms 202.84.138.45
11 151 ms 151 ms 151 ms 202.84.138.141
12 175 ms 175 ms 175 ms unknown.telstraglobal.net [202.127.78.186]
13 170 ms 170 ms 170 ms po110.bs-a.sech-hkg2.netarch.akamai.com [72.52.2.176]
14 174 ms 174 ms 174 ms po561-10.bs-a.sech-ams.netarch.akamai.com [72.52.1.149]
15 175 ms 175 ms 175 ms ae120.access-a.sech-hkg2.netarch.akamai.com [72.52.2.181]
16 224 ms 224 ms 241 ms 93.191.173.93
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.

Trace complete.

Once it hits the Netherlands Akamai server (93.191.173.93) ... bork.

Aquatorch

Has Akamai addressed this issue yet for anyone here?

MrGarlic

Aquatorch wrote: »

Has Akamai addressed this issue yet for anyone here?

Sadly, no.

SirAndy

Aquatorch wrote: »

Has Akamai addressed this issue yet for anyone here?

Akamai has clearly stated that they won't investigate unless that request comes from their customer, which in our case is ZOS.

So no ...

Elsonso

Lelldorano wrote: »

Tracing route to 198.20.200.155 over a maximum of 30 hops

1 <1 ms <1 ms <1 ms 10.254.106.1
2 174 ms 174 ms 174 ms 10.254.254.1
3 174 ms 174 ms 174 ms 10.255.242.46
4 * * * Request timed out.
5 196 ms 207 ms 187 ms 45.32.142.65
6 * * * Request timed out.
7 176 ms 178 ms 177 ms xe-0-0-14-2.a00.snjsca04.us.bb.gin.ntt.net [129.250.207.197]
8 175 ms 175 ms 176 ms ae-8.r11.snjsca04.us.ce.gin.ntt.net [165.254.191.154]
9 175 ms 175 ms 175 ms po110.bs-b.sech-sjc.netarch.akamai.com [209.200.184.200]
10 175 ms 175 ms 175 ms po587-10.bs-a.sech-ams.netarch.akamai.com [72.52.1.201]
11 175 ms 175 ms 176 ms ae121.access-a.sech-sjc.netarch.akamai.com [209.200.184.205]
12 * * * Request timed out.
13 214 ms 213 ms 213 ms 198.20.192.3
14 214 ms 213 ms 213 ms 198.20.200.155

Trace complete.

I just wanted to caution that the ping times are not accurate in that stack trace. The route is accurate, but the times are not.


Edit: Also want to point out that your before and after traces indicate (to me) that Akamai is dealing with a lot of malicious traffic in Hong Kong and that your VPN, by bypassing that, avoids whatever is going on there. Basically, it is the internet equivalent of going around an accident on a highway.

Gelmir

Simply put folks, its about contracts ZOS made a contract with Scrubber service provider, potentially at least for a year. And paid loads of money for it. They would cancel/bin it only if the damages caused by this service outweighs the cost of the contract. So I'd say, nothing will be fixed within 12 months since this dDOS protection was implemented.

Alinhbo_Tyaka

Gelmir wrote: »

Simply put folks, its about contracts ZOS made a contract with Scrubber service provider, potentially at least for a year. And paid loads of money for it. They would cancel/bin it only if the damages caused by this service outweighs the cost of the contract. So I'd say, nothing will be fixed within 12 months since this dDOS protection was implemented.

Any contract should have some service level agreements as well and it is up to the contractor to meet or exceed these requirements. However it is also usual that the customer has to drive this process so unless ZOS takes the lead and pushes Akamai to fix their issues nothing will change.

chaz

Zenimax! Are we still doing this?

It was ok for 3 weeks then it went back to this ess aich eye tea!

Arunei

I'm not having any issues like this, seeing as I'm in the US, but man that whole thing about better communication from Gina sure is holding up, ain't it? The Undaunted Event has to come and go before any real comment is made about the borked LFG tool, massive issues in Cyrodiil go unacknowledged, numerous threads all over the forums completely ignored...yup, reeeeaaaal good on the communication there, ZOS. A+ for effort.

I'm not one to shame people for spending money how they want, but the sad thing is the people who blow obscene amounts of money in the CS are likely the people making ZOS the most money, so even if droves of people cancel ESO+ and stop buying the occasional Crown packs, it's likely not hurting their bottom line enough to force them into actually putting some of the revenue they're getting back into the game. And, by extension, not hurting them enough to bother with any sort of communication to people being impacted by issues. The sad truth is a majority of players don't even know these forums exist most likely, let alone use them, so even though is this an 8-page topic, it's also taken several months for it to reach that number. In ZOS' eyes that probably means such a tiny percent of players seem to be having a problem so there must be no problem, even if hundreds or even thousands of NZ/AU players are actually experiencing this problem.

EDIT: Not to say that I'm excusing ZOS for their lack of communication or saying any of this is okay or the fault of the players. My point is that ZOS clearly doesn't care since in their eyes it's probably not really an issue. It's frustrating for me personally to see these threads getting no response from anyone at ZOS (aside from the 'we removed comments for baiting please keep discussion civil' posts from mods), even if I'm not having the problems spoken about, because you'd think they'd want to keep the people essentially paying their checks happy.

Elsonso

.

chaz

OMG The level of my frustration right now is beyond borderline blood pressure. This weekend, with every newbie spank that came out of the closet for the pvp midyear absolutely ruined my gaming experience. I could not stay logged in for any moment beyond 2 minutes, the ping has been a constant 999+ in red.

getting booted, getting kicked, having screen frozen and im the only one moving, getting into a good battle, then BAMMMMMMMMMMM (999+) ping, can't bar swap, can't cast skills, can't dodge roll can't block, then............screen goes still and im the only one moving, then,,,,,,meet the log in screen!

Livid, absoltely just livid! I am so tired of this. Here it is midnight and I'm thinking ok it's my time, and usually everyone logs off, 1 bar ep, 1 bar dc, 2 bards ad. NOPE! All alliances have been poplocked all day and night 24 hours a day unrelentlessly!

UNRELENTLESSLY!

So now cyrodiil, especially in the vivec campaign has every newbie pve player wearing merchant purchased gear running in cyrodiil thinking they're getting something special. They're not. None of are. Dude, don't even sway that way with me, I stayed in cyrodiil all day and night and the next day I had my other toon unpack all those silly boxes and nothing good comes from them. Nothing, junk, garbage. What kind of event is this?

Just marketing ploys to get more people to play, which like year after year, when the event end, we never see these people again. I attend these things every year, except this is the first year I'm not enjoying myself. I'm not entertained by this game. In fact, I should see a doctor because it's causing serious health issues and high blood pressure.

I would not be surprised after they find my dead body in a few weeks, the head stone will read:

Here lies Chaz. Just some poor azzhat that
died of aneurysm in head while playing ESO.

16BC to 2019AD

Funny huh. Listen, one server is clearly not enough. By the time you people ""QUOTE"", realize this and place another server in the oceanic area , I , meaning my body, will long have been turned to bug food and dust, to which at that point my immortal soul won't give a crap anymore.

I can ask you to fix this all day until my windpipe cracks, but we all know that's not going to happen and I just have to wait until your event ends for everyone else to stop playing like they do every year, then I can go back to normal trying to get around this stupid akamai so I can at least play.

It's funny, I have lived all this time on an Island while playing this game perfectly without turns of tide. Then all of a sudden September 2018, where absolutely nothing on my end changed, where something started to go wrong. So tell me, if I have been where I have been, and been reaching your server on the mainland all these years with no trouble, and loved this game so much I poured thousands of dollars into it, how is it that you still believe it's my fault that your server doesn't work anymore for people OVERSEAS???

Korax

I'm tired of TRYING to play this game. This is the 4th month I can't play this game reliably. I can only get a character to load once every like 100 tries and then only for a little while. No thinks, but this is not an online game.

Maybe ZOS don't care because I'm just a dumb **** here in South Africa, but by the 1st February I'm uninstalling this steaming pile of **** for good. I've already lost my guilds, lost the plot of where I actually am quest-wize with all my alts. Enough time has passed since September not to really miss this game and its issues.

If Blizzard games work just fine without issues, I'm sorry to inform you, ZOS, its your junk servers/whatever... not my router, isp or country etc.

Suddwrath

@ZOS_GinaBruno @ZOS_JessicaFolsom still no response to this? Really? How can an issue like this go on for several months without any acknowledgement?

Ulfson

Hello,
First, I'm no expert on the network topology and I don't like the game lag. This is IMHO only.
But this is a hot post and it seems like the pings are misunderstood. Pings are a one time measurement of a special TCP message trip through the network. I have noticed that pings like speed tests can fly through the net and provide no indication that the net is slow. I am told that is because lost packets are a bigger concern and they happen due to arriving too late to be valid. You may have a wonderful ping and loosing 5 in 15 packets which makes your message rate go down and that means lag goes up. Messages may be selectively delayed due to network priorities set by unknown people .

Lag is caused by information not arriving in time.

Pings do not measure successful information rates. At best they may be a theoretical maximum.

I doubt this is a ZOS problem. It's like my complaining to Subaru that Chevron is selling me junk gas. Or me complaining my gas mileage is really low using Mityfine Gas. You may believe that ZOS should be able to buy their way out of this but that may not be true. It could be too expensive. Who do you think will get priority on the net, NFL or ZOS?

I enclosed my tracert from tonight when i was in Cyrodiil during the winter mayhem extravaganza and huge lag.



Happy Trails

Korax

So how is it then that its only ZOS' MMO that doesn't work?

Seri

Arunei wrote: »

The sad truth is a majority of players don't even know these forums exist most likely, let alone use them, so even though is this an 8-page topic, it's also taken several months for it to reach that number. In ZOS' eyes that probably means such a tiny percent of players seem to be having a problem so there must be no problem, even if hundreds or even thousands of NZ/AU players are actually experiencing this problem.

The sadder truth is that the Oceanic latency issue is a routing issue - as best I can tell _all_ of the ISPs (at least in Australia) are now being shunted up into Asia before heading over to the USA - so it's not a per user issue. If ZOS believes that it's a small percentage of Oceanic players having the problem, that's awfully short-sighted of them.

Ulfson wrote: »

First, I'm no expert on the network topology and I don't like the game lag. This is IMHO only.
But this is a hot post and it seems like the pings are misunderstood. Pings are a one time measurement of a special TCP message trip through the network. I have noticed that pings like speed tests can fly through the net and provide no indication that the net is slow. I am told that is because lost packets are a bigger concern and they happen due to arriving too late to be valid. You may have a wonderful ping and loosing 5 in 15 packets which makes your message rate go down and that means lag goes up. Messages may be selectively delayed due to network priorities set by unknown people .

Lag is caused by information not arriving in time.

Pings do not measure successful information rates. At best they may be a theoretical maximum.

I doubt this is a ZOS problem. It's like my complaining to Subaru that Chevron is selling me junk gas. Or me complaining my gas mileage is really low using Mityfine Gas. You may believe that ZOS should be able to buy their way out of this but that may not be true. It could be too expensive. Who do you think will get priority on the net, NFL or ZOS?

This is actually why the net neutrality laws in the USA are important - all traffic should be considered equal and ISPs should not prioritise NFL than ZOS because the NFL will pay them to do so.

Ultimately there's three sources of 'lag' in play here. What you mention is about the bandwidth/congestion of the links between you and the servers. This is where packet loss occurs along the path (and in-game you get the random 999+ spurts, regardless of your in-game location). Sometimes this is your ISP, sometimes it is further along outside the ISP's control.

A second cause is the servers not keeping up - this is often what you see in Cyrodiil when things are sluggish, etc.

In both cases, if your game falls behind by any given amount, you usually get booted to the login screen. This is more noticable in Cyro as more is going on.

The third cause of lag is the route - which is what majority of this and the other Oceanic/VPN thread are about - traffic is heading a lot further than it needs to go, and ultimately additional distance = additional time. In this instance it's Akamai that is causing ESO traffic from the Asia-Pacific region to go to the HK Akamai node. Akamai says that their direct customer (ie, ZOS) needs to bring it up with them. It's been pointed out elsewhere that apparently Akamai have an Australian DDOS scrubbing presence so who knows why that isn't being used. Alternately, Australian and NZ players could be routed to a USA based scrubber rather than up into Asia.

Defilted

Net Neutrality is nothing but government over reach in an attempt to control the content of internet sites. The Net neutrality law would be used just like the commerce clause to regulate things way beyond its mandate.

This was defeated in the congress of the USA and we(USA) are all better for it.

tsaescishoeshiner

Seri wrote: »

Ultimately there's three sources of 'lag' in play here. What you mention is about the bandwidth/congestion of the links between you and the servers. This is where packet loss occurs along the path (and in-game you get the random 999+ spurts, regardless of your in-game location). Sometimes this is your ISP, sometimes it is further along outside the ISP's control.

Is there any way to check whether this is the case? For three weeks, I've had a support ticket open over this issue (PC NA, in Northern California) and don't want to be blaming the nice support people at ZOS for what might be my ISP's (Comcast's) issue. Support told me to contact my ISP with the message:

10 --- 1/ 1 =100% 0/ 1 = 0% 209.200.184.194

But all Comcast support can do is send someone to check our internet box (which appears to be working fine). Running a trace to that IP address above results in "response timed out" for every Akamai hop. When my ESO connection is stalling, other internet connections work fine. Issue started out of nowhere 3 weeks ago, worked fine before.

Point is, is there a way to check whether this issue is on my ISP's side or ZoS'? Have other people successfully fixed this by using a VPN? Thanks

Korax

I think just reading this thread with multiple different responders from different parts of the globe kinda tells you where the problem resides... its not your ISP for sure.

A non-response from ZOS, is in-fact also a response.