Horrible lag, and here's why ...

SirAndy · October 2017

tracert 198.20.200.23

Tracing route to 198.20.200.23 over a maximum of 30 hops

  1     Removed to protect the innocent
  2     Removed to protect the innocent
  3     Removed to protect the innocent
  4     Removed to protect the innocent
  5     Removed to protect the innocent
  6    38 ms    22 ms    23 ms  12.83.39.193
  7    26 ms    23 ms    23 ms  gar23.sffca.ip.att.net [12.122.114.5]
  8    24 ms    24 ms    38 ms  192.205.32.222
  9    26 ms    35 ms    25 ms  akamai-ic-317094-sjo-b21.c.telia.net [62.115.146.117]
 10    24 ms    30 ms    24 ms  a209-200-184-200.deploy.static.akamaitechnologies.com [209.200.184.200]
 11    25 ms    25 ms    27 ms  a209-200-184-205.deploy.static.akamaitechnologies.com [209.200.184.205]
 12     *        *        *     Request timed out.
 13    78 ms    61 ms    61 ms  198.20.192.4
 14    62 ms    76 ms    62 ms  198.20.200.23

Trace complete.

Cleanup at hop 12 please ...

SirAndy · October 2017

Still happening for me ...

phairdon · October 2017

Getting rolled back and disconnected.

Elsonso · October 2017

I am surprised you got results past the Akamai (Prolexic) network.

Alexandrious · October 2017

Get battleping/wtfast.

Problem solved.

A good dedicated Nic or a mobo with good onboard helps too.

kek.

Left4Daud · October 2017

That is similar to my tracert results as well. Timing out after the a209-200-144-197.deploy.static.akamaitechnologies.com node. No idea why, how to fix it, or who to contact to fix it. Tried ZOS support and my ISP and they both say talk to the other party.

From another thread on the subject:

xRIVALENx wrote: »

If you happen to be routing across a209-200-144-205.deploy.static.akamaitechnologies.com you may see some latency spikes. That node along the route seems to be all over the place in terms of latency.

From one of my many support tickets:

"Looking at your report, it looks like you keep losing data at one of the nodes 209.200.144.197."

From my tracert:

13 44 ms 47 ms 56 ms a209-200-144-192.deploy.static.akamaitechnologies.com [209.200.144.192]
14 48 ms 47 ms 49 ms a209-200-144-197.deploy.static.akamaitechnologies.com [209.200.144.197]
15 * * * Request timed out.
16 * * * Request timed out.
17 * * * Request timed out.
18 * * * Request timed out.
19 * * * Request timed out.
20 * * * Request timed out.
21 * * * Request timed out.
22 * * * Request timed out.
23 * * * Request timed out.
24 * * * Request timed out.
25 * * * Request timed out.
26 * * * Request timed out.
27 * * * Request timed out.
28 * * * Request timed out.
29 * * * Request timed out.
30 * * * Request timed out.

Trace complete.

Mureel · October 2017

Same issue here.

SirAndy · October 2017

@ZOS_GinaBruno
On the off chance that you are working today, can you summon your minions and have them talk to your hosting company?

It seems there is a router close to your NA server(s) that is about to burn up ...

Zagnut123Zagnut123 · October 2017

Is this a EU issue

Elsonso · October 2017

SirAndy wrote: »

@ZOS_GinaBruno
On the off chance that you are working today, can you summon your minions and have them talk to your hosting company?

It seems there is a router close to your NA server(s) that is about to burn up ...

Do you run this trace route enough to be able to tell that there is something abnormal going on? They might be mitigating a DDoS, as that whole system is "hop 12" in your trace route.

Mureel · October 2017

Zagnut123Zagnut123 wrote: »

Is this a EU issue

No.

SirAndy · October 2017

lordrichter wrote: »

Do you run this trace route enough to be able to tell that there is something abnormal going on? They might be mitigating a DDoS, as that whole system is "hop 12" in your trace route.

Well, either hop 12 is configured to not respond to ICMP packages (unlikely) or the hardware representing this hop has some serious problems.

It's close enough to the actual ZOS server that it could very well be a targeted DDoS attack on ZOS, or it's just some piece of equipment malfunctioning.

SirAndy · October 2017

SirAndy wrote: »

... the hardware representing this hop has some serious problems ...

I should probably clarify, i've been getting sporadic 999+ pings this morning along with horrible lag. Whenever one of these lag-spikes occurs, they last for several minutes and my traceroute looks like the one above.

Malic · October 2017

Every weekend the same song and dance.

Tuesday night prime time? I get 2 great hours of lag free PVP in CP and no CP.

so the "why" is your ridiculous expectations.

[Edit to remove bait]

akredon_ESO · October 2017

Doesnt ZOS have anything like a Network operations center ? i work for an ISP And we have people that monitor networks and stuff.... so like if a fiber line goes down you know theirs an outage/lag or some other type of issue... i would think this Trace route is a good example of a network issue...

SirAndy · October 2017

Malic wrote: »

Every weekend the same song and dance.
Tuesday night prime time? I get 2 great hours of lag free PVP in CP and no CP.
so the "why" is your ridiculous expectations.

The part you are missing is that this is not what a traceroute normally looks like for me.
I never get lag like this with 999+ spikes.

There's obviously something wrong, either failing hardware or some sort of DDoS attack.

[Edit to remove bait]

akredon_ESO · October 2017

@SirAndy Pay no heed to Malic he has been in the Salt mine for far too long

SirAndy · October 2017

akredon_ESO wrote: »

Doesnt ZOS have anything like a Network operations center ? i work for an ISP And we have people that monitor networks and stuff.... so like if a fiber line goes down you know theirs an outage/lag or some other type of issue... i would think this Trace route is a good example of a network issue...

There really should be someone on pager duty (why do they still call it that anyways?).

akredon_ESO · October 2017

Oh my company calls it the On call

some times that "on call" it out to lunch... or .... movies

Elsonso · October 2017

SirAndy wrote: »

lordrichter wrote: »

Do you run this trace route enough to be able to tell that there is something abnormal going on? They might be mitigating a DDoS, as that whole system is "hop 12" in your trace route.

Well, either hop 12 is configured to not respond to ICMP packages (unlikely) or the hardware representing this hop has some serious problems.

It's close enough to the actual ZOS server that it could very well be a targeted DDoS attack on ZOS, or it's just some piece of equipment malfunctioning.

My guess is that Hop 12 will be configured to not respond to ICMP, and it may be more than one router. Hop 12 is Prolexic, the DDoS mitigation company. It is a black box. A hole in reality that you cannot see into. Like I said, I am surprised you got a response at the exit of the box. I never do, no matter where I run the trace route, or who runs it.

SirAndy · October 2017

Well that is interesting, i just tried several IPs in the ZOS range and only some of them let the traceroute go through hop 12 while others keep timing out there.

JamilaRaj · October 2017

SirAndy wrote: »

lordrichter wrote: »

Do you run this trace route enough to be able to tell that there is something abnormal going on? They might be mitigating a DDoS, as that whole system is "hop 12" in your trace route.

Well, either hop 12 is configured to not respond to ICMP packages (unlikely)

It is common that some hops play dead. Test some non-ESO server, you will see.

Bryong9ub17_ESO · October 2017

Seem only some of you are having this issue. I've been playing all day with zero issues. Not one spike that I noticed. Latency has been around 90-110 all day for me.

(PC-NA Toledo, Ohio)

Elsonso · October 2017

Bryong9ub17_ESO wrote: »

Seem only some of you are having this issue. I've been playing all day with zero issues. Not one spike that I noticed. Latency has been around 90-110 all day for me.

(PC-NA Toledo, Ohio)

Mine has been running an average of 100 ms here, too. Max was just a smidge over 200ms

Supportic · October 2017

Have you ever considered the server maintenance? That's why it stopped tracing ^^

Seri · October 2017

lordrichter wrote: »

I am surprised you got results past the Akamai (Prolexic) network.

That changed for me with the NA server disconnect last Thursday morning (EST). It appeared Akamai was removed completely then but then re-added at some point between then and when I logged on the following night - Akamai was back then but I was able to traceroute through it. Unfortunately also started pushing my traffic the long way again so back to requiring battleping for something usable

Yarlenzey · October 2017

Well, I see my performance dropping as the player load increases.
By that I mean at 2PM on Saturday, in Australia, performance is great; but it declines around 4:30 as other players start showing up elsewhere in the world, for early morning gameplay.

What you players with less than 350 latency may not have experienced is the time it takes to render various world objects and NPCs. Players often appear as the default 'black character outline' before they render, one by one, as time passes.

My amateur view on all this is that as player load increases, network response is gradually eroded; which in turn reveals limitations in the way the the game is built. Specifically, I am referring to the way that game objects are loaded and rendered dynamically as a player enters a particular world space.

As an example, I was happily motoring along on the long road from Ashen road wayshrine to Steamcamp, when suddenly my FPS and latency figures started jumping all over the place for no apparent reason. An 'apparent' reason is sometimes Ash or rain falling and playing havoc with the rendering.

the real cause in this case was the four players who were behind me, finally catching me up and being rendered by the game.

Anyway, my literally unqualified opinion, is that this problem is one that wont be fixed by a dot-point patch.

Seri · October 2017

Yarlenzey wrote: »

Well, I see my performance dropping as the player load increases.
By that I mean at 2PM on Saturday, in Australia, performance is great; but it declines around 4:30 as other players start showing up elsewhere in the world, for early morning gameplay.

The flaw with that is 2PM for us in Australia is pretty close to prime-time on the NA server with most of the -5 to -8 time-zones are active. If performance drops after 4:30p through until 9p or so that could be internet issues with your ISP. The whole reason they do maintenance from 6ish AEST is because it's the start of the quieter server period.

THWIP71 · October 2017

Elsonso · October 2017

Seri wrote: »

Yarlenzey wrote: »

Well, I see my performance dropping as the player load increases.
By that I mean at 2PM on Saturday, in Australia, performance is great; but it declines around 4:30 as other players start showing up elsewhere in the world, for early morning gameplay.

The flaw with that is 2PM for us in Australia is pretty close to prime-time on the NA server with most of the -5 to -8 time-zones are active. If performance drops after 4:30p through until 9p or so that could be internet issues with your ISP. The whole reason they do maintenance from 6ish AEST is because it's the start of the quieter server period.

Well, @Yarlenzey did not say what they do in ESO, but the fact that they are in Australia certainly does not help. When the FPS and Latency numbers do a dance together, that tells me to start looking at problems closer to the player than the megaserver.

If I had to guess, I would suggest that their computer is underpowered or has a problem, they share internet with their neighbors (broadband), they might be using wifi to play the game, and the traffic from Australia to the megaserver is being routed through some yurt in rural Mongolia using kite string. Oh, and they play too much PVP in this game.

Seri · October 2017

lordrichter wrote: »

[...] the traffic from Australia to the megaserver is being routed through some yurt in rural Mongolia using kite string.

Close.

C:\Users>tracert 198.20.198.110

Tracing route to 198.20.198.110 over a maximum of 30 hops

  1    <1 ms     1 ms     2 ms  192.168.1.1
  2    12 ms    10 ms     9 ms  <snip>
  3     9 ms     8 ms    13 ms  <snip>
  4    10 ms    11 ms    17 ms  <snip>
  5   199 ms   195 ms   194 ms  bundle-3.mlx01.adl01.sa.vocus.net.au [114.31.198.102]
  6   168 ms   170 ms   170 ms  bundle-101.cor02.mel07.vic.vocus.net.au [114.31.198.40]
  7   169 ms   170 ms   166 ms  Hu-0-0-0-1.cor02.syd04.nsw.VOCUS.net.au [114.31.192.114]
  8   171 ms   168 ms   163 ms  Te-2-1-0.bdr01.hkg01.hkg.VOCUS.net [119.161.84.20]
  9   198 ms   196 ms   206 ms  32787.hkg.equinix.com [119.27.63.140]
 10   197 ms   197 ms   197 ms  a72-52-2-186.deploy.static.akamaitechnologies.com [72.52.2.186]
 11   170 ms   171 ms   171 ms  a72-52-2-189.deploy.static.akamaitechnologies.com [72.52.2.189]
 12     *        *        *     Request timed out.
 13     *        *        *     Request timed out.
 14   290 ms   292 ms   294 ms  198.20.192.3
 15   303 ms   305 ms   306 ms  198.20.198.110

Incidentally the path to EU jumps straight from Sydney to NA (which is also the route NA used to take until Akamai protection started).

<snip>
  6   183 ms   181 ms   181 ms  bundle-101.cor02.mel07.vic.vocus.net.au [114.31.198.40]
  7   189 ms   187 ms   187 ms  Hu-0-0-0-1.cor02.syd04.nsw.VOCUS.net.au [114.31.192.114]
  8   182 ms   180 ms   177 ms  bundle-10.cor01.syd04.nsw.vocus.net.au [175.45.72.78]
  9   176 ms   183 ms   177 ms  ten-0-0-0-3-7.cor01.sjc01.ca.vocus.net [114.31.199.87]
 10   178 ms   181 ms   179 ms  bundle-100.bdr01.sjc01.ca.vocus.net [49.255.255.19]
 11   180 ms   184 ms   180 ms  sjo-b21-link.telia.net [62.115.154.12]
 12   181 ms   181 ms   182 ms  palo-b22-link.telia.net [62.115.125.0]
 13   178 ms   185 ms   188 ms  87.190.234.45
 14   339 ms   341 ms   343 ms  217.239.44.22
 15   345 ms   346 ms   348 ms  dns11.parkpage.foundationapi.com [195.122.154.1]

It's unfortunate that ZOS are (apparently) unable to influence the Akamai path because from what I've seen this crazy routing is screwing over at least half my Aussie friends across a multitude of ISPs.