Packet loss affecting TWC to ATT networks at Peering point 192.205.36.113

lalley

ESO Network Team:

Is ATT your ISP for the NA megaserver? If so, can you get on the horn and yell at those bozos to fix their Peering Point Router at address: 192.205.36.113? It would appear that Internet traffic from the Midwest on TWC is having major issues when traversing to the ATT network at a very specific Peering Point. Below is a traceroute:

Tracing route to 198.20.200.153 over a maximum of 30 hops

1 * * * Request timed out.
2 1 ms 1 ms 1 ms 192.168.0.1
3 37 ms 29 ms 28 ms 142.254.156.61
4 11 ms 10 ms 10 ms tge7-5.hlrdoh0702h.midwest.rr.com [24.95.87.53]

5 13 ms 15 ms 15 ms 24.33.161.192
6 42 ms 62 ms 21 ms be27.clevohek01r.midwest.rr.com [65.29.1.38]
7 24 ms 27 ms 23 ms bu-ether37.vinnva0510w-bcr00.tbone.rr.com [107.1
4.19.58]
8 23 ms 25 ms 25 ms 0.ae3.pr0.dca20.tbone.rr.com [107.14.19.160]
9 29 ms 28 ms 27 ms 216.50.76.41
10 * * 64 ms 192.205.36.113
11 96 ms 99 ms * cr82.wshdc.ip.att.net [12.122.135.182]
12 151 ms * 139 ms cr2.wswdc.ip.att.net [12.122.134.94]
13 98 ms * * cr1.attga.ip.att.net [12.122.1.173]
14 100 ms 99 ms 95 ms cr2.dlstx.ip.att.net [12.122.2.110]
15 97 ms 155 ms 124 ms ggr3.dlstx.ip.att.net [12.122.138.25]

You can see the packet loss on hop 10 at the aforementioned router. This is causing the game to be unplayable... and not just ESO, but ANY online games that use ATT as their ISP for hosted services. It started happening about 2 am EDT this morning and hasn't let up since. I can't get into the game for more the 2-5 minutes before I get dropped to the login screen with an Error <<1>.

HELP!!!

Nestor

Is this repeatable? The reason I ask is some network equipment, namely Juniper stuff, will not respond to ICMP, or slowly respond, if they are busy (CPU usage threshold mostly)

Also, a jump from 27 to 64 MS is not that huge, 120 or 180 is.

Another place you can try to trace to is 4.2.2.1 or 4.2.2.2, they are DNS servers in Dallas, so would follow the same path most of the way.

Also, have you tried a Looking Glass Traceroute where you trace back to your IP from LG Server in the Texas area (which I am assuming the server farm is in, it could be somewhere else). Google Looking Glass Trace Route to find a list of the servers that do this.

lalley

Yes. This is repeatable. It's not the jump from 27ms to 64ms, it's that the router will continuously give no response. After it hits that IP, pings stop responding at most or all of the remaining hops. It also appears to get slightly better for a few minutes at a time before giving horrible pings and / or no response.

Traceroutes to 4.2.2.1 seem to be following a different route for some reason.

Tracing route to a.resolvers.level3.net [4.2.2.1]
over a maximum of 30 hops:

1 * * * Request timed out.
2 4 ms 7 ms 16 ms 192.168.0.1
3 35 ms 14 ms 31 ms 142.254.156.61
4 12 ms 10 ms 9 ms tge7-5.hlrdoh0702h.midwest.rr.com [24.95

5 12 ms 15 ms 11 ms 24.33.161.194
6 16 ms 25 ms 25 ms be27.clevohek01r.midwest.rr.com [65.29.1
7 24 ms 28 ms 27 ms bu-ether17.vinnva0510w-bcr00.tbone.rr.co
9.6.70]
8 65 ms 26 ms 25 ms 0.ae0.pr0.dca20.tbone.rr.com [107.14.19.
9 59 ms 26 ms 26 ms 66.109.7.162
10 27 ms 68 ms 25 ms ae-203-3607.edge2.Washington12.Level3.ne
159.169]
11 25 ms 25 ms 26 ms ae-204-3608.edge2.Washington12.Level3.ne
159.173]
12 27 ms 25 ms 25 ms a.resolvers.level3.net [4.2.2.1]

Trace complete.

lalley

Also, check this out:

ESO isn't the only game that appears to be getting affected by this issue:

http://us.battle.net/d3/en/forum/topic/17346266219?page=2#35

lalley

Switched off my Ethernet card and connected with WiFi to my phone's mobile hot spot and get this traceroute instead:

Tracing route to 198.20.200.153 over a maximum of 30 hops

1 2 ms 2 ms 2 ms 172.20.10.1
2 204 ms 41 ms 49 ms 172.26.96.169
3 41 ms 25 ms 50 ms 172.16.232.164
4 80 ms 39 ms 38 ms 12.249.2.9
5 112 ms 71 ms 57 ms 12.83.188.242
6 92 ms 72 ms 58 ms cr2.nsvtn.ip.att.net [12.122.28.73]
7 103 ms 68 ms 70 ms cr1.dlstx.ip.att.net [12.122.1.241]
8 113 ms 86 ms 54 ms 12.123.18.105

No disconnects, low server ping 100-140 average and smooth playing. Guess I have an answer for the time being... but - something should still be done about that pegged router in DC... just sayin'

jybgess

Im a TWC customer also and today ive been have crazy lag spikes

glak

Tracing route to 198.20.200.153 over a maximum of 30 hops

1 28 ms 42 ms 33 ms
2 14 ms 18 ms 22 ms
3 19 ms 15 ms 31 ms tge0-5-0-14.clmkohpe17r.midwest.rr.com [65.29.17.212]
4 17 ms 15 ms 15 ms 65.29.1.34
5 29 ms 27 ms 23 ms bu-ether45.chctilwc00w-bcr00.tbone.rr.com [107.14.19.36]
6 25 ms 23 ms 36 ms bu-ether11.chcgildt87w-bcr00.tbone.rr.com [66.109.6.20]
7 68 ms 22 ms 25 ms 0.ae1.pr1.chi10.tbone.rr.com [107.14.17.194]
8 42 ms 46 ms 95 ms 216.1.94.65
9 34 ms 38 ms 34 ms 207.88.15.90.ptr.us.xo.net [207.88.15.90]
10 61 ms 85 ms 58 ms 206.111.2.90.ptr.us.xo.net [206.111.2.90]
11 75 ms 64 ms 55 ms cr2.cgcil.ip.att.net [12.122.132.122]
12 55 ms 57 ms 55 ms cr2.sl9mo.ip.att.net [12.122.2.22]
13 55 ms 59 ms 73 ms cr1.sl9mo.ip.att.net [12.122.2.217]
14 61 ms 58 ms 63 ms cr2.dlstx.ip.att.net [12.122.3.221]
15 57 ms 55 ms 57 ms ggr3.dlstx.ip.att.net [12.122.138.25]
16 12.250.61.22 reports: Destination net unreachable.

lalley

glak wrote: »

Tracing route to 198.20.200.153 over a maximum of 30 hops

1 28 ms 42 ms 33 ms
2 14 ms 18 ms 22 ms
3 19 ms 15 ms 31 ms tge0-5-0-14.clmkohpe17r.midwest.rr.com [65.29.17.212]
4 17 ms 15 ms 15 ms 65.29.1.34
5 29 ms 27 ms 23 ms bu-ether45.chctilwc00w-bcr00.tbone.rr.com [107.14.19.36]
6 25 ms 23 ms 36 ms bu-ether11.chcgildt87w-bcr00.tbone.rr.com [66.109.6.20]
7 68 ms 22 ms 25 ms 0.ae1.pr1.chi10.tbone.rr.com [107.14.17.194]
8 42 ms 46 ms 95 ms 216.1.94.65
9 34 ms 38 ms 34 ms 207.88.15.90.ptr.us.xo.net [207.88.15.90]
10 61 ms 85 ms 58 ms 206.111.2.90.ptr.us.xo.net [206.111.2.90]
11 75 ms 64 ms 55 ms cr2.cgcil.ip.att.net [12.122.132.122]
12 55 ms 57 ms 55 ms cr2.sl9mo.ip.att.net [12.122.2.22]
13 55 ms 59 ms 73 ms cr1.sl9mo.ip.att.net [12.122.2.217]
14 61 ms 58 ms 63 ms cr2.dlstx.ip.att.net [12.122.3.221]
15 57 ms 55 ms 57 ms ggr3.dlstx.ip.att.net [12.122.138.25]
16 12.250.61.22 reports: Destination net unreachable.

Looks good. Especially since the offending router isn't present.

lalley

A couple posts from the Blizzard folks regarding this issue:

http://us.battle.net/heroes/en/forum/topic/17346286256?page=1

Sounds like they're already on the case. Perhaps this will get resolved sometime soon! Let's hope!

lalley

Look at this lovely traceroute with no packet loss:

Tracing route to 198.20.200.153 over a maximum of 30 hops

1 * * * Request timed out.
2 1 ms 1 ms 1 ms 192.168.0.1
3 29 ms 23 ms 17 ms 142.254.156.61
4 12 ms 12 ms 9 ms tge7-5.hlrdoh0702h.midwest.rr.com [24.95.87.53]

5 12 ms 15 ms 11 ms 24.33.161.192
6 19 ms 15 ms 19 ms be27.clevohek01r.midwest.rr.com [65.29.1.38]
7 26 ms 24 ms 26 ms bu-ether37.vinnva0510w-bcr00.tbone.rr.com [107.14.19.58]
8 25 ms 25 ms 25 ms 0.ae3.pr0.dca20.tbone.rr.com [107.14.19.160]
9 28 ms 26 ms 29 ms 216.50.76.41
10 29 ms 31 ms 27 ms 192.205.36.113
11 61 ms 63 ms 63 ms cr82.wshdc.ip.att.net [12.122.135.182]
12 66 ms 63 ms 67 ms cr2.wswdc.ip.att.net [12.122.134.94]
13 65 ms 63 ms 63 ms cr1.attga.ip.att.net [12.122.1.173]
14 64 ms 64 ms 63 ms cr2.dlstx.ip.att.net [12.122.2.110]
15 117 ms 63 ms 78 ms ggr3.dlstx.ip.att.net [12.122.138.25]

It would appear this issue has been rectified! WOOHOO!