I suspect it's ZOS's Network

Tomg999

I really have no way of knowing, but if the different platform servers are truly separate, and from what I am seeing, I suspect that the high ping, lag, lost connection, etc issues are an issue on the Hosts side prior to the server.
Maybe last mile connections, routers, load balancers, authentication, datacenter lan/switches, rack switches, etc. That or DoS attacks.
it's crazy, and it's a recent problem.

Anyone have any insight or better guesses than I?

Sigtric

DDoS as stated by ZOS over the weekend and as can be seen on internet attack data

Tomg999

Ah I missed that. Thanks

Bonzodog01

So, the server infrastructure is split into an EU datacentre in Frankfurt, Germany, and the US datacentre in Dallas, Texas.

In each of these datacentres, each platforms megaserver stands physically side by side with each other. Each of the megaservers are technically a cabinet of around 16 individual servers connected in such a way that they behave as a cluster, and ultimately as a single server.

Somehow, both the EU and US hosts are suffering from very poor performance and connections.

This is about as much knowledge as we have right now.

Jeremy

Tomg999 wrote: »

I really have no way of knowing, but if the different platform servers are truly separate, and from what I am seeing, I suspect that the high ping, lag, lost connection, etc issues are an issue on the Hosts side prior to the server.
Maybe last mile connections, routers, load balancers, authentication, datacenter lan/switches, rack switches, etc. That or DoS attacks.
it's crazy, and it's a recent problem.

Anyone have any insight or better guesses than I?

It's been consistent over a period of weeks now. I doubt that it is just some weekend hack job to blame.

My issues did not start until after that bumbled maintenance a month or so back. So that's when I suspect the real problem occurred. Might be something nefarious - or they may have downgraded their hardware to conserve money. But I'm fairly convinced that something happened during that downtime that negatively affected their server performance.

Glurin

Jeremy wrote: »

Tomg999 wrote: »

I really have no way of knowing, but if the different platform servers are truly separate, and from what I am seeing, I suspect that the high ping, lag, lost connection, etc issues are an issue on the Hosts side prior to the server.
Maybe last mile connections, routers, load balancers, authentication, datacenter lan/switches, rack switches, etc. That or DoS attacks.
it's crazy, and it's a recent problem.

Anyone have any insight or better guesses than I?

It's been consistent over a period of weeks now. I doubt that it is just some weekend hack job to blame.

My issues did not start until after that bumbled maintenance a month or so back. So that's when I suspect the real problem occurred. Might be something nefarious - or they may have downgraded their hardware to conserve money. But I'm fairly convinced that something happened during that downtime that negatively affected their server performance.

DDoS attacks quite often go on much longer than a single weekend. That is usually just the peak or a temporary spike in the attack.

You can't ransom the servers, for example, if your ransom note contains the phrase "But don't worry because we'll stop the attack at 3:15 tomorrow even if you don't pay us."

Jeremy

Glurin wrote: »

Jeremy wrote: »

Tomg999 wrote: »

I really have no way of knowing, but if the different platform servers are truly separate, and from what I am seeing, I suspect that the high ping, lag, lost connection, etc issues are an issue on the Hosts side prior to the server.
Maybe last mile connections, routers, load balancers, authentication, datacenter lan/switches, rack switches, etc. That or DoS attacks.
it's crazy, and it's a recent problem.

Anyone have any insight or better guesses than I?

It's been consistent over a period of weeks now. I doubt that it is just some weekend hack job to blame.

My issues did not start until after that bumbled maintenance a month or so back. So that's when I suspect the real problem occurred. Might be something nefarious - or they may have downgraded their hardware to conserve money. But I'm fairly convinced that something happened during that downtime that negatively affected their server performance.

DDoS attacks quite often go on much longer than a single weekend. That is usually just the peak or a temporary spike in the attack.

You can't ransom the servers, for example, if your ransom note contains the phrase "But don't worry because we'll stop the attack at 3:15 tomorrow even if you don't pay us."

Maybe it is DDoS attacks.

But It just seems like too much of a coincidence for all my problems to have started right after that maintenance a month or so back - the one where they had to continuously shut down the servers afterwards to try and fix what ever it was they did. This game has not been the same for me since.

NewBlacksmurf

It's always been ZOS server that's the issue.

DDOS attacks just are making it worse.

Ppl are acting like they didn't have problems each month previously.

elvenmad

This is what happends when companys hire cheap a** old and overloaded servers so they can save money, Its ZoS's fault for not paying for better quality server services, but obviouslly they dont care and we the customers have to put up with sub par service while they continue, as they have been doing for years now to state 'they are working on it' LOL

Lukums1

It's never been this bad, ever.

I guess we have to wait 2 weeks to get an official response.

As the last official response was 3 days AFTER the attack. LOL

Callous2208

This is the worst it's ever been for me as well. Bout a month so worth of constant spikes.

Glurin

NewBlacksmurf wrote: »

It's always been ZOS server that's the issue.

DDOS attacks just are making it worse.

Ppl are acting like they didn't have problems each month previously.

Probably because they didn't. A lag spike here, a disconnect there, but on the whole relatively stable. What we're experiencing now is different. It's constant and long lasting, and there's only so much ZOS can do to correct it. If there is a problem with the servers themselves, the DDoS is likely making it much harder to track down.

jedtb16_ESO

Glurin wrote: »

NewBlacksmurf wrote: »

It's always been ZOS server that's the issue.

DDOS attacks just are making it worse.

Ppl are acting like they didn't have problems each month previously.

Probably because they didn't. A lag spike here, a disconnect there, but on the whole relatively stable. What we're experiencing now is different. It's constant and long lasting, and there's only so much ZOS can do to correct it. If there is a problem with the servers themselves, the DDoS is likely making it much harder to track down.

and still aren't really. one failed log in on sunday and that is it.

Jeremy

Callous2208 wrote: »

This is the worst it's ever been for me as well. Bout a month so worth of constant spikes.

Same here.

For about the last month I've been having consistent latency spikes.

heaven13

I agree. That 3 day period of maintenance is when my issues seem to have really started and they've just gotten worse since then.

magnusthorek

IMHO it's easy to blame DDoS because they happen everywhere and don't require much effort. You download an script kiddie rookie All-in-One and that's it.

I'd go even further. These DDoS attacks are made by rage players themselves (not me, I swear XD) face of the ksck of transparency, respect, stability, content...

LadyLethalla

A friend in Japan who plays on PS4 EU has a theory that there aren't actually any servers in EU now... that they moved them all to US a few months back and there's some kind of VPN that makes it look as though there's still an EU server. (I'm not a techie, lol.) He suffers terrible lag on EU daily, late afternoon his time until after midnight.

For myself I find the lag is getting worse and worse on EU. I did play for a couple of months on the NA server February to April and I don't remember any such problems.

At any rate, ZoS aren't ever going to tell us what's going on. But if games like Battlefield and Destiny can operate largely lag-free, I don't get why the devs can't do something about ESO's poor - and worsening - performance.

NewBlacksmurf

Glurin wrote: »

NewBlacksmurf wrote: »

It's always been ZOS server that's the issue.

DDOS attacks just are making it worse.

Ppl are acting like they didn't have problems each month previously.

Probably because they didn't. A lag spike here, a disconnect there, but on the whole relatively stable. What we're experiencing now is different. It's constant and long lasting, and there's only so much ZOS can do to correct it. If there is a problem with the servers themselves, the DDoS is likely making it much harder to track down.

@Glurin

You gotta seperate "you" from "people"
The forums each week have in general, and technical threads posts about server issues outside of operator error or just not doing something correctly.

The statement I made should be understood collectively and not singularly.
Normal for me is dropping at least once per session on Xbox one if playing over one hour daily.

That's on a wired Google Fiber 1 gig ISP with all suggests ports setup and nothing else special running.

Sadly the drops tell me to log into Xbox live when most often I'm in Xbox party chat and no issues occur on the console other than just the game. That's been consistent for over 6 months.

Loading screens have been consistent for over 9 months

I even swapped ISPs and all hardware in 2016 trying to assume it was me.

There is literally a staff post once each month about a server issue.

Lukums1

I can tell everyone this:

If I'm from Australia ( with no lag EVER ) even in cryodill with 100+ zergs, and now "suddenly" lagging it's not me, my hardware or my "region".

7-10 days ago, it was smooth as, no lag, no spike no long loading screens nothing.

As soon as the DDOS attempts had begun... we experienced massive issues and still are quite frankly.

If I have to post full on networking logs to get some bloody attention from ZOS I will and explain what is going on.

I feel like every platform right now is getting ignored... which is unacceptable.

NewBlacksmurf

Lukums1 wrote: »

I can tell everyone this:

If I'm from Australia ( with no lag EVER ) even in cryodill with 100+ zergs, and now "suddenly" lagging it's not me, my hardware or my "region".

7-10 days ago, it was smooth as, no lag, no spike no long loading screens nothing.

As soon as the DDOS attempts had begun... we experienced massive issues and still are quite frankly.

If I have to post full on networking logs to get some bloody attention from ZOS I will and explain what is going on.

I feel like every platform right now is getting ignored... which is unacceptable.

Good point of view.

Conspiracy theory question to you @Lukums1

Do you accept that it's actually a DDOS attack or perhaps is it something else or something in addition to an attack?

Lukums1

NewBlacksmurf wrote: »

Lukums1 wrote: »

I can tell everyone this:

If I'm from Australia ( with no lag EVER ) even in cryodill with 100+ zergs, and now "suddenly" lagging it's not me, my hardware or my "region".

7-10 days ago, it was smooth as, no lag, no spike no long loading screens nothing.

As soon as the DDOS attempts had begun... we experienced massive issues and still are quite frankly.

If I have to post full on networking logs to get some bloody attention from ZOS I will and explain what is going on.

I feel like every platform right now is getting ignored... which is unacceptable.

Good point of view.

Conspiracy theory question to you @Lukums1

Do you accept that it's actually a DDOS attack or perhaps is it something else or something in addition to an attack?

@NewBlacksmurf

As a 10 year network/system admin from our largest telecommunications company in Australia...

I would only deem it "acceptable" if logs were posted on inbound packets on the hosted game servers to show the extent of the attack.

DDOS'ing is really not hard to do. it's more about sending millions of packets of "rubbish data" to a range of IP addresses which then have to process the inbound traffic.

I think they are trying to do shifty things without telling anyone.

let me explain:

I often move servers around (in work hours) while quickly moving people onto different server racks while I do maintenance, however this forms a type of bottleneck for existing customers which is "wrong to do" and bad practice.

Due to the PTS, and impending massive updates, I think a restructure may be happening, but I"m guessing out of my @ss at this point. I think there is a SERIOUS issue which they are working on without telling anyone it's happening.

Mentioning it 3 days ago ( ish ) didn't count, the issue is still occurring and needs to be FIRMLY ADDRESSED.

Syrani

LadyLethalla wrote: »

A friend in Japan who plays on PS4 EU has a theory that there aren't actually any servers in EU now... that they moved them all to US a few months back and there's some kind of VPN that makes it look as though there's still an EU server. (I'm not a techie, lol.) He suffers terrible lag on EU daily, late afternoon his time until after midnight.

For myself I find the lag is getting worse and worse on EU. I did play for a couple of months on the NA server February to April and I don't remember any such problems.

At any rate, ZoS aren't ever going to tell us what's going on. But if games like Battlefield and Destiny can operate largely lag-free, I don't get why the devs can't do something about ESO's poor - and worsening - performance.

I am on PC EU, but live in the US. The lag for me for the past several weeks has been absolutely horrible. If they moved the EU servers here to the US, then it hasn't made any difference.

LadyLethalla

Syrani wrote: »

I am on PC EU, but live in the US. The lag for me for the past several weeks has been absolutely horrible. If they moved the EU servers here to the US, then it hasn't made any difference.

But... if your connection to the PC EU server is going from where you live in the US to a virtual EU server (that's possibly based in Europe), then that wouldn't make any difference...? (Like I said, I'm not a techie.)
My friend's lag started around September, I think. And I do recall a couple of 8+ hour "mandatory infrastructure" maintenances around then.

Lukums1

Whoa, let's not get off the beaten track here.

Lag has always been apparent for some.

We are talking about from Friday the 13th ( funny enough ) to now. With the impending lag on all platforms not just one.
We have massive issues, cryodill (unplayable) which is normally 100% fine... Maelstrom ( blades freeze ) mobs teleport... not kidding.

Our connections are 80MB + with uploads of 30MB + it's not us, it's you.

Time to address.

Syrani

LadyLethalla wrote: »

Syrani wrote: »

I am on PC EU, but live in the US. The lag for me for the past several weeks has been absolutely horrible. If they moved the EU servers here to the US, then it hasn't made any difference.

But... if your connection to the PC EU server is going from where you live in the US to a virtual EU server (that's possibly based in Europe), then that wouldn't make any difference...? (Like I said, I'm not a techie.)
My friend's lag started around September, I think. And I do recall a couple of 8+ hour "mandatory infrastructure" maintenances around then.

Yeah, I have no idea. lol I can turn the computer on, and I can turn it off. That's the extent of my PC "expertise".

runagate

Syrani wrote: »

LadyLethalla wrote: »

Syrani wrote: »

I am on PC EU, but live in the US. The lag for me for the past several weeks has been absolutely horrible. If they moved the EU servers here to the US, then it hasn't made any difference.

But... if your connection to the PC EU server is going from where you live in the US to a virtual EU server (that's possibly based in Europe), then that wouldn't make any difference...? (Like I said, I'm not a techie.)
My friend's lag started around September, I think. And I do recall a couple of 8+ hour "mandatory infrastructure" maintenances around then.

Yeah, I have no idea. lol I can turn the computer on, and I can turn it off. That's the extent of my PC "expertise".

Cuz I had to.

Also, you'd think they'd have the common decency to have a better response time to global, all-platform, severe issues than 72 hours or so... which is when they mentioned there were some problems. They are still ongoing. ONE staff post saying, "Yep, you're not imagining what thousands are complaining about." That's it. That's their collective reaction to their customers.

On the forums, where hardly anyone goes. They can communicate in-game, via email, etc. But don't.

Lukums1

Zero communication. Again. Surprise Surprise.

Doctordarkspawn

I wonder -why- the DDOS attacks are going on.

NewBlacksmurf

Lukums1 wrote: »

NewBlacksmurf wrote: »

Lukums1 wrote: »

I can tell everyone this:

If I'm from Australia ( with no lag EVER ) even in cryodill with 100+ zergs, and now "suddenly" lagging it's not me, my hardware or my "region".

7-10 days ago, it was smooth as, no lag, no spike no long loading screens nothing.

As soon as the DDOS attempts had begun... we experienced massive issues and still are quite frankly.

If I have to post full on networking logs to get some bloody attention from ZOS I will and explain what is going on.

I feel like every platform right now is getting ignored... which is unacceptable.

Good point of view.

Conspiracy theory question to you @Lukums1

Do you accept that it's actually a DDOS attack or perhaps is it something else or something in addition to an attack?

@NewBlacksmurf

As a 10 year network/system admin from our largest telecommunications company in Australia...

I would only deem it "acceptable" if logs were posted on inbound packets on the hosted game servers to show the extent of the attack.

DDOS'ing is really not hard to do. it's more about sending millions of packets of "rubbish data" to a range of IP addresses which then have to process the inbound traffic.

I think they are trying to do shifty things without telling anyone.

let me explain:

I often move servers around (in work hours) while quickly moving people onto different server racks while I do maintenance, however this forms a type of bottleneck for existing customers which is "wrong to do" and bad practice.

Due to the PTS, and impending massive updates, I think a restructure may be happening, but I"m guessing out of my @ss at this point. I think there is a SERIOUS issue which they are working on without telling anyone it's happening.

Mentioning it 3 days ago ( ish ) didn't count, the issue is still occurring and needs to be FIRMLY ADDRESSED.

I like your logic here.

runagate

It doesn't matter why the attacks are happening. You plan ahead to mitigate them because this isn't 1997. Is it 1997? No, it is not.

Google "DDOS mitigation" and you will now know that you don't just sit there and let an attack happen if your business is based on servers communicating over the Internet..

What is likely happening, and ZOS feel free to chime in if you find this supposition true/sad/funny, but some middle management liaison is calling the cloud services company that rents the physical infrastructure and asking them "um, can you please make this stop happening?"

And they're all like, "oh my god, we are trying."

CombatPrayer

Jeremy wrote: »

Tomg999 wrote: »

I really have no way of knowing, but if the different platform servers are truly separate, and from what I am seeing, I suspect that the high ping, lag, lost connection, etc issues are an issue on the Hosts side prior to the server.
Maybe last mile connections, routers, load balancers, authentication, datacenter lan/switches, rack switches, etc. That or DoS attacks.
it's crazy, and it's a recent problem.

Anyone have any insight or better guesses than I?

It's been consistent over a period of weeks now. I doubt that it is just some weekend hack job to blame.

My issues did not start until after that bumbled maintenance a month or so back. So that's when I suspect the real problem occurred. Might be something nefarious - or they may have downgraded their hardware to conserve money. But I'm fairly convinced that something happened during that downtime that negatively affected their server performance.

Same thing for me. And to be quite frank, each update or patch for the last few months has added more crashes to my game. Far more than when I started. I truly doubt this is DDOS only. This weekend was likely DDOS as well, but this was going on weeks ago for me. Even about a month ago because the last maintenance was at least two weeks ago and I've began having problems before that.