I suspect it's ZOS's Network

Lukums1 · January 2017

NewBlacksmurf wrote: »

Lukums1 wrote: »

NewBlacksmurf wrote: »

Lukums1 wrote: »

I can tell everyone this:

If I'm from Australia ( with no lag EVER ) even in cryodill with 100+ zergs, and now "suddenly" lagging it's not me, my hardware or my "region".

7-10 days ago, it was smooth as, no lag, no spike no long loading screens nothing.

As soon as the DDOS attempts had begun... we experienced massive issues and still are quite frankly.

If I have to post full on networking logs to get some bloody attention from ZOS I will and explain what is going on.

I feel like every platform right now is getting ignored... which is unacceptable.

Good point of view.

Conspiracy theory question to you @Lukums1

Do you accept that it's actually a DDOS attack or perhaps is it something else or something in addition to an attack?

@NewBlacksmurf

As a 10 year network/system admin from our largest telecommunications company in Australia...

I would only deem it "acceptable" if logs were posted on inbound packets on the hosted game servers to show the extent of the attack.

DDOS'ing is really not hard to do. it's more about sending millions of packets of "rubbish data" to a range of IP addresses which then have to process the inbound traffic.

I think they are trying to do shifty things without telling anyone.

let me explain:

I often move servers around (in work hours) while quickly moving people onto different server racks while I do maintenance, however this forms a type of bottleneck for existing customers which is "wrong to do" and bad practice.

Due to the PTS, and impending massive updates, I think a restructure may be happening, but I"m guessing out of my @ss at this point. I think there is a SERIOUS issue which they are working on without telling anyone it's happening.

Mentioning it 3 days ago ( ish ) didn't count, the issue is still occurring and needs to be FIRMLY ADDRESSED.

I like your logic here.

@NewBlacksmurf

Thank you.

I'm just kinda saying it how it is you know? It would be nice to get an official response...

However: Being in these positions myself in the past and it's not fun, not one bit you don't want to go to your boss or CUSTOMERS and say.... ahhh I see an issue... but ahhh I don't have a solution "Yet"... so until I get a solution to at least "begin working on an issue" I don't mention it because it starts a *** storm.... I would think same issue here really.

As mentioned it's not your customers ZOS, this issue is your own. 7 Days now, 7 Days we the community and your players, customer base are just asking to know if "works" have started in relation to a solution.

Riggsy · January 2017

Is it possible that their homestead update is too much for their servers?

babedenny · January 2017

I don't believe this is a DDOS attack. I've been the victim of them before (as a consumer), and they were always a relentlessness barrage from the time they started, until the time they either stopped, or were thwarted.

I've been playing along at an average of 100ms ping for an hour, and then notice the "rubber band" effect. I look down and see my latency is 900ms. This lasts for about 10 seconds, and then everything is back to normal.

Glurin · January 2017

NewBlacksmurf wrote: »

Glurin wrote: »

NewBlacksmurf wrote: »

It's always been ZOS server that's the issue.

DDOS attacks just are making it worse.

Ppl are acting like they didn't have problems each month previously.

Probably because they didn't. A lag spike here, a disconnect there, but on the whole relatively stable. What we're experiencing now is different. It's constant and long lasting, and there's only so much ZOS can do to correct it. If there is a problem with the servers themselves, the DDoS is likely making it much harder to track down.

@Glurin

You gotta seperate "you" from "people"
The forums each week have in general, and technical threads posts about server issues outside of operator error or just not doing something correctly.

The statement I made should be understood collectively and not singularly.
Normal for me is dropping at least once per session on Xbox one if playing over one hour daily.

That's on a wired Google Fiber 1 gig ISP with all suggests ports setup and nothing else special running.

Sadly the drops tell me to log into Xbox live when most often I'm in Xbox party chat and no issues occur on the console other than just the game. That's been consistent for over 6 months.

Loading screens have been consistent for over 9 months

I even swapped ISPs and all hardware in 2016 trying to assume it was me.

There is literally a staff post once each month about a server issue.

Yes, yes, I'm well aware of that. My earlier statement still stands.

What you need to understand is that the internet is a hell of a lot more complex than you, your ISP and ZOS and that the server issues that the staff posts about are usually resolved fairly quickly, lasting maybe a day or two. One issue a month does not constitute the disaster that you're making it out to be.

runagate · January 2017

babedenny wrote: »

I don't believe this is a DDOS attack. I've been the victim of them before (as a consumer), and they were always a relentlessness barrage from the time they started, until the time they either stopped, or were thwarted.

I've been playing along at an average of 100ms ping for an hour, and then notice the "rubber band" effect. I look down and see my latency is 900ms. This lasts for about 10 seconds, and then everything is back to normal.

Well, that could just as easily be an effect of the attempts to mitigate the attack. DDOS attacks are adaptive nowadays otherwise they'd be extremely easy to simply filter out as a known attack.

babedenny · January 2017

runagate wrote: »

babedenny wrote: »

I don't believe this is a DDOS attack. I've been the victim of them before (as a consumer), and they were always a relentlessness barrage from the time they started, until the time they either stopped, or were thwarted.

I've been playing along at an average of 100ms ping for an hour, and then notice the "rubber band" effect. I look down and see my latency is 900ms. This lasts for about 10 seconds, and then everything is back to normal.

Well, that could just as easily be an effect of the attempts to mitigate the attack. DDOS attacks are adaptive nowadays otherwise they'd be extremely easy to simply filter out as a known attack.

True, and I considered that. In that case, they keep 'out adapting' each other. It could also be a failing piece of network gear.

billp_ESO · January 2017

So, is it still happening tonight? I haven't logged in yet, was dropped a bunch yesterday. Not sure I even want to try it ....

runagate · January 2017

babedenny wrote: »

runagate wrote: »

babedenny wrote: »

I don't believe this is a DDOS attack. I've been the victim of them before (as a consumer), and they were always a relentlessness barrage from the time they started, until the time they either stopped, or were thwarted.

I've been playing along at an average of 100ms ping for an hour, and then notice the "rubber band" effect. I look down and see my latency is 900ms. This lasts for about 10 seconds, and then everything is back to normal.

Well, that could just as easily be an effect of the attempts to mitigate the attack. DDOS attacks are adaptive nowadays otherwise they'd be extremely easy to simply filter out as a known attack.

True, and I considered that. In that case, they keep 'out adapting' each other. It could also be a failing piece of network gear.

Well, like everything else around here I suspect it ceases getting attention once 5:30 PM EST comes around and the bloke who makes polite phone calls to their cloud services provider isn't there to call and ask for an update anymore until 9 AM the next day. Or once the weekend or holiday is over. Not real hard to see a pattern in employee posts. No matter how severe the issue is. Course, that's not to say no one is up at midnight making angry calls. It's just that the people authorized to make one sentence PR communiques to pass on to community managers and their ilk to we wretched customers don't work those hours. Or care what we think til they get a bad quarterly review in comparison to some other unit of the parent company. This, of course, is ultimately why they will fail. Inattentive yet Orwellian middle management. It needn't be that way. It's just toxic company culture and you can see examples in every little twinge of their interactions in videos. The "not joke jokes."

NewBlacksmurf · January 2017

Glurin wrote: »

NewBlacksmurf wrote: »

Glurin wrote: »

NewBlacksmurf wrote: »

It's always been ZOS server that's the issue.

DDOS attacks just are making it worse.

Ppl are acting like they didn't have problems each month previously.

Probably because they didn't. A lag spike here, a disconnect there, but on the whole relatively stable. What we're experiencing now is different. It's constant and long lasting, and there's only so much ZOS can do to correct it. If there is a problem with the servers themselves, the DDoS is likely making it much harder to track down.

@Glurin

You gotta seperate "you" from "people"
The forums each week have in general, and technical threads posts about server issues outside of operator error or just not doing something correctly.

The statement I made should be understood collectively and not singularly.
Normal for me is dropping at least once per session on Xbox one if playing over one hour daily.

That's on a wired Google Fiber 1 gig ISP with all suggests ports setup and nothing else special running.

Sadly the drops tell me to log into Xbox live when most often I'm in Xbox party chat and no issues occur on the console other than just the game. That's been consistent for over 6 months.

Loading screens have been consistent for over 9 months

I even swapped ISPs and all hardware in 2016 trying to assume it was me.

There is literally a staff post once each month about a server issue.

Yes, yes, I'm well aware of that. My earlier statement still stands.

What you need to understand is that the internet is a hell of a lot more complex than you, your ISP and ZOS and that the server issues that the staff posts about are usually resolved fairly quickly, lasting maybe a day or two. One issue a month does not constitute the disaster that you're making it out to be.

@Glurin

While
I'm not a network admin my brother manages microsofts office 360 cloud. And I'm benefiting from learning a lot from him as he tests and breaks stuff using servers we built so I'm not clueless.

Also please understand I'm not suggesting a few issues I'm literally coming from a position to where I've personal balky worked with ZOS specifically since 2013 which began with their server and the missOmg red boxes to after the lighting update in 2014 for over 3 months of major lag and packet loss issues caused to thousands of customers due to their server.

Skipping forward as I moved to Xbox one, there are and continue to be issues where some have existed since Xbox one beta.

It's not an overreaction but a reality that there are real issues with zos' server.

Me being open and willing to test and try a lot of the feedback and pms from staff to resolve things over the years has also lead to us and them realizing and uncovering issues on their end.

I'm gonna assume you're a lot more network savvy than myself as I'd consider myself average having some common knowledge and my brother being an expert....I'd assume ZOS has a lot of experts from my interactions with them once escalated.

There are real issues bud. It's not ppl crying the sky is falling cause of a few hiccups.

Betheny · January 2017

I have never had any issues until the past week.

Which is when the "DDoS" thing started to happen, according to that one post from ZOS...

babedenny · January 2017

I'm averaging 100ms ping right now. Smooth sailing, so far.

ZOS_GregoryV · January 2017

We have decided to leave this thread open.

old_mufasa · January 2017

Well heck its been almost a year with buzzing crash bug on the xbox.. and we cant even get a "we are looking into it" response.. or the memory leak issue with crafting and dye stations that also will crash the game..

NewBlacksmurf · January 2017

ZOS_GregoryV wrote: »

We have decided to leave this thread open.

@Lukums1

Bruh the forums are like the servers now. Maybe it's all ZOS locking servers like threads

CombatPrayer · January 2017

I thought it was DDOS and maybe last weekend it was. But this is going on while with crashes among other issues. I tolerated the crashes, and this last weekend of random kicks mid dungeon, mid quest, mid trials. Certainly not renewing my ESO Plus. I'm glad I held off. It's up in < 1 day. I think I'll be taking some serious time off and just hop on to do what I need to for my guild.

Yep, time for a major time out. Just research for me. Andromeda is around the corner anyway. Good timing for me.

Good luck ZOS. I hope you fix the issues soon. ESO has a lot of unrealized potential. I know many are waiting for vvardenfell release, but who knows when that will be. Hopefully sometime this year.

Lukums1 · January 2017

ZOS_GregoryV wrote: »

We have decided to leave this thread open.

Thank you, Gregory.

It's been bad the last week since that Friday and it continues. I can say even right now there are issues.

Mobs are still teleporting in Maelstrom (even with a 100mb/80mb) connection.

Just giving an update...

Elsonso · January 2017

Lukums1 wrote: »

ZOS_GregoryV wrote: »

We have decided to leave this thread open.

Thank you, Gregory.

It's been bad the last week since that Friday and it continues. I can say even right now there are issues.

Mobs are still teleporting in Maelstrom (even with a 100mb/80mb) connection.

Just giving an update...

In your case, my suspicion runs to packet loss in the network between ZOS and you as the first reason for teleporting monsters that should not be teleporting. Historically, I frequently see it in one or two other players, not mobs and not the general player population, and my base assumption is that those other players are on a poor connection to the server.

Can what you see be due to the server? Yes, however, in a situation where we already know the servers have been under a DDoS, and without knowing the full status of such an event, network problems seem more likely to be at fault than server.

I want to add that your local internet connection speed means nothing. It is like expecting the road from Perth to Sydney to be smooth and free of bumps because your driveway is really wide. You should know that, though, so I wonder why you bring it up.

Lukums1 · January 2017

lordrichter wrote: »

Lukums1 wrote: »

ZOS_GregoryV wrote: »

We have decided to leave this thread open.

Thank you, Gregory.

It's been bad the last week since that Friday and it continues. I can say even right now there are issues.

Mobs are still teleporting in Maelstrom (even with a 100mb/80mb) connection.

Just giving an update...

In your case, my suspicion runs to packet loss in the network between ZOS and you as the first reason for teleporting monsters that should not be teleporting. Historically, I frequently see it in one or two other players, not mobs and not the general player population, and my base assumption is that those other players are on a poor connection to the server.

Can what you see be due to the server? Yes, however, in a situation where we already know the servers have been under a DDoS, and without knowing the full status of such an event, network problems seem more likely to be at fault than server.

I want to add that your local internet connection speed means nothing. It is like expecting the road from Perth to Sydney to be smooth and free of bumps because your driveway is really wide. You should know that, though, so I wonder why you bring it up.

Please don't talk to me like I have no idea what I'm talking about.

I "mention" it because have you seen the ticket responses people are getting from technical support staff mmm?

Blaming THEIR local connections. Honestly it's beyond a joke, sorry this is not an attack on you, it's an attack on THEIR technical support team which is really dropping the ball.

Sending helpless people on massive run arounds thinking their connections are to blame for the server being hit hard.

As mentioned I explained what I thought is happening, without a official post from ZOS it's a guessing game which I also mentioned previously.

letsdothedungeonslow · January 2017

LadyLethalla wrote: »

A friend in Japan who plays on PS4 EU has a theory that there aren't actually any servers in EU now... that they moved them all to US a few months back and there's some kind of VPN that makes it look as though there's still an EU server. (I'm not a techie, lol.) He suffers terrible lag on EU daily, late afternoon his time until after midnight.

For myself I find the lag is getting worse and worse on EU. I did play for a couple of months on the NA server February to April and I don't remember any such problems.

At any rate, ZoS aren't ever going to tell us what's going on. But if games like Battlefield and Destiny can operate largely lag-free, I don't get why the devs can't do something about ESO's poor - and worsening - performance.

As someone who was playing PS4 EU until a few weeks ago and who has been playing PC NA for the last few weeks, I can confirm that apart from some horrible spots (Elden Root, Mournhold,sometimes the odd dungeon) the PS4 EU server runs A WHOLE LOT BETTER than the PC NA one.

I'm enjoying the game overall, but am annoyed there is no way to transfer the character, and even more annoyed about the disconnections and the insane lag.

Bottom line: I, like tens of thousands of other people, are not going to subscribe all the while the game is in its current state. It's a crying shame that ZOS don't get their act together. If they did, I think they'd make a lot more money.

But crown crates, housing and the never-ending BS of costumes, wigs, moustaches and stupid pets only suits the dorks who stand around in Elden Root and Mournhold dropping their Ultimates.

People who actually want to play the game are being heavily dissuaded right now...

DagmarSLNY · January 2017

runagate wrote: »

It doesn't matter why the attacks are happening. You plan ahead to mitigate them because this isn't 1997. Is it 1997? No, it is not.

Google "DDOS mitigation" and you will now know that you don't just sit there and let an attack happen if your business is based on servers communicating over the Internet.

Lukums1 · January 2017

DagmarSLNY wrote: »

runagate wrote: »

It doesn't matter why the attacks are happening. You plan ahead to mitigate them because this isn't 1997. Is it 1997? No, it is not.

Google "DDOS mitigation" and you will now know that you don't just sit there and let an attack happen if your business is based on servers communicating over the Internet.

Perfect.

billp_ESO · January 2017

So let's say someone is using a botnet to DDOS the game.

Why?

Lukums1 · January 2017

billp_ESO wrote: »

So let's say someone is using a botnet to DDOS the game.

Why?

That's a simple one,

Because they CAN.

Sigtric · January 2017

elvenmad wrote: »

This is what happends when companys hire cheap a** old and overloaded servers so they can save money, Its ZoS's fault for not paying for better quality server services, but obviouslly they dont care and we the customers have to put up with sub par service while they continue, as they have been doing for years now to state 'they are working on it' LOL

How do you have any sort of idea what hardware they are or are not using? lol

Lukums1 · January 2017

Sigtric wrote: »

elvenmad wrote: »

This is what happends when companys hire cheap a** old and overloaded servers so they can save money, Its ZoS's fault for not paying for better quality server services, but obviouslly they dont care and we the customers have to put up with sub par service while they continue, as they have been doing for years now to state 'they are working on it' LOL

How do you have any sort of idea what hardware they are or are not using? lol

To be fair he doesn't. And shouldn't make those types of comments.

This is all about security this time around.

Glurin · January 2017

billp_ESO wrote: »

So let's say someone is using a botnet to DDOS the game.

Why?

Who knows. Maybe it's crooks trying to ransom the servers or break in to steal financial information. Maybe it's a rival company trying to take down their competition. Maybe it's some idiot activist throwing a hissy fit and protesting Trump's inauguration because big corporations are evil. Or maybe it's just some stupid kid that thinks it's funny.

Your guess is as good as mine.

Elsonso · January 2017

Lukums1 wrote: »

I "mention" it because have you seen the ticket responses people are getting from technical support staff mmm?

Blaming THEIR local connections. Honestly it's beyond a joke, sorry this is not an attack on you, it's an attack on THEIR technical support team which is really dropping the ball.

Sending helpless people on massive run arounds thinking their connections are to blame for the server being hit hard.

As mentioned I explained what I thought is happening, without a official post from ZOS it's a guessing game which I also mentioned previously.

The proper response of customer support to a network connectivity ticket is to have the user check their local connections.

Yes, ZOS should have a system in place so that when players are writing tickets about connection problems that they are told that there is a network disruption happening. Not in a forum post. Not in a service bulletin. Not in a Tweet. Right where the user goes to open a ticket. Sadly, this is not the way that ZOS thinks about communicating with the customer.

Because they do not do that, Chet has no idea what the problem is and can only assume that the first thing that needs to be done is to check the local connection. He is right. That is the place to start.

Sigtric wrote: »

How do you have any sort of idea what hardware they are or are not using? lol

We don't know what they are currently using, but back in 2014, the EU server was HP BladeSystem. Racks of what looked like c7000 enclosures with an uncountable number of blades and CPUs.

Lukums1 · January 2017

lordrichter wrote: »

Lukums1 wrote: »

I "mention" it because have you seen the ticket responses people are getting from technical support staff mmm?

Blaming THEIR local connections. Honestly it's beyond a joke, sorry this is not an attack on you, it's an attack on THEIR technical support team which is really dropping the ball.

Sending helpless people on massive run arounds thinking their connections are to blame for the server being hit hard.

As mentioned I explained what I thought is happening, without a official post from ZOS it's a guessing game which I also mentioned previously.

The proper response of customer support to a network connectivity ticket is to have the user check their local connections.

Yes, ZOS should have a system in place so that when players are writing tickets about connection problems that they are told that there is a network disruption happening. Not in a forum post. Not in a service bulletin. Not in a Tweet. Right where the user goes to open a ticket. Sadly, this is not the way that ZOS thinks about communicating with the customer.

Because they do not do that, Chet has no idea what the problem is and can only assume that the first thing that needs to be done is to check the local connection. He is right. That is the place to start.

Sigtric wrote: »

How do you have any sort of idea what hardware they are or are not using? lol

We don't know what they are currently using, but back in 2014, the EU server was HP BladeSystem. Racks of what looked like c7000 enclosures with an uncountable number of blades and CPUs.

Mate ZOS got no clue. (tech support)

When you have ALL platforms showing issues and forum posts and technical support sections which NEVER get responded to you have an issue.

It's not just 1 platform it's not just a few people this is hitting everyone hard.

So yes, getting told to go back and check your local connection seems like a pretty $!@tarded thing to suggest at this point agreed?

Messy1 · January 2017

This is getting tiresome, like everything, the conspiracies, the conjecture, the lack of transparency and responsiveness from ZOS.

I hope that they would be using cutting edge server technology, especially when one of there chief selling points is "megaserver" technology. I'm beginning to agree with someone who said ZOS should be sued for false advertising if things are not fixed. Big promises were made in the beginning of this game such as "lag free," "massive battles" etc.

Oh and another odd thing, It used to be that everyone's character appeared quickly and fast. Now you go to any capital or trade hubs and it takes up to a few mins for people's characters appearances to load. Could be that there are more players playing, but you think a company would provide a better foundation with their core server technology.

If ZOS cut corners and really skimped on their seever capabilities then they have totally earned this huge crash and burn. I want to keep playing this game. FIX IT PLEASE!!!

Messy1 · January 2017

letsdothedungeonslow wrote: »

LadyLethalla wrote: »

A friend in Japan who plays on PS4 EU has a theory that there aren't actually any servers in EU now... that they moved them all to US a few months back and there's some kind of VPN that makes it look as though there's still an EU server. (I'm not a techie, lol.) He suffers terrible lag on EU daily, late afternoon his time until after midnight.

For myself I find the lag is getting worse and worse on EU. I did play for a couple of months on the NA server February to April and I don't remember any such problems.

At any rate, ZoS aren't ever going to tell us what's going on. But if games like Battlefield and Destiny can operate largely lag-free, I don't get why the devs can't do something about ESO's poor - and worsening - performance.

As someone who was playing PS4 EU until a few weeks ago and who has been playing PC NA for the last few weeks, I can confirm that apart from some horrible spots (Elden Root, Mournhold,sometimes the odd dungeon) the PS4 EU server runs A WHOLE LOT BETTER than the PC NA one.

I'm enjoying the game overall, but am annoyed there is no way to transfer the character, and even more annoyed about the disconnections and the insane lag.

Bottom line: I, like tens of thousands of other people, are not going to subscribe all the while the game is in its current state. It's a crying shame that ZOS don't get their act together. If they did, I think they'd make a lot more money.

But crown crates, housing and the never-ending BS of costumes, wigs, moustaches and stupid pets only suits the dorks who stand around in Elden Root and Mournhold dropping their Ultimates.

People who actually want to play the game are being heavily dissuaded right now...

ZOS should circulate this as a memo and act on it! Good suggestions here.