Recent Issues with NA Megaserver

Mojmir

lordrichter wrote: »

Mojmir wrote: »

lordrichter wrote: »

Mojmir wrote: »

lordrichter wrote: »

rootimus wrote: »

lordrichter wrote: »

Amazon does not host ESO

Genuinely curious - do you have a source for this?

Has ZOS come out and said it? No.

they dont say alot of things, thats just one of their major problems.

They should tell us who provides and runs their data center?

i dont care who runs what for them, theyve been aware for awhile that they lack the measures to defend themselves. this isnt the first time nor the last. you sound like you could help them,here ya go:

Bah. They can't afford me.

LOL, you get an awesome

i do understand the severity and crippling effect such an attack has, i dont deny that. but there is way more to the problem i beleieve than just this.

m12d12_ESO

What this all says, is we are left in the dark by a company that won't even provide a proper customer service experience for their people. You, me, all of us, shouldnt have to scour the forums to find out why we cant access something we pay for.

Every company, small or large, provides some sense of customer service. Communications back and forth - not one sided - find it in the forums if you are lucky.

I haven't been able to play or log in for the better part of the event. Again today, all day, not able to log in.

Have any of you effected gotten an email, even a general email sent to thousands that say something like - "We see your account has been affected and you continued patience is important to us - and we are working on the problem.."?

No. I am sure no one has. So you have to look in the forums and see if someone from Zenimax has posted anything at all that day that pertains to your issues---maybe they have or mybe not.

Taysa

So no update on this?

Loading screens are still intermittently atrocious. Logging in is a chore as it's plagued with Log-In 200/108 errors. Client lock ups and crashes. Screen freezing and then logging me out.

This is getting ridiculous ZoS. We deserve more than "Oops, DDoS attack. Enjoy your weekend on our crappy servers ¯\_(ツ)_/¯"

CiliPadi

How does DDoS affect broken grouping system? Enlighten me, I'm not good at these networking things.
Lag is one thing, but modern MMOs have got grouping working 100%.

This game has been out for, what, 3yrs? And grouping is still broken.

Appreciate the update, but it is slow and way too late. A lot of my ESO friends have quit and gone to other MMOs.

MickeyBN

I'm seeing people complaining that the influx of PvPers are causing stress on the servers, then I'm seeing others begging for the PvP event to be extended...

Somebody help me understand this.

allup8679

I have a suggestion for you.

https://youtu.be/xjzzLelV0Y0

Caligamy_ESO

Whoever out there is responsible for it better be laying REAL low, this is a company that has no qualms about suing even giant corporations. Others have been sued and jailed over this very thing over the past 7-8 years.

russelmmendoza

Thanks for the info.
I am actually cursing whenever lag and kickouts hit me while playing.
Last boss dungeon you get kicked out. Doing nDSA you get kicked out.
I can accept dying because of lag at least you can resurrect there.
Getting kicked out of game while playing means you restart again.
Seriously WTF OMG. I gave up nDSA made me really sad.

Drummerx04

Can all the Sorc Haters please stop DDoSing ZoS's servers? Thank you.

Mojmir

Caligamy_ESO wrote: »

Whoever out there is responsible for it better be laying REAL low, this is a company that has no qualms about suing even giant corporations. Others have been sued and jailed over this very thing over the past 7-8 years.

Facebook is ddos attacking eso-confirmed

akl77

I'm not sure if I'll buy into that, sounds like an excuse for the lame server and cover up of the profit eating company.

DrPain

Thank you for this thread. Like many I was getting very frustrated at not knowing why this was happening, or why it started when it was going/running fine for so long.

I will remember this next time I have such issue's with lag, disco's and log outs.

Elsonso

CiliPadi wrote: »

How does DDoS affect broken grouping system? Enlighten me, I'm not good at these networking things. Lag is one thing, but modern MMOs have got grouping working 100%.

We don't know how their grouping system is designed to work. We know that it is a problem child and that it frequently has the appearance of not working. Making a simple grouping system is pretty much as easy as it sounds. It is when filters and criteria are put into place that things start to go bad. I think they use criteria and filters in an attempt to make the best and most fun group environment that they can.

I figure the situation with the grouping tool resembles the quote by Scotty from The Search For Spock: "The more they overthink the plumbing, the easier it is to stop up the drain."

DDoS is stopping up the grouping drain, somehow. Maybe they have a "latency criteria" in there so that everyone in the group has a similar latency? Maybe the grouping tool thinks the person is disconnected?

Talrol

@ZOS_GinaBruno It is appreciated that you replied and put this information out there. It would be much more helpful in the future if you alerted your web support team of these situations so that when we submit a support ticket they don't insist that it's a problem on our end.

As a support agent for one of the nations largest ISPs I can tell you that everything that the support agent stated in my answered ticket had absolutely nothing to do with the issues that I was experiencing, this creates a very poor taste in my mouth for the competency of your support team.

HalloweenWeed

ZOS_GinaBruno wrote: »

While we do have protection in place (which we obviously can’t discuss publicly), it hasn’t been enough. We’ve been working hard to implement additional security measures to make sure you’re able to log in – and stay logged in – whenever you want to play, and aren’t interrupted. We hope you understand why we’re being somewhat vague and not discussing specifics here. We take this issue very seriously, though, and are working hard to ensure everyone has the best experience possible.

Thank you for your continued support and patience.

What she is really getting at: A. It costs money to fight these things without reducing service quality quite a bit.
B. Countermeasures will usually affect (a small number of) players. Some valid player IP addresses, or "address translations," could be tagged as hostile (and blocked) during such countermeasures. The effects could be permanent inability to access the server (from your present IP address), for those players.
C. They don't have much experience combating DDoS attacks (of this magnitude).

This is why they are reluctant to do anything about the DDoS attacks. Very understandable.
Also, I wonder if this is why people reported certain network nodes near the server with high ping times - they were swamped.

NeoXanthus

DDoS is nothing new and should have been planned for from the beginning. There are many ways to mitigate the attacks. One approach is to use an on-data center appliance solution like Radware or Arbor with an additional cloud based volumetric service like Prolexic Akamai or Verisign DDoS mitigation services. These solutions are not cheap but are necessary in our day.

Elsonso

NeoXanthus wrote: »

DDoS is nothing new and should have been planned for from the beginning. There are many ways to mitigate the attacks. One approach is to use an on-data center appliance solution like Radware or Arbor with an additional cloud based volumetric service like Prolexic Akamai or Verisign DDoS mitigation services. These solutions are not cheap but are necessary in our day.

Honestly, they have been dealing with DDoS for quite a while. Gina's statement suggests that this most recent attack has bypassed existing DDoS mitigation. Given what we have actually seen in their response, it certainly looks that way.

Danika

Before I was not able to log in at all this last Friday, was having the ping jumping from low 100ms to 400ms+ for a while, then going over 900ms+. I guess that was the beginning of the end...

might be time to find a new game...

NeoXanthus

lordrichter wrote: »

NeoXanthus wrote: »

DDoS is nothing new and should have been planned for from the beginning. There are many ways to mitigate the attacks. One approach is to use an on-data center appliance solution like Radware or Arbor with an additional cloud based volumetric service like Prolexic Akamai or Verisign DDoS mitigation services. These solutions are not cheap but are necessary in our day.

Honestly, they have been dealing with DDoS for quite a while. Gina's statement suggests that this most recent attack has bypassed existing DDoS mitigation. Given what we have actually seen in their response, it certainly looks that way.

I assume Zenimax has something in place for SYN attacks (within the pipe size), Slowloris, and other attacks that and on premises device can defend against. The problem is when the attack is bigger than your internet pipes. A solution like Prolexic or Verisign cloud based solution uses GRE tunnels to divert good traffic back to your edge routers with a better matched eBGP announced prefix. For example, you announce a 23-bit to your internet edge and a Cloud based DDoS mitigation solution announces 2x 24-bit routes within your 23bit ARIN range. Because they have a better matched advertisement to the eBGP world traffic will flow to them and is scrubbed and returned to you over GRE tunnels. Most good Cloud based DDoS mitigation solutions have an unfathomable amount of traffic in the multi-terabit range so it is not possible for them to be hit with volumetric attack that cannot be filtered. These services are not free in fact they are very expensive but very needed.

Mojmir

NeoXanthus wrote: »

lordrichter wrote: »

NeoXanthus wrote: »

DDoS is nothing new and should have been planned for from the beginning. There are many ways to mitigate the attacks. One approach is to use an on-data center appliance solution like Radware or Arbor with an additional cloud based volumetric service like Prolexic Akamai or Verisign DDoS mitigation services. These solutions are not cheap but are necessary in our day.

Honestly, they have been dealing with DDoS for quite a while. Gina's statement suggests that this most recent attack has bypassed existing DDoS mitigation. Given what we have actually seen in their response, it certainly looks that way.

I assume Zenimax has something in place for SYN attacks (within the pipe size), Slowloris, and other attacks that and on premises device can defend against. The problem is when the attack is bigger than your internet pipes. A solution like Prolexic or Verisign cloud based solution uses GRE tunnels to divert good traffic back to your edge routers with a better matched eBGP announced prefix. For example, you announce a 23-bit to your internet edge and a Cloud based DDoS mitigation solution announces 2x 24-bit routes within your 23bit ARIN range. Because they have a better matched advertisement to the eBGP world traffic will flow to them and is scrubbed and returned to you over GRE tunnels. Most good Cloud based DDoS mitigation solutions have an unfathomable amount of traffic in the multi-terabit range so it is not possible for them to be hit with volumetric attack that cannot be filtered. These services are not free in fact they are very expensive but very needed.

sounds like they should allocate some of that lawsuit money into this,wont happen though. finance wont approve it since IT isnt a revenue generating dept of the company.

Elsonso

Mojmir wrote: »

NeoXanthus wrote: »

lordrichter wrote: »

NeoXanthus wrote: »

DDoS is nothing new and should have been planned for from the beginning. There are many ways to mitigate the attacks. One approach is to use an on-data center appliance solution like Radware or Arbor with an additional cloud based volumetric service like Prolexic Akamai or Verisign DDoS mitigation services. These solutions are not cheap but are necessary in our day.

Honestly, they have been dealing with DDoS for quite a while. Gina's statement suggests that this most recent attack has bypassed existing DDoS mitigation. Given what we have actually seen in their response, it certainly looks that way.

I assume Zenimax has something in place for SYN attacks (within the pipe size), Slowloris, and other attacks that and on premises device can defend against. The problem is when the attack is bigger than your internet pipes. A solution like Prolexic or Verisign cloud based solution uses GRE tunnels to divert good traffic back to your edge routers with a better matched eBGP announced prefix. For example, you announce a 23-bit to your internet edge and a Cloud based DDoS mitigation solution announces 2x 24-bit routes within your 23bit ARIN range. Because they have a better matched advertisement to the eBGP world traffic will flow to them and is scrubbed and returned to you over GRE tunnels. Most good Cloud based DDoS mitigation solutions have an unfathomable amount of traffic in the multi-terabit range so it is not possible for them to be hit with volumetric attack that cannot be filtered. These services are not free in fact they are very expensive but very needed.

sounds like they should allocate some of that lawsuit money into this,wont happen though. finance wont approve it since IT isnt a revenue generating dept of the company.

I would imagine that the bulk of the proceeds from the lawsuit, assuming they even have the money, will go to the studios working on VR, not to MMO studios like ZOS.

CaffeinatedMayhem

Thanks for being honest with us. It's much less frustrating to know what's going on, even if we can't have all the details.

TempusFugit

Came across related info (quoted below) that was written by Jonathan Leack, Executive Editor for the GameRevolution gaming site that is dated Friday, July 28 2017. ( http://www.gamerevolution.com/news/343117-elder-scrolls-onlines-servers-struggling-due-ddos-attacks) and thought to share...

The Elder Scrolls Online’s Servers Are Struggling Due to DDoS Attacks

During the past few days The Elder Scrolls Online players have been reporting issues with server instability. The issues appear to have ramped up during the past 48 hours, causing discussion on social media and forums.

According to developer ZeniMax Online Studios this has been caused by a series of DDoS attacks. The news was confirmed by admin Gina Bruno, who shared the following:

We’d like to briefly explain what’s been going on with the North American server stability lately. In short, we’ve been getting hit with many repeated DDoS attacks (Distributed Denial of Service). When this happens, it floods our service and prevents you from being able to log in, stay connected, or even group properly. Sometimes they only affect a handful of players, and other times they can cause a much greater and widespread impact. Either way, it’s a frustrating situation.

While we do have protection in place (which we obviously can’t discuss publicly), it hasn’t been enough. We’ve been working hard to implement additional security measures to make sure you’re able to log in – and stay logged in – whenever you want to play, and aren’t interrupted. We hope you understand why we’re being somewhat vague and not discussing specifics here. We take this issue very seriously, though, and are working hard to ensure everyone has the best experience possible.

Thank you for your continued support and patience.

The attack has arrived in the middle of a major PvP event for the game called Midyear Mayhem. Many players have reported losing progress during participation, summoning frustration among the playerbase.

It is unclear when the DDoS will subside, but ZeniMax has historically been effective in thwarting attacks.

These attacks are among many in 2017. Earlier in the year DDoS was a huge concern of Ubisoft when it launched For Honor, and we've seen seen mega companies such as Blizzard and Square Enix affected on a routine basis.

Best Regards,

Dracofyre

Neverwinters have that DDoS too, it was little over a year ago. lasted for 4-5 days. it was a single guy bragging in other website forum. calling himself as "God of Neverwinter" then he was never heard again,

jfdaly

When can we expect the log on problem to be resolved ?

CiliPadi

jfdaly wrote: »

When can we expect the log on problem to be resolved ?

When Star Citizen is finally released ... LOL :P

Dracofyre

i guess that we need to isolate North Korea's IP, possible for hackers, since they are technically still in war, so they may have cyber warriors.
back in 2002, i have heard from other players from different game, it was before "Call of Duty", it was other shooter game in open beta, they had elite marine sniper team to play and testing, and they all got one shot killed by a single player, they were all laid prone, near impossible, and in that dense snow blizzard, zero visibility sight, they were all shot one by one without any missed shot.

they wonder how it happen and how that player knew how many guys in the team, so they investigate,. someone used cheat code, devs had to shut down that game and lost investments and went broke since hackers use more cheat codes, hackers dont care about other players or dev teams who make games.

hackers are there to cheat and to destroy many games as possible before devs shutting down, they used bots for money and upped many items on the market.

so many hackers in different form of revenge or make money at all cost, or exploitings, or in attempts to shutting down as "DDoS", or to grief in some ways.

they were treated as rockstars in other countries.

Banana

Mine seems back to normal at the moment. Funny how the pvp event ended not long before. How are others connection?

Elsonso

Banana wrote: »

Mine seems back to normal at the moment. Funny how the pvp event ended not long before. How are others connection?

There are three issues going on all at the same time.

1. The DDoS. It is really happening, and they are working to mitigate it.
2. The Event. This is bringing people into Cyrodiil, which causes crowding and really highlights...
3. The Lag. The ongoing lag problem in Cyrodiil, which includes disconnects and loading screens, is made worse by both of the above.

It is really #3 that needs to be fixed, but even if they had already fixed #3, the first two would be enough to get people to complain about performance, dropped connections, and loading screens.

Dracofyre

still not able to log, and hadnt craft a writ today yet, and got less than 5 hrs before next writ are up next, plus my researching are on the clock and need scroll timers to save me a day closer.
tired of checking, relog, refresh launcher nearly all day.

i barely played other game and i get sleepy within few minutes. i just had nap while waiting.