SERVER CRASH AGAIN?!

SLy_Kyti

ZOS_MichaelServotte wrote: »

@Runs : All major updates with information relaterd to this maintenance have been devtracked, this way players can find them easily right here.
Also, we are making sure we are using the maximum of tools at our disposal to let players know when the maintenance is over: Facebook, Google+, Twitter, Service Alerts & the Service Alert messages on the game launcher.

Thank you @ZOS_MichaelServotte and @ZOS_GinaBruno for being the watch keepers through the night (and morning). It's reassuring to have Devs hang with us.

SLy_Kyti

NadiusMaximus wrote: »

Relaterd a word?

It is if you are a French speaking Dev. I figured he meant "related".

There was something going around the internet awhile that showed a person could reduce the letters in a word and still convey their meaning, much like text speak but on a whole new level. Pretty interesting but I'm sure the thought of it gives English grammar teachers nightmare's.

Enodoc

SLy_Kyti wrote: »

Pretty interesting but I'm sure the thought of it gives English grammar teachers nightmare's.

Was that intentional?

wraith808

ZOS_GinaBruno wrote: »

CheeseDivinity wrote: »

ZOS_GinaBruno wrote: »

Nah, we got ESO Live tomorrow! Hope everyone can tune in.

I keep hearing about ESO live, and it sounds quite informative whatever it is. How do we tune in? Is it like a podcast or something?

We're on the air every other Friday at 4:00pm EDT on our ZOS Twitch channel:

To kill some time, you can watch some past broadcasts! If you click on the Zenimax Online Studios link toward the top, it'll take you to the page (if I try to link it here, it just goes straight to the channel).

@ZOS_GinaBruno - it doesn't do that if you link it to text- it only does that if you put the link in bare.

SLy_Kyti

Belidos wrote: »

ViscousSummer88 wrote: »

at the end of the day Level 3 are not the only supplier, the data centre they use is not the only data centre.

For all intents and purposes in that area Level3 IS the only supplier, they own a large majority of the Backbone network, and almost all internet traffic in that area goes through them no matter which ISP you use, it wouldn't matter whether ZOS uses AT&T, Verizon, ComCast or any other provider, they all go through Level3's network. So no they don't really have a choice in the matter.

After figuring out I wouldn't be playing ESO for awhile I turned on the Roku streaming player & TV - got a real slow connection, thought what da heck? booted up the laptop and it sat there spinning like my car does in winter. Came back to forums to see it was a Lvl 3 outrage. It messed with more than just ESO. I have Comcast in the NW and this outrage affectedly made me pick up a book. It's a nice book though!

Volkodav

Awe,cmon.Its a National outage problem.NOT ZOS this time. Give em a break,ok?

Techlisp

Funny now, sources reporting that Battlefield 4 is having login issues across all platforms.

Laplace

So... What happened exactly? I got kicked from the server last night, my guild's Skype chat said they all did too, so I just rage quit and went to bed.

Somebody spill coffee in the server room?

Sensesfail13

Miss my darkshade pledge last night and get banished cells today... most liked dungeon to most hated dungeon... screw you servers... screw you.

Enodoc

Laplace wrote: »

So... What happened exactly? I got kicked from the server last night, my guild's Skype chat said they all did too, so I just rage quit and went to bed.

Somebody spill coffee in the server room?

@Laplace Major outage at the network provider, Level3.

Aeeeek

Enodoc wrote: »

Laplace wrote: »

So... What happened exactly? I got kicked from the server last night, my guild's Skype chat said they all did too, so I just rage quit and went to bed.

Somebody spill coffee in the server room?

@Laplace Major outage at the network provider, Level3.

So NSA got carried away, again.

ColtPython

Things are working smooth after that server reset. Thanks!

Darlgon

@ZOS_JessicaFolsom , @ZOS_RichLambert

I guess Gina is gonna need to sleep in today a bit, after working all night. Make sure she has a big cup of coffee when she gets in.

ZOS_MichaelServotte wrote: »

@SLy_Kyti it's already the morning around here.

Gina will get some well deserved sleep, and I'll be there to keep you updated as soon as I got news for you. Meanwhile I'm reading this thread and I just found the best picture ever. This is disturbing.

However, I have to ask @ZOS_MichaelServotte which picture?

Soulsix

http://fortune.com/2015/07/01/cutting-internet-cables/

seems some kids love to ruin things for everyone.

TalonShina

jonjees wrote: »

for eso plus members, it is still a subscription.
for whatever reasons eso is down, compensation needs to be given to subscription users.
in this era where redundancy, high availability and business continuity are all viable solutions, this should not be happening.

likewise down for patching, crashes and what nots, the subs needs to be compensated, because they have paid for the service.

I wasn't on because my kids wanted to do a popcorn movie night, but if I had been on as an ESO plus member I would not have expected compensation for this at all

kaorunandrak

https://www.youtube.com/watch?v=QXPAZJvbH1s&feature=youtu.be

wraith808

kaorunandrak wrote: »

https://www.youtube.com/watch?v=QXPAZJvbH1s&feature=youtu.be

A bit late. We've seen that one, the 10 hour one, elevator music for 10 hours, etc.

Darlgon wrote: »

ZOS_MichaelServotte wrote: »

Gina will get some well deserved sleep, and I'll be there to keep you updated as soon as I got news for you. Meanwhile I'm reading this thread and I just found the best picture ever. This is disturbing.

However, I have to ask @ZOS_MichaelServotte which picture?

Look earlier in the thread. It's one that Gina posted. Or go to the developer tracker.

kaorunandrak

yeah it was late but I was asleep through the fun.....so shut up *runs away flailing arms and crying hysterically*

Annalyse

jonjees wrote: »

for eso plus members, it is still a subscription.
for whatever reasons eso is down, compensation needs to be given to subscription users.
in this era where redundancy, high availability and business continuity are all viable solutions, this should not be happening.

likewise down for patching, crashes and what nots, the subs needs to be compensated, because they have paid for the service.

I don't understand this reasoning at all.

I pay a monthly fee for the use of my internet. Occasionally, something happens and I lose internet for a short period of time. Am I annoyed? Of course. But do I call my ISP and demand they reimburse me for that period in which I was unable to use the internet? No.

This is not something that was in their control, and they worked all night to get it fixed as fast as they could. Just be grateful that there are staff for this game that care about things like this.... I've seen a lot of games where things would take days to get rectified because of the lesser quality of support staff.

TalonShina

Annalyse wrote: »

jonjees wrote: »

for eso plus members, it is still a subscription.
for whatever reasons eso is down, compensation needs to be given to subscription users.
in this era where redundancy, high availability and business continuity are all viable solutions, this should not be happening.

likewise down for patching, crashes and what nots, the subs needs to be compensated, because they have paid for the service.

I don't understand this reasoning at all.

I pay a monthly fee for the use of my internet. Occasionally, something happens and I lose internet for a short period of time. Am I annoyed? Of course. But do I call my ISP and demand they reimburse me for that period in which I was unable to use the internet? No.

This is not something that was in their control, and they worked all night to get it fixed as fast as they could. Just be grateful that there are staff for this game that care about things like this.... I've seen a lot of games where things would take days to get rectified because of the lesser quality of support staff.

All I got to say is YES

UrQuan

SeptimusDova wrote: »

UrQuan do you work backhaul?

No, I don't even work in IT anymore - until recently I worked in IT doing banking software development & deployment. All of my experience with server issues was through my colleagues who were responsible for the servers (with the occasional "all hands on deck" moment when almost every warm body in IT was doing something to help put out fires or keep management and the executives informed about what was going on). Now I do data analytics. It's less stressful

Really interesting post though @SeptimusDova !

Inactive Account

This may not be totally over yet; do be surprised if the servers go down again sometime in the near future.

The system is still weak...according to...https://downdetector.com/status/level3/map/

Read the comments at the bottom....

SeptimusDova

UrQuan wrote: »

SeptimusDova wrote: »

UrQuan do you work backhaul?

No, I don't even work in IT anymore - until recently I worked in IT doing banking software development & deployment. All of my experience with server issues was through my colleagues who were responsible for the servers (with the occasional "all hands on deck" moment when almost every warm body in IT was doing something to help put out fires or keep management and the executives informed about what was going on). Now I do data analytics. It's less stressful

Really interesting post though @SeptimusDova !

Thank you for the reply UrQuan

This "solar" event is going to increase in frequency. As Microwave density increases and covers longer paths. Older analog microwave systems "Non QAM-64" remain in the network they act as feedpoints as well. Older waveguide systems with exposed waveguide on tracks outside are capable of collecting stray RF Harmonics and "solar noise" Analog was more tolerant of white noise and variation. Versus digital systems .

All it takes is one access point for an entire circuit to feel the effects.

While higher order modulation rates are able to offer much faster data rates and higher levels of spectral efficiency for the radio communications system, this comes at a price. The higher order modulation schemes are considerably less resilient to Solar noise and interference (older spark systems).

As a result of this, many radio communications systems now use dynamic adaptive modulation techniques (TDM). They sense the channel conditions and adapt the modulation scheme to obtain the highest data rate for the given conditions. As signal to noise ratios decrease errors will increase along with re-sends of the data, thereby slowing throughput. By reverting to a lower order modulation scheme the link can be made more reliable with fewer data errors and re-sends. Higher interference results in many resend's and a higher BER which can result in a QAM scale back for example QAM64 to QAM 32. Or QAM 256 to QAM-64.

Note a System can only scale back to the networks lowest QAM Scheme. So if you have a DS sytem that can downscale to QAM 32, but the network only allows for QAM-64. The Site that is QAM32 need's to have the configuration set to not allow QAM-32 attempts. If not configured right; solar noise can cause the radio to downscale to 32 whilst the rest of the network stays 64. The end result a red alert at the NOC.

UrQuan

angelyn wrote: »

https://youtube.com/watch?v=AvyVNmhT4V0

I would eat Moss alive

Was that a reference to the episode where Moss mistakenly answers an ad from an aspiring cannibal looking for someone to eat?

UrQuan

SeptimusDova wrote: »

UrQuan wrote: »

SeptimusDova wrote: »

UrQuan do you work backhaul?

No, I don't even work in IT anymore - until recently I worked in IT doing banking software development & deployment. All of my experience with server issues was through my colleagues who were responsible for the servers (with the occasional "all hands on deck" moment when almost every warm body in IT was doing something to help put out fires or keep management and the executives informed about what was going on). Now I do data analytics. It's less stressful

Really interesting post though @SeptimusDova !

Thank you for the reply UrQuan

This "solar" event is going to increase in frequency. As Microwave density increases and covers longer paths. Older analog microwave systems "Non QAM-64" remain in the network they act as feedpoints as well. Older waveguide systems with exposed waveguide on tracks outside are capable of collecting stray RF Harmonics and "solar noise" Analog was more tolerant of white noise and variation. Versus digital systems .

All it takes is one access point for an entire circuit to feel the effects.

While higher order modulation rates are able to offer much faster data rates and higher levels of spectral efficiency for the radio communications system, this comes at a price. The higher order modulation schemes are considerably less resilient to Solar noise and interference (older spark systems).

As a result of this, many radio communications systems now use dynamic adaptive modulation techniques (TDM). They sense the channel conditions and adapt the modulation scheme to obtain the highest data rate for the given conditions. As signal to noise ratios decrease errors will increase along with re-sends of the data, thereby slowing throughput. By reverting to a lower order modulation scheme the link can be made more reliable with fewer data errors and re-sends. Higher interference results in many resend's and a higher BER which can result in a QAM scale back for example QAM64 to QAM 32. Or QAM 256 to QAM-64.

Note a System can only scale back to the networks lowest QAM Scheme. So if you have a DS sytem that can downscale to QAM 32, but the network only allows for QAM-64. The Site that is QAM32 need's to have the configuration set to not allow QAM-32 attempts. If not configured right; solar noise can cause the radio to downscale to 32 whilst the rest of the network stays 64. The end result a red alert at the NOC.

Again, very interesting!

BlueGreenMikey

Annalyse wrote: »

I don't understand this reasoning at all.

I pay a monthly fee for the use of my internet. Occasionally, something happens and I lose internet for a short period of time. Am I annoyed? Of course. But do I call my ISP and demand they reimburse me for that period in which I was unable to use the internet? No.

But there's a major, major distinction here.

"Occasionally, something happens and I lose internet for a short period of time."

Yeah, I wouldn't complain about that either. I wouldn't demand money back for that either.

ESO's servers go down with some regularity, however. Players repeatedly complain about issues of getting booted in the middle of playing the game. Though it has been more reliable lately, weekend outages at peak popularity times still happen. And then there's the tone-deaf "customer service" response of blaming everyone else---Microsoft/Sony, your ISP, you---rather than taking responsibility for their own actions unless pressed.

(Also, most of the Internet was working last night. When ESO went down, many of us simply went and did other stuff. I watched Netflix, which worked without a hitch. Literally zero of the web technologies I tried last night other than ESO were broken. I don't know if it is because other companies anticipate problems better and come up with redundancies and plans. But given that and everything that we know about ESO and the lack of server reliability, I'd say that this has more to do than with just bad luck last night.)

I personally wouldn't demand money back or anything for what happened last night, or probably ever. I'm not a subscriber, and I feel like I've already gotten my $60 worth out of the game, so everything is gravy now. But it's not ridiculous, especially for subscribers, to see yet another outage (whether it be last night or the next one) as the last straw. Our copies of ESO might as well be paperweights every time that ESO has an outage.

For many of us (me included), ESO is the most unreliable technology we use, and there's nothing else that even comes close.

Llafn_Nos

Gina thanks for keeping us all entertained last night , you are awesome.

However I have a question. We know that the issue was level 3's and not yours, they are a NA service provider. So why did the European server go down? Is it because it's in the U.S. Of A?

UrQuan

Llafn_Nos wrote: »

Gina thanks for keeping us all entertained last night , you are awesome.

However I have a question. We know that the issue was level 3's and not yours, they are a NA service provider. So why did the European server go down? Is it because it's in the U.S. Of A?

Level3 operates infrastructure in various parts of Europe as well. Large portions of Europe (including portions of Germany, where the EU servers are located) were affected. Of course, I've heard that some people were able to log on to the EU servers during the outage, so it may have just been that most of the paths to the server were affected, but not all.

cjthibs

BlueGreenMikey wrote: »

Annalyse wrote: »

I don't understand this reasoning at all.

I pay a monthly fee for the use of my internet. Occasionally, something happens and I lose internet for a short period of time. Am I annoyed? Of course. But do I call my ISP and demand they reimburse me for that period in which I was unable to use the internet? No.

But there's a major, major distinction here.

"Occasionally, something happens and I lose internet for a short period of time."

Yeah, I wouldn't complain about that either. I wouldn't demand money back for that either.

ESO's servers go down with some regularity, however. Players repeatedly complain about issues of getting booted in the middle of playing the game. Though it has been more reliable lately, weekend outages at peak popularity times still happen. And then there's the tone-deaf "customer service" response of blaming everyone else---Microsoft/Sony, your ISP, you---rather than taking responsibility for their own actions unless pressed.

(Also, most of the Internet was working last night. When ESO went down, many of us simply went and did other stuff. I watched Netflix, which worked without a hitch. Literally zero of the web technologies I tried last night other than ESO were broken. I don't know if it is because other companies anticipate problems better and come up with redundancies and plans. But given that and everything that we know about ESO and the lack of server reliability, I'd say that this has more to do than with just bad luck last night.)

I personally wouldn't demand money back or anything for what happened last night, or probably ever. I'm not a subscriber, and I feel like I've already gotten my $60 worth out of the game, so everything is gravy now. But it's not ridiculous, especially for subscribers, to see yet another outage (whether it be last night or the next one) as the last straw. Our copies of ESO might as well be paperweights every time that ESO has an outage.

For many of us (me included), ESO is the most unreliable technology we use, and there's nothing else that even comes close.

You must not use many technologies or understand how the internet works.

I've been playing since beta...and nope, it's not really all that unreliable for an MMO. I also work in enterprise-level IT, managing clusters of compute nodes used for research. Stuff happens that just isn't in your control sometimes. Nobody advertises 100% uptime, because it isn't really do-able. It's always 99 and some fraction, which I'd say ZOS has delivered on.

I am pretty sure the total unplanned downtime experienced does not exceed 3.65 days in the past year, but even if it were 3.65 entire days would still put us at 99% uptime. Which is respectable for an operation this large.

If there was a routing outage...no amount of redundancy is necessarily going to save you. It really depends on where the outage happened. Back in the day the 'cloud' was a dirty word in networking not some awesome thing like it is now. It referred to the part of the network routes that were out of your control. Outside your firewall, cloud...doesn't matter how redundant your systems are if it hits the fan out there, you might be alright, you might not.

As far as your -luck- with the outage. Good for you, you were relatively unaffected. Look around the internet though, it was a major deal all over the place.

Spottswoode

Pendrillion wrote: »

Is it just me or do we look at a major Cyber Attack? I know you cannot just take guesses. But what I see when I see the outtages is that all major centers of US have been affected. Also some that harbor military installations.

But of course you would see the outtage best where there is intense use and network infrastructure.

https://imgur.com/a/9fNKD#0

@Pendrillion
That's a possiblity but it's improbable. Don't assume that just because military installations might be "affected" that it would be an effective attack. If the military was the target, Level3 was a poor point of attack. The Pentagon is filled to the brim with people more paranoid than I'll ever be and many of whom give reports directly to congressional committees and military committees who basically sit around and think of horrible things that might happen and effective means to deal with them. (Not that the military is immune to the Peter Principle or the Dilbert Principle.) Suffice it to say, that any network outage is likely to have minimal effect on military operations. Soldiers probably weren't able to watch Netflix, though.
Now that doesn't mean this wasn't an attack: the economic impact is quite clear. But this did happen in the middle of the night over here in the US (as opposed to the when the NYSE was open, which would have done substantially more economic damage) and didn't prevent a lot of major services from operating. Most banking systems still operated. (Also, why are flippin' Pet Rescue Saga and Runescape listed on that page, but we aren't?)
Natural disaster/accident seems much more likely.