SERVER CRASH AGAIN?!

kaorunandrak · August 2015

yeah it was late but I was asleep through the fun.....so shut up *runs away flailing arms and crying hysterically*

Annalyse · August 2015

jonjees wrote: »

for eso plus members, it is still a subscription.
for whatever reasons eso is down, compensation needs to be given to subscription users.
in this era where redundancy, high availability and business continuity are all viable solutions, this should not be happening.

likewise down for patching, crashes and what nots, the subs needs to be compensated, because they have paid for the service.

I don't understand this reasoning at all.

I pay a monthly fee for the use of my internet. Occasionally, something happens and I lose internet for a short period of time. Am I annoyed? Of course. But do I call my ISP and demand they reimburse me for that period in which I was unable to use the internet? No.

This is not something that was in their control, and they worked all night to get it fixed as fast as they could. Just be grateful that there are staff for this game that care about things like this.... I've seen a lot of games where things would take days to get rectified because of the lesser quality of support staff.

TalonShina · August 2015

Annalyse wrote: »

jonjees wrote: »

for eso plus members, it is still a subscription.
for whatever reasons eso is down, compensation needs to be given to subscription users.
in this era where redundancy, high availability and business continuity are all viable solutions, this should not be happening.

likewise down for patching, crashes and what nots, the subs needs to be compensated, because they have paid for the service.

I don't understand this reasoning at all.

I pay a monthly fee for the use of my internet. Occasionally, something happens and I lose internet for a short period of time. Am I annoyed? Of course. But do I call my ISP and demand they reimburse me for that period in which I was unable to use the internet? No.

This is not something that was in their control, and they worked all night to get it fixed as fast as they could. Just be grateful that there are staff for this game that care about things like this.... I've seen a lot of games where things would take days to get rectified because of the lesser quality of support staff.

All I got to say is YES

UrQuan · August 2015

SeptimusDova wrote: »

UrQuan do you work backhaul?

No, I don't even work in IT anymore - until recently I worked in IT doing banking software development & deployment. All of my experience with server issues was through my colleagues who were responsible for the servers (with the occasional "all hands on deck" moment when almost every warm body in IT was doing something to help put out fires or keep management and the executives informed about what was going on). Now I do data analytics. It's less stressful

Really interesting post though @SeptimusDova !

Inactive Account · August 2015

This may not be totally over yet; do be surprised if the servers go down again sometime in the near future.

The system is still weak...according to...https://downdetector.com/status/level3/map/

Read the comments at the bottom....

SeptimusDova · August 2015

UrQuan wrote: »

SeptimusDova wrote: »

UrQuan do you work backhaul?

No, I don't even work in IT anymore - until recently I worked in IT doing banking software development & deployment. All of my experience with server issues was through my colleagues who were responsible for the servers (with the occasional "all hands on deck" moment when almost every warm body in IT was doing something to help put out fires or keep management and the executives informed about what was going on). Now I do data analytics. It's less stressful

Really interesting post though @SeptimusDova !

Thank you for the reply UrQuan

This "solar" event is going to increase in frequency. As Microwave density increases and covers longer paths. Older analog microwave systems "Non QAM-64" remain in the network they act as feedpoints as well. Older waveguide systems with exposed waveguide on tracks outside are capable of collecting stray RF Harmonics and "solar noise" Analog was more tolerant of white noise and variation. Versus digital systems .

All it takes is one access point for an entire circuit to feel the effects.

While higher order modulation rates are able to offer much faster data rates and higher levels of spectral efficiency for the radio communications system, this comes at a price. The higher order modulation schemes are considerably less resilient to Solar noise and interference (older spark systems).

As a result of this, many radio communications systems now use dynamic adaptive modulation techniques (TDM). They sense the channel conditions and adapt the modulation scheme to obtain the highest data rate for the given conditions. As signal to noise ratios decrease errors will increase along with re-sends of the data, thereby slowing throughput. By reverting to a lower order modulation scheme the link can be made more reliable with fewer data errors and re-sends. Higher interference results in many resend's and a higher BER which can result in a QAM scale back for example QAM64 to QAM 32. Or QAM 256 to QAM-64.

Note a System can only scale back to the networks lowest QAM Scheme. So if you have a DS sytem that can downscale to QAM 32, but the network only allows for QAM-64. The Site that is QAM32 need's to have the configuration set to not allow QAM-32 attempts. If not configured right; solar noise can cause the radio to downscale to 32 whilst the rest of the network stays 64. The end result a red alert at the NOC.

UrQuan · August 2015

angelyn wrote: »

https://youtube.com/watch?v=AvyVNmhT4V0

I would eat Moss alive

Was that a reference to the episode where Moss mistakenly answers an ad from an aspiring cannibal looking for someone to eat?

UrQuan · August 2015

SeptimusDova wrote: »

UrQuan wrote: »

SeptimusDova wrote: »

UrQuan do you work backhaul?

No, I don't even work in IT anymore - until recently I worked in IT doing banking software development & deployment. All of my experience with server issues was through my colleagues who were responsible for the servers (with the occasional "all hands on deck" moment when almost every warm body in IT was doing something to help put out fires or keep management and the executives informed about what was going on). Now I do data analytics. It's less stressful

Really interesting post though @SeptimusDova !

Thank you for the reply UrQuan

This "solar" event is going to increase in frequency. As Microwave density increases and covers longer paths. Older analog microwave systems "Non QAM-64" remain in the network they act as feedpoints as well. Older waveguide systems with exposed waveguide on tracks outside are capable of collecting stray RF Harmonics and "solar noise" Analog was more tolerant of white noise and variation. Versus digital systems .

All it takes is one access point for an entire circuit to feel the effects.

While higher order modulation rates are able to offer much faster data rates and higher levels of spectral efficiency for the radio communications system, this comes at a price. The higher order modulation schemes are considerably less resilient to Solar noise and interference (older spark systems).

As a result of this, many radio communications systems now use dynamic adaptive modulation techniques (TDM). They sense the channel conditions and adapt the modulation scheme to obtain the highest data rate for the given conditions. As signal to noise ratios decrease errors will increase along with re-sends of the data, thereby slowing throughput. By reverting to a lower order modulation scheme the link can be made more reliable with fewer data errors and re-sends. Higher interference results in many resend's and a higher BER which can result in a QAM scale back for example QAM64 to QAM 32. Or QAM 256 to QAM-64.

Note a System can only scale back to the networks lowest QAM Scheme. So if you have a DS sytem that can downscale to QAM 32, but the network only allows for QAM-64. The Site that is QAM32 need's to have the configuration set to not allow QAM-32 attempts. If not configured right; solar noise can cause the radio to downscale to 32 whilst the rest of the network stays 64. The end result a red alert at the NOC.

Again, very interesting!

BlueGreenMikey · August 2015

Annalyse wrote: »

I don't understand this reasoning at all.

I pay a monthly fee for the use of my internet. Occasionally, something happens and I lose internet for a short period of time. Am I annoyed? Of course. But do I call my ISP and demand they reimburse me for that period in which I was unable to use the internet? No.

But there's a major, major distinction here.

"Occasionally, something happens and I lose internet for a short period of time."

Yeah, I wouldn't complain about that either. I wouldn't demand money back for that either.

ESO's servers go down with some regularity, however. Players repeatedly complain about issues of getting booted in the middle of playing the game. Though it has been more reliable lately, weekend outages at peak popularity times still happen. And then there's the tone-deaf "customer service" response of blaming everyone else---Microsoft/Sony, your ISP, you---rather than taking responsibility for their own actions unless pressed.

(Also, most of the Internet was working last night. When ESO went down, many of us simply went and did other stuff. I watched Netflix, which worked without a hitch. Literally zero of the web technologies I tried last night other than ESO were broken. I don't know if it is because other companies anticipate problems better and come up with redundancies and plans. But given that and everything that we know about ESO and the lack of server reliability, I'd say that this has more to do than with just bad luck last night.)

I personally wouldn't demand money back or anything for what happened last night, or probably ever. I'm not a subscriber, and I feel like I've already gotten my $60 worth out of the game, so everything is gravy now. But it's not ridiculous, especially for subscribers, to see yet another outage (whether it be last night or the next one) as the last straw. Our copies of ESO might as well be paperweights every time that ESO has an outage.

For many of us (me included), ESO is the most unreliable technology we use, and there's nothing else that even comes close.

Llafn_Nos · August 2015

Gina thanks for keeping us all entertained last night , you are awesome.

However I have a question. We know that the issue was level 3's and not yours, they are a NA service provider. So why did the European server go down? Is it because it's in the U.S. Of A?

UrQuan · August 2015

Llafn_Nos wrote: »

Gina thanks for keeping us all entertained last night , you are awesome.

However I have a question. We know that the issue was level 3's and not yours, they are a NA service provider. So why did the European server go down? Is it because it's in the U.S. Of A?

Level3 operates infrastructure in various parts of Europe as well. Large portions of Europe (including portions of Germany, where the EU servers are located) were affected. Of course, I've heard that some people were able to log on to the EU servers during the outage, so it may have just been that most of the paths to the server were affected, but not all.

cjthibs · August 2015

BlueGreenMikey wrote: »

Annalyse wrote: »

I don't understand this reasoning at all.

I pay a monthly fee for the use of my internet. Occasionally, something happens and I lose internet for a short period of time. Am I annoyed? Of course. But do I call my ISP and demand they reimburse me for that period in which I was unable to use the internet? No.

But there's a major, major distinction here.

"Occasionally, something happens and I lose internet for a short period of time."

Yeah, I wouldn't complain about that either. I wouldn't demand money back for that either.

ESO's servers go down with some regularity, however. Players repeatedly complain about issues of getting booted in the middle of playing the game. Though it has been more reliable lately, weekend outages at peak popularity times still happen. And then there's the tone-deaf "customer service" response of blaming everyone else---Microsoft/Sony, your ISP, you---rather than taking responsibility for their own actions unless pressed.

(Also, most of the Internet was working last night. When ESO went down, many of us simply went and did other stuff. I watched Netflix, which worked without a hitch. Literally zero of the web technologies I tried last night other than ESO were broken. I don't know if it is because other companies anticipate problems better and come up with redundancies and plans. But given that and everything that we know about ESO and the lack of server reliability, I'd say that this has more to do than with just bad luck last night.)

I personally wouldn't demand money back or anything for what happened last night, or probably ever. I'm not a subscriber, and I feel like I've already gotten my $60 worth out of the game, so everything is gravy now. But it's not ridiculous, especially for subscribers, to see yet another outage (whether it be last night or the next one) as the last straw. Our copies of ESO might as well be paperweights every time that ESO has an outage.

For many of us (me included), ESO is the most unreliable technology we use, and there's nothing else that even comes close.

You must not use many technologies or understand how the internet works.

I've been playing since beta...and nope, it's not really all that unreliable for an MMO. I also work in enterprise-level IT, managing clusters of compute nodes used for research. Stuff happens that just isn't in your control sometimes. Nobody advertises 100% uptime, because it isn't really do-able. It's always 99 and some fraction, which I'd say ZOS has delivered on.

I am pretty sure the total unplanned downtime experienced does not exceed 3.65 days in the past year, but even if it were 3.65 entire days would still put us at 99% uptime. Which is respectable for an operation this large.

If there was a routing outage...no amount of redundancy is necessarily going to save you. It really depends on where the outage happened. Back in the day the 'cloud' was a dirty word in networking not some awesome thing like it is now. It referred to the part of the network routes that were out of your control. Outside your firewall, cloud...doesn't matter how redundant your systems are if it hits the fan out there, you might be alright, you might not.

As far as your -luck- with the outage. Good for you, you were relatively unaffected. Look around the internet though, it was a major deal all over the place.

Spottswoode · August 2015

Pendrillion wrote: »

Is it just me or do we look at a major Cyber Attack? I know you cannot just take guesses. But what I see when I see the outtages is that all major centers of US have been affected. Also some that harbor military installations.

But of course you would see the outtage best where there is intense use and network infrastructure.

https://imgur.com/a/9fNKD#0

@Pendrillion
That's a possiblity but it's improbable. Don't assume that just because military installations might be "affected" that it would be an effective attack. If the military was the target, Level3 was a poor point of attack. The Pentagon is filled to the brim with people more paranoid than I'll ever be and many of whom give reports directly to congressional committees and military committees who basically sit around and think of horrible things that might happen and effective means to deal with them. (Not that the military is immune to the Peter Principle or the Dilbert Principle.) Suffice it to say, that any network outage is likely to have minimal effect on military operations. Soldiers probably weren't able to watch Netflix, though.
Now that doesn't mean this wasn't an attack: the economic impact is quite clear. But this did happen in the middle of the night over here in the US (as opposed to the when the NYSE was open, which would have done substantially more economic damage) and didn't prevent a lot of major services from operating. Most banking systems still operated. (Also, why are flippin' Pet Rescue Saga and Runescape listed on that page, but we aren't?)
Natural disaster/accident seems much more likely.

SeptimusDova · August 2015

Kaithuzar said

Q. I've been working in industry a long time & while I understand some of what you are saying can you clarify a few things?
Personally I can't remember a single outage on the carrier level due to "solar disturbance".
Are there notes on this happening or happening frequently at various locations?
You state "are disruptive to backhaul circuits", are there not multiple pathways & failovers? Is there no redundancy? (both on the carrier side & ZOS's side)

Traditionally, many larger companies have multiple carriers so if Verizon goes down, AT&T is still up & vice versa; you just have to reroute the traffic & everything still works.

A. I cannot speak for the Zos side.

I can speak for the co located sites and ATC sites and other .gov located sites. Microwave sites typically have a set path and the you cannot connect multiple radios to the same hardline or waveguide*Edited to add, Not with out some rather specialized equipment.* And no guarantee it will work**. Solar affects the Radio sites. It affects the receive frontend and the AGC of the equipment. As spectrum is saturated with white noise approaching S9 digital receivers go into a polling mode to make up for increased BER. If a dirty signal is generated, then a dirty signal is sent. GIGO.

I have to be careful about violating and still pending NDA's and Security issues with PKI. So bear with me.
colo microwave sites have fixed and limited resources. Racks, Power, Backup power, battery banks, inverters and most importantly Limited Towers. Wave guide and hardline trays are limited as well.
Many Colo facilities for AT&T are actually in trust to Chase Manhattan bank. And as such you are limited to any building modifications you can make to expand floor space. I worked those sites as well as ATC sites and later on FAA gov sites. Ownership is a murky river. Who owns what and ties into what can be a mess. I have seen 2.5 GHz analog Collins equipment (Circa 1959) alongside a newer Alactel system followed by an older Nortel RD5 system. You never know what is in a microwave site until you open that door.

Q. :"The OC-whatever & DS-whatever lines that are running underneath are just cabling right? So the cabling shouldn't be affected, it should just be the circuitry in the telco room at the colo & that room should be shielded by being a metal room or something that would not be conductive/affected by the "solar disturbance"."

A. DS in this case is across a microwave path instead of a copper or fiber line. We do have items we can place "In-Line" that help reduce the effect. What we cannot fix is the Ionosphere saturation. Also waveguide and hardline systems require maintenance and on some circuits that 5 minutes a year. Example a 911CLN contracted for 99.999% uptime. I had this issue at the Little Maria site just before I retired the entire site went dark. 64 hours later we had it up and running again. At a cost of 4.5 mil to AT&T reason. Sabotage to the Generator system and the site as well.We had to replace all of the equipment.

This picture should help clarify any questions.

I managed this site along with many, many more across 9 states it can get tiring. When you have issues with Microwave backhaul.

BlueGreenMikey · August 2015

cjthibs wrote: »

I am pretty sure the total unplanned downtime experienced does not exceed 3.65 days in the past year, but even if it were 3.65 entire days would still put us at 99% uptime. Which is respectable for an operation this large.

There are people who play this game other than PC users now. I'd be shocked if it wasn't more than 3.65 entire days for console users just since the XBOX/PS4 launches.

Ignoring the rest of your unnecessary, unjustified, and unwarranted condescension.

gard · August 2015

It cracks me up when people attempt to validate their opinions by saying, "I've been playing since beta"

It's a fact that internet connectivity is susceptible to failure.
Lets accept that, and forget for a moment that datacenters generally peer with multiple backbone providers and can manage traffic flow among them relatively easily.

What's yet to be explained is how a loss of internet connectivity can lead to a server crash, much less require a system restart.
I have hundreds of linux servers on the east and west coasts. Didn't have to reboot a single one.

Don't get me wrong, I'm not angry - I accept that things happen. I don't even feel that I'm owed an explanation.
I'm just curious as to what really happened, because what I've heard so far doesn't add up.

me_ming · August 2015

BlueGreenMikey wrote: »

Annalyse wrote: »

I don't understand this reasoning at all.

I pay a monthly fee for the use of my internet. Occasionally, something happens and I lose internet for a short period of time. Am I annoyed? Of course. But do I call my ISP and demand they reimburse me for that period in which I was unable to use the internet? No.

But there's a major, major distinction here.

"Occasionally, something happens and I lose internet for a short period of time."

Yeah, I wouldn't complain about that either. I wouldn't demand money back for that either.

ESO's servers go down with some regularity, however. Players repeatedly complain about issues of getting booted in the middle of playing the game. Though it has been more reliable lately, weekend outages at peak popularity times still happen. And then there's the tone-deaf "customer service" response of blaming everyone else---Microsoft/Sony, your ISP, you---rather than taking responsibility for their own actions unless pressed.

(Also, most of the Internet was working last night. When ESO went down, many of us simply went and did other stuff. I watched Netflix, which worked without a hitch. Literally zero of the web technologies I tried last night other than ESO were broken. I don't know if it is because other companies anticipate problems better and come up with redundancies and plans. But given that and everything that we know about ESO and the lack of server reliability, I'd say that this has more to do than with just bad luck last night.)

I personally wouldn't demand money back or anything for what happened last night, or probably ever. I'm not a subscriber, and I feel like I've already gotten my $60 worth out of the game, so everything is gravy now. But it's not ridiculous, especially for subscribers, to see yet another outage (whether it be last night or the next one) as the last straw. Our copies of ESO might as well be paperweights every time that ESO has an outage.

For many of us (me included), ESO is the most unreliable technology we use, and there's nothing else that even comes close.

Yeah, basically you saying "Literally zero of the web technologies I tried last night other than ESO were broken." is pretty hilarious, not because it's not true. I don't doubt that everything else was working on your "web technologies" other than ESO, but the fact that you don't get why ESO wasn't working is pretty funny. lol. Anyways, I'm pretty sure people [in this thread] have already extensively explained why you ESO was down despite the fact that you can go online, it's up to you if you want to read and at least try to understand.

cjthibs · August 2015

BlueGreenMikey wrote: »

cjthibs wrote: »

I am pretty sure the total unplanned downtime experienced does not exceed 3.65 days in the past year, but even if it were 3.65 entire days would still put us at 99% uptime. Which is respectable for an operation this large.

There are people who play this game other than PC users now. I'd be shocked if it wasn't more than 3.65 entire days for console users just since the XBOX/PS4 launches.

Ignoring the rest of your unnecessary, unjustified, and unwarranted condescension.

I can't speak for console users since I'm not one.
I do remember hearing that some of those outages were related to their respective console services. (XBL/PSN)

I probably also should've mentioned that, as far as I'm aware, ZOS makes no such guarantee of uptime anyhow. Which makes all of this purely academic.

Disagreement ≠ Condescension.

ColoursYouHave · August 2015

I, for one, am outraged, and fully expect compensation.

By my estimations, the server was down between 11PM and 6AM, for a total of around 7 hours downtime. Despite the fact that I was asleep for a large portion of that downtime, I was still unable to use a service that I had paid for. At $15 per month, I end up paying about $.50 per day, or a little over $.02 per hour, so I demand you reimburse me the $.14 that I lost during this downtime in order to make things right!

And yes, that 14 cents really is important to me, and I'm not just saying all of this as a passive-aggressive way to express my frustrations towards the company for something they could not have avoided!

Ravici · August 2015

You guys (and gals, *fluttering eyelashes @ Gina*), have no idea about frustration!

So, here's me last night, can't play, watching this thread and in the early hours trying to keep up with the darn thing. It may have slowed down later on but for the first couple hours, the post counts were nuts. Refresh the page to see new posts just to find you had two new pages of posts to get through

So I'm a hexbox player and of course that means we can play without giving our email details to one of my most loathed companies right now so they don't bombard me with junk that I want absolutely nothing to do with... problem!

So, here's the thing, until said company has your details, you are not capable (allowed doesn't even come into it) of posting in said hyperactive thread.

Not only could I not play the game, I couldn't even communicate on the forums! but... can it get any worse? Of course it can!

Here's the extra kicker. Not only were the game servers down, but the signup/account servers were too. This obviously for those not keeping up with my lame excuse of a post so far, will by now be realising that in order to post about the problem, I had to get past the problem in the first place. Talk about chicken and egg

(Ok, you put a colon followed be a capital D, worldwide general term for a laughing smiley and it gives you... dissappointed???? sheesh!)

So, after all that, Hi there from me. I promised I would never touch another Elder Scroll game after the abortion that was Skyrim, killed off what was once a favourite series but is actually enjoying this latest offering (even if I do *** and moan about it because in places it still sucks hehe) enough to maybe come back to the fold.

Oh, and anyone saying this wasn't their fault, the buck has to stop and this is where it does. Enough with this passing the blame game. For those saying the internet was effected worldwide... get a grip. That's just marketing speak for a get out of jail card. For those talking about there being only the one game server and the European one being a sham... the game servers may be in different places, but we all need to go through an initial account server and it would be beyond silly to have two of those running all the time

You can put your tin foil hats in the bin on the way out

Hi all *waves like a loony*

danew6 · August 2015

On the plus side the game kicking me out last night meant I went to bed earlier.

SeptimusDova · August 2015

I am still trying to wake up.

stevenbennett_ESO · August 2015

I can't believe I ate read the whole thread…

Man… I go to sleep early for just one night instead of playing ESO late as I usually do, and *this* thread happens. Why do I always miss all the awesomeness?

Still, huge kudos to Gina especially for herding the cats in this thread and keeping things entertaining and fairly light in tone. As long as this thread was, it was still a highly entertaining read…

Spottswoode · August 2015

SeptimusDova wrote: »

I am still trying to wake up.

Still coherent.

Elsonso · August 2015

gard wrote: »

What's yet to be explained is how a loss of internet connectivity can lead to a server crash, much less require a system restart.

I have hundreds of linux servers on the east and west coasts. Didn't have to reboot a single one.

The servers probably did not crash. They were probably all running normally and the monitors were showing a significant decrease in the number of players that were playing as each player lost connectivity.

They rebooted the servers as part of manual recovery step, and I would guess that it was due to thousands of players that were suddenly disconnected in the middle of whatever they were doing. The server reboot probably acts to clean things up so that when players log back in they do not encounter any problems.

SeptimusDova · August 2015

Spottswoode wrote: »

SeptimusDova wrote: »

I am still trying to wake up.

Still coherent.

I am going to google and order that.
Edited to add. Just ordered 5 pound bag. nice price.

Greeniewolfub17_ESO · August 2015

BlueGreenMikey wrote: »

cjthibs wrote: »

I am pretty sure the total unplanned downtime experienced does not exceed 3.65 days in the past year, but even if it were 3.65 entire days would still put us at 99% uptime. Which is respectable for an operation this large.

There are people who play this game other than PC users now. I'd be shocked if it wasn't more than 3.65 entire days for console users just since the XBOX/PS4 launches.

Ignoring the rest of your unnecessary, unjustified, and unwarranted condescension.

you're the one being condescending. He was just giving you the information you needed to stop being ignorant and being very tolerant and patient about it. Much more so than you deserve.

stevenbennett_ESO · August 2015

Oh, and for all those posting their confusion about how server X could be affected, etc… In this day an age, network topology is extremely complex even when you're talking about small systems. Databases often run on separate instances in the cloud than the servers which need to access them, and that communication happens through the internet. When you start getting into multiple interacting servers, the communications between each server happens through the internet. Often just the communications channels are complex, with server group A communicating to a bridge which is tunneling through the internet to server group B, C, D, etc. And that happens even with fairly simple websites and the like. There's a LOT of communication across the net which needs to happen or things stop working. And often, if those communications fail for more than just a few seconds, it causes program crashes or abnormal terminations, which then cascades to other failures.

It's often possible (and common) to have redundant backup communications, but when the problem happens in a major internet backbone, there's a decent chance your backups go through the same backbone -- it's out of your control if that happens. Usually the backbones have some redundancy, but the redundant circuits can get overloaded and drop a lot of traffic -- something which has low traffic needs might work fine with a bit of a slowdown, and if you've got a sufficiently distributed system, you might still work because your clients can go to alternate data servers, but when you get into MMOs, that's another animal.

MMO network topology, in particular, is a massive spiderweb of cross communications, some between local servers, some across the net to remote servers, some to databases which may be local or remote, some to admin / login servers which may be local / remote, with status monitoring, and synchronization issues which make normal network administrators shy away in terror at the complexity of it all. And unlike, say, a streaming server, or your average website, ALL of that communication needs to run at a fairly consistent high speed or the MMO fails.

Fortunately, an MMO is a game, not something critical. People will survive an 8 hour outage. OTOH, I'm sure there are a lot of companies who are dealing with some very serious issues this morning due to the outage last night.

QuebraRegra · August 2015

I have responsibility for services domestic and foreign, with a significant portion being provided via LEVEL3, and AT&T carriers.

Total number out outages/interruptions to services last night... ZERO. This is very notable, as while most of the network connections are designed with high availability/redundancy in mind, some of the links are single homed, and none appeared to have been affected (even LEVEL3 services in the Houston area, etc.).

Is it possible that a network interruption could have impacted server performance?... Possibly, but I'd take a serious look at how my processes function in relation to network continuity were that the case.

That stated, Gina's participation was appreciated none the less.

Ratbert · August 2015

QuebraRegra wrote: »

I have responsibility for services domestic and foreign, with a significant portion being provided via LEVEL3, and AT&T carriers.

Total number out outages/interruptions to services last night... ZERO. This is very notable, as while most of the network connections are designed with high availability/redundancy in mind, some of the links are single homed, and none appeared to have been affected (even LEVEL3 services in the Houston area, etc.).

Is it possible that a network interruption could have impacted server performance?... Possibly, but I'd take a serious look at how my processes function in relation to network continuity were that the case.

That stated, Gina's participation was appreciated none the less.

I have responsibilities for Level 3 communications connections for my company out of Dallas. I can tell you we WERE affected by this outage. Some services were available to us and others were not. Many websites were available to us to browse and many weren't. Our hosted exchange was one of those services affected and unreachable, yet I could browse Bestbuy.com for a new TV but not get to facebook.