Oh, and for all those posting their confusion about how server X could be affected, etc… In this day an age, network topology is extremely complex even when you're talking about small systems. Databases often run on separate instances in the cloud than the servers which need to access them, and that communication happens through the internet. When you start getting into multiple interacting servers, the communications between each server happens through the internet. Often just the communications channels are complex, with server group A communicating to a bridge which is tunneling through the internet to server group B, C, D, etc. And that happens even with fairly simple websites and the like. There's a LOT of communication across the net which needs to happen or things stop working. And often, if those communications fail for more than just a few seconds, it causes program crashes or abnormal terminations, which then cascades to other failures.

It's often possible (and common) to have redundant backup communications, but when the problem happens in a major internet backbone, there's a decent chance your backups go through the same backbone -- it's out of your control if that happens. Usually the backbones have some redundancy, but the redundant circuits can get overloaded and drop a lot of traffic -- something which has low traffic needs might work fine with a bit of a slowdown, and if you've got a sufficiently distributed system, you might still work because your clients can go to alternate data servers, but when you get into MMOs, that's another animal.

MMO network topology, in particular, is a massive spiderweb of cross communications, some between local servers, some across the net to remote servers, some to databases which may be local or remote, some to admin / login servers which may be local / remote, with status monitoring, and synchronization issues which make normal network administrators shy away in terror at the complexity of it all. And unlike, say, a streaming server, or your average website, ALL of that communication needs to run at a fairly consistent high speed or the MMO fails.

Fortunately, an MMO is a game, not something critical. People will survive an 8 hour outage. OTOH, I'm sure there are a lot of companies who are dealing with some very serious issues this morning due to the outage last night.

QuebraRegra wrote: »

I have responsibility for services domestic and foreign, with a significant portion being provided via LEVEL3, and AT&T carriers.

Total number out outages/interruptions to services last night... ZERO. This is very notable, as while most of the network connections are designed with high availability/redundancy in mind, some of the links are single homed, and none appeared to have been affected (even LEVEL3 services in the Houston area, etc.).

Is it possible that a network interruption could have impacted server performance?... Possibly, but I'd take a serious look at how my processes function in relation to network continuity were that the case.

That stated, Gina's participation was appreciated none the less.

I have responsibilities for Level 3 communications connections for my company out of Dallas. I can tell you we WERE affected by this outage. Some services were available to us and others were not. Many websites were available to us to browse and many weren't. Our hosted exchange was one of those services affected and unreachable, yet I could browse Bestbuy.com for a new TV but not get to facebook.

Ratbert wrote: »

QuebraRegra wrote: »

I have responsibility for services domestic and foreign, with a significant portion being provided via LEVEL3, and AT&T carriers.

Total number out outages/interruptions to services last night... ZERO. This is very notable, as while most of the network connections are designed with high availability/redundancy in mind, some of the links are single homed, and none appeared to have been affected (even LEVEL3 services in the Houston area, etc.).

Is it possible that a network interruption could have impacted server performance?... Possibly, but I'd take a serious look at how my processes function in relation to network continuity were that the case.

That stated, Gina's participation was appreciated none the less.

I have responsibilities for Level 3 communications connections for my company out of Dallas. I can tell you we WERE affected by this outage. Some services were available to us and others were not. Many websites were available to us to browse and many weren't. Our hosted exchange was one of those services affected and unreachable, yet I could browse Bestbuy.com for a new TV but not get to facebook.

Then I guess the lesson here is to pay for and ensure diversely routed/provided carrier services? What about server virtualization at a remote or offsite?

Overall, considering that we're talking about a videogame here, I guess the service metrics are acceptable. Wouldn't fly on my network.

@ZOS_GinaBruno @ZOS_JessicaFolsom I think its happening again...

https://downdetector.com/status/level3/map/

Yeah me and my buddies were experiencing heavy lag a short while ago. Very similar before it crashed last night.

Everyone had major lag spikes in Cyro (and not the usual ones) including people getting disconnected earlier, I didn't get anything but even our newly crowned emp had issues doing anything.

Again.

nine9six wrote: »

Again.

Yep, PC NA down

For realz.

Yikes. Totally frozen in block mode.

XBOX Server is down and can't connect.

Time to take the monthly shower and think about eating my one meal per week now. Good things come from server crashes!

crasshhshhsggghhhh

up again, in, crash again

In Bleakrock and Perk Rates were exceeding 1700 causing the game to kick me to the log in screen. Oh well game is un-playable right now so guess I will watch a movie. Figured ZOS would of had a handle on this by now.