SERVER CRASH AGAIN?!

SLy_Kyti · August 2015

kaithuzar wrote: »

I just woke up to this -__________-

Go back to sleep!?!

@Xyrqes we heard you the first time buddy.

Pennthuis · August 2015

Its Friday night! & yay the week end, I'll be online on ESO on Sunday as my Khajiit, seeing were I start off after getting booted today part way through a quest lol oh the excitement.

Xyrqes · August 2015

Yeah..............just like we all have generators hooked up to our houses if the electric goes out.

well yes I do
but I also don't have 100,000s of users depending on my services

Eshelmen · August 2015

Xyrqes wrote: »

Yeah..............just like we all have generators hooked up to our houses if the electric goes out.

well yes I do
but I also don't have 100,000s of users depending on my services

This reminds me of my favourite quote from the Liar Liar movie.

Fletch: You know what I'm going to do?
Impound Attendant: What?
Fletcher: Nothing. Because if I take you to small claims court it'll just drain eight hours out of my life and even if I got the judgment you'd probably stiff me anyway. So what I'm gonna do is *** and moan like an impotent jerk and then BEND OVER AND TAKE IT UP THE TAIL PIPE.
Impound Attendant: You've been here before haven't you?

kaithuzar · August 2015

Is there going to be an announcement made when we can log back in? I show realm status as online even though it says the server's are offline in a message box pop up at the start?
Also, is PTS affected by this too? (I'm assuming so but figured I would still ask)

Epona222 · August 2015

jonjees wrote: »

Xyrqes wrote: »

LOL the only reason they carried out any maintenance was to add a login timer to the log-in screen
so you can now only attempt to log in once per minute

this isn't a bad thing, since it will relieve the login server from silly people constantly clicking 'login' when the servers are finally online but struggling to cope with the masses trying to actually play

but suggesting it is 'maintenance' hmmm

some have said you can't blame zenimax for the problem
but of course you can
any system should be prepared, and have contingency options in place
failure to do so is unprofessional
Z should have procured an alternative backup feed, in the case of this eventuallity

finally someone that understands this concept.

Yep I may have mentioned it earlier, but I once worked in an office where half of the walls were taken out by a carbomb. Due to contingency planning, the office was back up and running in a different location within a couple of hours.

I know there will be some folks on here in a minute to say how that was not possible in this case, blah blah blah

Spottswoode · August 2015

jonjees wrote: »

Spottswoode wrote: »

jonjees wrote: »

couple of hours to reboot servers .....

A statement from someone who has no experience with business networking. If Google had to reboot all of their servers, it could take a few days. (That is to say, ALL of them including the datacenters and backup datacenters.) As ZOS probably doesn't have a backup center to operate from (or backup network) this kind of reset is a lot of work.

wow really a few days?
thats when you just showed how much u know about redundancy and high availability.
maybe you need to read a paper on google hadoop cluster, hdfs just doesnt work this way.

also in terms of networking, u do not reboot ur servers if you have a network failure.

but ya i really got no experience.

uhh....datacenters and backups are down? We're talking global EMP attacks and universal communications blackouts apocalypse scenario. (I mean if you want to go through the whole emergency plan we could....) Google does have backups and redundancy out the wazoo to prevent that from happening in the fist place, but IF it did happen it's still gonna take a few days to test all that. But still, the point being ZOS probably doesn't have anywhere near the amount of backup systems that any data company has.

Depends on the source of the failure, size of the server, and amount of data loss on whether or not you'd reboot the server in the event of network failure. Also depends on whether or not you're loading a full backup, the type of server, etc. etc. Generally speaking, though, you are correct.
You also might be forgetting what time it is over here in the US.
Still a silly statement.

I'll still apologize though. I'm sorry.

Belidos · August 2015

Xyrqes wrote: »

some have said you can't blame zenimax for the problem
but of course you can
any system should be prepared, and have contingency options in place
failure to do so is unprofessional
Z should have procured an alternative backup feed, in the case of this eventuallity

Except in this case you are entirely missing the point.

It is impossible for ZOS to have had a backup contingency for this because it wasn't just their ISP that went down, it was the network backbone that went down, you know the whole network for the state that every ISP goes through, even if they had another ISP to fall back on that one would have been down too.

bareheiny · August 2015

If it makes anybody feel better, it's almost 9pm Friday night here...prime gaming time for me.

Now I have to slum it with an inferior game

jonjees · August 2015

players just dont understand that this is a business as well.
eso being an online game needs to stay ONLINE.
15$ a pop for sub and at 10k subs, thats easily 150k a month and im sure theres more than 10k subs.

if this is a free mmo, where i just download and play free with cash shops, i wouldnt give a hoot if its down this much.
also for the console players, we just want to turn it on and play hassle free, thats the entire concept of going console.

6hrs of downtime and a few more hrs for maintenance ..... if i bought eso today as a console playerand tried playing during this time, i am so returning it.

ZOS_MichaelServotte · August 2015

kaithuzar wrote: »

Is there going to be an announcement made when we can log back in?

We'll let you know as soon as possible, in the meantime we ask you not to try log into the game. Our teams are working on the servers and it may be a couple hours to complete everything. Thanks again for your patience.

Eshelmen · August 2015

Well, it's a good thing you're not new eh?

Prof_Bawbag · August 2015

I'm just waiting for ZoS to announce all our data has been compromised by the Chinese mafia.

k2blader · August 2015

I wouldn't have been so irked if one of the ZOS people hadn't posted the problem was fixed and for anyone still experiencing issues to check with his/her ISP. The problem wasn't/isn't my ISP, ffs. Just totally irresponsible to post something like that.

Spottswoode · August 2015

Prof_Bawbag wrote: »

I'm just waiting for ZoS to announce all our data has been compromised by the Chinese mafia.

*cough* government *cough*

Techlisp · August 2015

A naughty Khajiit took a leak in the server room I hear.

Soulsix · August 2015

I guess i got the other game....

jonjees · August 2015

Spottswoode wrote: »

jonjees wrote: »

Spottswoode wrote: »

jonjees wrote: »

couple of hours to reboot servers .....

A statement from someone who has no experience with business networking. If Google had to reboot all of their servers, it could take a few days. (That is to say, ALL of them including the datacenters and backup datacenters.) As ZOS probably doesn't have a backup center to operate from (or backup network) this kind of reset is a lot of work.

wow really a few days?
thats when you just showed how much u know about redundancy and high availability.
maybe you need to read a paper on google hadoop cluster, hdfs just doesnt work this way.

also in terms of networking, u do not reboot ur servers if you have a network failure.

but ya i really got no experience.

uhh....datacenters and backups are down? We're talking global EMP attacks and universal communications blackouts apocalypse scenario. (I mean if you want to go through the whole emergency plan we could....) Google does have backups and redundancy out the wazoo to prevent that from happening in the fist place, but IF it did happen it's still gonna take a few days to test all that. But still, the point being ZOS probably doesn't have anywhere near the amount of backup systems that any data company has.

Depends on the source of the failure, size of the server, and amount of data loss on whether or not you'd reboot the server in the event of network failure. Also depends on whether or not you're loading a full backup, the type of server, etc. etc. Generally speaking, though, you are correct.
You also might be forgetting what time it is over here in the US.
Still a silly statement.

I'll still apologize though. I'm sorry.

no offence taken, so no apology needed, i know how the internet works.

but during this network outage players from kuwait were still able to be logged on, so its not a server issue.
for ps4 eu az campaign they had the biggest duel, 15 from Ep, 7 from Dc and 9 from AD.

either way they need to have another link for such outages, most business that depends on being online would have a couple of links to switch to but its been "6hrs since i cant play when i want to" and eso as a business has failed for me.

SeptimusDova · August 2015

Gina thank you so much for your dedication to the playerbase.

Solar storms at levels g4 and g5 are disruptive to backhaul circuits. Any DS3 or higher Microwave path is a feedpoint into the system. I am very tired my friends so I will post the graphs. I will answer any questions pertaining to solar weather after I rest. On the graphs you will see K4 which coincides with NOAA and NASA and GOES.

UrQuan do you work backhaul?

K index

7 day indice below

When High frequency radio is mentioned that is referring to 1.8-30Mhz radio traffic.

Backhaul and space diversity systems and wave-guide systems with degraded magnetic isolators can act as transmission paths for voltage induced by Solar variances. waveguide to radio, radio to mux, mux to bes56, bes 56 to CLN. Its been awhile since I was in a NOC or a backhaul site. Some .gov sites we hardened to help prevent damage to equipment. Running spares often became an issue due to tired techs and contractors trying to adhere to the five 9's. Parts would be used and not annotated as such. the used were put back in a box and would not be turned in on time. Which created problems. The wiltel FAA site in AZ had this issue on 2004 and so did the riverside site. Level3 com. I am not sure how they handle their operations.

Supply voltage side on some sites were filtered and regulated via DC. But that is not the topic of this thread. I feel sorry for the affected companies and for the providers. If indeed their systems were damaged it can get expensive. Even more so if they were running any old Nortel systems.

jonjees · August 2015

Prof_Bawbag wrote: »

I'm just waiting for ZoS to announce all our data has been compromised by the Chinese mafia.

time to buy cheap gold from the chinese farmers then.

Eshelmen · August 2015

jonjees wrote: »

Spottswoode wrote: »

jonjees wrote: »

Spottswoode wrote: »

jonjees wrote: »

couple of hours to reboot servers .....

A statement from someone who has no experience with business networking. If Google had to reboot all of their servers, it could take a few days. (That is to say, ALL of them including the datacenters and backup datacenters.) As ZOS probably doesn't have a backup center to operate from (or backup network) this kind of reset is a lot of work.

wow really a few days?
thats when you just showed how much u know about redundancy and high availability.
maybe you need to read a paper on google hadoop cluster, hdfs just doesnt work this way.

also in terms of networking, u do not reboot ur servers if you have a network failure.

but ya i really got no experience.

uhh....datacenters and backups are down? We're talking global EMP attacks and universal communications blackouts apocalypse scenario. (I mean if you want to go through the whole emergency plan we could....) Google does have backups and redundancy out the wazoo to prevent that from happening in the fist place, but IF it did happen it's still gonna take a few days to test all that. But still, the point being ZOS probably doesn't have anywhere near the amount of backup systems that any data company has.

Depends on the source of the failure, size of the server, and amount of data loss on whether or not you'd reboot the server in the event of network failure. Also depends on whether or not you're loading a full backup, the type of server, etc. etc. Generally speaking, though, you are correct.
You also might be forgetting what time it is over here in the US.
Still a silly statement.

I'll still apologize though. I'm sorry.

no offence taken, so no apology needed, i know how the internet works.

but during this network outage players from kuwait were still able to be logged on, so its not a server issue.
for ps4 eu az campaign they had the biggest duel, 15 from Ep, 7 from Dc and 9 from AD.

either way they need to have another link for such outages, most business that depends on being online would have a couple of links to switch to but its been "6hrs since i cant play when i want to" and eso as a business has failed for me.

And yet you'll come back and play another time.....
Why?

Greeniewolfub17_ESO · August 2015

jonjees wrote: »

players just dont understand that this is a business as well.
eso being an online game needs to stay ONLINE.
15$ a pop for sub and at 10k subs, thats easily 150k a month and im sure theres more than 10k subs.

if this is a free mmo, where i just download and play free with cash shops, i wouldnt give a hoot if its down this much.
also for the console players, we just want to turn it on and play hassle free, thats the entire concept of going console.

6hrs of downtime and a few more hrs for maintenance ..... if i bought eso today as a console playerand tried playing during this time, i am so returning it.

Obviously you don't understand it. That's clear with your statements that totally ignore the fact that as others pointed out Data centers across US and Europe went down today. Some as a result of Level 3 going down, others likely because of the solar storm. Entire cities were without net altogether. No net. At all.
Zos is not as big a company as say Linden Lab, Aeira, Gaia, or EA. At least one of the datacenters LL uses was down today as well. I don't play any of LL's games anymore so I don't know how that outage affected them, but still. I play Starcraft II and Heroes of the Storm. Occasionally battle.net goes down. While battle.net seemed to be largely intact today, many people who had net service were still unable to connect. Outages are a reality in any online game. Unless they stretch on to days, to whine and demand compensation for a downtime of 6 hours that was unexpected and largely unpredictable and tied to outages across not only the US but the entire globe is quite frankly silly.
And furthermore your argument that they need more datacenters in more locations falls flat. One game I play has three datacenters spread across three countries. All three data centers were hit by the Level 3 outage. They all were down longer than ESO has been. IF you want to return ESO do so. But please stop posting the same rant over and over.

Listatta · August 2015

I can see Google or Yahoo or my local bank shelling out the hundreds of thousands and thousands of dollars for fast and secure data back up and network redundancy protocols.... but shall I remind you all this is a video game.

While I completely get the reaction, we all desperately need our escape from reality, less we crack.. it's still just a video game.

Soulsix · August 2015

SeptimusDova wrote: »

Gina thank you so much for your dedication to the playerbase.

Solar storms at levels g4 and g5 are disruptive to backhaul circuits. Any DS3 or higher Microwave path is a feedpoint into the system. I am very tired my friends so I will post the graphs. I will answer any questions pertaining to solar weather after I rest. On the graphs you will see K4 which coincides with NOAA and NASA and GOES.

UrQuan do you work backhaul?

K index

7 day indice below

When High frequency radio is mentioned that is referring to 1.8-30Mhz radio traffic.

Backhaul and space diversity systems and wave-guide systems with degraded magnetic isolators can act as transmission paths for voltage induced by Solar variances. waveguide to radio, radio to mux, mux to bes56, bes 56 to CLN. Its been awhile since I was in a NOC or a backhaul site. Some .gov sites we hardened to help prevent damage to equipment. Running spares often became an issue due to tired techs and contractors trying to adhere to the five 9's. Parts would be used and not annotated as such. the used were put back in a box and would not be turned in on time. Which created problems. The wiltel FAA site in AZ had this issue on 2004 and so did the riverside site. Level3 com. I am not sure how they handle their operations.

Supply voltage side on some sites were filtered and regulated via DC. But that is not the topic of this thread. I feel sorry for the affected companies and for the providers. If indeed their systems were damaged it can get expensive. Even more so if they were running any old Nortel systems.

TY sir, informative and passed some time

Techlisp · August 2015

I feel sorry for the people who reply on Twitter to ESO's server is down tweet. It is almost like their lives depend on ESO being there for them. #sad

kevlarto_ESO · August 2015

I once encountered an internet backbone issue, I remember when I played LOTRO and all the sudden had a hard time logging in I ran a trace route to find I was going to France and then back to the Boston area, I was able to log in but slowly and play but going through the web from Las Vegas across the Atlantic and back to the US made for one heck of a ping rate..:) The internet works in mysterious ways..:)

I know this is different, I have worked as an Enterprise Administrator for a state agency and keeping 67 counties communicating can be quite a task.

k2blader · August 2015

Has anyone found news articles about the internet outage?

[edit]
Here is the one I found:
neowin.net/news/level3-suffering-major-outages-effects-are-felt-worldwide

jonjees · August 2015

Greeniewolfub17_ESO wrote: »

jonjees wrote: »

players just dont understand that this is a business as well.
eso being an online game needs to stay ONLINE.
15$ a pop for sub and at 10k subs, thats easily 150k a month and im sure theres more than 10k subs.

if this is a free mmo, where i just download and play free with cash shops, i wouldnt give a hoot if its down this much.
also for the console players, we just want to turn it on and play hassle free, thats the entire concept of going console.

6hrs of downtime and a few more hrs for maintenance ..... if i bought eso today as a console playerand tried playing during this time, i am so returning it.

Obviously you don't understand it. That's clear with your statements that totally ignore the fact that as others pointed out Data centers across US and Europe went down today. Some as a result of Level 3 going down, others likely because of the solar storm. Entire cities were without net altogether. No net. At all.
Zos is not as big a company as say Linden Lab, Aeira, Gaia, or EA. At least one of the datacenters LL uses was down today as well. I don't play any of LL's games anymore so I don't know how that outage affected them, but still. I play Starcraft II and Heroes of the Storm. Occasionally battle.net goes down. While battle.net seemed to be largely intact today, many people who had net service were still unable to connect. Outages are a reality in any online game. Unless they stretch on to days, to whine and demand compensation for a downtime of 6 hours that was unexpected and largely unpredictable and tied to outages across not only the US but the entire globe is quite frankly silly.
And furthermore your argument that they need more datacenters in more locations falls flat. One game I play has three datacenters spread across three countries. All three data centers were hit by the Level 3 outage. They all were down longer than ESO has been. IF you want to return ESO do so. But please stop posting the same rant over and over.

wow wait, no net at all?
so tell me, how do u post in this forum?

wait .... forums dont run on datacentres right?

battle net as an example is bad.
because starcraft u can play offline, likewise with diablo and also u dont pay a sub for battle net.

DaffyDilly · August 2015

k2blader wrote: »

I wouldn't have been so irked if one of the ZOS people hadn't posted the problem was fixed and for anyone still experiencing issues to check with his/her ISP. The problem wasn't/isn't my ISP, ffs. Just totally irresponsible to post something like that.

Unfortunately for most if not all MMO'S that is a standard issue response. Expecially early in a server crash.

ViscousSummer88 · August 2015

Eshelmen wrote: »

jonjees wrote: »

Spottswoode wrote: »

jonjees wrote: »

Spottswoode wrote: »

jonjees wrote: »

couple of hours to reboot servers .....

A statement from someone who has no experience with business networking. If Google had to reboot all of their servers, it could take a few days. (That is to say, ALL of them including the datacenters and backup datacenters.) As ZOS probably doesn't have a backup center to operate from (or backup network) this kind of reset is a lot of work.

wow really a few days?
thats when you just showed how much u know about redundancy and high availability.
maybe you need to read a paper on google hadoop cluster, hdfs just doesnt work this way.

also in terms of networking, u do not reboot ur servers if you have a network failure.

but ya i really got no experience.

uhh....datacenters and backups are down? We're talking global EMP attacks and universal communications blackouts apocalypse scenario. (I mean if you want to go through the whole emergency plan we could....) Google does have backups and redundancy out the wazoo to prevent that from happening in the fist place, but IF it did happen it's still gonna take a few days to test all that. But still, the point being ZOS probably doesn't have anywhere near the amount of backup systems that any data company has.

Depends on the source of the failure, size of the server, and amount of data loss on whether or not you'd reboot the server in the event of network failure. Also depends on whether or not you're loading a full backup, the type of server, etc. etc. Generally speaking, though, you are correct.
You also might be forgetting what time it is over here in the US.
Still a silly statement.

I'll still apologize though. I'm sorry.

no offence taken, so no apology needed, i know how the internet works.

but during this network outage players from kuwait were still able to be logged on, so its not a server issue.
for ps4 eu az campaign they had the biggest duel, 15 from Ep, 7 from Dc and 9 from AD.

either way they need to have another link for such outages, most business that depends on being online would have a couple of links to switch to but its been "6hrs since i cant play when i want to" and eso as a business has failed for me.

And yet you'll come back and play another time.....
Why?

That point is completely invalid, of course people will come back to play another time!

Yes, they have failed in foreseeing such issues on their systems, which to be honest they should have some redundancy for these things, which clearly they do not. It's perfectly acceptable for us as CUSTOMERS to be upset and disappointed and feel that we have been let down, don't let companies try and fool you, you paid for their services after all and maybe you still do pay for their services in the form of a subscription etc.

Complaining as a customer is nothing to be ashamed or looked down on for, actually you all as customers should be complaining regardless if they directly have control over it or not! Sure their ToS has legal rubbish to state no responsibility for xyz and but peoples voices are loud and they will hear.