The thing with server downtime (and why it is NOT a necessary evil)

PeggymoeXD

FLuFFyxMuFFiN wrote: »

Geez with this amount of whining over a few hours of downtime you would think people have nothing else to do with their lives. Yes you paid for the game but you also agreed to the ToS which clearly states that ZOS has the right to take the servers offline whenever they want with or without notifying you. Also I might add that even though you paid for the game you still don't own the account or anything attached to the account. So how about instead of whining and acting all entitled why don't some of you people go experience the outside world for once or maybe go play another game for a couple hours.

Look how cool this guy is. Dude seriously because of this statement you are better than all of us. I can only hope to achieve your edge and wit someday. But until that day I'm going to sit here and be a regular person who is irritated with server downtime.

idk

quadraxis666 wrote: »

Lol.

ElBiggus

HatchetHaro wrote: »

I have updated my post with images that should make the scenarios much easier to process. Hopefully it'll be clearer to you guys.

Current system:

My proposed system:

Perfect!

This is exactly how every company I've worked for in the past 25 years that has a large server-side infrastructure has worked, because it's the sensible way to do it. One, customers can still keep doing what they're doing, and two, if you accidentally pour bleach into the second tank instead of water you can scrub it out and start again and nobody will ever know you screwed up.

anitajoneb17_ESO

ElBiggus wrote: »

Just because it's entertainment doesn't mean you're not entitled to expect a certain level of service. If you went to the movies and the picture was upside down or out of focus or something, would you just walk out and say "ah well, never mind, it was only entertainment" or would you expect them to either fix it or give you your money back?

The movie isn't shown upside down. The movie theater is closed for a day. That's entirely different. So yeah you have an abo that lets you see the movies you want to see, on any day 365 days a year, BUT the terms of the abo specifically mentioned that sometimes, the movie theater will be closed for cleaning or maintenance or whatever.
Some theaters will prefer to clean/maintain one room at a time and never close, other theaters will prefer to close it entirely for a short while and reopen all rooms after the break.

If you think the former option is a better service, go to those theaters...

SnubbS

The comments defending ZoS are so cringey. I'm not even bothered by the downtime, my expectations for ESO are rock bottom -- I'm excited every time my character isn't stuck in Grahtwood. But to blindly defend them on the platform of "QuITuTUT UrRRrRRR BITtCHCHnNInNNN" is pathetic.

There's probably a logical reason as to why the servers are down for such substantial amounts of time, and why it happens so frequently. It's likely to be an issue of money -- there are IT guys here who could probably point to more specific reasons, but money has to be at the very top.

Bakkagami

HatchetHaro wrote: »

Bakkagami wrote: »

I always lol when people use the "I payed for this game" arguement. Read the terms & conditions.

So you're telling me I can't throw suggestions for the game around?
Cute.

Nope. All I was saying was that paying for the game isn't a good reasoning/excuse for what you used it for. Feel free to throw your suggestion around all you want.

Also, while your suggestion isn't particularly bad. You want ZOS to add a shadow server, actually multiple to accommodate each of the mega-servers, to the already struggling system they have. Camel meet straw...

Cously

I was going to call you a whining female dog but the fish illustration of the problem got me laughing so hard I just gave an awesome!

ElBiggus

anitajoneb17_ESO wrote: »

Some theaters will prefer to clean/maintain one room at a time and never close, other theaters will prefer to close it entirely for a short while and reopen all rooms after the break.

If you think the former option is a better service, go to those theaters...

The difference being that, in the case of running stuff on servers, you can make a whole new cinema next door that's nice and clean, send everyone to that one when they show up to watch a movie, and then knock the old one down when it's empty. It's how the rest of the world has been handling server updates since the 70s (and probably earlier), and the games industry are seemingly the only people who seem to think doing it the other way round -- knocking down the old cinema before they built the new one -- is a sensible approach.

I will say it again. There is no technical reason why server updates and maintenance cannot be done without 8 or more hours of downtime a week and in any other industry only managing 95% uptime would be cause for concern, and 99.999% is the target. (Just in case you're wondering, that means a few minutes downtime a year.)

To make it clear I am aware it's not the end of the world that ESO is offline, but that does not change the fact that there's room for improvement, and the means to improving it is well known and has been done by thousands upon thousands of businesses for decades -- it's not some clever secret that only super geniuses can grasp, it's a common industry practice that every entry-level server admin knows. Saying that it's too difficult or that they shouldn't have to bother is, frankly, daft....

Zamrod_beta

SnubbS wrote: »

The comments defending ZoS are so cringey. I'm not even bothered by the downtime, my expectations for ESO are rock bottom -- I'm excited every time my character isn't stuck in Grahtwood. But to blindly defend them on the platform of "QuITuTUT UrRRrRRR BITtCHCHnNInNNN" is pathetic.

There's probably a logical reason as to why the servers are down for such substantial amounts of time, and why it happens so frequently. It's likely to be an issue of money -- there are IT guys here who could probably point to more specific reasons, but money has to be at the very top.

I can only guess, but I have worked in Operations for a extremely large company doing updates on critical systems that NEED 24 hour a day up time. It wasn't an MMO but some of the concepts are the same.

I can tell you that even in this situation, there were times we needed to take the servers offline for updates. But we did it only twice a year, in the middle of the night, and we limited the downtime to no more than 60 minutes(but those were a tense 60 minutes for us). The vast majority of patches were done with the servers still up so there was no downtime.

Either way, my guess is that they know that people have grown to expect downtime and it's cheaper not to pay people overtime to stay up all night to do a patch. But it's likely that importing the database is the most important reason.

Most patches likely make changes to the database structure. So, all the information on all of the Guilds, PvP stats, and character records are all stored in a database. Any changes to the database structure tends to require an import process and with a database the size of the one required for a game like this probably takes hours to import. It also makes 2 versions of the game completely incompatible. So, if you try to run 2 copies of the game simultaneously (one with the old version and one with the new version) then the old version would stop working immediately as soon as you changed the database structure since it wouldn't be able to read or write to the database(and likely would cause a bunch of corruption and possible data loss). You could make a copy of the database, do the import and set up a second server while the first one was still running but then all of the changes made since you copied the database wouldn't be reflected on the new server. Which means that everything everyone did for the last 2 or 3 hours would be erased when they switched to the new server.

That's not a very good user experience. You might as well take down the servers rather than keep them up and then have everyone complain that everything they did for a couple of hours went away. It's technically possible to code something that could figure out all of the changes to the database in the last couple of hours and then apply the same changes to the new database. But you'd likely need to re-code the tool each time a patch happened. That adds extra work each time there's a patch. And there's a real risk that if the tool fails then you corrupt your database and you need to restore from backups, meaning you add another couple of hours to the time required to patch. Also, those changes take time to put into the new database as well, so anyone who has been playing while the patch process happened would likely need to wait at least some time for the last couple of hours to be imported into the new server. But they couldn't keep playing because it would mean more changes that need to be imported. So, there HAS to be some downtime for at least some people. If you have fast enough servers (which, or course, cost more money), you can make the import process quicker...but it really depends on the size of your database. It's likely that the Everquest database was a lot smaller and less complicated than the ESO one, for instance.

For smaller patches with no database changes, you probably could do this a lot easier. Make two copies of the servers and have them both access the same database. Then you can just shut one down and everyone can keep playing on the new one. But most changes of any significance likely require database changes.

anitajoneb17_ESO

ElBiggus wrote: »

anitajoneb17_ESO wrote: »

Some theaters will prefer to clean/maintain one room at a time and never close, other theaters will prefer to close it entirely for a short while and reopen all rooms after the break.

If you think the former option is a better service, go to those theaters...

The difference being that, in the case of running stuff on servers, you can make a whole new cinema next door that's nice and clean, send everyone to that one when they show up to watch a movie, and then knock the old one down when it's empty. It's how the rest of the world has been handling server updates since the 70s (and probably earlier), and the games industry are seemingly the only people who seem to think doing it the other way round -- knocking down the old cinema before they built the new one -- is a sensible approach.

I will say it again. There is no technical reason why server updates and maintenance cannot be done without 8 or more hours of downtime a week and in any other industry only managing 95% uptime would be cause for concern, and 99.999% is the target. (Just in case you're wondering, that means a few minutes downtime a year.)

To make it clear I am aware it's not the end of the world that ESO is offline, but that does not change the fact that there's room for improvement, and the means to improving it is well known and has been done by thousands upon thousands of businesses for decades -- it's not some clever secret that only super geniuses can grasp, it's a common industry practice that every entry-level server admin knows. Saying that it's too difficult of that they shouldn't have to bother is, frankly, daft....

I get your point. But just because there's technically actual room for improvement, doesn't mean a company has to take that step : it's a cost/benefit/risk issue.
What is the cost of shadowing all megaservers as you suggest ? I assume it's HUGE.
What's the benefit of doing it ? Well, keep players happy ? Players' "happiness" is only relevant when it converts into income. The question is "are they going to win players by having less downtime" ? Probably not. Are they going to lose players by having downtime ? Probably a few, but the loss is minimal as compared to the investment that would be required. Inversely, Google would have A LOT to loose with downtime, since it would take people about 30 seconds to find another search engine, and eventually like it and stick to it.
It just happens that in the MMO market, server uptime is not a key aspect of player retention. That's why they don't shadow servers. The cost/benefit ration would be negative.

That being said, I agree that ESO downtimes lately are a bit too much and they should be careful about it.

Chew_Magna

Typhoios wrote: »

For some reason MMO players have come to accept server downtime and random outages. People would be pissed if Netflix, Facebook, Google etc... went down for hours at a time, but an MMO? Apparently it is just accepted. If you mention it, you get yelled at for whining. For some reason people are against promoting and trying to implement better systems. There is nothing wrong with trying to do things better and keep the servers up as much as possible.

It's called cultural adaptation. The same thing happened with ads on websites and mobile apps, people used to hate it and complain constantly, now it's just accepted. Same thing with microtransactions in games and mobile apps, same thing with paid dlc for minimal content in games, same thing with data mining for personal information. It happens across just about everything, something absurd is introduced, people whine and complain about it, a couple years later it's the norm and is accepted by the vast majority.

As for the OP, I get what you're saying. I believe Guild Wars did the same thing, you could keep playing all day on an old version of the game once a new version was pushed out, all you had to do was relog because the patch updated in the background. But, VERY few games have done this. It's the norm to have downtime to apply patches and maintenance. Some game engines and server setups probably just can't support it, at least not without a total overhaul of core systems, which would lead to more downtime and cause people to complain more.