The thing with server downtime (and why it is NOT a necessary evil)

HatchetHaro

You've all experienced it: patch day, the day where you have to actually refrain from playing ESO and do something productive for once, Akatosh forbid! It's a "necessary evil" in order to keep the game in tip-top shape and add new content.

Here's the thing about server downtime: it's not a "necessary evil".
It's just a stupid and outdated system that belongs anywhere but a well-respected MMORPG of a famous franchise.

Take Warframe, for example. They do regular updates, patches, and hotfixes. You know what they do when they have to push a patch out? They send Red Text into the chat.

Then everyone quits their game and reopens the launcher to download the new update/patch/hotfix. People don't automatically get kicked off the server until a while later so they can finish what they were doing, grind for that last prime part, yadi yada. It's a good system. The only gameplay downtime is downloading the new content client-side. There is absolutely no downtime server-side. All is good, there are no complaints with the system Digital Extremes is using.

Here? No. We get unnecessary server downtime that cuts into a lot of our actual playtime and takes away from the value of our ESO+ subscriptions. It's an archaic and annoying system that just makes the game experience so much worse.

Just get a second bloody server and push the update onto that, then push relogging players onto the updated server. Servers aren't that expensive, and for a large company such as Zenimax, a new server machine is a piece of cake.

We paid and are still paying for the game. Give us the game.

EDIT: a post about how archaic this maintenance system is, and already everyone's on my arse instead of actually contributing to the discussion. Great job, ESO forum community, great job.

UPDATE: I have created two images that detail the scenarios in a simple analogy. Hopefully it'll make my message more clear.
Current system:

My proposed system:

quadraxis666

Bakkagami

I'm just guessing here but I think far too much of the processing goes on server-side for them to make changes while people are on. If they tried we'd all be hoping for just the bugs we have now.

HatchetHaro

quadraxis666 wrote: »

*hurr durr stop whining*

I paid for the bloody game and I have the right to whine all the hell I want about the bloody game. It's a subpar gaming experience, and while I do still enjoy this game, I definitely do not enjoy not being able to play this game.

It's called "criticism", sweetie. It's actually a thing.

Bakkagami wrote: »

I'm just guessing here but I think far too much of the processing goes on server-side for them to make changes while people are on. If they tried we'd all be hoping for just the bugs we have now.

That can easily be fixed by pushing the update onto a second server where there are no players online, which is exactly what I'm getting at in my post.

karldavy149b16_ESO

HatchetHaro wrote: »

You've all experienced it: patch day, the day where you have to actually refrain from playing ESO and do something productive for once, Akatosh forbid! It's a "necessary evil" in order to keep the game in tip-top shape and add new content.

Here's the thing about server downtime: it's not a "necessary evil".
It's just a stupid and outdated system that belongs anywhere but a well-respected MMORPG of a famous franchise.

Take Warframe, for example. They do regular updates, patches, and hotfixes. You know what they do when they have to push a patch out? They send Red Text into the chat.

Then everyone quits their game and reopens the launcher to download the new update/patch/hotfix. People don't automatically get kicked off the server until a while later so they can finish what they were doing, grind for that last prime part, yadi yada. It's a good system. The only gameplay downtime is downloading the new content client-side. There is absolutely no downtime server-side. All is good, there are no complaints with the system Digital Extremes is using.

Here? No. We get unnecessary server downtime that cuts into a lot of our actual playtime and takes away from the value of our ESO+ subscriptions. It's an archaic and annoying system that just makes the game experience so much worse.

Just get a second bloody server and push the update onto that, then push relogging players onto the updated server. Servers aren't that expensive, and for a large company such as Zenimax, a new server machine is a piece of cake.

We paid and are still paying for the game. Give us the game.

how much of that game is done client side jesus christ i bet that game been exploited more than a hooker on $1 night

notimetocare

You realize Warframe is about... a fifth the size of ESO just for client side data?
Probably also has very little experience with computers, computing, and programming to boot...

Bakkagami

HatchetHaro wrote: »

quadraxis666 wrote: »

*hurr durr stop whining*

I paid for the bloody game and I have the right to whine all the hell I want about the bloody game. It's a subpar gaming experience, and while I do still enjoy this game, I definitely do not enjoy not being able to play this game.

It's called "criticism", sweetie. It's actually a thing.

I always lol when people use the "I payed for this game" arguement. Read the terms & conditions.

Grabmoore

You don't have to be here, you know.;)

HatchetHaro

Bakkagami wrote: »

I always lol when people use the "I payed for this game" arguement. Read the terms & conditions.

So you're telling me I can't throw suggestions for the game around?
Cute.

Lokryn

I don't think you realize how complicated and technical an update like this really is. As an IT Analyst, there are so many pitfalls with a live update that it's always better to go offline in the long run. Some games may be able to do this but you have to consider the platform and infrastructure they are built on. You can't compare games from two different genres that have different IT requirements.

runagate

I suspect the real issue is that if you don't plan to have live updates from the very start that it'd be a harrowing nightmare to retrofit it into one's system.

Wolfhammer

Cheese OP?

ElBiggus

Imagine if, every time Google or Facebook made some changes, we all had to stop using them for 8 hours? Now tell me that ESO's server-side computing requirements, data centre complexities, or other technical aspects are more complex than Facebook.

Anyone who runs server-side services will have a production, test, and deployment server. You muck about on the test one, when it's working you push it to the deployment one, and when that's done you toggle a value somewhere and swap deployment and production over -- people using the original production server can carry on uninterrupted while the task is carried out, and when they log off you grab their data, transfer it to the new production server, and everyone's happy...

The regular, frequent, and lengthy downtimes for "maintenance" are not normal and people need to stop making excuses for them.

Sounomi

The thing with Warframe is that its heavily instanced game with most (if not all) of the instances being handled entirely client side and the server merely acts as a middle man between the players. You easily see this when the party leader lags out as suddenly you can't open up any doors anymore because that was being handled by their computer and not the server.

HatchetHaro

ElBiggus wrote: »

Imagine if, every time Google or Facebook made some changes, we all had to stop using them for 8 hours? Now tell me that ESO's server-side computing requirements, data centre complexities, or other technical aspects are more complex than Facebook.

Anyone who runs server-side services will have a production, test, and deployment server. You muck about on the test one, when it's working you push it to the deployment one, and when that's done you toggle a value somewhere and swap deployment and production over -- people using the original production server can carry on uninterrupted while the task is carried out, and when they log off you grab their data, transfer it to the new production server, and everyone's happy...

The regular, frequent, and lengthy downtimes for "maintenance" are not normal and people need to stop making excuses for them.

Thank you! This is exactly what I'm getting at. Kudos for the knowledge by the way.

DM_ESO

Guild Wars 2 does the same. The old version stays live for an extra hour, everyone just logs out, updates for a few minutes and logs back in. It never goes offline save for 1 time for an hour during it's 4 year lifetime. Meanwhile, this is the third (and counting?) 8 hour downtime in the last week.

Solariken

I've thought about that a lot too, OP. It can't be that hard to have two separate servers that alternate as the live server from patch to patch. So for patch X.1 server A is live, and when it comes time for patch X.2 server B is loaded and set to live and all new logins are directed to that, until the next patch where server A becomes live.

It boggles my mind that even ZOS can't figure this out.

Luigi_Vampa

ElBiggus wrote: »

Imagine if, every time Google or Facebook made some changes, we all had to stop using them for 8 hours? Now tell me that ESO's server-side computing requirements, data centre complexities, or other technical aspects are more complex than Facebook.

Anyone who runs server-side services will have a production, test, and deployment server. You muck about on the test one, when it's working you push it to the deployment one, and when that's done you toggle a value somewhere and swap deployment and production over -- people using the original production server can carry on uninterrupted while the task is carried out, and when they log off you grab their data, transfer it to the new production server, and everyone's happy...

The regular, frequent, and lengthy downtimes for "maintenance" are not normal and people need to stop making excuses for them.

For some reason MMO players have come to accept server downtime and random outages. People would be pissed if Netflix, Facebook, Google etc... went down for hours at a time, but an MMO? Apparently it is just accepted. If you mention it, you get yelled at for whining. For some reason people are against promoting and trying to implement better systems. There is nothing wrong with trying to do things better and keep the servers up as much as possible.

Rasimir

Well, Warframe isn't the only game with much better patch management. Look at guildwars 2, they have downtimes too, but much shorter ones. Look at neverwinter, they download/patch only what is immediately needed, and the other things get downloaded when you enter the areas.

Here we have gigantic downloads, we even get japanese voices even without ever setting the client to japanese. And we have gigantic downtimes as well.

But: this is very basic engine design stuff, and I very much doubt it can be changed without rewriting almost the entire game. perhaps Zenimx learns and does better with their next game if they ever do another MMO. We will have to live with it for ESO.

By the way: Quality of Life is very important for gamers. There are so much games around that even small annoyances drive people away. Downtimes, lags, visual bugs, clumsy UI, broken mechanics not getting fixed for months are showstoppers for many people, much more important than content. A smooth, comfortable user experience keeps players interested, and after that new content every now and then ...

mobicera

I can't tell if this is a serious post or a troll post.
Honestly...
If serious, it's a video game... Really a video game it isn't that important.
If troll sorry for responding.

ElBiggus

mobicera wrote: »

I can't tell if this is a serious post or a troll post.
Honestly...
If serious, it's a video game... Really a video game it isn't that important.
If troll sorry for responding.

Doesn't matter if it's a video game or something else, it's a service we've paid for. I find this attitude baffling; for some reason gamers have come to accept poor service, shoddy products, and terrible customer support, and any of us who dare to say "look, this sucks, fix it" are told we're either whining, stupid, or trolls. If your phone stops working do you just go "meh, it's fine", or do you complain to the phone company? If you paid $60 for a coffee machine and once a week it just refused to work for 8 hours, and every time you used it there was a 1 in 10 chance of it serving you something undrinkable would you shrug and say "it's just coffee" or would you take it back?

Seriously, if you'[re happy with it, then good for you. I think you'd probably be happier if there wasn't so much downtime, but until enough people say it needs to change it's just going to keep on happening.

anitajoneb17_ESO

"ElBiggus wrote: »

Now tell me that ESO's server-side computing requirements, data centre complexities, or other technical aspects are more complex than Facebook..

Of course ESO is FAR MORE complicated than Facebook ! There's no doubt about that. As far as I know, Facebook doesn't rely on milliseconds precise data exchange for simultaneous display in 3D on 100+ screens.

Besides, ESO is on a MEGASERVER, we're all sharing the same space, which is instanced on the fly depending on population. It's not like they can switch off one part of it and move us somewhere else.

Also, don't believe the megaserver infrastructure is cheap. This is most likely the most expensive part or running this game. It's a massive cost, hence why it's being rented and not owned. Duplicating that infrastructure just to avoid a few hours maintenance every week would be totally out of proportion.

mobicera

ElBiggus wrote: »

mobicera wrote: »

I can't tell if this is a serious post or a troll post.
Honestly...
If serious, it's a video game... Really a video game it isn't that important.
If troll sorry for responding.

Doesn't matter if it's a video game or something else, it's a service we've paid for. I find this attitude baffling; for some reason gamers have come to accept poor service, shoddy products, and terrible customer support, and any of us who dare to say "look, this sucks, fix it" are told we're either whining, stupid, or trolls. If your phone stops working do you just go "meh, it's fine", or do you complain to the phone company? If you paid $60 for a coffee machine and once a week it just refused to work for 8 hours, and every time you used it there was a 1 in 10 chance of it serving you something undrinkable would you shrug and say "it's just coffee" or would you take it back?

Seriously, if you'[re happy with it, then good for you. I think you'd probably be happier if there wasn't so much downtime, but until enough people say it needs to change it's just going to keep on happening.

I just don't care enough about a video game.
If it's offline ce la vie, I will go play some music, read a book, write something, listen to some music, watch a movie, go outside(I think you get the point).
When it's all said and done a video game is just entertainment, it's not important enough to get worked up about.
If bugs, etc. cause me to not receive entertainment value I will simply seek it elsewhere.
As for the sense of entitlement I find that just as baffling as you do my lack of concern.

There are actual problems in this world, a video game(or a coffee machine) don't rank among them.

I am sorry that you can not see that and for any distress my views may cause you, just relax, live life.
Take care

anitajoneb17_ESO

ElBiggus wrote: »

If you paid $60 for a coffee machine and once a week it just refused to work for 8 hours, and every time you used it there was a 1 in 10 chance of it serving you something undrinkable would you shrug and say "it's just coffee" or would you take it back?

I would call it a bad purchase, get another coffee machine and move on.
There's no better coffee machine but I still don't like that coffee ? I'd switch to tea. Problem solved.

To make my point clear : I hope you "complainers" realize that the way and the intensity with which you express your "frustration" with downtimes probably has the reverse effect of what you expect. Instead of telling ZOS "we don't care, if it sucks too much, we'll go play something else", the message you pass on is "we're so addicted to this game that no matter how much you suck, we'll stick around". Actually, ZOS is far more likely to loose a customer such as @mobicera over bad service than to loose you.

CapnPhoton

In regard to maintenance, it seems that you are comparing apples to oranges with the two games. However, hopefully we all have something else to do when there is a down time. Otherwise, this might be useful:

http://internetaddictiondisorder.org/addiction-to-mmorpgs/

vyndral13preub18_ESO

anitajoneb17_ESO wrote: »

"ElBiggus wrote: »

Now tell me that ESO's server-side computing requirements, data centre complexities, or other technical aspects are more complex than Facebook..

Of course ESO is FAR MORE complicated than Facebook ! There's no doubt about that. As far as I know, Facebook doesn't rely on milliseconds precise data exchange for simultaneous display in 3D on 100+ screens.

Besides, ESO is on a MEGASERVER, we're all sharing the same space, which is instanced on the fly depending on population. It's not like they can switch off one part of it and move us somewhere else.

Also, don't believe the megaserver infrastructure is cheap. This is most likely the most expensive part or running this game. It's a massive cost, hence why it's being rented and not owned. Duplicating that infrastructure just to avoid a few hours maintenance every week would be totally out of proportion.

Just like GW2. Last i played their servers acted pretty much just like esos mega server. So it really isnt an excuse why this couldnt be done.

It doesnt really matter to me one way or the other, but it isnt the mega server stopping them from doing it.

ElBiggus

anitajoneb17_ESO wrote: »

Of course ESO is FAR MORE complicated than Facebook

Citation needed. In 2011 Facebook (just the core services, and excluding support systems like the replication backend) consisted of around 10 million lines of code, and have you seen their datacentres?

It's not like they can switch off one part of it and move us somewhere else.

They don't need to, and that's not what I'm suggesting. Once the switch has been made, simply pop up a message in the client saying "new server has gone live; please log out when you've finished what you're doing and log in again to switch to the updated server." Maybe force people to log out after a couple of hours or something -- obviously they can't keep two running indefinitely -- but there is no need to shut the whole thing down for hours at a time every week, often more than once (like this week, for example).

Also, don't believe the megaserver infrastructure is cheap. This is most likely the most expensive part or running this game. It's a massive cost, hence why it's being rented and not owned. Duplicating that infrastructure just to avoid a few hours maitenance every week would be totally out of proportion.

Let's say there's a day grace period where they run old and new servers concurrently; they already have the PTS, and as that's kind of a "bonus if you want to get a sneak peek" and not the core product they could just leverage that -- it's much more customer friendly to shut that down when they're doing maintenance than it is to keep that running and shut down the main servers.

At the end of the day there are no technical reasons why things need to be done this way. It may be a little more involved to do it "right", but I see no reason to accept (or indeed actively defend) them just because they've come up with an architecture and service pattern that saves them some work at the expense of customer satisfaction.

mobicera wrote: »

I just don't care enough about a video game.
If it's offline ce la vie, I will go play some music, read a book, write something, listen to some music, watch a movie, go outside(I think you get the point).
When it's all said and done a video game is just entertainment, it's not important enough to get worked up about.
If bugs, etc. cause me to not receive entertainment value I will simply seek it elsewhere.
As for the sense of entitlement I find that just as baffling as you do my lack of concern.

There are actual problems in this world, a video game(or a coffee machine) don't rank among them.

I am sorry that you can not see that and for any distress my views may cause you, just relax, live life.
Take care

Just because it's entertainment doesn't mean you're not entitled to expect a certain level of service. If you went to the movies and the picture was upside down or out of focus or something, would you just walk out and say "ah well, never mind, it was only entertainment" or would you expect them to either fix it or give you your money back?

Lunarhound

vyndral13preub18_ESO wrote: »

Just like GW2. Last i played their servers acted pretty much just like esos mega server. So it really isnt an excuse why this couldnt be done.

It doesnt really matter to me one way or the other, but it isnt the mega server stopping them from doing it.

Pretty much this. While the OP's attitude may come across as "whiny", they aren't wrong about downtime on patch day beginning to feel outdated. As @vyndral13preub18_ESO pointed out, Guild Wars 2 works this way, and they utilize a megaserver system that, at least from a user standpoint, is very similar to ESO's. Granted, I have no idea how things are set up on the back end. I'm sure there's a good chance it's very different there.

Maintenance downtime isn't the end of the world, but I absolutely do think it's worth asking why, when other, similar games have been using better systems for years now, we shouldn't expect better of ours.

HatchetHaro

I have updated my post with images that should make the scenarios much easier to process. Hopefully it'll be clearer to you guys.

Current system:

My proposed system:

FLuFFyxMuFFiN

Geez with this amount of whining over a few hours of downtime you would think people have nothing else to do with their lives. Yes you paid for the game but you also agreed to the ToS which clearly states that ZOS has the right to take the servers offline whenever they want with or without notifying you. Also I might add that even though you paid for the game you still don't own the account or anything attached to the account. So how about instead of whining and acting all entitled why don't some of you people go experience the outside world for once or maybe go play another game for a couple hours.

Jade1986

Lokryn wrote: »

I don't think you realize how complicated and technical an update like this really is. As an IT Analyst, there are so many pitfalls with a live update that it's always better to go offline in the long run. Some games may be able to do this but you have to consider the platform and infrastructure they are built on. You can't compare games from two different genres that have different IT requirements.

Oh god. The IT analyst argument. Suddenly we are all professionals. Not saying you aren't, but people say that JUST so they look smart.