What does ZoS do on maintenance day?

AnyOldIron

I guess that they fix a few bugs and probably upload new new code/content. What else happens during maintenance though? Are the servers backed up or replaced. Can anybody shed any light on exactly what happens? Genuinely curious.

ehgz

Defragment their databases, I imagine.

As the game database is used saving all your inventory and CP choices, it grows more and more fragmented.

Defragmenting speeds things up.

zharkovian

They get the truffles and bonbons out and tickle each other and giggle.

Mythreindeer

Delete threads like this.

DaiKahn

Someone asked this within the last month and people speculated why

I'll see if I can find it with the awesome forum search we have here /s

Zulera301

With "work from home" they probably just smoke dope all afternoon and then buy a hamster or two to replace the ones that died during the week.

wenchmore420b14_ESO

I borowed this from another thread. Thank you @Goregrinder for this... Huzzah!

Goregrinder
✭✭✭✭✭
✭
Yes, they can take this long. So Single player games like Skyrim are ran 100% off your own computer...the maintenance you would perform there would simply to reboot your computer. But ESO is a Massively-Multiplayer Online Role-playing Game, or also known in short as an "MMORPG", or simply "MMO".

MMORPG's use some form of server, which every ESO player connects to from wherever they live. There are different server configurations MMOs utilize, but ESO uses "Megaservers" for each region. So EVERYONE that lives in NA and plays on PC connects to the exact same server, or "Megaserver". That means thousands upon thousands of connections and sessions are made to that server all day long...every day....all hours of the day and night.

That will eventually cause performance problems, much like never rebooting your personal computer for 3 months...eventually you'll notice weird behavior from Windows, like apps crashing, other programs running slow, things not closing correctly, overall slowness, etc. You reboot your machine, maybe update your drivers every so often, etc and then you're back and off to the races.

Servers require similar maintenance. They sometimes require reboots in order to restart services, or apply an update to something, etc. Often times though, maintanence is "scheduled", as part of a proactive effort to minimize downtime. You can either never reboot, wait until SERIOUS problems happen, then do something about it...OR you can reboot every week, even if there are no serious problems, to insure the server is always in a high performance state.

Problems can still happen, however it is common practice to do weekly reboots, which is what normal server maintenance entails, among other things. It's like this...do you wait until your house is really dusty and dirty to vacuum or clean it, only once every like 6 months? Or do you dust and vacuum your house every 2 weeks or so, and perform regular house maintenance (taking out the trash, wiping down counters, cleaning your toilet, etc). It's a similar thing with servers and computers in general.

furiouslog

They probably read these forum reactions, sigh loudly, and then go grab a triple scotch (neat) and take some "me time" until the maintenance is over.

MarioMario

Server down in EU primetime. Again.

Jacozilla

logan_ESO wrote: »

Server down in EU primetime. Again.

Is there a single EU based game that does maintenance during non-EU preferred hours?

e.g. I play a UK based game but live in NA, I'm not surprised nor upset it does maint. during U.K. preferred hours (their downtime, my primetime).

WuffyCerulei

Herding the hamsters back to their wheels.

Marto

Work

NupidStoob

Depending on what kind of maintenance you do and what is planned different things happen.

Here are some of the things that are typically done (really not an exhaustive list and some of these things can be done less often than others):

- Shutting down all servers (this actually can take some time).

- cleaning servers, removing dust etc.

- visual checkup to see if something is broken or seems faulty

- Replace/repair faulty servers

- backup data

- install updates

- run testing software

- boot the whole thing back up again


Now imagine you are in a 2-4 man team and have to do this for hundreds of servers and you can see why this can take so much time.

Alurria

NupidStoob wrote: »

Depending on what kind of maintenance you do and what is planned different things happen.

Here are some of the things that are typically done (really not an exhaustive list and some of these things can be done less often than others):

- Shutting down all servers (this actually can take some time).

- cleaning servers, removing dust etc.

- visual checkup to see if something is broken or seems faulty

- Replace/repair faulty servers

- backup data

- install updates

- run testing software

- boot the whole thing back up again


Now imagine you are in a 2-4 man team and have to do this for hundreds of servers and you can see why this can take so much time.

And you might be the guy who has to drive to the server farm in the middle of no where because that way people get better frame rates and get stuck on a highway with a major accident. (I don't know if this is true but i could happen)

Reverb

They apply OS and security patches to the servers and reboot everything. They likely reindex and optimize several of the database tables.

The length of time needed depends on the size of the patches being applied, the scope of DB maintenance, and whether or not issues are encountered in taking the farm down gracefully or bringing it back up.

Today it is likely that they are also pre-staging some components of the Gates of Oblivion chapter, which is something they’ve done the last few cycles to make the actual go-live day patching shorter and smoother.

Lerozain

Here's a tidbit I found posted by a MMO dev in answer to this question:

"Mainly we just reset all of the server processes. Server processes are the things that handle tasks and behavior like chat, login, world areas, combat, spawning, guild banks, mail, and the like. Resetting the processes clears out any fragmentation that may have occurred in the server’s memory, and it clears out any zombie processes that died during the week and needed to be restarted. Our automated scripts then process most of the week’s data (e.g. parsing and generating reports from the telemetry captured). Logs of what happened that week are flushed to disk. Data and databases are backed up. Then we restart the game processes, re-initializing timers for quests, rare spawns, weekly events, raid bosses, and so on.

During some (but not all) maintenance sessions, we will apply non-emergency patches to the server in order to update it with new code. Usually this entails bug fixes, optimizations, and support for new or upcoming features. It isn’t uncommon for us to update the live servers with bits and pieces of expansion code and systems before launch and simply lock it off until the client update goes live. Sometimes we also use maintenance to swap out or upgrade hardware. Sometimes we use the time to run scripts to do a variety of tasks, like clean up old accounts or handle server merges. Sometimes we will migrate data from one machine to another, or we’ll replace/upgrade old hardware with new machines. Given how much data MMOGs generate, we tend to run through a lot of hard drives.

Maintenance is really pretty boring. Most of it is automated and handled by tasks scheduled on the servers themselves. There’s also usually 1-2 live ops and/or server team engineers rotating to oversee it each week to make sure nothing really catastrophic happens while it goes on. Overall, though, that’s about it."

MarioMario

Jacozilla wrote: »

logan_ESO wrote: »

Server down in EU primetime. Again.

Is there a single EU based game that does maintenance during non-EU preferred hours?[...]

Dunno, I play only ESO

TwiceBornStar

I know what I'm doing on maintenance day..

Nothing!

jedtb16_ESO

the guys come over to my place we have a few beers, shoot the breeze then go down to the lake and throw marshmallows at ducks...

n.b. no ducks are harmed in this process.

TwinLamps

Im sure of these two things:
On day of maintenance they drink coffee
On day of maintenance they extend PC EU maintenance

Marto

Alurria wrote: »

NupidStoob wrote: »

Depending on what kind of maintenance you do and what is planned different things happen.

Here are some of the things that are typically done (really not an exhaustive list and some of these things can be done less often than others):

- Shutting down all servers (this actually can take some time).
- cleaning servers, removing dust etc.
- visual checkup to see if something is broken or seems faulty
- Replace/repair faulty servers
- backup data
- install updates
- run testing software
- boot the whole thing back up again


Now imagine you are in a 2-4 man team and have to do this for hundreds of servers and you can see why this can take so much time.

And you might be the guy who has to drive to the server farm in the middle of no where because that way people get better frame rates and get stuck on a highway with a major accident. (I don't know if this is true but i could happen)

Whenever annoying delays or cancellations happen to videogames, specially their updates, I always think back to Battlefield 4, which was supposed to get 2 or 3 night-time versions of maps. One was released, and was quite fun, with an outdoor area barely illuminated by moonlight, and an indoor area that was pitch black. They definitely put a lot of work and design into the lighting.

Despite the positive feedback and good response from the community, the other maps were cancelled.

Why? Because their main lighting engineer hated Los Angeles traffic, and decided it was healthier to just quit his job, and move to another state.

And I mean... Who wouldn't hate LA traffic? I've never been there, but I've seen videos and heard the stories.

S*** happens. That's the moral of the story.

Kamchuk

They sit back, down a few pints, read these forums - take bets on which ones are the most ridiculous, and laugh. Since they are in Bethesda Maryland and the traffic is absolutely the worst of any metropolitan city, (Washington DC area), they are relaxing until the traffic goes down so they can get home and play other computer games.

Vonkarolinas

zharkovian wrote: »

They get the truffles and bonbons out and tickle each other and giggle.

I love it. This is now, officially, what happens on maintenance day.

vgabor

majulook

Will it be up at 3:00PM EDT (19:00 UTC) ????????

Kamchuk

vgabor wrote: »

Awesome!

ZOS_ConnorG

Greetings all,

After review we have closed this thread as there are multiple other threads currently open about this topic.