Stop with the weekly maintenaces already!!!

cjthibs · October 2016

Bigevilpeter wrote: »

Well that was interesting, but it doesn't excuse the long downtimes and the downtimes because of the downtimes

"I reject your reality and substitute my own" --Bigevilpeter.

wenchmore420b14_ESO · October 2016

To comparing ESO to other MMO's.
Every game mentioned above is way older than ESO. We are only 2 1/2 years old. Remember WoW at 2 years old?? It was way worse than ESO!
Give the delvs a break! They have done a amazing job of fixing the problems in this game since launch and have done a better job than a lot of other games in the short time ESO has been live.
Are there problems? Yes. But complaining every Monday won't help.

Immortal_Dark410 · October 2016

wenchmore420b14_ESO wrote: »

To comparing ESO to other MMO's.
Every game mentioned above is way older than ESO. We are only 2 1/2 years old. Remember WoW at 2 years old?? It was way worse than ESO!
Give the delvs a break! They have done a amazing job of fixing the problems in this game since launch and have done a better job than a lot of other games in the short time ESO has been live.
Are there problems? Yes. But complaining every Monday won't help.

Yup 100% Agree

Kryptonite_Kent · October 2016

Solid_Metal wrote: »

Kryptonite_Kent wrote: »

old_mufasa wrote: »

Bigevilpeter wrote: »

Omg do them once a month, every time you do a maintenance you break the game and just to make a few fixes. Please for love of Akatosh stop the weekly maintenance.

Wow.. can you be more ignorant

Almost all major mmo's do server maintenance weekly.. and yes server can become unstable and shut done if you do not do the maintenance... even more stable mmos then ESO do it. Maybe you should educate yourself about why and the effects of not doing it before running you mouth off...

Yeah I have played reputable MMO's... you know where they actually tell you whats going on when something happens, they give you a timeline for maint downtime, and normal weekly maint is 2 - maybe 4 hours, not 8 lol.

Gina actually confirmed they know the problem, and take "couple" hours to fix it, but indeed i agree with the communication part
you know it take 8 right now because sh*t happen right?, normally it take 3-4 hours

Except uhh... pretty much every recent maintenance that I can recall has been at least double 3-4 hours and almost every one has been extended... yes every MMO does maintenance, but come on theres a bit of incompetence going on when its more common to have a problem with maintenance and it to last 8+ hours than for it to be a simple 3-4 hours and good to go.

Bigevilpeter · October 2016

cjthibs wrote: »

Bigevilpeter wrote: »

Well that was interesting, but it doesn't excuse the long downtimes and the downtimes because of the downtimes

"I reject your reality and substitute my own" --Bigevilpeter.

"I write weird stuff that don't make any sense" -- cjthibs

CapnPhoton · October 2016

Tandor wrote: »

Every time there's a week without maintenance the complainers moan about the servers performing badly and call for weekly maintenance. Every time they get weekly maintenance the complainers moan about the amount of downtime and the new ways ZOS have found of breaking the game.

Who'd be a developer? They can never do right for doing wrong.

With a world full of entitled people, trying to do the right thing will seem like a losing battle. But if people just do their work, succeed, and be positive, they can easily ignore those that make it their mission in life to be negative, and complain about everything.

Toybot · October 2016

wenchmore420b14_ESO wrote: »

To comparing ESO to other MMO's.
Every game mentioned above is way older than ESO. We are only 2 1/2 years old. Remember WoW at 2 years old?? It was way worse than ESO!
Give the delvs a break! They have done a amazing job of fixing the problems in this game since launch and have done a better job than a lot of other games in the short time ESO has been live.
Are there problems? Yes. But complaining every Monday won't help.

I disagree. Complaining can help because knowing problems allows us to invent improvements. Investigate Microservices because that's what allows companies like Netflix to deliver entertainment without downtimes like what we are currently experiencing.

cjthibs · October 2016

Bigevilpeter wrote: »

cjthibs wrote: »

Bigevilpeter wrote: »

Well that was interesting, but it doesn't excuse the long downtimes and the downtimes because of the downtimes

"I reject your reality and substitute my own" --Bigevilpeter.

"I write weird stuff that don't make any sense" -- cjthibs

Averya_Teira · October 2016

Weekly maintenance is pretty standard for MMOs.....

You must be new to online gaming?

Kryptonite_Kent · October 2016

wenchmore420b14_ESO wrote: »

To comparing ESO to other MMO's.
Every game mentioned above is way older than ESO. We are only 2 1/2 years old. Remember WoW at 2 years old?? It was way worse than ESO!
Give the delvs a break! They have done a amazing job of fixing the problems in this game since launch and have done a better job than a lot of other games in the short time ESO has been live.
Are there problems? Yes. But complaining every Monday won't help.

This is like saying, hey its ok that my hamburger is undercooked and has the wrong toppings because this restaurant is new compared to the one down the street that's been making hamburgers for 10 years...

imattc · October 2016

Just don't keep the hood open for too long ZoS, okay? We want to get back to the game.

Solid_Metal · October 2016

Kryptonite_Kent wrote: »

Solid_Metal wrote: »

Kryptonite_Kent wrote: »

old_mufasa wrote: »

Bigevilpeter wrote: »

Omg do them once a month, every time you do a maintenance you break the game and just to make a few fixes. Please for love of Akatosh stop the weekly maintenance.

Wow.. can you be more ignorant

Almost all major mmo's do server maintenance weekly.. and yes server can become unstable and shut done if you do not do the maintenance... even more stable mmos then ESO do it. Maybe you should educate yourself about why and the effects of not doing it before running you mouth off...

Yeah I have played reputable MMO's... you know where they actually tell you whats going on when something happens, they give you a timeline for maint downtime, and normal weekly maint is 2 - maybe 4 hours, not 8 lol.

Gina actually confirmed they know the problem, and take "couple" hours to fix it, but indeed i agree with the communication part
you know it take 8 right now because sh*t happen right?, normally it take 3-4 hours

Except uhh... pretty much every recent maintenance that I can recall has been at least double 3-4 hours and almost every one has been extended... yes every MMO does maintenance, but come on theres a bit of incompetence going on when its more common to have a problem with maintenance and it to last 8+ hours than for it to be a simple 3-4 hours and good to go.

nope, recent maintenance they never doubling anything AFAIK, top are 5 hours and thats it, this the only one so far that more than 5 hours, theres probably something like this yeah, but it couple months back

Kryptonite_Kent · October 2016

Solid_Metal wrote: »

Kryptonite_Kent wrote: »

Solid_Metal wrote: »

Kryptonite_Kent wrote: »

old_mufasa wrote: »

Bigevilpeter wrote: »

Omg do them once a month, every time you do a maintenance you break the game and just to make a few fixes. Please for love of Akatosh stop the weekly maintenance.

Wow.. can you be more ignorant

Almost all major mmo's do server maintenance weekly.. and yes server can become unstable and shut done if you do not do the maintenance... even more stable mmos then ESO do it. Maybe you should educate yourself about why and the effects of not doing it before running you mouth off...

Yeah I have played reputable MMO's... you know where they actually tell you whats going on when something happens, they give you a timeline for maint downtime, and normal weekly maint is 2 - maybe 4 hours, not 8 lol.

Gina actually confirmed they know the problem, and take "couple" hours to fix it, but indeed i agree with the communication part
you know it take 8 right now because sh*t happen right?, normally it take 3-4 hours

Except uhh... pretty much every recent maintenance that I can recall has been at least double 3-4 hours and almost every one has been extended... yes every MMO does maintenance, but come on theres a bit of incompetence going on when its more common to have a problem with maintenance and it to last 8+ hours than for it to be a simple 3-4 hours and good to go.

nope, recent maintenance they never doubling anything AFAIK, top are 5 hours and thats it, this the only one so far that more than 5 hours, theres probably something like this yeah, but it couple months back

Lolwut... are you playing the same game? Just like last week or the week before there was maintenance more than once and extended maintenance on that!

InvitationNotFound · October 2016

wenchmore420b14_ESO wrote: »

Bigevilpeter wrote: »

Omg do them once a month, every time you do a maintenance you break the game and just to make a few fixes. Please for love of Akatosh stop the weekly maintenance.

OMG! No! Who remembers the Christmas of 2014 when they went over a month with out maintenance? The game was almost unplayable!!

Servers won't crash if they don't do maintenaces ok maybe it will after a very long time without maintenance but that's actually what happens when they do, so if its going to crash anyways then at least it will be without the patching downtime and then they can patch when they actually need to, we get double patching time now

So, you are a computer coder? An IT tech? ME either. But! Unlike others here on forum, when I dont know what I'm talking about, I educate myself and learn about it. Do you know what "Maintenance actually entails @Bigevilpeter ?
Well here you are...(FYI..Thanks to @Cronopoly for posting this in another thread...)

*"Firewall's: Enterprise systems must use them for protected DMZ's, Demilitarized Zones as they are called which protect all network traffic within (Incoming and outgoing) as having Firewalls controlling data coming in and out by port and protocol to prevent unwarranted traffic & hackers etc... We used separate Firewall Zones for Initiall Web ingress of traffic, Credit Card zone, Database Zone, Application Zones, this way all different types of data was protected separately. So much easier to put rules around it that way, though not simple. I'll skip the talk of State tables, and NAT's getting full...

Most enterprise level Firewalls are configured in pairs with a crossover cable to make sure that if one side fails the other can take the traffic seamlessly without users knowing a failure occured. (redundancy is huge)

Firewalls maintenance windows for vendor updates (security etc) are required unless you want to be hacked sooner than later. this can sometimes take the Firewall team hours as they do one Firewall Pair at a time. Firewalls are most finicky when updating and sometimes require rebooting several times to become stable. And being that a leading vendor is in Israel (Checkpoint) makes this a pain if you have to contact them with timezones involved.

Application Servers / HTTP Servers Many applications can use app servers to host the users session, that in the case of an MMO might hold the states of a users session with the mirrored variables of the users location, stats, all actions etc.

App servers along with getting required security and O'/S patches which can take hours to apply, (shutdown, patching, startup), can be just that servers that get "Dirty" over time especially with the use of code that leverages JVM's (Java) that can get corrupted over time and have frequent memory problems. spelled pain in the butt. Required weekly reboots are common to prevent alot of problems. With an MMO there are many many servers to reboot and double check that they come back up clean, as users are pretty unforgiving if after maintenance something is not fully working...

Network Typically the most stable part of an enterprise once setup correctly, however with more and more exploits targeting Internet facing Routers with new vulnerabilities that you guessed it have to be Security/firmware patched.

ACE/VPN concentrators: Commonly used to provide a secure tunnel of connected users from the Internet to datacenter backend systems. These must always get patched like everything else and care taken to deprecate old protocols that over time always have vulnerabilities pop up. TLS etc...

One network component that does have a higher failure rate (and time fixing them) are Load Balancers / Content Switches. These can take more maintenance as they are typically setup to take a network request targeting a single IP address and rout traffic round robin etc to multiple backend servers. The pain this causes during longer maintenance window due to Network team members not being thorough or overworked cannot be understated. I always cringed when I saw a change control for a CSS (content switch).

Database servers I'll state this simply as I've gone down the wabbit hole too far already considering this is a gaming forum and not "Unpaid consulting hour" Obviously all your data in several databases in an MMO have millions of transactions of reads and writes. Databases need maintenance in order to optimize paths to the disk to shorten read and write times. Database Administrators can run Stats, Reorgs, create new Indexes or refresh index to keep your data access optimal.
Simply put on large databases this can take HOURS. Nothing is free here and it's typically not optional unless you want your system response time to slow to a crawl, and in an MMO that would kill your application for all users...

You have the same level of complexity with other related systems:

Storage Farms: Servers are typically connected to their remote NAS or SAN hard drives (EMC etc) which require periodic maintenance as well.

Backup and Recovery Servers need to be backed up just in case of catastopophic failure which does occur. Many backups need to take place when users are not on the system. This can sometimes take hour(s) depending on how much local storage of the O/S+ Application and Data are on the server. I can see this occuring 1st during a maintenance window.

TL:DR - Running an MMO requires mandatory maintenance to keep everything running and "secure". No free lunch.

what the f***?

I guess I've highlighted the most important part of your post. You wanted to say you have no idea what you're writing about, right?

I've just skimmed over it as it mostly seemed like you have no idea. Just a few examples:
Firewall: Great description but irrelevant and i doubt you understand how such devices can work. Anyway, the point is, updating such a device is usually pretty fast. You generally don't have a downtime when reconfiguring it. Therefore, this doesn't justify the current downtime at all.
Application Servers / HTTP Servers: Mostly same as above, if you're doing it right, the downtime is short. Really short. And you're going to test if everything will works fine in your test environment (which is mostly a clone of the production environment) before updating your production environment.
ACE/VPN concentrators: Has nothing to do with their productive (gaming) environment. Working on these components shouldn't have an effect on the game at all. It is very unlikely that this affects the maintenance. If you want to talk about network equipment in general see my Firewall comment above.
Storage Farms: You're talking about the network equipment. See my Firewall comment above. It's the same.
Backup and Recovery Servers: At this point, I even doubt they have such systems

If it's about the equipment: See my Firewall comment above.

At this point I have no idea what exactly they are doing during maintenance. But i doubt the only "update" some devices. I rather think they try to improve performance by cleaning up the database, improve indexing or whatever. But simply updating some devices sounds rather trivial, and by the way, most of the components could be updated without a downtime (redundancy, you know...). I highly doubt some updates would cause the current behavior. But feel free to ask ZOS what exactly they are doing, but your list above doesn't make sense in the current context.

edit: sorry for the edits, this quoting got messed up pretty much >_<

wenchmore420b14_ESO · October 2016

Toybot wrote: »

wenchmore420b14_ESO wrote: »

To comparing ESO to other MMO's.
Every game mentioned above is way older than ESO. We are only 2 1/2 years old. Remember WoW at 2 years old?? It was way worse than ESO!
Give the delvs a break! They have done a amazing job of fixing the problems in this game since launch and have done a better job than a lot of other games in the short time ESO has been live.
Are there problems? Yes. But complaining every Monday won't help.

I disagree. Complaining can help because knowing problems allows us to invent improvements. Investigate Microservices because that's what allows companies like Netflix to deliver entertainment without downtimes like what we are currently experiencing.

True !I agree. Constructive complaints can help to get results, but the same old "Maintenance again" threads every week gets really old after 3 years.
You know, they used to do maintenance twice a week? Mondays and Thursdays for the first year. Then they got stable and went to once a week.. Yes, they improved.
As far as the "Microservices", Zos uses a "Megaserver" tech that is unique to Zos, so I don't think it compares to Netfilx type of server as netflix doesnt have to keep record of millions of character accounts, guild infos, etc.

kevlarto_ESO · October 2016

If they let the servers go for more than 2 weeks with out some kind of down time the game would be unplayable, I am fine with down time, take your time get things right, down time is needed for online games, even phone apps have down time for maintenance.

I welcome down time this game needs a lot more because there is so many bugs and broken things and they don't get fixed without downtime. Shame is they have such poor in house testing the patches end up breaking new things with every fix /sigh

Kryptonite_Kent · October 2016

kevlarto_ESO wrote: »

If they let the servers go for more than 2 weeks with out some kind of down time the game would be unplayable, I am fine with down time, take your time get things right, down time is needed for online games, even phone apps have down time for maintenance.

I welcome down time this game needs a lot more because there is so many bugs and broken things and they don't get fixed without downtime. Shame is they have such poor in house testing the patches end up breaking new things with every fix /sigh

EXCEPT... they never fix the broken things... they fix some random crap thing like "oh some tree was a slightly wrong shade of brown, or some NPC was spawning too fast, or some grass wasn't rendering properly in some obscure location" meanwhile there are game breaking bugs like falling through the map while gap closing, gap closers not functioning properly due to terrain, cloaks not working correctly, etc, etc, etc...

Hempyre · October 2016

old_mufasa wrote: »

Bigevilpeter wrote: »

Omg do them once a month, every time you do a maintenance you break the game and just to make a few fixes. Please for love of Akatosh stop the weekly maintenance.

Wow.. can you be more ignorant

Almost all major mmo's do server maintenance weekly.. and yes server can become unstable and shut done if you do not do the maintenance... even more stable mmos then ESO do it. Maybe you should educate yourself about why and the effects of not doing it before running you mouth off...

Actually, you're the one being ignorant.

Most companies or "MMO's" as you put it, off-load and maintain service during maintenance.

I'm not saying I agree with the OP, but the way zos is doing maintenance is archaic. To top it off, there was a clear lack of testing on this patch before it went live. That's just pis-s poor planning and development.

I'm content to just sit here and twiddle my thumbs, at least until I see a joker like you "running your mouth".

Getting back to your ignorance. The use of ellipses requires three periods, not two, and it's used when omitting words or lines that are not relevant to your point. Not for whatever the *** you're struggling to spit out. Lets not talk about your spelling, punctuation and grammar.

Sit the *** down son.

Solid_Metal · October 2016

Kryptonite_Kent wrote: »

Solid_Metal wrote: »

Kryptonite_Kent wrote: »

Solid_Metal wrote: »

Kryptonite_Kent wrote: »

old_mufasa wrote: »

Bigevilpeter wrote: »

Omg do them once a month, every time you do a maintenance you break the game and just to make a few fixes. Please for love of Akatosh stop the weekly maintenance.

Wow.. can you be more ignorant

Almost all major mmo's do server maintenance weekly.. and yes server can become unstable and shut done if you do not do the maintenance... even more stable mmos then ESO do it. Maybe you should educate yourself about why and the effects of not doing it before running you mouth off...

Yeah I have played reputable MMO's... you know where they actually tell you whats going on when something happens, they give you a timeline for maint downtime, and normal weekly maint is 2 - maybe 4 hours, not 8 lol.

Gina actually confirmed they know the problem, and take "couple" hours to fix it, but indeed i agree with the communication part
you know it take 8 right now because sh*t happen right?, normally it take 3-4 hours

Except uhh... pretty much every recent maintenance that I can recall has been at least double 3-4 hours and almost every one has been extended... yes every MMO does maintenance, but come on theres a bit of incompetence going on when its more common to have a problem with maintenance and it to last 8+ hours than for it to be a simple 3-4 hours and good to go.

nope, recent maintenance they never doubling anything AFAIK, top are 5 hours and thats it, this the only one so far that more than 5 hours, theres probably something like this yeah, but it couple months back

Lolwut... are you playing the same game? Just like last week or the week before there was maintenance more than once and extended maintenance on that!

more like emergerncy maintenance after Tamriel One update, it also happen on WoD update for WoW mind you, dunno if it also happen in Legion
we talking about extended weekly maintenance like this, which is normally only 3-5 hours top
and also as i recall, those "emergency" maintenance also take very short

wenchmore420b14_ESO · October 2016

@InvitationNotFound
Hi! Thanks for the critique. Yes, I am a Chef in life, not a programmer. The above post was copied from a response to another "Maintenance" thread from a while back. The post was written by a 21 year computer programmer as a explanation of what a maintenance is and does in general. It was just for folks that think maintenance is not necessary.
But as I said, I know about as much about coding/programming as I do about Quantum Physics. But, I want to learn.
So, if you are a programmer and can clarify how the Megaserver works and why maintenance is necessary/ not necessary, please share with us.

Kryptonite_Kent wrote: »

wenchmore420b14_ESO wrote: »

To comparing ESO to other MMO's.
Every game mentioned above is way older than ESO. We are only 2 1/2 years old. Remember WoW at 2 years old?? It was way worse than ESO!
Give the delvs a break! They have done a amazing job of fixing the problems in this game since launch and have done a better job than a lot of other games in the short time ESO has been live.
Are there problems? Yes. But complaining every Monday won't help.

This is like saying, hey its ok that my hamburger is undercooked and has the wrong toppings because this restaurant is new compared to the one down the street that's been making hamburgers for 10 years...

LMAO! Never been to a soft open of a restaurant have you..Your response made me laugh! Thank you..

AsteriaStarfall · October 2016

Bigevilpeter wrote: »

Omg do them once a month, every time you do a maintenance you break the game and just to make a few fixes. Please for love of Akatosh stop the weekly maintenance.

Dear OP

I want to take a moment to possibly educate you a little bit in the world of I.T.

I work as a network admin for a company of just over 100 employees. We of course house our own servers for things like file shares, email, and websites. For a game and server complexity that ESO has, i want to share some of things that they have to do on a daily basis as well as when maintenance comes up.

My company runs an email server that uses greylisting as a form of spam prevention. The greylisting program utilizes MySQL as its database. This greylisting will listen to incoming email requests and reject them telling the sending server to resend the email after a specified amount of time. Now if things go wrong in the database table then a problem like we encountered just the other day would come up where legitimate email was being rejected and no record of it was being recorded in the log file. We had to take the server down for maintenance to rebuild the MySQL database. It took about 30 minutes and everyone was back up and running. The company is not large enough and not willing to invest in a secondary set of servers to be used as mirrors in case some have to go offline. So I have to manage maintenance in the evenings to hopefully cut down daytime maintenance that would affect the productivity of employees. But it does happen where things go offline during the day. It is unavoidable. There is no way to guarantee 100% server uptime.

Problems like the one I had above and crop up out of nowhere and until you narrow down what the problem is, you have to investigate and review hundreds of lines of log files to see where the issue might be.

So many things can come up and go wrong even in little server configurations such as small and medium size businesses. Now picture the ESO servers. These servers have multiple databases and multiple servers running in sync with each other. They have to not only know the position and movement of all player and NPC characters, but they also have to manage inventory, spell abilities, spell procs, countless guild trader transactions, and much more. Every little thing you do and all the players do in the game, those commands get sent to the server and they have to process them in real time and in such a way that players cannot exploit a weakness in it, such as duplicating items, or glitching through walls.

So when maintenance occurs, they run database integrity checks, make sure everything adds up (I am not a database admin and will not fake knowing everything that goes into it). They also check for bandwidth and bottleneck issues, are there any issues with the running processes themselves such as memory leaks or high cpu usage because a process is not executing properly?

I am quite sure there are many many more things that a much larger organization like Bethesda has to do during this time. But in the end, it is to make the game you play and enjoy that much better and more stable. Downtime does suck but if you are wanting a game that requires zero downtime, then i would suggest you play an offline game that does not require it talking to a server to play.

Kryptonite_Kent · October 2016

Solid_Metal wrote: »

Kryptonite_Kent wrote: »

Solid_Metal wrote: »

Kryptonite_Kent wrote: »

Solid_Metal wrote: »

Kryptonite_Kent wrote: »

old_mufasa wrote: »

Bigevilpeter wrote: »

Omg do them once a month, every time you do a maintenance you break the game and just to make a few fixes. Please for love of Akatosh stop the weekly maintenance.

Wow.. can you be more ignorant

Almost all major mmo's do server maintenance weekly.. and yes server can become unstable and shut done if you do not do the maintenance... even more stable mmos then ESO do it. Maybe you should educate yourself about why and the effects of not doing it before running you mouth off...

Yeah I have played reputable MMO's... you know where they actually tell you whats going on when something happens, they give you a timeline for maint downtime, and normal weekly maint is 2 - maybe 4 hours, not 8 lol.

Gina actually confirmed they know the problem, and take "couple" hours to fix it, but indeed i agree with the communication part
you know it take 8 right now because sh*t happen right?, normally it take 3-4 hours

Except uhh... pretty much every recent maintenance that I can recall has been at least double 3-4 hours and almost every one has been extended... yes every MMO does maintenance, but come on theres a bit of incompetence going on when its more common to have a problem with maintenance and it to last 8+ hours than for it to be a simple 3-4 hours and good to go.

nope, recent maintenance they never doubling anything AFAIK, top are 5 hours and thats it, this the only one so far that more than 5 hours, theres probably something like this yeah, but it couple months back

Lolwut... are you playing the same game? Just like last week or the week before there was maintenance more than once and extended maintenance on that!

more like emergerncy maintenance after Tamriel One update, it also happen on WoD update for WoW mind you, dunno if it also happen in Legion
we talking about extended weekly maintenance like this, which is normally only 3-5 hours top
and also as i recall, those "emergency" maintenance also take very short

So wait, you're saying yes there is "emergency" maintenance, but that's different than extended maintenance? At this point you sound like a paid spokesperson making up some cryptic excuses for obvious shortcomings... "oh this isn't 'extended' maintenance guys, its 'emergency' maintenance... and normal downtime is 3-5 hours 'usually', but sometimes we have slightly longer extended times due to emergencies and the sun was in my eyes and WoW has maintenance too and stuff and things"

Toybot · October 2016

wenchmore420b14_ESO wrote: »

True !I agree. Constructive complaints can help to get results, but the same old "Maintenance again" threads every week gets really old after 3 years.
You know, they used to do maintenance twice a week? Mondays and Thursdays for the first year. Then they got stable and went to once a week.. Yes, they improved.
As far as the "Microservices", Zos uses a "Megaserver" tech that is unique to Zos, so I don't think it compares to Netfilx type of server as netflix doesnt have to keep record of millions of character accounts, guild infos, etc.

I see your train of thought as to how the two services have differences but I cannot yet foresee how Microservice technology would be prohibitive for them.

Absolutely, it takes re-engineering but it's the future and one of the reasons why is because Microservices provides the opportunity to make planned maintenance system outages to be a thing of the past.

Bigevilpeter · October 2016

AsteriaStarfall wrote: »

Bigevilpeter wrote: »

Omg do them once a month, every time you do a maintenance you break the game and just to make a few fixes. Please for love of Akatosh stop the weekly maintenance.

Dear OP

I want to take a moment to possibly educate you a little bit in the world of I.T.

I work as a network admin for a company of just over 100 employees. We of course house our own servers for things like file shares, email, and websites. For a game and server complexity that ESO has, i want to share some of things that they have to do on a daily basis as well as when maintenance comes up.

My company runs an email server that uses greylisting as a form of spam prevention. The greylisting program utilizes MySQL as its database. This greylisting will listen to incoming email requests and reject them telling the sending server to resend the email after a specified amount of time. Now if things go wrong in the database table then a problem like we encountered just the other day would come up where legitimate email was being rejected and no record of it was being recorded in the log file. We had to take the server down for maintenance to rebuild the MySQL database. It took about 30 minutes and everyone was back up and running. The company is not large enough and not willing to invest in a secondary set of servers to be used as mirrors in case some have to go offline. So I have to manage maintenance in the evenings to hopefully cut down daytime maintenance that would affect the productivity of employees. But it does happen where things go offline during the day. It is unavoidable. There is no way to guarantee 100% server uptime.

So many things can come up and go wrong even in little server configurations such as small and medium size businesses. Now picture the ESO servers. These servers have multiple databases and multiple servers running in sync with each other. They have to not only know the position and movement of all player and NPC characters, but they also have to manage inventory, spell abilities, spell procs, countless guild trader transactions, and much more. Every little thing you do and all the players do in the game, those commands get sent to the server and they have to process them in real time and in such a way that players cannot exploit a weakness in it, such as duplicating items, or glitching through walls.

So when maintenance occurs, they run database integrity checks, make sure everything adds up (I am not a database admin and will not fake knowing everything that goes into it). They also check for bandwidth and bottleneck issues, are there any issues with the running processes themselves such as memory leaks or high cpu usage because a process is not executing properly?

I am quite sure there are many many more things that a much larger organization like Bethesda has to do during this time. But in the end, it is to make the game you play and enjoy that much better and more stable. Downtime does suck but if you are wanting a game that requires zero downtime, then i would suggest you play an offline game that does not require it talking to a server to play.

Customers want results not excuses, I don't care why I got my food uncooked, I don't care why the phone I bought didn't work etc.

There are other mmos who have waaaay less downtime and still offer a very good service. I'm not an IT expert and don't want to be, I want to be able to play the game I payed a lot of money in and subbed in

Zamuro · October 2016

i havent seen so bad servers and maintenances in other mmo. im saying servers because im playing with 100+ ping in eu server which is unacceptable, the normal ping should be 40-50(thats what i get in all other games)

Solid_Metal · October 2016

Kryptonite_Kent wrote: »

Solid_Metal wrote: »

Kryptonite_Kent wrote: »

Solid_Metal wrote: »

Kryptonite_Kent wrote: »

Solid_Metal wrote: »

Kryptonite_Kent wrote: »

old_mufasa wrote: »

Bigevilpeter wrote: »

Omg do them once a month, every time you do a maintenance you break the game and just to make a few fixes. Please for love of Akatosh stop the weekly maintenance.

Wow.. can you be more ignorant

Almost all major mmo's do server maintenance weekly.. and yes server can become unstable and shut done if you do not do the maintenance... even more stable mmos then ESO do it. Maybe you should educate yourself about why and the effects of not doing it before running you mouth off...

Yeah I have played reputable MMO's... you know where they actually tell you whats going on when something happens, they give you a timeline for maint downtime, and normal weekly maint is 2 - maybe 4 hours, not 8 lol.

Gina actually confirmed they know the problem, and take "couple" hours to fix it, but indeed i agree with the communication part
you know it take 8 right now because sh*t happen right?, normally it take 3-4 hours

Except uhh... pretty much every recent maintenance that I can recall has been at least double 3-4 hours and almost every one has been extended... yes every MMO does maintenance, but come on theres a bit of incompetence going on when its more common to have a problem with maintenance and it to last 8+ hours than for it to be a simple 3-4 hours and good to go.

nope, recent maintenance they never doubling anything AFAIK, top are 5 hours and thats it, this the only one so far that more than 5 hours, theres probably something like this yeah, but it couple months back

Lolwut... are you playing the same game? Just like last week or the week before there was maintenance more than once and extended maintenance on that!

more like emergerncy maintenance after Tamriel One update, it also happen on WoD update for WoW mind you, dunno if it also happen in Legion
we talking about extended weekly maintenance like this, which is normally only 3-5 hours top
and also as i recall, those "emergency" maintenance also take very short

So wait, you're saying yes there is "emergency" maintenance, but that's different than extended maintenance? At this point you sound like a paid spokesperson making up some cryptic excuses for obvious shortcomings... "oh this isn't 'extended' maintenance guys, its 'emergency' maintenance... and normal downtime is 3-5 hours 'usually', but sometimes we have slightly longer extended times due to emergencies and the sun was in my eyes and WoW has maintenance too and stuff and things"

you can perceive it as whatever you like really, for me the maintenance during big update is acceptable since *** can happen on those time, like almost all mmo with big update roll
and now weekly maintenance that usually only take 3-5 hours take a bit longer cause of *** happen, and this only happen once in a while
you probably can look up at google that other mmo also experience this kind of suff, how big your game, how detail are you on testing and preparing update, *** can happen

raglau · October 2016

Bigevilpeter wrote: »

wenchmore420b14_ESO wrote: »

Bigevilpeter wrote: »

Omg do them once a month, every time you do a maintenance you break the game and just to make a few fixes. Please for love of Akatosh stop the weekly maintenance.

OMG! No! Who remembers the Christmas of 2014 when they went over a month with out maintenance? The game was almost unplayable!!
[quote="Bigevilpeter;3473139"]Servers won't crash if they don't do maintenaces ok maybe it will after a very long time without maintenance but that's actually what happens when they do, so if its going to crash anyways then at least it will be without the patching downtime and then they can patch when they actually need to, we get double patching time now

So, you are a computer coder? An IT tech? ME either. But! Unlike others here on forum, when I dont know what I'm talking about, I educate myself and learn about it. Do you know what "Maintenance actually entails @Bigevilpeter ?
Well here you are...(FYI..Thanks to @Cronopoly for posting this in another thread...)

*"Firewall's: Enterprise systems must use them for protected DMZ's, Demilitarized Zones as they are called which protect all network traffic within (Incoming and outgoing) as having Firewalls controlling data coming in and out by port and protocol to prevent unwarranted traffic & hackers etc... We used separate Firewall Zones for Initiall Web ingress of traffic, Credit Card zone, Database Zone, Application Zones, this way all different types of data was protected separately. So much easier to put rules around it that way, though not simple. I'll skip the talk of State tables, and NAT's getting full...

Most enterprise level Firewalls are configured in pairs with a crossover cable to make sure that if one side fails the other can take the traffic seamlessly without users knowing a failure occured. (redundancy is huge)

Firewalls maintenance windows for vendor updates (security etc) are required unless you want to be hacked sooner than later. this can sometimes take the Firewall team hours as they do one Firewall Pair at a time. Firewalls are most finicky when updating and sometimes require rebooting several times to become stable. And being that a leading vendor is in Israel (Checkpoint) makes this a pain if you have to contact them with timezones involved.

Application Servers / HTTP Servers Many applications can use app servers to host the users session, that in the case of an MMO might hold the states of a users session with the mirrored variables of the users location, stats, all actions etc.

App servers along with getting required security and O'/S patches which can take hours to apply, (shutdown, patching, startup), can be just that servers that get "Dirty" over time especially with the use of code that leverages JVM's (Java) that can get corrupted over time and have frequent memory problems. spelled pain in the butt. Required weekly reboots are common to prevent alot of problems. With an MMO there are many many servers to reboot and double check that they come back up clean, as users are pretty unforgiving if after maintenance something is not fully working...

Network Typically the most stable part of an enterprise once setup correctly, however with more and more exploits targeting Internet facing Routers with new vulnerabilities that you guessed it have to be Security/firmware patched.

ACE/VPN concentrators: Commonly used to provide a secure tunnel of connected users from the Internet to datacenter backend systems. These must always get patched like everything else and care taken to deprecate old protocols that over time always have vulnerabilities pop up. TLS etc...

One network component that does have a higher failure rate (and time fixing them) are Load Balancers / Content Switches. These can take more maintenance as they are typically setup to take a network request targeting a single IP address and rout traffic round robin etc to multiple backend servers. The pain this causes during longer maintenance window due to Network team members not being thorough or overworked cannot be understated. I always cringed when I saw a change control for a CSS (content switch).

Database servers I'll state this simply as I've gone down the wabbit hole too far already considering this is a gaming forum and not "Unpaid consulting hour" Obviously all your data in several databases in an MMO have millions of transactions of reads and writes. Databases need maintenance in order to optimize paths to the disk to shorten read and write times. Database Administrators can run Stats, Reorgs, create new Indexes or refresh index to keep your data access optimal.
Simply put on large databases this can take HOURS. Nothing is free here and it's typically not optional unless you want your system response time to slow to a crawl, and in an MMO that would kill your application for all users...

You have the same level of complexity with other related systems:

Storage Farms: Servers are typically connected to their remote NAS or SAN hard drives (EMC etc) which require periodic maintenance as well.

Backup and Recovery Servers need to be backed up just in case of catastopophic failure which does occur. Many backups need to take place when users are not on the system. This can sometimes take hour(s) depending on how much local storage of the O/S+ Application and Data are on the server. I can see this occuring 1st during a maintenance window.

TL:DR - Running an MMO requires mandatory maintenance to keep everything running and "secure". No free lunch.

TBH, ZOS will not manage most of the items above, or do anything with them as part of their release/maintenance cycle, the majority will be supplied to them as solution building blocks in their data centre service. These items will all be highly resilient and not require outage for patching and fixing, and this will be baked into the contract with ZOS. At a bare minimum the uptime will be 99.9% as it's simply not possible to buy a lower uptime these days, in datacentre terms.

It is highly doubtful ZOS even supply their own servers, they will near definitely be supplied either as platform as a service, or infrastructure as a service. The only items ZOS will own are:

App itself
DB itself
Web code itself

The web and DB code will be largely static (in design terms) but the app itself - the entity that runs the game - will be subject to regular patching with game code. All of these items *could* be made resilient and also require no outage for patching, but it raises ZOS' costs to do so, in terms of licensing, kit etc. If the game is not that profitable, ZOS will not invest this money, or they simply may not invest it if they are just cheapskates. We can't know their rationale as they are not a quoted company.

Suffice to say, as it's just a game and no one dies when there is an outage, it's hard to put together a business case for redundancy across all tiers (app, web, DB).

Kryptonite_Kent · October 2016

Solid_Metal wrote: »

Kryptonite_Kent wrote: »

Solid_Metal wrote: »

Kryptonite_Kent wrote: »

Solid_Metal wrote: »

Kryptonite_Kent wrote: »

Solid_Metal wrote: »

Kryptonite_Kent wrote: »

old_mufasa wrote: »

Bigevilpeter wrote: »

Omg do them once a month, every time you do a maintenance you break the game and just to make a few fixes. Please for love of Akatosh stop the weekly maintenance.

Wow.. can you be more ignorant

Almost all major mmo's do server maintenance weekly.. and yes server can become unstable and shut done if you do not do the maintenance... even more stable mmos then ESO do it. Maybe you should educate yourself about why and the effects of not doing it before running you mouth off...

Yeah I have played reputable MMO's... you know where they actually tell you whats going on when something happens, they give you a timeline for maint downtime, and normal weekly maint is 2 - maybe 4 hours, not 8 lol.

Gina actually confirmed they know the problem, and take "couple" hours to fix it, but indeed i agree with the communication part
you know it take 8 right now because sh*t happen right?, normally it take 3-4 hours

Except uhh... pretty much every recent maintenance that I can recall has been at least double 3-4 hours and almost every one has been extended... yes every MMO does maintenance, but come on theres a bit of incompetence going on when its more common to have a problem with maintenance and it to last 8+ hours than for it to be a simple 3-4 hours and good to go.

nope, recent maintenance they never doubling anything AFAIK, top are 5 hours and thats it, this the only one so far that more than 5 hours, theres probably something like this yeah, but it couple months back

Lolwut... are you playing the same game? Just like last week or the week before there was maintenance more than once and extended maintenance on that!

more like emergerncy maintenance after Tamriel One update, it also happen on WoD update for WoW mind you, dunno if it also happen in Legion
we talking about extended weekly maintenance like this, which is normally only 3-5 hours top
and also as i recall, those "emergency" maintenance also take very short

So wait, you're saying yes there is "emergency" maintenance, but that's different than extended maintenance? At this point you sound like a paid spokesperson making up some cryptic excuses for obvious shortcomings... "oh this isn't 'extended' maintenance guys, its 'emergency' maintenance... and normal downtime is 3-5 hours 'usually', but sometimes we have slightly longer extended times due to emergencies and the sun was in my eyes and WoW has maintenance too and stuff and things"

you can perceive it as whatever you like really, for me the maintenance during big update is acceptable since *** can happen on those time, like almost all mmo with big update roll
and now weekly maintenance that usually only take 3-5 hours take a bit longer cause of *** happen, and this only happen once in a while
you probably can look up at google that other mmo also experience this kind of suff, how big your game, how detail are you on testing and preparing update, *** can happen

What big update... go look at the patch notes man lol... they fixed like a few minor things and the game has been down for going on 10 hours, 3-5 my ass!

MAOofDC · October 2016

toekster wrote: »

pls do it in the middle of night and not during day

It goes down at 04:00 EST which BTW was a good change from the way it's been for the last 2+ years. Today is unusual because the server went up then there was mass roll backs and crashes so down it went again. So they have been working on the server for the last 9 or so hours as of this posting 13:33 EST

ThePaleItalian · October 2016

Bigevilpeter wrote: »

cjthibs wrote: »

Bigevilpeter wrote: »

Well that was interesting, but it doesn't excuse the long downtimes and the downtimes because of the downtimes

"I reject your reality and substitute my own" --Bigevilpeter.

"I write weird stuff that don't make any sense" -- cjthibs

Doesn't.... sorry..it hurt my head.

Carry on!!!