Maintenance for the week of November 4:
• [COMPLETE] ESO Store and Account System for maintenance – November 6, 9:00AM EST (14:00 UTC) - 6:00PM EST (23:00 UTC)
• [COMPLETE] ESO Store and Account System for maintenance – November 6, 9:00AM EST (14:00 UTC) - 6:00PM EST (23:00 UTC)
Why no rolling updates?
DigiAngel
✭✭✭✭✭
From the thread:
Now that all scheduled maintenance has concluded on the PC, PTS and console megaservers, we’d like to outline next steps and what to expect in the coming days:
You don't see Azure going down for maintenance....or Oracle or Google. Why ESO?
Now that all scheduled maintenance has concluded on the PC, PTS and console megaservers, we’d like to outline next steps and what to expect in the coming days:
- Tomorrow, April 25 beginning at 6am EDT, we’ll be taking the PC NA megaserver offline for maintenance. During this time, we’ll be restoring affected PC NA accounts back to their state from Monday, April 15 at 9:30am EDT.
You don't see Azure going down for maintenance....or Oracle or Google. Why ESO?
0
-
TKo_ROUSE✭✭✭✭So maybe some of this has changed by now but ArenaNet has been doing this for years with Guild Wars 2. They use instances to do it. They bring up new updated instances and start letting characters move over as they pass zones. Eventually you will get an hour timer that your instance is dying and it will force you to move.
I think the issue is they patented that technology at the time for MMO games. I am not sure if the patent has expired yet but even if it did many of these games were developed at a time they could not do that so maybe they won't even move over to the system once the patent is up.Edited by TKo_ROUSE on April 25, 2024 1:11PM12 -
sarahthes✭✭✭✭✭
✭✭From the thread:
Now that all scheduled maintenance has concluded on the PC, PTS and console megaservers, we’d like to outline next steps and what to expect in the coming days:- Tomorrow, April 25 beginning at 6am EDT, we’ll be taking the PC NA megaserver offline for maintenance. During this time, we’ll be restoring affected PC NA accounts back to their state from Monday, April 15 at 9:30am EDT.
You don't see Azure going down for maintenance....or Oracle or Google. Why ESO?
I mean... they only have the one mega server. What is it supposed to be rolling with?5 -
Northwold✭✭✭✭✭
✭You're talking about very different kinds of software / service and, for example, enterprises using Oracle do often have to shut down their systems to update their software at the enterprise end unless they consider 24/7 uptime so critical that they run two sets of servers, in which case they update the redundant server and then do a rolling switch.
In the case of ESO, which uses a "single" server for all player activity, I'm not sure how that could be done in a way that is financially reasonable for what is, at the end of the day, a video game.Edited by Northwold on April 25, 2024 1:13PM2 -
TKo_ROUSE✭✭✭✭Just for frame of reference: The last Guild Wars 2 downtime was August 13 of 2016.
As far as how they can patent something like that I do not know. I am not an expert in copy right law.2 -
danno8✭✭✭✭✭
✭✭✭✭✭So maybe some of this has changed by now but ArenaNet has been doing this for years with Guild Wars 2. They use instances to do it. They bring up new updated instances and start letting characters move over as they pass zones. Eventually you will get an hour timer that your instance is dying and it will force you to move.
I think the issue is they patented that technology at the time for MMO games. I am not sure if the patent has expired yet but even if it did many of these games were developed at a time they could not do that so maybe they won't even move over to the system once the patent is up.
One of the lead developers of GW2 at the time said (I'm paraphrasing) that he didn't understand why other MMO's didn't use the same system as they do since it works so much better.
I don't believe there were any patents involved. I'm not even sure how you could patent something as broad as instanced worlds on a server or updating server code.5 -
barney2525✭✭✭✭✭
✭✭✭
yeah, but it's in the shop.
people in the future are evener meaner than what we have today.
4 -
Nerouyn✭✭✭✭✭
✭✭Just for frame of reference: The last Guild Wars 2 downtime was August 13 of 2016.
As far as how they can patent something like that I do not know. I am not an expert in copy right law.
According to this article, they used Amazon Web Services.
https://www.guildwars2.com/en/news/inside-arenanet-live-game-outage-analysis/
0 -
DigiAngel✭✭✭✭✭Also, considering that Microsoft owns Zenimax now....pretty sure they have the resources
1 -
xclassgaming✭✭✭✭✭
✭From the thread:
Now that all scheduled maintenance has concluded on the PC, PTS and console megaservers, we’d like to outline next steps and what to expect in the coming days:- Tomorrow, April 25 beginning at 6am EDT, we’ll be taking the PC NA megaserver offline for maintenance. During this time, we’ll be restoring affected PC NA accounts back to their state from Monday, April 15 at 9:30am EDT.
You don't see Azure going down for maintenance....or Oracle or Google. Why ESO?
those are....completely different things to a mmorpg? you cant compare them. Thats like comparing a apple to a carrot.Give us clannfear mounts!1 -
TKo_ROUSE✭✭✭✭Just for frame of reference: The last Guild Wars 2 downtime was August 13 of 2016.
As far as how they can patent something like that I do not know. I am not an expert in copy right law.
According to this article, they used Amazon Web Services.
https://www.guildwars2.com/en/news/inside-arenanet-live-game-outage-analysis/
Yes and several other MMO's use AWS and have down time. Remember AWS is just renting the server infrastructure of another company. The MMO's are an application that runs on top of some one else's infrastructure. So AWS has very little downtime typically but these companies still have to write their applications in a way that could allow updates to happen without taking the system offline.
ArenaNet accomplished this with Guild Wars 2. They probably spin up new servers with the update in AWS, transfer those clients between those servers and then take down the old servers. I probably am making that sound more simple than it actually is. I am sure it gets convoluted with things like the clients need to update, and how does a running instance of the application handle a version difference when they are handed off. I was under the impression that it was under a Patent but someone else pointed out I may be wrong. I really don't know for sure as I only know based on what others have told me. The only thing that might speak for patent is I would think other MMO's would have adopted these technologies, but maybe they are just cost prohibitive?
Also I did forget EU had a downtime that NA did not. There NA server has been up for over 8 years though.
Edited by TKo_ROUSE on April 25, 2024 2:23PM3 -
Northwold✭✭✭✭✭
✭Just for frame of reference: The last Guild Wars 2 downtime was August 13 of 2016.
As far as how they can patent something like that I do not know. I am not an expert in copy right law.
According to this article, they used Amazon Web Services.
https://www.guildwars2.com/en/news/inside-arenanet-live-game-outage-analysis/
Yes and several other MMO's use AWS and have down time. Remember AWS is just renting the server infrastructure of another company. The MMO's are an application that runs on top of some one else's infrastructure. So AWS has very little downtime typically but these companies still have to write their applications in a way that could allow updates to happen without taking the system offline.
ArenaNet accomplished this with Guild Wars 2. They probably spin up new servers with the update in AWS, transfer those clients between those servers and then take down the old servers. I probably am making that sound more simple than it actually is. I am sure it gets convoluted with things like the clients need to update, and how does a running instance of the application handle a version difference when they are handed off. I was under the impression that it was under a Patent but someone else pointed out I may be wrong. I really don't know for sure as I only know based on what others have told me. The only thing that might speak for patent is I would think other MMO's would have adopted these technologies, but maybe they are just cost prohibitive?
Also I did forget EU had a downtime that NA did not. There NA server has been up for over 8 years though.
If server infrastructure and the software itself allow for instances of a game to be run (and instances of parts of the game), it's certainly conceivable to move people off an instance while the server infrastructure is updated for that instance, and then on to the next and the next to get the process done. In essence, you can give a little bit of the load to another set of servers while you do the update elsewhere without requiring those servers to handle the full load of the whole game.
Probably less feasible with ESO given how the game was set up (at least, from what has been described of the architecture).Edited by Northwold on April 25, 2024 2:49PM0 -
EramTheLiar✭✭✭✭ESO came out in 2014 and AWS didn't start offering instanced servers until 2015. At that point ESO had a dedicated server environment that I think was standalone (i.e., not AWS). I'm impressed GW2 did what it did, _when_ it did it, but it was legitimately ballsy of them to do so - instancing was only in place for a few years and at least at first it wasn't the kind of thing you really did with heavily database-dependent environments because you had to figure out how to make sure database writes weren't being lost between instances. Obviously it's less of an issue these days but it does add complexity to databases and is more expensive to run (especially if you're on AWS - AWS does not provide flat rates. All AWS services are metered and every company I've ever worked for who moved from their own data center to an AWS data center has wound up paying more. In some cases that extra spend was offset by the greater reliability, because instancing can be genuinely useful when you use it right. In other cases... it was just more expensive.
Taking servers down to perform maintenance is a thing we're all used to and we grumble about it and for the most part it finishes up and we all start playing again. Occasionally you get interesting scenarios like where someone accidentally connects the wrong database to the wrong server and things get intensely exciting for a while, but I'd argue that if that specific scenario had occurred in an instanced environment it might have been significantly harder to fix depending on when the problem was detected.
But I don't actually know. This is all speculation on my part.1 -
Danikat✭✭✭✭✭
✭✭✭✭✭For info ArenaNet have actually been doing this since 2005 when Guild Wars 1 came out.
It might have been simpler with that game because everything outside of towns was instanced, but the way people are describing server updates it sounds like that doesn't make much difference. In terms of updates it was the same, even when an expansion launched you could be online the whole time and you'd just get a notification that an update was available and then had an hour or so to log off, download the update and log back in immediately.
The servers for both games have very rarely been offline and as far as I remember only when something seriously broke, or once because they were physically moving the data center.
I know for a lot of people MMOs going offline regularly for maintenance and updates is just what's expected, but for me it came as a big surprise when I started ESO and found out a modern MMO still had to do that. Between GW1 and GW2 being my only MMOs for a few years and the fact that no other online systems or services seem to have maintenance shut down times any more I just assumed it was something that wasn't needed if you were using modern technology.
Like how you used to need to defragment a PC every few months, and it would take hours (often over night), but these days it rarely needs doing at all and is much, much quicker when it is necessary.PC EU player | She/her/hers | PAWS (Positively Against Wrip-off Stuff) - Say No to Crown Crates!
"Remember in this game we call life that no one said it's fair"1 -
Sakiri✭✭✭✭✭
✭✭I think you guys forgot the critical point here.
This game isn't some kind of critical service. It can be down. it's okay. Go outside, cook a meal. Learn a new hobby. Wash the car, bathe the cat, then go to the emergency room.10 -
DigiAngel✭✭✭✭✭I think you guys forgot the critical point here.
This game isn't some kind of critical service. It can be down. it's okay. Go outside, cook a meal. Learn a new hobby. Wash the car, bathe the cat, then go to the emergency room.
Completely agree. But the TIMING.....events...really though?0 -
Sakiri✭✭✭✭✭
✭✭Timing sucks, sure, but still. Nothing ever goes the way you want it to. Learned that 20 years ago at least.3 -
Whose Internet infrastructure never ever goes down, as you can tell by how my organization (that used Microsoft Azure) has outages you could literally set your watch by.
A big difference is centralization. "Cloud" architecture is usually down for someone but never down for everyone. Centralized servers are more reliably available to everyone but have downtimes for everyone as well.Edited by ragnarok6644b14_ESO on April 25, 2024 5:13PM0 -
Funhammy✭✭✭It’s more because you references a cloud system with thousands of servers of which any one could go down and the cloud would at most have some performance issues but not go fully down.
As for the mega server spaghetti code they use for ESO; it’s just that, one server per server… no extra servers.
Which means when the server requires maintenance the entire server must go down to do it.
They don’t have a backend set up in order to allow hot fix maintenance because that requires modulating server load between multiple severs which they don’t have since they use a mega server.
Plus hotfixes are only so capable of fixing things, take RuneScape which hotfixes all the time, when they’ve had to do rollbacks they also have to turn all of their servers offline to do it.
You can’t modulate around the server load of major changes, one server would say that your account is in one state and another would say it’s in a different state and when they try to merge your save data who knows which it would pick (what happened with the PTS) so they have to turn servers off for anything beyond a small patch.
And as far as I’ve ever seen ESO doesn’t even have the capability to do a hotfix anyways.Edited by Funhammy on April 25, 2024 5:17PM0 -
Jimbru✭✭✭✭Because ZOS still thinks of ESO in terms of "Multiplayer Skyrim on Outdated Consoles", not a modern PC MMO, and its back end is structured accordingly.0
-
sarahthes✭✭✭✭✭
✭✭It’s more because you references a cloud system with thousands of servers of which any one could go down and the cloud would at most have some performance issues but not go fully down.
As for the mega server spaghetti code they use for ESO; it’s just that, one server per server… no extra servers.
Which means when the server requires maintenance the entire server must go down to do it.
They don’t have a backend set up in order to allow hot fix maintenance because that requires modulating server load between multiple severs which they don’t have since they use a mega server.
Plus hotfixes are only so capable of fixing things, take RuneScape which hotfixes all the time, when they’ve had to do rollbacks they also have to turn all of their servers offline to do it.
You can’t modulate around the server load of major changes, one server would say that your account is in one state and another would say it’s in a different state and when they try to merge your save data who knows which it would pick (what happened with the PTS) so they have to turn servers off for anything beyond a small patch.
And as far as I’ve ever seen ESO doesn’t even have the capability to do a hotfix anyways.
They can, but I think the scope of what they can do is incredibly limited.0
