Maintenance for the week of November 25:
• [COMPLETE] PC/Mac: NA and EU megaservers for maintenance – November 25, 4:00AM EST (9:00 UTC) - 7:00AM EST (12:00 UTC)
• Xbox: NA and EU megaservers for maintenance – November 27, 6:00AM EST (11:00 UTC) - 9:00AM EST (14:00 UTC)
• PlayStation®: NA and EU megaservers for maintenance – November 27, 6:00AM EST (11:00 UTC) - 9:00AM EST (14:00 UTC)
• [COMPLETE] PC/Mac: NA and EU megaservers for maintenance – November 25, 4:00AM EST (9:00 UTC) - 7:00AM EST (12:00 UTC)
• Xbox: NA and EU megaservers for maintenance – November 27, 6:00AM EST (11:00 UTC) - 9:00AM EST (14:00 UTC)
• PlayStation®: NA and EU megaservers for maintenance – November 27, 6:00AM EST (11:00 UTC) - 9:00AM EST (14:00 UTC)
Upcoming PC NA Datacenter Hardware Replacement
-
Epona222✭✭✭✭✭
✭✭✭✭Any chance of the daily login rewards being reworked so we don't miss out on todays?GM - Ghost Sea Trading Co - NA PC
Epona was a Romano-Celtic goddess dating back to around 1800 to 2000 years before computer games were invented.4 -
prettypony✭✭✭@ZOS_GinaBruno
You should give us a free week of traders spots and let us keep our spots from last week! For those questionable GMs, that did not get their bids in before the server went down
3 -
0 -
Shrinkwrap✭✭✭Grats on the new equipment... about time. Am I reading it right for server availability? 3:00PM PDT?0 -
Dr_Rektar✭✭✭The problem with ESO+ is somehow related to the extended maintenance?Engine guardian - best set ever0 -
kassandra86✭✭prettypony wrote: »@ZOS_GinaBruno
You should give us a free week of traders spots and let us keep our spots from last week! For those questionable GMs, that did not get their bids in before the server went down
This would be nice!!! I've been placing trader bids for years & have never had an issue! I was so tired last night, I figured I'd do it in the morning.
GM of The Ninja Squirrels - NA PC0 -
imno007b14_ESO✭✭✭Athanasia_Nocnitsa wrote: »
Exactly. There's probably some legalese buried in the agreement that says you are not guaranteed to be able to play all the days you pay for
0 -
Alarra✭✭✭Just putting in a word of appreciation for the ZOS folks working on this! I used to work in programming, and major updates like this can be rough and require long hours.UESP - Your source for The Elder Scrolls since 1995.
UESP forum/blog/Discord admin, wiki patroller, NA/PC Guild Advisor0 -
Doctor_Demento✭✭✭Cuddlypuff wrote: »pat_augustine wrote: »SimonThesis wrote: »The fact that they refuse to upgrade to servers that will have a noticeable impact on performance speaks volumes. Especially as they are bleeding players and guilds due to the never ending miserable performance in endgame.
I don't know what they are actually doing, but I have significant experience with Datacenter moves and upgrades. It's entirely possible that they aren't doing servers, but other infrastructure. Replacing all the switches and firewalls would also take a long time and wouldn't give any end-user noticeable performance gains.
Replacing servers isn't typically done in a single day event, but gradually over time. Routers and switches, on the other hand, pretty much have to be upgraded together.
Have they specified exactly what they are replacing?
It would be hard to replace 2012-era servers with modern hardware without some performance improvement... unless their backend code is mostly single-threaded. Modern computers simply throw more CPUs at the problem. The CPUs are faster due to better designs, but the biggest gain is in multi-threaded apps or running multiple apps so the more cores get used.
Database reads benefit from more cpus, but writes don't as the database is basically locked for any write. Building a database that can be written to by multiple cores at the same time without corruption remains a task we haven't solved yet.
If they are, in fact, replacing EVERY SERVER in the datacenter, that would be one hell of a project and I'm glad I don't have to do it. 8 hours shouldn't be sufficient for that for any data center of any significant size.
It's more likely that the new hardware is already imaged and configured and ready for deployment. Just needs to be connected and tested. As far as I've heard, everything is located in a little server room in the Zenimax HQ basement. It's not the same scale as an enterprise datacenter.
It's a 386SX Windows 95 box shoved under Matt's desk. He uses it as a footrest in the afternoons while drinking his coffee...
Actually, to the best of what I recall got figured out back in 2014, it is actually in a full fledged commercial data center somewhere in Dallas. Likely not on ZOS property. Leased space. My bet is on Level 3/CenturyLink, but there are plenty of possibilities.
Whether they will relocate it to a Microsoft data center as part of the "hardware replacement" is unknown, but also unlikely.
To handle the number of people who play this game on the PCNA servers you are correct as to its requirements for hardware. Servers, load balancers (ie: network load balancers, application load balancers, classic load balancers, gateway load balancers and even software load balancers) , fast switches, routers, interconnection switches, access switches, etc even the fiber optic links connection would have to be done. And lets not forget the Data Server Security Center that all large gaming centers use. I am shocked they were running it on equipment that old. That is ludicrous. The fact they are saying we will see small to little improvement is bad. Unless they are replacing it with cheap junk. Given the insane amount of money they make per month off millions and millions of players, wow. Can you say pure greed? But it would not shock me with the current mind set of most corps anymore. I hope things are actually are going to improve. It has needed it for some time now. And yes EU needs rebuilt from ground up as well. Oh and the banner now says the "maintenance" is to end at 4:00 PM EST. Keeping my fingers crossed performance improvement is better then they say. 0 -
DMuehlhausen✭✭✭✭✭
✭✭Scary to hear you're running this game on 10 year-old hardware. Glad to hear it's being replaced, best of luck with that transition.
That is actually about when most companies start switching out hardware. I work for a cloud provider and that's about the time they we were doing it. WoW had to do it at about the 3 year mark because the servers were so overloaded, but after that is like another 8 or 9 years before they did it again.
If ESO is still going they will do it again around 2030.0 -
prettypony✭✭✭kassandra86 wrote: »prettypony wrote: »@ZOS_GinaBruno
You should give us a free week of traders spots and let us keep our spots from last week! For those questionable GMs, that did not get their bids in before the server went down
This would be nice!!! I've been placing trader bids for years & have never had an issue! I was so tired last night, I figured I'd do it in the morning.
Hi Kass! I was actually referring to myself as questionable. I need to not do my bids at the last minute. I spend all week raising funds for the trader bids. I was until Tuesday mornings, so I have all the gold I will earn for the week. I have learned my lesson however to put my primary bids in early now. ZOS you have any day of the week to do this, why do it on a trader flip day? 2 -
ElfFromSpace✭✭✭✭kassandra86 wrote: »prettypony wrote: »@ZOS_GinaBruno
You should give us a free week of traders spots and let us keep our spots from last week! For those questionable GMs, that did not get their bids in before the server went down
This would be nice!!! I've been placing trader bids for years & have never had an issue! I was so tired last night, I figured I'd do it in the morning.
Yikes! This is very reminiscent of the long maintenance that made me up and quit as GM, saddling it on one of my poor officers.
At the time maintenance was Monday morning at 4 am and trader flip was Monday at 8 am. Bank balances were visible and you only had one bid which lead to rampant spying by checking bank balances and doing math. So all GMs got up to place bids right before 4, and we'd put in a bid and then slowly increase it in random amounts to make the spying harder, til the server went down. Then we set an alarm for 7:45 and basically watched til the server came up to run and hire. Ugh. The day I had enough the maintenance lasted until 5pm the next day and I felt like I had to rush if I even wanted to pee because I couldn't miss maintenance ending or I'd lose my chance to hire a backup!
What a mess this is!! Good luck to all GMs. Just like ZOS to do a major update like this without even thinking about the trader flip time!Edited by ElfFromSpace on May 3, 2022 6:57PMFormer GM Elder Scrolls Exchange3 -
ZOS_MattFiror wrote: »Greetings everyone – I’m very happy to be able to give you a progress update on our datacenter hardware refresh. We’ve talked about this a few times in the past but to remind everyone what this is, here are a few things to keep in mind:
- In most cases, we are still running on 2012-era server hardware and it needs to be updated before components start failing more than they are. So, we are literally swapping out (“refreshing”) all hardware in our two datacenters.
- This will not result in any appreciable performance gain in-game, but it will result in a more reliable service overall that needs fewer unplanned maintenances.
- This was originally planned for last year, but the pandemic pushed back the timelines as it has become difficult to acquire new hardware because of global shortages.
Our next step is to refresh the rest of the PC North American Datacenter, which will happen during the maintenance scheduled for May 3. Afterwards, everyone on PC NA should see a more reliable service experience with less unexpected maintenance downtimes.
We will be performing the PC EU and console datacenter hardware refresh as soon as we can, but as timelines for ordering/sourcing server hardware is still incredibly long, we don’t have a firm date for those yet. As soon as we have any additional information to pass along to all of you, we will do so.
As always, thank you so much for being a member of such a bright and vibrant fantasy virtual world.
Another great example of excellent communication, Thank you. A little disappointing that performance issues are going to take longer, but I appreciate the transparency and look forward to any future news regarding performance.0 -
Lady_Ems✭✭✭prettypony wrote: »kassandra86 wrote: »prettypony wrote: »@ZOS_GinaBruno
You should give us a free week of traders spots and let us keep our spots from last week! For those questionable GMs, that did not get their bids in before the server went down
This would be nice!!! I've been placing trader bids for years & have never had an issue! I was so tired last night, I figured I'd do it in the morning.
Hi Kass! I was actually referring to myself as questionable. I need to not do my bids at the last minute. I spend all week raising funds for the trader bids. I was until Tuesday mornings, so I have all the gold I will earn for the week. I have learned my lesson however to put my primary bids in early now. ZOS you have any day of the week to do this, why do it on a trader flip day?
Hello Ladies/Gents,
So glad I'm not the only one in this boat this week. This is why I think maintenance shouldn't happen on trader flip days. This one NEVER should have happened on a trader flip day. I generally get all my bids on Tuesday morning at the latest. Now I get to hope I can find one when the game finally does come back online.2 -
Merenwen_812✭✭✭I think got my daily log in last night before I went to bed...I sure do hope I did ....0
-
prettypony✭✭✭prettypony wrote: »kassandra86 wrote: »prettypony wrote: »@ZOS_GinaBruno
You should give us a free week of traders spots and let us keep our spots from last week! For those questionable GMs, that did not get their bids in before the server went down
This would be nice!!! I've been placing trader bids for years & have never had an issue! I was so tired last night, I figured I'd do it in the morning.
Hi Kass! I was actually referring to myself as questionable. I need to not do my bids at the last minute. I spend all week raising funds for the trader bids. I was until Tuesday mornings, so I have all the gold I will earn for the week. I have learned my lesson however to put my primary bids in early now. ZOS you have any day of the week to do this, why do it on a trader flip day?
Hello Ladies/Gents,
So glad I'm not the only one in this boat this week. This is why I think maintenance shouldn't happen on trader flip days. This one NEVER should have happened on a trader flip day. I generally get all my bids on Tuesday morning at the latest. Now I get to hope I can find one when the game finally does come back online.
Hi Lady Ems! I will be doing the same thing. We are all going to be racing when the server pops back up. I hope we do not crash it by all logging in. I also have to cancel raffles today. Oh, I get a day off of guild stuff after the rat race when we log in 0 -
Doctor_Demento✭✭✭The banner is now saying the end is 6:00 PM EST. They moved it forward 2 more hours.
0 -
ElfFromSpace✭✭✭✭prettypony wrote: »prettypony wrote: »kassandra86 wrote: »prettypony wrote: »@ZOS_GinaBruno
You should give us a free week of traders spots and let us keep our spots from last week! For those questionable GMs, that did not get their bids in before the server went down
This would be nice!!! I've been placing trader bids for years & have never had an issue! I was so tired last night, I figured I'd do it in the morning.
Hi Kass! I was actually referring to myself as questionable. I need to not do my bids at the last minute. I spend all week raising funds for the trader bids. I was until Tuesday mornings, so I have all the gold I will earn for the week. I have learned my lesson however to put my primary bids in early now. ZOS you have any day of the week to do this, why do it on a trader flip day?
Hello Ladies/Gents,
So glad I'm not the only one in this boat this week. This is why I think maintenance shouldn't happen on trader flip days. This one NEVER should have happened on a trader flip day. I generally get all my bids on Tuesday morning at the latest. Now I get to hope I can find one when the game finally does come back online.
Hi Lady Ems! I will be doing the same thing. We are all going to be racing when the server pops back up. I hope we do not crash it by all logging in. I also have to cancel raffles today. Oh, I get a day off of guild stuff after the rat race when we log in
As I just posted earlier, I can't agree more that maintenance and trader flips should never be the same time. It made me SO ANGRY when it was this was every week, I think that was 5 years ago. I certainly made this problem very clear to ZOS and did not let them ignore it til they finally moved the trader flip. You'd think they had learned. So sorry for all of the GMs who had no warning of this.Former GM Elder Scrolls Exchange3 -
ElfFromSpace✭✭✭✭Something we used to do when this was common was that a bunch of GMs would get together in chat to help each other find the open slots. The ones who were lucky and found they got their spot could help those less fortunate locate others to hire. Individual guilds might try to pull in any officers who have hiring permissions to help run.Former GM Elder Scrolls Exchange0
-
Gaeliannas✭✭✭✭✭Merenwen_812 wrote: »
But just imagine how great the game will perform now on this brand new equipment, totally worth the downtime... Oh wait, nevermind. 1 -
xXSilverDragonXx✭✭✭✭✭As much as I would like to be playing right now, I'm thrilled they are doing this updating and I truly hope we notice a very desirable improvement ASAP. Load screens had still been really long and watching my latency hit the red randomly or seeing sudden bouts of lag and freezing here and there was a bit disconcerting.Edited by xXSilverDragonXx on May 3, 2022 7:21PM0
-
Merenwen_812✭✭✭Gaeliannas wrote: »Merenwen_812 wrote: »
But just imagine how great the game will perform now on this brand new equipment, totally worth the downtime... Oh wait, nevermind.
IKR
*ponders whether to laugh or cry harder*0 -
Are you hopefully going to extend the collection of the daily gift? You make it almost impossible to collect it when you are down for more than half a day. I have a pretty liberal time that I can play, but if this isn't available to accept the days gift, it makes it impossible to get the big gifts at the end of the month. Please consider this.2 -
I am glad you are updating the hardware and such. So, what happens if we don't get the daily because of the downtime. The only reason I ask, I'm a 3rd shifter and i normally in the bed when the planned server up time suggestion.
Keep up the amazing work0
