Upcoming PC NA Datacenter Hardware Replacement

ZOS_MattFiror

Greetings everyone – I’m very happy to be able to give you a progress update on our datacenter hardware refresh. We’ve talked about this a few times in the past but to remind everyone what this is, here are a few things to keep in mind:

• In most cases, we are still running on 2012-era server hardware and it needs to be updated before components start failing more than they are. So, we are literally swapping out (“refreshing”) all hardware in our two datacenters.
• This will not result in any appreciable performance gain in-game, but it will result in a more reliable service overall that needs fewer unplanned maintenances.
• This was originally planned for last year, but the pandemic pushed back the timelines as it has become difficult to acquire new hardware because of global shortages.

It’s worth noting we installed and transitioned all the hardware for the PTS a few weeks ago, before the U34 PTS update, and we’re happy to report it has been working very smoothly. This was our major test case to ensure that everything would work as expected.

Our next step is to refresh the rest of the PC North American Datacenter, which will happen during the maintenance scheduled for May 3. Afterwards, everyone on PC NA should see a more reliable service experience with less unexpected maintenance downtimes.

We will be performing the PC EU and console datacenter hardware refresh as soon as we can, but as timelines for ordering/sourcing server hardware is still incredibly long, we don’t have a firm date for those yet. As soon as we have any additional information to pass along to all of you, we will do so.

As always, thank you so much for being a member of such a bright and vibrant fantasy virtual world.

WhiteCoatSyndrome

Thank you for the heads’ up! Much appreciated!

FeedbackOnly

O.O pts load screens were worse then live though...

ArchMikem

FeedbackOnly wrote: »

O.O pts load screens were worse then live though...

Pretty sure they mean connection/latency type of reliable.

FeedbackOnly

ArchMikem wrote: »

FeedbackOnly wrote: »

O.O pts load screens were worse then live though...

Pretty sure they mean connection/latency type of reliable.

Update 33 has given me PTSD with major changes

xaraan

fingers crossed

Sylvermynx

Thanks Matt. Always appreciate a heads-up!

Destai

Scary to hear you're running this game on 10 year-old hardware. Glad to hear it's being replaced, best of luck with that transition.

thejadefalcon

Can we ask why you're doing PC NA first, when PC EU historically has, by far, the most problems? Particularly with unplanned maintenance, which you say this will help to solve?

ApoAlaia

thejadefalcon wrote: »

Can we ask why you're doing PC NA first, when PC EU historically has, by far, the most problems? Particularly with unplanned maintenance, which you say this will help to solve?

It did cause me to raise an eyebrow too.

PC/EU is the most populous and the most troublesome of all the megaservers.

Oh well, at least we won't have maintenance on May Day so we have that going for us I guess.

FeedbackOnly

ApoAlaia wrote: »

thejadefalcon wrote: »

Can we ask why you're doing PC NA first, when PC EU historically has, by far, the most problems? Particularly with unplanned maintenance, which you say this will help to solve?

It did cause me to raise an eyebrow too.

PC/EU is the most populous and the most troublesome of all the megaservers.

Oh well, at least we won't have maintenance on May Day so we have that going for us I guess.

NA had some worse problems of late then EU.

S0Z0H

ZOS_MattFiror wrote: »

• This will not result in any appreciable performance gain in-game, but it will result in a more reliable service overall that needs fewer unplanned maintenances.

Thanks but what other sort of upgrade will be needed to better in game performance?
This is what everyone is looking forward to, an upgrade that grants strong in game performance.

ApoAlaia

FeedbackOnly wrote: »

ApoAlaia wrote: »

thejadefalcon wrote: »

Can we ask why you're doing PC NA first, when PC EU historically has, by far, the most problems? Particularly with unplanned maintenance, which you say this will help to solve?

It did cause me to raise an eyebrow too.

PC/EU is the most populous and the most troublesome of all the megaservers.

Oh well, at least we won't have maintenance on May Day so we have that going for us I guess.

NA had some worse problems of late then EU.

Not sure about that, both Jester's and Anniversary were pretty ropey in PC/EU.

Regardless, that is irrelevant to the matter at hand because:

timelines for ordering/sourcing server hardware is still incredibly long

Which means that this was a decision that was made a long time ago. Replacing the hardware in the NA datacentre first that is.

redlink1979

Thanks for the update!

SeaGtGruff

thejadefalcon wrote: »

Can we ask why you're doing PC NA first, when PC EU historically has, by far, the most problems? Particularly with unplanned maintenance, which you say this will help to solve?

I play on both servers, and PC EU has generally been more reliable for me lately than PC NA, but both need this upgrading for sure.

Kesstryl

Communication is, as always, greatly appreciated. Thank you for the update!

WhiteCoatSyndrome

ApoAlaia wrote: »

PC/EU is the most populous

This right here strikes me as a reason not to do EU first. Consider: if you had to risk a group of people, would you rather risk a group of forty or a group of sixty? Yes it’s bad if something happens either way but you want to minimize risk. If NA dies during the upgrade they have a smaller proportion of their player base unable to play, and either way they have more practice for when they get to EU.

Pyr0xyrecuprotite

thejadefalcon wrote: »

Can we ask why you're doing PC NA first, when PC EU historically has, by far, the most problems? Particularly with unplanned maintenance, which you say this will help to solve?

keep in mind that the EU players were originally hosted on the PC/NA servers at launch in 2014, and only got a separate server setup quite a few months later (after a lot of performance complaints). That is, the EU servers are of about 2 years more recent vintage than the NA server hardware. I'm sure that ZoS is aware of needing to refresh them, but it hasn't been quite as urgent as for the NA servers. It gets REALLY hard to source compatible spare parts on 10-year old hardware (or anything more than 5 years old I suppose).

Stefirex

FeedbackOnly wrote: »

ApoAlaia wrote: »

thejadefalcon wrote: »

Can we ask why you're doing PC NA first, when PC EU historically has, by far, the most problems? Particularly with unplanned maintenance, which you say this will help to solve?

It did cause me to raise an eyebrow too.

PC/EU is the most populous and the most troublesome of all the megaservers.

Oh well, at least we won't have maintenance on May Day so we have that going for us I guess.

NA had some worse problems of late then EU.

Such as?

FlopsyPrince

Who thought 10-year old hardware was good to rely on? Most businesses have a 3-5 year refresh cycle.

I will bet the system was not designed to swap in new hardware pieces without hurting production. We were doing that for a major airline's website in 2007 or earlier!

Are most MMO companies this behind the times?

FlopsyPrince

Pyr0xyrecuprotite wrote: »

thejadefalcon wrote: »

Can we ask why you're doing PC NA first, when PC EU historically has, by far, the most problems? Particularly with unplanned maintenance, which you say this will help to solve?

keep in mind that the EU players were originally hosted on the PC/NA servers at launch in 2014, and only got a separate server setup quite a few months later (after a lot of performance complaints). That is, the EU servers are of about 2 years more recent vintage than the NA server hardware. I'm sure that ZoS is aware of needing to refresh them, but it hasn't been quite as urgent as for the NA servers. It gets REALLY hard to source compatible spare parts on 10-year old hardware (or anything more than 5 years old I suppose).

It is possible they were doing lots of system-specific things in code, but normally updating hardware is not this hard.

They better consider that before they fall into failure because of needing that really old hardware!

FeedbackOnly

Stefirex wrote: »

FeedbackOnly wrote: »

ApoAlaia wrote: »

thejadefalcon wrote: »

Can we ask why you're doing PC NA first, when PC EU historically has, by far, the most problems? Particularly with unplanned maintenance, which you say this will help to solve?

It did cause me to raise an eyebrow too.

PC/EU is the most populous and the most troublesome of all the megaservers.

Oh well, at least we won't have maintenance on May Day so we have that going for us I guess.

NA had some worse problems of late then EU.

Such as?

You are joking? Update 33 for first 2 weeks was barely playable most of the time. Our group finder still isn't fixed fully either

WhiteCoatSyndrome

FeedbackOnly wrote: »

Stefirex wrote: »

FeedbackOnly wrote: »

ApoAlaia wrote: »

thejadefalcon wrote: »

Can we ask why you're doing PC NA first, when PC EU historically has, by far, the most problems? Particularly with unplanned maintenance, which you say this will help to solve?

It did cause me to raise an eyebrow too.

PC/EU is the most populous and the most troublesome of all the megaservers.

Oh well, at least we won't have maintenance on May Day so we have that going for us I guess.

NA had some worse problems of late then EU.

Such as?

You are joking? Update 33 for first 2 weeks was barely playable most of the time. Our group finder still isn't fixed fully either

This, and we’re still getting login queues that claim to be one second but last longer, and for me at least any big fights (EX: Trials) end up freezing up for several seconds during intensive bits.

meekmiko

S0Z0H wrote: »

ZOS_MattFiror wrote: »

• This will not result in any appreciable performance gain in-game, but it will result in a more reliable service overall that needs fewer unplanned maintenances.

Thanks but what other sort of upgrade will be needed to better in game performance?
This is what everyone is looking forward to, an upgrade that grants strong in game performance.

^ 100% This

Gaeliannas

I am very curious how you can upgrade from 2012 hardware to 2022 and not get a performance increase? Regardless of how bad your code it, running it on faster processors, with more cores, and faster memory and I would hope way faster storage since flash arrays are relatively cheap now, add up to a guaranteed performance increase. So how is ZOS managing to not allow that to happen? And better yet, why?

Sylvermynx

It's not how many of the EU players there are; it's that most of them play in a nearly singular EU TZ. Far less differentiated than the (at least) three disparate TZs on NA.

That said, I have two accounts both PC NA and EU. And I actually play on EU prime time. I don't have any trouble - but then again, I don't do group content or pvp....

sarahthes

thejadefalcon wrote: »

Can we ask why you're doing PC NA first, when PC EU historically has, by far, the most problems? Particularly with unplanned maintenance, which you say this will help to solve?

I believe EU supply chain for computer hardware is considerably worse than NA, so I'm suspecting the orders were placed at the same time, but NA's was received first.

ApoAlaia

sarahthes wrote: »

thejadefalcon wrote: »

Can we ask why you're doing PC NA first, when PC EU historically has, by far, the most problems? Particularly with unplanned maintenance, which you say this will help to solve?

I believe EU supply chain for computer hardware is considerably worse than NA, so I'm suspecting the orders were placed at the same time, but NA's was received first.

That is entirely feasible.

At least is the EU, getting hold of server hardware in post-Brexit Britain is even more of a kerfuffle.

Either way I hope the upgrade goes smoothly, ultimately someone had to go first; it would be very reassuring if it did.

LalMirchi

@ZOS_MattFiror

While this is good news, I'm still puzzled.

The Akamai overlay is still a connection/performance roadblock that seriously hampers connections and diminishes gameplay.

Why not go to a cloud-based system such as Azure or AWS? they have smoothly running servers that could alleviate your hardware problems?

FlopsyPrince

LalMirchi wrote: »

@ZOS_MattFiror

While this is good news, I'm still puzzled.

The Akamai overlay is still a connection/performance roadblock that seriously hampers connections and diminishes gameplay.

Why not go to a cloud-based system such as Azure or AWS? they have smoothly running servers that could alleviate your hardware problems?

They would probably have to do a significant redesign of the system for that. It is probably built to use Akamai if that is true and switching is not trivial. That said, I am surprised Akamai hasn't adapted to the times for what they provide....

HertoginJanneke

thejadefalcon wrote: »

Can we ask why you're doing PC NA first, when PC EU historically has, by far, the most problems? Particularly with unplanned maintenance, which you say this will help to solve?

I have no probem with using the trial-and-error tactic on PC NA, as long as the learn and do it first-time-right on the PC EU server.