Maintenance for the week of March 25:
• [COMPLETE] ESO Store and Account System for maintenance – March 28, 9:00AM EDT (13:00 UTC) - 12:00PM EDT (16:00 UTC)

Upcoming PC NA Datacenter Hardware Replacement

ZOS_MattFiror
Greetings everyone – I’m very happy to be able to give you a progress update on our datacenter hardware refresh. We’ve talked about this a few times in the past but to remind everyone what this is, here are a few things to keep in mind:
  • In most cases, we are still running on 2012-era server hardware and it needs to be updated before components start failing more than they are. So, we are literally swapping out (“refreshing”) all hardware in our two datacenters.
  • This will not result in any appreciable performance gain in-game, but it will result in a more reliable service overall that needs fewer unplanned maintenances.
  • This was originally planned for last year, but the pandemic pushed back the timelines as it has become difficult to acquire new hardware because of global shortages.
It’s worth noting we installed and transitioned all the hardware for the PTS a few weeks ago, before the U34 PTS update, and we’re happy to report it has been working very smoothly. This was our major test case to ensure that everything would work as expected.

Our next step is to refresh the rest of the PC North American Datacenter, which will happen during the maintenance scheduled for May 3. Afterwards, everyone on PC NA should see a more reliable service experience with less unexpected maintenance downtimes.

We will be performing the PC EU and console datacenter hardware refresh as soon as we can, but as timelines for ordering/sourcing server hardware is still incredibly long, we don’t have a firm date for those yet. As soon as we have any additional information to pass along to all of you, we will do so.

As always, thank you so much for being a member of such a bright and vibrant fantasy virtual world.
Matt Firor
Studio Director, ZeniMax Online Studios
Staff Post
  • WhiteCoatSyndrome
    WhiteCoatSyndrome
    ✭✭✭✭✭
    ✭✭✭✭
    Thank you for the heads’ up! :smiley: Much appreciated!
    #proud2BAStarObsessedLoony
    PAWS (Positively Against Wrip-off Stuff) - Say No to Crown Crates!
    A useful explanation for how RNG works
  • FeedbackOnly
    FeedbackOnly
    ✭✭✭✭✭
    ✭✭
    O.O pts load screens were worse then live though...
  • ArchMikem
    ArchMikem
    ✭✭✭✭✭
    ✭✭✭✭✭
    O.O pts load screens were worse then live though...

    Pretty sure they mean connection/latency type of reliable.
    CP1,900+ Master Explorer - AvA One Star General - Console Peasant - The Clan
    Quest Objective: OMG Go Talk To That Kitty!
  • FeedbackOnly
    FeedbackOnly
    ✭✭✭✭✭
    ✭✭
    ArchMikem wrote: »
    O.O pts load screens were worse then live though...

    Pretty sure they mean connection/latency type of reliable.

    Update 33 has given me PTSD with major changes
  • xaraan
    xaraan
    ✭✭✭✭✭
    ✭✭✭✭✭
    fingers crossed
    -- @xaraan --
    nightblade: Xaraan templar: Xaraan-dar dragon-knight: Xaraanosaurus necromancer: Xaraan-qa warden: Xaraanodon sorcerer: Xaraan-ra
    AD • NA • PC
  • Sylvermynx
    Sylvermynx
    ✭✭✭✭✭
    ✭✭✭✭✭
    Thanks Matt. Always appreciate a heads-up!
  • Destai
    Destai
    ✭✭✭✭✭
    ✭✭
    Scary to hear you're running this game on 10 year-old hardware. Glad to hear it's being replaced, best of luck with that transition.
  • thejadefalcon
    thejadefalcon
    ✭✭✭✭✭
    Can we ask why you're doing PC NA first, when PC EU historically has, by far, the most problems? Particularly with unplanned maintenance, which you say this will help to solve?
  • ApoAlaia
    ApoAlaia
    ✭✭✭✭✭
    ✭✭
    Can we ask why you're doing PC NA first, when PC EU historically has, by far, the most problems? Particularly with unplanned maintenance, which you say this will help to solve?

    It did cause me to raise an eyebrow too.

    PC/EU is the most populous and the most troublesome of all the megaservers.

    Oh well, at least we won't have maintenance on May Day so we have that going for us I guess.
  • FeedbackOnly
    FeedbackOnly
    ✭✭✭✭✭
    ✭✭
    ApoAlaia wrote: »
    Can we ask why you're doing PC NA first, when PC EU historically has, by far, the most problems? Particularly with unplanned maintenance, which you say this will help to solve?

    It did cause me to raise an eyebrow too.

    PC/EU is the most populous and the most troublesome of all the megaservers.

    Oh well, at least we won't have maintenance on May Day so we have that going for us I guess.

    NA had some worse problems of late then EU.
  • S0Z0H
    S0Z0H
    ✭✭✭
    • This will not result in any appreciable performance gain in-game, but it will result in a more reliable service overall that needs fewer unplanned maintenances.

    Thanks but what other sort of upgrade will be needed to better in game performance?
    This is what everyone is looking forward to, an upgrade that grants strong in game performance.
  • ApoAlaia
    ApoAlaia
    ✭✭✭✭✭
    ✭✭
    ApoAlaia wrote: »
    Can we ask why you're doing PC NA first, when PC EU historically has, by far, the most problems? Particularly with unplanned maintenance, which you say this will help to solve?

    It did cause me to raise an eyebrow too.

    PC/EU is the most populous and the most troublesome of all the megaservers.

    Oh well, at least we won't have maintenance on May Day so we have that going for us I guess.

    NA had some worse problems of late then EU.

    Not sure about that, both Jester's and Anniversary were pretty ropey in PC/EU.

    Regardless, that is irrelevant to the matter at hand because:

    timelines for ordering/sourcing server hardware is still incredibly long

    Which means that this was a decision that was made a long time ago. Replacing the hardware in the NA datacentre first that is.
  • redlink1979
    redlink1979
    ✭✭✭✭✭
    ✭✭✭
    Thanks for the update!
    "Sweet Mother, sweet Mother, send your child unto me, for the sins of the unworthy must be baptized in blood and fear"
    • Sons of the Night Mother [PS5][EU] 2165 CP
    • Daggerfall's Mightiest [PS5][NA] 1910 CP
    • SweetTrolls [PC][EU] 1950 CP
    • Bacon Rats [PC][NA] 1850 CP
  • SeaGtGruff
    SeaGtGruff
    ✭✭✭✭✭
    ✭✭✭✭✭
    Can we ask why you're doing PC NA first, when PC EU historically has, by far, the most problems? Particularly with unplanned maintenance, which you say this will help to solve?

    I play on both servers, and PC EU has generally been more reliable for me lately than PC NA, but both need this upgrading for sure.
    I've fought mudcrabs more fearsome than me!
  • Kesstryl
    Kesstryl
    ✭✭✭✭✭
    ✭✭
    Communication is, as always, greatly appreciated. Thank you for the update!
    HEARTHLIGHT - A guild for housing enthusiasts! Contact @Kesstryl in-game to join.
  • WhiteCoatSyndrome
    WhiteCoatSyndrome
    ✭✭✭✭✭
    ✭✭✭✭
    ApoAlaia wrote: »
    PC/EU is the most populous

    This right here strikes me as a reason not to do EU first. Consider: if you had to risk a group of people, would you rather risk a group of forty or a group of sixty? Yes it’s bad if something happens either way but you want to minimize risk. If NA dies during the upgrade they have a smaller proportion of their player base unable to play, and either way they have more practice for when they get to EU.
    #proud2BAStarObsessedLoony
    PAWS (Positively Against Wrip-off Stuff) - Say No to Crown Crates!
    A useful explanation for how RNG works
  • Pyr0xyrecuprotite
    Pyr0xyrecuprotite
    ✭✭✭✭✭
    Can we ask why you're doing PC NA first, when PC EU historically has, by far, the most problems? Particularly with unplanned maintenance, which you say this will help to solve?

    keep in mind that the EU players were originally hosted on the PC/NA servers at launch in 2014, and only got a separate server setup quite a few months later (after a lot of performance complaints). That is, the EU servers are of about 2 years more recent vintage than the NA server hardware. I'm sure that ZoS is aware of needing to refresh them, but it hasn't been quite as urgent as for the NA servers. It gets REALLY hard to source compatible spare parts on 10-year old hardware (or anything more than 5 years old I suppose).
  • Stefirex
    Stefirex
    ✭✭✭
    ApoAlaia wrote: »
    Can we ask why you're doing PC NA first, when PC EU historically has, by far, the most problems? Particularly with unplanned maintenance, which you say this will help to solve?

    It did cause me to raise an eyebrow too.

    PC/EU is the most populous and the most troublesome of all the megaservers.

    Oh well, at least we won't have maintenance on May Day so we have that going for us I guess.

    NA had some worse problems of late then EU.

    Such as?
  • FlopsyPrince
    FlopsyPrince
    ✭✭✭✭✭
    ✭✭
    Who thought 10-year old hardware was good to rely on? Most businesses have a 3-5 year refresh cycle.

    I will bet the system was not designed to swap in new hardware pieces without hurting production. We were doing that for a major airline's website in 2007 or earlier!

    Are most MMO companies this behind the times?
    PC
    PS4/PS5
  • FlopsyPrince
    FlopsyPrince
    ✭✭✭✭✭
    ✭✭
    Can we ask why you're doing PC NA first, when PC EU historically has, by far, the most problems? Particularly with unplanned maintenance, which you say this will help to solve?

    keep in mind that the EU players were originally hosted on the PC/NA servers at launch in 2014, and only got a separate server setup quite a few months later (after a lot of performance complaints). That is, the EU servers are of about 2 years more recent vintage than the NA server hardware. I'm sure that ZoS is aware of needing to refresh them, but it hasn't been quite as urgent as for the NA servers. It gets REALLY hard to source compatible spare parts on 10-year old hardware (or anything more than 5 years old I suppose).

    It is possible they were doing lots of system-specific things in code, but normally updating hardware is not this hard.

    They better consider that before they fall into failure because of needing that really old hardware!
    PC
    PS4/PS5
  • FeedbackOnly
    FeedbackOnly
    ✭✭✭✭✭
    ✭✭
    Stefirex wrote: »
    ApoAlaia wrote: »
    Can we ask why you're doing PC NA first, when PC EU historically has, by far, the most problems? Particularly with unplanned maintenance, which you say this will help to solve?

    It did cause me to raise an eyebrow too.

    PC/EU is the most populous and the most troublesome of all the megaservers.

    Oh well, at least we won't have maintenance on May Day so we have that going for us I guess.

    NA had some worse problems of late then EU.

    Such as?

    You are joking? Update 33 for first 2 weeks was barely playable most of the time. Our group finder still isn't fixed fully either
  • WhiteCoatSyndrome
    WhiteCoatSyndrome
    ✭✭✭✭✭
    ✭✭✭✭
    Stefirex wrote: »
    ApoAlaia wrote: »
    Can we ask why you're doing PC NA first, when PC EU historically has, by far, the most problems? Particularly with unplanned maintenance, which you say this will help to solve?

    It did cause me to raise an eyebrow too.

    PC/EU is the most populous and the most troublesome of all the megaservers.

    Oh well, at least we won't have maintenance on May Day so we have that going for us I guess.

    NA had some worse problems of late then EU.

    Such as?

    You are joking? Update 33 for first 2 weeks was barely playable most of the time. Our group finder still isn't fixed fully either

    This, and we’re still getting login queues that claim to be one second but last longer, and for me at least any big fights (EX: Trials) end up freezing up for several seconds during intensive bits.
    #proud2BAStarObsessedLoony
    PAWS (Positively Against Wrip-off Stuff) - Say No to Crown Crates!
    A useful explanation for how RNG works
  • meekmiko
    meekmiko
    ✭✭✭✭✭
    S0Z0H wrote: »
    • This will not result in any appreciable performance gain in-game, but it will result in a more reliable service overall that needs fewer unplanned maintenances.

    Thanks but what other sort of upgrade will be needed to better in game performance?
    This is what everyone is looking forward to, an upgrade that grants strong in game performance.

    ^ 100% This
    🌟PC/NA CP2000+ [Been playing since 2016]
    vMA / vVH / vDSA / vBRP / vAA HM / vSO HM / vHRC HM / vMoL HM / vHoF HM / vAS+1& +1 / vCR+2 / vSS / vRG
    • 🌩️ ⚔️ EP - Jessamine Seed-Nightrun, Dro-m'Athra Destroyer / Bosmer StamSorc (DPS) -Main DPS-
    • ☀️ 🛡️ EP - Mihi'Mai-Ra, Plague of Peryite / Khajiit StamPlar (Tank) -Main Tank-
    • ☀️ ✨ EP - Lady Lapa'au, The Magnanimous / Khajiit MagPlar (Healer) -Main Healer-
    • 🌩️ ☄️ EP - Ra'venk, Style Master / Khajiit MagSorc (DPS)
    • 🗡️ ⚔️ EP - Bird gra-Shuzgub, Forge Breaker / Orc StamBlade (DPS)
    • 🗡️ ✨ EP - Blades-at-the-Ready, Witch / Argonian MagBlade (Healer)
    • ☀️ ⚔️ EP - Lady Fortuna the Blessed, Grand Champion / Imperial StamPlar (DPS)
    • 🔥 🛡️ EP - Plays-in Volcanoes, Mystic / Argonian StamDK (Tank)
    • 💀 ☄️ EP - Ko'shamari the Doomweaver, Dovahkriid / Khajiit MagNecro (DPS)
    • 🌱 ✨ EP - Sorvete, Countess / Khajiit StamDen (Healer)
    • 🔥 🏹 EP - Falora Veloth, The Merciless / Dunmer StamDK (DPS)
    • 🌩️ 🛡️ AD - Shimmers-with-Static, Spark of Vengeance / Argonian StamSorc (Tank)
    • ☀️ ⚔️ EP - Amarri-do the Magnificent, Clan Mother / Khajiit StamPlar (DPS)
    • 🌩️ ⚔️ EP - Z'majii-dar the Quick, Battlegrounds Butcher / Khajiit StamSorc (DPS)
    • 🔥 ⚔️ EP - Habasi the Glamorous, Seeker of Artifacts / Khajiit StamDK (DPS)
    • ❄️ 🛡️ EP - Elísabet Skull-Smasher, Thane of Falkreath / Nord StamDen (Tank)
    • 🗡️ 🛡️ EP - Yana-la the Iron Lotus, Silver Knight / Khajiit StamBlade (Tank)
    • 👁️ ☄️ EP - Mog gra-Ushug the Scholar, Master Historian / Orc MagCanist (DPS)
    • 👁️ 🔨 EP - Mamaea the Sledgehammer, Alpha Predator / Khajiit StamCanist (DPS)
    • 👁️ 🏹 AD - Elsyiir Lichenhollow, Lady / Bosmer StamCanist (DPS)
    • & I only dabble on the PC/EU server sometimes
    • 🗡️ ⚔️ AD - Jessamine Seed-Nightrun / Bosmer StamBlade (DPS)
    • 🌩️ ⚔️ EP - Sings-a-Song-of-Storms / Argonian StamSorc (DPS)
    • 🗡️ ☄️ DC - Helainie the Shadebringer / Breton MagBlade (DPS)
    • 🌩️ ☄️ EP - Steals-Many-Hearts / Argonian MagSorc (DPS)
    • 🗡️ ⚔️ AD - Pashet the Nimble / Khajiit StamBlade (DPS)
    • 🔥 🏹☄️ EP - Furoni the Ember / Dunmer HybridDK (DPS)
    • 👁️ ☄️ EP - Am-Kesh / Argonian MagCanist (DPS)
    • 💀 ⚔️ AD - Dro'Vashpar the Corrupted / Khajiit StamNecro (DPS)
    • 🔥 ⚔️ AD - Eldrinthr the Flame-Heart / Bosmer StamDK (DPS)
    • 👁️ ☄️ AD - Bajaa the Blackened Beast / Khajiit MagNecro (DPS)
  • Gaeliannas
    Gaeliannas
    ✭✭✭✭✭
    I am very curious how you can upgrade from 2012 hardware to 2022 and not get a performance increase? Regardless of how bad your code it, running it on faster processors, with more cores, and faster memory and I would hope way faster storage since flash arrays are relatively cheap now, add up to a guaranteed performance increase. So how is ZOS managing to not allow that to happen? And better yet, why?
  • Sylvermynx
    Sylvermynx
    ✭✭✭✭✭
    ✭✭✭✭✭
    It's not how many of the EU players there are; it's that most of them play in a nearly singular EU TZ. Far less differentiated than the (at least) three disparate TZs on NA.

    That said, I have two accounts both PC NA and EU. And I actually play on EU prime time. I don't have any trouble - but then again, I don't do group content or pvp....
    Edited by Sylvermynx on April 29, 2022 2:32AM
  • sarahthes
    sarahthes
    ✭✭✭✭✭
    Can we ask why you're doing PC NA first, when PC EU historically has, by far, the most problems? Particularly with unplanned maintenance, which you say this will help to solve?

    I believe EU supply chain for computer hardware is considerably worse than NA, so I'm suspecting the orders were placed at the same time, but NA's was received first.
  • ApoAlaia
    ApoAlaia
    ✭✭✭✭✭
    ✭✭
    sarahthes wrote: »
    Can we ask why you're doing PC NA first, when PC EU historically has, by far, the most problems? Particularly with unplanned maintenance, which you say this will help to solve?

    I believe EU supply chain for computer hardware is considerably worse than NA, so I'm suspecting the orders were placed at the same time, but NA's was received first.

    That is entirely feasible.

    At least is the EU, getting hold of server hardware in post-Brexit Britain is even more of a kerfuffle.

    Either way I hope the upgrade goes smoothly, ultimately someone had to go first; it would be very reassuring if it did.
  • LalMirchi
    LalMirchi
    ✭✭✭✭✭
    @ZOS_MattFiror

    While this is good news, I'm still puzzled.

    The Akamai overlay is still a connection/performance roadblock that seriously hampers connections and diminishes gameplay.

    Why not go to a cloud-based system such as Azure or AWS? they have smoothly running servers that could alleviate your hardware problems?
  • FlopsyPrince
    FlopsyPrince
    ✭✭✭✭✭
    ✭✭
    LalMirchi wrote: »
    @ZOS_MattFiror

    While this is good news, I'm still puzzled.

    The Akamai overlay is still a connection/performance roadblock that seriously hampers connections and diminishes gameplay.

    Why not go to a cloud-based system such as Azure or AWS? they have smoothly running servers that could alleviate your hardware problems?

    They would probably have to do a significant redesign of the system for that. It is probably built to use Akamai if that is true and switching is not trivial. That said, I am surprised Akamai hasn't adapted to the times for what they provide....
    PC
    PS4/PS5
  • HertoginJanneke
    HertoginJanneke
    ✭✭✭✭
    Can we ask why you're doing PC NA first, when PC EU historically has, by far, the most problems? Particularly with unplanned maintenance, which you say this will help to solve?

    I have no probem with using the trial-and-error tactic on PC NA, as long as the learn and do it first-time-right on the PC EU server.
Sign In or Register to comment.