Maintenance for the week of May 27:
• PC/Mac: No maintenance – May 27

Upcoming PC NA Datacenter Hardware Replacement

  • tomfant
    tomfant
    ✭✭✭✭
    Gaeliannas wrote: »
    I am very curious how you can upgrade from 2012 hardware to 2022 and not get a performance increase? Regardless of how bad your code it, running it on faster processors, with more cores, and faster memory and I would hope way faster storage since flash arrays are relatively cheap now, add up to a guaranteed performance increase. So how is ZOS managing to not allow that to happen? And better yet, why?

    Yep, very strange. Code must be really bad, or the new hardware is just a few i3...
    Options
  • Sylvermynx
    Sylvermynx
    ✭✭✭✭✭
    ✭✭✭✭✭
    LalMirchi wrote: »
    @ZOS_MattFiror

    While this is good news, I'm still puzzled.

    The Akamai overlay is still a connection/performance roadblock that seriously hampers connections and diminishes gameplay.

    Why not go to a cloud-based system such as Azure or AWS? they have smoothly running servers that could alleviate your hardware problems?

    As someone whose reseller provider switched to AWS last year from his own server farm, I can say with perfect truth that it's a damn good thing he gave me a 50% annual discount when he moved, because while my uptime on his former setup was 99.99%, my uptime on AWS is down to around 89%.

    Now that might not seem to be a big issue, but my usage of AWS is small compared to something as big as ZOS - and when I've had issues on AWS, so have really major corporations, including Amazon itself....

    I don't know anything about Azure, except that it's MS's cloud.
    Options
  • Gaeliannas
    Gaeliannas
    ✭✭✭✭✭
    Sylvermynx wrote: »
    LalMirchi wrote: »
    @ZOS_MattFiror

    While this is good news, I'm still puzzled.

    The Akamai overlay is still a connection/performance roadblock that seriously hampers connections and diminishes gameplay.

    Why not go to a cloud-based system such as Azure or AWS? they have smoothly running servers that could alleviate your hardware problems?

    As someone whose reseller provider switched to AWS last year from his own server farm, I can say with perfect truth that it's a damn good thing he gave me a 50% annual discount when he moved, because while my uptime on his former setup was 99.99%, my uptime on AWS is down to around 89%.

    Now that might not seem to be a big issue, but my usage of AWS is small compared to something as big as ZOS - and when I've had issues on AWS, so have really major corporations, including Amazon itself....

    I don't know anything about Azure, except that it's MS's cloud.

    That would be because your reseller cheaped out. You don't toss all your bananas in one basket (one AWS/Azure datacenter), you configure failover between AWS East/West or Pacific Northwest(which is cheaper) and you will have 5 nines uptime. Better yet, you failover between AWS/Azure. Last year(earlier this year?) AWS East crapped out, it brought down a ton of large companies, who were all under the impression being in a single datacenter would be fine, they learned better. At least 5 services we used went down with AWS East, every one of them promised to provide redundancy in the future in their apology letters.

    It is basically the same reason you have multiple Internet providers for your connectivity, and those providers have multiple routes configured, in case one goes down things still work.
    Options
  • gariondavey
    gariondavey
    ✭✭✭✭✭
    ✭✭
    My data is going to be SO refreshed after this.
    Hooray!
    PC NA @gariondavey, BG, IC & Cyrodiil Focused Since October 2017 Stamplar (main), Magplar, Magsorc, Stamsorc, StamDK, MagDK, Stamblade, Magblade, Magden, Stamden
    Options
  • Gaeliannas
    Gaeliannas
    ✭✭✭✭✭
    [snip]

    Actually he said 2012-Era, some of it was probably bought as far back as 2010.

    [edited to remove quote]
    Edited by ZOS_Icy on April 29, 2022 5:18PM
    Options
  • wenchmore420b14_ESO
    wenchmore420b14_ESO
    ✭✭✭✭✭
    ✭✭✭
    Gaeliannas wrote: »
    [snip]

    Actually he said 2012-Era, some of it was probably bought as far back as 2010.

    Or older, as ESO started development in 2007.
    :)

    [edited to remove quote]
    Edited by ZOS_Icy on April 29, 2022 5:19PM
    Drakon Koryn~Oryndill, Rogue~Mage,- CP ~Doesn't matter any more
    NA / PC Beta Member since Nov 2013
    GM~Conclave-of-Shadows, EP Social Guild, ~Proud member of: The Wandering Merchants, Phoenix Rising, Imperial Trade Union & Celestials of Nirn
    Sister Guilds with: Coroner's Report, Children of Skyrim, Sunshine Daydream, Tamriel Fisheries, Knights Arcanum and more
    "Not All Who Wander are Lost"
    #MOREHOUSINGSLOTS
    “When the people that can make the company more successful are sales and marketing people, they end up running the companies. The product people get driven out of the decision making forums, and the companies forget what it means to make great products.”

    _Steve Jobs (The Lost Interview)
    Options
  • KMarble
    KMarble
    ✭✭✭✭✭
    Gaeliannas wrote: »
    I am very curious how you can upgrade from 2012 hardware to 2022 and not get a performance increase? Regardless of how bad your code it, running it on faster processors, with more cores, and faster memory and I would hope way faster storage since flash arrays are relatively cheap now, add up to a guaranteed performance increase. So how is ZOS managing to not allow that to happen? And better yet, why?

    The choice of words used in the OP leads me to think that they'll only be replacing *some* parts, not the whole machines (similar to how some of us might add more memory, or a better (but still outdated) GC to an older computer).
    Options
  • Ingenon
    Ingenon
    ✭✭✭✭✭
    KMarble wrote: »
    The choice of words used in the OP leads me to think that they'll only be replacing *some* parts, not the whole machines (similar to how some of us might add more memory, or a better (but still outdated) GC to an older computer).
    [*] In most cases, we are still running on 2012-era server hardware and it needs to be updated before components start failing more than they are. So, we are literally swapping out (“refreshing”) all hardware in our two datacenters.
    [*] This will not result in any appreciable performance gain in-game, but it will result in a more reliable service overall that needs fewer unplanned maintenances.

    Just curious, how does "swapping out all hardware in our two data centers" lead you to think they'll only be replacing "some" parts?

    Swapping out all hardware in the two datacenters to me means every part.
    Options
  • Vaoh
    Vaoh
    ✭✭✭✭✭
    ✭✭✭✭✭
    This is very good news!

    Would be strange not to notice performance improvements though.
    Options
  • Gaeliannas
    Gaeliannas
    ✭✭✭✭✭
    Gaeliannas wrote: »
    [snip]

    Actually he said 2012-Era, some of it was probably bought as far back as 2010.

    Or older, as ESO started development in 2007.
    :)

    [edited to remove quote]

    Probably dumb of me to assume, but I am assuming the stuff from 2007 is what the PTS was running on until last week.
    Options
  • kojou
    kojou
    ✭✭✭✭✭
    ✭✭
    Gaeliannas wrote: »
    I am very curious how you can upgrade from 2012 hardware to 2022 and not get a performance increase? Regardless of how bad your code it, running it on faster processors, with more cores, and faster memory and I would hope way faster storage since flash arrays are relatively cheap now, add up to a guaranteed performance increase. So how is ZOS managing to not allow that to happen? And better yet, why?

    It does feel odd that they would say that, but my guess is that the server hardware that they are replacing isn't the overall bottleneck in what we perceive as "bad" system performance.

    My other interpretation is that they aren't targeting system performance, so therefore they aren't going to mention it. Anything that you notice that does perform better as a result of this is just an unintended benefit.

    In general, when you are an organization sizing these types of systems you look at the level of load and you design a hardware configuration to process that load. This isn't Linus Tech Tips where they are trying to destroy benchmarks with overclocking everything to the max. This is about a set of hardware performing a job consistently and with minimal unplanned downtime for a long time so they get a good return on investment.
    Playing since beta...
    Options
  • Ingenon
    Ingenon
    ✭✭✭✭✭
    Gaeliannas wrote: »
    I am very curious how you can upgrade from 2012 hardware to 2022 and not get a performance increase? Regardless of how bad your code it, running it on faster processors, with more cores, and faster memory and I would hope way faster storage since flash arrays are relatively cheap now, add up to a guaranteed performance increase. So how is ZOS managing to not allow that to happen? And better yet, why?

    I expect they have performed or at least looked at some benchmarks to determine how many new hardware to purchase. If they replace a room full of 2012 era servers with a smaller number of 2022 era servers, we could end up with the same overall performance.
    Options
  • Sylvermynx
    Sylvermynx
    ✭✭✭✭✭
    ✭✭✭✭✭
    Gaeliannas wrote: »
    Sylvermynx wrote: »
    LalMirchi wrote: »
    @ZOS_MattFiror

    While this is good news, I'm still puzzled.

    The Akamai overlay is still a connection/performance roadblock that seriously hampers connections and diminishes gameplay.

    Why not go to a cloud-based system such as Azure or AWS? they have smoothly running servers that could alleviate your hardware problems?

    As someone whose reseller provider switched to AWS last year from his own server farm, I can say with perfect truth that it's a damn good thing he gave me a 50% annual discount when he moved, because while my uptime on his former setup was 99.99%, my uptime on AWS is down to around 89%.

    Now that might not seem to be a big issue, but my usage of AWS is small compared to something as big as ZOS - and when I've had issues on AWS, so have really major corporations, including Amazon itself....

    I don't know anything about Azure, except that it's MS's cloud.

    That would be because your reseller cheaped out. You don't toss all your bananas in one basket (one AWS/Azure datacenter), you configure failover between AWS East/West or Pacific Northwest(which is cheaper) and you will have 5 nines uptime. Better yet, you failover between AWS/Azure. Last year(earlier this year?) AWS East crapped out, it brought down a ton of large companies, who were all under the impression being in a single datacenter would be fine, they learned better. At least 5 services we used went down with AWS East, every one of them promised to provide redundancy in the future in their apology letters.

    It is basically the same reason you have multiple Internet providers for your connectivity, and those providers have multiple routes configured, in case one goes down things still work.

    Yes, I'm aware of that (in fact, he himself told me that's why he was giving me a big discount - as well as me being the only remaining client from the month he began the reseller program 16 years ago; we go back a LONG way).

    I was just avoiding drawing any invidious comparisons anywhere - as I like my posting privs, y'know?
    Options
  • Gaeliannas
    Gaeliannas
    ✭✭✭✭✭
    Ingenon wrote: »
    Gaeliannas wrote: »
    I am very curious how you can upgrade from 2012 hardware to 2022 and not get a performance increase? Regardless of how bad your code it, running it on faster processors, with more cores, and faster memory and I would hope way faster storage since flash arrays are relatively cheap now, add up to a guaranteed performance increase. So how is ZOS managing to not allow that to happen? And better yet, why?

    I expect they have performed or at least looked at some benchmarks to determine how many new hardware to purchase. If they replace a room full of 2012 era servers with a smaller number of 2022 era servers, we could end up with the same overall performance.

    That's actually what I figure as well, which is sad. If they did a like for like replacement, performance would shoot through the roof. And through the roof I mean, make the game playable again in all scenarios.
    Options
  • Gaeliannas
    Gaeliannas
    ✭✭✭✭✭
    kojou wrote: »
    Gaeliannas wrote: »
    I am very curious how you can upgrade from 2012 hardware to 2022 and not get a performance increase? Regardless of how bad your code it, running it on faster processors, with more cores, and faster memory and I would hope way faster storage since flash arrays are relatively cheap now, add up to a guaranteed performance increase. So how is ZOS managing to not allow that to happen? And better yet, why?

    It does feel odd that they would say that, but my guess is that the server hardware that they are replacing isn't the overall bottleneck in what we perceive as "bad" system performance.

    My other interpretation is that they aren't targeting system performance, so therefore they aren't going to mention it. Anything that you notice that does perform better as a result of this is just an unintended benefit.

    In general, when you are an organization sizing these types of systems you look at the level of load and you design a hardware configuration to process that load. This isn't Linus Tech Tips where they are trying to destroy benchmarks with overclocking everything to the max. This is about a set of hardware performing a job consistently and with minimal unplanned downtime for a long time so they get a good return on investment.

    Oh yeah, I am generally the one sizing these types of jobs wherever I happen to be. They should also be looking 5+ years out, 10+ years in their case apparently. In which case you would think performance would jump considerably, then degrade over time as they add content/load, like they have been doing. Or they could just be installing enough to keep the game running as is, and hopefully have extras in boxes to use when they add load, so it won't degrade over time again, which would be nice as well. But the game definitely needs a huge performance boost now, not hope that whatever code untangling they are doing will maybe fix the issue a year+ from now.
    Options
  • Paske
    Paske
    ✭✭✭✭
    "In most cases, we are still running on 2012-era server hardware"

    "This will not result in any appreciable performance gain in-game"

    Translation - we got the cheapest stuff on the market.

    Only reason we are upgrading is because old hardware is End Of Life and no longer supported by vendor.

    New hardware will be barely bit better then what we have now,
    Options
  • FlopsyPrince
    FlopsyPrince
    ✭✭✭✭✭
    ✭✭✭
    Paske wrote: »
    "In most cases, we are still running on 2012-era server hardware"

    "This will not result in any appreciable performance gain in-game"

    Translation - we got the cheapest stuff on the market.

    Only reason we are upgrading is because old hardware is End Of Life and no longer supported by vendor.

    New hardware will be barely bit better then what we have now,

    An underlying question is if the operating system they run on can run on the latest hardware.

    That is not true for all of them! It also introduces huge security risks to be running on an old OS that has many known security holes. Too many companies failed to address that in the past and it bit many of them.
    PC
    PS4/PS5
    Options
  • Jaraal
    Jaraal
    ✭✭✭✭✭
    ✭✭✭✭✭
    Vaoh wrote: »
    This is very good news!

    Would be strange not to notice performance improvements though.

    Because all the data still has to go through the Akamai DDoS bottleneck, regardless of hardware quality.
    RIP Bosmer Nation. 4/4/14 - 2/25/19.
    Options
  • ArchangelIsraphel
    ArchangelIsraphel
    ✭✭✭✭✭
    ✭✭✭✭
    It’s worth noting we installed and transitioned all the hardware for the PTS a few weeks ago, before the U34 PTS update, and we’re happy to report it has been working very smoothly. This was our major test case to ensure that everything would work as expected.

    I have my doubts that this is working as smoothly as you say it is on the PTS.

    At the moment, the PTS is running worse than the live server for me, with barely anyone around.

    I have been experiencing:

    1. Infinite load screens and disconnects when logging into the server, switching zones, and, most notably, teleporting between houses.

    2. Infinite 1 second login queues followed by either a disconnect from the server or the need to relog.

    3. Initial startup of the test server always results in an infinite loadscreen/crash. I then need to restart the PTS, sit on an incredibly long load screen, and pray to whatever god is listening that I get to the character select.

    Overall worse performance, though since you said that we wont see performance improvements...I guess this is normal for ESO? >_>
    Legends never die
    They're written down in eternity
    But you'll never see the price it costs
    The scars collected all their lives
    When everything's lost, they pick up their hearts and avenge defeat
    Before it all starts, they suffer through harm just to touch a dream
    Oh, pick yourself up, 'cause
    Legends never die
    Options
  • Jaraal
    Jaraal
    ✭✭✭✭✭
    ✭✭✭✭✭
    It’s worth noting we installed and transitioned all the hardware for the PTS a few weeks ago, before the U34 PTS update, and we’re happy to report it has been working very smoothly. This was our major test case to ensure that everything would work as expected.

    I have my doubts that this is working as smoothly as you say it is on the PTS.

    At the moment, the PTS is running worse than the live server for me, with barely anyone around.

    Well that certainly doesn't bode well for the PC NA "upgrade," then. If it's working very smoothly on their end, but not on ours, then it must be bad code, or bad pathing causing the problems.


    Edited by Jaraal on April 30, 2022 5:19PM
    RIP Bosmer Nation. 4/4/14 - 2/25/19.
    Options
  • LalMirchi
    LalMirchi
    ✭✭✭✭✭
    Somehow this vanity project (own data-centers and related hardware) does not appeal to me, a mere user of this system. YMMV
    Options
  • AnonomissX
    AnonomissX
    ✭✭✭✭
    Since they were purchased by Microsoft why no usage of the Azure servers?
    Ebonheart Pact, Nord Templar/healer on NA Xbox server. My main toon WAS a tall and foxy redhead - now she has been gamma-irradiated and has green skin and black hair. 3 other characters I only use for writs. Can't be bothered to create multiple toons - EXCEPT now my WW is getting spooled up for Cyrodiil - Blood For The Pact! IRL cranky sometimes redhead chick at large in Las Vegas, NV
    Options
  • Sylvermynx
    Sylvermynx
    ✭✭✭✭✭
    ✭✭✭✭✭
    AnonomissX wrote: »
    Since they were purchased by Microsoft why no usage of the Azure servers?

    No clue, but it does make one wonder.
    Options
  • ZeroDPS
    ZeroDPS
    ✭✭✭✭
    GJ Matt and all ZOS team! Please take some time to upgrade everything which is possible from hardware perspective!!!
    Options
  • FlopsyPrince
    FlopsyPrince
    ✭✭✭✭✭
    ✭✭✭
    Sylvermynx wrote: »
    AnonomissX wrote: »
    Since they were purchased by Microsoft why no usage of the Azure servers?

    No clue, but it does make one wonder.

    Because moving would take quite a bit of effort. It is not a simple matter of just reloading the code in a different spot when moving "to The Cloud"....
    PC
    PS4/PS5
    Options
  • HonestLoverr
    HonestLoverr
    ✭✭✭
    "This will not result in any appreciable performance gain in-game"

    So when you finally get better hardware after 10 years, it will not lead to noticable performance gain. I mean.. seriously? No.1 complaint of players is the miserable performance! Could have had it fixed with this. So why not invest in good hardware then? Money is the problem? Can't Microsoft help out? If so, why? Heck you could have even opened up some crowdfunding to make better servers happen.

    In short: Nothing will change besides less maintainance..
    Options
  • Madhatten512
    Madhatten512
    ✭✭✭✭
    If it is not going to fix performance, then why even do it? Put money and effort into something that will. The game is seven years old its spent the last three or more being nearly unplayable most of the time and completely unplayable in prime time so what is the point if this is not going to directly impact the performance??? I baffles me that a company can get away with this stuff. The "year of performance" was how long ago?
    Options
  • LashanW
    LashanW
    ✭✭✭✭✭
    • In most cases, we are still running on 2012-era server hardware and it needs to be updated before components start failing more than they are. So, we are literally swapping out (“refreshing”) all hardware in our two datacenters.
    • This will not result in any appreciable performance gain in-game, but it will result in a more reliable service overall that needs fewer unplanned maintenances.
    What era hardware you getting as replacement for current 10 year old servers? Hopefully they are atleast from 2020.

    Will we see less major desyncs and more consistent loading screens atleast? My loading screen times are all over the place, anything from 10 seconds to 3 minutes, this is not reliable service. Even for same scenario like loading a character. It wasn't like this a few months back, nothing changed from my end. I hope your new servers are better.
    ---No longer active in ESO---
    Platform: PC-EU
    CP: 2500+
    Trial Achievements
    Godslayer, Gryphon Heart, Tick-Tock Tormentor, Immortal Redeemer, Dro-m'Athra Destroyer, vMoL no death

    Arena Achievements
    vMA Flawless, vVH Spirit Slayer

    DLC Dungeon Trifectas
    Scalecaller Peak, Fang Lair, Depths of Malatar, Icereach
    Options
  • Danikat
    Danikat
    ✭✭✭✭✭
    ✭✭✭✭✭
    Thank you for scheduling this maintenance for Tuesday, instead of Monday which is a holiday in a lot of Europe and I think some other places. It will be nice to be able to play ESO on the extra day off. :)
    PC EU player | She/her/hers | PAWS (Positively Against Wrip-off Stuff) - Say No to Crown Crates!

    "Remember in this game we call life that no one said it's fair"
    Options
  • Meiox
    Meiox
    ✭✭✭✭
    ApoAlaia wrote: »
    PC/EU is the most populous

    This right here strikes me as a reason not to do EU first. Consider: if you had to risk a group of people, would you rather risk a group of forty or a group of sixty? Yes it’s bad if something happens either way but you want to minimize risk. If NA dies during the upgrade they have a smaller proportion of their player base unable to play, and either way they have more practice for when they get to EU.

    If that is the reason, wouldn't they do it first on one of the console servers, since I don't think that PC NA is the server with the least amount of players.
    Options
Sign In or Register to comment.