Upcoming PC NA Datacenter Hardware Replacement

tomfant

Gaeliannas wrote: »

I am very curious how you can upgrade from 2012 hardware to 2022 and not get a performance increase? Regardless of how bad your code it, running it on faster processors, with more cores, and faster memory and I would hope way faster storage since flash arrays are relatively cheap now, add up to a guaranteed performance increase. So how is ZOS managing to not allow that to happen? And better yet, why?

Yep, very strange. Code must be really bad, or the new hardware is just a few i3...

Sylvermynx

LalMirchi wrote: »

@ZOS_MattFiror

While this is good news, I'm still puzzled.

The Akamai overlay is still a connection/performance roadblock that seriously hampers connections and diminishes gameplay.

Why not go to a cloud-based system such as Azure or AWS? they have smoothly running servers that could alleviate your hardware problems?

As someone whose reseller provider switched to AWS last year from his own server farm, I can say with perfect truth that it's a damn good thing he gave me a 50% annual discount when he moved, because while my uptime on his former setup was 99.99%, my uptime on AWS is down to around 89%.

Now that might not seem to be a big issue, but my usage of AWS is small compared to something as big as ZOS - and when I've had issues on AWS, so have really major corporations, including Amazon itself....

I don't know anything about Azure, except that it's MS's cloud.

Gaeliannas

Sylvermynx wrote: »

LalMirchi wrote: »

@ZOS_MattFiror

While this is good news, I'm still puzzled.

The Akamai overlay is still a connection/performance roadblock that seriously hampers connections and diminishes gameplay.

Why not go to a cloud-based system such as Azure or AWS? they have smoothly running servers that could alleviate your hardware problems?

As someone whose reseller provider switched to AWS last year from his own server farm, I can say with perfect truth that it's a damn good thing he gave me a 50% annual discount when he moved, because while my uptime on his former setup was 99.99%, my uptime on AWS is down to around 89%.

Now that might not seem to be a big issue, but my usage of AWS is small compared to something as big as ZOS - and when I've had issues on AWS, so have really major corporations, including Amazon itself....

I don't know anything about Azure, except that it's MS's cloud.

That would be because your reseller cheaped out. You don't toss all your bananas in one basket (one AWS/Azure datacenter), you configure failover between AWS East/West or Pacific Northwest(which is cheaper) and you will have 5 nines uptime. Better yet, you failover between AWS/Azure. Last year(earlier this year?) AWS East crapped out, it brought down a ton of large companies, who were all under the impression being in a single datacenter would be fine, they learned better. At least 5 services we used went down with AWS East, every one of them promised to provide redundancy in the future in their apology letters.

It is basically the same reason you have multiple Internet providers for your connectivity, and those providers have multiple routes configured, in case one goes down things still work.

gariondavey

My data is going to be SO refreshed after this.
Hooray!

Gaeliannas

Grandchamp1989 wrote: »

[snip]

Actually he said 2012-Era, some of it was probably bought as far back as 2010.

[edited to remove quote]

wenchmore420b14_ESO

Gaeliannas wrote: »

Grandchamp1989 wrote: »

[snip]

Actually he said 2012-Era, some of it was probably bought as far back as 2010.

Or older, as ESO started development in 2007.


[edited to remove quote]

KMarble

Gaeliannas wrote: »

I am very curious how you can upgrade from 2012 hardware to 2022 and not get a performance increase? Regardless of how bad your code it, running it on faster processors, with more cores, and faster memory and I would hope way faster storage since flash arrays are relatively cheap now, add up to a guaranteed performance increase. So how is ZOS managing to not allow that to happen? And better yet, why?

The choice of words used in the OP leads me to think that they'll only be replacing *some* parts, not the whole machines (similar to how some of us might add more memory, or a better (but still outdated) GC to an older computer).

Ingenon

KMarble wrote: »

The choice of words used in the OP leads me to think that they'll only be replacing *some* parts, not the whole machines (similar to how some of us might add more memory, or a better (but still outdated) GC to an older computer).

ZOS_MattFiror wrote: »

[*] In most cases, we are still running on 2012-era server hardware and it needs to be updated before components start failing more than they are. So, we are literally swapping out (“refreshing”) all hardware in our two datacenters.
[*] This will not result in any appreciable performance gain in-game, but it will result in a more reliable service overall that needs fewer unplanned maintenances.

Just curious, how does "swapping out all hardware in our two data centers" lead you to think they'll only be replacing "some" parts?

Swapping out all hardware in the two datacenters to me means every part.

Vaoh

This is very good news!

Would be strange not to notice performance improvements though.

Gaeliannas

wenchmore420b14_ESO wrote: »

Gaeliannas wrote: »

Grandchamp1989 wrote: »

[snip]

Actually he said 2012-Era, some of it was probably bought as far back as 2010.

Or older, as ESO started development in 2007.


[edited to remove quote]

Probably dumb of me to assume, but I am assuming the stuff from 2007 is what the PTS was running on until last week.

kojou

Gaeliannas wrote: »

I am very curious how you can upgrade from 2012 hardware to 2022 and not get a performance increase? Regardless of how bad your code it, running it on faster processors, with more cores, and faster memory and I would hope way faster storage since flash arrays are relatively cheap now, add up to a guaranteed performance increase. So how is ZOS managing to not allow that to happen? And better yet, why?

It does feel odd that they would say that, but my guess is that the server hardware that they are replacing isn't the overall bottleneck in what we perceive as "bad" system performance.

My other interpretation is that they aren't targeting system performance, so therefore they aren't going to mention it. Anything that you notice that does perform better as a result of this is just an unintended benefit.

In general, when you are an organization sizing these types of systems you look at the level of load and you design a hardware configuration to process that load. This isn't Linus Tech Tips where they are trying to destroy benchmarks with overclocking everything to the max. This is about a set of hardware performing a job consistently and with minimal unplanned downtime for a long time so they get a good return on investment.

Ingenon

Gaeliannas wrote: »

I am very curious how you can upgrade from 2012 hardware to 2022 and not get a performance increase? Regardless of how bad your code it, running it on faster processors, with more cores, and faster memory and I would hope way faster storage since flash arrays are relatively cheap now, add up to a guaranteed performance increase. So how is ZOS managing to not allow that to happen? And better yet, why?

I expect they have performed or at least looked at some benchmarks to determine how many new hardware to purchase. If they replace a room full of 2012 era servers with a smaller number of 2022 era servers, we could end up with the same overall performance.

Sylvermynx

Gaeliannas wrote: »

Sylvermynx wrote: »

LalMirchi wrote: »

@ZOS_MattFiror

While this is good news, I'm still puzzled.

The Akamai overlay is still a connection/performance roadblock that seriously hampers connections and diminishes gameplay.

Why not go to a cloud-based system such as Azure or AWS? they have smoothly running servers that could alleviate your hardware problems?

As someone whose reseller provider switched to AWS last year from his own server farm, I can say with perfect truth that it's a damn good thing he gave me a 50% annual discount when he moved, because while my uptime on his former setup was 99.99%, my uptime on AWS is down to around 89%.

Now that might not seem to be a big issue, but my usage of AWS is small compared to something as big as ZOS - and when I've had issues on AWS, so have really major corporations, including Amazon itself....

I don't know anything about Azure, except that it's MS's cloud.

That would be because your reseller cheaped out. You don't toss all your bananas in one basket (one AWS/Azure datacenter), you configure failover between AWS East/West or Pacific Northwest(which is cheaper) and you will have 5 nines uptime. Better yet, you failover between AWS/Azure. Last year(earlier this year?) AWS East crapped out, it brought down a ton of large companies, who were all under the impression being in a single datacenter would be fine, they learned better. At least 5 services we used went down with AWS East, every one of them promised to provide redundancy in the future in their apology letters.

It is basically the same reason you have multiple Internet providers for your connectivity, and those providers have multiple routes configured, in case one goes down things still work.

Yes, I'm aware of that (in fact, he himself told me that's why he was giving me a big discount - as well as me being the only remaining client from the month he began the reseller program 16 years ago; we go back a LONG way).

I was just avoiding drawing any invidious comparisons anywhere - as I like my posting privs, y'know?

Gaeliannas

Ingenon wrote: »

Gaeliannas wrote: »

I am very curious how you can upgrade from 2012 hardware to 2022 and not get a performance increase? Regardless of how bad your code it, running it on faster processors, with more cores, and faster memory and I would hope way faster storage since flash arrays are relatively cheap now, add up to a guaranteed performance increase. So how is ZOS managing to not allow that to happen? And better yet, why?

I expect they have performed or at least looked at some benchmarks to determine how many new hardware to purchase. If they replace a room full of 2012 era servers with a smaller number of 2022 era servers, we could end up with the same overall performance.

That's actually what I figure as well, which is sad. If they did a like for like replacement, performance would shoot through the roof. And through the roof I mean, make the game playable again in all scenarios.

Gaeliannas

kojou wrote: »

Gaeliannas wrote: »

I am very curious how you can upgrade from 2012 hardware to 2022 and not get a performance increase? Regardless of how bad your code it, running it on faster processors, with more cores, and faster memory and I would hope way faster storage since flash arrays are relatively cheap now, add up to a guaranteed performance increase. So how is ZOS managing to not allow that to happen? And better yet, why?

It does feel odd that they would say that, but my guess is that the server hardware that they are replacing isn't the overall bottleneck in what we perceive as "bad" system performance.

My other interpretation is that they aren't targeting system performance, so therefore they aren't going to mention it. Anything that you notice that does perform better as a result of this is just an unintended benefit.

In general, when you are an organization sizing these types of systems you look at the level of load and you design a hardware configuration to process that load. This isn't Linus Tech Tips where they are trying to destroy benchmarks with overclocking everything to the max. This is about a set of hardware performing a job consistently and with minimal unplanned downtime for a long time so they get a good return on investment.

Oh yeah, I am generally the one sizing these types of jobs wherever I happen to be. They should also be looking 5+ years out, 10+ years in their case apparently. In which case you would think performance would jump considerably, then degrade over time as they add content/load, like they have been doing. Or they could just be installing enough to keep the game running as is, and hopefully have extras in boxes to use when they add load, so it won't degrade over time again, which would be nice as well. But the game definitely needs a huge performance boost now, not hope that whatever code untangling they are doing will maybe fix the issue a year+ from now.

Paske

"In most cases, we are still running on 2012-era server hardware"

"This will not result in any appreciable performance gain in-game"

Translation - we got the cheapest stuff on the market.

Only reason we are upgrading is because old hardware is End Of Life and no longer supported by vendor.

New hardware will be barely bit better then what we have now,

FlopsyPrince

Paske wrote: »

"In most cases, we are still running on 2012-era server hardware"

"This will not result in any appreciable performance gain in-game"

Translation - we got the cheapest stuff on the market.

Only reason we are upgrading is because old hardware is End Of Life and no longer supported by vendor.

New hardware will be barely bit better then what we have now,

An underlying question is if the operating system they run on can run on the latest hardware.

That is not true for all of them! It also introduces huge security risks to be running on an old OS that has many known security holes. Too many companies failed to address that in the past and it bit many of them.

Jaraal

Vaoh wrote: »

This is very good news!

Would be strange not to notice performance improvements though.

Because all the data still has to go through the Akamai DDoS bottleneck, regardless of hardware quality.

ArchangelIsraphel

It’s worth noting we installed and transitioned all the hardware for the PTS a few weeks ago, before the U34 PTS update, and we’re happy to report it has been working very smoothly. This was our major test case to ensure that everything would work as expected.

I have my doubts that this is working as smoothly as you say it is on the PTS.

At the moment, the PTS is running worse than the live server for me, with barely anyone around.

I have been experiencing:

1. Infinite load screens and disconnects when logging into the server, switching zones, and, most notably, teleporting between houses.

2. Infinite 1 second login queues followed by either a disconnect from the server or the need to relog.

3. Initial startup of the test server always results in an infinite loadscreen/crash. I then need to restart the PTS, sit on an incredibly long load screen, and pray to whatever god is listening that I get to the character select.

Overall worse performance, though since you said that we wont see performance improvements...I guess this is normal for ESO? >_>

Jaraal

ArchangelIsraphel wrote: »

It’s worth noting we installed and transitioned all the hardware for the PTS a few weeks ago, before the U34 PTS update, and we’re happy to report it has been working very smoothly. This was our major test case to ensure that everything would work as expected.

I have my doubts that this is working as smoothly as you say it is on the PTS.

At the moment, the PTS is running worse than the live server for me, with barely anyone around.

Well that certainly doesn't bode well for the PC NA "upgrade," then. If it's working very smoothly on their end, but not on ours, then it must be bad code, or bad pathing causing the problems.

LalMirchi

Somehow this vanity project (own data-centers and related hardware) does not appeal to me, a mere user of this system. YMMV

AnonomissX

Since they were purchased by Microsoft why no usage of the Azure servers?

Sylvermynx

AnonomissX wrote: »

Since they were purchased by Microsoft why no usage of the Azure servers?

No clue, but it does make one wonder.

ZeroDPS

GJ Matt and all ZOS team! Please take some time to upgrade everything which is possible from hardware perspective!!!

FlopsyPrince

Sylvermynx wrote: »

AnonomissX wrote: »

Since they were purchased by Microsoft why no usage of the Azure servers?

No clue, but it does make one wonder.

Because moving would take quite a bit of effort. It is not a simple matter of just reloading the code in a different spot when moving "to The Cloud"....

HonestLoverr

"This will not result in any appreciable performance gain in-game"

So when you finally get better hardware after 10 years, it will not lead to noticable performance gain. I mean.. seriously? No.1 complaint of players is the miserable performance! Could have had it fixed with this. So why not invest in good hardware then? Money is the problem? Can't Microsoft help out? If so, why? Heck you could have even opened up some crowdfunding to make better servers happen.

In short: Nothing will change besides less maintainance..

Madhatten512

If it is not going to fix performance, then why even do it? Put money and effort into something that will. The game is seven years old its spent the last three or more being nearly unplayable most of the time and completely unplayable in prime time so what is the point if this is not going to directly impact the performance??? I baffles me that a company can get away with this stuff. The "year of performance" was how long ago?

LashanW

ZOS_MattFiror wrote: »

• In most cases, we are still running on 2012-era server hardware and it needs to be updated before components start failing more than they are. So, we are literally swapping out (“refreshing”) all hardware in our two datacenters.
• This will not result in any appreciable performance gain in-game, but it will result in a more reliable service overall that needs fewer unplanned maintenances.

What era hardware you getting as replacement for current 10 year old servers? Hopefully they are atleast from 2020.

Will we see less major desyncs and more consistent loading screens atleast? My loading screen times are all over the place, anything from 10 seconds to 3 minutes, this is not reliable service. Even for same scenario like loading a character. It wasn't like this a few months back, nothing changed from my end. I hope your new servers are better.

Danikat

Thank you for scheduling this maintenance for Tuesday, instead of Monday which is a holiday in a lot of Europe and I think some other places. It will be nice to be able to play ESO on the extra day off.

Meiox

WhiteCoatSyndrome wrote: »

ApoAlaia wrote: »

PC/EU is the most populous

This right here strikes me as a reason not to do EU first. Consider: if you had to risk a group of people, would you rather risk a group of forty or a group of sixty? Yes it’s bad if something happens either way but you want to minimize risk. If NA dies during the upgrade they have a smaller proportion of their player base unable to play, and either way they have more practice for when they get to EU.

If that is the reason, wouldn't they do it first on one of the console servers, since I don't think that PC NA is the server with the least amount of players.