"Outdated Server Hardware Failures" and Their Impact on Performance

Cuddlypuff

14 months ago, in ZOS' own words (https://twitter.com/tesonline/status/1354891564447911940):
We'll be updating all datacenter hardware to ensure we have a modern infrastructure and to support the game for many more years. This is not new/more megaservers and is unlikely to have a direct impact on performance, but will ensure that outdated hardware failures do not occur.
7:38 AM · Jan 29, 2021

Servers consist of real hardware which fail over time according to statistical models (every component typically has a MTBF or mean-time-between-failure). Due to supply chain issues, apparently no server updates have occurred yet and over 10,000 hours has elapsed since the announcement. According to some generic server failure stats (https://statista.com/statistics/430769/annual-failure-rates-of-servers/), one could expect at least 15-20% of the servers to have failed by now.

On top of this, it appears that the servers were already choking under load well before 2021 which would imply no over-provisioning of bandwidth or resources to accommodate for additional failures. Clearly an extra loss of capacity would be unsustainable, so it appears that ZOS is addressing this by:
1. Reducing database footprint (eg cold storage, AWA)
2. Heavier rate limiting on logins, mail, trader requests (including booted from server for learning enchanting or turning in master writs too quickly)
3. (Speculative, but most likely true from evidence) Re-allocating servers to more critical/profitable areas of the game. For example PC-EU may be allocated more server time than PC-NA due to their population peak being 3-4 times higher (and possibly explaining why PC-NA had dungeon finder issues after U33 but not PC-EU). It's also reasonable to expect that server time was taken away from Cyrodiil after decreasing the poplock caps, which explains why the lag is even worse despite pvp population reaching an all-time low.

Understandably the cost-benefit calculations of overpriced server upgrades during a pandemic are an upper management decision, but in hindsight it appears that inaction was a poor choice.

phaneub17_ESO

Considering the pandemic caused an unforeseen massive backlog of shipment and a chip shortage around the world, can they be faulted on the follow through that came from it? Square-Enix had that problem too, unable to import new server hardware to Japan they made due of opening new servers to support the Oceanic community to relieve congestion.

Abigail

Cuddlypuff wrote: »

Understandably the cost-benefit calculations of overpriced server upgrades during a pandemic are an upper management decision, but in hindsight it appears that inaction was a poor choice.

I get it and I ought to feel some empathy for ZOS. But if the problem is really situationally caused inability to replace tired hardware, then subscription rates should have been dropped appropriately. Truth is, if ZOS isn't investing in new equipment for whatever reason, then they are profiting from COVID.

I feel no pity whatsoever. So sad, too bad.

Gaeliannas

phaneub17_ESO wrote: »

Considering the pandemic caused an unforeseen massive backlog of shipment and a chip shortage around the world, can they be faulted on the follow through that came from it? Square-Enix had that problem too, unable to import new server hardware to Japan they made due of opening new servers to support the Oceanic community to relieve congestion.

When designing a datacenter and the hardware in it to support whatever it is you are running, in this case ESO. What would normally happen is a 5 year plan be made, and the system designed to support the growth for the next 5 years. Sometimes, if you are not confident about your product, or there are a slew of variables the business folks can not account for, you go with a 3 year plan instead.

Once approaching the 5 year mark, you re-evaluate your plan and make a new one for the next 5 years. You then start purchasing replacement hardware and new hardware to support the products continued growth over time. By the 7 year mark most original hardware should have been replaced (the servers and storage at the very least, network hardware lasts longer) and your product should be running fine, with enough new hardware coming in at regular intervals to support new features that have been added, in this case Chapters, Zones, DLC's, etc...

Figuring ZOS built at least some of their datacenter back in 2012, replacements should have started in 2017 and been completed by 2019, with new servers to support the next 5 years growth, taking them right through Covid, the supply chain issues and well into 2022 before having to repeat the process. Yet, here we are in 2022 with what seems to be no hardware replaced or new hardware added to support the growth of the game over the past 8 years it has been live. At least I assume that is the case by them never mentioning they did it, and mentioning a year ago they were just starting the process, which should have already been completed by normal standards.

Never mind the fact that the original hardware was bought to support the original game design, and One Tamriel changed everything. Instead of predictable player loads due to level progression through zones, now everyone can play everywhere, and they do, which requires a lot more computing power than the original design did.

Jusey1

phaneub17_ESO wrote: »

Considering the pandemic caused an unforeseen massive backlog of shipment and a chip shortage around the world, can they be faulted on the follow through that came from it? Square-Enix had that problem too, unable to import new server hardware to Japan they made due of opening new servers to support the Oceanic community to relieve congestion.

Also scalpers and NFT scammers as well whom probably are also buying out the new hardware for their own scams. I have heard that this had impacted the market in hardware by a lot since those types of people buy them out by the masses consistently.

_Zathras_

phaneub17_ESO wrote: »

Considering the pandemic caused an unforeseen massive backlog of shipment and a chip shortage around the world, can they be faulted on the follow through that came from it? Square-Enix had that problem too, unable to import new server hardware to Japan they made due of opening new servers to support the Oceanic community to relieve congestion.

Riiight, but at least they stopped the influx of sales and character creation when they realized they reached a critical point.

ESO kept it's doors open, and now they are trimming core game features to try to fix the hemorrhaging. Even that isn't working.

Also, this is the way to handle community PR, direct from their Producer.

Dietche

I'm sorry I have to disagree wholeheartedly.

They merged with Microsoft. Microsoft has all the hardware in the world that they need to easily run 2000 copies of ESO vis-a-vis Azure.

Thus, I am wholly unwilling to excuse ZOS based on the idea they don't have access to server equipment that can run ESO, regardless of supply chain issues, covid, or otherwise.

katorga

Dietche wrote: »

I'm sorry I have to disagree wholeheartedly.

They merged with Microsoft. Microsoft has all the hardware in the world that they need to easily run 2000 copies of ESO vis-a-vis Azure.

Thus, I am wholly unwilling to excuse ZOS based on the idea they don't have access to server equipment that can run ESO, regardless of supply chain issues, covid, or otherwise.

You would think they would halt the server replacements, and shift the project to migrating to MS infrastructure. Granted, The game is 7 years old, which means it was developed 10-12 years ago using the data center models common to MMOs a decade ago. The game probably needs some rewriting to run on modern cloud data centers.

Dietche

katorga wrote: »

You would think they would halt the server replacements, and shift the project to migrating to MS infrastructure. Granted, The game is 7 years old, which means it was developed 10-12 years ago using the data center models common to MMOs a decade ago. The game probably needs some rewriting to run on modern cloud data centers.

I still rather have to disagree. One word: Virtualization.
Azure is built from the ground up to host numerous types of operating systems and programs, from nearly any year that computers have .... uhmmm... computed.

ESO can run in Azure with very little modification or work, I'm fairly certain, via VMWare, Parallels, or most likely, Microsoft Hyper-V. Especially, Hyper-V, since they own the [snip] thing.

[edited for profanity bypass]

EmEm_Oh

Jusey1 wrote: »

phaneub17_ESO wrote: »

Considering the pandemic caused an unforeseen massive backlog of shipment and a chip shortage around the world, can they be faulted on the follow through that came from it? Square-Enix had that problem too, unable to import new server hardware to Japan they made due of opening new servers to support the Oceanic community to relieve congestion.

Also scalpers and NFT scammers as well whom probably are also buying out the new hardware for their own scams. I have heard that this had impacted the market in hardware by a lot since those types of people buy them out by the masses consistently.

Microsoft has had an unofficial "Hands-Off" when it came to ESO, but maybe they need to intervene and get the equipment that is necessary. They have the buying power and resources, regardless of the scalping.

With looming issues with China and Taiwan, it is essential that MS and ZOS get ESO on the right track now, or it won't be good for future players. Because sourcing will be a serious problem if Taiwan is cut off from the world and will cause tech panic like no other.

Cuddlypuff

katorga wrote: »

Dietche wrote: »

I'm sorry I have to disagree wholeheartedly.

They merged with Microsoft. Microsoft has all the hardware in the world that they need to easily run 2000 copies of ESO vis-a-vis Azure.

Thus, I am wholly unwilling to excuse ZOS based on the idea they don't have access to server equipment that can run ESO, regardless of supply chain issues, covid, or otherwise.

You would think they would halt the server replacements, and shift the project to migrating to MS infrastructure. Granted, The game is 7 years old, which means it was developed 10-12 years ago using the data center models common to MMOs a decade ago. The game probably needs some rewriting to run on modern cloud data centers.

Just looking at the way lag bleeds into every facet of the game whether its mail, pvp, login, traders, pve, instances or open world etc, it's very obvious that the server architecture is fundamentally monolithic in nature. It also doesn't help that the ESO game engine is entirely proprietary and probably inspired by/based off HERO engine which ended development in 2012. On-premise to cloud migration is not as simple as just hitting a switch. The entire game architecture probably needs to be redesigned from the group up around microservices, distributed processing and containerization etc to truly make the most of and save money from going cloud.

You know what does run as a microservice though? The crown store is always rock solid no matter how borked the game itself is.

Dietche

Microsoft doesn't need to buy *anything* to run ESO, EmEm_Oh.
Azure has MORE than enough spare horsepower to run something as piddly as ESO is.
I'd wager you wouldn't even see 1% extra load overall on their server farms.
More likely, a tiny fraction of that.

And it would take a very, very short time to spin up a Virtual Server to hold ESO inside of, burp the database over on a 10Gig line (or simply carry a book drive), and problems solved.

*you hear 2 pennies hit a bucket from far away*

ShawnLaRock

Doesn’t matter how new your server hardware is, when your ~15yr old engine is ancient and decrepit.

S.

Cuddlypuff

Dietche wrote: »

Microsoft doesn't need to buy *anything* to run ESO, EmEm_Oh.
Azure has MORE than enough spare horsepower to run something as piddly as ESO is.
I'd wager you wouldn't even see 1% extra load overall on their server farms.
More likely, a tiny fraction of that.

And it would take a very, very short time to spin up a Virtual Server to hold ESO inside of, burp the database over on a 10Gig line (or simply carry a book drive), and problems solved.

*you hear 2 pennies hit a bucket from far away*

Just because Microsoft owns Zenimax doesn't make Azure free...they will still have to pay an internal rate for cloud services which is probably far higher than their existing on-premise OPEX. Also the database of this scale is going to be multiple replicated clusters rather than a book drive. You can see they have already tried to cut expenses by moving the majority of the data to cold storage.

Dietche

@Cuddlypuff
"Microsoft owns Zenimax"
"Someone will have to pay for Azure"


uhmmmmm????
Those thoughts of yours contradict themselves.
You are trying to suggest that the owner of Azure will charge the owner of ESO, when, in fact, that's the same company.

I do not believe Microsoft would charge *themselves* to run something on their hardware that they already OWN.

Dietche

Also? As far as "replicated clusters"???
Uhhhmmmm
"Replicated" means.... a copy.

All databases can be compacted down to a single file, transferred, and reinstated on new hardware. The fact they are on redundant, replicated clusters doesn't mean anything, because those are all simply copies running concurrently of the same primary data.

Elementalism77

Dietche wrote: »

@Cuddlypuff
"Microsoft owns Zenimax"
"Someone will have to pay for Azure"


uhmmmmm????
Those thoughts of yours contradict themselves.
You are trying to suggest that the owner of Azure will charge the owner of ESO, when, in fact, that's the same company.

I do not believe Microsoft would charge *themselves* to run something on their hardware that they already OWN.

Yes they charge their own divisions for using Azure. You think the VPs and C-level players on Azure side are going to miss out on their bonus because the VPs and C-Suite at Zenimax want something for free?

Every company I have worked for which has divisions or business units that use a service from another are charged for that service. If it were given away for free it would be more difficult to hide the true cost. Which becomes a problem for not only inter-agency planning but tax purposes and public disclosure of costs to the shareholders.

Elementalism77

Gaeliannas wrote: »

phaneub17_ESO wrote: »

Considering the pandemic caused an unforeseen massive backlog of shipment and a chip shortage around the world, can they be faulted on the follow through that came from it? Square-Enix had that problem too, unable to import new server hardware to Japan they made due of opening new servers to support the Oceanic community to relieve congestion.

When designing a datacenter and the hardware in it to support whatever it is you are running, in this case ESO. What would normally happen is a 5 year plan be made, and the system designed to support the growth for the next 5 years. Sometimes, if you are not confident about your product, or there are a slew of variables the business folks can not account for, you go with a 3 year plan instead.

Once approaching the 5 year mark, you re-evaluate your plan and make a new one for the next 5 years. You then start purchasing replacement hardware and new hardware to support the products continued growth over time. By the 7 year mark most original hardware should have been replaced (the servers and storage at the very least, network hardware lasts longer) and your product should be running fine, with enough new hardware coming in at regular intervals to support new features that have been added, in this case Chapters, Zones, DLC's, etc...

Figuring ZOS built at least some of their datacenter back in 2012, replacements should have started in 2017 and been completed by 2019, with new servers to support the next 5 years growth, taking them right through Covid, the supply chain issues and well into 2022 before having to repeat the process. Yet, here we are in 2022 with what seems to be no hardware replaced or new hardware added to support the growth of the game over the past 8 years it has been live. At least I assume that is the case by them never mentioning they did it, and mentioning a year ago they were just starting the process, which should have already been completed by normal standards.

Never mind the fact that the original hardware was bought to support the original game design, and One Tamriel changed everything. Instead of predictable player loads due to level progression through zones, now everyone can play everywhere, and they do, which requires a lot more computing power than the original design did.

COVID has thrown a lot of last two years out to door with planning. Money cutoff due to not knowing ramifications of mandates and shutdowns. When it started opening back up supply chain issues. I ordered 7 figures worth of Cisco switching, DNA infrastructure, and WAPs with controllers June 30th last summer. We just received some of the chassis and WAPs in the last two weeks. Not expected to have the full order arriving until mid June if we are lucky. 12 months from order date until arrival. Insane.

Jaraal

Abigail wrote: »

Cuddlypuff wrote: »

Understandably the cost-benefit calculations of overpriced server upgrades during a pandemic are an upper management decision, but in hindsight it appears that inaction was a poor choice.

I get it and I ought to feel some empathy for ZOS. But if the problem is really situationally caused inability to replace tired hardware, then subscription rates should have been dropped appropriately. Truth is, if ZOS isn't investing in new equipment for whatever reason, then they are profiting from COVID.

I feel no pity whatsoever. So sad, too bad.

Yes, and they have specifically blamed “work from home” for part of the current problems.

I’m sorry, but excuses don’t really cut it in the business world. What other business types would expect their consumers to be happy with an inferior, shoddily implemented and severely problematic product?

AnonomissX

In the sight of high speed internet connections, virtual w/in virtual computers and multiple platforms that let you use audio and video conferencing, Microsoft Teams and Zoom to name a couple...what is up with the need to work from home being a stumbling block anyways?

I work from home and I have to collaborate from time to time, and even see this as a boost to productivity so I don't buy the remote work excuse. The need to do so started in ernest in 03/2020. 2 years ago most companies have benefited from the remote worker situation in that time. And Azure servers? I thought they would have got the benefit of that the day I heard Microsoft purchased the company outright. Would that not have been cheaper then waiting for servers that never came? Hardware cost vs rental cost?

Gaeliannas

Elementalism77 wrote: »

Gaeliannas wrote: »

phaneub17_ESO wrote: »

Considering the pandemic caused an unforeseen massive backlog of shipment and a chip shortage around the world, can they be faulted on the follow through that came from it? Square-Enix had that problem too, unable to import new server hardware to Japan they made due of opening new servers to support the Oceanic community to relieve congestion.

When designing a datacenter and the hardware in it to support whatever it is you are running, in this case ESO. What would normally happen is a 5 year plan be made, and the system designed to support the growth for the next 5 years. Sometimes, if you are not confident about your product, or there are a slew of variables the business folks can not account for, you go with a 3 year plan instead.

Once approaching the 5 year mark, you re-evaluate your plan and make a new one for the next 5 years. You then start purchasing replacement hardware and new hardware to support the products continued growth over time. By the 7 year mark most original hardware should have been replaced (the servers and storage at the very least, network hardware lasts longer) and your product should be running fine, with enough new hardware coming in at regular intervals to support new features that have been added, in this case Chapters, Zones, DLC's, etc...

Figuring ZOS built at least some of their datacenter back in 2012, replacements should have started in 2017 and been completed by 2019, with new servers to support the next 5 years growth, taking them right through Covid, the supply chain issues and well into 2022 before having to repeat the process. Yet, here we are in 2022 with what seems to be no hardware replaced or new hardware added to support the growth of the game over the past 8 years it has been live. At least I assume that is the case by them never mentioning they did it, and mentioning a year ago they were just starting the process, which should have already been completed by normal standards.

Never mind the fact that the original hardware was bought to support the original game design, and One Tamriel changed everything. Instead of predictable player loads due to level progression through zones, now everyone can play everywhere, and they do, which requires a lot more computing power than the original design did.

COVID has thrown a lot of last two years out to door with planning. Money cutoff due to not knowing ramifications of mandates and shutdowns. When it started opening back up supply chain issues. I ordered 7 figures worth of Cisco switching, DNA infrastructure, and WAPs with controllers June 30th last summer. We just received some of the chassis and WAPs in the last two weeks. Not expected to have the full order arriving until mid June if we are lucky. 12 months from order date until arrival. Insane.

Point was they should have been upgrading all along as they added more load and the existing equipment was aging, which happened well before covid. Also, had they actually ordered the gear a year+ ago as they announced they did, it should have been here by now. I have the same experience as you with ordering, most of my stuff came in within 6 months though, there were some outlier pieces that took about 11 months to show up, but that was mostly network equipment which Cisco seems behind on, all our servers and storage has showed up fairly quickly though all things considered.

Anyhow, what has happened isn't actually a surprise to me at all. Quite a few others have posted it and I have been watching where this has been going for years. It started in Cyrodiil where it looks like they robbed Peter to pay Paul in order to keep pushing out new zones, but it seems to have caught up to them over a year ago, when PVE also started becoming unstable and laggy as happened in Cyrodiil, and they kept dropping new zones/content regardless. The performance degradation was pretty obvious over the last year as they pushed out new content, and it was bound to break down at some point, regardless of how hard they are trying to code/optimize their way around it.

Anyhow, thats simply my take on the matter for what it's worth, from the little information they actually share, and the quite obvious situation we have all found ourselves in. It seems like the latest DLC, may have broke the proverbial camels back.

DarrowLykos

Who said you can't still make money from spit and duct tape

tsaescishoeshiner

I love threads like this because you really get a sense of being exposed to knowledge—without actually learning anything!

Highly qualified business and server experts are wielding their jargonic buzzwords against OTHER experts—and they're all coming to different conclusions.

Everyone poses as an expert with insider knowledge and touts universal business principles, without knowing what's actually going on inside the company they're talking about. I wonder if these forum experts have strangers posting online about their business and how it supposedly operates. *eats more popcorn*

Cadbury

tsaescishoeshiner wrote: »

I love threads like this because you really get a sense of being exposed to knowledge—without actually learning anything!

Highly qualified business and server experts are wielding their jargonic buzzwords against OTHER experts—and they're all coming to different conclusions.

Everyone poses as an expert with insider knowledge and touts universal business principles, without knowing what's actually going on inside the company they're talking about. I wonder if these forum experts have strangers posting online about their business and how it supposedly operates. *eats more popcorn*

It's the internet. Anyone can be an "expert" or an "industry insider".
*Steals some popcorn*

WhyMustItBe

phaneub17_ESO wrote: »

...chip shortage around the world...

So tired of hearing companies throw this around as an excuse for price gouging and complacency. Global chip production was back at pre-covid levels in April of 2021, 1 YEAR AGO now.

There is only so long NVIDIA can charge tripple price for cards and keep supply scarce while blaming covid and chip shortages that don't exist before some regulatory body has to step in and investigate them for antitrust and fraud.

At some point you really need to drop the excuses and just get the damn job done, and just sell a good product at good prices and make good profits and stop obsessing over needing to make more than last month every month forever which is impossible.

But the world didn't need to contract covid to lose their fracking minds.

Arthtur

I love this thread. Keep going
Its a lot better to read about how servers are just bad and not how we have to remove half of the game in order to stop lag.

*brings more popcorn*

Jaraal

Arthtur wrote: »

I love this thread. Keep going
Its a lot better to read about how servers are just bad and not how we have to remove half of the game in order to stop lag.

*brings more popcorn*

Yeah the behind the scenes details don't really matter. Trying to log in to a broken, dysfunctional game tells us all we really need to know.

Cuddlypuff

tsaescishoeshiner wrote: »

I love threads like this because you really get a sense of being exposed to knowledge—without actually learning anything!

Highly qualified business and server experts are wielding their jargonic buzzwords against OTHER experts—and they're all coming to different conclusions.

Everyone poses as an expert with insider knowledge and touts universal business principles, without knowing what's actually going on inside the company they're talking about. I wonder if these forum experts have strangers posting online about their business and how it supposedly operates. *eats more popcorn*

I don't recommend using internet forums as your source of learning

LalMirchi

"There is no try only do"

[snip]
[edited for bashing]

ADarklore

LalMirchi wrote: »

"There is no try only do"

[snip]

Yes, in the current world where nobody wants to work. Almost every place around here is HIRING but nobody wants to actually work. I work in a factory and we constantly have to shut machines down due to employee shortage (no new hires coming in, people constantly calling in but we can't fire them because we need them when they DO show up, etc).

[edited to remove quote]

ADarklore

It's interesting how people don't want to accept the chip and other component shortages yet, car manufacturers are shutting down lines due to shortage, game consoles cannot make enough for demand due to chip shortage... BUT... ESO should certainly be able to get some because their more important.