Why can't ZOS simply throw more servers to the cluster to handle the load?

mikeabboudb14_ESO · February 2017

the lag /low framerate is making me sick ive set everything to low and the framerate is still coming in low teens even in to the single digits in large cities it need to be fixed or ill need to stop paying for my sub.

akl77 · February 2017

lordrichter wrote: »

Pibbles wrote: »

Xundiin wrote: »

As I was trying to make him understand, though I'm convinced that he can't understand this for fear that his piece of paper means little to nothing because of it, is you if you have the passion to learn the field, you don't need to have the job to have the knowledge. All the problems that can cause issues with servers and networking is well documented in various sources of information. Information that most techs that aren't conceded will use as a reference to help them do their job. But again... he can't see the difference. To him knowledge = skill.

Knowledge alone can only take you so far, you can only become a skilled practitioner of anything by application of knowledge in a real world environment.

For example, if I say to you that I have a desktop running an application that is performing badly, what are your first 3 troubleshooting steps?

You only need one. Switch from Windows to Linux.

Yea.

josh.lackey_ESO · February 2017

Pibbles wrote: »

TequilaFire wrote: »

cjthibs wrote: »

TequilaFire wrote: »

Let's also not forget about poorly optimized client side code either and flat out bugs like memory leaks.
After all this time you would think game devs could take advantage of multicore processing client side as well.
Servers are an issue but it takes two to tango.

When ESO was coded multicore gaming was still very much in its infancy.

But multicore computing was not, also the 64 bit client got a workover.

I can't get away coding old school in my industry, no excuse for game development.

A significant issue is the API they are coding for. DirectX, up to and including 11, uses a single-threaded renderer. Therefore you will see that most games have a 'main thread' that is responsible for squirting stuff into the renderer. If you run up FF14 you'll see this, ESO, any number of games. There is little point having a multi-threaded process in the app delivering data for processing by a single-threaded procces. Some people have tried to get around this with buffers but they still cannot overcome this inherent limitation of a single-threaded renderer. Trion famously multi-threaded Rift by adding a load of buffers and after all the work, people said there was little improvement.

Now, Vulkan and DX12 are multi-threaded renderers and it really makes a difference. I run the Vulkan version of Doom on my 12 core PC and it flies along at 120FPS with everything on, and only loads each core by about 20%.

So once ESO move to DX12 we'll open the door to improvement in the client, but they still need to recode the actual game, because in that scenario the multi-threaded API will be sat waiting for the single-threaded process to give it something to do, until such time as they increase the bandwidth in the pipeline process feeding the renderer. So, don't hold your breath ;-)

This is accurate, but it not why there is server lag.

raglau · February 2017

josh.lackey_ESO wrote: »

Pibbles wrote: »

TequilaFire wrote: »

cjthibs wrote: »

TequilaFire wrote: »

Let's also not forget about poorly optimized client side code either and flat out bugs like memory leaks.
After all this time you would think game devs could take advantage of multicore processing client side as well.
Servers are an issue but it takes two to tango.

When ESO was coded multicore gaming was still very much in its infancy.

But multicore computing was not, also the 64 bit client got a workover.

I can't get away coding old school in my industry, no excuse for game development.

A significant issue is the API they are coding for. DirectX, up to and including 11, uses a single-threaded renderer. Therefore you will see that most games have a 'main thread' that is responsible for squirting stuff into the renderer. If you run up FF14 you'll see this, ESO, any number of games. There is little point having a multi-threaded process in the app delivering data for processing by a single-threaded procces. Some people have tried to get around this with buffers but they still cannot overcome this inherent limitation of a single-threaded renderer. Trion famously multi-threaded Rift by adding a load of buffers and after all the work, people said there was little improvement.

Now, Vulkan and DX12 are multi-threaded renderers and it really makes a difference. I run the Vulkan version of Doom on my 12 core PC and it flies along at 120FPS with everything on, and only loads each core by about 20%.

So once ESO move to DX12 we'll open the door to improvement in the client, but they still need to recode the actual game, because in that scenario the multi-threaded API will be sat waiting for the single-threaded process to give it something to do, until such time as they increase the bandwidth in the pipeline process feeding the renderer. So, don't hold your breath ;-)

This is accurate, but it not why there is server lag.

I agree. Someone was asking about why they don't just go full multithreaded, and I was covering off the basics. Even so, whilst multithreading will help some things (game client is most definitely CPU bound), it won't deal with any back-end issues, and these do exist IME. I did some detailed Perform analysis of ESO and whilst a lot of the trouble is caused by being CPU bound, there are also performance anomalies that can only be explained by back-end issues.

dvstansb14_ESO · February 2017

Pibbles wrote: »

ESOUser0x00 wrote: »

I'm not familiar with MMORPG tech but for cloud to scale up, you simply throw more machine resources.

Not quite that simple. There are two approaches to scaling IT services:

Scale up - i.e. add more resource too each node, such as more CPUs, more RAM, faster hard drives, more NICs etc
Scale out - i.e. add more nodes to the cluster/load-balanced array

Most services use a combination of the approaches but crucially, each has laws of diminishing returns depending on the software architecture. Some services, say Exchange Server, Active Directory and SQL Server, scale out excellently, in some cases hundreds of nodes can be uniformly load balanced automatically. Other services, e.g. vanilla LDAP directories, scale out badly and should be scaled up, but you then reach physical limitations of the host, e.g. perhaps it can only hold 32 CPUs or something. Some hardware architectures scale up better - NUMA for example - but the hosted service must run NUMA aware code (such as Exchange or SQL) or that service will fail to scale with it, in fact it may perform worse the more CPUs you add.

Now, we don't know how ZOS have done this but let's be honest, they're not really on the ball in terms of modernity of their code, so we can easily envisage a case where they're in a corner due to weak architecture decisions early in development and therefore cannot easily take advantage of hardware scalability.

Good answer! I've been building HPC/distributed systems for years, and the short answer is: it's difficult, expensive, and requires very specialized skills. Alas, if ZOS hires Cray or IBM to solve their problems, I think the subscription price might go up a bit (as in add a few zeros to the price).

Elsonso · February 2017

dvstansb14_ESO wrote: »

Good answer! I've been building HPC/distributed systems for years, and the short answer is: it's difficult, expensive, and requires very specialized skills.

I get the impression that ZOS is their own expert on this subject.

dvstansb14_ESO · February 2017

Balamoor wrote: »

t3hdubzy wrote: »

Its about game design, no amount of extra servers is going to solve this problem because the way the game was designed, the calculations, and the amount of players interacting at one time unless you are suggesting that everyone gets their own personal sever like the days of ultima online where everyone had their turnkey.

This is unmitigated horse crap. the lag is due to a combination of network transport latency and server processing lag not game design. What needs to be addressed are sever and network load balancing and it isn't as easy as it sounds, I know because before I retired I was a Consulting Engineer contracted to Digital Reality. We provided server systems for many online games.

Also if these Data centers are anywhere near North Dallas there is a very good chance they are on Century link which has the single worst infrastructure in the country, also those data centers won't be changing anytime soon, so the best we can hope for is a new contract with new data centers which brings its own stabbing your eye out with a fork issues.

If you people are going to ***, *** about the right things not something as idiotic as bad design.

While no one knows what the actual underlying problems with ESO really are, I would like to point out that it is possible for the game design to be a (significant) part of the problem. An MMO can be thought of as any distributed application - my expertise is in distributed (HPC) scientific applications. There are many software factors - including design and resulting communication patterns - that can affect how well an application scales up or out on available hardware. Many of the scaling bottlenecks are not intuitive and to get around them (if possible) can take writing application-layer code that is hardware/network aware. In short, it's not easy - especially for developers not experienced in this area. And yes, you can code yourself into a corner, so to speak, as I've done it myself and had to go back and refactor major chunks of an architecture that looked great on the whiteboard.