I think ESO has grown too big for a single mega server anymore.

Foto1

Tammany wrote: »

Due to russian localisation there will be million new players.

you exaggerate a lot

Tapio75

NoNameNamer wrote: »

There's only so many design & code optimizations and hardware upgrades you can do. At some point you reach the theoretical limit on how much load a server can bare. WoW went through a lot of changes to minimize server loads (remember when they pruned most of the combat logging?) but even then they had to implement sharding. Players in the same area are phased into multiple instances to balance the load. This means they would not always see each other despite being in the same location unless they form a party which pulls everyone in the same phase/instance. This allowed them to completely get rid of login queues. Obviously, they couldn't bring this tech to Classic WoW so they had to preserve the login queues for those servers.

That was not done to help with server loads. It was to help "Empty" realms have more population and to make people on the same realm be in different phases was originally for the quests to have more impact on game world which it did like it does in ESO.

In ESO though things are just bigger, but the problems megaserver system does, are more community related just like in WoW rather than performance issues, but the fact that there are so many players makes it problematic to have system such as rthis. It makes performance issues when too many players are allowed in same instance for example.

Most problems thpough, are community based. There can be no real community, when people are randomly put on separate instances of a wrold.

I woukld rather see less player but know the server community than be placed with random differet players each time i log in. Even guildies are in different instances which is a nuisance and make people to switch instance all the time when grouping up. Also removes a chance to randomly see guildies on your travels or to have possibility to RP with same players .

Anotherone773

fred4 wrote: »

As far as I understand their setup, it is NOT a single server. What they call a megaserver is in reality a server farm. Whenever you change zone the client makes a new connection to a different IP-address - a different server. In order to spread the load there are, as we all know, multiple instances of highly populated zones, such as the ones containing the big trading hubs. Each is handled by a different server or, at the very least, different threads of high core-count CPUs.

Basically. It will be a group of blade servers that work together. You are passed between the servers depending on what you are doing. Servers have different functions such as one for login only. A database server Cyro and probably BGs will be together on a server. Etc etc. Blade servers are the standard as they are easy to upgrade and and change depending on ongoing needs. Each blade can be customized like a computer to provide a certain function.

The problem is that servers require special rooms that are clean rooms and climate controlled. Also has to be X distance between enclosures so physical space is a limiting factor. Servers are expensive to upkeep, far more expensive than most people think. One megaserver and its support systems( IE: HVAC) can use as much electric as a small town.

Dr_Sinister

HaruKamui wrote: »

walterhwilliamsb14_ESO wrote: »

I am an ESO plus and paying good money and I can not get on. That is stealing my money.

have you been to a theme park and had to wait in line before?

I've never had to wait 24 hours to get on a ride that someone who didn't pay to get in got on for free

Dr_Sinister

I personally think they do need more servers but not for the reasons the OP stated. I would like to see a possible first person only server. I think this would make things a lot more hardcore and challenging. I've done a lot of dungeon runs in first person and they are fun as hell but I know for a fact if I had mentioned that in my group I would have been kicked because, elitism

MartiniDaniels

Anotherone773 wrote: »

fred4 wrote: »

As far as I understand their setup, it is NOT a single server. What they call a megaserver is in reality a server farm. Whenever you change zone the client makes a new connection to a different IP-address - a different server. In order to spread the load there are, as we all know, multiple instances of highly populated zones, such as the ones containing the big trading hubs. Each is handled by a different server or, at the very least, different threads of high core-count CPUs.

Basically. It will be a group of blade servers that work together. You are passed between the servers depending on what you are doing. Servers have different functions such as one for login only. A database server Cyro and probably BGs will be together on a server. Etc etc. Blade servers are the standard as they are easy to upgrade and and change depending on ongoing needs. Each blade can be customized like a computer to provide a certain function.

The problem is that servers require special rooms that are clean rooms and climate controlled. Also has to be X distance between enclosures so physical space is a limiting factor. Servers are expensive to upkeep, far more expensive than most people think. One megaserver and its support systems( IE: HVAC) can use as much electric as a small town.

We don't know how big ZOS "megaserver" is. But fact is that IT companies, with much larger number of clients/subscribers somehow manage to handle necessary capacity and expand it to avoid "queues". Lack of space in equipment room is not an excuse. Problem doesn't appeared yesterday, problem was acknowledged long ago and they have plenty of time to upgrade current megaservers or create a new one.
Hell, we don't even need to go out of gaming industry. There are dozens of games with hundreds thousands of simultaneously active users, created by companies much less then Zenimax, with no subscription pay and somehow they manage to organize their work properly.

kojou

It seems like an East Asia/Australia/NZ server would be worth it at this point, but I would miss all my friends on that side of the globe, so I have mixed feelings about it.

Coppes

Dr_Sinister wrote: »

HaruKamui wrote: »

walterhwilliamsb14_ESO wrote: »

I am an ESO plus and paying good money and I can not get on. That is stealing my money.

have you been to a theme park and had to wait in line before?

I've never had to wait 24 hours to get on a ride that someone who didn't pay to get in got on for free

But everyone pays to get in...

twev

volkeswagon wrote: »

just a more powerful server

Maybe, sure, that might be enough, possibly.

But we don't know that the game engine itself, as implemented, could handle more connections from players, either.

Dr_Sinister

Coppes wrote: »

Dr_Sinister wrote: »

HaruKamui wrote: »

walterhwilliamsb14_ESO wrote: »

I am an ESO plus and paying good money and I can not get on. That is stealing my money.

have you been to a theme park and had to wait in line before?

I've never had to wait 24 hours to get on a ride that someone who didn't pay to get in got on for free

But everyone pays to get in...

No, some people have been playing the game in the new area without having to wait in queue or even pay for the expansion/chapter. Yes they are working on it but that wasn't part of the context

MyKillv2.0

The game was about as poor of a pre-conceived game as I can ever remember. Almost every choice that they made has been proven to be a 'bad choice' and they have had to almost rebuild the game from the ground up. To their credit (and personally, I give them a MASSIVE amount of credit for their persistence to improve the product) they have turned the game around into being one of the top second tier MMOs on the market. However some mistakes you just cannot overcome and I am afraid that the one server per region & platform/mega server choice is one of them. Maybe I am wrong, not an IT guy. Maybe someone who has excellent game coding knowledge can speak on the issue better than me. I hope I am wrong but I have to believe that almost every other MMO on the market has a reason for having multiple servers per region.

mairwen85

MyKillv2.0 wrote: »

The game was about as poor of a pre-conceived game as I can ever remember. Almost every choice that they made has been proven to be a 'bad choice' and they have had to almost rebuild the game from the ground up. To their credit (and personally, I give them a MASSIVE amount of credit for their persistence to improve the product) they have turned the game around into being one of the top second tier MMOs on the market. However some mistakes you just cannot overcome and I am afraid that the one server per region & platform/mega server choice is one of them. Maybe I am wrong, not an IT guy. Maybe someone who has excellent game coding knowledge can speak on the issue better than me. I hope I am wrong but I have to believe that almost every other MMO on the market has a reason for having multiple servers per region.

They have multiple servers per region. The term megaserver refers to multiple servers working as one.

Several posts refer to this in this thread. (one of mine on previous page explain in simple terms).

Elsonso

MartiniDaniels wrote: »

We don't know how big ZOS "megaserver" is. But fact is that IT companies, with much larger number of clients/subscribers somehow manage to handle necessary capacity and expand it to avoid "queues". Lack of space in equipment room is not an excuse. Problem doesn't appeared yesterday, problem was acknowledged long ago and they have plenty of time to upgrade current megaservers or create a new one.

We have seen pictures of the PC EU megaserver. I wish they would update those pictures, as they are 6 years old now. Based on those pictures, we know (knew) what hardware platform they were on. We know they are in a data center. We know from job postings what OS they run. We know from multiple points of data that they do maintain these systems, and even add server hardware to them. This is not a fly-by-night system running in the corner of Lambert's office where he can use the CD tray as a glass holder and put his feet on at the end of the day.

ItsJustHashtag

I’ve heard they’ve reduced server capacity over the years to save money.

xeNNNNN

Aptonoth wrote: »

I hate to say it but these huge wait ques and long load times are a sign even the American server is no longer enough. I think ESO needs another NA server and another EA server as well. If ESO wants to keep growing I think this is the only viable way now as annoying as it will be for them its for the best of their games and player base future it can't grow if there's no space left on any of the servers.

Also congrats ESO and Zenimax for making a great game. So great it needs more servers lol.

The route cause and many of us have been saying this for years now is not capacity its transmission and flow of data. There is a very clear and rather large data bottleneck in their servers anyone with a tech background can see that just by the performance of the last 5 years. The processing power of their hardware leaves much to be desired AND the location of the server for EU happens to be in Frankfurt, the worst location possible for a Multiplayer game in Europe. They SHOULD transfer the game to a data centre in the Netherlands but that costs "money" and we all know how ZoS feels about money.

Fengrush actually said a really good line when reading the patch notes: "You can only put so much Data (water) into a pipe till its at max" (not an exact quote but close enough - i am not going through that 6 hour video to find one line lol)

To further this statement, if the pipe is angled in too many directions on its way to 1 direction then it will take longer to reach its destination. The same applies to the servers, I did a trace route and my connection to the game (I tested other MMOs as well) bounced from London to Portugal then to France and then to Poland and finally Germany instead of at the very lead just connecting to Belgium first and then to Germany. This is not a user end thing... this is the server redirecting traffic for one reason or another...whether it be intentionally or unintentionally this is hindering the servers for some people (nobody has the same route at all times). Basically my connection to the game for some absurd reason goes south first then goes north to go east its odd as hell.

fred4

NoNameNamer wrote: »

There's only so many design & code optimizations and hardware upgrades you can do.

I understand that some people experience lag in trials now. I cannot believe this is due to anything but underinvestment in server capacity, as that is and always was just 12 people per instance. Same with BGs.

The only problem that may be hard to solve is Cyrodiil. Something caught my eye about that the other day. A ZOS developer comment mentioned adjustments to armor set scripting. That word implies there is server side code written in Lua. It's hard to say, but an interpreted language tends to be an order of magnitude slower than C/C++, which is what the core of the game engine is probably written in. Even if Lua is JIT compiled, and apparently Lua is good in this regard, there is still the possibility that a 2x to 3x performance improvement could be had for current Lua code by recoding some functionality in C/C++. I may well be wrong, but I cringed when they mentioned server-side scripting.

fred4

xeNNNNN wrote: »

Aptonoth wrote: »

I hate to say it but these huge wait ques and long load times are a sign even the American server is no longer enough. I think ESO needs another NA server and another EA server as well. If ESO wants to keep growing I think this is the only viable way now as annoying as it will be for them its for the best of their games and player base future it can't grow if there's no space left on any of the servers.

Also congrats ESO and Zenimax for making a great game. So great it needs more servers lol.

The route cause and many of us have been saying this for years now is not capacity its transmission and flow of data. There is a very clear and rather large data bottleneck in their servers anyone with a tech background can see that just by the performance of the last 5 years. The processing power of their hardware leaves much to be desired AND the location of the server for EU happens to be in Frankfurt, the worst location possible for a Multiplayer game in Europe. They SHOULD transfer the game to a data centre in the Netherlands but that costs "money" and we all know how ZoS feels about money.

Fengrush actually said a really good line when reading the patch notes: "You can only put so much Data (water) into a pipe till its at max" (not an exact quote but close enough - i am not going through that 6 hour video to find one line lol)

To further this statement, if the pipe is angled in too many directions on its way to 1 direction then it will take longer to reach its destination. The same applies to the servers, I did a trace route and my connection to the game (I tested other MMOs as well) bounced from London to Portugal then to France and then to Poland and finally Germany instead of at the very lead just connecting to Belgium first and then to Germany. This is not a user end thing... this is the server redirecting traffic for one reason or another...whether it be intentionally or unintentionally this is hindering the servers for some people (nobody has the same route at all times). Basically my connection to the game for some absurd reason goes south first then goes north to go east its odd as hell.

I am located in Ireland. I play on PC EU and on PC NA. PC NA consistently has 200+ ping. That can cause issues, but they are few and far between, compared to the consistent feeling of molasses and dropping of skill casts when Cyrodiil on either platform is full. The transatlantic lag I experience when playing on NA is inconsistent, but is at this stage by far the lesseer problem. This leads me to believe the servers and/or ZOS internal network are at fault.

The network traffic between the game client and the servers is really low - tiny! - for one individual player. It's possible that, by the time that hits Frankfurt and/or ZOS internal networks, it adds up to cause a congested network connection somewhere. If that was the case, though, then everyone would have the same severity of problems at the same time, throughout the game. This is not the case. When Cyrodiil is unplayable, I go to IC and, thus far - touch wood, IC has always been fine.

TL;DR: Network issues / bad routing can contribute, but are not the main issue.

validifyedneb18_ESO

I think you missunderstand how a "mega server" works.

scorpius2k1

Aptonoth wrote: »

I hate to say it but these huge wait ques and long load times are a sign even the American server is no longer enough. I think ESO needs another NA server and another EA server as well. If ESO wants to keep growing I think this is the only viable way now as annoying as it will be for them its for the best of their games and player base future it can't grow if there's no space left on any of the servers.

Also congrats ESO and Zenimax for making a great game. So great it needs more servers lol.

6+ years on and it's only getting worse. I think ESO has outgrown it's current development team if you want an honest opinion about it. That, or the game/server engine/infrastructure was horribly undershot in it's design.

On the other side of the coin, I do agree that several instances aka "shards" in other MMO's is the proper way to fix so many problems. ESO isn't a special exception to the rule that every other MMO has had to follow, eventually they will realize this. Hopefully it's not too late though.

Pinkes

Some other games let you choose your instance, so can you decide for yourself if you want to play in an instance with a lot of people or not. This game has never been properly optimized, there is basic stuff that still works poorly since launch ( group finder ahem). I like to play cyrodill a lot, but its almost always lagging, so much as in unable to play.
I spent a lot of money on this gameand i like it, but I'm getting tired of always having to deal with the same stuff over and over again.

Megatto

I've been saying this for a while. Give a couple of servers to Asia and one for Australia

TropicsDelight

I can fathom WHY people get ****hurt by these warnings (etc.), I would value such an asset worth twice its weight in gold.

Suits at the ground level who manage technical staff work under higher level suits.

Higher level suits in every business are normally focused on cost reductions in their departments and their own bonuses normally directly correlate to those cost savings. Proving increased spending resulted in increased revenue is very hard and often suits get personally finacially burned for spending more money even if it was the right move.

Better server performance is hard to quantify as are player number increases resulting directly from performance enhacements instead of say an expansion or covid 19, ect..

A suits most critical focus is their yearly performance review and resulting bonus or raise/promotion. Their own paycheck/net worth/retirement bank account will always take priority. As such they are pressured to not spend money, even at the expense of lost revenue potential.

I have watched that game in multi-billion dollar businesses for a long time..

redgreensunset

Dr_Sinister wrote: »

I personally think they do need more servers but not for the reasons the OP stated. I would like to see a possible first person only server. I think this would make things a lot more hardcore and challenging. I've done a lot of dungeon runs in first person and they are fun as hell but I know for a fact if I had mentioned that in my group I would have been kicked because, elitism

As long as you pull your weight and don't mess it up for the rest of us, I couldn't care less if you do the run standing on your head. Whatever floats your boat my dude.

Dr_Sinister

redgreensunset wrote: »

Dr_Sinister wrote: »

I personally think they do need more servers but not for the reasons the OP stated. I would like to see a possible first person only server. I think this would make things a lot more hardcore and challenging. I've done a lot of dungeon runs in first person and they are fun as hell but I know for a fact if I had mentioned that in my group I would have been kicked because, elitism

As long as you pull your weight and don't mess it up for the rest of us, I couldn't care less if you do the run standing on your head. Whatever floats your boat my dude.

Playing in first person doesn't affect rotation,dps, healing or tanking abilities. It may just make them a little harder as you now have to actually be aware of your surroundings without the use of the third person camera

daim

Eifleber wrote: »

Wasn't there rumours that there will be a Russian/cyrillic server?

Yes please! The cyrillic chat filter addon just quit working

Casterial

Spin up more hardware, more resources, and more engineers on the server.

fred4

Pinkes wrote: »

Some other games let you choose your instance, so can you decide for yourself if you want to play in an instance with a lot of people or not ... I like to play cyrodill a lot, but its almost always lagging, so much as in unable to play.

You can choose a low pop instance of Cyrodiil. It's called a campaign. If that's your preferred solution, you already have it, yet you chose going into the main (lagging) campaign anyway. The campaigns are all getting very full these days, so if you want to play no CP on PC EU then, sure, you have a point. We possibly need more Cyrodiil instances now. That hasn't been true historically, nor across all platforms, though.

To the many people who wrote something similar, e.g. the game would work better if there were more megaservers, I say this:

(1) Cyrodiil is possibly unfixable (or would take a large amount of work). There isn't another game designed around that many players on a single huge map.

(2) Every zone has multiple instances. Every trial, every dungeon is a separate instance. Every instance could run on a separate physical server. If people are having problems in those (yesterday it was Greenshade on PC EU), ZOS could easily throw more servers and/or enhanced network infrastructure at it within their two existing data centers.

(3) The only reason to have more megaservers would be to service geographical regions better, e.g. to improve lag experienced by players from Asia and Oceania.

Elsonso

fred4 wrote: »

(2) Every zone has multiple instances. Every trial, every dungeon is a separate instance. Every instance could run on a separate physical server. If people are having problems in those (yesterday it was Greenshade on PC EU), ZOS could easily throw more servers and/or enhanced network infrastructure at it within their two existing data centers.

To expand upon this... (note: massive speculation ahead)

My feeling is that they have a ton of "instance servers" for PVE content, but that each one handles more than one copy of a zone. Obviously, I don't know if each one handles the same number, or to what extent different content gets spread across different physical servers. For example, they can probably run a few more "houses" on a single physical server than they can zones.

Ultimately, I expect that each physical server can handle, approximately, the population of a single Cyrodiil campaign. How that breaks down to what we see in the game would be part of the magic that they don't talk about.

ZOS talks about something called a "Channel" (N Konkle interview), and this is what we call "the zone we are in" using common forum parlance. I don't think that each copy of the zone we are in is a separate "instance", rather there are one or more "instances" of a zone that host multiple channels for that zone. There could be several, even dozens, of these channels for a particular zone. I imagine that there could be multiple instances, and probably multiple physical servers, involved.

From the beta weekend (no NDA), I learned (and witnessed) that they can seamlessly merge these channels together, which implies that some, or all, may be co-located on a single physical server, separated by only a "channel ID". The purpose of the channels seems to be to separate players in the game to avoid the appearance of overcrowding and to limit the number of calculations and updates that have to be done whenever something changes. The latter thought is based on comments they made about Cyrodiil performance a few years ago in ESO Live streams talking about steps they were taking to address performance issues.

We don't see this work, and it is probably impossible catch it when it happens. I think that, as the population of a zone increases and decreases, I expect that these channels are managed so that, at any given time, the zone looks as populated as possible without going over, or going under, certain limits. This means that channels might be merged and split as needed to keep the channel population within the bounds they set. I doubt that the "last one out, turn off the lights" scenario happens based on the fact that I have never seen such a thing.

As server population increases, I would imagine they run into situations where there are not enough physical servers for that channel management to work properly. Channels get too large, and the worst case, there are no zone instances or channels to place someone. I see adding hardware as a way to allow additional player capacity so that the zones can be managed properly.

I am guessing that the dungeons, trials, and other "instanced" PVE content are handled much the same, but with much smaller channels and less channel management. Additional hardware allows more people to be in this instanced content.

Naturally, this is probably a lot more complicated than what I say. We have yet to talk about databases and database servers that store accounts, characters, inventory, guild banks, and all that. They probably have a ton of network attached storage, in addition to physical storage on each physical server. They might use a container system to quickly spin up virtual servers to act as on-demand "instance servers". I have no idea where ZOS stopped with flexibility and complexity.

emeraldshado

this really depends on what the term "mega server" is. is it an HP Elite desktop?

is it an HPE BladeSystem c7000

The enclosure is 10U high and holds up to 16 server and/or storage blades plus optional redundant network and storage interconnect modules. It includes a shared 7.1 Tbps high-speed NonStop mid-plane for wire-once connectivity of server blades to network and shared storage. Power is delivered through a pooled-power backplane, and power input flexibility is provided with choices of single-phase AC input, 3-phase AC input, -48V DC input, and high voltage DC input.


or is it a single HPDL380 G10 with 512G of RAM, 72 cpu, 2 physical, and 10TB of storage, though i find both of those options would be potential over kill.

but i'm not sure their Mega server is so mega.

They could run 10G nics to their demarcation point, so their auth servers and the game server, if not hosted on same host if vms, both would have 10G, but then the ISP would still be the network based bottle neck.

it really comes down to the definition of mega and the underlying infrastructure.

Alomar

Lol, lead devs admited they were sorely needing server improvements in April 2019.... None have happened, in fact there's speculation they reduced capacity after their very minor and didn't really do anything fps "performance improvement." The years have very clearly demonstrated where ZOS's values lie.