Why was the mega server running better during beta?

Baphomet · June 2020

This is not a rant - I am genuinely curious to find out!

Sure, during beta we had the occational crash, fps drop etc. but overall the server was running much, much better.

We actually had 100 v 100 v 100 combat scenarios without server lag or delay - so what happened?

Now the server is struggling as soon there are five players in the same vicinity, and there is lag, delay, ruber-banding, disconnects etc.

So what killed the server?

Is it all the new PvE zones which have accumulated over the years, is it all the inventories the game has to keep track off, is it the collision system or something else entirely?

Does anyone have any real idea what could kill the server performance to such an extend we see today when it once was capable of handling five times the max population we see today?

redgreensunset · June 2020

Fewer players over all?

Like Cyrodiil might have had more players and bigger battles, but on the whole the servers had less players and therefore less strain on thr code and hardware. We've all seen how all zones, not just Cyro, suffers when put under strain and how the whole game suffers when there are too many players on at once (points to PC/EU).
In short, the game gpt more players when released. Then more and even more later, straining a code and hardware that can't handle it.

Can it be fixed? Possibly. But it will take major recoding and since they're noy willing to take the game down while they do it but insist on doing it to a live game (a Suit decision I suspect, I doubt any dev would do it this way if given a choice) it is goingto be an incredibly bumpy ride to get there. If the fixes does what they're meant to at all, something for which we have no guarantee.

Septimus_Magna · June 2020

In the beginning people had basic gear (no proc sets), barely any passives, no champion points and most (like myself) had no clue how to play yet so low APM. The performance was really good a while after launch, probably due to a combination of these factors. After the infamous lightning patch more tasks were moved to the server side to decrease the amount of bots/cheaters and this was very noticeable in latency and responsiveness.

Kadoin · June 2020

We don't talk about that...

Elsonso · June 2020

In general, closed beta testing is useless for validating performance.

Joy_Division · June 2020

Elsonso wrote: »

In general, closed beta testing is useless for validating performance.

Pretty sure the OP means the public sessions.

There is a bit of nostalgia with the Beta with some folks claiming that performance was ideal, but I watched a video I made not too long ago and there were definitely server hiccups and sometimes skill delays in Cyrodiil.

That being said, there was probably more than ten times the amount of people logged on in Cyrodiil and the performance was more consistent, stable, and all around better.

You're not going to find out from ZOS, who has never been forthright about the pop cap, nor the changes they do make (the infamous Lighting Patch and the crashing "fix" from a few months ago to be specific) and you're not going to find out on the forums where various overly reductionist hypotheses have abounded and changed over the years: I still remember when many were convinced the meteor ultimate was the main contributor.

I'd bet real money the issue boils down to a lot more than just the commonly touted culprit of server calculations. A contributing factor, sure, but probably just that. It's a little too convenient on ZOS's end in that it insinuates the problem isn;t so much on their end or with their servers, rather the sheer amount of computations required to have an interesting game. When ZOS did their no CP experiment a few years back, the performance on the PvP server that everyone plays on was just as crappy, and when I'm in an PvE instance like Maelstrom or DSA, I can predict exactly when performance will go to crap, even when I'm not using any skills. It also does not explain the immediate and severe deterioration we have all seen in updates such as the Lighting Patch and the recent mass crashing "fix".

mav1234 · June 2020

I have no doubt some deal of performance problems do relate actually to the calculations. While the populations are a rough estimate for how laggy a campaign is, there is a definitely an effect of more people in one location, especially when those people are stacking a lot of hots/other buffs on themselves.

but a lot of the performance problems are basically stacked on top of each other,a nd it is hard to know what is the problem. pvp, even without lag, has ability failure/lag on a consistent basis.

and the whole "I wish we could fix performance" feels hollow when it is followed by "Servers are doing great

"

brandonv516 · June 2020

They broke their game and don't know how to fix it. That's the simple answer.

Goregrinder · June 2020

Is this a serious question?

Moloch1514 · June 2020

brandonv516 wrote: »

They broke their game and don't know how to fix it. That's the simple answer.

Truth. Anyone here remember the patch note explaining all deer in Cyrodiil were removed to improve performance. That was after everything went server side and they have been trying whatever they can lol.

kaithuzar · June 2020

I speculate it’s possible they added more virtual machines without increasing the hardware capacity. I would imagine when they brought out player housing more VM’s were needed and the same for new dungeons a new zones. From what I recall I’ve only ever heard of one hardware upgrade being done in a patch and I think that was two or three years after hearing they were downgraded.

If it were me, every new DLC or expansion should have estimated hardware requirements for that content so the hardware should be upgraded each time.

If not then your capacity and planning team should estimate the required or upgraded hardware needed to assist for accounting for all patches in a given fiscal year.

The end result of not accounting for these things and just pushing content means your infrastructure isn’t scalable and so we have crap performance.
Just opening your wallet and buying server grade SSD’s could be a huge push in the right direction.

smee_z · June 2020

Simply put.

In the current version, the server side is the bottle-neck. Game client now is just a dummy doing no calculations and just renders the game.

In the release version (beta), the server side was not the bottle-neck at all as calculations are shared and off-loaded to the game client.

Curtdogg47 · June 2020

I wonder if it would help if they had Separate PVE and PVP version of Cyrodiil? At least for PVP lag and issues.

kaithuzar · June 2020

I’ve heard rumors that some other games don’t have the server keep track of the players location, which sounds like a bad thing but it seems to save a lot of server resources. Another MMO that I play got away with doing many things client side just like ESO did originally. The difference is, they paid for a rather expensive Anti-cheat Technology to hide their client side issues and started banning people if the protection mechanism was triggered.

It seems to have paid off for the other company and might be worth looking into.

Neloth · June 2020

smee_z wrote: »

Simply put.

In the current version, the server side is the bottle-neck. Game client now is just a dummy doing no calculations and just renders the game.

In the release version (beta), the server side was not the bottle-neck at all as calculations are shared and off-loaded to the game client.

This, at some point instead of investing into anti-cheat system they just decided to move most calculations to the server side, and that was the starting point of performance degradation.

This primary happened because they never test new features in real-world scenarios - PTS servers have very low population, so they couldn’t realize drawbacks of new system in time, since on paper (and on PTS) it worked well.

In addition, they add more complex computational logic each patch (like new vamp or NB stun, when server has to check the direction of each affected player, new sets, etc), and it further bottlenecks the performance when large fights occur.

Adding new hardware won’t change the situation a lot: sometimes, we have 4-5 second of server latency, which means they need to speed up server computations in 10-20 times. If their scalability is perfect (which I doubt), it means that they have to expand their PvP server infrastructure in the same amount of times, and it’s too expensive.

WuffyCerulei · June 2020

I’m pretty sure it’s the server itself having issues with all the new players, and just Cyrodiil not getting some TLC. ESO having fresh content every year isn’t the issue.
Also I’m pretty sure that in late 2013 to early 2014, there weren’t nearly as many players. Like a fraction. Wondering why a 6+ year-old overrun megaserver isn’t running as smoothly as it did at the beginning with a smaller population is like wondering why the old 3-person canoe is tipping over when 10 people just suddenly got aboard.

Neloth · June 2020

WuffyCerulei wrote: »

I’m pretty sure it’s the server itself having issues with all the new players, and just Cyrodiil not getting some TLC. ESO having fresh content every year isn’t the issue.
Also I’m pretty sure that in late 2013 to early 2014, there weren’t nearly as many players. Like a fraction. Wondering why a 6+ year-old overrun megaserver isn’t running as smoothly as it did at the beginning with a smaller population is like wondering why the old 3-person canoe is tipping over when 10 people just suddenly got aboard.

I don't think that new players can significantly affect the performance.

First, the largest part of eso player base is formed from casual players, doing overland quests. Compare the amount of server computations required when a player hits a wolf in PvE zone, and when players casts northern storm in the middle of enemy zerg. Im pretty sure that the first one has linear complexity, while second one includes processing many nested loops, and that's the reason why we have lag in Cyro/trials, but don't have it in overland PvE.

Second, with the chapter release a lot of people come back to the game, the player base increases. By your logic it should reduce Cyrodiil performance, but in fact it is as usual improved compared to days before update, since many PvP players are still farming gear and not logging into Cyro. But when more players return back to Cyro (especially organized guilds), it will be as bad as usual.

JTorus · June 2020

Baphomet wrote: »

Is it all the new PvE zones which have accumulated over the years, is it all the inventories the game has to keep track off, is it the collision system or something else entirely?

12.19.2013
https://gameranx.com/updates/id/19480/article/the-elder-scrolls-online-pvp-map-size-factions-and-guilds-explained/

"Having worked on games with player collision in PVP before, we found that it was far more valuable to offer players larger battles with better server and client performance than smaller battles with player collision due to the hit servers and clients take when implementing such a feature."

Collision was initially off for pvp.

Elsonso · June 2020

Neloth wrote: »

This primary happened because they never test new features in real-world scenarios - PTS servers have very low population, so they couldn’t realize drawbacks of new system in time, since on paper (and on PTS) it worked well.

They have actually said similar on a couple of occasions. The conclusion is that the only place they can test some of the performance stuff is on Live because the conditions don't exist anywhere else.

kaithuzar · June 2020

Elsonso wrote: »

Neloth wrote: »

This primary happened because they never test new features in real-world scenarios - PTS servers have very low population, so they couldn’t realize drawbacks of new system in time, since on paper (and on PTS) it worked well.

They have actually said similar on a couple of occasions. The conclusion is that the only place they can test some of the performance stuff is on Live because the conditions don't exist anywhere else.

InaMoonlight · June 2020

Less players ^^

Vaoh · June 2020

Like everyone else has said, the biggest reason that performance tanked in the past is because more tasks were moved onto the server (away from the client).

HOWEVER

Game performance on my PS4 Pro has drastically degraded every year since Morrowind launched. They also:
• Erased all allied non-synergy AoE ability effects
• Erased a large number of audio from sources not directly on your screen
• Introduced countless new bugs which greatly hamper combat gameplay fluidity
• Reduced Draw Distance
• Loading takes longer, textures load slower or simply fail to load, other players who load are “un-loaded” when you move too far or look away

In other words, the game has drastically deteriorated due to neglect by the developers. ESO felt great in Homestead, yet it feels like an absolute mess in Harrowstorm (and soon Greymoor). Moving checks server-side happened many years prior to Homestead.

Mortiis13 · June 2020

Moloch1514 wrote: »

brandonv516 wrote: »

They broke their game and don't know how to fix it. That's the simple answer.

Truth. Anyone here remember the patch note explaining all deer in Cyrodiil were removed to improve performance. That was after everything went server side and they have been trying whatever they can lol.

Oh dear., XD

Alomar · June 2020

It's a whole list of complications. Not upgrading server hardware, the removal of many calculations being client-side and moving them server side, the 1.2 lighting patch, layers upon layers of horrible jumbled-up coding by amateur developers post-launch compared to the ones who launched the game, never getting balance and ttk right with horrible flip flopping balancing, etc.

Sephyr · June 2020

Moloch1514 wrote: »

brandonv516 wrote: »

They broke their game and don't know how to fix it. That's the simple answer.

Truth. Anyone here remember the patch note explaining all deer in Cyrodiil were removed to improve performance. That was after everything went server side and they have been trying whatever they can lol.

RIP all those deer too. They did something similar to the butterflies near Northpoint in Rivenspire as well. Totally nuked those completely.

Elsonso · June 2020

Moloch1514 wrote: »

brandonv516 wrote: »

They broke their game and don't know how to fix it. That's the simple answer.

Truth. Anyone here remember the patch note explaining all deer in Cyrodiil were removed to improve performance. That was after everything went server side and they have been trying whatever they can lol.

It is one conclusion, but I do not think that is the truth. They have had years to study the Cyrodiil performance problem, and I think they have known for years what it is going to take to fix it. I think the fix is huge. I think the fix is to a core flaw in the game design. I think they know how to fix it, but it is akin to replacing the foundation of a sinking skyscraper. Expensive and risky.

They might be working on that major fix. It would take time, and might have many parts, and we might not see it for years, but my gut says this is not what they are doing. I think they are running about trying everything except for that fix in an attempt to "make the patient comfortable" until the end. When the list of things to look at gets all the way down to something like auditing combat skills to identify server performance problems, and you are trying to fix crippling combat problems in Cyrodiil, that is a major red flag.

This is not all bad, as everyone outside of Cyrodiil can see incremental performance increases as they come across and fix a number of things. Greymoor cross-zone and intra-zone loading screens are shorter than I have seen them in a long time.