The reason why server performance is bad and a way to adress it

Willard · July 2019

Xerikten wrote: »

you forgot addons.

PS4 has no addons, yet it is plagued with performance issues.

zaria · July 2019

Nairinhe wrote: »

Move calculations to client, aaand... Cheaters, cheaters everywhere.

Ways to fix this with anti cheat programs, yes you need to maintain this but its also something who can be monetized. You can unlock the game again then caught cheating with buying an new version of ESO as an fine the first time, obviously the fine goes up every time you are caught. No cooldown or upper limit. say an 40-100% increase each time.
If unlocked you are an usual suspect who will be monitored way more both by AI and humans

And posters ideas would simply ruin the game, yes it will solve the performance issue very well as most will leave.
It looks however as the balance team has the same ideas.

Girl_Number8 · July 2019

Delparis wrote: »

ArenGesus wrote: »

So, basically you'd like for them to make the game super slow and basically suck instead of invest in hardware capable of running a good game. Seems legit. But no worries, because they aren't going to do either of those things.

Upgrading hardware cost money so I don't expect them to spend any penny on it.
However the calculation on the server side can be improved which will make the game tuns better.
Did anybody notice that when starting a trial before boss pull you got like +60 fps and 70 ms and the moment the fight starts you drop to 30 fps and the ping get higher? That's the server struggling to do all 12 players calculations.

You sound like a shill. They certainly do have the money to upgrade after all these years. That is the solution rather then your nonsense. Pffft.

Delpi · July 2019

Wolfpaw · July 2019

Girl_Number8 wrote: »

Delparis wrote: »

ArenGesus wrote: »

So, basically you'd like for them to make the game super slow and basically suck instead of invest in hardware capable of running a good game. Seems legit. But no worries, because they aren't going to do either of those things.

Upgrading hardware cost money so I don't expect them to spend any penny on it.
However the calculation on the server side can be improved which will make the game tuns better.
Did anybody notice that when starting a trial before boss pull you got like +60 fps and 70 ms and the moment the fight starts you drop to 30 fps and the ping get higher? That's the server struggling to do all 12 players calculations.

You sound like a shill. They certainly do have the money to upgrade after all these years. That is the solution rather then your nonsense. Pffft.

I would like zos to upgrade everything also, but does zos really have the say to make that financial investment?

SaucyMcSauceface · July 2019

Delparis wrote: »

ArenGesus wrote: »

So, basically you'd like for them to make the game super slow and basically suck instead of invest in hardware capable of running a good game. Seems legit. But no worries, because they aren't going to do either of those things.

Upgrading hardware cost money so I don't expect them to spend any penny on it.
However the calculation on the server side can be improved which will make the game tuns better.
Did anybody notice that when starting a trial before boss pull you got like +60 fps and 70 ms and the moment the fight starts you drop to 30 fps and the ping get higher? That's the server struggling to do all 12 players calculations.

I am in Australia, so fighting a ping of 250-280 with a tunneling service, or 300-400 without it, but that stays constant in trials, out of trials, in open world etc. The only places that majorly affect it are Cyro, and, strangely, Maelstrom Arena. In all of those though, my fps sits at between 80 and 144. If your FPS is dropping so much in trials, it may be your rig not the server. The fact that there is a drop in performance in Maelstrom suggests that the issue is not the server having to make multiple calculations for multiple people, but rather that it is an issue with the base code. If it were just the number of simultaneous calculations, a solo arena should have better performance than a trial, and yet Maelstrom has substantially worse performance than anywhere but Cyro.

Androconium · July 2019

A player with 810CP is carting around huge amounts of variables for their character, when compared with someone who is for example, at 180CP with relatively few achievements recorded across the board.

Then the obvious solution would be to move the long-term players "along" somehow. Either to another game (fallout76?); or just change things so that they become part of natural attrition...

Delparis · July 2019

or they will just move to another mmo if things don't get any better

https://www.youtube.com/watch?v=aIN8pKAPeew

CambionDaemon · July 2019

Didn't they admit years ago that the base code was a mess.

SweepsAllClowns · July 2019

Androconium wrote: »

A player with 810CP is carting around huge amounts of variables for their character, when compared with someone who is for example, at 180CP with relatively few achievements recorded across the board.

Then the obvious solution would be to move the long-term players "along" somehow. Either to another game (fallout76?); or just change things so that they become part of natural attrition...

I think I have the perfect solution for eliminating these problematic veteran players once and for all, I'm definitely one of those unwanted too high cp players having cp 1575, 2 Grand Overlord and other quite high level characters, it's clear that I've played too much over the past years and I'm the root of many problems so I must go. My game bounces back to character selection screen every time when I try to log in with my Xbox EU account, this started immediately after I downloaded Wrathstone so it's been stuck for exactly 4 months now, with this amazing feature I'm totally unable to cause any lag at all. Support told me they are aware and investigating the issue so many probably understand quite fast that all hope is gone with my account by now, I've even closed all my tickets simply out of frustration so it's all good for Zenimax. How about if this amazing "no logs for you" feature could be somehow donated for too high cp level veteran players with every dlc, that would definitely make ESO great again!

Skinzz · July 2019

Pretty sure it has something to do with new content being added to the game. Gets worst with each update/dlcs.

Itzmichi · July 2019

Delparis wrote: »

Nairinhe wrote: »

Move calculations to client, aaand... Cheaters, cheaters everywhere.

maybe but cheaters can't have that huge impact as they do things in a way that canno't be seen normaly

also cheaters are not that many on Pc and especially on consols

R u from China, young farmer?

idk · July 2019

Read it, as to the reasons, and the title Capt Obvious came to mind. Clearly someone who has read some of the many threads on the subject. Solving this issue is not as simple as OP has made it out though. But good try.

nafensoriel · July 2019

BrooksP wrote: »

I doubt any of these are the issue as most of these could be simply solved by hardware upgrades. If I had to guess the issue probably stems back to the revamp and the rushed scaling/phasing systems which have only compounded since. They keep adding onto a weak foundation

While they won't do it, but probably the best thing for them to do is to freeze all content updates and clean and restructure their backend.

E: Addons have no overall impact since they are almost completely client side.

Hardware upgrades mean exactly jack and an argonian squat if the system is bottlenecked. Adding more threads or more boxes doesn't help if you have a single thread controlling it all... and that thread is at capacity. Changing this aint easy or really even possible. The only way is to reduce or segment things so each tick does more work.

@Delparis
Calculations arent really all that slow. Computers are basically designed to handle them. Having to do a calculation, wait for a reply, then do more calculations is the problem. GDC and light attack weaving will likely have no effect on the performance of the server wise. Reducing buffs by 50% also will likely have a minimal impact. Things like that can be stuffed into a "container" and dumped for a really quick calculation elsewhere and then repackaged to move on to the next part of the process. Changing whatever is INSIDE that container isn't going to really hurt performance. What hurts performance is when you have to move that box around. If you want a real-world example of this check out EVE:Onlines "brain in a box" dev posts. They break it down pretty far for people who don't exactly understand coding or networks.

Unfortunately, most of what you suggest just won't do anything.

reprosal · July 2019

I wish people would realize it’s not a hardware bottleneck and it’s really really poor coding and netcode protocols.

If you look at older MMO’s like Dark Age of Camelot and even Warhammer online, there was hardly any lag even on way less throughput. It was just written much better.

Large scale open pvp on both and it still worked out just fine.

nafensoriel · July 2019

reprosal wrote: »

I wish people would realize it’s not a hardware bottleneck and it’s really really poor coding and netcode protocols.

If you look at older MMO’s like Dark Age of Camelot and even Warhammer online, there was hardly any lag even on way less throughput. It was just written much better.

Large scale open pvp on both and it still worked out just fine.

Comparing two entirely different platforms from two entirely different eras of gaming is like comparing your kitchen stove to a campfire.

Jhalin · July 2019

nafensoriel wrote: »

reprosal wrote: »

I wish people would realize it’s not a hardware bottleneck and it’s really really poor coding and netcode protocols.

If you look at older MMO’s like Dark Age of Camelot and even Warhammer online, there was hardly any lag even on way less throughput. It was just written much better.

Large scale open pvp on both and it still worked out just fine.

Comparing two entirely different platforms from two entirely different eras of gaming is like comparing your kitchen stove to a campfire.

ESO is the campfire here, they can put their disorganized jumbled code into a fancy box, but’s it’s still inferior programming

Jeremy · July 2019

Delparis wrote: »

imo the server got this bad performance because all the calculation responsable for the lag and frame rate drop are made server side. Those calculations are liked to:
many skills have multiple effects (dmg, buffs, debuffs) which make the calculation more complicated and time consuming

skills don't have a long enough global cooldown to prevent too much ping a request to the server

animation cancelling with LA weaving is increasing the stress on servers

there is too many buffs and debuffs available and they are splitted into major and minor buffs

objects, containers and npc are dynamic so the server has to tell the client were they are (npc) and what they are and what they content (objects and container)

to name a few..

i really think Zenimax devs should have a look on those points and improve them instead of putting more ressources in hardware.
a way to adress those issues is as follow:
Make skills have only 1 dmg, heal or buff effect

Increase the global cooldown to skills to 1 sec and make LA share the same global cooldown (clunky combat maybe but at least you don't play on a slideshow)

remove the animation cancelling on LA as they share the same gcd with skills

standerdize buffs and debuffs, no more minor and major one (less factors for the calculation), and make some of those buffs and debuffs available through foods, drinks and potions

npc position should be known from the client rather than the server so less useless informations transiting, the same thing for objects

container should have an RNG tool when interacting with it, so the server won't have to comunicate to the client what they contain. also since rng players will be able to interact with the same container and get different loot depending on their rng. each container have 1 loot.

hope this can help you Zenimax devs

Most calculations in respect to damage, healing, or buffs is just math. If their game engines can't handle math equations then we're in serious trouble no matter what.

What causes the lag in my experience is easy enough to see - lots of people in the zone or on the screen at once. Their servers simply can't handle high usage. I think what made them think it was excessive calculations that caused it was because the CP campaigns lag a lot more often than the none CP campaigns do. But I believe that's simply due to the higher populations. The CP campaigns have way way more traffic than the none CP campaigns. And I suspect if the none CP campaigns had the same population the CP Campaigns had it would lag just as bad there - with or without the added calculations of CP.

They just need better servers. There is no other way around this I don't think.

Cavedog · July 2019

Knightpanther wrote: »

Trust me its addons, I never use them and I only ever experience bad server performance in Cyrodiil, and that is pretty rare for me.

Be safe

It's not just add ons. Most of us play with very few or none at all and we still get horrendous lag during peak times

Jeremy · July 2019

Cavedog wrote: »

Knightpanther wrote: »

Trust me its addons, I never use them and I only ever experience bad server performance in Cyrodiil, and that is pretty rare for me.

Be safe

It's not just add ons. Most of us play with very few or none at all and we still get horrendous lag during peak times

I use zero addons (never have - mostly out of principle) and I lag horribly in a full Cyrodil as well.

So it can't be addons causing it.

idk · July 2019

reprosal wrote: »

I wish people would realize it’s not a hardware bottleneck and it’s really really poor coding and netcode protocols.

If you look at older MMO’s like Dark Age of Camelot and even Warhammer online, there was hardly any lag even on way less throughput. It was just written much better.

Large scale open pvp on both and it still worked out just fine.

You are correct about this not being a hardware problem but you are being overly simplistic on what you blame. This game is much more complex and has many more calculations comparatively than the games you mention.

idk · July 2019

Jeremy wrote: »

Delparis wrote: »

imo the server got this bad performance because all the calculation responsable for the lag and frame rate drop are made server side. Those calculations are liked to:
many skills have multiple effects (dmg, buffs, debuffs) which make the calculation more complicated and time consuming

skills don't have a long enough global cooldown to prevent too much ping a request to the server

animation cancelling with LA weaving is increasing the stress on servers

there is too many buffs and debuffs available and they are splitted into major and minor buffs

objects, containers and npc are dynamic so the server has to tell the client were they are (npc) and what they are and what they content (objects and container)

to name a few..

i really think Zenimax devs should have a look on those points and improve them instead of putting more ressources in hardware.
a way to adress those issues is as follow:
Make skills have only 1 dmg, heal or buff effect

Increase the global cooldown to skills to 1 sec and make LA share the same global cooldown (clunky combat maybe but at least you don't play on a slideshow)

remove the animation cancelling on LA as they share the same gcd with skills

standerdize buffs and debuffs, no more minor and major one (less factors for the calculation), and make some of those buffs and debuffs available through foods, drinks and potions

npc position should be known from the client rather than the server so less useless informations transiting, the same thing for objects

container should have an RNG tool when interacting with it, so the server won't have to comunicate to the client what they contain. also since rng players will be able to interact with the same container and get different loot depending on their rng. each container have 1 loot.

hope this can help you Zenimax devs

Most calculations in respect to damage, healing, or buffs is just math. If their game engines can't handle math equations then we're in serious trouble no matter what.

What causes the lag in my experience is easy enough to see - lots of people in the zone or on the screen at once. Their servers simply can't handle high usage. I think what made them think it was excessive calculations that caused it was because the CP campaigns lag a lot more often than the none CP campaigns do. But I believe that's simply due to the higher populations. The CP campaigns have way way more traffic than the none CP campaigns. And I suspect if the none CP campaigns had the same population the CP Campaigns had it would lag just as bad there - with or without the added calculations of CP.

They just need better servers. There is no other way around this I don't think.

It is the calculations that you call "just math" that are the issue. When we have a large number of players at the same location we are loading the same piece of hardware with all that math. The math for one skill takes into account passive percentages, buffs and debuffs, set bonus percentages all on the casters and the targets times all those people in the same area.

The server load became even more after the CE issue because Zos has to move a lot of "math" our clients used to keep track of to the servers.

While some of the changes going on this year will reduce some of that math I doubt it will be near enough. I do not think Zos possesses the technical knowledge to develop an economical solution.

Jeremy · July 2019

idk wrote: »

Jeremy wrote: »

Delparis wrote: »

imo the server got this bad performance because all the calculation responsable for the lag and frame rate drop are made server side. Those calculations are liked to:
many skills have multiple effects (dmg, buffs, debuffs) which make the calculation more complicated and time consuming

skills don't have a long enough global cooldown to prevent too much ping a request to the server

animation cancelling with LA weaving is increasing the stress on servers

there is too many buffs and debuffs available and they are splitted into major and minor buffs

objects, containers and npc are dynamic so the server has to tell the client were they are (npc) and what they are and what they content (objects and container)

to name a few..

i really think Zenimax devs should have a look on those points and improve them instead of putting more ressources in hardware.
a way to adress those issues is as follow:
Make skills have only 1 dmg, heal or buff effect

Increase the global cooldown to skills to 1 sec and make LA share the same global cooldown (clunky combat maybe but at least you don't play on a slideshow)

remove the animation cancelling on LA as they share the same gcd with skills

standerdize buffs and debuffs, no more minor and major one (less factors for the calculation), and make some of those buffs and debuffs available through foods, drinks and potions

npc position should be known from the client rather than the server so less useless informations transiting, the same thing for objects

container should have an RNG tool when interacting with it, so the server won't have to comunicate to the client what they contain. also since rng players will be able to interact with the same container and get different loot depending on their rng. each container have 1 loot.

hope this can help you Zenimax devs

Most calculations in respect to damage, healing, or buffs is just math. If their game engines can't handle math equations then we're in serious trouble no matter what.

What causes the lag in my experience is easy enough to see - lots of people in the zone or on the screen at once. Their servers simply can't handle high usage. I think what made them think it was excessive calculations that caused it was because the CP campaigns lag a lot more often than the none CP campaigns do. But I believe that's simply due to the higher populations. The CP campaigns have way way more traffic than the none CP campaigns. And I suspect if the none CP campaigns had the same population the CP Campaigns had it would lag just as bad there - with or without the added calculations of CP.

They just need better servers. There is no other way around this I don't think.

It is the calculations that you call "just math" that are the issue. When we have a large number of players at the same location we are loading the same piece of hardware with all that math. The math for one skill takes into account passive percentages, buffs and debuffs, set bonus percentages all on the casters and the targets times all those people in the same area.

The server load became even more after the CE issue because Zos has to move a lot of "math" our clients used to keep track of to the servers.

While some of the changes going on this year will reduce some of that math I doubt it will be near enough. I do not think Zos possesses the technical knowledge to develop an economical solution.

But the part I bolded is the real problem. Not extra calculations.

Adding 100 extra math problems for the game engine to process isn't going to cause a problem - especially simple ones like healing, buffs and damage effects. Even an ancient game engine could process those with no problem. Adding an extra 100 players is. So good luck trying to solve this problem by focusing on reducing the amount of calculations, They are going to need it.

Lower population restraints or increasing the capacity of their servers would be far more effective. I have to head out so if I don't respond that's why. Which is probably a good thing considering what you left in my mail box last time we talked.

Trancestor · July 2019

Rather play with lag at the evenings than play without ani canceling and weaving. What is wrong with you to even suggest such a thing?

nafensoriel · July 2019

When you have "alot of players" in one area its not the calculations causing lag... its the TRANSITION DATA bottlenecking the server.

Ever wonder why you come up to a fight and get dropped from the server if you move too quickly? That's because you moved from one part of the system to another... and that part was under stress. People get dropped in this situation because hamsters dislike being on fire.

Calculations are stupid easy for modern processors to do. Moving data between two points is always a bottleneck because it will always be slower than the CPU calcs. The instant you have to write or read your performance dies especially if you suddenly have to read the entire library of congress in 1.2 ms because a horde of yellow chihuahuas combines with a full-on DC/EP fight.

mairwen85 · July 2019

^ hamsters on fire...

Delparis wrote: »

imo the server got this bad performance because all the calculation responsable for the lag and frame rate drop are made server side. Those calculations are liked to:
many skills have multiple effects (dmg, buffs, debuffs) which make the calculation more complicated and time consuming

skills don't have a long enough global cooldown to prevent too much ping a request to the server

animation cancelling with LA weaving is increasing the stress on servers

there is too many buffs and debuffs available and they are splitted into major and minor buffs

objects, containers and npc are dynamic so the server has to tell the client were they are (npc) and what they are and what they content (objects and container)

to name a few..

i really think Zenimax devs should have a look on those points and improve them instead of putting more ressources in hardware.
a way to adress those issues is as follow:
Make skills have only 1 dmg, heal or buff effect

Increase the global cooldown to skills to 1 sec and make LA share the same global cooldown (clunky combat maybe but at least you don't play on a slideshow)

remove the animation cancelling on LA as they share the same gcd with skills

standerdize buffs and debuffs, no more minor and major one (less factors for the calculation), and make some of those buffs and debuffs available through foods, drinks and potions

npc position should be known from the client rather than the server so less useless informations transiting, the same thing for objects

container should have an RNG tool when interacting with it, so the server won't have to comunicate to the client what they contain. also since rng players will be able to interact with the same container and get different loot depending on their rng. each container have 1 loot.

hope this can help you Zenimax devs

Title: the reason == definitive, absolute information based on fact.

Post: in my opinion == implied information based on assumption

1) skills are delivery mechanisms for applying debuff/buff on target (damage is a health debuff), and/or caster - - the volume in or direction (1way or 2way) of that delivery isn't an issue, the number of complex and multi variable calculations behind each are. Which why ZOS ability audit is standardising calculations for skills functionally by type.

2) skill cool down is already ~1 second. Putting LA on the same cool down will not eliminate weaving, but make it harder for people who already struggle with it, or have high latency as there is now an even tighter window to get both upstream requests in.

3) see above. The only way round the new problem you've created is auto weaving. However, you don't want to weave every skill. You don't want healers pre emptively debuffing to pull everything all the time. And what of channelled skills and cast timers... You end up with more conditions. 2 separate attack windows works because you choose what to weave, and can compensate for latency... You can also play more Strategically jn pvp.

4) there should be sources of buffs at different degrees of strength. I can't see any argument or valid reason to consolidate.

5) npc locations, and containers are static - - npcs sometimes have a fixed path they wander in. The location is a larger radius than fixed npc. Moot point.

6) server will always tell client what's in container. Else: cue cheat engine insert hex at address for maelstrom bow on barrel click.

Mettaricana · July 2019

Delparis wrote: »

don't forget that consol player got the same server issues but they don't/can't use addons

This is accurate.... doubt it has anything to do with skills and add ons vs just beig a poorly made game by developers who didnt have a clue coupled with non stop management changes and a big wig holding the "fixthisshit" button hostage til he milks the game to death then when the game becomes a ghost town he'll press the fix button and relaunch it as pay to win totally free to play model to rope on the last few straggling dollars before pullin the plug or just letting the game go for years without content til it dies.

EmEm_Oh · July 2019

Delparis wrote: »

Xerikten wrote: »

you forgot addons.

addons depends on the skill of who made them, if the code is clean there shouldn't be any issues
however if the code is bad you should expect issues yes

I think it's more of the user exploiting what is designed to be exploited in the addon. We're not talking avout a simple motif tracker or recipe maker. The ones that do damage are the addons which allow easy manipulation of skill mechanics and actions.

Billdor · July 2019

Delparis wrote: »

imo the server got this bad performance because all the calculation responsable for the lag and frame rate drop are made server side. Those calculations are liked to:
many skills have multiple effects (dmg, buffs, debuffs) which make the calculation more complicated and time consuming

skills don't have a long enough global cooldown to prevent too much ping a request to the server

animation cancelling with LA weaving is increasing the stress on servers

there is too many buffs and debuffs available and they are splitted into major and minor buffs

objects, containers and npc are dynamic so the server has to tell the client were they are (npc) and what they are and what they content (objects and container)

to name a few..

i really think Zenimax devs should have a look on those points and improve them instead of putting more ressources in hardware.
a way to adress those issues is as follow:
Make skills have only 1 dmg, heal or buff effect

Increase the global cooldown to skills to 1 sec and make LA share the same global cooldown (clunky combat maybe but at least you don't play on a slideshow)

remove the animation cancelling on LA as they share the same gcd with skills

standerdize buffs and debuffs, no more minor and major one (less factors for the calculation), and make some of those buffs and debuffs available through foods, drinks and potions

npc position should be known from the client rather than the server so less useless informations transiting, the same thing for objects

container should have an RNG tool when interacting with it, so the server won't have to comunicate to the client what they contain. also since rng players will be able to interact with the same container and get different loot depending on their rng. each container have 1 loot.

hope this can help you Zenimax devs

Performance is bad because of the crappy CDN they use.

Also I don't think the Sysadmins over at ZoS need your advice.

RinaldoGandolphi · July 2019

The problem with games performance issues likes solely with their use of the TCP protocols. TCP has a ridiculous amount of overhead, and every single packet or group of packets had to be ACK. To make matters worse TCP has built in congestion controls that assume packets not ACK or revived with the TTL value are assumed to be loss due to network congestion, so it will dial
Back throughput and attempt to resend.

When you got two zergs in pvp all mashing skills and sending the server packets it must ACK and process with TCP what do you think is going to happen?

There is reason no other triple A studio since the early 2000’s has used TCP for any real time or FPS combat game.

ZOS is t going to re write their net ode to properly use a UDP implementation so no amount of game changes short of straight cooldowns is going to do much of anything to fix the issues, 99% of the problem lies with the idiotic choice to use TCP which scales poorly and has a huge amount of overhead. TCP wasn’t designed for this, there is a reason other real time games started using UDP instead with their own custom stack on top of it and client prediction.

Good luck