Performance - population - godmode...A Fix

Rune_Relic · April 2016

Performance.
Many believe having unlimited players hit will kill everyone.
For me this very act of killing more people in less time just adds more code pumped through the servers and more clients that need updating. ie lag.
It can only work if people die instantly and if people die instantly all you did was create god mode and ridiculously OP skills/sets.
So I don't buy the performance argument for 1 second.
What I do buy is people want godmode......don't hate me yet....read on.

Population.
There is nothing more demoralising than seeing a red, blue or yellow map and knowing there is nothing you can do to change it when an alliance decides to make it their home.
You are permanently outgunned unless you catch them with there pants down for a limited amount of time.
Then normal business resumes.
The current system of bonuses that reward you for dominance makes the game an uphill challenge that cant be met without a mass exodus from other campaigns and destabilising the whole game with campaign tourists.

Godmode.
The lust for power is strong in this game and yet there are many wiser heads that abound and can see beyond the testosterone addiction.
I was young once. I have not forgotten the taste of domination. But I don't let it rule me.
DPS want to take on impossible numbers and win against impossible odds.
Tanks want to take on impossible numbers and survive against impossible odds.
Even the healers want to keep everyone alive when they should be long dead.
You are... the HERO.....after all.
Many lust for the days of dynamic ult gen to give back the sense of power and supremacy.
But that single player ES mentality cant be met in an MMO....or can it ?

What if I told you you could have it all ?
Yes...let that sink in for a while......and think about how that would feel before reading on.....

Let the population imbalance control the alliance AoE caps.
If you have a low population and the campaign is dominated;
1. the dominant alliance has very low AoE target cap. Almost relegated to single target skills in fact. This encourages the use of single target skills and builds to be optimal in any fights and to hold onto dominance. This in turn naturally reduces server load as the highly populated alliance hits much fewer people with any skill or ultimate or siege or heals. This encourages ideas on how to improve single target skills and gains experience with them.
2. the dominated alliance in turn can have much higher AoE caps. Maybe even unlimited. If this applied to the whole population it would and does cripple the servers. But as the alliance population numbers are so low in this case, the server impact is much, much lower. This enable the focus of AoE and gives the 1vX or small group godmode, that is coveted by many, but impossible in reality (for everyone at the same time at least).
3. By seriously limiting AoE caps except for the smallest populations we have increased performance.
4. By giving the dominant faction a penalty and the dominated faction a boost we end campaign domination that does nothing but destroy morale and frustrate every player. Either through lack of hope or no one to fight with.
5. It encourages inventive use of single target skills as well as AoE skills to make the best of the tools that either side has and ends the era of AoE dominated cyrodiil.

TL;DR
We can now go full *** without worrying about server performance.

https://www.youtube.com/watch?v=oAKG-kbKeIo

smacx250 · April 2016

So it has no change on the 3-faction pop locked campaigns that everyone is complaining about the lag in, since all the pops are the same?

Rune_Relic · April 2016

smacx250 wrote: »

So it has no change on the 3-faction pop locked campaigns that everyone is complaining about the lag in, since all the pops are the same?

You are using variable AoE caps.
So pop locked campaigns on all sides will be practically single target for everyone.
ie no lag.
If you hit one person....that's you and them that needs to be updated at the same time.
If you hit 10 people...that's you and 10 others that have to be updated at the same time.
If you hit loads of people at a flag, breach or kill box....everyone has to be updated at the same time.
Multiply that by the number of people on screen.

This generates a crapstorm of code all at the same time with the inevitable hit on concurrent data-access, network bandwidth (with everyones game client being updated at the same time) and server load trying to do a magnitude more work in the same time.
Add in a base population at prime time which is already pushing the game system towards its limits and you end up with freezeframe and no response.

At the moment all we have is siege + prox det + vicious death among other inter related AoE.
We stand in a sea of red.
We can not dodge roll out of it because the whole area is red and we run out of stamina trying.
We can not hold block because there is no regen and we run out of stamina trying.
As soon as we are out of stamina it is 1 or 2 hits and dead.
It is like the carpet bombing practice of world war two.
Where they used the wind and heat to fan the flames across a whole city.
Destroying anything and everything within its area.

So counter measures to this ?
Stacking even more smart healing to the point of heal spam.
You run your arse off to that green ring and join all the other people standing in that green ring.
Because that gives you more chance of staying alive than the instant death of standing in that sea of red.

Sej on Azura EU was like this the other night.
If you went outside into the sea of red you would die.
If you split up you would be picked of by the gank squads laying in wait with the ambush.
You couldn't take down the ambush without entering the sea of red in numbers.
The only way to stay alive in any kind of numbers was non stop heal spam.

So you stay inside.
They cant enter because the entry is a kill box.
You cant go outside because of the sea of red.
So both sides are desperately trying to get a chain reaction of vicious death going.
Occasional charge with ultimates hitting even more people.

The game is a cluster* where one broken OP skills is created to counter another broken OP skill created to overcome game imbalances created by bad game design.
People are just chucking nukes at each other hoping to wipe out everyone in one go.
I am tired of shouting in the wind.
Enjoy the lag everyone.

I started this PvP and thoroughly enjoyed it at the time as it was pretty much hand to hand combat (aside from MMO vets that arrived and bought bomb groups and zergs with them).
There is no hand to hand combat anymore.
There is only a sea of red or gang ganked.

KenaPKK · April 2016

Varying AoE caps based on population levels would create messy code that just complicates the system, possibly making it function worse. Let's be honest -- do we really trust ZoS to implement such a system efficiently? Instead of trying to reduce server calculations by complicating the server.....try simplifying it instead.

The best option for simplifying the code and reducing server calculations is and always has been the outright removal of Magicka Detonation, Vicious Death, and AoE caps.

Removing AoE caps reduces server load, and removing mag det and VD prevents individuals from instakilling groups without counterplay. A few surgical nerfs may be also be called for, namely reduction in the radii of Bat Swarm and Steel Tornado, but that cannot be determined until the above changes are made.

Rune_Relic · April 2016

KenaPKK wrote: »

Varying AoE caps based on population levels would create messy code that just complicates the system, possibly making it function worse. Let's be honest -- do we really trust ZoS to implement such a system efficiently? Instead of trying to reduce server calculations by complicating the server.....try simplifying it instead.

The best option for simplifying the code and reducing server calculations is and always has been the outright removal of Magicka Detonation, Vicious Death, and AoE caps.

Removing AoE caps reduces server load, and removing mag det and VD prevents individuals from instakilling groups without counterplay. A few surgical nerfs may be also be called for, namely reduction in the radii of Bat Swarm and Steel Tornado, but that cannot be determined until the above changes are made.

I understand what you are saying about trusting ZOS with convoluted code.
I am inclined to agree.

Det and vicious death I agree.....the AoE caps I don't.
That will simplify the code (agree there) ...but the code still remains.
You are still hitting untold people...especially in the case of siege and such.
Every player hit...is more code to be calculated and more client connections and more database access
Removing 1% of the code of each player hit by selecting everyone in a red circle instead of just some of the people in that red circle will have little effect.
100+x 99% code (uncapped AoE hitting as many people in a circle as can be bothered) > 1x 100% code (1 person hit)

Throw in camps and instant rezz.
People are never out of the fight.
So there is 100% uptime of mass AoE in effect.
There is no break in the combat as players are delayed getting into battle.
With more players on line, and more players on screen....all you did is kick the *** out the server.
Then wonder why nothing is happening as the server chokes.

Vicous death was hailed as a cure for zergs....and as zrergs caused lag the lag would go.
Mass siege had no caps so would kill everyone off rapidly.
Guess what ?...its still lagging.
In fact...ITS LAGGING EVEN WORSE!
We just have even more powerful gods running around with instakill of whole groups instead of just instakill 1v1

KenaPKK · April 2016

Rune_Relic wrote: »

KenaPKK wrote: »

Varying AoE caps based on population levels would create messy code that just complicates the system, possibly making it function worse. Let's be honest -- do we really trust ZoS to implement such a system efficiently? Instead of trying to reduce server calculations by complicating the server.....try simplifying it instead.

The best option for simplifying the code and reducing server calculations is and always has been the outright removal of Magicka Detonation, Vicious Death, and AoE caps.

Removing AoE caps reduces server load, and removing mag det and VD prevents individuals from instakilling groups without counterplay. A few surgical nerfs may be also be called for, namely reduction in the radii of Bat Swarm and Steel Tornado, but that cannot be determined until the above changes are made.

I understand what you are saying about trusting ZOS with convoluted code.
I am inclined to agree.

Det and vicious death I agree.....the AoE caps I don't.
That will simplify the code (agree there) ...but the code still remains.
You are still hitting untold people...especially in the case of siege and such.
Every player hit...is more code to be calculated and more client connections and more database access
Removing 1% of the code of each player hit by selecting everyone in a red circle instead of just some of the people in that red circle will have little effect.
100+x 99% code (uncapped AoE hitting as many people in a circle as can be bothered) > 1x 100% code (1 person hit)

Throw in camps and instant rezz.
People are never out of the fight.
So there is 100% uptime of mass AoE in effect.
There is no break in the combat as players are delayed getting into battle.
With more players on line, and more players on screen....all you did is kick the *** out the server.
Then wonder why nothing is happening as the server chokes.

Vicous death was hailed as a cure for zergs....and as zrergs caused lag the lag would go.
Mass siege had no caps so would kill everyone off rapidly.
Guess what ?...its still lagging. We just have even more powerful gods running around with instakill of whole groups instead of just instakill 1v1

Without AoE caps, each player hit is a single server side damage calculation.

With AoE caps, the server has a much higher load per person inside an AoE spell, even if fewer take damage. This is because the server must first count the players within the AoE, then it must choose ones for each tier of AoE damage (full, half, quarter, etc), and then calculate the damage amount for each player struck based on those tiers. That last step along outnumbers the calculations done without AoE caps simply because the full damage load is calculate for each player, and then some number of players receive an additional division step as the AoE cap reduces their damage.

It seems you do not fully understand the complexity of the code structure.

Rune_Relic · April 2016

In a nutshell.
Mass AoE as a cure for lag is FAIL.
Superpowers as cure for lag is FAIL.
All it did was make gods among men and destroyed hand to hand combat .

Yes being outnumbered is an issue.
Making people gods to compensate is an answer to this....but not to lag.
As has been proven patch after patch after patch after patch after patch after patch after patch after patch after patch after patch after patch after patch after patch

When the server is pop locked their is no population outnumbering issue (or shouldn't be)
Thus superpower and super AoE (aka full *** godmode) are not required.

Rune_Relic · April 2016

KenaPKK wrote: »

Rune_Relic wrote: »

KenaPKK wrote: »

Varying AoE caps based on population levels would create messy code that just complicates the system, possibly making it function worse. Let's be honest -- do we really trust ZoS to implement such a system efficiently? Instead of trying to reduce server calculations by complicating the server.....try simplifying it instead.

The best option for simplifying the code and reducing server calculations is and always has been the outright removal of Magicka Detonation, Vicious Death, and AoE caps.

Removing AoE caps reduces server load, and removing mag det and VD prevents individuals from instakilling groups without counterplay. A few surgical nerfs may be also be called for, namely reduction in the radii of Bat Swarm and Steel Tornado, but that cannot be determined until the above changes are made.

I understand what you are saying about trusting ZOS with convoluted code.
I am inclined to agree.

Det and vicious death I agree.....the AoE caps I don't.
That will simplify the code (agree there) ...but the code still remains.
You are still hitting untold people...especially in the case of siege and such.
Every player hit...is more code to be calculated and more client connections and more database access
Removing 1% of the code of each player hit by selecting everyone in a red circle instead of just some of the people in that red circle will have little effect.
100+x 99% code (uncapped AoE hitting as many people in a circle as can be bothered) > 1x 100% code (1 person hit)

Throw in camps and instant rezz.
People are never out of the fight.
So there is 100% uptime of mass AoE in effect.
There is no break in the combat as players are delayed getting into battle.
With more players on line, and more players on screen....all you did is kick the *** out the server.
Then wonder why nothing is happening as the server chokes.

Vicous death was hailed as a cure for zergs....and as zrergs caused lag the lag would go.
Mass siege had no caps so would kill everyone off rapidly.
Guess what ?...its still lagging. We just have even more powerful gods running around with instakill of whole groups instead of just instakill 1v1

Without AoE caps, each player hit is a single server side damage calculation.

With AoE caps, the server has a much higher load per person inside an AoE spell, even if fewer take damage. This is because the server must first count the players within the AoE, then it must choose ones for each tier of AoE damage (full, half, quarter, etc), and then calculate the damage amount for each player struck based on those tiers. That last step along outnumbers the calculations done without AoE caps simply because the full damage load is calculate for each player, and then some number of players receive an additional division step as the AoE cap reduces their damage.

It seems you do not fully understand the complexity of the code structure.

That's not how it works my friend.
60 players are not all updated in parallel.
60players are updated one after another in serial sequence.
Whether you do this from the recipient view or instigator view is arbitrary.
Depends if you want to hook recipient updates one by one or do a group update for group damage.

First is to receive an attack packet from the client.
Step 1 is to see who if anyone is going to receive that damage and then build a list of everyone in the red/green circle or damage radius (accounting for LOS and such).
Step 2 is to filter the closest to the center of radius in order of radius or those with lowest health in the case of smart heals.
Step 3 is to have a conditional branch (which is in the process of being removed) where group 1 receive damage multiplayer A, group 2 receive damage multiplyer B and group 3 receive damage multiplyer C.
Loop (x players hit using damage modifier in radius order) starts here to ensure concurrency....
Step 4 is then to collect all of the attackers stats and find the current status, buffs, passives and debuffs (dynamic and static).
Step 5 is then to get the 1st recipents status and find the current status, buffs, passives and debuffs (dynamic and static)
Step 6 is then to get the results of the damage vs mitigation (allowing for all the transient states buff debuffs and modifiers including armour sets and such..blocking/dodging/stunned) and then apply them to the recipient record.
Step 7 is then to modify the recipient database.
Step 8 is then to notify the recipient client.
Step 9....repeat for every other player that has been hit within the red/green circle.
...Loop end here
Step 10 is then to charge the attacker the required costing including any modifiers that are currently in effect.
Step 11 is then to modify the attacker database.
Step 12 is then to notify the attacker client
Step 13....send occasional full status packets to ensure client and server are in sync.
(NB all the time the AoE updates is being performed the database for each player concerned is probably locked to ensure concurrency. Priority one is to ensure the database remains accessible and is released ASAP).

So like I said.
The piece you are trying to remove is step 3 above which probably accounts for about 1% of the code involved in each client update.
The remaining 99% of the code is repeated for every player hit like above.
10 players....10x loop code.... 10x traffic...database lockdown
100 players....100x loop code....100x traffic...extensive database lockdown

EDIT: reviewed attacker costing update outside loop.

Jimbohere · April 2016

Or you could regulate the server population. I don't want to say it but it's true. If red has one bar on then it caps till yellow and blue have one bar and so on. That would make sure there's a fairly even number of players. It would defiantly suck but it would keep it competitive. Cause let's be honest there's nothing competitive about trying to defend a 100 man Zerg when your faction only has 40 on. Plus they need to remove imperial city from the actual Cyrodil count.

AlexHo1982 · April 2016

@ OP: The system sounds interesting, but I would just expect some major (game breaking) issue arising from implementing this system. They should probably simplify the game mechanics by just removing AoE skills altogether.

At this point I would be happy to see them:
- adding an adaptive pop lock that does not allow players to enter a campaign as soon as another alliance has less players, until all alliances are balanced out (you know, like in those FPS games), forcing people to spread in different campaigns or to wait.
- remove all AoE related spells and sets entirely from the game and rework / rebalance AoE related content (skills, sets and PvE balance) accordingly.
- remove any buffs from holding Keeps, Scrolls and Resources and add instead a passive AP gain per captured objective for each player in the alliance (e.g.: 50 AP per hour for each Resource / 150 AP per hour for each Scroll / 250 AP per hour for each Keep).

Not that any of this is probable or will be popular with the majority of players...

RinaldoGandolphi · April 2016

On the topic of AOE Caps, discussing them from a "game performance perspective" and a "game balance perspective" are two completely different things. AOE caps actually lessen lag IF they are implemented correctly. the key word is IF.

If you jump into a zerg ball of 60 people and your AOE hits 60 people your looking at 60 + calcs
If you jump into a zerg ball with a straight AOE cap of 10(no drop off just 10 people take full damage) then your sending 10 calcs to the server

The performance problems with this game are two fold.

1. The choice of TCP over UDP was a poor choice for a real time combat game like ESO. TCP due to its nature as a a guanrateed delivery system and the fact that it assumes every lost packet is caused by congestion and its built-in mechanism for dealing with said congestion, and its requirements on packet delivery simply make it a poor choice for the game and its netcode.

2. Simply put, a game like ESO, with as close to a real-time combat system you will get with an MMO, was designed to have resource management take the place of cooldowns. then they introduced the Champion System that gave everyone unlimited resources, along with the 20% Buff to resource regen with Pots, adding an absurd amount of regen with drinks, suddenly resource management no longer matters as literlaly everyone can spam X skill of their choice and NEVER run out of resources.

So where do we go forward from here? Its obvious ZOS sin't going to re-write the game code or the netcode, but there are things they can do to fix the issues.

A:- Introduce a Softcap System in Cyrodiil and disable the CS in every campaign its no secret that PVP and lag got worse with the CS due to resource management no longer mattering. They MUST change this...

they need reasonable caps...

hardcap magic and stamina regen at 1000,
cap weapon/spell damage at 3k,
Cap cost reductions to 7%
remove the 50% damage reduction from Battle Spirit,
remove the nerfs to movement and the buffs to snares....

They need to lower time to kill while at the same time lowering resources available to players to use skills. Making infinite resource builds impossible to make reduces the button spamming and overall reduces the server stress.

B:- AOE caps need tweaked AOE Caps need tweaked. Right now the falloff where after 6 people X amount of people take half damage actually puts more load on the server to calculate. Removing the cap from a performance perspective is counter intuitive because hitting 60 people with an AOE with no caps would cause more stress on the server then the current hit 6 people and rest take half damage we have now.

AOE caps need to be set to a flat number...say 25 people take a flat full damage and thats it...after 25 no one takes damage at all...compromises have to be made weighing balance vs performance...AOE caps need to register numbers the server can handle...the falloff damage is obviously too high a number to calculate, so just increase the base number and get rid of the falloff component.

C - Player Base AOE skills need to be removed from the game and replaced with ground targeted AOE with cast times - PBAOE, the way the current AOE caps are, and the netcode issues further exacerbate these problems. Changing PBAOE to Ground Target AOE with cast times gets rid of the stack and spam meta all together, its simply gone. the stack and spam meta is the final nail that makes the Cyrodiil lag unbearable.

In short nothing short of drastic changes to player behavior in conjunction with tweaking AOE caps is going to fix the lag in PVP. Remove the AOE caps would actually make things worse from a performance perspective. I would love to have AOE caps removed from a balance perspective, but from a performance perspective it would induce even more server side stress on the game.

The lag in ESO PVP will never be fixed 100% without a major engine and netcode re-write, ZOS however can improve performance of the game CONSIDERABLY by altering the AOE caps and game skills that encourage the conditions that cause the performance issues.

You stop feeding the stray cat, he quits coming to your house for food....remove the meta and playstyles that lag your game, and the lag will stop...i know many of the old people that would comeback if these things were addrssed.

At the end of the day, the game both server and client side doesn't scale well enough to support what we have currently, so you have tailor it to scale to what you cna support, thats going to require some drastic changes, and the sooner they are made the better.

Rune_Relic · April 2016

Jimbohere wrote: »

Or you could regulate the server population. I don't want to say it but it's true. If red has one bar on then it caps till yellow and blue have one bar and so on. That would make sure there's a fairly even number of players. It would defiantly suck but it would keep it competitive. Cause let's be honest there's nothing competitive about trying to defend a 100 man Zerg when your faction only has 40 on. Plus they need to remove imperial city from the actual Cyrodil count.

Yes...of course you can cut the population too.
But where does that leave us ?

Is the idea to have just a group on group game with a massive deserted map ?
Is the idea to have faction vs faction on a highly populated massive map ?

I don't object to the people that want to have team vs team game.
But I personally bought ESO for the massive fights.....like we once had....without any lag.
That was in the age of innocence when most of the players like me had no concept of bomb groups, pain trains and zerg balls.
It was L2P nubs that allowed Cyrdodiil to function while the others tried to melt it in the name of the god AP.
Now there are much fewer if any nubs using single target and hand to hand combat.
Everyone just stacks the latest bestest gear or skills that do the max damage to the most people in the minimum time.
Hence FoTM builds....prox det...siege spam...VD

Rune_Relic · April 2016

AlexHo1982 wrote: »

@ OP: The system sounds interesting, but I would just expect some major (game breaking) issue arising from implementing this system. They should probably simplify the game mechanics by just removing AoE skills altogether.

At this point I would be happy to see them:
- adding an adaptive pop lock that does not allow players to enter a campaign as soon as another alliance has less players, until all alliances are balanced out (you know, like in those FPS games), forcing people to spread in different campaigns or to wait.
- remove all AoE related spells and sets entirely from the game and rework / rebalance AoE related content (skills, sets and PvE balance) accordingly.
- remove any buffs from holding Keeps, Scrolls and Resources and add instead a passive AP gain per captured objective for each player in the alliance (e.g.: 50 AP per hour for each Resource / 150 AP per hour for each Scroll / 250 AP per hour for each Keep).

Not that any of this is probable or will be popular with the majority of players...

Fair comment.....and not wrong...on the likely busted game.
I think that will happen regardless though

The adaptive pop lock has been suggested by a few over the history of the forum.
Its a good idea except for the fatal flaw of endless queues for any popular faction.
Admittedly it gives the players a choice of balancing the factions or waiting in an endless queue.
BUT...players always had the option to swap faction and avoid queuing and yet only a few did.
So I don't see that changing.

Removing AoE...as much as I would like that personally as I prefer hand to hand combat and 'fair' combat;
I understand many people did come here for that aspect of the game too.
They did have dynamic ultimate when they came here which gave them that sense of godhead.
We can argue if that was ever a good idea in hindsight.....but many did get a taste for blood back then.
So my compromise was to give those people a taste of godhood......just not at the expense of crippling the whole game.

AP....oh boy. Yes the incentives need a rethink on many fronts.
Your AP idea I think would just lead to faction ensuring they possess all resources and keeps.
Without population balance or some way to address the difference in power...you can do nothing about that
But there is a balance to be struck. There not only has to be a need for dominated sides to play the game but also for the dominating to play the game too.
The dominating can be enticed through AP/greed.
The dominated can be entices through power/godmode.
Both need something to fight for.
Faction loyalty is worth jack at this point with no penalty for treachery.
Currently I can just have a guild with faction on both sides and AP farm away to my hearts content.
Free money....thanks very much ZOS.

AlexHo1982 · April 2016

Rune_Relic wrote: »

The adaptive pop lock has been suggested by a few over the history of the forum.
Its a good idea except for the fatal flaw of endless queues for any popular faction.
Admittedly it gives the players a choice of balancing the factions or waiting in an endless queue.
BUT...players always had the option to swap faction and avoid queuing and yet only a few did.
So I don't see that changing.

AP....oh boy. Yes the incentives need a rethink on many fronts.
Your AP idea I think would just lead to faction ensuring they possess all resources and keeps.
Without population balance or some way to address the difference in power...you can do nothing about that
But there is a balance to be struck. There not only has to be a need for dominated sides to play the game but also for the dominating to play the game too.
The dominating can be enticed through AP/greed.
The dominated can be entices through power/godmode.
Both need something to fight for.
Faction loyalty is worth jack at this point with no penalty for treachery.
Currently I can just have a guild with faction on both sides and AP farm away to my hearts content.
Free money....thanks very much ZOS.

Since the story mode allows players to fight for all factions, they could also allow them to queue for any faction in PvP (without going against their beloved lore), if a pop lock system was implemented. This could be handled as basically playing a mercenary role with the same restrictions of a guest campaign (e.g. no Emp., but AP gain).

I think this would at least help to bring balance to the population issue in campaigns and solve problems of one faction controlling the entire map. A problem that already exists or at least existed (buff campaigns) and leads to dead campaigns.
My AP idea would only be viable with a pop lock system. Otherwise it would become a terrible AP farm "exploit".

Removing all AoE skills was btw. not meant entirely seriously and more a sign of my resignation on the balancing of this game. What probably would be more viable ist to set a fixed AoE cap (full damage and heal) to group size (24?) and not scale it. This reduces server load due to less calculations that need to be handled and all skills would have the same game mechanic (no varying by skill like it is right now). Group heals should also only work on group members.

They also really should start reviewing all ability costs. Right now the ratio of cost / efficiency is totally imbalanced, comparing the skill lines and AoE vs. single target dps / heals.

I wished they were better in balancing the game and finding smart simple solutions to address the problems, since right now it cannot live up to its premise at all.

Rune_Relic · April 2016

RinaldoGandolphi wrote: »

On the topic of AOE Caps, discussing them from a "game performance perspective" and a "game balance perspective" are two completely different things. AOE caps actually lessen lag IF they are implemented correctly. the key word is IF.
If you jump into a zerg ball of 60 people and your AOE hits 60 people your looking at 60 + calcs

If you jump into a zerg ball with a straight AOE cap of 10(no drop off just 10 people take full damage) then your sending 10 calcs to the server

Yep....my main point I guess and the problem with siege spam among other skills.

The performance problems with this game are two fold.

1. The choice of TCP over UDP was a poor choice for a real time combat game like ESO. TCP due to its nature as a a guanrateed delivery system and the fact that it assumes every lost packet is caused by congestion and its built-in mechanism for dealing with said congestion, and its requirements on packet delivery simply make it a poor choice for the game and its netcode.

I guess the automated sequential packets made managing concurrency a hell of a lot easier. Although they could have had the application layer deal with timestamping and concurrency instead. So yes I agree that UDP would have been a better choice from that aspect. But then again they chose a client server model which was mostly authoritive too. UDP can be a fickle thing with packet loss, fragmentation and firewall implementations dropping fragmented packets. I have used some VPN systems that offered UDP or TCP and the UDP simply wouldn't work with all the various MTU settings across the net. The problem with TCP may be that you have to receive everything. But the problem with UDP is that you are not guaranteed to receive anything. I think I would have gone the TCP way too in the end. Although probably IPv6 TCP and a monitored peer-peer gated virtual network with a layer 4 virtual routing protocol for zoning players.

2. Simply put, a game like ESO, with as close to a real-time combat system you will get with an MMO, was designed to have resource management take the place of cooldowns. then they introduced the Champion System that gave everyone unlimited resources, along with the 20% Buff to resource regen with Pots, adding an absurd amount of regen with drinks, suddenly resource management no longer matters as literlaly everyone can spam X skill of their choice and NEVER run out of resources.

But you cant run any game without cooldowns. Imagine a macro to hit 5 skills pretty much in parallel. Whoever hits first wins as long as they have a big enough resource pool to launch 5 attacks in parallel. The whole point of cooldowns and timing control is to ensure the enemy has a chance to respond. That simply doesnt happen most of the time with animation cancelling and lag. So any cooldowns have to allow for that lag too.

But yes...resource management...or lack of. Agreed. The fact you can have a big enough resource pool to take someone out without response (regardless of 1,2,3 or 5 shot) is the main problem for me though. So scaling off stamina or magicka to not only do more damage but also more often is a recipe for disaster with really awkward balancing issues. That's why I kicked up such a stink about cost/hit being irrelevant and cost/dps being the regulator.

So where do we go forward from here? Its obvious ZOS sin't going to re-write the game code or the netcode, but there are things they can do to fix the issues.

A:- Introduce a Softcap System in Cyrodiil and disable the CS in every campaign its no secret that PVP and lag got worse with the CS due to resource management no longer mattering. They MUST change this...

they need reasonable caps...

hardcap magic and stamina regen at 1000,
cap weapon/spell damage at 3k,
Cap cost reductions to 7%
remove the 50% damage reduction from Battle Spirit,
remove the nerfs to movement and the buffs to snares....

They need to lower time to kill while at the same time lowering resources available to players to use skills. Making infinite resource builds impossible to make reduces the button spamming and overall reduces the server stress.

Again. See cost/dps vs cost/hit. In the days of soft caps we just ended up with certain configs that could max out dps healing and mitigation all at once. You used that configuration or you weren't competitive. Sound familiar ?

B:- AOE caps need tweaked AOE Caps need tweaked. Right now the falloff where after 6 people X amount of people take half damage actually puts more load on the server to calculate. Removing the cap from a performance perspective is counter intuitive because hitting 60 people with an AOE with no caps would cause more stress on the server then the current hit 6 people and rest take half damage we have now.

This is the hitting more people adds more load.....dead people don't add load...paradox. You have to increase the load to reduce the load. Variable damage is ridiculously inefficient because you are using the power to hit someone else but the negligible probablility of death ensures that its wasted cycles. So yes the variable damage has to go. Removing the cap on the players hit though is just instant server death. Especially when people can be rezzed, forward camps are everywhere putting you straight back in the fight, animation cancelling speeds up the AoE adding to even more calcs, smart heals being ridiculously efficient using smart heals instead of proximity heals ...etc etc

AOE caps need to be set to a flat number...say 25 people take a flat full damage and thats it...after 25 no one takes damage at all...compromises have to be made weighing balance vs performance...AOE caps need to register numbers the server can handle...the falloff damage is obviously too high a number to calculate, so just increase the base number and get rid of the falloff component.

Agreed as above. The AoE caps and population caps should have been used to model worst case load, traffic, i/o counts. They know what cycle/s, memory allocation, bandwidth and such is provided by their hardware. They can crunch the numbers and figure out exactly where that cap should be. AND/OR....you can also use that information to increase the AoE cap as the population decreases for the alliance concerned. It matters not as long as the equation balances.

C - Player Base AOE skills need to be removed from the game and replaced with ground targeted AOE with cast times - PBAOE, the way the current AOE caps are, and the netcode issues further exacerbate these problems. Changing PBAOE to Ground Target AOE with cast times gets rid of the stack and spam meta all together, its simply gone. the stack and spam meta is the final nail that makes the Cyrodiil lag unbearable.

Stack and spam with PBAoE. Does it matter if you used ranged aoe targeted at your feet or pbaoe targeted at your feet ? You have to enforce minimum distance on all ranged aoe too or removing pbaoe counts for nothing. Even then you still create a loophole. Simply use two groups at minimum range healing each other across the minimum range gap. I guess two zerg balls would be better then one......on second thoughts...maybe not

In short nothing short of drastic changes to player behavior in conjunction with tweaking AOE caps is going to fix the lag in PVP. Remove the AOE caps would actually make things worse from a performance perspective. I would love to have AOE caps removed from a balance perspective, but from a performance perspective it would induce even more server side stress on the game.

This is kind of the whole point of the original post. When the server is under extreme load the AoE target caps are set to low or 1. They don't increase unless the population is low enough not to kill the server.

The lag in ESO PVP will never be fixed 100% without a major engine and netcode re-write, ZOS however can improve performance of the game CONSIDERABLY by altering the AOE caps and game skills that encourage the conditions that cause the performance issues.

Agreed. Like I said....I would have gone monitored, authenticated and gated peer-peer with encrypted IPv6. That's not going to happen though and is fine in hindsight or foresight...whatever. The AoE target limits in conjuction with population and load management are the key though. Streamlining the loop code should help....which is en-route too.

You stop feeding the stray cat, he quits coming to your house for food....remove the meta and playstyles that lag your game, and the lag will stop...i know many of the old people that would comeback if these things were addrssed.

At the end of the day, the game both server and client side doesn't scale well enough to support what we have currently, so you have tailor it to scale to what you cna support, thats going to require some drastic changes, and the sooner they are made the better.

Whole heartedly agree on changing the code to work within the physical limitations and not way beyond them.

Thanks @RinaldoGandolphi always enjoy your in depth input and thoughts.

RinaldoGandolphi · April 2016

No problem @Rune_Relic I enjoy reading your thoughts as well, and your bring up some very good points here in how they could address things moving forward.

Rune_Relic · April 2016

AlexHo1982 wrote: »

Rune_Relic wrote: »

The adaptive pop lock has been suggested by a few over the history of the forum.
Its a good idea except for the fatal flaw of endless queues for any popular faction.
Admittedly it gives the players a choice of balancing the factions or waiting in an endless queue.
BUT...players always had the option to swap faction and avoid queuing and yet only a few did.
So I don't see that changing.

AP....oh boy. Yes the incentives need a rethink on many fronts.
Your AP idea I think would just lead to faction ensuring they possess all resources and keeps.
Without population balance or some way to address the difference in power...you can do nothing about that
But there is a balance to be struck. There not only has to be a need for dominated sides to play the game but also for the dominating to play the game too.
The dominating can be enticed through AP/greed.
The dominated can be entices through power/godmode.
Both need something to fight for.
Faction loyalty is worth jack at this point with no penalty for treachery.
Currently I can just have a guild with faction on both sides and AP farm away to my hearts content.
Free money....thanks very much ZOS.

Since the story mode allows players to fight for all factions, they could also allow them to queue for any faction in PvP (without going against their beloved lore), if a pop lock system was implemented. This could be handled as basically playing a mercenary role with the same restrictions of a guest campaign (e.g. no Emp., but AP gain).

I think this would at least help to bring balance to the population issue in campaigns and solve problems of one faction controlling the entire map. A problem that already exists or at least existed (buff campaigns) and leads to dead campaigns.
My AP idea would only be viable with a pop lock system. Otherwise it would become a terrible AP farm "exploit".

Removing all AoE skills was btw. not meant entirely seriously and more a sign of my resignation on the balancing of this game. What probably would be more viable ist to set a fixed AoE cap (full damage and heal) to group size (24?) and not scale it. This reduces server load due to less calculations that need to be handled and all skills would have the same game mechanic (no varying by skill like it is right now). Group heals should also only work on group members.

They also really should start reviewing all ability costs. Right now the ratio of cost / efficiency is totally imbalanced, comparing the skill lines and AoE vs. single target dps / heals.

I wished they were better in balancing the game and finding smart simple solutions to address the problems, since right now it cannot live up to its premise at all.

Yes..the 4th faction.
Suggested the same thing myself at one point.
The 4th faction become a surplus buffer for everyone beyond equality range.
ie.
90 AD
100 EP
110 DC
200 AD 4th faction (Imperial or Rogue or Daedric)

I do foresee an issue even here though where the 4th faction that would receive friendly fire from their own faction can still work in league with their own faction regardless.
So it become... 290 AD vs 100 EP vs 110 DC.....and then we are back to square one.
/shrugs.... I like the idea but it will be abused too easily.

Fair point about the AoE in jest. I feel the same way these days.

They are moving over to group only AoE where it ca be done.
They have started limiting the worst AoE target cap offenders too...some of which have been dropped to 6 without any scaling.
But then they do things like completely uncapped siege, prox det and VD......and then I sit there scratching my head thinking WTF just happended ?!?!

Balancing.....single target efficiency vs aoe efficiency.
Doesn't need to be merited with a comment really...everyone knows the skills everyone is adopting.
Speaks for itself.
The endless hunt for the most OP/broken skill/gear/configs you can setup + L2P noob get rekt....rather than hey, it doesn't matter.
Play as you want...you wont be gimped whatever combo you use.
As long as you play to your strengths.

That fact everyone is adopting the most broke op AoE they could use is one of the reasons behind the idea of the highly restrictive cap when populations are full.
Hopefully it will show people that there is another way to play....without the lag.
Hopefully it would return us to the noobville nirvana before pain trains, zerg balls and bomb groups.

AlexHo1982 · April 2016

Rune_Relic wrote: »

Yes..the 4th faction.
Suggested the same thing myself at one point.
The 4th faction become a surplus buffer for everyone beyond equality range.
ie.
90 AD
100 EP
110 DC
200 AD 4th faction (Imperial or Rogue or Daedric)

I do foresee an issue even here though where the 4th faction that would receive friendly fire from their own faction can still work in league with their own faction regardless.
So it become... 290 AD vs 100 EP vs 110 DC.....and then we are back to square one.
/shrugs.... I like the idea but it will be abused too easily.

Since I agree with everything else you said, please just let me clarify above point.

I was actually not thinking of a fourth faction. I was going more into the following direction: Let's assume you have 200 AD, 150 EP and 50 DC in a campaign, AD gets locked and you will have to wait in line until EP and DC catch up to 200 or AD drops to the second highest population on the server. There those factions get locked until the third faction has caught up. This will permanently continue and rotate automatically as necessary.
However, instead of not being able to play PvP in this campaign at all while your faction is pop. locked, your AD character gets the option to join the lowest populated faction (DC in this example ) instead as a mercenary. The available faction for you instead of queuing for your own faction (waiting is only possible in the own faction to avoid other exploits), will always depend on which side has the lowest amount of players.

I believe that this avoids what you described above and would give an incentive to self balance the population for the players, since they can chose to gain AP, instead of waiting. The majority of players are not going for Emp. anyway, so this would not be a problem. The only downside would be, not being able to receive campaign rewards. On the other hand the monster set vendor in Cyrodil at least gives you rewards for AP and while you are waiting in a queue you are not get anything anyway.

Rune_Relic · April 2016

AlexHo1982 wrote: »

Rune_Relic wrote: »

Yes..the 4th faction.
Suggested the same thing myself at one point.
The 4th faction become a surplus buffer for everyone beyond equality range.
ie.
90 AD
100 EP
110 DC
200 AD 4th faction (Imperial or Rogue or Daedric)

I do foresee an issue even here though where the 4th faction that would receive friendly fire from their own faction can still work in league with their own faction regardless.
So it become... 290 AD vs 100 EP vs 110 DC.....and then we are back to square one.
/shrugs.... I like the idea but it will be abused too easily.

Since I agree with everything else you said, please just let me clarify above point.

I was actually not thinking of a fourth faction. I was going more into the following direction: Let's assume you have 200 AD, 150 EP and 50 DC in a campaign, AD gets locked and you will have to wait in line until EP and DC catch up to 200 or AD drops to the second highest population on the server. There those factions get locked until the third faction has caught up. This will permanently continue and rotate automatically as necessary.
However, instead of not being able to play PvP in this campaign at all while your faction is pop. locked, your AD character gets the option to join the lowest populated faction (DC in this example ) instead as a mercenary. The available faction for you instead of queuing for your own faction (waiting is only possible in the own faction to avoid other exploits), will always depend on which side has the lowest amount of players.

I believe that this avoids what you described above and would give an incentive to self balance the population for the players, since they can chose to gain AP, instead of waiting. The majority of players are not going for Emp. anyway, so this would not be a problem. The only downside would be, not being able to receive campaign rewards. On the other hand the monster set vendor in Cyrodil at least gives you rewards for AP and while you are waiting in a queue you are not get anything anyway.

So home/guest campaigns are a preference or best effort thing depending on population balance.
Clever.
I'd still worry about treacherous AD put into DC with another campaign though or insert factions of your choice.
Pointing out all enemy positions, where all the ambushes are, what the chat is in faction zone, joining groups to divulge intel to the enemy, giving scrolls to the enemy...etc...
Still... we have that now anyway

Merlight · April 2016

Rune_Relic wrote: »

KenaPKK wrote: »

Without AoE caps, each player hit is a single server side damage calculation.

With AoE caps, the server has a much higher load per person inside an AoE spell, even if fewer take damage. This is because the server must first count the players within the AoE, then it must choose ones for each tier of AoE damage (full, half, quarter, etc), and then calculate the damage amount for each player struck based on those tiers. That last step along outnumbers the calculations done without AoE caps simply because the full damage load is calculate for each player, and then some number of players receive an additional division step as the AoE cap reduces their damage.

It seems you do not fully understand the complexity of the code structure.

So like I said.
The piece you are trying to remove is step 3 above which probably accounts for about 1% of the code involved in each client update.
The remaining 99% of the code is repeated for every player hit like above.
10 players....10x loop code.... 10x traffic...database lockdown
100 players....100x loop code....100x traffic...extensive database lockdown

EDIT: reviewed attacker costing update outside loop.

I agree with a lot of what you're saying about the state of PvP, the big picture. When it comes to implementation details, I disagree, and since you keep repeating it, I'll keep refuting it. Let me revise your list of steps with how I think it should work:

Rune_Relic wrote: »

First is to receive an attack packet from the client.
Step 1 is to see who if anyone is going to receive that damage and then build a list of everyone in the red/green circle or damage radius (accounting for LOS and such).

This is by far the most expensive operation when there are many people in an area. Everything else is just juggling with bytes.

Rune_Relic wrote: »

Step 2 is to filter the closest to the center of radius in order of radius or those with lowest health in the case of smart heals.

This is at least a partial sort, with O(n log m) complexity (n=number of players coming from step 1). If they're doing it correctly, then m=target cap and the complexity is actually O(n) since log m is constant (for a given ability). If they're doing it wrong, then m=n.

Rune_Relic wrote: »

Step 3 is to have a conditional branch (which is in the process of being removed) where group 1 receive damage multiplayer A, group 2 receive damage multiplyer B and group 3 receive damage multiplyer C.

This is the "AoE damage falloff", right? You're correct that this operation alone is cheap, and removing it wouldn't reduce the amount of work done per ability-cast. What you don't acknowledge is that this step helps people survive bathing in AoE, allowing them to spam more AoE. Consequently it generates much more server load than just the calculations it makes.

Rune_Relic wrote: »

Loop (x players hit using damage modifier in radius order) starts here to ensure concurrency....
Step 4 is then to collect all of the attackers stats and find the current status, buffs, passives and debuffs (dynamic and static).
Step 5 is then to get the 1st recipents status and find the current status, buffs, passives and debuffs (dynamic and static)

"Collecting" stats every time you're hit would be the stupidest implementation imaginable. That stuff has to be in memory, updated whenever you receive buff/debuff, swap weapon, change equipment, etc. (all of which happens much less often than you're hit). These two steps are mere memory accesses.

Rune_Relic wrote: »

Step 6 is then to get the results of the damage vs mitigation (allowing for all the transient states buff debuffs and modifiers including armour sets and such..blocking/dodging/stunned) and then apply them to the recipient record.

Pure arithmetic, simple.

Rune_Relic wrote: »

Step 7 is then to modify the recipient database.

You certainly don't want to go to the database with each hit.

Rune_Relic wrote: »

Step 8 is then to notify the recipient client.

Nope. Here you just queue this attack to be sent in the next packet to that client at the end of the server frame. No networking here, only synchronized memory copy (possible contention, though). Whether you're being hit by 1 person or 20 people, you'll receive their attacks in one packet (provided they fit into one, and all happen within one server frame).

Rune_Relic wrote: »

- Step 9....repeat for every other player that has been hit within the red/green circle.
...Loop end here
- Step 10 is then to charge the attacker the required costing including any modifiers that are currently in effect.
- Step 11 is then to modify the attacker database.
- Step 12 is then to notify the attacker client
- Step 13....send occasional full status packets to ensure client and server are in sync.

I have doubts about the last one... ever heard of health desync? j/k

Rune_Relic · April 2016

Merlight wrote: »

Rune_Relic wrote: »

KenaPKK wrote: »

Without AoE caps, each player hit is a single server side damage calculation.

With AoE caps, the server has a much higher load per person inside an AoE spell, even if fewer take damage. This is because the server must first count the players within the AoE, then it must choose ones for each tier of AoE damage (full, half, quarter, etc), and then calculate the damage amount for each player struck based on those tiers. That last step along outnumbers the calculations done without AoE caps simply because the full damage load is calculate for each player, and then some number of players receive an additional division step as the AoE cap reduces their damage.

It seems you do not fully understand the complexity of the code structure.

So like I said.
The piece you are trying to remove is step 3 above which probably accounts for about 1% of the code involved in each client update.
The remaining 99% of the code is repeated for every player hit like above.
10 players....10x loop code.... 10x traffic...database lockdown
100 players....100x loop code....100x traffic...extensive database lockdown

EDIT: reviewed attacker costing update outside loop.

I agree with a lot of what you're saying about the state of PvP, the big picture. When it comes to implementation details, I disagree, and since you keep repeating it, I'll keep refuting it. Let me revise your list of steps with how I think it should work:

Rune_Relic wrote: »

First is to receive an attack packet from the client.
Step 1 is to see who if anyone is going to receive that damage and then build a list of everyone in the red/green circle or damage radius (accounting for LOS and such).

This is by far the most expensive operation when there are many people in an area. Everything else is just juggling with bytes.

Rune_Relic wrote: »

Step 2 is to filter the closest to the center of radius in order of radius or those with lowest health in the case of smart heals.

This is at least a partial sort, with O(n log m) complexity (n=number of players coming from step 1). If they're doing it correctly, then m=target cap and the complexity is actually O(n) since log m is constant (for a given ability). If they're doing it wrong, then m=n.

Rune_Relic wrote: »

Step 3 is to have a conditional branch (which is in the process of being removed) where group 1 receive damage multiplayer A, group 2 receive damage multiplyer B and group 3 receive damage multiplyer C.

This is the "AoE damage falloff", right? You're correct that this operation alone is cheap, and removing it wouldn't reduce the amount of work done per ability-cast. What you don't acknowledge is that this step helps people survive bathing in AoE, allowing them to spam more AoE. Consequently it generates much more server load than just the calculations it makes.

Rune_Relic wrote: »

Loop (x players hit using damage modifier in radius order) starts here to ensure concurrency....
Step 4 is then to collect all of the attackers stats and find the current status, buffs, passives and debuffs (dynamic and static).
Step 5 is then to get the 1st recipents status and find the current status, buffs, passives and debuffs (dynamic and static)

"Collecting" stats every time you're hit would be the stupidest implementation imaginable. That stuff has to be in memory, updated whenever you receive buff/debuff, swap weapon, change equipment, etc. (all of which happens much less often than you're hit). These two steps are mere memory accesses.

Rune_Relic wrote: »

Step 6 is then to get the results of the damage vs mitigation (allowing for all the transient states buff debuffs and modifiers including armour sets and such..blocking/dodging/stunned) and then apply them to the recipient record.

Pure arithmetic, simple.

Rune_Relic wrote: »

Step 7 is then to modify the recipient database.

You certainly don't want to go to the database with each hit.

Rune_Relic wrote: »

Step 8 is then to notify the recipient client.

Nope. Here you just queue this attack to be sent in the next packet to that client at the end of the server frame. No networking here, only synchronized memory copy (possible contention, though). Whether you're being hit by 1 person or 20 people, you'll receive their attacks in one packet (provided they fit into one, and all happen within one server frame).

Rune_Relic wrote: »

- Step 9....repeat for every other player that has been hit within the red/green circle.
...Loop end here
- Step 10 is then to charge the attacker the required costing including any modifiers that are currently in effect.
- Step 11 is then to modify the attacker database.
- Step 12 is then to notify the attacker client
- Step 13....send occasional full status packets to ensure client and server are in sync.

I have doubts about the last one... ever heard of health desync? j/k

Step1
You state the list building is the "most" expensive part rather than the loop.
Without seeing the code and infrastructure none of us can state that.
If you want to offer up the actual code to prove your point then by all means...please do.
It is no doubt more expensive than most of the activites.
On that we will agree....except for the loop.
They have stated on record this part is expensive anyway

Step3.
I have never argued about the removal of falloff, only unlimited AoE caps (removal).
This is part of the loop Paradox.
You are telling the server to hit more people....in order to reduce the amount of people being hit (awesome).
Yet....you see no problem with that logic.
And it ONLY WORKS if you actually kill people instantly = godmode + TTK issue.
It appears I am the only one that acknowledges the logic failure of hitting an unlimted amount of people...and MAYBE killing them.
Even though they get straight back into the fight with rez/camps making the ability to kill people enmasse instantly (to reduce load) a moot point.
You also seem to be completely oblivious to the physical capabilities of the design which creates lag in the 1st place.
ie...EXCEEDING THOSE LIMITS ! Which you appear to want to exceed even further.

Step 4&5
In a flag situation any player is being hit repeatedly with your unlimited caps.
Multiplied by everyone on the flag and hitting the flag with AoE.
Should be in memory, in an ideal world, with unlimited physical memory to hold 1000s of peoples details all at once ?
Maybe they have the hardware and clustering software (MPI) for that...maybe they cache x% of the playerbase...maybe they commit... maybe they use network ram disks...maybe they use distributed computing....who knows.
I dont for sure....just making a rough guess.
With 1000s of attacks happening almost simultaneously on what might be more than one server,
ensuring that data is concurrent across multiple servers might not allow for a player to exist in "local" RAM.
I worked on the assumption not all players can be in memory.

Step 7.
You might not have a choice.
Across multiple servers this might be the only way to ensure concurrency.

Step 8.
No networking ? How on earth do you update a client on a completely different network across the internet ?
Neither of us know the exact messaging setup ZOS uses.
Even routing protocols use instant (on change) + periodic (sync) updates though.
Which were 'ground up' designed to make the most optimal use of network bandwidth and latency.
How many client can be notified with one packet depends on the netcode we have no access to.
It may be 1...it may be 100.
Is it better to send 1 large packet copied to 100 people or 100 unique small packets to 100 people.
Regardless...you still have to use the bandwidth to accomodate 100 updated players.
I would rather send 2 packets (aoe cap = 1) than 101 (aoe cap = 100).
Plus the client needs to be notified immediately about any changes that specifically effect them.
Plus we still dont know how this is affected by clustering.

Step 9.
Not really surpising periodic updates cant get through if dyamic updates cant get through either.
Plus we get rubberbanding so we know periodic updates overwrite client data.
At the very least we know zoning syncs CP and XP points.
Joking aside

Merlight · April 2016

Rune_Relic wrote: »

Step1
You state the list building is the "most" expensive part rather than the loop.
Without seeing the code and infrastructure none of us can state that.
If you want to offer up the actual code to prove your point then by all means...please do.
It is no doubt more expensive than most of the activites.
On that we will agree....except for the loop.
They have stated on record this part is expensive anyway

Yes, I'm 100% guessing. All actors in Cyrodiil (players, NPCs, walls) have to be kept in a data structure that allows spatial queries. I'm assuming octree -- it's simple, yet efficient, and Wheeler's drawing on ESO live suggests they might be using that. When an AoE hits, it covers several cells containing potential targets. For each potential target, you must check that they're alive, not too far away, and for some abilities line of sight -- and that's what I think is the most expensive part. You need to check every potential target in the area, regardless of what the AoE cap is, you pay the price for everyone who shows up to the party.

From this first step filter you get a list of hit targets, unto whom later steps apply a sequence of actions. It seems to me that all the operations on targets that actually get hit are relatively simple, compared to the spatial and line of sight filter.

Rune_Relic wrote: »

Step3.
I have never argued about the removal of falloff, only unlimited AoE caps (removal).
This is part of the loop Paradox.
You are telling the server to hit more people....in order to reduce the amount of people being hit (awesome).
Yet....you see no problem with that logic.
And it ONLY WORKS if you actually kill people instantly = godmode + TTK issue.
It appears I am the only one that acknowledges the logic failure of hitting an unlimted amount of people...and MAYBE killing them.
Even though they get straight back into the fight with rez/camps making the ability to kill people enmasse instantly (to reduce load) a moot point.
You also seem to be completely oblivious to the physical capabilities of the design which creates lag in the 1st place.
ie...EXCEEDING THOSE LIMITS ! Which you appear to want to exceed even further.

Whether AoE is hard-capped at 60 or uncapped is irrelevant. That hard-cap will hardly ever be reached, and when it's reached, you've already paid the price of 60+ LoS checks and the server is crumbling.

I never advocated for instant kills. There's a middle ground between "people never drop" and "people drop instantly", you know? But when a character finally dies, you don't need to do any LoS checks on them for a while. And if they resurrect at another keep or camp, they're out of this cell, further offloading the first step. Yes they will return, but by then someone else will have died...

Rune_Relic wrote: »

Step 4&5
In a flag situation any player is being hit repeatedly with your unlimited caps.
Multiplied by everyone on the flag and hitting the flag with AoE.
Should be in memory, in an ideal world, with unlimited physical memory to hold 1000s of peoples details all at once ?
Maybe they have the hardware and clustering software (MPI) for that...maybe they cache x% of the playerbase...maybe they commit... maybe they use network ram disks...maybe they use distributed computing....who knows.
I dont for sure....just making a rough guess.
With 1000s of attacks happening almost simultaneously on what might be more than one server,
ensuring that data is concurrent across multiple servers might not allow for a player to exist in "local" RAM.
I worked on the assumption not all players can be in memory.

Seriously, do you have an idea of how much memory is needed to hold the stats and buffs of one character? I'm going to greatly exaggerate and say it's 10kB. Throw in 1000 people and 1000 NPCs, that makes 20MB!

Character data are a drop in an ocean; for example terrain data needed for LoS checks, which need to be kept in memory as well, are probably hundreds of megabytes.

Rune_Relic wrote: »

Step 7.
You might not have a choice.
Across multiple servers this might be the only way to ensure concurrency.

1 campaign = 1 server. Theoretically 1 server could host multiple campaigns, but they said buying more servers won't remove lag, so I assume 1 server hosts 1 campaign.

Rune_Relic wrote: »

Step 8.
No networking ? How on earth do you update a client on a completely different network across the internet ?
Neither of us know the exact messaging setup ZOS uses.
Even routing protocols use instant (on change) + periodic (sync) updates though.
Which were 'ground up' designed to make the most optimal use of network bandwidth and latency.
How many client can be notified with one packet depends on the netcode we have no access to.
It may be 1...it may be 100.
Is it better to send 1 large packet copied to 100 people or 100 unique small packets to 100 people.
Regardless...you still have to use the bandwidth to accomodate 100 updated players.
I would rather send 2 packets (aoe cap = 1) than 101 (aoe cap = 100).
Plus the client needs to be notified immediately about any changes that specifically effect them.
Plus we still dont know how this is affected by clustering.

You completely missed my point. You don't send any data over the network from this loop. This step throws a message in a per-client buffer, i.e. when 3 enemies are attacking Joe, those 3 run this loop, each puts "received X damage" in Joe's buffer (and "dealt X damage" in their buffers), and at the end of the server frame, the 4 buffers (Joe's containing 3 messages, enemies containing 1 message) are sent. Each client receives different set of updates (the 3 enemies usually don't deal the exact same damage), you can't really broadcast this stuff.

Rune_Relic · April 2016

Merlight wrote: »

Rune_Relic wrote: »

Step1
You state the list building is the "most" expensive part rather than the loop.
Without seeing the code and infrastructure none of us can state that.
If you want to offer up the actual code to prove your point then by all means...please do.
It is no doubt more expensive than most of the activites.
On that we will agree....except for the loop.
They have stated on record this part is expensive anyway

Yes, I'm 100% guessing. All actors in Cyrodiil (players, NPCs, walls) have to be kept in a data structure that allows spatial queries. I'm assuming octree -- it's simple, yet efficient, and Wheeler's drawing on ESO live suggests they might be using that. When an AoE hits, it covers several cells containing potential targets. For each potential target, you must check that they're alive, not too far away, and for some abilities line of sight -- and that's what I think is the most expensive part. You need to check every potential target in the area, regardless of what the AoE cap is, you pay the price for everyone who shows up to the party.

From this first step filter you get a list of hit targets, unto whom later steps apply a sequence of actions. It seems to me that all the operations on targets that actually get hit are relatively simple, compared to the spatial and line of sight filter.

Rune_Relic wrote: »

Step3.
I have never argued about the removal of falloff, only unlimited AoE caps (removal).
This is part of the loop Paradox.
You are telling the server to hit more people....in order to reduce the amount of people being hit (awesome).
Yet....you see no problem with that logic.
And it ONLY WORKS if you actually kill people instantly = godmode + TTK issue.
It appears I am the only one that acknowledges the logic failure of hitting an unlimted amount of people...and MAYBE killing them.
Even though they get straight back into the fight with rez/camps making the ability to kill people enmasse instantly (to reduce load) a moot point.
You also seem to be completely oblivious to the physical capabilities of the design which creates lag in the 1st place.
ie...EXCEEDING THOSE LIMITS ! Which you appear to want to exceed even further.

Whether AoE is hard-capped at 60 or uncapped is irrelevant. That hard-cap will hardly ever be reached, and when it's reached, you've already paid the price of 60+ LoS checks and the server is crumbling.

I never advocated for instant kills. There's a middle ground between "people never drop" and "people drop instantly", you know? But when a character finally dies, you don't need to do any LoS checks on them for a while. And if they resurrect at another keep or camp, they're out of this cell, further offloading the first step. Yes they will return, but by then someone else will have died...

Rune_Relic wrote: »

Step 4&5
In a flag situation any player is being hit repeatedly with your unlimited caps.
Multiplied by everyone on the flag and hitting the flag with AoE.
Should be in memory, in an ideal world, with unlimited physical memory to hold 1000s of peoples details all at once ?
Maybe they have the hardware and clustering software (MPI) for that...maybe they cache x% of the playerbase...maybe they commit... maybe they use network ram disks...maybe they use distributed computing....who knows.
I dont for sure....just making a rough guess.
With 1000s of attacks happening almost simultaneously on what might be more than one server,
ensuring that data is concurrent across multiple servers might not allow for a player to exist in "local" RAM.
I worked on the assumption not all players can be in memory.

Seriously, do you have an idea of how much memory is needed to hold the stats and buffs of one character? I'm going to greatly exaggerate and say it's 10kB. Throw in 1000 people and 1000 NPCs, that makes 20MB! Character data are a drop in an ocean; for example terrain data needed for LoS checks, which need to be kept in memory as well, are probably hundreds of megabytes.

Rune_Relic wrote: »

Step 7.
You might not have a choice.
Across multiple servers this might be the only way to ensure concurrency.

1 campaign = 1 server. Theoretically 1 server could host multiple campaigns, but they said buying more servers won't remove lag, so I assume 1 server hosts 1 campaign.

Rune_Relic wrote: »

Step 8.
No networking ? How on earth do you update a client on a completely different network across the internet ?
Neither of us know the exact messaging setup ZOS uses.
Even routing protocols use instant (on change) + periodic (sync) updates though.
Which were 'ground up' designed to make the most optimal use of network bandwidth and latency.
How many client can be notified with one packet depends on the netcode we have no access to.
It may be 1...it may be 100.
Is it better to send 1 large packet copied to 100 people or 100 unique small packets to 100 people.
Regardless...you still have to use the bandwidth to accomodate 100 updated players.
I would rather send 2 packets (aoe cap = 1) than 101 (aoe cap = 100).
Plus the client needs to be notified immediately about any changes that specifically effect them.
Plus we still dont know how this is affected by clustering.

You completely missed my point. You don't send any data over the network from this loop. This step throws a message in a per-client buffer, i.e. when 3 enemies are attacking Joe, those 3 run this loop, each puts "received X damage" in Joe's buffer (and "dealt X damage" in their buffers), and at the end of the server frame, the 4 buffers (Joe's containing 3 messages, enemies containing 1 message) are sent. Each client receives different set of updates (the 3 enemies usually don't deal the exact same damage), you can't really broadcast this stuff.

We are talking at cross purposes I think.
I will reply tommorrow when I am less tired.

Rune_Relic · April 2016

@Melright

And for those trying to follow I shall try to explain everything as best I can.
I make no apologies for this wall of text, as its my exhaustive final thoughts along with the reasoning and explanations:

Access Latency.

1. Supercomputers have local ram and higher latency remote RAM for SMP via an on chip interconnect.
2. Clusters have similar (cheaper) CPU power at the expense of (even higher latency) far RAM access when using shared memory across seperate servers.
This internconnect access could be via backplane, fibre or copper depending on cost and infrastructure.
Essentially the RAM is abstracted across combined physical servers into a single unfied block.
http://www.enterprisetech.com/2014/06/30/shared-memory-clusters-101/
http://www.enterprisetech.com/2014/07/09/shared-memory-clusters-numa-cache-latencies/
http://www.enterprisetech.com/2014/07/21/shared-memory-clusters-observations-capacity-response-time/

The point of clustering is for the cpu intensive processes to be spread across servers to balance the processing or CPU load and run them in parallel.
The reason to do so is supercomputers are very expensive.
Ideally any storage those processes need/use will be working storage on local/remote RAM only.
Meaning only enough RAM memory volume is required for those locally isolated but distributed processes.
Ideally they bring minimal to no data with them, generate and store any data on local site (physical server).
In reality many processes have to talk and exchange memory volume across the network interconnects (physical servers).
Hence shared RAM across a network.
That network latency and bandwidth limits performance in comparison to supercomputers.

This has been true across the history of computing.
The greater the distance between storage and cpu...the higher the latency and slower the bandwidth.
Hence the use of compressed RAM techniques on windows 10 rather than use the swap disk when RAM is being exhausted.
Thus sacrificing CPU performance to increase storage access speeds.
Compromising CPU performance increases overall performance in this case.
As disk access was the bottleneck.
Nice idea in principal when its works properly.

Read-Only vs Write access.

The second problem is although many programs can/do generate a READ-ONLY copy/version of a volume of data to work on and use it for secondary processes,
(be it a file, database record or class instance),
there can only every be "one" AUTHORITIVE volume/chunk of data that is available for WRITE-ACCESS.
Its this "solitary" single-access authoritive source, across a finite infrastructure of any kind and size, that is the bottleneck.
Especially when you throw in the distance/latency between where the authoritive source is stored and the cpu that needs it.

To put it in context of modern object oriented programming;
there is only one authoritive version of each player.
There may be 100/1000s of inferior outdated READ-ONLY copies/versions/instances of my player being used for many purposes at exactly the same time.
But there is only one AUTHORITIVE master with current data.
AND only ONE process can "write" to this AUTHORITIVE MASTER at any time.
This is the way it has to be, to ensure the sytem knows which outdated version of the original is the most valid and current.

Concurrency.

Imagine you gap-close/stun then use skill/buff then heavy attack someone.
If the system decided to heavy attack (miss), skill/buff (miss) and then gap-close/stun instead, you'd be pretty annoyed and probably smacking that keyboard on the desk before rage quitting.
So the game server has to observe strict control over the order of events.
Everything has to be processed sequentially one action after another in timestamp order.
Not just from your point of view, but from the view of every player within PvP or PvE and especially the server cluster.

Concurreny vs Write access vs Far Memory.
(This is where we come to the major stumbling block of MMOs and AoE).

Imagine if you will that no one is attacking me or buffing me because there is only single target skills.
I attack other people OR buff myself (one or the other).
My instructions jump across the internet to the game server cluster.
It wants to do loads of stuff to other people and then update my AUTHORITIVE WRITE-LOCKED reference version of me (which may be on any clustered physical server anywhere depending on load).
As noone else is attacking me or buffing me...I get instant unchallenged access and sole use of this reference version of 'me' that becomes write-locked to everyone else.
My cost is removed from the resource stack and buffs applied to the 'authoritive me'.
Excellent. The server is really responsive and does exactly what I ask pretty much whenever I ask (internet network latency allowing).
There were 10 other poeple on screen but they neither buffed or attacked me.
As its sequential single target, there can never be anymore than 10 attacks OR 10 buffs going on at any time from the servers viewpoint.
My 'local version of me' thus tried changing the 'authoritive version of me' on the server over the internet.
Which then updated my inferior version of me in response, over the internet, if the change was valid (or over-ruled my current stats if I also did them clientside in parallel).

Now imagine those 10 people are going to attack me with single target skills when I buff myself or heal.
I am simply being gang ganked with Xv1. This is recently what some people consider a bomb group apparently

Every one of those 10 players and me now need to make changes to that sequential single access write locked authoritive reference file/object.
Now access to that data is contended. I have to share its access with 10 other players.
That means if it took 1ms to access and change my data it is now going to take 10x longer or 10ms before I get sole/any access to that object (authoritive me).
I cant get on with anything else as this would break the ordered sequence of events, where I am doing stuff on the client before I have completed stuff on the server.
At this same time, 10 other players have also tried updating my authoritve information.
So the server needs to notify me of those 10 sequential changes that were made to my data (over the internet), to make sure my healthbar and such are in sync as well as my resources.
Not so good but tolerable. My character responsiveness has slowed down and got a bit laggy. Not unplayable but a little annoying.
Access Latency was increased as was network bandwith with either the number of packets or the size of the packet.
Those 10 other players were neither being buffed nor attacked as I was the one being attacked.
Only 10 attacks could take place with single target....but only my access latency hit the fan.
Their access latency was perfect and unaffected....so their costing update was instant and uncontended for their authoritive version of them.
However, Each one of them also had to wait in a queue to update the authoritive version of me and my health resource or other data.
Thus minimal authoritive locks and server latency was in effect when viewed as a whole for all 11 players.
But 'my' network bandwidth and 'my' access latency took a hit.
The attackers latency might also have taken a hit depending on if they needed to wait for a 'successful attack' update or not.
[NB. If costing is applied 'only' after damage is 'successful', the attacker latency is also high.
If costing is applied out of sequence (with possible concurrency issues), then attacker latency is low as they dont have to wait].

Now imagine I am in a confined area and all 10 other players are going to hit each other with uncapped AoE.
At this point everyone will attack everyone at the same time.
From my point of view I am still seeing 10 incoming hits + my cost/buff, just like the last paragraph scenario.
Except now ...10 other players are also experiencing the exact same authoritive version access latency issues....and their own increased client update size or frequency, burning network bandwidth.
Now we end up with a write-locking storm on the server, plus a 10 fold increase in CPU load/traffic over the single target version plus a 10 fold increase in the packet size or frequency that have to enter and exit the server to the clients.
Its now going to take upto 10x longer for every player to access their own authoritive version while other players lock them out.
Its now going to use upto 10x more bandwidth than single target updates for every player to ensure every action occurs independantly and in sequence (10 events rather than 1).
Its now going to create upto 10x the single target cycle load on the server as 10x10 players are being hit rather than 10x1.
Its now going to generate upto 10x more bandwidth than single target on the server itself, to cover the 10x10 (players x hits) events rather than the 10x1 events

Now scale this up to 60 player target caps (or unlimited in the case of siege) and populations of several 100.
Now put the bulk of those players in one single area at the same time.
With 60 cap the worst case is several 100 x 60.
With no cap the worst case is not worth pondering because the server will have long crashed or frozen.

TL;DR
1. Find the maximum campaign population that generates zero lag, with a per campaign, PvP only, AoE cap of 1. Use this as the performance base line.
2. Use campaign population to vary the PvP only, AoE caps, so that the cap x campaign population does not exceed the player x hits lagfree baseline.
3. Optionally use variable, per faction, per camapign, PvP only, AoE cap (to make up for population imbalance and faction dominance), while ensuring the server remains lagfree.
4. Queues or not....the more attacks ....the longer I have to wait for service. Pretending access is instant because i am 'instantly' stuffed at the back of an evergrowing queue is nothing more than a strawman.
5. Take all this with a pinch of salt. Its a best guess. No one but ZOS really knows how their megaserver infrastructure and netcode is built and organised physically and logically.