Server Multithreading Update – July 2023

FrankonPC

blktauna wrote: »

yes, so who won the pool on that. The bets have been going since the delayed xbox server announcement.

This is why there is no trust. They just sweep it aay and hope people move on rather than directly addressing the issues.

I think I did. I called out that the majority of the time that Mr. Firor posts something it goes on uncompleted.

Sadly the re-architecture time period revolved around a rather tumultuous time for ESO and it looked then like it looks now:

A way to distract the player base from the direction the game is heading.

I would love to be wrong though and would love to see if the re-architecture that they had said is different from the multi-threading they're bringing up now.

blktauna

I guess that means the pool is back on.

FrankonPC wrote: »

blktauna wrote: »

yes, so who won the pool on that. The bets have been going since the delayed xbox server announcement.

This is why there is no trust. They just sweep it aay and hope people move on rather than directly addressing the issues.

I think I did. I called out that the majority of the time that Mr. Firor posts something it goes on uncompleted.

Sadly the re-architecture time period revolved around a rather tumultuous time for ESO and it looked then like it looks now:

A way to distract the player base from the direction the game is heading.

I would love to be wrong though and would love to see if the re-architecture that they had said is different from the multi-threading they're bringing up now.

You and me both, man. But we both know the reality.

Lags

what an awful night in gray host. Unbelievably laggy, and still zos has its head in the sand ignoring everyone in this thread. Nice.

Janni

laniakea_0 wrote: »

TaSheen wrote: »

Or the new IP....

doubt they are going to downsize the ESO dev team while it's still growing in popularity

These don't have to be mutually exclusive. If their new project is an MMO there would be a lot of incentive to try to develop a system that can be simultaneously used for both. If you are already spending the money anyway it would totally be worth trying to develop it in a way that works for ESO first since it means you get to try it in a live working game where real world metrics can be used to measure the success. The bonus is that you now have battle tested core game engine that can be used for your new project AND you get a currently live product working more efficiently with hardware to reduce operation costs in electricty all for the price of one.

reazea

virtus753 wrote: »

SaffronCitrusflower wrote: »

I see now that they've just recently unpinned this discussion from the announcements section.

That’s odd. It still appears pinned in General for me. Maybe it’s a glitch?

Is the thread pinned or unpinned now?

virtus753

reazea wrote: »

virtus753 wrote: »

SaffronCitrusflower wrote: »

I see now that they've just recently unpinned this discussion from the announcements section.

That’s odd. It still appears pinned in General for me. Maybe it’s a glitch?

Is the thread pinned or unpinned now?

It is still pinned for me. My understanding is that it was unpinned but then repinned after a few people commented, and I missed that between forum visits.

Darethran

I've gone through a repeated cycle with the game. Gear up, get into PvP, quit because the lag and latency becomes unbearable. Forget how bad it was about a year later, repeat.

Don't think I've seen it quite as awful as this in just the PvE content though, on EU server.

Matt said during the NA hardware update, that ESO's servers were running Sandybridge CPUs and DDR3 RAM (2012-era hardware). Do we know what the new hardware is? If Zenimax is using Zen 3 Epycs then server performance would be a several times improvement. But if using crummy Intel Atom-tier hardware, it's not going to be much better if at all.

I_killed_Vivec

It ain't working.

NorfolkXX

StihlReign

We'll know it's working when it's fast enough to release us from combat.

I_killed_Vivec

StihlReign wrote: »

We'll know it's working when it's fast enough to release us from combat.

We'll know when it's working when it will let us log in and get to fight!

Paske

The lovely 1 hour + que states that you did a good job Matt

/s

ProudMary

More server resources please. It's obvious the game will benefit greatly from more servers.

ioResult

Hey I know what will improve server performance! Let's go a free ESO Plus event [snip] What a great idea to improve server performance!

[snip]

The cause of lag is simple Matt. I don't understand WHY it is so difficult for your team to understand. The cause of the lag is that in PvP where you have more than 20 units in an area, the server must attempt to calculate the effects of AoE attacks and defenses within the single global cooldown period and the way the combat abilities are designed now, those calculations and spin locks (for lack of a better term) they cause because the server can't do them all in one global cooldown.

Who causes all of these AoE calculations to be necessary EVERY SINGLE global cooldown?

BALL GROUPS!

All they do is spam AoEs, [snip] Spam AoE heals CONSTANTLY until everyone has Ulti enough to all drop *AoE* Ultis and other attacks simultaneously.

[snip]

When you have 100+ units (players and NPCs) in an area doing AoE attacks and heals, the server has to check EACH AND EVERY UNIT just to see if its in the AoE area and then if the AoE attack or heal affects each unit. If it affects a unit then the server has to generate a random number in a certain range which had to be calculated when the caster cast the ability, so if its an instant ability that calculation also had to be done within the same global cooldown. Then once the server decides an AoE ability did affect a particular unit then it needs to compute a random number to figure out if it was a crit and then another to figure out the damage to that unit. And then it also needs to do more calculations to determine if Status effects proc'd. Oh and lets throw in if the attack proc'd a proc set. No wonder they put a 1 second delay on proc sets because the server needs to do all those calculations next cooldown for THAT AoE proc which just proc'd as a result of the AoE attack.

The solution is simple.

Record everything that is going on when ball groups are running around. Profile each ability they spam. It'll be easy to see abilities they spam. I will be any money that any ZOS dev wants to bet me that the number one thing clogging the global cooldown is the Resto staff that seemingly everyone in a Cyrodiil, let alone the ball groups, has on their back bar.

Everything that skill line does is pure AoE! Make one simple change to all those abilities, just one, and that is that Resto staff abilities are no longer pure AoE. Change them all from saying "you and your allies" to say "you and your group members" and watch lag in Cyrodiil decrease.

Then change a version of each Resto staff ability so it is significantly better for self-heals rather than group heals so that most people using Resto staves in Cyrodiil pick THAT morph instead. Why? Because people who are truly "solo" (what I call "zerg-adjacent") will always choose that morph and the ball group [snip] will still think they can PvP as the other version of each ability will still heal the people in their group.

This change to one weapon skill line will significantly reduce per GCD AoE calculations in Cyrodiil.

Outside of Cyrodiil these changes will not affect PvE at all. The Resto staff abilities are not used outside of a group in any instanced content like Dungeons, Trials, Arenas outside of Public Dungeons, where there is not any reason for a resto staff to be healing some rando in your Resto staff's AoE.

Besides the Resto staff abilities, Echoing Vigor is the other big "you and your allies" offender. So change this ability to be "you and your group" too.

This means all AoE calculations will be offensive.

Which leads to the other part of what I believe is the lag, all the AoE calculations necessary when people are throwing SIEGE!

Why are people throwing siege? Because as the game stands now, it is the only way to TRY to kill these [snip] ball groups in Cyrodiil. They heal through everything else. If it didn't take 20 people pouring oils and firing Coldfire siege just to TRY (and usually failing) to kill the ball groups, you wouldn't have all this siege contributing to the GCD AoE calculation overload that produces the lag.

What is the solution to this?

Simple, put stacking caps on these healing abilities!

In PvE there really aren't more than 2 healers in a Trial group or one in a Dungeon/Arena group. So make two the cap on stacking HoTs! I watch YouTube content of some of the ball group members who think they can actually PvP and on their screen we can SEE that there are seven, eight, even TEN stacks of Echoing Vigor on them. Cap that nonsense at two stacks and make it an in-group only heal and then we can see which ball groups can actually farm inside a keep or while running the walls of the keep. ZOS you've made ball groups EZ Mode AP farming which is why more and more players turn to it. Make it a real challenge for the folks who want to perfect the playstyle and at the same time you'll be reducing lag even more.

[snip]

So, hopefully mentioning these suggestions in a public forum rather than just in bug reports might help get them to at least put them in the PTS.

Oh Matt, every PTS cycle come up with a reward, a reward that you get in your NORMAL account if you help come test PvP and Cyrodiil on the PTS. Copy everyone's current toons over to PTS, pick a day and a time period and get all the PvPers onto PTS to test changes you make. The more they help the more rewards they get, just like the Dragon Rising event back in the day. It's the only way to test what you're doing to fix this lag without [snip] people off when stuff you put on live doesn't work as you intended.

If I ran development for ZOS that's what I'd do. Hell if I worked for Microsoft, that's what I'd mandate that ZOS do.

[edited for bashing]

Lags

hey just another reminder about how bad performance is. 100 fps, 100 ping, both of which are normal for eso (but not normal compared to other mmos), and my normal internet speed, and cyrodil is just utterly terrible. So incredibly delayed for me and the 2 other players im with.

Nothing is fixed, nothing is better, there is no victory here. Its not ping, its not internet, its not fps. Its just an issue with the core of the game, and everytime there is a que into cyrodil it is bad. And chasing away pvpers by the boatloads, so that there is never a que, isnt a solution to the problem.

Hrafngrimr3

Other game studios are beginning to see the appeal of mmorpg games again and it’s only a matter of time before ZOS and TESO has some serious serious competition across all platforms. Their PvP system is unique but easily replaceable with a working system. Square Enix just announced the launch of FF14 on Xbox in spring 2024, Chrono Odyssey revealed first gameplay trailer a few months ago. I’ve loved playing ESO for many reasons but I’m not sure ZOS is keeping themselves relevant anymore - extremely poor communication (again and again), lack of accepting responsibility and/or feedback from the player base, falling behind in their ability to balance play between PVP and PVE separately (every other MMO does this) which feeds into frequent and sweeping changes which inhibits any actual conversation around what’s needed/wanted. This is on top of failing performance in both game zones (pvp and pve). I’ll throw my hands up on this one, but if something better comes along I’ll be one of the first to throw in the towel.

GooGa592

Hrafngrimr3 wrote: »

Other game studios are beginning to see the appeal of mmorpg games again and it’s only a matter of time before ZOS and TESO has some serious serious competition across all platforms. Their PvP system is unique but easily replaceable with a working system. Square Enix just announced the launch of FF14 on Xbox in spring 2024, Chrono Odyssey revealed first gameplay trailer a few months ago. I’ve loved playing ESO for many reasons but I’m not sure ZOS is keeping themselves relevant anymore - extremely poor communication (again and again), lack of accepting responsibility and/or feedback from the player base, falling behind in their ability to balance play between PVP and PVE separately (every other MMO does this) which feeds into frequent and sweeping changes which inhibits any actual conversation around what’s needed/wanted. This is on top of failing performance in both game zones (pvp and pve). I’ll throw my hands up on this one, but if something better comes along I’ll be one of the first to throw in the towel.

I've pre-puchased Ashes of Creation already.

DirtyDeeds765

ioResult wrote: »

Hey I know what will improve server performance! Let's go a free ESO Plus event [snip] What a great idea to improve server performance!

[snip]

The cause of lag is simple Matt. I don't understand WHY it is so difficult for your team to understand. The cause of the lag is that in PvP where you have more than 20 units in an area, the server must attempt to calculate the effects of AoE attacks and defenses within the single global cooldown period and the way the combat abilities are designed now, those calculations and spin locks (for lack of a better term) they cause because the server can't do them all in one global cooldown.

Who causes all of these AoE calculations to be necessary EVERY SINGLE global cooldown?

BALL GROUPS!

All they do is spam AoEs, [snip] Spam AoE heals CONSTANTLY until everyone has Ulti enough to all drop *AoE* Ultis and other attacks simultaneously.

[snip]

When you have 100+ units (players and NPCs) in an area doing AoE attacks and heals, the server has to check EACH AND EVERY UNIT just to see if its in the AoE area and then if the AoE attack or heal affects each unit. If it affects a unit then the server has to generate a random number in a certain range which had to be calculated when the caster cast the ability, so if its an instant ability that calculation also had to be done within the same global cooldown. Then once the server decides an AoE ability did affect a particular unit then it needs to compute a random number to figure out if it was a crit and then another to figure out the damage to that unit. And then it also needs to do more calculations to determine if Status effects proc'd. Oh and lets throw in if the attack proc'd a proc set. No wonder they put a 1 second delay on proc sets because the server needs to do all those calculations next cooldown for THAT AoE proc which just proc'd as a result of the AoE attack.

The solution is simple.

Record everything that is going on when ball groups are running around. Profile each ability they spam. It'll be easy to see abilities they spam. I will be any money that any ZOS dev wants to bet me that the number one thing clogging the global cooldown is the Resto staff that seemingly everyone in a Cyrodiil, let alone the ball groups, has on their back bar.

Everything that skill line does is pure AoE! Make one simple change to all those abilities, just one, and that is that Resto staff abilities are no longer pure AoE. Change them all from saying "you and your allies" to say "you and your group members" and watch lag in Cyrodiil decrease.

Then change a version of each Resto staff ability so it is significantly better for self-heals rather than group heals so that most people using Resto staves in Cyrodiil pick THAT morph instead. Why? Because people who are truly "solo" (what I call "zerg-adjacent") will always choose that morph and the ball group [snip] will still think they can PvP as the other version of each ability will still heal the people in their group.

This change to one weapon skill line will significantly reduce per GCD AoE calculations in Cyrodiil.

Outside of Cyrodiil these changes will not affect PvE at all. The Resto staff abilities are not used outside of a group in any instanced content like Dungeons, Trials, Arenas outside of Public Dungeons, where there is not any reason for a resto staff to be healing some rando in your Resto staff's AoE.

Besides the Resto staff abilities, Echoing Vigor is the other big "you and your allies" offender. So change this ability to be "you and your group" too.

This means all AoE calculations will be offensive.

Which leads to the other part of what I believe is the lag, all the AoE calculations necessary when people are throwing SIEGE!

Why are people throwing siege? Because as the game stands now, it is the only way to TRY to kill these [snip] ball groups in Cyrodiil. They heal through everything else. If it didn't take 20 people pouring oils and firing Coldfire siege just to TRY (and usually failing) to kill the ball groups, you wouldn't have all this siege contributing to the GCD AoE calculation overload that produces the lag.

What is the solution to this?

Simple, put stacking caps on these healing abilities!

In PvE there really aren't more than 2 healers in a Trial group or one in a Dungeon/Arena group. So make two the cap on stacking HoTs! I watch YouTube content of some of the ball group members who think they can actually PvP and on their screen we can SEE that there are seven, eight, even TEN stacks of Echoing Vigor on them. Cap that nonsense at two stacks and make it an in-group only heal and then we can see which ball groups can actually farm inside a keep or while running the walls of the keep. ZOS you've made ball groups EZ Mode AP farming which is why more and more players turn to it. Make it a real challenge for the folks who want to perfect the playstyle and at the same time you'll be reducing lag even more.

[snip]

So, hopefully mentioning these suggestions in a public forum rather than just in bug reports might help get them to at least put them in the PTS.

Oh Matt, every PTS cycle come up with a reward, a reward that you get in your NORMAL account if you help come test PvP and Cyrodiil on the PTS. Copy everyone's current toons over to PTS, pick a day and a time period and get all the PvPers onto PTS to test changes you make. The more they help the more rewards they get, just like the Dragon Rising event back in the day. It's the only way to test what you're doing to fix this lag without [snip] people off when stuff you put on live doesn't work as you intended.

If I ran development for ZOS that's what I'd do. Hell if I worked for Microsoft, that's what I'd mandate that ZOS do.

You wrote all of that just to be completely wrong. The solution isn't to restrict people from playing the game in the most optimal way. The solution is to fix the code and upgrade the hardware to the level the game requires.

@DirtyDeeds765

[edited to remove quote]

blktauna

But that plainly isn't going to happen so we'll need to work with what we have.

Wolfkeks

Not sure if it was asked already but are multithreading and the code-rewrite two different things or are they the same?
Haven't heard anything regarding the rewrite so that's why I'm asking ^^

RMW

Wolfkeks wrote: »

Not sure if it was asked already but are multithreading and the code-rewrite two different things or are they the same?
Haven't heard anything regarding the rewrite so that's why I'm asking ^^

Should be two seperate things

Four_Fingers

Well, I don't think I will be buying into the new IP or Starfield for that matter.

laniakea_0

Four_Fingers wrote: »

Well, I don't think I will be buying into the new IP or Starfield for that matter.

not the same studio...

blktauna

RMW wrote: »

Wolfkeks wrote: »

Not sure if it was asked already but are multithreading and the code-rewrite two different things or are they the same?
Haven't heard anything regarding the rewrite so that's why I'm asking ^^

Should be two seperate things

I agree, but after matt changed the wording to mulithreading no further mention was made about a rewrite.

ioResult

DirtyDeeds765 wrote: »

You wrote all of that just to be completely wrong. The solution isn't to restrict people from playing the game in the most optimal way. The solution is to fix the code and upgrade the hardware to the level the game requires.

You sound like you know a lot about how to "fix the code", so please explain how you would fix the GCD AoE problem given the time span of the GCD. I offered a way to fix it given the constraint of the GCD isn't going to change much no matter how much hardware you throw at it because round trip packet time will still become the limiter but if you have another suggestion for how to fix it, I'm all ears.

blktauna

No one can comment on the code as we hve no idea of the codebase or the DB schema.

Still they said they were going to address issues in pvp, yet after this 'multithreading' try, it looks like they've given up.
Very dissappointing ZOS

(ed. no longer pinned either)

laniakea_0

why do you think that? all he said was that it their efforts fell short of expectations and they had to undo some steps. it says nothing about them stopping the work on that all together. they'll continue working on it for the foreseeable future.

TaSheen

laniakea_0 wrote: »

why do you think that? all he said was that it their efforts fell short of expectations and they had to undo some steps. it says nothing about them stopping the work on that all together. they'll continue working on it for the foreseeable future.

Maybe. There will definitely be a point of diminishing returns though....

blktauna

This thread is unpinned. No further comment by ZOS was offered in the thread. No possible continuance was offered. And sadly, years of experience dealing with how ZOS aproaches things.

Zabagad

blktauna wrote: »

This thread is unpinned. No further comment by ZOS was offered in the thread. No possible continuance was offered. And sadly, years of experience dealing with how ZOS aproaches things.

I agree to your conclusion, but for me the thread is stilled pinned in "GENERAL ESO DISCUSSION"...