Server Multithreading Update – July 2023

etchedpixels

ghost_bg_ESO wrote: »

hmm... what was it : "adding more/new servers won't help performance..."


of course i know it has to be software and hardware optimization, but i wonder if marketing team understands that game servers can't be run on fridge door...

Unfortunately for complex multi process and multi-node systems it's usually the case that putting in something bigger doesn't actually help. Lots of things don't scale so the "We've put it on a 64 core threadripper" is followed by "and nothing changed". It works for overland sure because nobody notices if you have 1 Alik'r or 50 but not Cyrodiil

GooGa592

One question that needs to be answered by the dev team at ZOS:

How is it that in 2016-2018 ZOS was able to deliver fantastic performance in Cyrodiil with 24 man group size and quadruple the population cap than we have today, but in 2023 this is not possible?

Stamicka

Well I'm not surprised. [snip] You really shouldn't announce things that may not work out. I wonder if anything was ever in the works at all.

[edited for bashing]

h9dlb

How much fun PvP is? Lol even the scoreboards don't work

RicAlmighty

GooGa592 wrote: »

One question that needs to be answered by the dev team at ZOS:

How is it that in 2016-2018 ZOS was able to deliver fantastic performance in Cyrodiil with 24 man group size and quadruple the population cap than we have today, but in 2023 this is not possible?

This is the pertinent question. It’s not as though this game has never had good performance and this was an attempt to make it better. It *was* better, and now its worse. Why? What caused it? And what can we do to mitigate the performance degradation that those changes introduced?

FrankonPC

One last thing,

What's the difference between re-architecture and multi-threading?

RicAlmighty wrote: »

GooGa592 wrote: »

One question that needs to be answered by the dev team at ZOS:

How is it that in 2016-2018 ZOS was able to deliver fantastic performance in Cyrodiil with 24 man group size and quadruple the population cap than we have today, but in 2023 this is not possible?

This is the pertinent question. It’s not as though this game has never had good performance and this was an attempt to make it better. It *was* better, and now its worse. Why? What caused it? And what can we do to mitigate the performance degradation that those changes introduced?

This is exactly my argument when people bring up the old consoles as the reason why the game performance can't improve.

Performance WAS better back in 2018, on the older xbox compared to now on PC with a computer I bought last year. Why i have no idea, but performance really plummeted in early 2019

blktauna

FrankonPC wrote: »

One last thing,

What's the difference between re-architecture and multi-threading?

I've asked this several times to no avail. Once on stream and was timed out for 10 mins...

Arrow312

In German forum they deleted Posts in this thread

Finedaible

I honestly don't know if I can ever trust anymore. There's been years of empty promises, dead ends, and ever slower resolutions and I'm starting to wonder if we were just being fed hopium all this time. From the sound of it things are probably worse than we previously thought, but thanks for the update at any rate.

Wuduwasa13

vsrs_au wrote: »

ZOS_MattFiror wrote: »

Please remember that we must be very, very careful when making changes of this complexity to ESO’s massive codebase.

This is the most important part of the announcement. Refactoring code to add more support for multithreading is very difficult when the software is this large and complex, and it's so, so easy to introduce bugs in the process. It's easy to disparage ZOS for not having already done it, when you're not in the position of having to do such a difficult code update.

This rather pained update gives us very little information at all to alleviate our concerns around the present & persistent issues we as customers endure and instead leaves us with more questions than answers, a lot of uncertainty & little clarity.

Suggesting we ought to afford some type of privileged leniency in spite of our own very valid frustrations completely overlooks the myriad other issues surrounding how this is all being handled & is completely tone deaf to the majority of complaints and feedback being given here…which are by no means unwarranted nor are they unfair.

We as customers understand what is being said:

- It is a large & particularly complex project
- It has not progressed as initially anticipated
- It’s value has previously / is currently being re-evaluated.

What is missing:

- Details: why has performance degraded so noticeably? Many different reasons have been given to date only later to be discarded. Does ZOS even know or are they simply taking educated guesses or shots in the dark at this point?

- Setting expectations: what customers situated in the pvp community can expect in the foreseeable future. (If performance is going to remain terrible, acknowledge & own it, we will respect you far more for it).

- Next steps: Information & updates on other potentially viable solutions & possibilities being explored / considered?

- Whether or not ZOS intend to / are planning to introduce new content to the other PVP gameplay activities within ESO’s broad pvp definition (BG’s & IC predominately) whilst they review Cyrodil (admittedly a half-step, but better than leaving your customers ‘high & dry’ with nothing).

This to me is the bare minimum I would expect any type of subscription based paid service provider to give me if I was continuing to pay for their service & there was a marked decrease in quality (& by extension, value for money to me the customer)... it really isn’t extraordinary & plenty of other companies within the same industry manage it quite successfully I might add.

Marto

RicAlmighty wrote: »

GooGa592 wrote: »

One question that needs to be answered by the dev team at ZOS:

How is it that in 2016-2018 ZOS was able to deliver fantastic performance in Cyrodiil with 24 man group size and quadruple the population cap than we have today, but in 2023 this is not possible?

This is the pertinent question. It’s not as though this game has never had good performance and this was an attempt to make it better. It *was* better, and now its worse. Why? What caused it? And what can we do to mitigate the performance degradation that those changes introduced?

Rich Lambert actually talked about this during some of his livestreams.

It's not the only reason, but one of the biggest factors was the addition of Volendrung, and destructible gates/bridges. For some unknown reason, the addition of those things dramatically reduced Cyrodiil performance.

Paradoxically, removing those features in their internal test servers did not result in performance returning to the state it was before. Whatever was broken could not be un-broken.

We know from datamining that ZOS originally planned and developed more content for Cyrodiil, like more artifact weapons and more map features. But all that content had to be put on hold until performance was improved. That's why they've spent years and years doing multiple public and private tests with things like skill cooldowns, AoE limits, proc-sets, champion points, etc.

This is really not ZOS being "lazy" or trying to do things for cheap. The cheapest and easiest option would be for them to just release all that content, and care not how bad performance gets. Instead they spent years and years working on all sorts of improvements of hardware and software.

But honestly, I also feel people are looking at the past with rose tinted glasses. Performance in 2016-2018 was absolutely not better than it is right now. Different issues existed, like packet congestion as a result of too much data being handled by the server, poor lag compensation, and poor particle-related performance.

BlindingBright

PVP performance in Grey Host during sieges this past week have been a mixed bag. At the start of mayhem, ball groups didn't lag the game out enough to not be able to respond/use skills. As the event went on I noticed server performance during prime time getting worse.

Also noticed server side lag in trials(Along with others reporting it) was during primetime hours, and aligns with cyrodil performing like poo.

Also, the scoring system is broken for Mayhem. Not even a single word from ZoS about it during their last big pvp event of the year?! for shame. Hopefully we'll get a surprise PVP event later on to make up for this... mayhem. I just hope that if they were making changes to server software/hardware they gained some info from this years events.

Please fix PVP, ESO still has one of the best MMO PVP experiences... when it works.

VaranisArano

Marto wrote: »

RicAlmighty wrote: »

GooGa592 wrote: »

One question that needs to be answered by the dev team at ZOS:

How is it that in 2016-2018 ZOS was able to deliver fantastic performance in Cyrodiil with 24 man group size and quadruple the population cap than we have today, but in 2023 this is not possible?

This is the pertinent question. It’s not as though this game has never had good performance and this was an attempt to make it better. It *was* better, and now its worse. Why? What caused it? And what can we do to mitigate the performance degradation that those changes introduced?

Rich Lambert actually talked about this during some of his livestreams.

It's not the only reason, but one of the biggest factors was the addition of Volendrung, and destructible gates/bridges. For some unknown reason, the addition of those things dramatically reduced Cyrodiil performance.

Paradoxically, removing those features in their internal test servers did not result in performance returning to the state it was before. Whatever was broken could not be un-broken.

We know from datamining that ZOS originally planned and developed more content for Cyrodiil, like more artifact weapons and more map features. But all that content had to be put on hold until performance was improved. That's why they've spent years and years doing multiple public and private tests with things like skill cooldowns, AoE limits, proc-sets, champion points, etc.

This is really not ZOS being "lazy" or trying to do things for cheap. The cheapest and easiest option would be for them to just release all that content, and care not how bad performance gets. Instead they spent years and years working on all sorts of improvements of hardware and software.

But honestly, I also feel people are looking at the past with rose tinted glasses. Performance in 2016-2018 was absolutely not better than it is right now. Different issues existed, like packet congestion as a result of too much data being handled by the server, poor lag compensation, and poor particle-related performance.

I so wish that Rich Lambert had gotten to talk about this stuff from a more official, more accessible platform.

And yes, as to rose-colored glasses, I remember that Warden blizzards were really laggy when used en masse, and the Summerset MYM was a PITA because I was crashing during big defenses and missing the giant AP rewards.

Janni

blktauna wrote: »

FrankonPC wrote: »

One last thing,

What's the difference between re-architecture and multi-threading?

I've asked this several times to no avail. Once on stream and was timed out for 10 mins...

In short, large scale multi threading requires a different fundamental architecture. When they started developing this game it was still fairly uncommon to lean into multi threading as a way of improving performance due to the hardware of the time. So in order to make those changes now would require fundamentally changing the core of how the server software is designed... its architecture.

blktauna

Janni wrote: »

blktauna wrote: »

FrankonPC wrote: »

One last thing,

What's the difference between re-architecture and multi-threading?

I've asked this several times to no avail. Once on stream and was timed out for 10 mins...

In short, large scale multi threading requires a different fundamental architecture. When they started developing this game it was still fairly uncommon to lean into multi threading as a way of improving performance due to the hardware of the time. So in order to make those changes now would require fundamentally changing the core of how the server software is designed... its architecture.

I know this and you know this but they do not tell people this. If you are not a developer you are mystefied. My point is that they should be explaining this to the cusomer base as they know their own systems (one hopes) better than we can guess.

Initially the response was 'we need to re-do from the ground up', which is what I was expecting considering the age and state of the code. Then suddenly out of nowhere it became, 'no I nver said redo, we are just multithreading... ' So you see the concern. They have been saying they are working on performance since 2019... yet its always silence. No milestones are given, and if they are even hinted at they are then glossed over at every ask. Now after all these years theres the hint that nothing has worked.

Its immensly frustrating. If I treated my clients like this, I wouldn't have any. I don't think that's the goal here, so someone has got to lay out the plan so expectations are managed.

SimonThesis

Disappointed in this announcement to say the least.

Performance is not good on PCNA, server wide slowbugs, frequent crashes, skill input delay causing players to cast skills 5 times for them to go off, invisible aoes, desyncing Oil and Dot damage, frequent positional desyncs and Rollbacks, the dreaded stuck in combat bug. While performance was greatly improved for a few weeks following the new servers, performance rapidly degraded to preupgrade levels. If anything this most recent MYM should have shown how bad performance has gotten.

If the devs are under the impression performance is better now, we as a community need to start putting in those bug reports for cyrodiil.

@ZOS_MattFiror Will there be other avenues explored to improve performance? If not will major systems and qol improvements be added to cyro. If groups were halved to improve performance can they go back to 24? Will deer and destructable structures come back to cyrodiil?

Janni

blktauna wrote: »

Janni wrote: »

blktauna wrote: »

FrankonPC wrote: »

One last thing,

What's the difference between re-architecture and multi-threading?

I've asked this several times to no avail. Once on stream and was timed out for 10 mins...

In short, large scale multi threading requires a different fundamental architecture. When they started developing this game it was still fairly uncommon to lean into multi threading as a way of improving performance due to the hardware of the time. So in order to make those changes now would require fundamentally changing the core of how the server software is designed... its architecture.

I know this and you know this but they do not tell people this. If you are not a developer you are mystefied. My point is that they should be explaining this to the cusomer base as they know their own systems (one hopes) better than we can guess.

Initially the response was 'we need to re-do from the ground up', which is what I was expecting considering the age and state of the code. Then suddenly out of nowhere it became, 'no I nver said redo, we are just multithreading... ' So you see the concern. They have been saying they are working on performance since 2019... yet its always silence. No milestones are given, and if they are even hinted at they are then glossed over at every ask. Now after all these years theres the hint that nothing has worked.

Its immensly frustrating. If I treated my clients like this, I wouldn't have any. I don't think that's the goal here, so someone has got to lay out the plan so expectations are managed.

Yeah. I can't really argue with you on any of those points. But to play devil's advocate I'm pretty sure they *never* intended to bring any of this up until they suddenly felt it was a way to direct attention away from what I perso ally consider a non-event. Even where I work management is super tight lipped about stuff partly for ad campaign reasons and partly because they want to reserve the option to roll stuff back quietly if it doesn't pan out.

But since the cat was out of the bag they really should have had someone dedicated to this line of communication since the reveal.

SaffronCitrusflower

Finedaible wrote: »

I honestly don't know if I can ever trust anymore. There's been years of empty promises, dead ends, and ever slower resolutions and I'm starting to wonder if we were just being fed hopium all this time. From the sound of it things are probably worse than we previously thought, but thanks for the update at any rate.

Cyrodiil used to work much better than it does now, and pop caps were four times higher and group sizes were double what they are now.

What happened?

Twig_Garlicshine

Reverb wrote: »

To everyone saying that performance on PCNA is as bad as it’s ever been - I gotta disagree.

Yes, performance is bad in cyrodiil. Not only IN big fights, but if there is a big fight anywhere on the map even if you aren’t near it. Yes there are still skill delays, invisible AOE, siege that won’t work because it’s “busy”, and occasional rubber banding.

The server upgrades brought a glimpse of what cyrodiil could be like without any lag or performance issues, and it was amazing. But it was short lived and soon things were like before.

But what they aren’t is worse than ever. I spent some time recently going through old vids from Thornblade, and was reminded how much worse things used to be from a performance experience. Late 2014 through all of 2017 was far worse than anything I experience now, or have for the last couple years. Things are objectively better.

Would I trade the improvements for a higher pop cap for more dynamic fights? Yeah I think I would.

However, the stuck in combat bug is truly worse than ever.

It was amazing, and very short lived.
"Here's the way it could be, we are now reverting you to what you are used to."

kyle.wilson

This update announcement sounds like 2 years of unfulfilled promises, and a surrender.

Elsonso

kyle.wilson wrote: »

This update announcement sounds like 2 years of unfulfilled promises, and a surrender.

Definitely sounded like a surrender...

PAFF06

Wuduwasa13 wrote: »

ZOS_MattFiror wrote: »

In fact, the recent Whitestrake’s Mayhem event in May showed just how much fun ESO PvP is with the new hardware in place.

Appreciate the post & I am sincerely glad for those of other servers who have and are able to enjoy the event, however for us poor folk on Xbox EU it’s been a rather woeful time bereft of nearly all enjoyment, given how appalling the performance in cyrodil is.

With that fact in mind, do you perhaps have something a little less nebulous to offer us (customers who pay the same subscription fees as your players hosted elsewhere) as to just specifically when we might expect the server update, other than just “scheduled later this year” or an insight into how preparations towards it are proceeding? (After all, I can tell anyone I’ve scheduled something down the line and do nothing, then cancel it approaching it’s due date citing any number of reasons).

Lastly, because a picture says a thousand words: here is the recent server population during the Xbox EU Whitestrake event. Personally I’ve never seen it this low for the event in the many years I’ve been playing as with past years we endured long queues to enter campaigns. I can rather confidently say it factors almost exclusively due to the incredible lag & unacceptably awful performance based on feedback in guilds and various alternative forums.

Many thanks.

Xbox eu population died. After Zos annaunced the server change late, most of the endgamers started to quit. My trial guilds are playing with other games now cause the population so low they cant even do a craglorn hm lol. Theres no people for anything..
After I heard whats up with our servers I went to NA to try it. 401 cp atm and it works pretty well!
On xbox Na after the server maintenance theres no lagg on 3-3-3 or any issue.
So on EU we pay for the same content and we get the worst performance. This is just not fair.

If the servers wont be changed asap, those who still play going to quit soon too.
I play since day1 and couldnt even imagine I gonna stop playing, now I did. This “performance” is unplayable.

Soraka

Welp. Hope something else gets tried and this isn't a surrender.

Arrow312

PAFF06 wrote: »

Wuduwasa13 wrote: »

ZOS_MattFiror wrote: »

In fact, the recent Whitestrake’s Mayhem event in May showed just how much fun ESO PvP is with the new hardware in place.

Appreciate the post & I am sincerely glad for those of other servers who have and are able to enjoy the event, however for us poor folk on Xbox EU it’s been a rather woeful time bereft of nearly all enjoyment, given how appalling the performance in cyrodil is.

With that fact in mind, do you perhaps have something a little less nebulous to offer us (customers who pay the same subscription fees as your players hosted elsewhere) as to just specifically when we might expect the server update, other than just “scheduled later this year” or an insight into how preparations towards it are proceeding? (After all, I can tell anyone I’ve scheduled something down the line and do nothing, then cancel it approaching it’s due date citing any number of reasons).

Lastly, because a picture says a thousand words: here is the recent server population during the Xbox EU Whitestrake event. Personally I’ve never seen it this low for the event in the many years I’ve been playing as with past years we endured long queues to enter campaigns. I can rather confidently say it factors almost exclusively due to the incredible lag & unacceptably awful performance based on feedback in guilds and various alternative forums.

Many thanks.

Xbox eu population died. After Zos annaunced the server change late, most of the endgamers started to quit. My trial guilds are playing with other games now cause the population so low they cant even do a craglorn hm lol. Theres no people for anything..
After I heard whats up with our servers I went to NA to try it. 401 cp atm and it works pretty well!
On xbox Na after the server maintenance theres no lagg on 3-3-3 or any issue.
So on EU we pay for the same content and we get the worst performance. This is just not fair.

If the servers wont be changed asap, those who still play going to quit soon too.
I play since day1 and couldnt even imagine I gonna stop playing, now I did. This “performance” is unplayable.

Problem is as long as so many ppl will Log in daily nothing Change....was the Last two days only since First MyM and WE Had 2 Bars in PvP and AD 0. Normally you only have 2 choices as EU player Switch to NA or quit. Lag was Back Yesterday and at 7pm many Blues logged Out cuz map was dead.

GooGa592

I'm still not understanding how a working cyrodiil with a much higher population and larger group sizes used to be possible, but somehow it's not possible now. Server hardware has advanced a long way in the last 5 years. Seems more like a cost benefit analysis is what's holding things back.

laniakea_0

GooGa592 wrote: »

I'm still not understanding how a working cyrodiil with a much higher population and larger group sizes used to be possible, but somehow it's not possible now. Server hardware has advanced a long way in the last 5 years. Seems more like a cost benefit analysis is what's holding things back.

too many new players, too much new content, too many new systems. the servers have to handle everything at the same time and they the current setup is not capable of providing that much capacity. wait for the server code rework to be concluded and then we'll see if it's going to improve or not.

virtus753

laniakea_0 wrote: »

GooGa592 wrote: »

I'm still not understanding how a working cyrodiil with a much higher population and larger group sizes used to be possible, but somehow it's not possible now. Server hardware has advanced a long way in the last 5 years. Seems more like a cost benefit analysis is what's holding things back.

too many new players, too much new content, too many new systems. the servers have to handle everything at the same time and they the current setup is not capable of providing that much capacity. wait for the server code rework to be concluded and then we'll see if it's going to improve or not.

I might have misunderstood Matt Firor’s post, but to me the takeaway is that the code rework isn’t having the desired impact on performance and has caused significant issues. His post suggested to me that the rework might well be considered concluded with respect to their original plans for it. I wasn’t clear on how aggressively they would be pursuing further changes to the code given the disappointing results and major issues encountered to date. “We’ll keep looking for areas of improvement” suggests to me that they are switching to a more passive “we’ll keep an eye out” approach after re-evaluating whatever was left of the project.

I found it interesting to hear that Rich Lambert has been saying that undoing some things that have worsened performance does not result in better performance, like reverting destructible gates and the bridge. It seems like it just isn’t possible to get that performance cost back now, on top of the increases in population, content, and systems you mentioned.

laniakea_0

@virtus753 that post was just about the multi-threading. that's just one of several things they are working on. as far as I know, the multi-threading and the server code rework overlap but aren't the same.

blktauna

laniakea_0 wrote: »

@virtus753 that post was just about the multi-threading. that's just one of several things they are working on. as far as I know, the multi-threading and the server code rework overlap but aren't the same.

I think you are being optimistic here. We've asked specifically for details on this and there has been complete silence. Its an easy answer yes there are other things going on or no we're giving up. Nothing.

virtus753

laniakea_0 wrote: »

@virtus753 that post was just about the multi-threading. that's just one of several things they are working on. as far as I know, the multi-threading and the server code rework overlap but aren't the same.

That’s also a question that I think we all deserve more clarification on.

In the past ZOS seems to have used various terms without being very clear about how exactly they relate (e.g. rearchitecture, code rework, server multithreading, etc.). But the initial announcement and several of Kevin's subsequent comments have suggested that they are the same project, and the timing of Matt's post from the other week seems to confirm that. I would, however, like confirmation, especially since it seems that a lot of players are still holding out hope that there is more coming here.

In the initial announcement, from January 2022, they use a variety of phrases to describe a singular project:

"In short – just like we did for the client a year or so back when we introduced multithreaded rendering to increase client frame rates – we are going to rearchitect our server. The version of ESO in 2022 is many magnitudes larger and more complex than the ESO that launched in 2014. So, in order to give everyone a good play experience in high-intensity situations like PvP and Trials, we need to essentially rewrite some of the foundational server code to account for it. This should dramatically increase server performance, but obviously we will need to test and evaluate as we go along."

https://forums.elderscrollsonline.com/en/discussion/595442/eso-pvp-update-january-2022/p1

This comment came in a thread about PvP updates, but the work is aimed at addressing both PvP and trials, where they have acknowledged performance is also an issue.

In his letter of December 12, 2022, Matt Firor said: "Our multi-threading work continues, as we announced a month or so back. Just to remind everyone, this is the initiative that will result in better overall server performance." In the same post, he announced that we could expect an update on this work in the patch notes for the second quarter of this year.

https://www.elderscrollsonline.com/en-us/news/post/63363

The singular "the initiative" suggests to me that the multithreading work and the code rework and the server rearchitecture are all the same thing in this context.

Furthermore, at the beginning of February, in response to comments that we had not gotten an update on the "base code rewrite/reworking" project, Kevin pointed to Matt's comment above about "multithreading" work as an update:

"Here is what he [Matt] said, 'Our multi-threading work continues, as we announced a month or so back.'" etc.

https://forums.elderscrollsonline.com/en/discussion/comment/7805010/#Comment_7805010

Now Kevin's response confirms that the multithreading work is at least part of the code rewrite/rework without explicitly addressing whether there are any other parts to the code project. That said, if there are other parts, we have not heard about them since the initial announcement in over 18 months. We have had updates only on multithreading work and on the hardware replacement/refresh.

So if there is more to the code rework, what is it and where is it? (That is not rhetorical. I do hope we get an answer either way.)