Server Multithreading Update – July 2023

React

virtus753 wrote: »

laniakea_0 wrote: »

@virtus753 that post was just about the multi-threading. that's just one of several things they are working on. as far as I know, the multi-threading and the server code rework overlap but aren't the same.

That’s also a question that I think we all deserve more clarification on.

In the past ZOS seems to have used various terms without being very clear about how exactly they relate (e.g. rearchitecture, code rework, server multithreading, etc.). But the initial announcement and several of Kevin's subsequent comments have suggested that they are the same project, and the timing of Matt's post from the other week seems to confirm that. I would, however, like confirmation, especially since it seems that a lot of players are still holding out hope that there is more coming here.

In the initial announcement, from January 2022, they use a variety of phrases to describe a singular project:

"In short – just like we did for the client a year or so back when we introduced multithreaded rendering to increase client frame rates – we are going to rearchitect our server. The version of ESO in 2022 is many magnitudes larger and more complex than the ESO that launched in 2014. So, in order to give everyone a good play experience in high-intensity situations like PvP and Trials, we need to essentially rewrite some of the foundational server code to account for it. This should dramatically increase server performance, but obviously we will need to test and evaluate as we go along."

https://forums.elderscrollsonline.com/en/discussion/595442/eso-pvp-update-january-2022/p1

This comment came in a thread about PvP updates, but the work is aimed at addressing both PvP and trials, where they have acknowledged performance is also an issue.

In his letter of December 12, 2022, Matt Firor said: "Our multi-threading work continues, as we announced a month or so back. Just to remind everyone, this is the initiative that will result in better overall server performance." In the same post, he announced that we could expect an update on this work in the patch notes for the second quarter of this year.

https://www.elderscrollsonline.com/en-us/news/post/63363

The singular "the initiative" suggests to me that the multithreading work and the code rework and the server rearchitecture are all the same thing in this context.

Furthermore, at the beginning of February, in response to comments that we had not gotten an update on the "base code rewrite/reworking" project, Kevin pointed to Matt's comment above about "multithreading" work as an update:

"Here is what he [Matt] said, 'Our multi-threading work continues, as we announced a month or so back.'" etc.

https://forums.elderscrollsonline.com/en/discussion/comment/7805010/#Comment_7805010

Now Kevin's response confirms that the multithreading work is at least part of the code rewrite/rework without explicitly addressing whether there are any other parts to the code project. That said, if there are other parts, we have not heard about them since the initial announcement in over 18 months. We have had updates only on multithreading work and on the hardware replacement/refresh.

So if there is more to the code rework, what is it and where is it? (That is not rhetorical. I do hope we get an answer either way.)

Wonderfully composed comment. I too would like clarification on whether these are separate projects or the same project, and what we should expect from them moving forward.

@ZOS_Kevin @ZOS_GinaBruno

Could either of you share any insight into whether the "server multithreading" project mentioned in this post is the same project described in the January 2022 post as a "code rewrite"? Would really help clear up a lot of confusion.

GooGa592

laniakea_0 wrote: »

GooGa592 wrote: »

I'm still not understanding how a working cyrodiil with a much higher population and larger group sizes used to be possible, but somehow it's not possible now. Server hardware has advanced a long way in the last 5 years. Seems more like a cost benefit analysis is what's holding things back.

too many new players, too much new content, too many new systems. the servers have to handle everything at the same time and they the current setup is not capable of providing that much capacity. wait for the server code rework to be concluded and then we'll see if it's going to improve or not.

There aren't more players in cyrodiil, there are fewer, far fewer. Population caps are 1/4 of what they used to be. There is no new content or systems in Cyrodiil. If server capacity is the issue, what is stopping ZOS from adding more server capacity to support Cyrodiil?

laniakea_0

@GooGa592 this isn't about Cyrodiil pop. it's about amount of players in general. it all impacts the servers. if you assign additional resources to Cyrodiil, that means taking away resources from other activities and vice versa. the player base is so large that they apparently can't bolster cyrodiil without compromising other areas of the game anymore.

reazea

laniakea_0 wrote: »

@GooGa592 this isn't about Cyrodiil pop. it's about amount of players in general. it all impacts the servers. if you assign additional resources to Cyrodiil, that means taking away resources from other activities and vice versa. the player base is so large that they apparently can't bolster cyrodiil without compromising other areas of the game anymore.

Don't most game companies provide more server resources when they add more content so all content continues to perform as intended?

blktauna

reazea wrote: »

laniakea_0 wrote: »

@GooGa592 this isn't about Cyrodiil pop. it's about amount of players in general. it all impacts the servers. if you assign additional resources to Cyrodiil, that means taking away resources from other activities and vice versa. the player base is so large that they apparently can't bolster cyrodiil without compromising other areas of the game anymore.

Don't most game companies provide more server resources when they add more content so all content continues to perform as intended?

One would think. Yet here we are,

laniakea_0

and they did. the server code just can't handle it yet.

blktauna

Well they stated replacement, not expansion, but as they are so stingy with specifics we don't actually kno.

TechMaybeHic

I'm pretty sure the ships sailed. The gig is up. Nothing else is being done. And honestly; I'd appreciate that over hearing another "improvements are coming next year" and have nothing come from it. It's just more honest.

Aardappelboom

It's been what? 4 years? I remember this going on since I started playing three years ago. It's a sad string of developments but to be fair just admitting things aren't working out is the right thing to do at this point.

Instead of focussing on glories of the past, how about looking at new ways to make pvp interesting, there's just so much stuff here to at least make it interesting for the next couple of years.

I have to admit I am a bit tired of the rethoric used to not create new PVP content, maybe now, since there is no plan anymore, and after all servers have new hardware, they can start looking at something fresh?

I'm not getting my hopes up though. 😊

laniakea_0

blktauna wrote: »

Well they stated replacement, not expansion, but as they are so stingy with specifics we don't actually kno.

one and the same in this case. if they replace server hardware with new more recent models on the same surface area, that also increases processing capacity.

blktauna

Are you sure? Who knows what it was replaced with. In any case the code seemingly is unable to take advantage of the new speed and/or capacity so we have a problem still.

laniakea_0

yes. hardware has come a long way in the past 10 years. unless they downsized the floor space their server rooms take up, the replacement absolutely improved Processing capacity.
also, try to look at the clues their behavior gives. there won't be a Q3 DLC and they already confirmed there won't be any new area in Q4. this would make it seem like there are a lot of people at ZOS working on little to nothing right now. but that can't be the case. no, if they are falling short of their usual development cycle, it's because they are working on something that won't be ready for release for a while yet. it's gonna have to be something big. like the code rework and subsequent major bug fixes.

TaSheen

Or the new IP....

blktauna

Howw much of the new hardware capacity as even allotted to ESO. They have several properties. And again I agree, the capacity will be infinitely greater. I use machines that are 15 years old, 10 years old and 4 years old. The difference is vast. But the reason I still use a 15 year old machine is the necessary software on it won't run on a modern machine...

I live in deep hope that they will say 'AHA e trolled all of you. Here's the revamped awesomely functional new base code!'

laniakea_0

TaSheen wrote: »

Or the new IP....

doubt they are going to downsize the ESO dev team while it's still growing in popularity

TaSheen

Well, according to what was initially stated about the server hardware, it was to effect changes that would make "forward compatibility" better in the future. That's paraphrased - it's just what I took their statements to mean in light of them not saying "yes we're replacing all the old server racks with shiny new stuff".

And I don't know that I really believe they are downsizing the team - but it just seems that normal oversight of systems in this game is not actually being provided....

React

laniakea_0 wrote: »

yes. hardware has come a long way in the past 10 years. unless they downsized the floor space their server rooms take up, the replacement absolutely improved Processing capacity.
also, try to look at the clues their behavior gives. there won't be a Q3 DLC and they already confirmed there won't be any new area in Q4. this would make it seem like there are a lot of people at ZOS working on little to nothing right now. but that can't be the case. no, if they are falling short of their usual development cycle, it's because they are working on something that won't be ready for release for a while yet. it's gonna have to be something big. like the code rework and subsequent major bug fixes.

That is a very optimistic perspective to have despite the history of performance in TESO, and the inability of the studio to make good on their promises.

I'd be much more inclined to believe that the reduction in content is due to a shift in focus by the studio to their new IP. They dress Q3 up as a "bug fixing and QOL patch", but when it comes to performance and combat there are almost 0 signifcant fixes or improvements.

Aardappelboom

laniakea_0 wrote: »

yes. hardware has come a long way in the past 10 years. unless they downsized the floor space their server rooms take up, the replacement absolutely improved Processing capacity.
also, try to look at the clues their behavior gives. there won't be a Q3 DLC and they already confirmed there won't be any new area in Q4. this would make it seem like there are a lot of people at ZOS working on little to nothing right now. but that can't be the case. no, if they are falling short of their usual development cycle, it's because they are working on something that won't be ready for release for a while yet. it's gonna have to be something big. like the code rework and subsequent major bug fixes.

You'd think that, yes, but I don't believe they are working on anything in particular related to pvp or the code base.

I also don't think they are working on something "big" they're just working towards the next update and the one after that, it's been like this forever.

As far as I know ZOS from the last three years: they don't really have big surprises, what you see is what you get, the only thing to look forward to right now is the endless dungeon and we'll now what to look forward to next year when the global reveal hits.

It's rince and repeat.

Knockmaker

It seems the good times of new servers better performance is already coming to an end. The performance in Cyrodiil right now is abysmal again and it has been for the last few days. I really don't believe anymore this game is going to perform normally ever again...

SaffronCitrusflower

Aardappelboom wrote: »

laniakea_0 wrote: »

yes. hardware has come a long way in the past 10 years. unless they downsized the floor space their server rooms take up, the replacement absolutely improved Processing capacity.
also, try to look at the clues their behavior gives. there won't be a Q3 DLC and they already confirmed there won't be any new area in Q4. this would make it seem like there are a lot of people at ZOS working on little to nothing right now. but that can't be the case. no, if they are falling short of their usual development cycle, it's because they are working on something that won't be ready for release for a while yet. it's gonna have to be something big. like the code rework and subsequent major bug fixes.

You'd think that, yes, but I don't believe they are working on anything in particular related to pvp or the code base.

I also don't think they are working on something "big" they're just working towards the next update and the one after that, it's been like this forever.

As far as I know ZOS from the last three years: they don't really have big surprises, what you see is what you get, the only thing to look forward to right now is the endless dungeon and we'll now what to look forward to next year when the global reveal hits.

It's rince and repeat.

The group size has been cut in half and the population cap is a fraction of what it used to be since they said they were working on making things better in Cyrodiil.

Aardappelboom

SaffronCitrusflower wrote: »

Aardappelboom wrote: »

laniakea_0 wrote: »

yes. hardware has come a long way in the past 10 years. unless they downsized the floor space their server rooms take up, the replacement absolutely improved Processing capacity.
also, try to look at the clues their behavior gives. there won't be a Q3 DLC and they already confirmed there won't be any new area in Q4. this would make it seem like there are a lot of people at ZOS working on little to nothing right now. but that can't be the case. no, if they are falling short of their usual development cycle, it's because they are working on something that won't be ready for release for a while yet. it's gonna have to be something big. like the code rework and subsequent major bug fixes.

You'd think that, yes, but I don't believe they are working on anything in particular related to pvp or the code base.

I also don't think they are working on something "big" they're just working towards the next update and the one after that, it's been like this forever.

As far as I know ZOS from the last three years: they don't really have big surprises, what you see is what you get, the only thing to look forward to right now is the endless dungeon and we'll now what to look forward to next year when the global reveal hits.

It's rince and repeat.

The group size has been cut in half and the population cap is a fraction of what it used to be since they said they were working on making things better in Cyrodiil.

I don't know why you quoted me but so much more has been tried without succes. I was hopeful when they announced a server-side code rewrite and I'm sure they will keep optimizing code (you can see the combat team trying every patch optimizing skills for performance and saying as much) one day the issues might fade, it often goes that way with things like this, small changes stack up in the end.

I just wouldn't expect another big masterplan unless they announce it officially and even then, after so many years I expect them to look ahead instead of looking back.

There is room for PVP outside of Cyro, the ranking system of TOT is ten times better than what we have for BG to just say one thing. It just comes down to whether they want to or not. Giving up on this endless quest to fix cyrodiil, after all they've done, would be a respectful decision in my opinion.

Lebkuchen

Aardappelboom wrote: »

[...] the ranking system of TOT is ten times better than what we have for BG to just say one thing. It just comes down to whether they want to or not. Giving up on this endless quest to fix cyrodiil, after all they've done, would be a respectful decision in my opinion.

And that's the big question, right? Are they really trying to fix it, or just telling us some stories about mysterious projects year after year, without actually trying.

They accidentally fixed the performance once, when the new servers came. And we saw that performance problems have nothing to do with code or whatever. We just need more server power, or whatever they dialed back on PC NA after a few months. Because when Playstation got the new servers, one year after PC NA, Cyrodiil worked great on Playstation, while the performance on PC was already bad again. That doesn't make sense.

Aardappelboom

Lebkuchen wrote: »

Aardappelboom wrote: »

[...] the ranking system of TOT is ten times better than what we have for BG to just say one thing. It just comes down to whether they want to or not. Giving up on this endless quest to fix cyrodiil, after all they've done, would be a respectful decision in my opinion.

And that's the big question, right? Are they really trying to fix it, or just telling us some stories about mysterious projects year after year, without actually trying.

They accidentally fixed the performance once, when the new servers came. And we saw that performance problems have nothing to do with code or whatever. We just need more server power, or whatever they dialed back on PC NA after a few months. Because when Playstation got the new servers, one year after PC NA, Cyrodiil worked great on Playstation, while the performance on PC was already bad again. That doesn't make sense.

I truly believe (most) companies don't lie about trying to fix things. Especially in the game industry these guys put their creative sould into this game. Whether or not they're making the right decisions or just holding on to some old mindset or dream is another question. (same goes for the entire overland discussion btw)

In any case, my point was to not expect anything, ZOS is pretty clear on what they're going to do and make and you can hear it every year during the global reveal, if it's not mentioned there, it's not happening.

blktauna

The Year of Improvements thing from a couple years back was in the Global Reveal...

And yet here we are.

SaffronCitrusflower

Lebkuchen wrote: »

Aardappelboom wrote: »

[...] the ranking system of TOT is ten times better than what we have for BG to just say one thing. It just comes down to whether they want to or not. Giving up on this endless quest to fix cyrodiil, after all they've done, would be a respectful decision in my opinion.

And that's the big question, right? Are they really trying to fix it, or just telling us some stories about mysterious projects year after year, without actually trying.

They accidentally fixed the performance once, when the new servers came. And we saw that performance problems have nothing to do with code or whatever. We just need more server power, or whatever they dialed back on PC NA after a few months. Because when Playstation got the new servers, one year after PC NA, Cyrodiil worked great on Playstation, while the performance on PC was already bad again. That doesn't make sense.

It's a fact that we have repeatedly, over several years of more, seen that when they upgrade servers that performance improves. Long ago we used to see improved performance after maintenance right before the MYM events, then performance would then degrade again after the next maintenance after the MYM events were over. I'm wholly confident people at ZOS know why and what's going on.

SaffronCitrusflower

I see now that they've just recently unpinned this discussion from the announcements section.

blktauna

yes, so who won the pool on that. The bets have been going since the delayed xbox server announcement.

This is why there is no trust. They just sweep it aay and hope people move on rather than directly addressing the issues.

virtus753

SaffronCitrusflower wrote: »

I see now that they've just recently unpinned this discussion from the announcements section.

That’s odd. It still appears pinned in General for me. Maybe it’s a glitch?

Lags

React wrote: »

ZOS_MattFiror wrote: »

It is approaching midyear and time to give an update on how our multithreading work is going. If you remember, back at the end of last year, I gave the news that we would be performing an ongoing series of server work aimed at spreading out ESO processes across more CPU cores, with the goal that it would result in a substantial increase in server performance. Over Update 37 – launched back in March – and in Update 38, we have completed some of this work.

Initial metrics, however, show that these steps in multithreading are not having as much of a performance improvement as desired, and so we had to roll back one of the process changes because it introduced an unacceptable level of instability. Fortunately, the recent upgrade to server hardware has resulted in more customer-facing server performance improvements than any of our multithreading work in all public realms where we’ve completed the work. (Xbox EU is still scheduled for later this year.) In fact, the recent Whitestrake’s Mayhem event in May showed just how much fun ESO PvP is with the new hardware in place.

All this to say, we continue to look for areas to spread process to optimize our server CPU capabilities. Please remember that we must be very, very careful when making changes of this complexity to ESO’s massive codebase. However, we will continue to dedicate resources to keep analyzing and looking for ways to utilize multithreading to improve server performance, and we will update everyone on any successes that we have when we find solutions.

Thanks for being the best community in gaming!

Matt

Thanks for the update, matt.

You reference the performance improvements that have been a result of the replaced hardware, but are you aware of the drastic decline in performance PC NA has experienced since it's hardware replacement? The performance is seemingly just as bad as it was pre-upgrade, with campaigns seeing signifcant action delays as soon as there is any signifcant population.

I have to say, assuming we're still talking about the project that you announced in January of 2022 (there is a bit of confusion here, as it was initially referred to as a "code rewrite" but later redefined as a "server hyperthreading"), this update is extremely disappointing to see. It has been 19 months since the initial announcement, at which time we were told there would be no PVP content until the work was finished. This post seems to indicate there is no actual end in sight anymore.

It has been 4.5 years since the last PVP content was added to the game - which was a single BG map. We keep seeing the excuse of "we simply can't touch cyrodiil until we fix the performance, as we will just exacerbate the issues", but what about imperial city? What about battlegrounds? What about dueling, or potential arenas? None of these things suffer the same performance issues that cyrodiil does.

I held out hope for a long time that this project was being conducted in good faith, and that we'd see some progress from it that could then lead to a rejuvenation of PVP in TESO. As you said yourself - when there is population and decent performance, it is truly fun and honestly miles above any other PVP mmo on the market. But the lack of communication from developers regarding PVP, the lack of content in nearly 5 years, the continually degrading performance, and now the lack of any tangible progress on this now 19 month long project have finally crushed my ability to remain hopeful.

EDIT: Just wanted to add, your comment about the midyear performance on new hardware seems misleading to me. The community knows that additional server resources are allocated to cyrodiil for duration of the event, resulting in improved performance. This has been observable going back years, far before any server hardware was replaced. To imply that the enjoyable performance we're seeing this midyear is a result of new hardware exclusively seems disingenuous to me, especially since there has been no zenimax acknowledgement of how the PC NA cyrodiil performance has degraded in the past year since the hardware was replaced.

Ya ngl this is about what i expected. It seems like they always put their head in the sand and pretend things are not the way they are. Being positive is fine but this is just unreal and so far from the truth. Nothing ever changes with eso, and what does change is either worse, like combat, or takes many many years. I find myself playing less and less every year, and this year ive barely played at all outside of mid year.

Its just sad all around. Its hard to play a game with bad performance. Its hard to play a game with a terrible reward system. Its hard to play a game that keeps making terrible changes to core systems that were perfectly fine, like combat, to cater to low effort players. Its hard to play a game that cannot nail down balance and refuses to try things differently, such as balancing cyro/bgs/pve separately, to do it. And its especially hard to keep playing when all these issues are the problem, with performance at the forefront.

I always had the extremely small shred of hope that they would improve performance, hopefully this year. But after the restructure and things got better, only to get worse, i lost a lot of that hope. And now with this post it doesnt look promising. The restructure was not a win. Performance got better, then it got worse. And has stayed worse.

SaffronCitrusflower

virtus753 wrote: »

SaffronCitrusflower wrote: »

I see now that they've just recently unpinned this discussion from the announcements section.

That’s odd. It still appears pinned in General for me. Maybe it’s a glitch?

Not a glitch. After I and blktauna posted about the unpinning, they repinned the thread.