Server Multithreading Update – July 2023

ZOS_MattFiror

It is approaching midyear and time to give an update on how our multithreading work is going. If you remember, back at the end of last year, I gave the news that we would be performing an ongoing series of server work aimed at spreading out ESO processes across more CPU cores, with the goal that it would result in a substantial increase in server performance. Over Update 37 – launched back in March – and in Update 38, we have completed some of this work.

Initial metrics, however, show that these steps in multithreading are not having as much of a performance improvement as desired, and so we had to roll back one of the process changes because it introduced an unacceptable level of instability. Fortunately, the recent upgrade to server hardware has resulted in more customer-facing server performance improvements than any of our multithreading work in all public realms where we’ve completed the work. (Xbox EU is still scheduled for later this year.) In fact, the recent Whitestrake’s Mayhem event in May showed just how much fun ESO PvP is with the new hardware in place.

All this to say, we continue to look for areas to spread process to optimize our server CPU capabilities. Please remember that we must be very, very careful when making changes of this complexity to ESO’s massive codebase. However, we will continue to dedicate resources to keep analyzing and looking for ways to utilize multithreading to improve server performance, and we will update everyone on any successes that we have when we find solutions.

Thanks for being the best community in gaming!

Matt

Wuduwasa13

ZOS_MattFiror wrote: »

In fact, the recent Whitestrake’s Mayhem event in May showed just how much fun ESO PvP is with the new hardware in place.

Appreciate the post & I am sincerely glad for those of other servers who have and are able to enjoy the event, however for us poor folk on Xbox EU it’s been a rather woeful time bereft of nearly all enjoyment, given how appalling the performance in cyrodil is.

With that fact in mind, do you perhaps have something a little less nebulous to offer us (customers who pay the same subscription fees as your players hosted elsewhere) as to just specifically when we might expect the server update, other than just “scheduled later this year” or an insight into how preparations towards it are proceeding? (After all, I can tell anyone I’ve scheduled something down the line and do nothing, then cancel it approaching it’s due date citing any number of reasons).

Lastly, because a picture says a thousand words: here is the recent server population during the Xbox EU Whitestrake event. Personally I’ve never seen it this low for the event in the many years I’ve been playing as with past years we endured long queues to enter campaigns. I can rather confidently say it factors almost exclusively due to the incredible lag & unacceptably awful performance based on feedback in guilds and various alternative forums.

Many thanks.

Arrow312

You will dont get another answer....this will kill X EU PvP

React

ZOS_MattFiror wrote: »

It is approaching midyear and time to give an update on how our multithreading work is going. If you remember, back at the end of last year, I gave the news that we would be performing an ongoing series of server work aimed at spreading out ESO processes across more CPU cores, with the goal that it would result in a substantial increase in server performance. Over Update 37 – launched back in March – and in Update 38, we have completed some of this work.

Initial metrics, however, show that these steps in multithreading are not having as much of a performance improvement as desired, and so we had to roll back one of the process changes because it introduced an unacceptable level of instability. Fortunately, the recent upgrade to server hardware has resulted in more customer-facing server performance improvements than any of our multithreading work in all public realms where we’ve completed the work. (Xbox EU is still scheduled for later this year.) In fact, the recent Whitestrake’s Mayhem event in May showed just how much fun ESO PvP is with the new hardware in place.

All this to say, we continue to look for areas to spread process to optimize our server CPU capabilities. Please remember that we must be very, very careful when making changes of this complexity to ESO’s massive codebase. However, we will continue to dedicate resources to keep analyzing and looking for ways to utilize multithreading to improve server performance, and we will update everyone on any successes that we have when we find solutions.

Thanks for being the best community in gaming!

Matt

Thanks for the update, matt.

You reference the performance improvements that have been a result of the replaced hardware, but are you aware of the drastic decline in performance PC NA has experienced since it's hardware replacement? The performance is seemingly just as bad as it was pre-upgrade, with campaigns seeing signifcant action delays as soon as there is any signifcant population.

I have to say, assuming we're still talking about the project that you announced in January of 2022 (there is a bit of confusion here, as it was initially referred to as a "code rewrite" but later redefined as a "server hyperthreading"), this update is extremely disappointing to see. It has been 19 months since the initial announcement, at which time we were told there would be no PVP content until the work was finished. This post seems to indicate there is no actual end in sight anymore.

It has been 4.5 years since the last PVP content was added to the game - which was a single BG map. We keep seeing the excuse of "we simply can't touch cyrodiil until we fix the performance, as we will just exacerbate the issues", but what about imperial city? What about battlegrounds? What about dueling, or potential arenas? None of these things suffer the same performance issues that cyrodiil does.

I held out hope for a long time that this project was being conducted in good faith, and that we'd see some progress from it that could then lead to a rejuvenation of PVP in TESO. As you said yourself - when there is population and decent performance, it is truly fun and honestly miles above any other PVP mmo on the market. But the lack of communication from developers regarding PVP, the lack of content in nearly 5 years, the continually degrading performance, and now the lack of any tangible progress on this now 19 month long project have finally crushed my ability to remain hopeful.

EDIT: Just wanted to add, your comment about the midyear performance on new hardware seems misleading to me. The community knows that additional server resources are allocated to cyrodiil for duration of the event, resulting in improved performance. This has been observable going back years, far before any server hardware was replaced. To imply that the enjoyable performance we're seeing this midyear is a result of new hardware exclusively seems disingenuous to me, especially since there has been no zenimax acknowledgement of how the PC NA cyrodiil performance has degraded in the past year since the hardware was replaced.

MetallicMonk

Bro those chaos ball weekends go HARD though.

alternatelder

Wuduwasa13 wrote: »

ZOS_MattFiror wrote: »

In fact, the recent Whitestrake’s Mayhem event in May showed just how much fun ESO PvP is with the new hardware in place.

Appreciate the post & I am sincerely glad for those of other servers who have and are able to enjoy the event, however for us poor folk on Xbox EU it’s been a rather woeful time bereft of nearly all enjoyment, given how appalling the performance in cyrodil is.

With that fact in mind, do you perhaps have something a little less nebulous to offer us (customers who pay the same subscription fees as your players hosted elsewhere) as to just specifically when we might expect the server update, other than just “scheduled later this year” or an insight into how preparations towards it are proceeding? (After all, I can tell anyone I’ve scheduled something down the line and do nothing, then cancel it approaching it’s due date citing any number of reasons).

Lastly, because a picture says a thousand words: here is the recent server population during the Xbox EU Whitestrake event. Personally I’ve never seen it this low for the event in the many years I’ve been playing as with past years we endured long queues to enter campaigns. I can rather confidently say it factors almost exclusively due to the incredible lag & unacceptably awful performance based on feedback in guilds and various alternative forums.

Many thanks.

This great performance was a "here today, gone tomorrow" experience on all realms. I experienced no change on PC NA. On PS NA, it was mostly fine, but then back to garbage performance afterwards. I heard this same experience from many others.

blktauna

To say I'm dissappointed is putting it mildly. Unfortunately this is exactly the answer I was expecting. I had so hoped to proven wrong.

BlessedThistle

It would be nice if the game didn’t unexpectedly and seemingly randomly freeze every 20-30 minutes or so while in Cyrodiil for this Whitestrake’s Mayhem. I quit playing earlier than anticipated yesterday after freezing for the 5th time, and loading back in to find my Fire Lancer was no longer there with me. Very frustrating. (PS5 NA, btw).

Hrafngrimr3

ZOS_MattFiror wrote: »

Initial metrics, however, show that these steps in multithreading are not having as much of a performance improvement as desired, and so we had to roll back one of the process changes because it introduced an unacceptable level of instability. Fortunately, the recent upgrade to server hardware has resulted in more customer-facing server performance improvements than any of our multithreading work in all public realms where we’ve completed the work. (Xbox EU is still scheduled for later this year.) In fact, the recent Whitestrake’s Mayhem event in May showed just how much fun ESO PvP is with the new hardware in place.

All this to say, we continue to look for areas to spread process to optimize our server CPU capabilities. Please remember that we must be very, very careful when making changes of this complexity to ESO’s massive codebase. However, we will continue to dedicate resources to keep analyzing and looking for ways to utilize multithreading to improve server performance, and we will update everyone on any successes that we have when we find solutions.

Thanks for being the best community in gaming!

Matt

What about the “unacceptable instability” which currently plagues PvP and, as many actual players have addressed, the server upgrades only marginally (if that) improved performance? We’re expected to accept that as paying customers? Have any of the developers actually logged on to Xbox EU or PC-NA to experience it? If not I’m sure many of us here can share videos detailing the degradation and the “unacceptable instability” as you call it.
If I could petition for a refund of my many years of ESO+ at this point I would. Massively disappointed.

FrankonPC

The performance has devolved on PC NA though Mr. Firor. Surely this is an issue long term for all platforms, no?

RicAlmighty

Performance on PC NA especially with regards to server side processing lag is as bad as it has ever been in my years of playing the game. Actions will be triggered and take upwards of 500-1000ms to actually execute, which is simply unacceptable for an action based game like this.

This has nothing to do with my connection or ping, in fact my ping most times is 100 or less, but that's irrelevant as this is a server side lag, so even if my ping were 0, there would still be delayed actions due to the server side processing being bottlenecked. This is rendering the game close to unplayable and if it does not improve I can certainly see players leaving en masse.

Wuduwasa13

alternatelder wrote: »

This great performance was a "here today, gone tomorrow" experience on all realms. I experienced no change on PC NA. On PS NA, it was mostly fine, but then back to garbage performance afterwards. I heard this same experience from many others.

So there really is no hope in sight then. How terribly disappointing.

GooGa592

This sounds like they are shelving the effort to improve performance in Cyrodiil. We are forever stuck with the current performance situation, still massively lower population caps, and only 12 man group sizes.

PvP used to be a priority and used to work so much better and be so much more fun than it is now, so I find the dev comment about how fun PvP is now to be a bit off the mark. ZOS used to deliver a far, far superior PvP environment than they do today. I noticed he didn't comment on a week of broken scoreboards during a PvP event or a two month long campaign with only one month of rewards etc.

Disappointed doesn't even begin to cover how most people are going to process this announcement.

VaranisArano

So correct me if I'm wrong, but it sounds like that if our server has the upgrades already, that's as good as we can expect it to get? Even though performance is not great, population caps are still low, and group size is half of what it used to be?

Ooof.

Thanks for the update, though. Bad news is still better than no news.

Destai

ZOS_MattFiror wrote: »

It is approaching midyear and time to give an update on how our multithreading work is going. If you remember, back at the end of last year, I gave the news that we would be performing an ongoing series of server work aimed at spreading out ESO processes across more CPU cores, with the goal that it would result in a substantial increase in server performance. Over Update 37 – launched back in March – and in Update 38, we have completed some of this work.

Initial metrics, however, show that these steps in multithreading are not having as much of a performance improvement as desired, and so we had to roll back one of the process changes because it introduced an unacceptable level of instability. Fortunately, the recent upgrade to server hardware has resulted in more customer-facing server performance improvements than any of our multithreading work in all public realms where we’ve completed the work. (Xbox EU is still scheduled for later this year.) In fact, the recent Whitestrake’s Mayhem event in May showed just how much fun ESO PvP is with the new hardware in place.

All this to say, we continue to look for areas to spread process to optimize our server CPU capabilities. Please remember that we must be very, very careful when making changes of this complexity to ESO’s massive codebase. However, we will continue to dedicate resources to keep analyzing and looking for ways to utilize multithreading to improve server performance, and we will update everyone on any successes that we have when we find solutions.

Thanks for being the best community in gaming!

Matt

So does this mean you're done with the multi-threading effort and are back to ground zero? Don't you think an AMA would be appropriate at this point?

blktauna

So the actual question is, are you re scoping the project and continuing? dropping it? continuing (sic) as current? What can we expect?

Lebkuchen

Years ago, when i first had the idea that this game is actually just a big social experiment, to test how much a company in the gaming industry can disrespect their customers before they have to shut down their game, i thought it was funny. It's not funny anymore...

DrNukenstein

So does this mean it's okay to get new pvp content now? or.....

colossalvoids

It sounds really bad tbh, kind of like "we're shelving the work for the better times and no pvp content in foreseeable future still stands".

blktauna

colossalvoids wrote: »

It sounds really bad tbh, kind of like "we're shelving the work for the better times and no pvp content in foreseeable future still stands".

That's exactly the feeling I'm getting. No call to action in this update tells me nothing further is slated to happen.

Arrow312

So shortcut:
No new content for any Kind of PvP means Last stand 2016....check some Longtime Players maybe leave.
A complete PvP is going to Break down X EU Check. I think x EU Server will not updated before Sep or later this means around 1/2 later then PS and NA so what do you think will happen with all the EU Players on NA do you think they Go Back to EU when they Grind CP and all the stuff. I don't think so. Look at the Screenshots Here it is Event time and No Queue in X EU. It feels Like X EU Players are Second class customers and you don't earn enough Money from them and therefore you don't Upgrade the Server because it is to expensive.

vsrs_au

ZOS_MattFiror wrote: »

Please remember that we must be very, very careful when making changes of this complexity to ESO’s massive codebase.

This is the most important part of the announcement. Refactoring code to add more support for multithreading is very difficult when the software is this large and complex, and it's so, so easy to introduce bugs in the process. It's easy to disparage ZOS for not having already done it, when you're not in the position of having to do such a difficult code update.

FrankonPC

blktauna wrote: »

So the actual question is, are you re scoping the project and continuing? dropping it? continuing (sic) as current? What can we expect?

Yes this would be good to know.

jhall03

@ZOS_MattFiror xbox EU upgrades are “scheduled’? If it is actually scheduled then please give us a date/timeframe. If that is still not determined, then it’s not actually scheduled.honesty and transparency are much appreciated

Reverb

To everyone saying that performance on PCNA is as bad as it’s ever been - I gotta disagree.

Yes, performance is bad in cyrodiil. Not only IN big fights, but if there is a big fight anywhere on the map even if you aren’t near it. Yes there are still skill delays, invisible AOE, siege that won’t work because it’s “busy”, and occasional rubber banding.

The server upgrades brought a glimpse of what cyrodiil could be like without any lag or performance issues, and it was amazing. But it was short lived and soon things were like before.

But what they aren’t is worse than ever. I spent some time recently going through old vids from Thornblade, and was reminded how much worse things used to be from a performance experience. Late 2014 through all of 2017 was far worse than anything I experience now, or have for the last couple years. Things are objectively better.

Would I trade the improvements for a higher pop cap for more dynamic fights? Yeah I think I would.

However, the stuck in combat bug is truly worse than ever.

jle30303

I would just like to say that, at times when there's any significant population on - even if the population is not where I am - I am currently getting horrendous lag and ping issues whenever I change zone *even in PVE areas*.

And this isn't just "the newest zones and the crafting zones". It's *anywhere*. Like sometimes my FPS drops to literally single figures and my ping goes up to 900+.

It is particularly bad when I change zone.

How come it was *ever* better? Because apparently it was, once.

markulrich1966

I tried xbox EU today, was almost like on the screenshot, but DC had 3 bars, AD and EP 1 bar. DC dominated the map over a long period, unlike on NA, where you have 3 bars for each faction, and gameplay is very dynamic.
As a result, it took forever to gain AP, so I finally resignated at 11 pm and switched to NA again, my main server meanwhile for PVP and PVE.

On NA, grey host is not perfect, but skills fire 80% of the time, unlike before the upgrade. At least during WSM, I don't play cyro apart from events.
EU was not as bad as last time, so I think perormance is not so much driving players away, it is more the general being fed up with the low pop on EU. It is dead boring, always the same players and strategies, if 1 faction takes a lead, it is clear they will dominate the map for the rest of the campaign. With alliance lock, there is no chance this will change like before the lock was introduced. So the others resignate and play something different.

redlink1979

Thanks for the info.

But during that process, my latency to the closest server of my location (EU) has quadrupled at the very least - it now averages between 400 to 1200ms. PvP and PvE zones.
It happens everyday regardless of how populated the server is (week day and time of day).

To make it awkard, my latencty on PC NA remains similar to values before the server update (180-200ms).

I already sent several tickets but so far I only got automated replies suggesting a resolution, that I already had wrote down on the ticket that it didn't worked (the proof that the reply wasn't made by a human).

I_killed_Vivec

"the recent upgrade to server hardware has resulted in more customer-facing server performance improvements than any of our multithreading work in all public realms where we’ve completed the work"

So the logical way to proceed is to save money by not updating the software and spend the cash on making sure the hardware is at least adequate for purpose... with the added bonus of avoiding the "unacceptable level of instability" brought on by modifying the software.

ghost_bg_ESO

hmm... what was it : "adding more/new servers won't help performance..."


of course i know it has to be software and hardware optimization, but i wonder if marketing team understands that game servers can't be run on fridge door...

VaranisArano

vsrs_au wrote: »

ZOS_MattFiror wrote: »

Please remember that we must be very, very careful when making changes of this complexity to ESO’s massive codebase.

This is the most important part of the announcement. Refactoring code to add more support for multithreading is very difficult when the software is this large and complex, and it's so, so easy to introduce bugs in the process. It's easy to disparage ZOS for not having already done it, when you're not in the position of having to do such a difficult code update.

Personally, I expected it to take longer than anticipated. I remember past times ZOS has redone their code like the Groupfinder Rework, and there are always problems. It's just the nature of ideal timelines vs the real world.

However, I am still disappointed that for everyone who's already gotten the server upgrades, we can essentially expect the status quo to continue indefinitely. I didn't start playing PVP until 2016, which is well after the lighting patch that moved code client-side because of cheating that really altered performance, but it's still sobering to realiz how much performance degraded in the time I played even as population caps and group sizes were slashed. The status quo is certainly better that Cyrodiil at its worst; its certainly worse than Cyrodiil at its best.

While I understand that it's going to take time for ZOS to reevaluate their options now that multithreading hasn't fully panned out and find a new path forward, I'm still looking at Cyrodiil and thinking "How much do I want to play like this indefinitely, knowing there's no improvement coming?"


It might be helpful to judge PVPer reactions in terms of the cycle of grief. I'm sure ZOS will keep trying, but as an anaolgy this feels like being told by a doctor that Cyrodiil has a chronic illness and the hoped-for treatment has reached the limits of its effectiveness with no clear way forwards.

We're going to see denial and hope, anger and bargaining, and eventually acceptance from players who've been clinging to small hopes amidst disappointment for years.