Server Multithreading Update – July 2023

Janni

RMW wrote: »

Wolfkeks wrote: »

Not sure if it was asked already but are multithreading and the code-rewrite two different things or are they the same?
Haven't heard anything regarding the rewrite so that's why I'm asking ^^

Should be two seperate things

Multithreading on this scale requires a massive code rewrite. It is a subset of what the rearcitecturing facilitates and is one of the most likely to give significant performance gains. So effectively from our standpoint we might as well view them as the same thing. Internally they would be highly related enough that they couldn't really work on one without taking the other into consideration.

React

laniakea_0 wrote: »

why do you think that? all he said was that it their efforts fell short of expectations and they had to undo some steps. it says nothing about them stopping the work on that all together. they'll continue working on it for the foreseeable future.

Because we have seen this exact scenario pan out over and over again with this company.

We had the cyrodiil performance testing period, where they ran a series of restrictive tests over the course of months, that they said would give them the information needed to fix the game. The end conclusion of these tests, months after they actually ended, was that they didn't get the information they needed and couldn't positively impact performance with what they had learned. They abandoned this project completely and haven't even bothered since to adjust or revert the final test of "no proc cyrodiil" which is still present in ravenwatch and no cp ic.

We had the 2020 "year of performance" where they promised to deliver a much better performing game each quarter that year. We ended 2020 with the worst cyrodiil performance in history at that point.

We've had the code rewrite project - announced in January of 2022, with an estimated completion date of December 2022, which we did not even receive an update on until November of 2022 (after being promised regular updates and communication on the project), at which point we were then told would begin rolling out in Q2 2023. This is nowhere to be found. It's actually confusing whether this project is the same one as the "server multithreading" one from this thread - because despite being asked numerous times, zenimax hasn't even bothered to clarify this. There is even further confusion on this subject by Matt's OP here which states that the "project was announced last month" from the time of this posting.

We get an annual posting from Matt or Brian where they make fluffy remarks about how they're continuing to work on performance, what they'll deliver this year, and every single time they fail to make good on these promises.

So I'd like to ask, why exactly should we believe they are working on this project in good faith? Why should we expect something good to come from this, when they can't even take the time to communicate with their customers about it? Believing these things in spite of the enormous historical evidence to the contrary is, in my opinion, willful ignorance.

Edit: Just wanted to add that amidst all of this, they claim current server/code projects are "preventing them" from adding pvp content when there hasn't been pvp content in almost 5 years now. These are projects 8 months past their initial projected completion date that we have seen 0 observable progress on whatsoever. Really makes it feel like these are just an excuse to continue neglecting PVP like they've done for the better part of a decade now.

Lags

React wrote: »

laniakea_0 wrote: »

why do you think that? all he said was that it their efforts fell short of expectations and they had to undo some steps. it says nothing about them stopping the work on that all together. they'll continue working on it for the foreseeable future.

Because we have seen this exact scenario pan out over and over again with this company.

We had the cyrodiil performance testing period, where they ran a series of restrictive tests over the course of months, that they said would give them the information needed to fix the game. The end conclusion of these tests, months after they actually ended, was that they didn't get the information they needed and couldn't positively impact performance with what they had learned. They abandoned this project completely and haven't even bothered since to adjust or revert the final test of "no proc cyrodiil" which is still present in ravenwatch and no cp ic.

We had the 2020 "year of performance" where they promised to deliver a much better performing game each quarter that year. We ended 2020 with the worst cyrodiil performance in history at that point.

We've had the code rewrite project - announced in January of 2022, with an estimated completion date of December 2022, which we did not even receive an update on until November of 2022 (after being promised regular updates and communication on the project), at which point we were then told would begin rolling out in Q2 2023. This is nowhere to be found. It's actually confusing whether this project is the same one as the "server multithreading" one from this thread - because despite being asked numerous times, zenimax hasn't even bothered to clarify this. There is even further confusion on this subject by Matt's OP here which states that the "project was announced last month" from the time of this posting.

We get an annual posting from Matt or Brian where they make fluffy remarks about how they're continuing to work on performance, what they'll deliver this year, and every single time they fail to make good on these promises.

So I'd like to ask, why exactly should we believe they are working on this project in good faith? Why should we expect something good to come from this, when they can't even take the time to communicate with their customers about it? Believing these things in spite of the enormous historical evidence to the contrary is, in my opinion, willful ignorance.

Edit: Just wanted to add that amidst all of this, they claim current server/code projects are "preventing them" from adding pvp content when there hasn't been pvp content in almost 5 years now. These are projects 8 months past their initial projected completion date that we have seen 0 observable progress on whatsoever. Really makes it feel like these are just an excuse to continue neglecting PVP like they've done for the better part of a decade now.

I love how they just stick their head in the sand when things go badly. And how people like the one who asked you the question have nothing to say when given the exact correct answer of why we should have zero faith. But the thing that really gets me while reading your post, and really every day when dealing with this performance, is the fact that he is basically claiming victory with the hardware refresh.

"Fortunately, the recent upgrade to server hardware has resulted in more customer-facing server performance improvements than any of our multithreading work in all public realms where we’ve completed the work." This statement is is so out of touch its laughable. I have experienced lag in the last month like i never have before, and tonight is no different. Which is why im here, looking to see if this guy ever decided to respond or just keep hiding. But i already knew the answer.

This game performs worse than any other game i play. An average of 100 fps and 100 ping and its terrible, but its nothing to do with the fps or ping. The core of the game is fundamentally flawed in some way. Because the biggest issue is ability delay when everything else is fine. Low fps or something is understandable at some fights, but having a 3-5 second ability delay in the middle of nowhere on a tuesday afternoon is just a joke. No other mmo is like this. And ill never understand how they cannot get to the bottom of it.

Lags

ZOS_MattFiror wrote: »

It is approaching midyear and time to give an update on how our multithreading work is going. If you remember, back at the end of last year, I gave the news that we would be performing an ongoing series of server work aimed at spreading out ESO processes across more CPU cores, with the goal that it would result in a substantial increase in server performance. Over Update 37 – launched back in March – and in Update 38, we have completed some of this work.

Initial metrics, however, show that these steps in multithreading are not having as much of a performance improvement as desired, and so we had to roll back one of the process changes because it introduced an unacceptable level of instability. Fortunately, the recent upgrade to server hardware has resulted in more customer-facing server performance improvements than any of our multithreading work in all public realms where we’ve completed the work. (Xbox EU is still scheduled for later this year.) In fact, the recent Whitestrake’s Mayhem event in May showed just how much fun ESO PvP is with the new hardware in place.

All this to say, we continue to look for areas to spread process to optimize our server CPU capabilities. Please remember that we must be very, very careful when making changes of this complexity to ESO’s massive codebase. However, we will continue to dedicate resources to keep analyzing and looking for ways to utilize multithreading to improve server performance, and we will update everyone on any successes that we have when we find solutions.

Thanks for being the best community in gaming!

Matt

just another friendly reminder, from 3am on a tuesday in a very laggy gray host campaign, that the "recent upgrade to server hardware" has not improved performance. Cheers.

GooGa592

Lags wrote: »

ZOS_MattFiror wrote: »

It is approaching midyear and time to give an update on how our multithreading work is going. If you remember, back at the end of last year, I gave the news that we would be performing an ongoing series of server work aimed at spreading out ESO processes across more CPU cores, with the goal that it would result in a substantial increase in server performance. Over Update 37 – launched back in March – and in Update 38, we have completed some of this work.

Initial metrics, however, show that these steps in multithreading are not having as much of a performance improvement as desired, and so we had to roll back one of the process changes because it introduced an unacceptable level of instability. Fortunately, the recent upgrade to server hardware has resulted in more customer-facing server performance improvements than any of our multithreading work in all public realms where we’ve completed the work. (Xbox EU is still scheduled for later this year.) In fact, the recent Whitestrake’s Mayhem event in May showed just how much fun ESO PvP is with the new hardware in place.

All this to say, we continue to look for areas to spread process to optimize our server CPU capabilities. Please remember that we must be very, very careful when making changes of this complexity to ESO’s massive codebase. However, we will continue to dedicate resources to keep analyzing and looking for ways to utilize multithreading to improve server performance, and we will update everyone on any successes that we have when we find solutions.

Thanks for being the best community in gaming!

Matt

just another friendly reminder, from 3am on a tuesday in a very laggy gray host campaign, that the "recent upgrade to server hardware" has not improved performance. Cheers.

lag so bad today during guild raid that I disconnected six times and finally gave up and decided not to try logging back on for the seventh time. It takes 2-3 login attempts to get back on. Lag spikes to 2000+ milliseconds and it's like watching a slide show during big fights. PC NA

Twohothardware

Can we get a statement as to whether the team has looked into the ability to add Crossplay down the road? We need it on console with the current level of active players.

alternatelder

Twohothardware wrote: »

Can we get a statement as to whether the team has looked into the ability to add Crossplay down the road? We need it on console with the current level of active players.

No, we don't need it, a small group of people want it to ruin the game. Queues are bad enough, economy is screwed enough, trolling and toxicity is bad enough on console, I look to PC to escape it.

Twohothardware

alternatelder wrote: »

Twohothardware wrote: »

Can we get a statement as to whether the team has looked into the ability to add Crossplay down the road? We need it on console with the current level of active players.

No, we don't need it, a small group of people want it to ruin the game. Queues are bad enough, economy is screwed enough, trolling and toxicity is bad enough on console, I look to PC to escape it.

Crossplay would not increase queues, it would decrease them. The economy would not be screwed, it would be volatile for a couple of weeks and then stabilize at a new level that works for everyone. And players on PC are no more exempt from trolling and toxicity than anyone else, in fact most cheaters are on PC.

blktauna

The devs have repeatedly said crossplay will not happen.

Also lag is now interrupting my overland experience on PCEU and PCNA. Stutter lag and boomeranging ping. Please can this be acknowledged and addressed at some point?

alternatelder

Twohothardware wrote: »

alternatelder wrote: »

Twohothardware wrote: »

Can we get a statement as to whether the team has looked into the ability to add Crossplay down the road? We need it on console with the current level of active players.

No, we don't need it, a small group of people want it to ruin the game. Queues are bad enough, economy is screwed enough, trolling and toxicity is bad enough on console, I look to PC to escape it.

Crossplay would not increase queues, it would decrease them. The economy would not be screwed, it would be volatile for a couple of weeks and then stabilize at a new level that works for everyone. And players on PC are no more exempt from trolling and toxicity than anyone else, in fact most cheaters are on PC.

My experience on PC after being on console for 6+ is that PC is a lot more friendly, there is way less trolling.

Lags

alternatelder wrote: »

Twohothardware wrote: »

Can we get a statement as to whether the team has looked into the ability to add Crossplay down the road? We need it on console with the current level of active players.

No, we don't need it, a small group of people want it to ruin the game. Queues are bad enough, economy is screwed enough, trolling and toxicity is bad enough on console, I look to PC to escape it.

its so crazy that you people cant see that you're actually the toxic ones. Labeling an entire platform toxic because you had some bad experiences. Well i hate to break it to you but people are toxic everywhere. Ignore them and move on. And wdym ques are bad enough? As in it takes 10 years to get a dungeon que? Or do you mean ques into pvp, because thats just not true anymore. Unless you mean the like one day we had a que to log in, which almost never ever happens.

As for the economy, the economy is just fine on xbox. You can thank addons like tamriel trade center for the terrible pc economy. Among other things, like bots. tbh idk why zos allows these things.

In reality, with how many people zos has chased away, cross play would be ideal. Unfortunately i dont think the game is stable enough in general to handle any of that

blktauna

This is not the cross play thread, it's the Multithreading update/once as re achitecture thread.

Has any mention been made of this in the upcoming Q4 updated? I am not on twitter and wondered if they've said anything.

Lags

blktauna wrote: »

This is not the cross play thread, it's the Multithreading update/once as re achitecture thread.

Has any mention been made of this in the upcoming Q4 updated? I am not on twitter and wondered if they've said anything.

oh it is? I had no idea. And i havent seen anything else about it, but i doubt it considering the tone of the post.

blktauna

That's the problem. Another empty promise and thats extremely dissappointing.

RMW

But I'm waitin', yeah, I'm waitin'
For you to pick up the phone
And tell me that you're done with the code-rewrite
Yeah, I'm waitin', I'm waitin'
Every minute of every day
Just to hear you say
"New PvP Update coming soon!"
So I'm waitin'

https://www.youtube.com/watch?v=CGyfu6Hfy7U

For the wait

Twohothardware

This game has been out for almost 10 years. Why is there not an Elder Scrolls Online 2 on a newly built engine on the near horizon? One with full crossplay of all platforms and no in combat bug.

Lags

Twohothardware wrote: »

This game has been out for almost 10 years. Why is there not an Elder Scrolls Online 2 on a newly built engine on the near horizon? One with full crossplay of all platforms and no in combat bug.

I would never trust this company again. They have failed completely with performance, and then act like its better now. And then ignore everyone questioning that statement. No thanks.

This reminds me of around 2018 when they would tell you to check with your ISP when you would report how terrible cyrodil performance was.

Everything that matters with zos is put on the back burner, while pointless things get priority. Performance should have been priority number 1 for many years, and they should have spent whatever amount of money needed to fix the issue. But the truth is, its never really mattered much because it doesnt impact the super casual audience they are targeting. So they will work on it, slowly over many years. And it will stay terrible. Very unfortunate situation.

ProudMary

Lags wrote: »

Twohothardware wrote: »

This game has been out for almost 10 years. Why is there not an Elder Scrolls Online 2 on a newly built engine on the near horizon? One with full crossplay of all platforms and no in combat bug.

I would never trust this company again. They have failed completely with performance, and then act like its better now. And then ignore everyone questioning that statement. No thanks.

This reminds me of around 2018 when they would tell you to check with your ISP when you would report how terrible cyrodil performance was.

Everything that matters with zos is put on the back burner, while pointless things get priority. Performance should have been priority number 1 for many years, and they should have spent whatever amount of money needed to fix the issue. But the truth is, its never really mattered much because it doesnt impact the super casual audience they are targeting. So they will work on it, slowly over many years. And it will stay terrible. Very unfortunate situation.

The game has definitely changed focus dramatically since release. No debating that.

blktauna

So has any progress been made to report on?

Lag has invaded my experiences overland in the newest area of the Peninsula. PVP has dragged down incredibly.

Please do not suggest your silly (fixes) they do not work and never have.

@ZOS_MattFiror where are in this?

Wolfkeks

Any update?

Elsonso

My expectation is that near the end of the year, they will let us know that they stopped working on it.

blktauna

Elsonso wrote: »

My expectation is that near the end of the year, they will let us know that they stopped working on it.

I'm amazed you think they will actuall communicate that solidly.

I expect more prevarication and dithering to stem the bad PR that will come.

RMW

At this point the new PvP update will come 2030

OtarTheMad

Marto wrote: »

RicAlmighty wrote: »

GooGa592 wrote: »

One question that needs to be answered by the dev team at ZOS:

How is it that in 2016-2018 ZOS was able to deliver fantastic performance in Cyrodiil with 24 man group size and quadruple the population cap than we have today, but in 2023 this is not possible?

This is the pertinent question. It’s not as though this game has never had good performance and this was an attempt to make it better. It *was* better, and now its worse. Why? What caused it? And what can we do to mitigate the performance degradation that those changes introduced?

Rich Lambert actually talked about this during some of his livestreams.

It's not the only reason, but one of the biggest factors was the addition of Volendrung, and destructible gates/bridges. For some unknown reason, the addition of those things dramatically reduced Cyrodiil performance.

Paradoxically, removing those features in their internal test servers did not result in performance returning to the state it was before. Whatever was broken could not be un-broken.

We know from datamining that ZOS originally planned and developed more content for Cyrodiil, like more artifact weapons and more map features. But all that content had to be put on hold until performance was improved. That's why they've spent years and years doing multiple public and private tests with things like skill cooldowns, AoE limits, proc-sets, champion points, etc.

This is really not ZOS being "lazy" or trying to do things for cheap. The cheapest and easiest option would be for them to just release all that content, and care not how bad performance gets. Instead they spent years and years working on all sorts of improvements of hardware and software.

But honestly, I also feel people are looking at the past with rose tinted glasses. Performance in 2016-2018 was absolutely not better than it is right now. Different issues existed, like packet congestion as a result of too much data being handled by the server, poor lag compensation, and poor particle-related performance.

^^this. I agree completely.

I feel like ZOS is trying when it comes to PvP and I remember those streams where Rich mentioned those things and it made me realize they do in fact try but it’s harder than we all thought.

One thing that I feel like was also lost to time was when ZOS came out, believe it was on a ESO Live or a Q&A, I don’t remember but they told everyone basically that it’s hard for them to find why PvP is the way it is because they cannot duplicate Live Cyrodiil conditions internally. That is why we had those live tests, it was easier on them.

I miss Rich’s streams because we got so much information about what is going on behind closed doors and I liked it and I think it was good for the game.

OtarTheMad

Also, this is where my confusion comes in on what is what. I thought they were already working on multi threading when they found the issue with PvP and started to plan a re-architecture. This is the forum thread about the PvP code:

https://forums.elderscrollsonline.com/en/discussion/595442/eso-pvp-update-updated-june-2022/p1

So is the multi threading now also including the PvP re-write?

I see a lot of people saying that 2020, the year of performance, was a fail because things got worse and they did. 2020 was the first year I really had PvP lag BUT what we fail to remember is that’s when the world shut down unexpectedly. The devs made all these plans and then, boom, had to work from home and everything had to change. Things even became harder because certain things they needed for fixes were on massive back order because of COVID and the shut downs.

We really need to be updated more often though about what is going on even if it is just a blip during an ESO Live promoting a new DLC.

reazea

OtarTheMad wrote: »

Marto wrote: »

RicAlmighty wrote: »

GooGa592 wrote: »

One question that needs to be answered by the dev team at ZOS:

How is it that in 2016-2018 ZOS was able to deliver fantastic performance in Cyrodiil with 24 man group size and quadruple the population cap than we have today, but in 2023 this is not possible?

This is the pertinent question. It’s not as though this game has never had good performance and this was an attempt to make it better. It *was* better, and now its worse. Why? What caused it? And what can we do to mitigate the performance degradation that those changes introduced?

Rich Lambert actually talked about this during some of his livestreams.

It's not the only reason, but one of the biggest factors was the addition of Volendrung, and destructible gates/bridges. For some unknown reason, the addition of those things dramatically reduced Cyrodiil performance.

Paradoxically, removing those features in their internal test servers did not result in performance returning to the state it was before. Whatever was broken could not be un-broken.

We know from datamining that ZOS originally planned and developed more content for Cyrodiil, like more artifact weapons and more map features. But all that content had to be put on hold until performance was improved. That's why they've spent years and years doing multiple public and private tests with things like skill cooldowns, AoE limits, proc-sets, champion points, etc.

This is really not ZOS being "lazy" or trying to do things for cheap. The cheapest and easiest option would be for them to just release all that content, and care not how bad performance gets. Instead they spent years and years working on all sorts of improvements of hardware and software.

But honestly, I also feel people are looking at the past with rose tinted glasses. Performance in 2016-2018 was absolutely not better than it is right now. Different issues existed, like packet congestion as a result of too much data being handled by the server, poor lag compensation, and poor particle-related performance.

^^this. I agree completely.

I feel like ZOS is trying when it comes to PvP and I remember those streams where Rich mentioned those things and it made me realize they do in fact try but it’s harder than we all thought.

One thing that I feel like was also lost to time was when ZOS came out, believe it was on a ESO Live or a Q&A, I don’t remember but they told everyone basically that it’s hard for them to find why PvP is the way it is because they cannot duplicate Live Cyrodiil conditions internally. That is why we had those live tests, it was easier on them.

I miss Rich’s streams because we got so much information about what is going on behind closed doors and I liked it and I think it was good for the game.

Many of us that have been PvP players since 2014-2015 have seen enough empty PR that we are highly skeptical of any PR regarding PVP and those creating it.

Have you seen the "waaah waaah waaah" video? That's the straw that broke the proverbial camels back for a lot of us.

OtarTheMad

reazea wrote: »

OtarTheMad wrote: »

Marto wrote: »

RicAlmighty wrote: »

GooGa592 wrote: »

One question that needs to be answered by the dev team at ZOS:

How is it that in 2016-2018 ZOS was able to deliver fantastic performance in Cyrodiil with 24 man group size and quadruple the population cap than we have today, but in 2023 this is not possible?

This is the pertinent question. It’s not as though this game has never had good performance and this was an attempt to make it better. It *was* better, and now its worse. Why? What caused it? And what can we do to mitigate the performance degradation that those changes introduced?

Rich Lambert actually talked about this during some of his livestreams.

It's not the only reason, but one of the biggest factors was the addition of Volendrung, and destructible gates/bridges. For some unknown reason, the addition of those things dramatically reduced Cyrodiil performance.

Paradoxically, removing those features in their internal test servers did not result in performance returning to the state it was before. Whatever was broken could not be un-broken.

We know from datamining that ZOS originally planned and developed more content for Cyrodiil, like more artifact weapons and more map features. But all that content had to be put on hold until performance was improved. That's why they've spent years and years doing multiple public and private tests with things like skill cooldowns, AoE limits, proc-sets, champion points, etc.

This is really not ZOS being "lazy" or trying to do things for cheap. The cheapest and easiest option would be for them to just release all that content, and care not how bad performance gets. Instead they spent years and years working on all sorts of improvements of hardware and software.

But honestly, I also feel people are looking at the past with rose tinted glasses. Performance in 2016-2018 was absolutely not better than it is right now. Different issues existed, like packet congestion as a result of too much data being handled by the server, poor lag compensation, and poor particle-related performance.

^^this. I agree completely.

I feel like ZOS is trying when it comes to PvP and I remember those streams where Rich mentioned those things and it made me realize they do in fact try but it’s harder than we all thought.

One thing that I feel like was also lost to time was when ZOS came out, believe it was on a ESO Live or a Q&A, I don’t remember but they told everyone basically that it’s hard for them to find why PvP is the way it is because they cannot duplicate Live Cyrodiil conditions internally. That is why we had those live tests, it was easier on them.

I miss Rich’s streams because we got so much information about what is going on behind closed doors and I liked it and I think it was good for the game.

Many of us that have been PvP players since 2014-2015 have seen enough empty PR that we are highly skeptical of any PR regarding PVP and those creating it.

Have you seen the "waaah waaah waaah" video? That's the straw that broke the proverbial camels back for a lot of us.

I have been playing since beta so I get that. For me, I didn’t have issues with PvP and lag really until 2020 but I’ve seen a lot of guildies and friends leave this game over the years.

I’ve seen that video, I was watching live and I truly think that got blown WAY out of proportion. The person who asked was doing a laugh emoji in chat and saying LOL to her answer. It was almost like they asked as a joke and it just went nuclear.

The timing overall was bad I get it because that was after COVID and other things completely changed any plans which made players mad. The lack of communication made it seem that devs weren’t working on fixing anything etc.

The issue I really think is just constant communication. ZOS should probably put PvP re-architecture and lag fix on a bulletin board and seriously just update us all at least quarterly when they have those ESO Lives promoting the next DLC. Even making it part of the article afterwards for those who missed it, a small thing like that will go far I think. At least for me it will.

TechMaybeHic

I can't believe people don't understand they abandoned it, if they ever were even actually doing it. Any of it. They just changed what they called it a few times, then said "welp, it broke more than it fixed so we're not going forward with it"

RMW

OtarTheMad wrote: »

The issue I really think is just constant communication. ZOS should probably put PvP re-architecture and lag fix on a bulletin board and seriously just update us all at least quarterly when they have those ESO Lives promoting the next DLC. Even making it part of the article afterwards for those who missed it, a small thing like that will go far I think. At least for me it will.

On this we can all agree. We just get no news and always asking for it only to be met by silence is just frustrating.
ZOS also has to realize that at some point you will lose people. You can't really expect everyone to stick around after promising something and then just stay silent for months on end.

ProudMary

OtarTheMad wrote: »

reazea wrote: »

OtarTheMad wrote: »

Marto wrote: »

RicAlmighty wrote: »

GooGa592 wrote: »

One question that needs to be answered by the dev team at ZOS:

How is it that in 2016-2018 ZOS was able to deliver fantastic performance in Cyrodiil with 24 man group size and quadruple the population cap than we have today, but in 2023 this is not possible?

This is the pertinent question. It’s not as though this game has never had good performance and this was an attempt to make it better. It *was* better, and now its worse. Why? What caused it? And what can we do to mitigate the performance degradation that those changes introduced?

Rich Lambert actually talked about this during some of his livestreams.

It's not the only reason, but one of the biggest factors was the addition of Volendrung, and destructible gates/bridges. For some unknown reason, the addition of those things dramatically reduced Cyrodiil performance.

Paradoxically, removing those features in their internal test servers did not result in performance returning to the state it was before. Whatever was broken could not be un-broken.

We know from datamining that ZOS originally planned and developed more content for Cyrodiil, like more artifact weapons and more map features. But all that content had to be put on hold until performance was improved. That's why they've spent years and years doing multiple public and private tests with things like skill cooldowns, AoE limits, proc-sets, champion points, etc.

This is really not ZOS being "lazy" or trying to do things for cheap. The cheapest and easiest option would be for them to just release all that content, and care not how bad performance gets. Instead they spent years and years working on all sorts of improvements of hardware and software.

But honestly, I also feel people are looking at the past with rose tinted glasses. Performance in 2016-2018 was absolutely not better than it is right now. Different issues existed, like packet congestion as a result of too much data being handled by the server, poor lag compensation, and poor particle-related performance.

^^this. I agree completely.

I feel like ZOS is trying when it comes to PvP and I remember those streams where Rich mentioned those things and it made me realize they do in fact try but it’s harder than we all thought.

One thing that I feel like was also lost to time was when ZOS came out, believe it was on a ESO Live or a Q&A, I don’t remember but they told everyone basically that it’s hard for them to find why PvP is the way it is because they cannot duplicate Live Cyrodiil conditions internally. That is why we had those live tests, it was easier on them.

I miss Rich’s streams because we got so much information about what is going on behind closed doors and I liked it and I think it was good for the game.

Many of us that have been PvP players since 2014-2015 have seen enough empty PR that we are highly skeptical of any PR regarding PVP and those creating it.

Have you seen the "waaah waaah waaah" video? That's the straw that broke the proverbial camels back for a lot of us.

I have been playing since beta so I get that. For me, I didn’t have issues with PvP and lag really until 2020 but I’ve seen a lot of guildies and friends leave this game over the years.

I’ve seen that video, I was watching live and I truly think that got blown WAY out of proportion. The person who asked was doing a laugh emoji in chat and saying LOL to her answer. It was almost like they asked as a joke and it just went nuclear.

The timing overall was bad I get it because that was after COVID and other things completely changed any plans which made players mad. The lack of communication made it seem that devs weren’t working on fixing anything etc.

The issue I really think is just constant communication. ZOS should probably put PvP re-architecture and lag fix on a bulletin board and seriously just update us all at least quarterly when they have those ESO Lives promoting the next DLC. Even making it part of the article afterwards for those who missed it, a small thing like that will go far I think. At least for me it will.

It's the lack of results over many years of promises to fix issues in cyrodiil that make it seem that devs aren't working on fixing anything. It's the lack of results far more than lack of communication.

We've all seen many times over and over again that ZOS can make Cyrodiil perform better by making improvements on their server end. If you've been around as long as you say, then you remember how much better cyrodiil would perform during MYM events right after maintenance, then decline again when the event was over. And you remember how much better performance was when they upgraded their servers. It didn't last long, but it was noticeable, temporary improvements after ZOS made adjustments to their servers that resulted in better performance. So we know ZOS can make it better if they want to. Are they still censoring the mention of cyodiil or PvP in their live streams?