ESO PvP Update [Updated June 2022]

Sylvermynx

FrancisCrawford wrote: »

BloodyStigmata wrote: »

Aside from the badly needed PVP performance fixes that this will supposedly address, what else can we expect from this revision of the foundational code? Will this potentially open up other avenues for different types of content or customization that weren't possible before?

They don't know. Because when they fix the worst bottlenecks they now know of, hard-to-predict new ones will come to the fore.

https://www.dbms2.com/2009/08/21/bottleneck-whack-a-mole/

That's a great article - and actually it's still spot on so many years later. Or at least it is with the database systems I use (mostly SQL and its offshoots).

I've always wondered what database systems ZOS uses (just like I did with WoW and RIFT when I played those games)....

VaranisArano

FrancisCrawford wrote: »

BloodyStigmata wrote: »

Aside from the badly needed PVP performance fixes that this will supposedly address, what else can we expect from this revision of the foundational code? Will this potentially open up other avenues for different types of content or customization that weren't possible before?

They don't know. Because when they fix the worst bottlenecks they now know of, hard-to-predict new ones will come to the fore.

https://www.dbms2.com/2009/08/21/bottleneck-whack-a-mole/

That's pretty much what happened with the Groupfinder rework, from my outside perspective.

Lots of "we figured it out!"
"No, wait, the next step in line failed."
"We fixed that problem!"
"Turns out something else failed under load..."
"We need more time..."
"We really got it this time!"

Celephantsylvius_Bornasfinmo

I do like the wording and promise of 'A plan' and 're-architecture of the server'.
Thank you for the update, Mr Firor.

wildbear247

Thanks for the update Matt. While the ongoing Cyrodiil performance experiment continues, can we get some discussion about creating a mode that would satisfy PvPers who really enjoy the Cyrodiil PvP experience but don't want to deal with the horrendous lag, unbalanced alliance populations, etc? For example, an instanced 30v30 castle siege mode where players have a time limit to take the flags and flip the keep? Doesn't have to be exactly that, most importantly it's just ZOS engaging the players to explore ways to ensure PvP stays fun.

I've been playing ESO since 2015 and Cyrodiil performance seems to have gotten worse since the old days. And if it's going to take at least a year before the planned overhauls makes a difference, then I think it would be wise to discuss other modes in the interim.

Elsonso

.

Musax wrote: »

Pretty telling in what state the remaining PvP playerbase is when the message "after 3years of investigating we have now decided to start from scratch, don't expect any performance improvememts for another year" is widely seen as a huge positive thing...how dire must the mindset be?

Well, stop and think about what is happening here. They could have just said "as-is" and put no further work into PVP or Cyrodiil performance. I mean, that is what it feels like they have been doing all this time, anyway. Saying it would have gotten people up in arms, but many would have expected it.

What that are actually doing is spending money to basically rewrite a portion of the "engine" (server code), which is not a trivial, or cheap, prospect. You can probably tell from the way he talked about rolling it out that it is not insignificant work. It is like changing drivers in a car while the car is motoring down a curvy highway at 70 MPH.

Yeah, the timeline is a lot longer than ideal, but it is a huge positive thing, if they can pull it off.

Destai wrote: »

neferpitou73 wrote: »

Madhatten512 wrote: »

I find the timing of this post interesting. So if your creative directors wife hadn't rammed her foot in her mouth and insulted a portion of the player base would this post have even been made? People have been telling you for years the servers need updated this isn't news.

That's my concern. As this thread and the thread on 'the incident' show there's not a lot of trust between the devs and players. ZOS needs to do something to fix that over the next year while they (hopefully) fix the servers.

Well, trust is earned.

On both sides.

Destai

Elsonso wrote: »

.

Musax wrote: »

Pretty telling in what state the remaining PvP playerbase is when the message "after 3years of investigating we have now decided to start from scratch, don't expect any performance improvememts for another year" is widely seen as a huge positive thing...how dire must the mindset be?

Well, stop and think about what is happening here. They could have just said "as-is" and put no further work into PVP or Cyrodiil performance. I mean, that is what it feels like they have been doing all this time, anyway. Saying it would have gotten people up in arms, but many would have expected it.

What that are actually doing is spending money to basically rewrite a portion of the "engine" (server code), which is not a trivial, or cheap, prospect. You can probably tell from the way he talked about rolling it out that it is not insignificant work. It is like changing drivers in a car while the car is motoring down a curvy highway at 70 MPH.

Yeah, the timeline is a lot longer than ideal, but it is a huge positive thing, if they can pull it off.

Destai wrote: »

neferpitou73 wrote: »

Madhatten512 wrote: »

I find the timing of this post interesting. So if your creative directors wife hadn't rammed her foot in her mouth and insulted a portion of the player base would this post have even been made? People have been telling you for years the servers need updated this isn't news.

That's my concern. As this thread and the thread on 'the incident' show there's not a lot of trust between the devs and players. ZOS needs to do something to fix that over the next year while they (hopefully) fix the servers.

Well, trust is earned.

On both sides.

I spent $300 on crown store stuff over the holidays, have recruited multiple friends, and played the game for 7 years. Yeah. Trust is earned.

rexagamemnon

Awesome! This is great news indeed.
Just a friendly reminder, server transfers if at all possible lets make it happen after this is resolved.
Also, for weekend events, might i suggest double AP when ever the Golden vendor is available.

frozywozy

ZOS_MattFiror wrote: »

We’ve been planning this for the last quarter and we’re starting work on it right now.

I think we can all appreciate the honesty that nothing serious has been done in the first 7 years to deal with server performances and database optimization. It feels good to hear that somebody at Zenimax has finally been brave enough to say it. I will certainly be keeping an eye out for further communication and improvements. Hopefully the riddim at which we get these updates will be better than once every year or two.

Also, even if I don't play the game anymore, I really hope for all current players that Zenimax will do a big effort to surround themselves with active pvpers who play in organized groups on a regular basis and who know what the real problems are.

Finally, I hope that after every major update, Zenimax will start monitoring the game which means going in Cyrodiil for several hours straight everyday after the patch hits and paying attention to the zone chat to read what the new main problems are, then putting them on paper and associating them to that specific patch, so devs later on have a better idea where to look in the code.

The reason why I'm insisting on those particular points is because I have had proof in the past that some staff at Zenimax are not aware at all of what's going on and when problems got introduced. It really shows that they don't have staff members monitoring after big updates or discussing with active players in Cyrodiil.

Izanagi.Xiiib16_ESO

ZOS_MattFiror wrote: »

.

In the meantime, we’ll consider additional ways to keep Cyrodiil and Battlegrounds interesting and exciting (for example, potentially special rulesets or weekend events) and will continue to work on bug fixes affecting combat, but we won’t be adding any new features to PvP until the work mentioned above is complete. Doing so would not only prolong the fixes we want to implement but would also exacerbate the issue of poor performance.

@ZOS_Kevin perhaps some additional points could be addressed.

1) Why is it the PvP Dev team has to work on server upgrades instead of PvP content, wouldn't such a fundamental core change be best suited to experienced Devs from each team not just 1?

2) PvP has not received any features (and in fact has had content removed more then added) for many years at this point
Why are changes to content which could help performance not being considered or not impactful changes made in addition to this behind the scenes work? E.g. some sort of population balancing, changes to volendruung, PvP rewards and incentives, group duelling and most importantly major PvP bug fixes (e.g. 49000 day debuffs and positional desyncs after knock back or up CCs)

3) why has it taken so long to inform us of any progress. The original tests were run in 2020. All that came out of them from a players eyes is limited sets No CP campaign and 12m group size with 1 essentially announcement of each. No real follow up to player questions or further detail.

Thanks

Miracle19

still doesn't changed the fact deathmatch que is going away. bad call

Cireous

You may be saying PVP, but when I see server rewrite and dramatic performance increases, I am instead reading: Furnishing limit increases and unlimited Furnishing Storage.

VaranisArano

Cireous wrote: »

You may be saying PVP, but when I see server rewrite and dramatic performance increases, I am instead reading: Furnishing limit increases and unlimited Furnishing Storage.

We don't know, but I'm hopeful that if ZOS can address the performance impact of lots of effects going off in the same small area in Cyrodiil, housing can also benefit.

Alucu

Now this is the statement I was looking for. I'm sure many of us will keep tabs on your progress and help with the tests if we can

VvwvenomwvV

"We need to essentially rewrite some of the foundational server code to account for it. This should dramatically increase server performance, but obviously we will need to test and evaluate as we go along".

They are going to rearchitect the server. 

Only after every other option was exhausted was the the main problem, and probably most expensive problem finally addressed.

I hope that "we will need to test and evaluate as we go along" isn't another way of saying that they will only do what is necessary to see some improvements and stop at that point.

I'm sure it will be an expensive fix, but it is necessary for the survival of Eso.

The_Lex

Although I may not be happy with the details or with how long this has been dragging on before an update, I am truly grateful for the response.

caindele

The issues are in the foundational server core software...as Matt said, you have to be very careful because you don't want to blow up a live game. It will take a lot of time because basically you need to jack up the house and replace the foundation while people are still living in the house....and you have to maintain all of the functionality of that house at the same time. If they put an addition onto that house that is on a bad foundation, the whole house can collapse.

If it was an easy fix, they would have done it already. ZoS has nothing to gain by not fixing it, if it was easy. Making changes to the server code can be tested, but you will never be able to get the same or simulate the server load that you can on a live server. So, it will go through extensive QA, then onto PTS, but they really won't know how much of an improvement until it is live and players start playing on it.

Matt's statement just reinforces what Rich has been saying, that they have a handle on the issue and it is deep in the code and will take a long time to fix. I still don't get what people want them to do....I guess there is no pleasing some people. Look at it this way, they could have made the decision to just remove PVP from the game, but they decided instead to keep working on it, to make it a better experience.

Anyway, thanks to ZOS for continuing to work on ESO and continually looking to improve it.

blktauna

They've known since probably before the year of testing what was needed. They should have started their branch there and got to work. Even if it was after that testing, the process to rework the code should have begun then. Yes it will take forever, I have no doubts on that, but the fact they've only JUST STARTED is maddening.

Lebkuchen

@caindele
"but they decided instead to keep working on it"
yes, they really love "working on it" instead of fixing it.

"I still don't get what people want them to do" ...
i guess most people just want to play the game they are paying for (without all the problems that are being "worked on" for years)

Trevore

caindele wrote: »

The issues are in the foundational server core software...as Matt said, you have to be very careful because you don't want to blow up a live game. It will take a lot of time because basically you need to jack up the house and replace the foundation while people are still living in the house....and you have to maintain all of the functionality of that house at the same time. If they put an addition onto that house that is on a bad foundation, the whole house can collapse.

If it was an easy fix, they would have done it already. ZoS has nothing to gain by not fixing it, if it was easy. Making changes to the server code can be tested, but you will never be able to get the same or simulate the server load that you can on a live server. So, it will go through extensive QA, then onto PTS, but they really won't know how much of an improvement until it is live and players start playing on it.

Matt's statement just reinforces what Rich has been saying, that they have a handle on the issue and it is deep in the code and will take a long time to fix. I still don't get what people want them to do....I guess there is no pleasing some people. Look at it this way, they could have made the decision to just remove PVP from the game, but they decided instead to keep working on it, to make it a better experience.

Anyway, thanks to ZOS for continuing to work on ESO and continually looking to improve it.

I mean, I'm sure there will be a lot of downtime maintenance when that day comes (which I personally would welcome, since there's other things I could do), but if it actually fixes the game rather than break it, I'm all for it. Even if they have to even restructure the hardware itself to get the job done, that's fine. Who knows, maybe they'll end up making room for opportunities we didn't even expect, but I won't push my luck with that; even a small change is better than none.

Karivaa

Thank you soooo much!

Wolfshade

Respect and thanks for this decision . I would not have expected this fundamental but necessary step. Preparing eso for the next decade, is the right step at the right time!

Lysette

This is a brave and necessary decision imo, because the server technology was designed under premises which are no longer valid and anything what could be done with the old one would more or less just be band aids and workarounds. Designing it from the ground up is a risk to take, especially when the transition into the live game has to be made, that is as well why I think this is a really brave decision to make - I just hope, when the time to transition to the new technology comes, that we won't experience major service interruptions, so please take the time to test it as much as possible before putting it live - and please plan for a rewind, just in case it leads to major service disruptions - it is a risky step and a fallback should be possible.

Sylvermynx

Actually, there's no need to bollix live servers while they tweak (um.... that's a very poor word for massive server architecture changes.... sorry) the base code. When you do stuff like this, you set up a separate test area, copy the current code there, and only then start whacking. And you keep redundant backups of what you do on an hourly basis.

With some program code - especially database - you have to be very careful how you iterate a live server on a test platform: as a long time wordpress user/site manager, I can guarantee that if you don't manage a dup'd database appropriately, you WILL have major issues. And an SQL wordpress database is probably kindergarten compared to what ZOS is dealing with.

I'm sure the database techs for this AAA game company will manage things in the safest way possible.

Kwoung

If this actually happens before something better to play comes out or New World pulls a "ESO" and turns their game around to be something great, that would be cool. But at this point, what reassurances do we have as players that this isn't just another in the very long line of yearly posts saying something will be done, but isn't? In just the recent years we had... Year of performance improvements... complete bust. Year of new servers... someone forgot to order them apparently.

Also, not to be confrontational, but a recent video made fun of PVPers because it is supposedly a well know fact that PVP was being worked on forever... and here we now hear "We’ve been planning this for the last quarter and we’re starting work on it right now"?

All I can say is, please deliver and provide updates on progress to the community along the way. We can be very understanding and if you hit a bump in the road, let us know, make a breakthrough, let us know. Communication is key and a very simple method of keeping you customers happy. Also, most MMO's now have a public roadmap posted in one form or another, please consider doing the same. It doesn't have to contain spoilers, just a general idea of what is being worked on and the progress of those items.

Thrasher91604

Fantastic news! Ths has been recognized by many as something that has been needed for years. So glad ZOS is signing up to it!

Lysette

Kwoung wrote: »

If this actually happens before something better to play comes out or New World pulls a "ESO" and turns their game around to be something great, that would be cool. But at this point, what reassurances do we have as players that this isn't just another in the very long line of yearly posts saying something will be done, but isn't? In just the recent years we had... Year of performance improvements... complete bust. Year of new servers... someone forgot to order them apparently.

Also, not to be confrontational, but a recent video made fun of PVPers because it is supposedly a well know fact that PVP was being worked on forever... and here we now hear "We’ve been planning this for the last quarter and we’re starting work on it right now"?

All I can say is, please deliver and provide updates on progress to the community along the way. We can be very understanding and if you hit a bump in the road, let us know, make a breakthrough, let us know. Communication is key and a very simple method of keeping you customers happy. Also, most MMO's now have a public roadmap posted in one form or another, please consider doing the same. It doesn't have to contain spoilers, just a general idea of what is being worked on and the progress of those items.

Nah, this step is admitting - we can't really achieve it with the old design, so if we want to get it done (and we do), we really need to redesign it. because the old technology was based on premises, which have changed so drastically, that it requires a redesign to address the issues. This is about what Matt's decision is based on and I think he is completely right with it.

A roadmap in regards to server redesign - nah, what would you do with it - give them a chance to be flexible and innovative, don't put them into a cage of a roadmap to achieve - this is creative work, redesigning the server architecture, just let them do it and be free to make changes or even discard steps they took, because they found a better solution meanwhile - a roadmap could hinder this creative freedom. "It will take most of this year" is as good as it gets as a roadmap at this point in time - they don't really know it, it is an estimate.

And it is not about ordering new servers, but changing the information processes and how different services work together.

NanoTechnicianHQ

A lot has happened in terms of server networking since using 2011 protocols, Its great to hear something is happening and thanks for keeping us informed!

Kwoung

Lysette wrote: »

Kwoung wrote: »

If this actually happens before something better to play comes out or New World pulls a "ESO" and turns their game around to be something great, that would be cool. But at this point, what reassurances do we have as players that this isn't just another in the very long line of yearly posts saying something will be done, but isn't? In just the recent years we had... Year of performance improvements... complete bust. Year of new servers... someone forgot to order them apparently.

Also, not to be confrontational, but a recent video made fun of PVPers because it is supposedly a well know fact that PVP was being worked on forever... and here we now hear "We’ve been planning this for the last quarter and we’re starting work on it right now"?

All I can say is, please deliver and provide updates on progress to the community along the way. We can be very understanding and if you hit a bump in the road, let us know, make a breakthrough, let us know. Communication is key and a very simple method of keeping you customers happy. Also, most MMO's now have a public roadmap posted in one form or another, please consider doing the same. It doesn't have to contain spoilers, just a general idea of what is being worked on and the progress of those items.

Nah, this step is admitting - we can't really achieve it with the old design, so if we want to get it done (and we do), we really need to redesign it. because the old technology was based on premises, which have changed so drastically, that it requires a redesign to address the issues. This is about what Matt's decision is based on and I think he is completely right with it.

A roadmap in regards to server redesign - nah, what would you do with it - give them a chance to be flexible and innovative, don't put them into a cage of a roadmap to achieve - this is creative work, redesigning the server architecture, just let them do it and be free to make changes or even discard steps they took, because they found a better solution meanwhile - a roadmap could hinder this creative freedom.

Umm ok, so basically you are saying "it will be different this time". I would love to believe that, but after 6+ years of "we are working on it", I am a tad jaded and feel a bit more than yearly words are required at this point. Also due to timing, this seems a lot more like damage control to a viral video than a sincere attempt at improving the players game experience, which could in fact, have been done at any time in years past.

If it took that video to spur actual action, then that is awesome, because quite honestly, I love playing PVP in this game and if it actually performed, I will probably play ESO for another decade, even though the PVE end of things has pretty much dried up for me. The yearly chapters or whatever they are called are great most of the time, but that's like a weeks worth of play and doesn't come near to filling my time in game... PVP keeps it fresh and since I am here, I still spend about $1000 in the crown store every year.

Lysette

Kwoung wrote: »

Lysette wrote: »

Kwoung wrote: »

If this actually happens before something better to play comes out or New World pulls a "ESO" and turns their game around to be something great, that would be cool. But at this point, what reassurances do we have as players that this isn't just another in the very long line of yearly posts saying something will be done, but isn't? In just the recent years we had... Year of performance improvements... complete bust. Year of new servers... someone forgot to order them apparently.

Also, not to be confrontational, but a recent video made fun of PVPers because it is supposedly a well know fact that PVP was being worked on forever... and here we now hear "We’ve been planning this for the last quarter and we’re starting work on it right now"?

All I can say is, please deliver and provide updates on progress to the community along the way. We can be very understanding and if you hit a bump in the road, let us know, make a breakthrough, let us know. Communication is key and a very simple method of keeping you customers happy. Also, most MMO's now have a public roadmap posted in one form or another, please consider doing the same. It doesn't have to contain spoilers, just a general idea of what is being worked on and the progress of those items.

Nah, this step is admitting - we can't really achieve it with the old design, so if we want to get it done (and we do), we really need to redesign it. because the old technology was based on premises, which have changed so drastically, that it requires a redesign to address the issues. This is about what Matt's decision is based on and I think he is completely right with it.

A roadmap in regards to server redesign - nah, what would you do with it - give them a chance to be flexible and innovative, don't put them into a cage of a roadmap to achieve - this is creative work, redesigning the server architecture, just let them do it and be free to make changes or even discard steps they took, because they found a better solution meanwhile - a roadmap could hinder this creative freedom.

Umm ok, so basically you are saying "it will be different this time". I would love to believe that, but after 6+ years of "we are working on it", I am a tad jaded and feel a bit more than yearly words are required at this point. Also due to timing, this seems a lot more like damage control to a viral video than a sincere attempt at improving the players game experience, which could in fact, have been done at any time in years past.

If it took that video to spur actual action, then that is awesome, because quite honestly, I love playing PVP in this game and if it actually performed, I will probably play ESO for another decade, even though the PVE end of things has pretty much dried up for me. The yearly chapters or whatever they are called are great most of the time, but that's like a weeks worth of play and doesn't come near to filling my time in game... PVP keeps it fresh and since I am here, I still spend about $1000 in the crown store every year.

When the game was designed, they thought a lot of the computation can be outsourced to the client machines - and that is why they had good performance at start. Problem with this approach is, that it opened up the possibility to cheat quite drastically - (because it was breaking the basic rule of client-server architecture "never trust the client") - so they transferred vital steps back to the servers, which now choked the servers, because they were never designed to have this kind of workload. All the steps which have been done to better the situation, where just band aids and workaround basically, because they had to do it on a server-architecture, which was never designed to deal with this kind of workload.

A redesign will address this major design problem and I am quite hopeful, that this will not just benefit pvp, but as well give us better performance in other crowded areas - it is a step into the right direction, imo.

Kwoung

Lysette wrote: »

Kwoung wrote: »

Lysette wrote: »

Kwoung wrote: »

If this actually happens before something better to play comes out or New World pulls a "ESO" and turns their game around to be something great, that would be cool. But at this point, what reassurances do we have as players that this isn't just another in the very long line of yearly posts saying something will be done, but isn't? In just the recent years we had... Year of performance improvements... complete bust. Year of new servers... someone forgot to order them apparently.

Also, not to be confrontational, but a recent video made fun of PVPers because it is supposedly a well know fact that PVP was being worked on forever... and here we now hear "We’ve been planning this for the last quarter and we’re starting work on it right now"?

All I can say is, please deliver and provide updates on progress to the community along the way. We can be very understanding and if you hit a bump in the road, let us know, make a breakthrough, let us know. Communication is key and a very simple method of keeping you customers happy. Also, most MMO's now have a public roadmap posted in one form or another, please consider doing the same. It doesn't have to contain spoilers, just a general idea of what is being worked on and the progress of those items.

Nah, this step is admitting - we can't really achieve it with the old design, so if we want to get it done (and we do), we really need to redesign it. because the old technology was based on premises, which have changed so drastically, that it requires a redesign to address the issues. This is about what Matt's decision is based on and I think he is completely right with it.

A roadmap in regards to server redesign - nah, what would you do with it - give them a chance to be flexible and innovative, don't put them into a cage of a roadmap to achieve - this is creative work, redesigning the server architecture, just let them do it and be free to make changes or even discard steps they took, because they found a better solution meanwhile - a roadmap could hinder this creative freedom.

Umm ok, so basically you are saying "it will be different this time". I would love to believe that, but after 6+ years of "we are working on it", I am a tad jaded and feel a bit more than yearly words are required at this point. Also due to timing, this seems a lot more like damage control to a viral video than a sincere attempt at improving the players game experience, which could in fact, have been done at any time in years past.

If it took that video to spur actual action, then that is awesome, because quite honestly, I love playing PVP in this game and if it actually performed, I will probably play ESO for another decade, even though the PVE end of things has pretty much dried up for me. The yearly chapters or whatever they are called are great most of the time, but that's like a weeks worth of play and doesn't come near to filling my time in game... PVP keeps it fresh and since I am here, I still spend about $1000 in the crown store every year.

When the game was designed, they thought a lot of the computation can be outsourced to the client machines - and that is why they had good performance at start. Problem with this approach is, that it opened up the possibility to cheat quite drastically - (because it was breaking the basic rule of client-server architecture "never trust the client") - so they transferred vital steps back to the servers, which now choked on this because they were never designed to have this kind of workload. All the steps which have been done to better the situation, where just band aids and workaround basically, because they had to do it on a server-architecture, which was never designed to deal with this kind of workload.

A redesign will address this major design problem and I am quite hopeful, that this will not just benefit pvp, but as well give us better performance in other crowded areas - it is a step into the right direction, imo.

I am well aware of all of that, I just want to see them deliver on what was just posted, not have to wait to hear another excuse come Jan 2023 of why performance is still abysmal. And quite honestly, I have a side bet with a friend that them adding this next chapter, will destroy PVE completely as well, as the last 2 brought the game to it's knee's and those long term PVP issues, are full on in PVE content now.