Code-Rewrite/Server Re-architecture - Is there any update on this?

React

In January, we received the update below from Matt Firor regarding the state of combat performance in ESO. We were told that there would be no new PVP related content in the mean time, as the resources required for developing that content would be focused solely on this "Code Rewrite" that would apparently fix the server and combat performance.

It is now September, 8 months later. Since this code rewrite was supposed to "take up much of 2022", and we are now nearing the end of 2022, could we please have an update on the status of this? Many of us are eager to know what to expect in the coming months, and also if we can expect PVP related DLC content to come in Q1/Q2 2023. After all, by that time it will have been four years since our last PVP content was added in the form of a single BG map.

There are so many great ideas floating around the community for content that could be added, too. Group-Dueling. A reworked imperial city where there are no factions, just a group limit and free for all PVP. New reward structures to make things like earning AP/TelVar more meaningful. A resized cyrodiil to match the lower population caps. A new PVP zone altogether. Custom battlegrounds matchmaking.

I really hope we can get some sort of update on this soon.

@ZOS_GinaBruno @ZOS_MattFiror @ZOS_Kevin @ZOS_BrianWheeler

ZOS_MattFiror wrote: »

We’ve been planning this for the last quarter and we’re starting work on it right now. But, as you can imagine, changing the fundamentals of a huge live game such as ESO is a delicate and multi-stepped process, so expect this to take up much of 2022. Much of the early work is going to happen behind the scenes and we are definitely not going to rush it. A re-architecture of this magnitude will require the entire game to be re-tested and evaluated, as this particular code is the foundation on which the game is built. It will take tons of QA and testing time as well, and I’m sure that when the time comes, we’ll do as large a test as we can on the PTS.

In the meantime, we’ll consider additional ways to keep Cyrodiil and Battlegrounds interesting and exciting (for example, potentially special rulesets or weekend events) and will continue to work on bug fixes affecting combat, but we won’t be adding any new features to PvP until the work mentioned above is complete. Doing so would not only prolong the fixes we want to implement but would also exacerbate the issue of poor performance.

Knockmaker

Totally in agreement with everything except for the resized Cyrodiil. I doubt that is a good idea, nor have I heard from anyone proposing such a thing. A re-write of the code should render such an idea unnecessary in which we would drift further away from large-scale, large map, large-scale fights concept which was their sales pitch in the very begining.
With that being said, it makes me and probably many others echo the same question: Whap happened to the re-writing of the code? It should be on testing stage by now. When are you testing it on PTS?

Dangranma_Burgrukgad

Do people really believe that they are going to do such great effort for a game that is in a clear declining state?
People that need better performance are PvPers and trials end gamer and they showed very clearly that they don't care about them.
This game is becoming more and more a sandbox for very casual players and they don't need such great performance.

This is, obviously, just my opinion.

GreatGildersleeve

MrLasagna wrote: »

Do people really believe that they are going to do such great effort for a game that is in a clear declining state?
People that need better performance are PvPers and trials end gamer and they showed very clearly that they don't care about them.
This game is becoming more and more a sandbox for very casual players and they don't need such great performance.

This is, obviously, just my opinion.

Yes, but it’s a sound opinion. We’re 8 years into the life span of this game. How many more years does ZoS expect to squeeze out of it before putting it into maintenance mode? (Some would argue we’re there now given new zones are getting smaller and smaller with no new game systems). A task as Herculean as a complete code re-write? I had my doubts when I first heard it and I’ll be without socks if it does actually happen.

Besides, look at Matt’s blurb. Special weekend events or rule sets. How many have we had… one? I seriously doubt a code re-write is on the horizon.

Marto

8 months is sadly too little of a timeframe to expect an update on a change of that scope.

Even if ZOS were to genuinely and honestly answer you, with all the desire to be transparent, the answer would most likely boil down to "We're working on it".

Dangranma_Burgrukgad

Marto wrote: »

8 months is sadly too little of a timeframe to expect an update on a change of that scope.

Even if ZOS were to genuinely and honestly answer you, with all the desire to be transparent, the answer would most likely boil down to "We're working on it".

Are they "working on it" like they were "working on it" for the U35 Q&A ?
I firmly believe that the outcome will be the same.

Hotdog_23

Would not expect any new PVP content until the server upgrade is complete on all the servers. Still have PC/EU and consoles to go.

From the August Hardwire update.
“Based on what we know today, we anticipate everything being updated, tested, and running the live version of the game around March 2023. We expect the hardware upgrade for the console servers to be sometime soon after that.”

https://forums.elderscrollsonline.com/en/discussion/616021/pc-eu-hardware-update-august-2022/p1

Stay safe

Onomog

My friend and I were talking about this and all the seemingly unfocused changes happening to the game. Maybe it was the whisky talking, but we landed upon the idea that maybe all these changes are prep work for moving the game to a newer engine.

Seems outlandish, I know (remember - whisky), but not impossible. Will be interesting to see what, if anything, happens.

FluffyReachWitch

There are still three-and-a-half months of 2022 left, and things related to game development can stretch out much longer than expected. It wouldn’t surprise me if we didn’t hear back until December, or perhaps Q1 2023.

I wonder if the speculated special PvP weekends got converted into the XP buff we saw in an update earlier this year?

SaffronCitrusflower

MrLasagna wrote: »

Do people really believe that they are going to do such great effort for a game that is in a clear declining state?
People that need better performance are PvPers and trials end gamer and they showed very clearly that they don't care about them.
This game is becoming more and more a sandbox for very casual players and they don't need such great performance.

This is, obviously, just my opinion.

Not just your opinion. All evidence points to this is the primary focus of the game last couple plus years.

React

Hotdog_23 wrote: »

Would not expect any new PVP content until the server upgrade is complete on all the servers. Still have PC/EU and consoles to go.

From the August Hardwire update.
“Based on what we know today, we anticipate everything being updated, tested, and running the live version of the game around March 2023. We expect the hardware upgrade for the console servers to be sometime soon after that.”

https://forums.elderscrollsonline.com/en/discussion/616021/pc-eu-hardware-update-august-2022/p1

Stay safe

This is probably true, but it is in contrast with the January update. As far as I am aware, the code rewrite and server replacements are two separate things.

I just would like some sort of update on this. 8 months is a long time to go without any word on this massive project, especially with the massive rift in PVP content we've had.

Atrael7

If there was an official update regarding server rewrite and future PvP content right now, I think it'd look kind of like this:

https://www.youtube.com/watch?v=cOCD_Jzre3c

React

Atrael7 wrote: »

If there was an official update regarding server rewrite and future PvP content right now, I think it'd look kind of like this:

https://www.youtube.com/watch?v=cOCD_Jzre3c

Future PVP content, maybe. But 8 months into a project that was estimated to take up the better part of 12 months, I feel like there must be some sort of development that could be shared with the community.

React

@ZOS_GinaBruno @ZOS_Kevin @ZOS_MattFiror

Any idea if the developers are planning to update us soon about the code rewrite? We're 10 months in now to a project that was projected to take "much of 2022". We are eager to hear an update on this.

rpa

If they are actually still doing the rewrite and have not found the task of untangling the spaghetti impossible, I hope they take their time and only release it when it is ready.

React

rpa wrote: »

If they are actually still doing the rewrite and have not found the task of untangling the spaghetti impossible, I hope they take their time and only release it when it is ready.

I also hope they do not release it until it is ready. However, communication on this is 100% necessary. This project was claimed as the reason we would not receive any PVP content in 2022, despite there having been four consecutive years now since PVP content was released.

ZOS_Kevin

No update as of right now, @React. But when we have an update, we will share across our social channels and on the forum in a specific post highlighting any updates.

React

ZOS_Kevin wrote: »

No update as of right now, @React. But when we have an update, we will share across our social channels and on the forum in a specific post highlighting any updates.

Thank you for the reply, Kevin. Looking forward to this, and hopeful that it comes sooner rather than later.

Dangranma_Burgrukgad

Well, sad to say but that was expected.
All we can get is an update to tell us there is no update.

spartaxoxo

MrLasagna wrote: »

Well, sad to say but that was expected.
All we can get is an update to tell us there is no update.

Better than silence. I hope they have something new to share by the end of the year.

Lumsdenml

MrLasagna wrote: »

Do people really believe that they are going to do such great effort for a game that is in a clear declining state?
People that need better performance are PvPers and trials end gamer and they showed very clearly that they don't care about them.
This game is becoming more and more a sandbox for very casual players and they don't need such great performance.

This is, obviously, just my opinion.

I disagree that this game is in a "clear state of decline. " Looking at the population trend from launch it is clearly increasing in population. The current downtrend could be seen as simple seasonal decline, and is still too little data to accurately call a trend, much less a "clear state of decline. "

Amottica

Lumsdenml wrote: »

MrLasagna wrote: »

Do people really believe that they are going to do such great effort for a game that is in a clear declining state?
People that need better performance are PvPers and trials end gamer and they showed very clearly that they don't care about them.
This game is becoming more and more a sandbox for very casual players and they don't need such great performance.

This is, obviously, just my opinion.

I disagree that this game is in a "clear state of decline. " Looking at the population trend from launch it is clearly increasing in population. The current downtrend could be seen as simple seasonal decline, and is still too little data to accurately call a trend, much less a "clear state of decline. "

This.

It is so easy to make claims the game is dying or in decline, What is interesting is there have been people saying this game was dying and dead for years and it is clear their claims were baseless.

dsalter

Knockmaker wrote: »

Totally in agreement with everything except for the resized Cyrodiil. I doubt that is a good idea, nor have I heard from anyone proposing such a thing. A re-write of the code should render such an idea unnecessary in which we would drift further away from large-scale, large map, large-scale fights concept which was their sales pitch in the very begining.
With that being said, it makes me and probably many others echo the same question: Whap happened to the re-writing of the code? It should be on testing stage by now. When are you testing it on PTS?

i actually mentioned shrinking Cyrodiil like 2 years ago, the clutter size of the map contributes part to the lag, especially the pointless delves. dolmens and quest hubs.
i get it they add some RP element to the map but it comes at the cost of more entities and calculations to the map.
if you were to cut out a huge amount of the clutter, make it so keeps 2 links down are unsiegable (to reduce this stupid cut off ball group crap) and then rework major and minor buffs into just singular buffs it would go a massive way to helping elivate the strain on the servers hamster.

besides reworking the buffs to just be singular types could help level the power creep, would reduce the max achievable stat while raising the minimum so that it becomes more about rotation and skill over stack buffs=win, both PvP and PvE would benefit from it

kargen27

"and also if we can expect PVP related DLC content to come in Q1/Q2 2023. After all, by that time it will have been four years since our last PVP content was added in the form of a single BG map."

I hope not unless it is a change in dynamics with existing content. We don't need to spread the already thin PvP population any thinner.

A smaller Cyrodiil is a bad idea. A smaller map would take away much of the strategy in controlling the map. All PvP needs is performance improvements. Improvements will bring the players back.

React

kargen27 wrote: »

"and also if we can expect PVP related DLC content to come in Q1/Q2 2023. After all, by that time it will have been four years since our last PVP content was added in the form of a single BG map."

I hope not unless it is a change in dynamics with existing content. We don't need to spread the already thin PvP population any thinner.

A smaller Cyrodiil is a bad idea. A smaller map would take away much of the strategy in controlling the map. All PvP needs is performance improvements. Improvements will bring the players back.

I mean, every single thing I suggested is a rework to existing content apart from the new pvp zone.

You seem worried about spreading the population, but the reality of the population is that a good deal of those players quit because of the lack of content. This has been especially noticeable on PC NA where we have had good performance for about 6 months now since the servers were replaced. The first couple months there was insanely high population, but now it has dwindled back to pre-server upgrade levels.

Good performance is amazing and should be the number one priority, but new content should be a close second. Even if that content is just reworking what we already have.

Kessra

It's always easy to scream for throwing newer and faster hardware on a problem though this might never solve the core issue. ZOS has analyzed and measured the throughput under various scenarios in the last year but wasn't really explaining the results of the investigations to us players. We therefore can only speculate what might have lead to performance degradation over the years. Sure, splitting the DB into cold and hot accounts has helped to some degree but it was just one part of the puzzle.

Years ago there was already a login-issue with chars that have quested a lot as the DB was just overburdened with loading data for each player back then. ZOS managed to "fix" this issue some weeks later, mostly by throwing bigger hardware at the problem. Whether they also changed something in the way data is loaded is beyond my knowledge. I at least can clearly see a difference in terms of login-times of characters/accounts that haven't done many quests or explored much of the zones and ones that did all the content. There is a difference of almost 2-3 minutes noticeable in that regards. That at least leaves me with the conclusion that the data-model each char carries around, where the progress of each zone and so forth is stored, is just very large and takes some time to load in comparison to ones that hardly have done anything.

A rewrite here might eventually split up the data or allow to stream only data in that is required for the current zone or the like. I have no insights in what databases they are using or about their internals. Maybe they switch part of their backend to a more graph-based DB like Neo4J or the like. In Star Citizen i.e. the devs changed their backend to such a graph-based DB to help to stream only data in that is needed for the current portion of the area they are in and clear out any unneeded data. They called this technique persistent entity streaming and it took them over 4 years of implementation and they still are working on it.

While I do find the rewrite necessary on the long run, it though can't be an excuse for not listening to their PvP community for years, and I am by no means a PvPer at all. I literally have to force myself to go to a PvP-Zone when one of the events is live and was only in a battleground once when someone of the group queued for battleground instead of the daily dungeon he actually looked for. An arena-like PvP skirmish shouldn't be to hard to implement and should also not be very demanding on network/database bandwidth. A 2v2, 3v3, 4v4, 12v12 with ranking option and eventual matchmaking and/or monthly tournaments can't be that hard to implement.

kargen27

"You seem worried about spreading the population, but the reality of the population is that a good deal of those players quit because of the lack of content."

No, the vast majority left because of performance issues. You can almost pinpoint the dates of mass exodus. First was when ZoS had to make more calculations server side to deal with cheats. That really hurt performance and players left. They from time to time check back to see if performance has improved. There have been other significant (though not as significant) events that caused players to leave.
With PvP the players provide the dynamics of the area. It is good to change things up from time to time to shake up player actions some but new areas isn't what PvP needs.
I do think they could do some fun things with Imperial City and have voiced ideas on that in the past. I think the flags in Imperial City was a mistake. Events where the flag doesn't matter for spawning in one district from time to time I think would get a good number of people participating.

And I stick by my opinion that a smaller Cyrodiil would be a bad idea. It needs to be large to allow for a variety of strategies in controlling the map. If/when ZoS figures out the performance issues all kinds of people will come back to PvP.

React

kargen27 wrote: »

"You seem worried about spreading the population, but the reality of the population is that a good deal of those players quit because of the lack of content."

No, the vast majority left because of performance issues. You can almost pinpoint the dates of mass exodus. First was when ZoS had to make more calculations server side to deal with cheats. That really hurt performance and players left. They from time to time check back to see if performance has improved. There have been other significant (though not as significant) events that caused players to leave.
With PvP the players provide the dynamics of the area. It is good to change things up from time to time to shake up player actions some but new areas isn't what PvP needs.
I do think they could do some fun things with Imperial City and have voiced ideas on that in the past. I think the flags in Imperial City was a mistake. Events where the flag doesn't matter for spawning in one district from time to time I think would get a good number of people participating.

And I stick by my opinion that a smaller Cyrodiil would be a bad idea. It needs to be large to allow for a variety of strategies in controlling the map. If/when ZoS figures out the performance issues all kinds of people will come back to PvP.

Yes, the majority left because of performance issues. Again, performance is the top priority.

However, on PC NA, they "fixed" the performance issues. The population rose for several months, but in the past two months has dwindled back to pre server upgrade numbers. Therefore, I disagree with your sentiment that "fixing performance will bring these players back".

I know from talking to many of these people myself that their issue playing the game now, post performance fix, is that they are bored of doing the exact same thing in the exact same environments with no new rewards or incentives.

Jaraal

There’s only so much they can do as long as they continue to support the first gen consoles. The hardware limitations dictate what can and can’t be added to the game. They have stated that console capabilities is what currently prevents them from raising the housing item limits. So I wouldn’t expect any major system upgrades until all platforms are on the same performance page.

Just my personal opinion.

SPR_of_HA_community

It is not a big problem now. Server optimization will not help a lot. If servers are 7 years old+ the hardware just do not work good enough for the game to work on it.

All they really could do - was check and fix critical code mistakes. Game stop hard lugs.

In PVP zone client side optimization was made OK. The problem is or on server side with sending to mutch information in the same time or with calculations due to it on server side. So with such thing - only change of Hardwear can help.

The more big problem is in balance of sets - because some sets are like - it gives you 10hp and nothing else why others look like 1000+ HP and some thing else.

It is bad if they do not see such problems. I do not say about some really good powerfull set, but - just as example - set plague breaker. It is good (perfect) in some content. But really bad in other. At least change 20 seconds calldown to some thing like 5/10 seconds can make set good on solo targets too so make set more usefull.

The same time if you have sets like - give you 300 SPD or give you 300 spd on some action like dodge - it just look strange. Bad power up bonus for hard action. Not enough for be good in PVP and PVE.

There are to much old underperforming sets now. It would be good if they get updated to better values or better bonuses.

Destai

ZOS_Kevin wrote: »

No update as of right now, @React. But when we have an update, we will share across our social channels and on the forum in a specific post highlighting any updates.

Maybe you missed it @ZOS_Kevin , but there was an update in an interview. The link is here. I discovered this through Nefas.

Text from Matt Firor:
"The studio are hard at work on finishing content for this year's major update, High Isle, but still have plenty in the pipeline for the future, including 2023 ESO content.
"We're also doing more mundane things like upgrades, updating all of the hardware and all of our data centres because it's, you know, we have to do that and people don't realise that making games like this, it's not just about the game side of it, it's also about the IT infrastructure and making sure that everything is up and running," he said.
While there's been some setbacks in upgrading data centres due to COVID-19, Firor says it's a project that they expect to be completed over the next six months."

This creates some questions Kevin. Hopefully you can explain the disconnect.
1. Why wasn't this linked here in some fashion? Players ask for updates, even if it's interim updates like this interview, and we're given what sounds like a canned response. It's hard not to interpret your "updates" as of late as canned responses handed down to you and the mods. "No update" sounds dismissive - even high-level updates as seen in the interview would be more appreciated.
2. When you provide updates - are you actually reaching out to the owners of those issues or are you echoing canned responses provided by the CMs and/or Devs? Given how you responded to this and the Q&A requests, it sounds much like that latter. Hopefully you can clear that up; perhaps it's a mix of the two. If you are getting the status from actual stakeholders, hopefully you can provide more details beyond what we've seen lately - much like the interview.
3. Are mods capable of providing status updates on deliverables? Perhaps this could help with any time constraints you might have. Based on my link, sounds like they can and do.