Code-Rewrite/Server Re-architecture - Is there any update on this?

Gaebriel0410 · October 2022

MrLasagna wrote: »

Do people really believe that they are going to do such great effort for a game that is in a clear declining state?
People that need better performance are PvPers and trials end gamer and they showed very clearly that they don't care about them.
This game is becoming more and more a sandbox for very casual players and they don't need such great performance.

This is, obviously, just my opinion.

Huh interesting. On one hand you say that pvp and trial players need good performance, but the devs don't care about them.
And then you mention "casual players" don't need good performance, which sounds like you don't care about them in turn.

That's a bit pot and kettle, shouldn't good performance be for everyone? It's not like some groups of players have more rights to it.

TechMaybeHic · October 2022

So here is the canned response that will happen over the course of time so y'all are prepared:

"No update at this time as we continue to work on it" or some form of update that doesn't really give progress as far as timing

Repeat until game shutdown

SaffronCitrusflower · October 2022

Does ZOS even have anyone on staff that can do any significant rewriting of the engine code? The Hero engine was not developed by ZOS. A modified Hero engine is what is used for ESO.

SaffronCitrusflower · October 2022

Lumsdenml wrote: »

MrLasagna wrote: »

Do people really believe that they are going to do such great effort for a game that is in a clear declining state?
People that need better performance are PvPers and trials end gamer and they showed very clearly that they don't care about them.
This game is becoming more and more a sandbox for very casual players and they don't need such great performance.

This is, obviously, just my opinion.

I disagree that this game is in a "clear state of decline. " Looking at the population trend from launch it is clearly increasing in population. The current downtrend could be seen as simple seasonal decline, and is still too little data to accurately call a trend, much less a "clear state of decline. "

PvP and end game activities most certainly are in decline. The end game player base are giving up and no longer believe the years of unfulfilled promises. U35 was the last straw for many, many end game players. Sure, they're still playing some, but they aren't leading raids or subbing anymore. They're just passing the time until something new comes along now.

SaffronCitrusflower · October 2022

React wrote: »

kargen27 wrote: »

"You seem worried about spreading the population, but the reality of the population is that a good deal of those players quit because of the lack of content."

No, the vast majority left because of performance issues. You can almost pinpoint the dates of mass exodus. First was when ZoS had to make more calculations server side to deal with cheats. That really hurt performance and players left. They from time to time check back to see if performance has improved. There have been other significant (though not as significant) events that caused players to leave.
With PvP the players provide the dynamics of the area. It is good to change things up from time to time to shake up player actions some but new areas isn't what PvP needs.
I do think they could do some fun things with Imperial City and have voiced ideas on that in the past. I think the flags in Imperial City was a mistake. Events where the flag doesn't matter for spawning in one district from time to time I think would get a good number of people participating.

And I stick by my opinion that a smaller Cyrodiil would be a bad idea. It needs to be large to allow for a variety of strategies in controlling the map. If/when ZoS figures out the performance issues all kinds of people will come back to PvP.

Yes, the majority left because of performance issues. Again, performance is the top priority.

However, on PC NA, they "fixed" the performance issues. The population rose for several months, but in the past two months has dwindled back to pre server upgrade numbers. Therefore, I disagree with your sentiment that "fixing performance will bring these players back".

I know from talking to many of these people myself that their issue playing the game now, post performance fix, is that they are bored of doing the exact same thing in the exact same environments with no new rewards or incentives.

A lot of PvP players left due to U35 as well.

Dangranma_Burgrukgad · October 2022

Gaebriel0410 wrote: »

Huh interesting. On one hand you say that pvp and trial players need good performance, but the devs don't care about them.
And then you mention "casual players" don't need good performance, which sounds like you don't care about them in turn.

That's a bit pot and kettle, shouldn't good performance be for everyone? It's not like some groups of players have more rights to it.

What I am saying is that in my experience I have never seen people in PvE zone chat screaming at the game because no skill works and rubberbanding teleport you from a place to a place 10m afar.
Normal PvE zones works almost fine, nothing compared to that shameful lag fest that is Cyrolag.
I have nothing against people that enjoys the game in a different way to mine and hope the game will become better for just everyone.

ZOS_Kevin · October 2022

@Destai some follow ups to your questions:

The community team's purpose is not to be a news round-up site. We would rather share information through our channels coming from the team. We know the information being shared with news outlets and we often do share that info when it is available. So for example, the text you shared is nothing new compared to what Gina shared back in June. Matt updated the community about back in August regarding the data center updates and IT infrastructure. (That point is also highlighted in the video you linked to.) Hence the usage of "no update". And we do not have additional feedback regarding the engine work right now. But we are passing that along so the team knows the question is being asked.

We do speak to stakeholders about questions and feedback posted here on the forum and across our other communication channels. But if there is nothing to update players on at the time, I would rather say "no update but we're still keeping track" rather than just ignoring. When there are status updates, we are more than happy to update on those. Sometimes these things take time. And while we understand that players want things done as soon as possible, we try our best to only update when we are sure. The intent is not meant to sound canned. But if we don't have additional information to share, there isn't much we can do outside of letting you know that.

Lastly, mods on the forum are here specifically to moderate. If there are things they can help with when relaying information, they certainly will, but their primary job is to make sure the forums are safe and address player issues with each other and against the forum code of conduct.

Hope this helps to clear up any of the questions posed. The community team is constantly checking in with stakeholders to see what is ready to share with all of you.

Amottica · October 2022

Destai wrote: »

ZOS_Kevin wrote: »

No update as of right now, @React. But when we have an update, we will share across our social channels and on the forum in a specific post highlighting any updates.

Maybe you missed it @ZOS_Kevin , but there was an update in an interview. The link is here. I discovered this through Nefas.

Text from Matt Firor:
"The studio are hard at work on finishing content for this year's major update, High Isle, but still have plenty in the pipeline for the future, including 2023 ESO content.
"We're also doing more mundane things like upgrades, updating all of the hardware and all of our data centres because it's, you know, we have to do that and people don't realise that making games like this, it's not just about the game side of it, it's also about the IT infrastructure and making sure that everything is up and running," he said.
While there's been some setbacks in upgrading data centres due to COVID-19, Firor says it's a project that they expect to be completed over the next six months."

This creates some questions Kevin. Hopefully you can explain the disconnect.
1. Why wasn't this linked here in some fashion? Players ask for updates, even if it's interim updates like this interview, and we're given what sounds like a canned response. It's hard not to interpret your "updates" as of late as canned responses handed down to you and the mods. "No update" sounds dismissive - even high-level updates as seen in the interview would be more appreciated.
2. When you provide updates - are you actually reaching out to the owners of those issues or are you echoing canned responses provided by the CMs and/or Devs? Given how you responded to this and the Q&A requests, it sounds much like that latter. Hopefully you can clear that up; perhaps it's a mix of the two. If you are getting the status from actual stakeholders, hopefully you can provide more details beyond what we've seen lately - much like the interview.
3. Are mods capable of providing status updates on deliverables? Perhaps this could help with any time constraints you might have. Based on my link, sounds like they can and do.

@Destai That is great information about the hardware replacement project which has been completed on PC/NA but obtaining the hardware to complete the project has been slow due to COVID.

This thread is about the re-architecture of the server-side code, which is a different project.

React · October 2022

ZOS_Kevin wrote: »

@Destai some follow ups to your questions:

The community team's purpose is not to be a news round-up site. We would rather share information through our channels coming from the team. We know the information being shared with news outlets and we often do share that info when it is available. So for example, the text you shared is nothing new compared to what Gina shared back in June. Matt updated the community about back in August regarding the data center updates and IT infrastructure. (That point is also highlighted in the video you linked to.) Hence the usage of "no update". And we do not have additional feedback regarding the engine work right now. But we are passing that along so the team knows the question is being asked.

We do speak to stakeholders about questions and feedback posted here on the forum and across our other communication channels. But if there is nothing to update players on at the time, I would rather say "no update but we're still keeping track" rather than just ignoring. When there are status updates, we are more than happy to update on those. Sometimes these things take time. And while we understand that players want things done as soon as possible, we try our best to only update when we are sure. The intent is not meant to sound canned. But if we don't have additional information to share, there isn't much we can do outside of letting you know that.

Lastly, mods on the forum are here specifically to moderate. If there are things they can help with when relaying information, they certainly will, but their primary job is to make sure the forums are safe and address player issues with each other and against the forum code of conduct.

Hope this helps to clear up any of the questions posed. The community team is constantly checking in with stakeholders to see what is ready to share with all of you.

Thank you for the thorough reply to @Destai 's question, and for your continued communication. The number one concern for many people, including myself, is radio silence on things like this. It is so much easier for us to manage expectations when we are given updates as frequently as possible.

FluffWit · October 2022

They talk about updates but console players haven't had so much as an estimate. At the rate they're going I cant see xbox getting done until late 2023 if we're lucky and as for Playstation? Lol gl.

ZOS_Kevin · October 2022

Right and we totally get that, @React. The intent is not to have radio silence. This is a complicated process that has a lot of moving parts and one that cannot be taken lightly. So we do our best to update when something important has changed or there is something new to report.

Amottica · October 2022

FluffWit wrote: »

They talk about updates but console players haven't had so much as an estimate. At the rate they're going I cant see xbox getting done until late 2023 if we're lucky and as for Playstation? Lol gl.

@FluffWit

If you are speaking about the hardware update, Zenimax is not able to provide an update as the ability to source the hardware is out of their control due to COVID shortages. They do not know how long it will take to procure the hardware to do the next server let alone all of them.

As it is, you may be correct that Xbox may not be done until 2023 or even later. It may be done sooner.

Below is a link to a recent article explaining the shortage and that it is still an issue today.

https://www.jabil.com/blog/global-chip-shortages.html

FluffWit · October 2022

Amottica wrote: »

FluffWit wrote: »

They talk about updates but console players haven't had so much as an estimate. At the rate they're going I cant see xbox getting done until late 2023 if we're lucky and as for Playstation? Lol gl.

@FluffWit

If you are speaking about the hardware update, Zenimax is not able to provide an update as the ability to source the hardware is out of their control due to COVID shortages. They do not know how long it will take to procure the hardware to do the next server let alone all of them.

As it is, you may be correct that Xbox may not be done until 2023 or even later. It may be done sooner.

Below is a link to a recent article explaining the shortage and that it is still an issue today.

https://www.jabil.com/blog/global-chip-shortages.html

They did provide an update. Kevin linked to it. That's what I was replying to. Sorry for the confusion.

Destai · October 2022

ZOS_Kevin wrote: »

@Destai some follow ups to your questions:

The community team's purpose is not to be a news round-up site. We would rather share information through our channels coming from the team. We know the information being shared with news outlets and we often do share that info when it is available. So for example, the text you shared is nothing new compared to what Gina shared back in June. Matt updated the community about back in August regarding the data center updates and IT infrastructure. (That point is also highlighted in the video you linked to.) Hence the usage of "no update". And we do not have additional feedback regarding the engine work right now. But we are passing that along so the team knows the question is being asked.

We do speak to stakeholders about questions and feedback posted here on the forum and across our other communication channels. But if there is nothing to update players on at the time, I would rather say "no update but we're still keeping track" rather than just ignoring. When there are status updates, we are more than happy to update on those. Sometimes these things take time. And while we understand that players want things done as soon as possible, we try our best to only update when we are sure. The intent is not meant to sound canned. But if we don't have additional information to share, there isn't much we can do outside of letting you know that.

Lastly, mods on the forum are here specifically to moderate. If there are things they can help with when relaying information, they certainly will, but their primary job is to make sure the forums are safe and address player issues with each other and against the forum code of conduct.

Hope this helps to clear up any of the questions posed. The community team is constantly checking in with stakeholders to see what is ready to share with all of you.

Thanks Kevin, I appreciate the detailed response.

Some additional considerations:
1. Short snips from CMs and Mods can be interpreted as either aloof or dismissive and defensive. Hoping the tone can be tweaked so we know something is happening - planning, ordering, whatever. Details help your guys' image.
2. We need to know what the major endeavors are, just at a high-level. As evidenced by some comments here, there's some confusion about what each of the performance upgrades are and their statuses. Having some high level overview would help us better understand those things and better target our questions. The expectation isn't to know every bug in progress, I understand that's a separate issue as we've discussed earlier.
3. Regular updates that work for you guys - quarterly for big ones, monthly for small ones like the Q&A. Of course there's oodles of complexities in conveying information, let alone the actual logistics of things. Detailed regular updates help us get a sense of those complexities - as evidenced by your reply.
4. I appreciate your patience and candor in dealing with these questions and concerns.

Kingsindarkness · October 2022

TechMaybeHic wrote: »

So here is the canned response that will happen over the course of time so y'all are prepared:

"No update at this time as we continue to work on it" or some form of an update that doesn't really give progress as far as timing

Repeat until game shutdown

I simply don't believe that Matt comes from a PVP MMO, I think he is dealing with the spaghetti code and the fact that open-world PVP wasn't the smash hit he thought it was going to be. So more than likely there is going to be a code restructure and a reimagining of PVP, because let's face it the war can not last forever, and no matter how smooth PVP is the current system just won't fill the seats.

The most logical thing that ZoS can do I think is bolster the arena system and or add smaller Castlekeep sieges or PVPVE areas as there is no reason to just take the PVP game out back and put a knife in its back.

But the current system won't stand, and remember what we get or don't get is all up to the moneymen, a case has to be made that PVP is going to rake in the cash...and I would imagine that is really going to be a tough sell.

spartaxoxo · October 2022

ZOS_Kevin wrote: »

Right and we totally get that, React. The intent is not to have radio silence. This is a complicated process that has a lot of moving parts and one that cannot be taken lightly. So we do our best to update when something important has changed or there is something new to report.

I, for one, am grateful for that, @ZOS_Kevin And really glad that you prefer a "no news" update to radio silence, which I know from experience is sometimes the only alternative. Thanks for the detailed feedback on the process you used when relaying feedback. I have a question about the server re-architecure, could you guys maybe explain a bit the difference between this and the server replacement? I think know one is a code thing and the other is a hardware thing? But beyond that? ELI5

Tandor · October 2022

ZOS_Kevin wrote: »

Right and we totally get that, @React. The intent is not to have radio silence. This is a complicated process that has a lot of moving parts and one that cannot be taken lightly. So we do our best to update when something important has changed or there is something new to report.

That's appreciated @ZOS_Kevin , as would be more regular updates along the lines of "Still in progress, and on course" or even "Still in progress, albeit slower than we would have liked". You don't have to have an important change or something new to report, just confirming that it's still being pursued is a worthwhile update in itself.

Lebkuchen · October 2022

Can someone please help me understand?

In January 2022 they told us "we are going to rearchitect our server" and "rewrite some of the foundational server code" in order to "give everyone a good play experience in high-intensity situations like PvP and Trials". (1)

A few months later they told us we would get new "server hardware". But "This will not result in any appreciable performance gain in-game, but it will result in a more reliable service overall that needs fewer unplanned maintenances." (2)

PC NA already got it. PC EU will maybe get it in March 2023. And consoles maybe sometime after that. (3)

Now what i do not understand: PC NA players reported better performance after the new "server hardware" 6 months ago. Does this mean they already got the "rearchitecture of the server and rewrite of the code" from link (1)? Or did they just switch some 10 year old hardware, and the game worked better because of it?

I am asking because, for Playstation EU players, it would be interesting to know if the "rearchitecture and rewrite" that will "take up much of 2022" (1) can give us better performance before the "hardware update" sometime after March 2023? Or is this the same project and one thing can not work without the other?

(1) https://forums.elderscrollsonline.com/en/discussion/595442/eso-pvp-update-updated-june-2022/p1
(2) https://forums.elderscrollsonline.com/en/discussion/604073/upcoming-pc-na-datacenter-hardware-replacement/p1
(3) https://forums.elderscrollsonline.com/en/discussion/616021/pc-eu-hardware-update-august-2022/p1

TechMaybeHic · October 2022

Lebkuchen wrote: »

Can someone please help me understand?

In January 2022 they told us "we are going to rearchitect our server" and "rewrite some of the foundational server code" in order to "give everyone a good play experience in high-intensity situations like PvP and Trials". (1)

A few months later they told us we would get new "server hardware". But "This will not result in any appreciable performance gain in-game, but it will result in a more reliable service overall that needs fewer unplanned maintenances." (2)

PC NA already got it. PC EU will maybe get it in March 2023. And consoles maybe sometime after that. (3)

Now what i do not understand: PC NA players reported better performance after the new "server hardware" 6 months ago. Does this mean they already got the "rearchitecture of the server and rewrite of the code" from link (1)? Or did they just switch some 10 year old hardware, and the game worked better because of it?

I am asking because, for Playstation EU players, it would be interesting to know if the "rearchitecture and rewrite" that will "take up much of 2022" (1) can give us better performance before the "hardware update" sometime after March 2023? Or is this the same project and one thing can not work without the other?

(1) https://forums.elderscrollsonline.com/en/discussion/595442/eso-pvp-update-updated-june-2022/p1
(2) https://forums.elderscrollsonline.com/en/discussion/604073/upcoming-pc-na-datacenter-hardware-replacement/p1
(3) https://forums.elderscrollsonline.com/en/discussion/616021/pc-eu-hardware-update-august-2022/p1

Just hardware PCNA so far.

Lebkuchen · October 2022

@spartaxoxo looks like we had the same question, but i walked away for an hour to eat while i was writing my post

ZOS_Kevin · October 2022

@spartaxoxo You're right on the money. The server refresh is ordering and replacing physical hardware in server centers. The rewriting of foundational server code is the technical code that the game is built on. I wish I could give a better breakdown than that, but that is well beyond my paygrade. But, when all of this is ready for players, we will ask for a more detailed breakdown for those who are interested in all of the technicals behind a process like this.

@Lebkuchen we have only replaced PC NA hardware. That also saw a performance increase for those on PC NA. We are working on replicating that for PC EU in 2023 and then console after that. But as noted before, supply chain issues have severely impacted the timeline here.

jcasini222ub17_ESO · October 2022

PC NA here. There was a marked improvement after the hardware was replaced. That has worn off, still better but not the smooth butter right after upgrade.

From my understanding the re-architecture is not done for PC NA. That is the real hope for a smooth Cyro.

What we experienced was more akin to MYM performance of that past. As in Cyro was all full throttle when it comes to server access and I’m fairly confident for a few days most of the anti-cheat checks were lifted. It felt like old school client side checks and played that way. It was glorious and on NA we locked each campaign at prime time through what boiled down to a whisper campaign of “cyros better” throughout the various gaming communities.

Of course as time as passed those cheat checks rightfully were put back and the Cyro server portion returned to what it would be outside of MYM.

As Kevin stated this is complex and there is a ton of moving parts. The hardware was a necessary step 1 and as Firor stated true performance benefits will be seen once the re-architecture is complete.

virtus753 · October 2022

I wonder how much fixing Path is going to help with performance. That one thing alone has been cutting my fps by over 30 in dungeons and trials - who knows what it’s doing in PvP. I hope other skills will be checked to ensure they are not similarly counteracting the effect of the server replacement (I’m on PC-NA) and the code rework.

Triplesixtyson · October 2022

Anytime ive heard "re-write" in a mmorpg, its always been a massive undertaking. Im no computer wiz but you can guarntee if they are in fact doing a re-write, it will not be done this year. It always takes longer in my experience.

Wolfkeks · October 2022

Ayy, 'no new update' news is still better than radio silence. Thank you Kevin!

ErMurazor · October 2022

Well, if they actually comes through with this. Rewriting the code and new server hardware i know our pvp guild of 20 ppl would come back for sure and many more that are on standby and playing other pvp games atm. ESO gameplay is one of its kind. If it runs smooth I'm sure it will be a massive influx of old and new players when the word starts spreading. I lost hope though after so many trials and errors so I'm not expecting anything to change sadly.

LittlePinkDot · November 2022

Woo hoo! Put ESO on Unreal Engine 5! 😁

ashadris · November 2022

LittlePinkDot wrote: »

Woo hoo! Put ESO on Unreal Engine 5! 😁

Unreal or not an engine upgrade would be veeeery sweet! I want ESO to flourish because we all love this game and a potential engine upgrade would definitely reinvigorate the player base and general enthusiasm as soon as the word goes out imo.

I think essentially this game needs a one tamriel v2 moment.

blktauna · November 2022

please remember ToT is touted as pvp content, so I wouldn't assume meaning.