Hardware Refresh Work Update – March 31, 2023

Varana

ZOS_Kevin wrote: »

Tommy_The_Gun wrote: »

It would be ideal if ZOS would communicate internally better so that the team responsible for the in-game events would know in advance when there are some potenial server down-times and plan events in different days.

Wanted to chime in here and provide a little bit of context. The in-game events team is aware and part of the process of the hardware refresh. In fact, many teams need to be on the same page for a maintenance like this to happen. One thing to remember is, there are many teams working together to pull this off. We understand sometimes that isn't the best timing for events and how that is frustrating. As a reminder, that is why we built in the extra day for Jester's, to try and balance that out. However, we also have to take into account the teams involved and where those teams are. The teams have to be available to do the work. And pushing out one of these dates would inevitably impact when other servers get their refresh. The team considers multiple angles when choosing a time for these and work to create the least friction possible, when possible. Hopefully this provides some context for timings, as events are not the only factor the team has to consider when planning out the maintenance here.

While we know it is frustrating, we to appreciate everyone's patience and understanding for these maintenance windows.

Feedback like this is really appreciated. It would've been helpful to have posts like this appear on the Dev Tracker as well, though.

thejadefalcon

Sidewaves89 wrote: »

[snip] tickets, you getting new servers. [snip] relax and enjoy.

[edited for baiting]

Everyone is getting or has gotten new servers anyway. Is it too much to ask that it doesn't clash with events? It's not one or the other.

Zodiarkslayer wrote: »

This is nonsensical.

@thejadefalcon You are from the UK, right? In the UK the servers go down on 1 a.m. April 4th and are up at 3 p.m. the same day. This should be plenty of opportunity to get all possible tickets.

The times seem to be deliberately chosen to avoid EU primetime.

Ah, yes, because ZOS has never gone over their estimate on maintenance or had major problems after even smaller ones. And they've never had it consistently with PC EU. But even if they finish on time, with no issues (doubt. If I recall, PC NA had days of problems because they plugged something in wrong), it's not about primetime, it's about the fact it's happening during an event at all. And ZOS clearly feels the same because they extended the event to account for it!

For those newer players among us, this is an unusually stable period for PC EU. But those of us who have been around for a while know that, historically, PC EU has broken first, broken longest and gotten ignored consistently regarding that, despite having, by ZOS' own words, the highest population of all servers. The first time PC NA had even a taste of the problems that regularly plagued PC EU, ZOS threw out a bunch of 150% XP scrolls to soothe the booboo. And now PC EU gets major maintenance scheduled at a time that literally only inconveniences that server? Come on...

ApoAlaia wrote: »

The whole 'thing' with the current event aside (which is already been covered elsewhere so I won't repeat it here) this is what it looks to me.

They are starting the maintenance at 1AM BST, and they are not doing it during the anniversary event, I might be suffering from Stockholm syndrome but it seems pretty considerate to me?

Anniversary would, ironically, actually be better. This Jester's Festival is awful and clear evidence that ZOS either doesn't listen to our feedback or just doesn't care (which is worse). We have yet another event where we have a once a day chance to earn one of the new rewards, one of which requires ten copies, or we could get previous style pages that are still worthless a year on or we could get nothing at all. And they were originally going to make this a day shorter? I'm sorry, but that's garbage. An in-game event should always give players an opportunity to earn everything in it without needing to resort to what ZOS are clearly trying to push is towards (buying event tickets for real money).

ArchMikem wrote: »

Need remind you that the planet has different time zones. Maintenance times will ALWAYS inconvenience someone, regardless of when it's scheduled.

I shouldn't need to explain this but, apparently, I do. It's not about the fact maintenance is happening. It's about the fact it is happening during an event. If it was outside of an event, I wouldn't have said a word, that's a great idea. But ZOS have clearly anticipated disruption to the event by the fact they extended the event pre-emptively.

But since we're on the topic of maintenance being scheduled in a way that inconveniences people, how about the not exactly unusual maintenance timed specifically to screw with trader swap on Tuesdays? Literally any other time of the week and they make guild leaders' lives harder than they already are with the complete lack of guild tools in this game? And ZOS are fully aware of this as there have been numerous complaints about it. As suggested before, they're just completely ignoring feedback.

Zodiarkslayer

thejadefalcon wrote: »

Ah, yes, because ZOS has never gone over their estimate on maintenance or had major problems after even smaller ones. And they've never had it consistently with PC EU. But even if they finish on time, with no issues (doubt. If I recall, PC NA had days of problems because they plugged something in wrong), it's not about primetime, it's about the fact it's happening during an event at all. And ZOS clearly feels the same because they extended the event to account for it!

For those newer players among us, this is an unusually stable period for PC EU. But those of us who have been around for a while know that, historically, PC EU has broken first, broken longest and gotten ignored consistently regarding that, despite having, by ZOS' own words, the highest population of all servers. The first time PC NA had even a taste of the problems that regularly plagued PC EU, ZOS threw out a bunch of 150% XP scrolls to soothe the booboo. And now PC EU gets major maintenance scheduled at a time that literally only inconveniences that server? Come on...

First, roll outs of new hardware or software are messy by nature. That is due to the complex and integrated nature of IT systems.

Second, I am sure the employees at ZOS have learned from their mistakes with the PC/NA rollout. I'd like to see PC/NA as the guinea pigs. It's healthier this way.

Third, I am sure the complexity and scale of this hardware rollout is intentionally kept out of the discussion on ZOSs side, because divulging such sensitive information could be leveraged against them finacially and ultimately also economically.

Fourth, said complexity necessitates coordination between a lot of employees, stretching not just over different teams, like @ZOS_Kevin said, but also different countries, time zones and corporate entities. Possibly even engaging external contractors, that have more experience on the subject or are former employees, that have detailed knowledge of the legacy system.
Getting all of these to agree on one timeframe is an organisational challenge in itself.

Fifth, keeping all of the above in mind, prioritising the convenience of a (however significant) part of the customers is irresponsible, from a management perspective, if it would endanger the project otherwise.

I am personally happy with resulting compromise as it is now. Even if there is six or even eight hours more maintenance down time, there will still be some time left to do the daily for the three tickets.

We will be fine.

thejadefalcon

Zodiarkslayer wrote: »

I'd like to see PC/NA as the guinea pigs. It's healthier this way.

That would be a nice change, ha!

Zodiarkslayer wrote: »

Fourth, said complexity necessitates coordination between a lot of employees, stretching not just over different teams, like @ZOS_Kevin said, but also different countries, time zones and corporate entities. Possibly even engaging external contractors, that have more experience on the subject or are former employees, that have detailed knowledge of the legacy system.

I can understand that, but given they're more than capable of shifting events around on a whim, I don't think moving Jesters or Anniversary, as odd as they would be, would be a challenge. After all, for a long time, we had "Midyear Mayhem" in the middle of January. Even the Anniversary could have been a few weeks off and no-one would have cared and I cannot possibly conceive a scenario where delaying an event (or even bringing Jesters et al forward!) could have negative consequences for any team, let alone the whole game. In fact, bringing it forward would help, given so many people are complaining about event burnout.

thejadefalcon

Oh, here's something special for you I'd actually forgotten about until someone reminded me.

@ZOS_MattFiror said this in January, the last time we actually got an update on this:

ZOS_MattFiror wrote: »

we need to move some events around on the calendar to give the infrastructure team some clear air to do the maintenance without impacting any ongoing events.

Ah, yes, so the solution to giving the infrastructure team time to work without impacting events... was to schedule it during the longest continuous amount of events this game has. ZOS had months (heck, about a year since the PC NA upgrades when you get down to it) to work out the time schedule if organising teams was truly this difficult.

What changed, @ZOS_GinaBruno @ZOS_Kevin ? Why is ZOS still constantly saying one thing, then completely contradicting it the very next post with no acknowledgement? Talk about gaslighting your players.

Elsonso

thejadefalcon wrote: »

I can understand that, but given they're more than capable of shifting events around on a whim, I don't think moving Jesters or Anniversary, as odd as they would be, would be a challenge. After all, for a long time, we had "Midyear Mayhem" in the middle of January. Even the Anniversary could have been a few weeks off and no-one would have cared and I cannot possibly conceive a scenario where delaying an event (or even bringing Jesters et al forward!) could have negative consequences for any team, let alone the whole game. In fact, bringing it forward would help, given so many people are complaining about event burnout.

Instead, they extended Jester's by one day. That seems like a very easy accommodation, even if it adds a day to event fatigue..

ebix_

Lets hope this is not your April fools joke.

React

thejadefalcon wrote: »

Oh, here's something special for you I'd actually forgotten about until someone reminded me.

@ZOS_MattFiror said this in January, the last time we actually got an update on this:

ZOS_MattFiror wrote: »

we need to move some events around on the calendar to give the infrastructure team some clear air to do the maintenance without impacting any ongoing events.

Ah, yes, so the solution to giving the infrastructure team time to work without impacting events... was to schedule it during the longest continuous amount of events this game has. ZOS had months (heck, about a year since the PC NA upgrades when you get down to it) to work out the time schedule if organising teams was truly this difficult.

What changed, @ZOS_GinaBruno @ZOS_Kevin ? Why is ZOS still constantly saying one thing, then completely contradicting it the very next post with no acknowledgement? Talk about gaslighting your players.

Ah, so it's fine to move midyear mayhem to an undetermined future date, but we'll leave the jesters fest and anniversary events smack in the middle of this server replacement.

Makes me wonder if the rumors regarding them adding/reducing cyrodiil server resources temporarily are accurate. Would make a ton of sense given the insanely flawless performance for PC NA following the replacement, which then mysteriously deteriorated out of nowhere after 3-4 months later, as well as the unwillingness to let the event fall during a hardware replacement.

Zodiarkslayer

Nice quote.

I guess this was the plan in January. And honestly Matt Firor's entire post says more. For example the date for the cutover stayed the same. And it seems the intend was to not have the cutover coincide with any event. Only later it became clear that it was impossible. And that wasn't communicated.

I remember one of Kevin's posts that they didn't know when the Jester's Festival is going to be. And I think that was at the end of February. I could be wrong, though.
They probably wanted to stay as flexible as possible for as long as possible.

In the end they had to compromise. At my old company we would've called that a Tuesday Afternoon. 😅

thejadefalcon

Zodiarkslayer wrote: »

And that wasn't communicated.

Yeah. It's a pattern with this company. The only thing regularly communicated is promises of better communication.

kyle.wilson

Why is a company now owned by Microsoft having issues sourcing server equipment?
@ZOS_MattFiror

WoppaBoem

ZOS, The news hit our Xbox EU community hard. Many players where back and left again or where ready to come back with the hardware update. So please use the additional servers from PC EU and PS EU to increase capacity for Xbox EU up until the hardware refresh. The game does work perfectly on low population so we need extra capacity for peak times with the hardware refresh old capacity should become available USE this for Xbox EU!!!!

Elrond87

React wrote: »

Destai wrote: »

Thanks for the update, good to finally get one. Can you ask what are the specific performance issues this fixes?

When they replaced the hardware on PC NA, for a short period(3 or 4 months) cyrodiil server performance was perfect. No lag at all in pop locked campaigns with massive sieges happening.

After about three or four months, the performance suddenly and drastically decreased. Now, almost a year later, the server performance is quite literally back to where it was pre-hardware refresh. Every night when the campaigns pop lock, they become unplayable. It's very odd that the performance increased so drastically and then decreased so drastically. Almost feels like there were additional (necessary) server resources at the start, that were then turned off or reallocated elsewhere in the later part of the year.

Strange how it is the same for client end well for me anyway, after prolonged login fps gets eaten up by something unknown during action packed moments, but upon first login action packed moment have high fps, is there some problem with the game eating up resources server and client end i wonder, also when logging out after prolonged login it just hangs until pc is reset but logouts are fine if briefly done.

Blackbird_V

ZOS_Kevin wrote: »

Tommy_The_Gun wrote: »

It would be ideal if ZOS would communicate internally better so that the team responsible for the in-game events would know in advance when there are some potenial server down-times and plan events in different days.

Wanted to chime in here and provide a little bit of context. The in-game events team is aware and part of the process of the hardware refresh. In fact, many teams need to be on the same page for a maintenance like this to happen. One thing to remember is, there are many teams working together to pull this off. We understand sometimes that isn't the best timing for events and how that is frustrating. As a reminder, that is why we built in the extra day for Jester's, to try and balance that out. However, we also have to take into account the teams involved and where those teams are. The teams have to be available to do the work. And pushing out one of these dates would inevitably impact when other servers get their refresh. The team considers multiple angles when choosing a time for these and work to create the least friction possible, when possible. Hopefully this provides some context for timings, as events are not the only factor the team has to consider when planning out the maintenance here.

While we know it is frustrating, we to appreciate everyone's patience and understanding for these maintenance windows.

Not to mention, and to give you credit - there've been events in the past that overflowed into maintenance, extended maintenance or plagued with bugs and you guys have always extended the events in that case

Here's a few:

• Anniversary Event 2016
• Anniversary Event 2021
• Witches Festival 2016
• Witches Festival 2019

If there are major issues throughout the next few days, then I am sure ZoS would see that 1 day extra is not adequate and would adjust end date.

thejadefalcon

Blackbird_V wrote: »

If there are major issues throughout the next few days, then I am sure ZoS would see that 1 day extra is not adequate and would adjust end date.

They can't extend Jester's Festival because they've scheduled Anniversary directly after it. And, by their own words, moving events is a titanic task. So the only solution would be for a second Jester's Festival in August, I guess.

ZOS_Kevin

AdexiTheOneEyed wrote: »

Kevin, is there any update on Whitestrake's Mayhem? I'm still holding out hope we'll get the event, at this point, it feels largely unlikely.

Have an update here for ya!

Zodiarkslayer

kyle.wilson wrote: »

Why is a company now owned by Microsoft having issues sourcing server equipment?
@ZOS_MattFiror

Because everybody has problems.

Zodiarkslayer

ZOS_Kevin wrote: »

AdexiTheOneEyed wrote: »

Kevin, is there any update on Whitestrake's Mayhem? I'm still holding out hope we'll get the event, at this point, it feels largely unlikely.

Have an update here for ya!

And in between, the Necrom launch on june 5th? Talk about a full plate.

I don't envy you guys. 😅

Paske

May blessings of Meridia fall upon your kind soul.

Finally the day is upon us.

Let us rejoice !

I shall open a bottle of single malt in your honour.

Ratzkifal

I want to be hopeful about this, but I've been told the NA performance has massively deteriorated since the refresh was done. So what is being done to prevent this from happening in EU as well?
Also, it was said on multiple occassions that this is not a silver bullet for performance, yet we've also been told that you now know what needs to be done to fix performance. Well, after the refresh, what's next for PC EU? How will you ensure performance stays good?
I want to know if it's worth coming back to this game or not. And just so you know, no answer is also an answer.

disintegr8

Yes, it's not how the game performs immediately after the upgrade, it's about how it's running down the track, weeks or months after the upgrade that matters.

I'm on PS4-NA and because of my location, do not expect our refresh (when we get it) to do much to improve the performance I get, especially when I'm in Cyrodiil.

FeedbackOnly

thejadefalcon wrote: »

"why we preemptively added an extra day to the beginning of the event."

So you're anticipating people can't play and earn tickets and your solution is to benefit other servers more than PC EU (since everyone gets the extra day, not just PC EU), rather than just... schedule the maintenance some other time?

Don't get me wrong, yay for smoother Cyrodiil, but seriously, why are you so determined to consistently schedule maintenance at the worst possible times for your players? This isn't even an event thing, it's just even more ludicrous when it is.

Na lost 10 days last year, it could be worse

Mayrael

In the context of what @React wrote, hold back your unrestrained joy, I want to see how the situation will be in a few months. Why my lack of enthusiasm? Because I have 9 years of experience with this company, more than some of its employees.

LarsS

I am playing on both PC/NA and PC/EU. NA is still much better than EU at respective primetimes.

The real game changer though it the introduction of multi-threaded code for the server, the first part of that change should come in June.

fred4

React wrote: »

ZOS_Kevin wrote: »

Tommy_The_Gun wrote: »

It would be ideal if ZOS would communicate internally better so that the team responsible for the in-game events would know in advance when there are some potenial server down-times and plan events in different days.

Wanted to chime in here and provide a little bit of context. The in-game events team is aware and part of the process of the hardware refresh. In fact, many teams need to be on the same page for a maintenance like this to happen. One thing to remember is, there are many teams working together to pull this off. We understand sometimes that isn't the best timing for events and how that is frustrating. As a reminder, that is why we built in the extra day for Jester's, to try and balance that out. However, we also have to take into account the teams involved and where those teams are. The teams have to be available to do the work. And pushing out one of these dates would inevitably impact when other servers get their refresh. The team considers multiple angles when choosing a time for these and work to create the least friction possible, when possible. Hopefully this provides some context for timings, as events are not the only factor the team has to consider when planning out the maintenance here.

While we know it is frustrating, we to appreciate everyone's patience and understanding for these maintenance windows.

I know it likely isn't at the top of the priority list, but are the developers aware of the timeline with PC NA's server performance in the past year? Are there any plans to address this?

When we got the fresh hardware, cyrodiil ran flawlessly. I don't say that lightly - at the end of week one post server replacement, I had one particular fight where I was in a keep with over 100 players on the player counter, multiple "ball groups" fighting with one another, over 30 siege deployed across the 3 factions, in a pop locked gray host during the prime hours. There was no ability delay or lag at all. This was completely unheard of - the game never had run this well in it's entire life. Everybody I know that PVPs was ecstatic, and I had numerous friends returning to the game.

Fast forward 3-4 months post server replacement, and suddenly the performance drastically decreases. GrayHost begins to become "unplayable" again each night once it fills, and any time there is a large conflict (particularly between "ball groups"), the entire campaign experienced skill delay. It seemingly just happened out of nowhere.

Now, one year post hardware replacement, both CP cyrodiil campaigns become extremely laggy as soon as they fill. There is significant skill delay every single night, and it is very close to the level of unbearable performance we had right before the hardware refresh.

I'm no server engineer. But I do have friends that work in that field, and what they've suggested to me is that it seems like when the servers were first replaced, there were more resources dedicated to cyrodiil allowing for the flawless performance we didn't think was possible. They have implied that the sudden and drastic decrease 3-4 months later would indicate that some of those server resources were reallocated.

I've wondered whether ZOS are tweaking the population cap behind the scenes. Could performance deterioration be due to a much higher population cap putting the NA servers back to square one?

method__01

⚠️The maintenance for the
@TESOnline
European PC/Mac megaserver has been extended to 2:00PM EDT / 18:00 UTC. #ESO

Zodiarkslayer

@thejadefalcon It seems my high hopes have been misplaced. 😅

You may now say: I told you so! 🤷‍♂️

Elsonso

Someone forgot to tightly close the lid on the bin holding the random numbers for the server RNG. Now they are all sitting around the server thinking of random numbers to replace them.

Eldartar

thejadefalcon wrote: »

"why we preemptively added an extra day to the beginning of the event."

So you're anticipating people can't play and earn tickets and your solution is to benefit other servers more than PC EU (since everyone gets the extra day, not just PC EU), rather than just... schedule the maintenance some other time?

Don't get me wrong, yay for smoother Cyrodiil, but seriously, why are you so determined to consistently schedule maintenance at the worst possible times for your players? This isn't even an event thing, it's just even more ludicrous when it is.

WOW, I mean, just WOW!! Read the room man.

thejadefalcon

Zodiarkslayer wrote: »

@thejadefalcon It seems my high hopes have been misplaced. 😅

You may now say: I told you so! 🤷‍♂️

I nearly did ask at what point does the maintenance affect "primetime", but I thought I'd wait and see what the revised estimate was. Now I'm just too tired to give a damn. All of this could have been so easily avoided. I just want to make sure my guild has a trader, for god's sake...