Hardware Refresh Work Update – March 31, 2023

ZOS_Kevin

Hi All, we wanted to share with everyone the latest on our ongoing hardware refresh work. The work on PC EU is nearing completion. We are doing the cutover during an extended maintenance that will start in the evening on April 3 and end in the morning on April 4. Tentative times are April 3 at 8:00 PM EDT to April 4 at 10:00 AM EDT (1:00 AM BST to 3:00 PM BST on April 4). Once the maintenance is complete, we will monitor the PC EU server performance. This maintenance does occur during the Jester’s Festival and is why we preemptively added an extra day to the beginning of the event.

For the remaining servers, below are the latest planned timelines. Please note that these timelines could change if we run into any issues with the hardware refresh process or other issues arise with the ESO live service that takes priority.

• PlayStation EU: This server is next, and that work will be completed later this month (April.) We will let you all know when we’ve locked in a date.
• Xbox NA and PlayStation NA: This work is scheduled for completion sometime in May. We will provide dates when we have them.
• Xbox EU: Additional work is needed in our Frankfurt datacenter before we can do the Xbox EU hardware refresh work. We recognize that this server coming last is not in-line with what we originally communicated in December. This is the server experiencing the least amount of performance/stability issues. The hardware refresh work for this server is planned for some time later this year. We’ll provide updates as we have them.

Thank you!

thejadefalcon

"why we preemptively added an extra day to the beginning of the event."

So you're anticipating people can't play and earn tickets and your solution is to benefit other servers more than PC EU (since everyone gets the extra day, not just PC EU), rather than just... schedule the maintenance some other time?

Don't get me wrong, yay for smoother Cyrodiil, but seriously, why are you so determined to consistently schedule maintenance at the worst possible times for your players? This isn't even an event thing, it's just even more ludicrous when it is.

AdexiTheOneEyed

Kevin, is there any update on Whitestrake's Mayhem? I'm still holding out hope we'll get the event, at this point, it feels largely unlikely. Last year, ZOS swapped out the Imperial City event to put in their High Isle Event. As a result, the price of Hakeijo in the market is through the roof, it doesn't help that Whitestrake's hasn't happened yet and that has further increased the prices of certain items that we'd normally get in Mayhem event boxes.

As always, we sincerely hope for changes and fixed to both Overland and PvP zones.

ZOS_Kevin

AdexiTheOneEyed wrote: »

Kevin, is there any update on Whitestrake's Mayhem?

Nothing to announce right now, but we will have an update on Whitestrake's Mayhem soon.

Sidewaves89

thejadefalcon wrote: »

"why we preemptively added an extra day to the beginning of the event."

So you're anticipating people can't play and earn tickets and your solution is to benefit other servers more than PC EU (since everyone gets the extra day, not just PC EU), rather than just... schedule the maintenance some other time?

Don't get me wrong, yay for smoother Cyrodiil, but seriously, why are you so determined to consistently schedule maintenance at the worst possible times for your players? This isn't even an event thing, it's just even more ludicrous when it is.

[snip] tickets, you getting new servers. [snip] relax and enjoy.

[edited for baiting]

Cobble123

thejadefalcon wrote: »

"why we preemptively added an extra day to the beginning of the event."

So you're anticipating people can't play and earn tickets and your solution is to benefit other servers more than PC EU (since everyone gets the extra day, not just PC EU), rather than just... schedule the maintenance some other time?

Don't get me wrong, yay for smoother Cyrodiil, but seriously, why are you so determined to consistently schedule maintenance at the worst possible times for your players? This isn't even an event thing, it's just even more ludicrous when it is.

Wow. Just wow.

Reverb

I’ll be very interested in hearing if PC EU sees the same pvp performance gains that we did on NA for the first week after the hardware upgrades.

Ingenon

Thanks for providing this schedule update!

I am looking forward to the PlayStation NA server upgrade, and getting to play using server hardware that is newer than the PS4 that I already traded in for a PS5.

Gadamlub14_ESO

an extra day for the event because you expect many people to miss a day when the event was already specifically designed to not give you enough shards to earn new cosmetics in the first place even with that extra day?

Well gee, thanks for forcing me to buy even more event tickets if i want anything from this event since this is during the only time i can play.

colossalvoids

Thanks for the update, new hardware would be the best anniversary present to the community if everything would be set up correctly and next patch won't introduce some additional over the top calculations rendering it useless. Fingers crossed!

Poss

Xbox EU “This is the server experiencing the least amount of performance/stability issues.”

Lol ok

Destai

ZOS_Kevin wrote: »

Hi All, we wanted to share with everyone the latest on our ongoing hardware refresh work. The work on PC EU is nearing completion. We are doing the cutover during an extended maintenance that will start in the evening on April 3 and end in the morning on April 4. Tentative times are April 3 at 8:00 PM EDT to April 4 at 10:00 AM EDT (1:00 AM BST to 3:00 PM BST on April 4). Once the maintenance is complete, we will monitor the PC EU server performance. This maintenance does occur during the Jester’s Festival and is why we preemptively added an extra day to the beginning of the event.

For the remaining servers, below are the latest planned timelines. Please note that these timelines could change if we run into any issues with the hardware refresh process or other issues arise with the ESO live service that takes priority.

• PlayStation EU: This server is next, and that work will be completed later this month (April.) We will let you all know when we’ve locked in a date.
• Xbox NA and PlayStation NA: This work is scheduled for completion sometime in May. We will provide dates when we have them.
• Xbox EU: Additional work is needed in our Frankfurt datacenter before we can do the Xbox EU hardware refresh work. We recognize that this server coming last is not in-line with what we originally communicated in December. This is the server experiencing the least amount of performance/stability issues. The hardware refresh work for this server is planned for some time later this year. We’ll provide updates as we have them.

Thank you!

Thanks for the update, good to finally get one. Can you ask what are the specific performance issues this fixes?

React

Destai wrote: »

ZOS_Kevin wrote: »

Hi All, we wanted to share with everyone the latest on our ongoing hardware refresh work. The work on PC EU is nearing completion. We are doing the cutover during an extended maintenance that will start in the evening on April 3 and end in the morning on April 4. Tentative times are April 3 at 8:00 PM EDT to April 4 at 10:00 AM EDT (1:00 AM BST to 3:00 PM BST on April 4). Once the maintenance is complete, we will monitor the PC EU server performance. This maintenance does occur during the Jester’s Festival and is why we preemptively added an extra day to the beginning of the event.

For the remaining servers, below are the latest planned timelines. Please note that these timelines could change if we run into any issues with the hardware refresh process or other issues arise with the ESO live service that takes priority.

• PlayStation EU: This server is next, and that work will be completed later this month (April.) We will let you all know when we’ve locked in a date.
• Xbox NA and PlayStation NA: This work is scheduled for completion sometime in May. We will provide dates when we have them.
• Xbox EU: Additional work is needed in our Frankfurt datacenter before we can do the Xbox EU hardware refresh work. We recognize that this server coming last is not in-line with what we originally communicated in December. This is the server experiencing the least amount of performance/stability issues. The hardware refresh work for this server is planned for some time later this year. We’ll provide updates as we have them.

Thank you!

Thanks for the update, good to finally get one. Can you ask what are the specific performance issues this fixes?

When they replaced the hardware on PC NA, for a short period(3 or 4 months) cyrodiil server performance was perfect. No lag at all in pop locked campaigns with massive sieges happening.

After about three or four months, the performance suddenly and drastically decreased. Now, almost a year later, the server performance is quite literally back to where it was pre-hardware refresh. Every night when the campaigns pop lock, they become unplayable. It's very odd that the performance increased so drastically and then decreased so drastically. Almost feels like there were additional (necessary) server resources at the start, that were then turned off or reallocated elsewhere in the later part of the year.

Zodiarkslayer

thejadefalcon wrote: »

"why we preemptively added an extra day to the beginning of the event."

So you're anticipating people can't play and earn tickets and your solution is to benefit other servers more than PC EU (since everyone gets the extra day, not just PC EU), rather than just... schedule the maintenance some other time?

Don't get me wrong, yay for smoother Cyrodiil, but seriously, why are you so determined to consistently schedule maintenance at the worst possible times for your players? This isn't even an event thing, it's just even more ludicrous when it is.

This is nonsensical.

@thejadefalcon You are from the UK, right? In the UK the servers go down on 1 a.m. April 4th and are up at 3 p.m. the same day. This should be plenty of opportunity to get all possible tickets.

The times seem to be deliberately chosen to avoid EU primetime.

ApoAlaia

Zodiarkslayer wrote: »

thejadefalcon wrote: »

"why we preemptively added an extra day to the beginning of the event."

So you're anticipating people can't play and earn tickets and your solution is to benefit other servers more than PC EU (since everyone gets the extra day, not just PC EU), rather than just... schedule the maintenance some other time?

Don't get me wrong, yay for smoother Cyrodiil, but seriously, why are you so determined to consistently schedule maintenance at the worst possible times for your players? This isn't even an event thing, it's just even more ludicrous when it is.

This is nonsensical.

@thejadefalcon You are from the UK, right? In the UK the servers go down on 1 a.m. April 4th and are up at 3 p.m. the same day. This should be plenty of opportunity to get all possible tickets.

The times seem to be deliberately chosen to avoid EU primetime.

The whole 'thing' with the current event aside (which is already been covered elsewhere so I won't repeat it here) this is what it looks to me.

They are starting the maintenance at 1AM BST, and they are not doing it during the anniversary event, I might be suffering from Stockholm syndrome but it seems pretty considerate to me?

DirtyDeeds765

ZOS_Kevin wrote: »

Hi All, we wanted to share with everyone the latest on our ongoing hardware refresh work. The work on PC EU is nearing completion. We are doing the cutover during an extended maintenance that will start in the evening on April 3 and end in the morning on April 4. Tentative times are April 3 at 8:00 PM EDT to April 4 at 10:00 AM EDT (1:00 AM BST to 3:00 PM BST on April 4). Once the maintenance is complete, we will monitor the PC EU server performance. This maintenance does occur during the Jester’s Festival and is why we preemptively added an extra day to the beginning of the event.

For the remaining servers, below are the latest planned timelines. Please note that these timelines could change if we run into any issues with the hardware refresh process or other issues arise with the ESO live service that takes priority.

• PlayStation EU: This server is next, and that work will be completed later this month (April.) We will let you all know when we’ve locked in a date.
• Xbox NA and PlayStation NA: This work is scheduled for completion sometime in May. We will provide dates when we have them.
• Xbox EU: Additional work is needed in our Frankfurt datacenter before we can do the Xbox EU hardware refresh work. We recognize that this server coming last is not in-line with what we originally communicated in December. This is the server experiencing the least amount of performance/stability issues. The hardware refresh work for this server is planned for some time later this year. We’ll provide updates as we have them.

Thank you!

Great! Happy to see an update for PSNA and glad it is happening soon! I was disappointed we were back to no concrete-as-can-be-date but you guys got us back to one quickly.

We know you guys don't hear it enough - we do appreciate what you do.

@DirtyDeeds765

Tommy_The_Gun

I think I am gonna leave some quests done, but not completed, so if I will be lucky I will just login & get event ticket ASAP. At least, this is the only countermeasure I can think of.

It would be ideal if ZOS would communicate internally better so that the team responsible for the in-game events would know in advance when there are some potenial server down-times and plan events in different days.

vgabor

You guys just moved the weekly timers reset to Tuesday, so how about you stop doing maintenance on Tuesdays? There are plenty enough days on a week which not collide with the reset times, why not pick one of those?

@ZOS_Kevin

ZOS_Kevin

Tommy_The_Gun wrote: »

It would be ideal if ZOS would communicate internally better so that the team responsible for the in-game events would know in advance when there are some potenial server down-times and plan events in different days.

Wanted to chime in here and provide a little bit of context. The in-game events team is aware and part of the process of the hardware refresh. In fact, many teams need to be on the same page for a maintenance like this to happen. One thing to remember is, there are many teams working together to pull this off. We understand sometimes that isn't the best timing for events and how that is frustrating. As a reminder, that is why we built in the extra day for Jester's, to try and balance that out. However, we also have to take into account the teams involved and where those teams are. The teams have to be available to do the work. And pushing out one of these dates would inevitably impact when other servers get their refresh. The team considers multiple angles when choosing a time for these and work to create the least friction possible, when possible. Hopefully this provides some context for timings, as events are not the only factor the team has to consider when planning out the maintenance here.

While we know it is frustrating, we to appreciate everyone's patience and understanding for these maintenance windows.

React

ZOS_Kevin wrote: »

Tommy_The_Gun wrote: »

It would be ideal if ZOS would communicate internally better so that the team responsible for the in-game events would know in advance when there are some potenial server down-times and plan events in different days.

Wanted to chime in here and provide a little bit of context. The in-game events team is aware and part of the process of the hardware refresh. In fact, many teams need to be on the same page for a maintenance like this to happen. One thing to remember is, there are many teams working together to pull this off. We understand sometimes that isn't the best timing for events and how that is frustrating. As a reminder, that is why we built in the extra day for Jester's, to try and balance that out. However, we also have to take into account the teams involved and where those teams are. The teams have to be available to do the work. And pushing out one of these dates would inevitably impact when other servers get their refresh. The team considers multiple angles when choosing a time for these and work to create the least friction possible, when possible. Hopefully this provides some context for timings, as events are not the only factor the team has to consider when planning out the maintenance here.

While we know it is frustrating, we to appreciate everyone's patience and understanding for these maintenance windows.

I know it likely isn't at the top of the priority list, but are the developers aware of the timeline with PC NA's server performance in the past year? Are there any plans to address this?

When we got the fresh hardware, cyrodiil ran flawlessly. I don't say that lightly - at the end of week one post server replacement, I had one particular fight where I was in a keep with over 100 players on the player counter, multiple "ball groups" fighting with one another, over 30 siege deployed across the 3 factions, in a pop locked gray host during the prime hours. There was no ability delay or lag at all. This was completely unheard of - the game never had run this well in it's entire life. Everybody I know that PVPs was ecstatic, and I had numerous friends returning to the game.

Fast forward 3-4 months post server replacement, and suddenly the performance drastically decreases. GrayHost begins to become "unplayable" again each night once it fills, and any time there is a large conflict (particularly between "ball groups"), the entire campaign experienced skill delay. It seemingly just happened out of nowhere.

Now, one year post hardware replacement, both CP cyrodiil campaigns become extremely laggy as soon as they fill. There is significant skill delay every single night, and it is very close to the level of unbearable performance we had right before the hardware refresh.

I'm no server engineer. But I do have friends that work in that field, and what they've suggested to me is that it seems like when the servers were first replaced, there were more resources dedicated to cyrodiil allowing for the flawless performance we didn't think was possible. They have implied that the sudden and drastic decrease 3-4 months later would indicate that some of those server resources were reallocated.

pecheckler

Is this going to make cyrodiil not constantly lag like crazy on ps5? It's quite frustrating having abilities not trigger nearly all the time.

ZOS_Kevin

@React The team is monitoring PC NA performance, but that is all we have to share at this time. However, we will make sure to share (and have been sharing) feedback like this as well.

React

ZOS_Kevin wrote: »

@React The team is monitoring PC NA performance, but that is all we have to share at this time. However, we will make sure to share (and have been sharing) feedback like this as well.

Thanks for the reply.

ArchMikem

thejadefalcon wrote: »

"why we preemptively added an extra day to the beginning of the event."

So you're anticipating people can't play and earn tickets and your solution is to benefit other servers more than PC EU (since everyone gets the extra day, not just PC EU), rather than just... schedule the maintenance some other time?

Don't get me wrong, yay for smoother Cyrodiil, but seriously, why are you so determined to consistently schedule maintenance at the worst possible times for your players? This isn't even an event thing, it's just even more ludicrous when it is.

Need remind you that the planet has different time zones. Maintenance times will ALWAYS inconvenience someone, regardless of when it's scheduled.

Destai

ZOS_Kevin wrote: »

Tommy_The_Gun wrote: »

It would be ideal if ZOS would communicate internally better so that the team responsible for the in-game events would know in advance when there are some potenial server down-times and plan events in different days.

Wanted to chime in here and provide a little bit of context. The in-game events team is aware and part of the process of the hardware refresh. In fact, many teams need to be on the same page for a maintenance like this to happen. One thing to remember is, there are many teams working together to pull this off. We understand sometimes that isn't the best timing for events and how that is frustrating. As a reminder, that is why we built in the extra day for Jester's, to try and balance that out. However, we also have to take into account the teams involved and where those teams are. The teams have to be available to do the work. And pushing out one of these dates would inevitably impact when other servers get their refresh. The team considers multiple angles when choosing a time for these and work to create the least friction possible, when possible. Hopefully this provides some context for timings, as events are not the only factor the team has to consider when planning out the maintenance here.

While we know it is frustrating, we to appreciate everyone's patience and understanding for these maintenance windows.

Fantastic reply, Kevin. Context like this is really appreciated! Thanks man.

vsrs_au

Destai wrote: »

ZOS_Kevin wrote: »

Tommy_The_Gun wrote: »

It would be ideal if ZOS would communicate internally better so that the team responsible for the in-game events would know in advance when there are some potenial server down-times and plan events in different days.

Wanted to chime in here and provide a little bit of context. The in-game events team is aware and part of the process of the hardware refresh. In fact, many teams need to be on the same page for a maintenance like this to happen. One thing to remember is, there are many teams working together to pull this off. We understand sometimes that isn't the best timing for events and how that is frustrating. As a reminder, that is why we built in the extra day for Jester's, to try and balance that out. However, we also have to take into account the teams involved and where those teams are. The teams have to be available to do the work. And pushing out one of these dates would inevitably impact when other servers get their refresh. The team considers multiple angles when choosing a time for these and work to create the least friction possible, when possible. Hopefully this provides some context for timings, as events are not the only factor the team has to consider when planning out the maintenance here.

While we know it is frustrating, we to appreciate everyone's patience and understanding for these maintenance windows.

Fantastic reply, Kevin. Context like this is really appreciated! Thanks man.

I wholeheartedly endorse that comment, and @ZOS_Kevin you get an 'awesome' from me for every post in this thread. They're much appreciated.

karthrag_inak

Khajiit is certainly impressed with the performance he's been having this evening. Lagged out to launch screen 3 times.

LadyLethalla

@ZOS_Kevin Please please please no maintenance on April 5th... it's my birthday...!

Sarannah

Awesome!!! And thanks for adding an extra day of jester's festival in advance, for those who might miss out on a day of tickets.

Would love it if we could get an update on the gameplay implications of the server hardware refresh/upgrade. Like will this allow for more character slots per server, more inventory space, more bank storage space, higher furniture limits in houses, etc? Does this refresh/upgrade have any other gameplay implications?
The only thing we hear about this is that the hardware refresh/upgrade should make the game/Cyrodiil run smoother, but any other possible implications have never been officially addressed.

Northwold

vgabor wrote: »

You guys just moved the weekly timers reset to Tuesday, so how about you stop doing maintenance on Tuesdays? There are plenty enough days on a week which not collide with the reset times, why not pick one of those?

@ZOS_Kevin

I'm not understanding why this is significant? You might equally complain if it was Monday that that leaves you a day less to meet the reset timers, or if it was Wednesday that that means you've lost a day from the timers. There is no eighth day to do maintenance on that will suit everyone's personal niggles.

Lebkuchen

@React I hope one day you will get at least one real answer to one of your questions.