Status on Hardware Refresh & Whitestrake’s Mayhem Event [January 2023]

React

Elsonso wrote: »

TechMaybeHic wrote: »

Still plan to have it twice this year, but given the timeline, it seems the first will be April/May when the original was June/July, and with how people already get upset that there is a PvP event for tickets already; I can't see how we still don't lose out by having it pushed back a few months to then cram in a 2nd one by the end of the year.

In my mind, the bigger question is "why now?"

Clearly they had no reservations about running it in the past. ZOS has showcased their server limitations many times before. Anyone remember trying to get to the Chef during early anniversary events? ZOS knew their servers couldn't handle it, but they ran the event anyway.

So, what changed?

My guess is that they run additional hardware or increase the capacity typically during midyear mayhems, and since the hardware refresh for one server was seemingly falling during the event, they felt it would be problematic to do the swap while running that increased capacity. Or perhaps they didn't want to risk downtime for a server during the pvp event, although that seems unlikely.

Elsonso

React wrote: »

Elsonso wrote: »

TechMaybeHic wrote: »

Still plan to have it twice this year, but given the timeline, it seems the first will be April/May when the original was June/July, and with how people already get upset that there is a PvP event for tickets already; I can't see how we still don't lose out by having it pushed back a few months to then cram in a 2nd one by the end of the year.

In my mind, the bigger question is "why now?"

Clearly they had no reservations about running it in the past. ZOS has showcased their server limitations many times before. Anyone remember trying to get to the Chef during early anniversary events? ZOS knew their servers couldn't handle it, but they ran the event anyway.

So, what changed?

My guess is that they run additional hardware or increase the capacity typically during midyear mayhems, and since the hardware refresh for one server was seemingly falling during the event, they felt it would be problematic to do the swap while running that increased capacity. Or perhaps they didn't want to risk downtime for a server during the pvp event, although that seems unlikely.

A lot of people think they added hardware during the Mayhem event, but that doesn't ring right. My thought is that if they could run additional hardware that easily, they would not have been waiting 18 months to get hardware. If they were able to source some of the hardware, enough to make Mayhem work, they certainly would not have removed it. That is just extra work putting it in and taking it out when the goal is to put it in and not take it out.

My gut tells me that they did something server side... a sacrifice of something... and they were willing to do that for a week for previous events, but not this time around.

Jaraal

Elsonso wrote: »

React wrote: »

Elsonso wrote: »

TechMaybeHic wrote: »

Still plan to have it twice this year, but given the timeline, it seems the first will be April/May when the original was June/July, and with how people already get upset that there is a PvP event for tickets already; I can't see how we still don't lose out by having it pushed back a few months to then cram in a 2nd one by the end of the year.

In my mind, the bigger question is "why now?"

Clearly they had no reservations about running it in the past. ZOS has showcased their server limitations many times before. Anyone remember trying to get to the Chef during early anniversary events? ZOS knew their servers couldn't handle it, but they ran the event anyway.

So, what changed?

My guess is that they run additional hardware or increase the capacity typically during midyear mayhems, and since the hardware refresh for one server was seemingly falling during the event, they felt it would be problematic to do the swap while running that increased capacity. Or perhaps they didn't want to risk downtime for a server during the pvp event, although that seems unlikely.

A lot of people think they added hardware during the Mayhem event, but that doesn't ring right. My thought is that if they could run additional hardware that easily, they would not have been waiting 18 months to get hardware. If they were able to source some of the hardware, enough to make Mayhem work, they certainly would not have removed it. That is just extra work putting it in and taking it out when the goal is to put it in and not take it out.

My gut tells me that they did something server side... a sacrifice of something... and they were willing to do that for a week for previous events, but not this time around.

My opinion is that they may have loosened the data bottleneck at the Akamai scrubbing service. It's a well known fact, supported by a lot of tracert documentation that much of the lag originates at the anti-DDoS servers that all game data is routed through. A simple search for Akamai here will bring up all the information you need.

React

Elsonso wrote: »

React wrote: »

Elsonso wrote: »

TechMaybeHic wrote: »

Still plan to have it twice this year, but given the timeline, it seems the first will be April/May when the original was June/July, and with how people already get upset that there is a PvP event for tickets already; I can't see how we still don't lose out by having it pushed back a few months to then cram in a 2nd one by the end of the year.

In my mind, the bigger question is "why now?"

Clearly they had no reservations about running it in the past. ZOS has showcased their server limitations many times before. Anyone remember trying to get to the Chef during early anniversary events? ZOS knew their servers couldn't handle it, but they ran the event anyway.

So, what changed?

My guess is that they run additional hardware or increase the capacity typically during midyear mayhems, and since the hardware refresh for one server was seemingly falling during the event, they felt it would be problematic to do the swap while running that increased capacity. Or perhaps they didn't want to risk downtime for a server during the pvp event, although that seems unlikely.

A lot of people think they added hardware during the Mayhem event, but that doesn't ring right. My thought is that if they could run additional hardware that easily, they would not have been waiting 18 months to get hardware. If they were able to source some of the hardware, enough to make Mayhem work, they certainly would not have removed it. That is just extra work putting it in and taking it out when the goal is to put it in and not take it out.

My gut tells me that they did something server side... a sacrifice of something... and they were willing to do that for a week for previous events, but not this time around.

My understanding is that it has to do with allocating additional existing server resources away from other areas of the game to cyrodiil for the duration of the event. Not adding additional physical hardware.

Ingenon

React wrote: »

My understanding is that it has to do with allocating additional existing server resources away from other areas of the game to cyrodiil for the duration of the event. Not adding additional physical hardware.

Agreed. And it ends up being a zero-sum game. In order for Mayhem event to get more of the server resources, everyone playing elsewhere in the game gets less. I believe in the past they were willing to do that, and this year they are not. If some of the servers are nearly maxed out now, there simply may not be any slack to reallocate for Mayhem event campaigns, without increasing lag a lot for everything else. Whereas in the past, there was some slack.

geonsocal

ZOS_MattFiror wrote: »

I’d like to give everyone an update on the server hardware refresh, where we are completely replacing hardware in our North American and European datacenters.

• We are running 1 or 2 weeks behind schedule because of shipping slowdowns. This is entirely out of our control and I apologize for it. This two week delay puts the refresh right on top of our March launch of Update 37, which for obvious reasons we don’t want to schedule simultaneously. So the new date for PC EU getting new hardware will be in early April, and PlayStation EU should follow very shortly after. In order to begin prepping for this work, we’re scheduling a maintenance on all EU realms that will occur on February 7.
  
• The March/April ESO timeframe is always very busy. We have a lot of events and almost always have a major Update. Because of this, we need to move some events around on the calendar to give the infrastructure team some clear air to do the maintenance without impacting any ongoing events. We are moving Whitestrake’s Mayhem to later in the year to ensure the datacenter hardware refresh is complete before we hold the event. We still intend to have two Whitestrake’s Mayhem events in 2023.

just kidding...it is what it is...we're all here by choice...

hopefully one of these days our favorite dungeon masters get to write a message where things are moving along smoothly and game performance is hardly an issue at all...

that would be nice...

jle30303

Instead of having *no* event at a time when there should have been a Mayhem event, why wasn't it replaced with another event in a "known" format that doesn't put so much strain on the servers because it's held in a ton of different places, and separates people out into instances?

An Undaunted event would have been the perfect filler, but add some event tickets. For instance... 3 tickets per day (since Whitestrake Mayhem has 3 per day, 2 for Cyrodiil and 1 for Imp City)... 1 for a Bolgrul Daily Delve. 1 for a Base Game pledge (Maj or Glirion). 1 for a DLC Pledge (Urgalag).

Since Imperial City is free, and its dungeons are among Urgalag's, all players would have access to all three daily tickets.

virtus753

jle30303 wrote: »

Instead of having *no* event at a time when there should have been a Mayhem event, why wasn't it replaced with another event in a "known" format that doesn't put so much strain on the servers because it's held in a ton of different places, and separates people out into instances?

An Undaunted event would have been the perfect filler, but add some event tickets. For instance... 3 tickets per day (since Whitestrake Mayhem has 3 per day, 2 for Cyrodiil and 1 for Imp City)... 1 for a Bolgrul Daily Delve. 1 for a Base Game pledge (Maj or Glirion). 1 for a DLC Pledge (Urgalag).

Since Imperial City is free, and its dungeons are among Urgalag's, all players would have access to all three daily tickets.

Imperial City Prison and White Gold Tower aren’t daily pledges every day, even if that’s the only DLC you own. You would have to hope that they come up in the rotation during the event, and then you’d only get them up to once each. People who have only that DLC would not be able to get their DLC pledge quest done on any other days.

There are also people who might not be willing to get IC, since that means that they would be eligible to get ICP and WGT for random normals forever after. Their choice, but a poor choice for ZOS to have put us in the position to make.

LarsS

@ZOS_Kevin can we get an update on the date for the new EU/PC server?

Jaavaa

@ZOS_Kevin xbox Eu Servers? Any News here?

Sun7dance

Jaavaa wrote: »

@ZOS_Kevin xbox Eu crossplay Servers? Any News here?

zaria

Jaraal wrote: »

Elsonso wrote: »

React wrote: »

Elsonso wrote: »

TechMaybeHic wrote: »

Still plan to have it twice this year, but given the timeline, it seems the first will be April/May when the original was June/July, and with how people already get upset that there is a PvP event for tickets already; I can't see how we still don't lose out by having it pushed back a few months to then cram in a 2nd one by the end of the year.

In my mind, the bigger question is "why now?"

Clearly they had no reservations about running it in the past. ZOS has showcased their server limitations many times before. Anyone remember trying to get to the Chef during early anniversary events? ZOS knew their servers couldn't handle it, but they ran the event anyway.

So, what changed?

My guess is that they run additional hardware or increase the capacity typically during midyear mayhems, and since the hardware refresh for one server was seemingly falling during the event, they felt it would be problematic to do the swap while running that increased capacity. Or perhaps they didn't want to risk downtime for a server during the pvp event, although that seems unlikely.

A lot of people think they added hardware during the Mayhem event, but that doesn't ring right. My thought is that if they could run additional hardware that easily, they would not have been waiting 18 months to get hardware. If they were able to source some of the hardware, enough to make Mayhem work, they certainly would not have removed it. That is just extra work putting it in and taking it out when the goal is to put it in and not take it out.

My gut tells me that they did something server side... a sacrifice of something... and they were willing to do that for a week for previous events, but not this time around.

My opinion is that they may have loosened the data bottleneck at the Akamai scrubbing service. It's a well known fact, supported by a lot of tracert documentation that much of the lag originates at the anti-DDoS servers that all game data is routed through. A simple search for Akamai here will bring up all the information you need.

They add hardware as in add more Cyrodil and IC instances during mayhem. And Akamai might be an major issue here to.
But during Mayhem you get lots of more players in Cyrodil who breaks up the huge PvP guild groups who can not get on their favorite server so you don't get the long and large fights between ball groups.
But yes if they could simply get an more expensive server for the main Cyrodil campaigns they would done that is its way cheaper than working for years trying to improve stuff.

But let us not ignore the client in the room, ZoS keeps adding more mounts, styles, outfits and costumes, all who has to be loaded into memory. This is purely an client issue as for the server its just an group of numbers.
In short join the Alki'r dolmen runs and note your FPS, if you get lag here your computer is not up for Cyrodil.
Now I long asked for extra graphic settings for Cyrodil and trials who graphic effect heavy and you need high frame rate.

ZOS_Kevin

LarsS wrote: »

@ZOS_Kevin can we get an update on the date for the new EU/PC server?

No date yet. We are working on prep for the new servers currently and working on getting past Scribes of Fate/ U37 launch first. Once we have a final date to share, we'll let everyone know. But the plan is still aiming for Early April.

ZOS_Kevin

Jaavaa wrote: »

@ZOS_Kevin xbox Eu Servers? Any News here?

No news on this front yet. As Matt outlined, we're working on PC EU getting new hardware in early April, and PlayStation EU should follow very shortly after. However, we'll get this info back that some community members are asking for Xbox dates as well.

Wolfkeks

ZOS_Kevin wrote: »

LarsS wrote: »

@ZOS_Kevin can we get an update on the date for the new EU/PC server?

No date yet. We are working on prep for the new servers currently and working on getting past Scribes of Fate/ U37 launch first. Once we have a final date to share, we'll let everyone know. But the plan is still aiming for Early April.

Thank you for the update Kevin! Might not be the right thread but do you know if there is an update for the sever re-architecture (I think this was what it's called)?

ZOS_Kevin

Wolfkeks wrote: »

Thank you for the update Kevin! Might not be the right thread but do you know if there is an update for the sever re-architecture (I think this was what it's called)?

The team is working on this. But this is also a long and slow process. So there are not any like "milestone" progress points to announce yet. But the best answer right now is work is ongoing.

Wolfkeks

ZOS_Kevin wrote: »

Wolfkeks wrote: »

Thank you for the update Kevin! Might not be the right thread but do you know if there is an update for the sever re-architecture (I think this was what it's called)?

The team is working on this. But this is also a long and slow process. So there are not any like "milestone" progress points to announce yet. But the best answer right now is work is ongoing.

Gotcha! Thank you for the answer Kevin! Appreciated!

LarsS

ZOS_Kevin wrote: »

LarsS wrote: »

@ZOS_Kevin can we get an update on the date for the new EU/PC server?

No date yet. We are working on prep for the new servers currently and working on getting past Scribes of Fate/ U37 launch first. Once we have a final date to share, we'll let everyone know. But the plan is still aiming for Early April.

Thanks for the info!

DirtyDeeds765

Wolfkeks wrote: »

ZOS_Kevin wrote: »

LarsS wrote: »

@ZOS_Kevin can we get an update on the date for the new EU/PC server?

No date yet. We are working on prep for the new servers currently and working on getting past Scribes of Fate/ U37 launch first. Once we have a final date to share, we'll let everyone know. But the plan is still aiming for Early April.

Thank you for the update Kevin! Might not be the right thread but do you know if there is an update for the sever re-architecture (I think this was what it's called)?

If they are to stick to their word, the re-architecturing will begin with the Chapter in June.

From the studio director's letter in December 2022 -

"Our multi-threading work continues, as we announced a month or so back. Just to remind everyone, this is the initiative that will result in better overall server performance. As we already announced, we will be phasing these changes in over the course of 2023, starting with the Chapter in Q2. We’ll keep everyone updated in patch notes going forward."

@DirtyDeeds765

Sun7dance

ZOS_Kevin wrote: »

Wolfkeks wrote: »

Thank you for the update Kevin! Might not be the right thread but do you know if there is an update for the sever re-architecture (I think this was what it's called)?

The team is working on this. But this is also a long and slow process. So there are not any like "milestone" progress points to announce yet. But the best answer right now is work is ongoing.

That's really funny!
At the beginning of 2022 you could still predict exactly that this rework would still happen in the same year.
But now it's suddenly a long-term process without any notable progress so far?
What does that tell us now?

ichibe

What’s the timeframe for XBOX NA?

malistorr

ichibe

Some time in the future. That sounds like a smartas* answer but it is their official answer at this point. Sounds like a bad joke right?

msteentjes

ApoAlaia wrote: »

thejadefalcon wrote: »

ZOS_MattFiror wrote: »

we’re scheduling a maintenance on all EU realms that will occur on February 7.

Will this interfere with the event?

ZOS_MattFiror wrote: »

So the new date for PC EU getting new hardware will be in early April
expect about a 14 hour downtime when the actual cutover happens.

Will *this* interfere with the Anniversary event?

Yes to the first.

The event ends on the 7th @3PM UTC and they usually start maintenance at 9AM UTC so yeah, the last day of the event will be a writeoff for many.

So much so to follow 'we don't want maintenance to interfere with events' with a maintenance that interferes with the event

Doesn't fill me with confidence regarding your second question...

Yeah, probably will, just like we missed the last day of the dark heart of skyrim last year because of server issue's and instead of extending it a day or giving any kind of compensation... as usual nothing, not even an apology. Also the season of the dragons lost a day (at least in EU) because of maintanance.
Makes me wonder what person makes these decisions, you know up front there is an event going on and you still plan maintanance on the last day of the event. bye bye at least two tickets and all the loot. for both events, and again no apology and no compensation.
It would have been easy to extend both events with 1 day or gift everyone the tickets that where earnable on these day's.
I've had it with eso anyway, for different reasons. But i see this happening again.

It's also completeely unacceptable that maintance every other week takes place at the same time on both servers.
In the states NA people should be asleep, but in europe EU server, it's from 10 in the morning till 3 or 4 in the afternoon.
why not schedule the EU maintenance also during the night?

We in europe are being screwed in all possible ways, from the maintenance times to server replacement a year later even though the EU server is the most heavy loaded and has the most issue's.

Just my two cents...

Kaysha

The PC NA servers were older and had to be replaced first. Easy as that. And yes, I play on PC EU (try to).

Elsonso

Kaysha wrote: »

The PC NA servers were older and had to be replaced first. Easy as that. And yes, I play on PC EU (try to).

Seems to me they also said that the EU server was larger, which probably made getting enough hardware more difficult.

Lebkuchen

ZOS_Kevin wrote: »

The team is working on this. But this is also a long and slow process. So there are not any like "milestone" progress points to announce yet. But the best answer right now is work is ongoing.

Maybe if we knew how big the teams are, we could manage our expectations better. Because "working on it, no ETA" really doesn't give us a lot of information. Are there 2 people working on it or more? Is it just 1 unpaid intern?

How big is the team for performance and bugs usually? Is there one?

What will change later this year when they are going to focus more on performance and bugs than ever before? How many people are going to work on this, and what have they been doing until now?

Elsonso

Lebkuchen wrote: »

ZOS_Kevin wrote: »

The team is working on this. But this is also a long and slow process. So there are not any like "milestone" progress points to announce yet. But the best answer right now is work is ongoing.

Maybe if we knew how big the teams are, we could manage our expectations better. Because "working on it, no ETA" really doesn't give us a lot of information. Are there 2 people working on it or more? Is it just 1 unpaid intern?

In-game login screen credits will give you a feel for how large the different teams are. Naturally, you won't find a heading for "SERVER REFRESH INTERN" but you will see all of the behind the scenes departments. Not only for ZOS, but for other companies, like Bethesda and Bethesda Softworks. The credits are long. Really long. They seemingly include everyone. For all I know, they included the name of the guy who walks past the studio on the way to work every day. You'd have to watch them for something that matches the department that manages the live servers. It may be more than one department, and from more than one company, involved in the hardware refresh.

ItsNotLiving

Any word on when Whitestrakes Mayhem is coming?

Aka_

Given how horrible, abysmal even, the performance in Cyrodiil is becoming in PC NA, including Blackreach, I do not believe this is going to be nearly enough for PC EU to notice a difference, but I hope they address this sooner rather than later. It's genuinely breaking down in quality every new day, I swear.

Katheriah

It's almost april... Update time?