Can we please get an update about the residual impacts of the datacenter outage?

Desiato · December 2024

ZOS_MattFiror wrote:

Hey everyone, I wanted to update everyone that we are recovering from the most severe system outage in ten years of ESO. Thank you all for being patient as we recover from this near worst case scenario where one of our datacenters lost all power for the first time ever. We will be dealing with the internal ramifications of this for weeks, but for now, all public live Realms are back online. I want to also extend thanks and gratitude to the many (many!) people at ZOS who have pitched in on 24-hour schedules to right the ship. This work continues, and shows the dedication and extreme pride in ESO that everyone at ZOS holds dear.

https://forums.elderscrollsonline.com/en/discussion/670628/datacenter-follow-up

Many players believe we've been experiencing server performance issues since the datacenter outage referenced above, especially after the event began.

I notice it in PVE instances of all kinds, but especially trials when a block, roll dodge or ability doesn't register. I notice it is common to be stuck in combat for a little longer after a fight ends. There are also freezes and rubberbanding, even in 4 player instances.

I've been pinging the ip address zos provides for troubleshooting gameplay issues (198.20.200.1) and the avg ping is around 50ms with no packet loss or spikes that correspond with the in-game lag I am experiencing.

Have we been experiencing residual effect of the datacenter outage? If so, when can we expect these issues to be resolved?

@ZOS_Kevin

moderatelyfatman · December 2024

I suspect everyone is on holiday for another week or six.

WhiteCoatSyndrome · December 2024

moderatelyfatman wrote: »

I suspect everyone is on holiday for another week or six.

Yup. They’re supposed to be back after New Year.

React · December 2024

Yep, I've been having issues that I have never once experienced in the history of the game since the outage. Namely the game is frequently freezing, then rubberbanding moments later. This same exact problem is happening to a number of other players I've spoken, as well as several content creators (juggy, imperial).

Never had these problems prior to the datacenter outage. Really hope they acknowledge this ASAP when they're back in the office, presumably after new years. The game is as good as unplayable for me.

Here is a thread with some videos of the issue I'm referring to.

https://forums.elderscrollsonline.com/en/discussion/670960/pc-na-constant-game-freezes#latest

Desiato · December 2024

I truly hope ZOS staff are enjoying their vacations.

However, for their customers to better enjoy their time off work, we would very much appreciate an update about clear performance issues that seem be affecting everyone.

Even if it can't be immediately remedied, it's easier for many to deal with if they at least have an understanding of the issue.

OR maybe it has nothing to do with the datacenter issue and there's something that needs to and can be fixed immediately and they just don't know about it yet.

Either way, this issue deserves quick attention.

virtus753 · December 2024

Jessica mentioned that they also took measures in response to “malicious activity” that she said would impact performance, separately from the outage.

https://forums.elderscrollsonline.com/en/discussion/comment/8238380#Comment_8238380

So there is more going on, and they’ve acknowledged it. Hopefully we’ll get some follow up after the holidays.

edward_frigidhands · December 2024

virtus753 wrote: »

Jessica mentioned that they also took measures in response to “malicious activity” that she said would impact performance, separately from the outage.

https://forums.elderscrollsonline.com/en/discussion/comment/8238380#Comment_8238380

So there is more going on, and they’ve acknowledged it. Hopefully we’ll get some follow up after the holidays.

Are these measures still active?

virtus753 · December 2024

edward_frigidhands wrote: »

virtus753 wrote: »

Jessica mentioned that they also took measures in response to “malicious activity” that she said would impact performance, separately from the outage.

https://forums.elderscrollsonline.com/en/discussion/comment/8238380#Comment_8238380

So there is more going on, and they’ve acknowledged it. Hopefully we’ll get some follow up after the holidays.

Are these measures still active?

As I said, that is something they should follow up on when they get back from the holidays, as the measures are either still in place or were disabled without a proper comment. We won’t know without further information. What we do know is that they were put in place and were said to be likely to hurt performance. That’s in addition to any aftereffects from the outage and to the normal stress from an event, which typically hurts performance in very noticeable ways as well.

We also know ZOS currently has lower staffing due to the holidays, which leaves a game and its community more vulnerable to the effects of said malicious activity, so it wouldn’t surprise me if the measures were still in place to try to forestall any further issues until everyone’s back to handle anything that arises.

Desiato · December 2024

Whatever the case is, it would be nice to have the very recent issues acknowledged with assurances they know what it is and how to resolve it.

I feel the performance issues have killed Trials during the holidays. There aren't as many open and group finder trials running as I expected, and I have a presence in a lot of trial communities to give me some perspective. One RL in particular I knew who had planned to run Trials over the holidays decided it's not worth the trouble.

I was in a vCR group that had to call it early last night despite a strong group comp because some players had recurring disconnects while most others had intermittent performance issues like freezing followed by rubber banding.

Then it got worse when I made the mistake of doing a 3 dps vdsa run. Though I knew the other players had an understanding of the performance issues affecting ESO, it was still embarrassing to die to lag a few times, which happened to all four of us.

Thrasher91604 · December 2024

It definitely needs a follow-up by ZoS.

Server performance has been very bad since since the data center shutdown and it's after the recovery.

I am experiencing bad lag in PvE and Cyrodiil areas, as well as a few daily crashes in both areas.

Rkindaleft · December 2024

I just finished watching my partner play in the background and half their trial group had a rubberband at the same time and a couple of them got disconnected afterwards - these issues have been going on for nearly 3 weeks now ever since the server outage.

I’ve posted this in a different thread before but this is exactly the same thing that was happening to us in Dreadsail a week ago, this rubberband affected nearly everyone in the group and a number of people got kicked off after it finished. I understand people may still be on holidays, but we really need some communication about this because any instanced PvE is pretty much unplayable.

https://youtu.be/AdxZsPtXDfY?si=3Wd8SJHligoxKkZ5

Amottica · December 2024

moderatelyfatman wrote: »

I suspect everyone is on holiday for another week or six.

And it is known. It is a pinned message informing us all. I doubt any response we will get this week would be very informative.

silky_soft · December 2024

I don't think we are running on the same servers and network hardware as before. The lag is crazy, it's all over the shop. There is no consistency.

Desiato · December 2024

A member of the vdsa group I mentioned earlier described it best: "I feel like I'm in Cyrodiil!"

This is obviously more than a ddos as there are clear signs of server processing lag and bottlenecks.

Aurielle · December 2024

Desiato wrote: »

A member of the vdsa group I mentioned earlier described it best: "I feel like I'm in Cyrodiil!"

This is obviously more than a ddos as there are clear signs of server processing lag and bottlenecks.

I said those exact words…back in May, when these problems first started happening to me while I was CRAFTING WRITS in Rawl Kha, of all things.

We definitely need some answers… is ESO running on the same servers it was running on pre-outage? What caused the outage in the first place? If ESO is indeed still running on the same servers, are the post-outage latency / disconnect problems related to the same problems a number of us started experiencing after that patch maintenance back in May?

Desiato · January 12

So this is still an issue any time the game seems active like has been the case today. It sure seems like the servers can't handle the activity.

I was in two different trials groups in different trials back to back this evening and the lag was brutal.

I pinged the ip ZOS provides for troubleshooting and there were no indications of a network related issue:

Ping statistics for 198.20.200.1:
    Packets: Sent = 338, Received = 338, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 47ms, Maximum = 91ms, Average = 54ms

When I asked others if they were extremely lagged, most others responded yes.

@ZOS_Kevin Can we please get an update?

TaSheen · January 12

Didn't affect me at all, so can't help you.;

TheMajority · January 12

getting cyrodiil like lag all alone in the infinite archive is weird

Desiato · January 12

TheMajority wrote: »

getting cyrodiil like lag all alone in the infinite archive is weird

Yeah, I've had some deaths in trial instances that were very Cyrodiil-esque. Like my controls became mushy even as other players around me continued to still move, I couldn't bar swap for a few seconds, and then I get the death screen with my character running in spot instead of laying dead. Until recently, I can't recall that happening to me in PVE ESO.

All the while I'm pinging the ESO test IP with normal ping results and no packet loss.

Dungeon instances are normally okay, but there are misc issues with ability responsiveness, but in every vdsa run I've had lately there have been more significant issues that pop up. It's pretty easy so we complete it without issue, but embarrassing to players. Like the tank kept missing taunts which was frustrating, but it was because of the lag.

Desiato · January 13

Mass disconnects seem to be happening atm. Almost my entire trial group has been disconnected twice. I was presented with a 30s queue when I reconnected.

Most group members agreed the lag and instability has been brutal recently.

We deserve some kind of response.

Rkindaleft · January 13

It seemed to be okay for the last week or so but the first 2-3 weeks after the outage were really inconsistent, rubberbanding and disconnections were prevalent and happening at seemingly random times.

And now everyone got kicked.

DragonRacer · January 13

Honestly, the disconnects are just so frustrating. I would get them occasionally - almost always during a ToT match, which is equally frustrating since the game logs that as a loss AND puts you in the deserter timeout queue for something out of your control.

But now it is multiple times within only a few hours of play, and even during the most mundane things such as decorating a house, scrolling through my craftable furnishings list, typing in the social screen, going through my bank inventory, putting up trader listings, or simply riding on the back of my friend's 2-seat mount while collecting skyshards on a baby toon in an old unpopulated base game zone (Shadowfen).

These disconnects are 100% far more increased ever since The Great Datacenter Meltdown.

SerafinaWaterstar · January 13

Would perhaps be helpful if posters stated what server they are on, and if cable or wifi internet.

On PSEU with LAN cabled internet and have had no issues at all - even in Cyrodiil.

Desiato · January 13

SerafinaWaterstar wrote: »

Would perhaps be helpful if posters stated what server they are on, and if cable or wifi internet.

On PSEU with LAN cabled internet and have had no issues at all - even in Cyrodiil.

The datacenter outage refers to a total shutdown of the NA datacenter where the ESO login, account and NA game servers are hosted. Though both regions were affected because the login servers are hosted there, the EU server did not actually go down while the NA server did and was more heavily affected.

Please refer to the OP for the specific quote by Matt where he says "We will be dealing with the internal ramifications of this for weeks".

Since the outage, the PC/NA server seems to be more affected by instability and disconnects, especially during peak times. I seem to recall NA console users describing more problems, but I'm not positive. The basic question I'm asking is if this is because of the "internal ramifications for weeks" Matt referenced.

If you read my posts in this thread, you can see I use troubleshooting tools like ping to try to identify a network related issue. Except for the issue last night which appeared to be more likely a ddos, there hasn't been any network issues I can correlate to the ESO performance issues. There definitely isn't one on my end and I only game on a wired connection.

It is has been well-documented by ZOS that Cyrodiil's performance issues are server related. Several people have noted most the recent performance issue feel like Cyrodiil lag and are unlike anything I had previously experienced in PVE instances. See my other posts in this thread for some details.

It seems like PC/NA either has less server processing capacity or is dealing with internal bottlenecks of some kind.

allochthons · January 13

I wonder if the changes ZoS made in December to mitigate the 'malicious activity' threw more people into the pool being affected by the same long-standing issues mentioned in this thread? ZOS, Massive Spike in Ping/LAG in Recent Days - What Gives

For instance, If the mitigation is anti-DDoS, more network paths could be going through Akamai than before?

SerafinaWaterstar · January 13

@Desiato I am fully aware of the recent outage!

What I was suggesting was for posters here to say what server they are on. Easier to see if it is indeed adversely affecting one more than the others, which you suggest is the case.

DragonRacer · January 13

In my signature, PS5 NA.

My disconnects are way, way increased since the datacenter outage. Yes, I play on WiFi as I do not have an option to be hard-wired given my distance from the modem/router, but my connection to ESO is night and day from playing a LOT before the datacenter outage and TRYING to play the same amount ever since then.

Desiato · January 13

SerafinaWaterstar wrote: »

I am fully aware of the recent outage!

What I was suggesting was for posters here to say what server they are on. Easier to see if it is indeed adversely affecting one more than the others, which you suggest is the case.

@SerafinaWaterstar
I apologize for the defensive tone in my response. I agree, it is a good idea to provide more details.

I also suggest submitting in-game reports while the lag is happening. I don't know if it's true -- it would be logical if it were so -- but I've read that in-game reports include situational technical information that may not be normally captured. It's possible there are issues affecting servers that ZOS is not yet aware of and such reports can help them be identified.

In regards to what someone wrote about Akamai: Akamai tends to be a red herring around here IMO. Though there can be a performance impact to active DDOS mitigation, it will always be better than dealing with unmitigated DDOSes! And Akamai has a good reputation in the industry.

In any case, the performance issues I am referring to don't correlate to any kind of internet latency or packet loss, but there could certainly be internal network bottlenecks within the Datacenter.

Even though I added to this thread last night, in hindsight the mass disconnects last night did bear the signatures of a DDOS and the interruptions correlated with interruptions to my internet connection with the ZOS servers.