ZOS, Massive Spike in Ping/LAG in Recent Days - What Gives?

sarahthes

Jaraal wrote: »

sarahthes wrote: »

JustLovely wrote: »

Aurielle wrote: »

ForumSavant wrote: »

Something is going on with the way individuals are being routed or there is just some account specific problem that is randomly distributed when you log in, no idea what it is though.

DDoS attacks, and possibly something else. It has been confirmed by ZOS, and discussed during the last page or so of replies.

Obscure posts made on Reddit don't count as an official announcement. If it's not on this forum, it's not official.

I mean, the Reddit thread was announced on the forums by Gina, moderated by her, and she's going to be highlighting the more interesting responses and posting them apparently (perhaps only in other languages?)

A lot of us don't use Reddit, or Twitter, or Twitch, etc. Any official information should be posted here, on the official The Elder Scrolls Online website. People shouldn't be expected to join every available social media site to piece together necessary and relevant information.

She posted a link here, apparently. I know it came up in dev tracker for me. To be honest, letting people know there's information and providing a direct link to it is more than sufficient.

ZOS_Kevin

eveboufa wrote: »

@ZOS_Kevin @ZOS_RichLambert @ZOS_GinaBruno

Can you provide your community an official update on its official forums?

An acknowledgment of issues was posted by Rich on Reddit but I don’t see one here. Maybe I just can’t see it?

https://www.reddit.com/r/elderscrollsonline/comments/1d3fs29/comment/l67ekx7/?context=3&share_id=fw5nkA67Mn0RU45jY5aDD&utm_content=1&utm_medium=ios_app&utm_name=ioscss&utm_source=share&utm_term=1

Following up on this. As Rich mentioned during the AMA on Reddit, there are some DDoS issues we have been working through. Specifically Rich said:

"We are most definitely aware. It's been a rough few weeks - DDOS attacks have been a big part of perf issues, but aren't the only issue we're looking into.

However, as he also mentioned, there are some other issue at play that we are actively investigating regarding the performance issues. We are trying to provide our engineers with as much information as possible to narrow down what is occurring, hence the previous ask for ticket numbers. We are working to figure this out, so additional tickets, examples, videos, etc. will help us to narrow down what is happening here. (Thanks @Photosniper89 for all of the tickets sent so far)

Thanks in advance for all the continued help.

Destai

ZOS_Kevin wrote: »

eveboufa wrote: »

@ZOS_Kevin @ZOS_RichLambert @ZOS_GinaBruno

Can you provide your community an official update on its official forums?

An acknowledgment of issues was posted by Rich on Reddit but I don’t see one here. Maybe I just can’t see it?

https://www.reddit.com/r/elderscrollsonline/comments/1d3fs29/comment/l67ekx7/?context=3&share_id=fw5nkA67Mn0RU45jY5aDD&utm_content=1&utm_medium=ios_app&utm_name=ioscss&utm_source=share&utm_term=1

Following up on this. As Rich mentioned during the AMA on Reddit, there are some DDoS issues we have been working through. Specifically Rich said:

"We are most definitely aware. It's been a rough few weeks - DDOS attacks have been a big part of perf issues, but aren't the only issue we're looking into.

However, as he also mentioned, there are some other issue at play that we are actively investigating regarding the performance issues. We are trying to provide our engineers with as much information as possible to narrow down what is occurring, hence the previous ask for ticket numbers. We are working to figure this out, so additional tickets, examples, videos, etc. will help us to narrow down what is happening here. (Thanks @Photosniper89 for all of the tickets sent so far)

Thanks in advance for all the continued help.

I'm glad an explanation was provided somewhere, but I'm a bit disappointed that the threads here didn't warrant a reply sooner.

Two questions:

1. Why were the related threads on the official forums not treated with a higher priority? This thread has been opened since May 7 and now, 23 days later, we get an update.
2. What are the other issues? Surely this is an anticipated question, as you all should know by now that performance concerns are quite visible. Can we get some background on what the other issues are?

Hrafngrimr3

Xbox EU - the lag and performance issues have been degrading the game for some time before updates, around January of this year it started to become more noticeable. Pve content plays like PvP cyrodiil before the server “refresh” about 98% of the time. In the last month several endgame trial players decided to call it quits over performance and move to other games. It’s becoming a dead zone at this rate as more and more players head for exit. Honestly ZOS, at this point I’d recommend taking the server down for a period to actually fix what clearly is an issue that extends beyond a band aid patch fix or just make a whole new base game (like FF14 did). It’s not really worth my money to play anymore.

ZOS_Kevin

Destai wrote: »

Two questions:

1. Why were the related threads on the official forums not treated with a higher priority? This thread has been opened since May 7 and now, 23 days later, we get an update.
2. What are the other issues? Surely this is an anticipated question, as you all should know by now that performance concerns are quite visible. Can we get some background on what the other issues are?

1. You're assuming this was not being taken seriously. As Rich mentioned, we have been dealing with this for the last few weeks. Also to note, we normally do not comment on DDoS attacks. Rich felt it appropriate to mention as it came up during the AMA, but we've been tracking this thread for awhile now.
2. Again as stated earlier, we are still investigating. I'm going into another meeting about this shortly to see what other information could be provided to assist in resolving this issue. Anything provided now would be a guess at best. Once we have a better idea of what else could be causing issues, we'll communicate that.

manukartofanu

[snip]

Have you ever considered the resources needed to launch a DDoS attack against such a large company? This isn’t a tiny indie project with just a few servers; we're talking about a massive infrastructure with dozens of servers capable of handling hundreds of thousands of users simultaneously. The login servers are separate from those that manage game sessions. Yet, we're not seeing issues at login, where DDoS attacks would most likely occur, but rather during gameplay. Moreover, these problems have been ongoing for weeks, and the longer a DDoS continues, the more expensive it becomes—costing millions, not just thousands of dollars. It’s laughable to read about some suggesting that disgruntled players could orchestrate such a DDoS to disrupt others.

[snip]
[edited for bashing]

Tandor

The best thing they could do now is delay the chapter launch. Putting it out now while players are experiencing game-breaking connection and performance issues which whole teams are dedicated to working on is simply asking for trouble, and I can't off-hand think of any previous update that wouldn't have benefitted from some additional time on the PTS.

Aurielle

JustLovely wrote: »

Aurielle wrote: »

ForumSavant wrote: »

Something is going on with the way individuals are being routed or there is just some account specific problem that is randomly distributed when you log in, no idea what it is though.

DDoS attacks, and possibly something else. It has been confirmed by ZOS, and discussed during the last page or so of replies.

Obscure posts made on Reddit don't count as an official announcement. If it's not on this forum, it's not official. Every social media platform is a cesspool of misinformation and many of us will never read anything posted on any social media platform.

Uh…

It’s not an obscure post on Reddit. It’s a response from Rich Lambert on the official ZOS ask-us-anything Reddit post…

Edit: Furthermore, is this official enough for you?

ZOS_Kevin wrote: »

eveboufa wrote: »

@ZOS_Kevin @ZOS_RichLambert @ZOS_GinaBruno

Can you provide your community an official update on its official forums?

An acknowledgment of issues was posted by Rich on Reddit but I don’t see one here. Maybe I just can’t see it?

https://www.reddit.com/r/elderscrollsonline/comments/1d3fs29/comment/l67ekx7/?context=3&share_id=fw5nkA67Mn0RU45jY5aDD&utm_content=1&utm_medium=ios_app&utm_name=ioscss&utm_source=share&utm_term=1

Following up on this. As Rich mentioned during the AMA on Reddit, there are some DDoS issues we have been working through. Specifically Rich said:

"We are most definitely aware. It's been a rough few weeks - DDOS attacks have been a big part of perf issues, but aren't the only issue we're looking into.

However, as he also mentioned, there are some other issue at play that we are actively investigating regarding the performance issues. We are trying to provide our engineers with as much information as possible to narrow down what is occurring, hence the previous ask for ticket numbers. We are working to figure this out, so additional tickets, examples, videos, etc. will help us to narrow down what is happening here. (Thanks @Photosniper89 for all of the tickets sent so far)

Thanks in advance for all the continued help.

vsrs_au

Yamenstein wrote: »

Quackery wrote: »

WhiteCoatSyndrome wrote: »

I’ve had the game crashing my desktop a few times over the last week or so. 😕 PC/NA

Be glad that the game doesn't freeze your pc on a black screen that then gives you no option but to force restart. This is the kind of stuff that destroys a PC.

What the! This happened to me today while I was playing! Just finished running Dragon Star Arena, going up the funnel of light at the end then bam! Black screen. Had to hard restart to get everything working again.

You mean you couldn't even run Task Manager to force-kill the game? You had to turn off your PC?

Aurielle

manukartofanu wrote: »

[snip]

Have you ever considered the resources needed to launch a DDoS attack against such a large company? This isn’t a tiny indie project with just a few servers; we're talking about a massive infrastructure with dozens of servers capable of handling hundreds of thousands of users simultaneously. The login servers are separate from those that manage game sessions. Yet, we're not seeing issues at login, where DDoS attacks would most likely occur, but rather during gameplay. Moreover, these problems have been ongoing for weeks, and the longer a DDoS continues, the more expensive it becomes—costing millions, not just thousands of dollars. It’s laughable to read about some suggesting that disgruntled players could orchestrate such a DDoS to disrupt others.

[snip]

Keep in mind that DDOS attacks are not SOLELY to blame here — Rich states they’re actively looking at other issues as well. DDOS attacks are simply part of it. It’s not “gullible” to believe an official statement when there is sufficient evidence supporting the statement. People ARE having log in issues in addition to gameplay issues.

[edited to remove quote]

Izanagi.Xiiib16_ESO

ZOS_Kevin wrote: »

eveboufa wrote: »

@ZOS_Kevin @ZOS_RichLambert @ZOS_GinaBruno

Can you provide your community an official update on its official forums?

An acknowledgment of issues was posted by Rich on Reddit but I don’t see one here. Maybe I just can’t see it?

https://www.reddit.com/r/elderscrollsonline/comments/1d3fs29/comment/l67ekx7/?context=3&share_id=fw5nkA67Mn0RU45jY5aDD&utm_content=1&utm_medium=ios_app&utm_name=ioscss&utm_source=share&utm_term=1

Following up on this. As Rich mentioned during the AMA on Reddit, there are some DDoS issues we have been working through. Specifically Rich said:

"We are most definitely aware. It's been a rough few weeks - DDOS attacks have been a big part of perf issues, but aren't the only issue we're looking into.

However, as he also mentioned, there are some other issue at play that we are actively investigating regarding the performance issues. We are trying to provide our engineers with as much information as possible to narrow down what is occurring, hence the previous ask for ticket numbers. We are working to figure this out, so additional tickets, examples, videos, etc. will help us to narrow down what is happening here. (Thanks @Photosniper89 for all of the tickets sent so far)

Thanks in advance for all the continued help.

@ZOS_Kevin if it helps changing DNS has seemed to help a number of people. Generally going to 8.8.8.8 or 1.1.1.1 has almost solve all the 'crashes' just the slower general performance remains. A lot of this is quite commonly also seen when PTS is up on PCNA - I wonder if the engineers could check if the PTS servers are stealing resources from Live?

ZOS_Kevin

Izanagi.Xiiib16_ESO wrote: »

@ZOS_Kevin if it helps changing DNS has seemed to help a number of people. Generally going to 8.8.8.8 or 1.1.1.1 has almost solve all the 'crashes' just the slower general performance remains. A lot of this is quite commonly also seen when PTS is up on PCNA - I wonder if the engineers could check if the PTS servers are stealing resources from Live?

Thanks for this bit of feedback. Will pass this along now for the engineers.

Seravi

I've sent tickets in as well, quite a few. It was so bad today I couldn't even send a ticket in during an issue. I listed them in my posts. I have a 2-3ms route to the game. There has to be something going on once I am connected that bottlenecks my traffic. Is it a dDos attack, hardware or tired hamsters, who knows. I can say with 100 % certainty it isn't me.

WIZZARD2K

ZOS_Kevin wrote: »

However, as he also mentioned, there are some other issue at play that we are actively investigating regarding the performance issues. We are trying to provide our engineers with as much information as possible to narrow down what is occurring, hence the previous ask for ticket numbers. We are working to figure this out, so additional tickets, examples, videos, etc. will help us to narrow down what is happening here. (Thanks @Photosniper89 for all of the tickets sent so far)

Thanks in advance for all the continued help.

I didn't make a ticket, but here's a video I recorded in glenumbra yesterday at limbscather. was experiencing some severe server-side spikes, as the ping didn't seem affected, and after one kill, my arcanist got stuck like this. on one bar, the book seemed to be in the ground, but on the other bar, it was bouncing as seen in the video. killing another wb didn't help, nor did reloadui. only porting away fixed it. later that evening I was experiencing extra-heavy ping (3400+) doing some light alchemy farming in alik'r around the survey point.

https://youtu.be/vyAHeaNAFTc

BetweenMidgets

Tickets from yesterday:
240530-000069
240529-005821
240529-005744

Jaraal

Aurielle wrote: »

People ARE having log in issues in addition to gameplay issues.

Yeah, about 50% of the time when I make it past the black splash screen and get the Elven/Breton/Nord Hero or the Gold Coast loading screen, the game freezes, screen goes gray, and the Windows working icon runs forever (AKA: app hang), and I have to force quit the game. However, it works 100% of the time (so far) on the second attempt. And this has only been happening since the emergency maintenance after the inadvertent PTS/Live merger.

manukartofanu

Aurielle wrote: »

Keep in mind that DDOS attacks are not SOLELY to blame here — Rich states they’re actively looking at other issues as well. DDOS attacks are simply part of it. It’s not “gullible” to believe an official statement when there is sufficient evidence supporting the statement. People ARE having log in issues in addition to gameplay issues.

I know, that's why I'm not saying he lied. And he really didn't just blame the DDoS attacks for the current problems. It's just that somehow most people forgot about this when they came to the forum, and are only blaming the DDoS attacks. Yet, he didn't even say that DDoS attacks are the main issue. It's just the part that he can at least mention.

ZOS_Kevin

WIZZARD2K wrote: »

ZOS_Kevin wrote: »

However, as he also mentioned, there are some other issue at play that we are actively investigating regarding the performance issues. We are trying to provide our engineers with as much information as possible to narrow down what is occurring, hence the previous ask for ticket numbers. We are working to figure this out, so additional tickets, examples, videos, etc. will help us to narrow down what is happening here. (Thanks @Photosniper89 for all of the tickets sent so far)

Thanks in advance for all the continued help.

I didn't make a ticket, but here's a video I recorded in glenumbra yesterday at limbscather. was experiencing some severe server-side spikes, as the ping didn't seem affected, and after one kill, my arcanist got stuck like this. on one bar, the book seemed to be in the ground, but on the other bar, it was bouncing as seen in the video. killing another wb didn't help, nor did reloadui. only porting away fixed it. later that evening I was experiencing extra-heavy ping (3400+) doing some light alchemy farming in alik'r around the survey point.

https://youtu.be/vyAHeaNAFTc

Thanks. Also shared this video in our report.

ZOS_Kevin

BetweenMidgets wrote: »

Tickets from yesterday:
240530-000069
240529-005821
240529-005744

Thanks, added these tickets to the report as well.

thinkaboutit

I have the following.

240528-007947
240529-005820

irswat

Most of my raid team made tickets last night. Interestingly the rubber banding and lag spike affected people at the exact same times.

Destai

ZOS_Kevin wrote: »

Destai wrote: »

Two questions:

1. Why were the related threads on the official forums not treated with a higher priority? This thread has been opened since May 7 and now, 23 days later, we get an update.
2. What are the other issues? Surely this is an anticipated question, as you all should know by now that performance concerns are quite visible. Can we get some background on what the other issues are?

1. You're assuming this was not being taken seriously. As Rich mentioned, we have been dealing with this for the last few weeks. Also to note, we normally do not comment on DDoS attacks. Rich felt it appropriate to mention as it came up during the AMA, but we've been tracking this thread for awhile now.
2. Again as stated earlier, we are still investigating. I'm going into another meeting about this shortly to see what other information could be provided to assist in resolving this issue. Anything provided now would be a guess at best. Once we have a better idea of what else could be causing issues, we'll communicate that.

No, you misunderstood my point. I’m not assuming anything. This isn’t about what the technical teams take seriously, it’s about what’s conveyed back to us. Given your aforementioned policy on DDOS announcements, it wasn't a priority to tell us until Rich mentioned it on the AUA.

You guys expect perfect trust but then there's some internal policy preventing us from getting updates on the very official channels that people are asking. We have no way of knowing what you guys track if you don't tell us. And if the issue is ongoing, and there's already longstanding issues, what are we to believe?

Do you think it's fair to us to go 3 weeks without an update, only for it to be on Reddit? That doesn't seem fair, especially when there's other issues, and especially when there's a longstanding request for the forums to come first because they're an official source.

I hope you guys can get it figured out, I am sure it’s frustrating to deal with.

bloodandiamonds

Well I'm glad I've held off on buying Gold Road. I can barely make it through doing daily crafting.

BardInSolitude

Heard y'all devs are looking for video evidence of this particular issue? I have some footage from a couple of months ago that showcases the same issue: https://www.youtube.com/watch?v=4E_amloO0xg

This really is not a plug, it's literally all I could find in my records. On this video, you see a server-wide incidence (i.e. every single player in the entire server, as far as we know, experienced this), whereas this Monday we had incidents that were experienced by some players but not others, and at different times. Another slight difference is that on the video my character is able to move around during the incident, whereas the more recent issue we've had involves everything coming to a complete stop. Unfortunately I don't have footage from our escapades on Monday, so hopefully the video I posted gives y'all an idea of what the issue looks like.

KaosWarMonk

KaosWarMonk wrote: »

FWIW - to PCNA from AU, was in the IC about 16 hours ago, had lag spikes every few minutes. Didn't get booted but I only stayed long enough to finish endeavours.

Again, FWIW, haven't seen the same problem in the two sessions I've played since the above.

akdave0

Xbox NA last night, numerous guild discords blew up at raid time. Rubberbanding, dashboards, 307 errors, desync issues made the game unplayable and many teams are now on hold for running 12 man content until resolution. Fix your game, now.

twev

Photosniper89 wrote: »

twev wrote: »

Photosniper89 wrote: »

ALL - as @ZOS_Kevin has asked please submit tickets in game regarding the issues you're experiencing and then drop the ticket numbers in here. They are tracking the thread and trying to resolve. Flood them with as much evidence as possible to assist.

https://help.elderscrollsonline.com/#en/home/category/861/feedback/866

otoh - posting ticket numbers for the complaint department to notice implies preferential treatment, while anyone who doesn't post a ticket number gets pushed further towards the back of the line while the 'posted ticket numbers' are given priority.

I think internally they are wanting to prioritize this issue due to the impact it's having on the game as a whole so posting numbers here helps them consolidate ones that are specific to this issue.. vs going one by one through all the random ticket sections to try to pin point ones that are related to this. I highly doubt every single support team member is only working on tickets posted in this thread.

For starters - They should prioritize manpower for the issue - with additional staff.
I'm not disagreeing that it's a priority issue, I'm simply stating that they need at least a few additional people on it, since the issue has ballooned as it has.
I'm not averse to 'seasoned' staff handling the prioritized issue, with some fill-in temps handling the other/lower problem issues (everyone has to get XP somewhere) but notice the number of service tickets in just one day between a single player's first ticket and forth third ticket:

Tickets submitted today in game so far in post #269 in this thread:

240529-002379
240529-002857
240529-003331
Edited by Seravi on May 29, 2024 2:01PM

Thats almost 1,000 tickets in just one day, that one single person has record of, with no info available as to how many were posted afterwards, or timestamps between an assumed -000001 and Seravi's last ticket, but Seravi edited it at 2:01PM, probably to add the info for the 3rd ticket.
I'd also be disinclined to think/believe that the ticket numbers issued were randomly numbered or non-sequential.

OTOH - I'd have to make a bold assumption to point out that the tickets date coded for: 240529 probably didn't start at '-002378', but I don't think it's a huge leap to believe they didn't start at '-002378 for that date code.
If the tickets are coded to start on May first at '-000001, and each day started at the previous day's last number +1, that implies an average of 84+ tickets filed every day until the 29th, which is hard to believe.

But an almost 1,000 ticket spread for a presumed single day might be indicative of something substantial.

No, I don't 'KNOW', but I know what I believe to be the case.

Photosniper89

ZOS_Kevin wrote: »

eveboufa wrote: »

@ZOS_Kevin @ZOS_RichLambert @ZOS_GinaBruno

Can you provide your community an official update on its official forums?

An acknowledgment of issues was posted by Rich on Reddit but I don’t see one here. Maybe I just can’t see it?

https://www.reddit.com/r/elderscrollsonline/comments/1d3fs29/comment/l67ekx7/?context=3&share_id=fw5nkA67Mn0RU45jY5aDD&utm_content=1&utm_medium=ios_app&utm_name=ioscss&utm_source=share&utm_term=1

Following up on this. As Rich mentioned during the AMA on Reddit, there are some DDoS issues we have been working through. Specifically Rich said:

"We are most definitely aware. It's been a rough few weeks - DDOS attacks have been a big part of perf issues, but aren't the only issue we're looking into.

However, as he also mentioned, there are some other issue at play that we are actively investigating regarding the performance issues. We are trying to provide our engineers with as much information as possible to narrow down what is occurring, hence the previous ask for ticket numbers. We are working to figure this out, so additional tickets, examples, videos, etc. will help us to narrow down what is happening here. (Thanks @Photosniper89 for all of the tickets sent so far)

Thanks in advance for all the continued help.

Greatly appreciate the communication in this thread since the first post you've made @ZOS_Kevin. I know people love to throw shade (me included) at zos coms but I have nothing but positive things to say about the zos coms throughout this thread.

For those of you saying well they gave a half answer or vague response... we bicker when they wait weeks to find out the answer and solution before they say something to us and we complain when they jump in early and don't give us a 100% answer... gotta give them some slack.

Logging on now and will see if I can get some more documentation. Does anyone know if there is any kind of logging software that could be used during the game to see if packets or something are being dropped or what that traffic looks like in general?

Photosniper89

Anywaysssssssssss....

I'm noticing a big drop in CPU usage right around the time the lag spikes happen. Not sure if its just because the game stops so the cpu is like what do I do fam? or if it's something causing cpu to tank which in turn makes the game do weird stuff.

Wuduwasa13

Also, yes, the lag & dc’ing recently has been ridiculous.