ZOS, Massive Spike in Ping/LAG in Recent Days - What Gives?

LadyGP · January 13

SeveN085 wrote: »

Man... If I was working in ZOS I'd legit feel embarrassed by this .... can you imagine going to WoW or FF14 forums and seeing a 104 pages threads about server issues? I surely do not. I wonder what kind of impression this leaves on a potential new players when they want to check the forums for some info and they see this thread. Not like it bothers me much, since I don't play this anymore, but still check the forums occasionally for the state of the game and seeing that this problem and thread are still present is really funny(or really unfunny if I was to still play this game).

TBH the barrier to entry with the forums is difficult - I doubt many "new players" come to the forums.

But, I see your point.

LadyGP · January 13

Arunei wrote: »

show previous quotes

ZOS_Kevin wrote: »

SaintJohnHM wrote: »

almost everyone in our trials got kicked tonight.

@ZOS_Kevin Why does ZOS think this is ok to not at least address the issue of ongoing terrible service?

Just wanted to follow up here. We posted an update on the Dev Tracker early this morning. Copied below. This update happened around 2am ET today.

ZOS_Kevin wrote: »

Just following up here. We've noticed a spike in malicious activity targeting our servers. So we made some adjustments to counter this. You may experience decreased game performance while these adjustments are in place.

We’ll continue to monitor and adjust as needed. Thanks for your patience.

@ZOS_Kevin
It hit ALL the servers? In the thread I was posting in I saw people on console having the same issues with getting booted, getting the login queue, and so on.

Do you guys have any way of tracking these attacks? The fact that they keep happening in such a short amount of time is...weird. Maybe it's the same group?

I'm sure they do but it's one of those things.. it's gonna happen. The Christmas day attacks on Xbox and Sony a few years back were insane... and if those companies with basically infinite resources can't keep it from happening I don't expect ZoS to keep it from happening.

This one, I'll give them a break on. Bad actors gonna bad act.

SaintJohnHM · January 14

A few folks got DC'd from our trial trifecta prog again tonight.

@ZOS_Kevin is this more 'DDOS' attacks, or is this the months long general ongoing problem with multiplayer DCs that ZOS has been ignoring?

Renato90085 · January 14

i solo IA and got kick, same time i see some stream got kick too

edward_frigidhands · January 14

ZOS_Kevin wrote: »

show previous quotes

SaintJohnHM wrote: »

almost everyone in our trials got kicked tonight.

@ZOS_Kevin Why does ZOS think this is ok to not at least address the issue of ongoing terrible service?

Just wanted to follow up here. We posted an update on the Dev Tracker early this morning. Copied below. This update happened around 2am ET today.
show previous quotes

ZOS_Kevin wrote: »

Just following up here. We've noticed a spike in malicious activity targeting our servers. So we made some adjustments to counter this. You may experience decreased game performance while these adjustments are in place.

We’ll continue to monitor and adjust as needed. Thanks for your patience.

Hello. Has normal operation resumed at this time?

Seravi · January 14

Lots of lag and 999+ in Alik'r Desert at Hoonding's Watch wayshrine area, 12:49 PM CST PC NA.

Not doing much. Simply ported there to chat with Azandar about his quest. Took several minutes for the game to allow me to target him and hit E.

Rohamad_Ali · January 14

I want to sub and play the game but I can not justify the expense when I wait in que for 20 minutes and then get server kicked from lag the first fight in. I can't. This needs fixing.

LadyGP · January 14

Rohamad_Ali wrote: »

I want to sub and play the game but I can not justify the expense when I wait in que for 20 minutes and then get server kicked from lag the first fight in. I can't. This needs fixing.

Welcome to ESO. Hopefully their upcoming Cyro test will help resolve this issue - I understnad your frustration.

ZOS_Kevin · January 14

Just wanted to follow up here. We just had Studio Director, Matt Firor post a community update about some of the DDoS attacks that have been happening. Link to statement is here.

LadyGP · January 14

ZOS_Kevin wrote: »

Just wanted to follow up here. We just had Studio Director, Matt Firor post a community update about some of the DDoS attacks that have been happening. Link to statement is here.

Appreciate the update and transparency! Keep it up

M33tr3 · January 15

ZOS_Kevin wrote: »

Just wanted to follow up here. We just had Studio Director, Matt Firor post a community update about some of the DDoS attacks that have been happening. Link to statement is here.

so in a nutshell, to combat ddos attacks you basicly ddos your customers?!?!?

sshogrin · January 15

YandereGirlfriend wrote: »

show previous quotes

LadyGP wrote: »

Sluggy wrote: »

manukartofanu wrote: »

edward_frigidhands wrote: »

Did we ever get a definitive answer regarding the use of Bandits UI?

Have people experimented with disabling it or add-ons in general to see if that helps with the issue in it's current state?

The addon developer has removed all pings from it. There's nothing to look for or respond to anymore.
If you don't believe me, you can find all the lines with PingMap in the addon; they are all commented out.

As for other addons, yes, it has been mentioned hundreds of times here that disabling them doesn't make any difference. The same issue occurs on consoles, which rules out even the theoretical possibility of addons having any impact.

To further hammer that second point home - if this is an actual issue with the API then it doesn't matter if BanditsUI disabled it or not. Some bad actor(s) will inevitably create their own addons to specifically abuse this loophole. I would hope that ZoS would be aware of such issues and immediately take action if they really did exist.

I'm sure they are having interal conversations about their API as a whole since Bandits was brought up - to make sure there aren't any other methods of sending information/causing issues like Bandits may have been doing.

Bandits most definitely is/was an issue. You can basically mark that one down as 100% confirmed.

Since raiders in my guild have purged it from their mod folders (from a high point of about 3/4 of the raid running it) the incidence of random kicks from Cyro from has decreased from several dozen per raid to virtually zero. Absolutely night and day difference.

Also fully agree that industrious minds will be hard at work to discover and implement new workarounds to the map pings method and that those should be eliminated/heavily regulated as well. The free-for-all approach to mods leaves WAY too much latitude for developers to, knowingly or not, release sketchy products.

Bandit's isn't "100% the issue"...explain why console players have experienced all this, and at the same rate as players on PC. As a matter of fact, since the data center issue, console players have seen a rise in getting kicked, not being able to login etc. They don't have addons.
As an addon developer, I have a better insight to what's going on. Update 45 is going to have a change in how MapPing works (which is a built in function to the base game). It will limit how often MapPing can be done.
One of the problems with data sharing is that there are multiple addons that share data like that. If you have multiple addons that you have turned sharing on in, they will cause conflicts and basically spam MapPing, that's why there's a Library that should be used, but since some developers don't want to integrate the use of that Library to manage it all, there is a problem
My experience is that I have run in trial groups where not one person ran Bandits and we've had half the group get kicked out of game, then other runs no issue what so ever. I've been in runs with people that Bandits was mixed, some had it, others didn't, and again some runs people got kicked, some runs absolutely no issue. I've been in runs where everybody had Bandits going and absolutely no body got kicked, and some runs were people got kicked. If it's Bandits, then explain how any of that points to Bandits. Not one person has been able to explain it. Better yet is a lot of people that are blaming Bandits don't actually write addons and don't know the API and what can and can't be done.

TaSheen · January 15

Huh. Y'all should move to satellite connect. I never have any of these issues - even in the middle of a ddos.

sshogrin · January 15

sarahthes wrote: »

show previous quotes

LadyGP wrote: »

Sluggy wrote: »

manukartofanu wrote: »

edward_frigidhands wrote: »

Did we ever get a definitive answer regarding the use of Bandits UI?

Have people experimented with disabling it or add-ons in general to see if that helps with the issue in it's current state?

The addon developer has removed all pings from it. There's nothing to look for or respond to anymore.
If you don't believe me, you can find all the lines with PingMap in the addon; they are all commented out.

As for other addons, yes, it has been mentioned hundreds of times here that disabling them doesn't make any difference. The same issue occurs on consoles, which rules out even the theoretical possibility of addons having any impact.

To further hammer that second point home - if this is an actual issue with the API then it doesn't matter if BanditsUI disabled it or not. Some bad actor(s) will inevitably create their own addons to specifically abuse this loophole. I would hope that ZoS would be aware of such issues and immediately take action if they really did exist.

I'm sure they are having interal conversations about their API as a whole since Bandits was brought up - to make sure there aren't any other methods of sending information/causing issues like Bandits may have been doing.

It is my understanding from talking to some add-on devs who are in communication with the ZOS devs who manage the API, that solutions are being worked on. It sounds like the current functionality for data transfer will be completely removed, i.e. Hodors and Bandits and other add-ons that share ultimate and DPS information will no longer work. You can look at the lib add-ons for Hodor for example in esoui for evidence.

I have also been told that zos is working with the add-on devs to find an alternate solution so PC raid leads don't have to go back to timing ultimates with stop watches.

There is going to be 2 other Libraries that will address the MapPing timing. You won't be able to make calls every second like it currently is, it will be more on a 10 second call instead, which will kind of mess things up.
The biggest problem is that some addon developers didn't convert to LibMapPing to make these calls, which would have made that date share work with all addons and make a single call for the same type of data. Instead developers had to try and use their own call with MapPing which ends up "spamming" the same data type.
This is not the entire issue though, there are other issues at hand.

sshogrin · January 15

Sluggy wrote: »

show previous quotes

sarahthes wrote: »

LadyGP wrote: »

Sluggy wrote: »

manukartofanu wrote: »

edward_frigidhands wrote: »

Did we ever get a definitive answer regarding the use of Bandits UI?

Have people experimented with disabling it or add-ons in general to see if that helps with the issue in it's current state?

The addon developer has removed all pings from it. There's nothing to look for or respond to anymore.
If you don't believe me, you can find all the lines with PingMap in the addon; they are all commented out.

As for other addons, yes, it has been mentioned hundreds of times here that disabling them doesn't make any difference. The same issue occurs on consoles, which rules out even the theoretical possibility of addons having any impact.

To further hammer that second point home - if this is an actual issue with the API then it doesn't matter if BanditsUI disabled it or not. Some bad actor(s) will inevitably create their own addons to specifically abuse this loophole. I would hope that ZoS would be aware of such issues and immediately take action if they really did exist.

I'm sure they are having interal conversations about their API as a whole since Bandits was brought up - to make sure there aren't any other methods of sending information/causing issues like Bandits may have been doing.

It is my understanding from talking to some add-on devs who are in communication with the ZOS devs who manage the API, that solutions are being worked on. It sounds like the current functionality for data transfer will be completely removed, i.e. Hodors and Bandits and other add-ons that share ultimate and DPS information will no longer work. You can look at the lib add-ons for Hodor for example in esoui for evidence.

I have also been told that zos is working with the add-on devs to find an alternate solution so PC raid leads don't have to go back to timing ultimates with stop watches.

Honestly that's great new to hear. Slightly concerning that such a hole was opened up in there system for so long without them realizing it. But still good to hear that they are addressing it. Personally, I don't really see the need for an addon to time stuff like that. It's not like they aren't in chat already. They could... couldn't they... would it be possible to... *talk* to each other?

MapPing is actually part of the base game...it's not a "hole" as you might think.
As far as being on coms together, you run into issues with a bunch of people trying to call out their own Ultis being ready, plus callouts, plus all the other conversations happening during a trial.
The bigger problem is people downloading multiple data sharing addons and thinking they have to turn on data sharing in every addon they have. The other problem is addon developers not using data sharing Libraries to handle those functions. LibMapPing handles all those calls and makes only one call for the same data type, like dps share, or ultimate share. When you have multiple addons making their own call for dps share and not using the Library, you have that many addons making the same exact call.
I raid lead many times weekly, and you'd be surprised how much chatter there is in voice, and you can try to keep people from talking about things that have zero to do with the trial, but it's going to happen. If you are going for trifecta trial runs, you need to be able to time things correctly. If you're looking at some stopwatch and trying to start stop it and depending on people hitting their ultis exactly when you make the call...it's not going to happen.

sshogrin · January 15

KaironBlackbard wrote: »

show previous quotes

sarahthes wrote: »

I have also been told that zos is working with the add-on devs to find an alternate solution so PC raid leads don't have to go back to timing ultimates with stop watches.

Here's an idea, everyone use discord since most guilds make a discord server, and everyone stream to the voice channel. Raid lead has an extra monitor to show all the streams so he can see all their ults. Probably on a TV screen so they can see them all.
Wouldn't really need stopwatches for that. Also, stopwatches can be inaccurate, especially if someone isn't proccing ult gen as often as the lead is timing it. Or if someone is running ult gen setup and forgot to mention that to lead, having their ult more often than the lead times it.

Because then the raid lead wouldn't be participating in the trial...you can't watch 11 other streams and see all that.
I raid lead many times a week...what you're suggesting is impossible. Get into a vet trial sometime, maybe you'll understand how much different things are than simple normal trials or some normal dungeon.

Rohamad_Ali · January 15

TaSheen wrote: »

Huh. Y'all should move to satellite connect. I never have any of these issues - even in the middle of a ddos.

Are you running starlink?

code65536 · January 15

sshogrin wrote: »

show previous quotes

sarahthes wrote: »

LadyGP wrote: »

Sluggy wrote: »

manukartofanu wrote: »

edward_frigidhands wrote: »

Did we ever get a definitive answer regarding the use of Bandits UI?

Have people experimented with disabling it or add-ons in general to see if that helps with the issue in it's current state?

The addon developer has removed all pings from it. There's nothing to look for or respond to anymore.
If you don't believe me, you can find all the lines with PingMap in the addon; they are all commented out.

As for other addons, yes, it has been mentioned hundreds of times here that disabling them doesn't make any difference. The same issue occurs on consoles, which rules out even the theoretical possibility of addons having any impact.

To further hammer that second point home - if this is an actual issue with the API then it doesn't matter if BanditsUI disabled it or not. Some bad actor(s) will inevitably create their own addons to specifically abuse this loophole. I would hope that ZoS would be aware of such issues and immediately take action if they really did exist.

I'm sure they are having interal conversations about their API as a whole since Bandits was brought up - to make sure there aren't any other methods of sending information/causing issues like Bandits may have been doing.

It is my understanding from talking to some add-on devs who are in communication with the ZOS devs who manage the API, that solutions are being worked on. It sounds like the current functionality for data transfer will be completely removed, i.e. Hodors and Bandits and other add-ons that share ultimate and DPS information will no longer work. You can look at the lib add-ons for Hodor for example in esoui for evidence.

I have also been told that zos is working with the add-on devs to find an alternate solution so PC raid leads don't have to go back to timing ultimates with stop watches.

There is going to be 2 other Libraries that will address the MapPing timing. You won't be able to make calls every second like it currently is, it will be more on a 10 second call instead, which will kind of mess things up.
The biggest problem is that some addon developers didn't convert to LibMapPing to make these calls, which would have made that date share work with all addons and make a single call for the same type of data. Instead developers had to try and use their own call with MapPing which ends up "spamming" the same data type.
This is not the entire issue though, there are other issues at hand.

Read the U45 API patch notes posted at ESOUI. There's an entirely new data broadcast API specifically for use by addons, the specifications of which you can check out.

Just_Attivi · January 15

"An error has occurred, ..."
This is my favorite bit of Lore from the game, I am glad it shows this more than any other cinematic, and that its unskippable, too!

Just_Attivi · January 15

"Youve been placed in a queue to login"

my second favorite piece of lore.... servers dying again? another " " DDoS" " perhaps?

DragonRacer · January 15

Just_Attivi wrote: »

"Youve been placed in a queue to login"

my second favorite piece of lore.... servers dying again? another " " DDoS" " perhaps?

And why is 9:15am Eastern on a Wednesday considered a great time to DDoS the NA server? Vast majority of North America is either at work or just waking up to get ready for work, depending on time zone… and yet, there we were, a massive server boot of the population then a login queue before the server decided we were allowed back in again…

Just_Attivi · January 15

DragonRacer wrote: »

show previous quotes

Just_Attivi wrote: »

"Youve been placed in a queue to login"

my second favorite piece of lore.... servers dying again? another " " DDoS" " perhaps?

And why is 9:15am Eastern on a Wednesday considered a great time to DDoS the NA server? Vast majority of North America is either at work or just waking up to get ready for work, depending on time zone… and yet, there we were, a massive server boot of the population then a login queue before the server decided we were allowed back in again…

very mysterious... must be DDOS

sarahthes · January 15

Just_Attivi wrote: »

show previous quotes

DragonRacer wrote: »

Just_Attivi wrote: »

"Youve been placed in a queue to login"

my second favorite piece of lore.... servers dying again? another " " DDoS" " perhaps?

And why is 9:15am Eastern on a Wednesday considered a great time to DDoS the NA server? Vast majority of North America is either at work or just waking up to get ready for work, depending on time zone… and yet, there we were, a massive server boot of the population then a login queue before the server decided we were allowed back in again…

very mysterious... must be DDOS

DDOS has a specific feel to it and the issues noted recently in ESO definitely feel like that. There's no reason to disbelieve them.

I remember a community hosted server for a survival game I used to play was once ddosed for 4 weeks straight by a single disgruntled banned former player. Nobody could even log in.

LadyGP · January 15

TaSheen wrote: »

Huh. Y'all should move to satellite connect. I never have any of these issues - even in the middle of a ddos.

I've tried 3 ISPs - one of them being Starlink - and I've had the issues still that have been explained in this thread. The ONLY time I can get the issues to go away (somewhat never 100%) is when I use a VPN.

sshogrin · January 15

code65536 wrote: »

show previous quotes

sshogrin wrote: »

sarahthes wrote: »

LadyGP wrote: »

Sluggy wrote: »

manukartofanu wrote: »

edward_frigidhands wrote: »

Did we ever get a definitive answer regarding the use of Bandits UI?

Have people experimented with disabling it or add-ons in general to see if that helps with the issue in it's current state?

The addon developer has removed all pings from it. There's nothing to look for or respond to anymore.
If you don't believe me, you can find all the lines with PingMap in the addon; they are all commented out.

As for other addons, yes, it has been mentioned hundreds of times here that disabling them doesn't make any difference. The same issue occurs on consoles, which rules out even the theoretical possibility of addons having any impact.

To further hammer that second point home - if this is an actual issue with the API then it doesn't matter if BanditsUI disabled it or not. Some bad actor(s) will inevitably create their own addons to specifically abuse this loophole. I would hope that ZoS would be aware of such issues and immediately take action if they really did exist.

I'm sure they are having interal conversations about their API as a whole since Bandits was brought up - to make sure there aren't any other methods of sending information/causing issues like Bandits may have been doing.

It is my understanding from talking to some add-on devs who are in communication with the ZOS devs who manage the API, that solutions are being worked on. It sounds like the current functionality for data transfer will be completely removed, i.e. Hodors and Bandits and other add-ons that share ultimate and DPS information will no longer work. You can look at the lib add-ons for Hodor for example in esoui for evidence.

I have also been told that zos is working with the add-on devs to find an alternate solution so PC raid leads don't have to go back to timing ultimates with stop watches.

There is going to be 2 other Libraries that will address the MapPing timing. You won't be able to make calls every second like it currently is, it will be more on a 10 second call instead, which will kind of mess things up.
The biggest problem is that some addon developers didn't convert to LibMapPing to make these calls, which would have made that date share work with all addons and make a single call for the same type of data. Instead developers had to try and use their own call with MapPing which ends up "spamming" the same data type.
This is not the entire issue though, there are other issues at hand.

Read the U45 API patch notes posted at ESOUI. There's an entirely new data broadcast API specifically for use by addons, the specifications of which you can check out.

Yes, I know about that, been on there. If you look at the LibMapPing change logs, you'll see the name of the 2 new Libraries for data broadcasting.

SaffronCitrusflower · January 15

This morning we had a near entire faction disconnect in Cyrodiil. Then we couldn't log back in. Then there was a queue to log back in..then after about 10 minutes most of us could get back into Cyrodiil.

This is very obviously not a result of dDos attacks. This is very obviously an issue at ZOS' server end. How is this not fixed yet after on and off issues like this for years?

Aurielle · January 15

Has ESO been experiencing daily DDoS attacks since May 2024? I mean, that WAS around the time of the PTS fiasco, so maybe someone got overly angry about how ZOS handled that and chose to lash out, but I have my doubts that it has been going on for this long.

I mean, I’ll take ZOS’s word for it that it’s what’s happening RIGHT NOW, but I hope people don’t latch onto it and say “it’s just an ongoing DDoS attack, nothing more they can do” with a shrug when these issues have been happening for so many months now. One presumes other big MMOs like WoW and such are targeted by malicious actors on a daily or near daily basis as well, but you don’t hear about players in those games having major connectivity issues for months.

loosej · January 15

M33tr3 wrote: »

show previous quotes

ZOS_Kevin wrote: »

Just wanted to follow up here. We just had Studio Director, Matt Firor post a community update about some of the DDoS attacks that have been happening. Link to statement is here.

so in a nutshell, to combat ddos attacks you basicly ddos your customers?!?!?

I think it's actually more like "we can't always tell the difference between legitimate connections and the ddos, so we throttle everything".

meekmiko · January 15

Aurielle wrote: »

Has ESO been experiencing daily DDoS attacks since May 2024? I mean, that WAS around the time of the PTS fiasco, so maybe someone got overly angry about how ZOS handled that and chose to lash out, but I have my doubts that it has been going on for this long.

I mean, I’ll take ZOS’s word for it that it’s what’s happening RIGHT NOW, but I hope people don’t latch onto it and say “it’s just an ongoing DDoS attack, nothing more they can do” with a shrug when these issues have been happening for so many months now. One presumes other big MMOs like WoW and such are targeted by malicious actors on a daily or near daily basis as well, but you don’t hear about players in those games having major connectivity issues for months.

That's around the same time that Final Fantasy XIV had major DDoS attacks last year.

Edited to add: I'm just pointing this out as I don't believe it's just a disgruntled PTS user(s). 🤷‍♀️

Aurielle · January 15

meekmiko wrote: »

show previous quotes

Aurielle wrote: »

Has ESO been experiencing daily DDoS attacks since May 2024? I mean, that WAS around the time of the PTS fiasco, so maybe someone got overly angry about how ZOS handled that and chose to lash out, but I have my doubts that it has been going on for this long.

I mean, I’ll take ZOS’s word for it that it’s what’s happening RIGHT NOW, but I hope people don’t latch onto it and say “it’s just an ongoing DDoS attack, nothing more they can do” with a shrug when these issues have been happening for so many months now. One presumes other big MMOs like WoW and such are targeted by malicious actors on a daily or near daily basis as well, but you don’t hear about players in those games having major connectivity issues for months.

That's around the same time that Final Fantasy XIV had major DDoS attacks last year.

Your AI summary does not state that FFXIV players have been having connectivity issues daily and consistently for months on end since the initial May DDoS attack — just that they experienced some connectivity problems during those attacks, which happened on three separate occasions. That’s quite different to what a number of us have been experiencing here in ESO.