ZOS, Massive Spike in Ping/LAG in Recent Days - What Gives?

Destai

Rkindaleft wrote: »

Appreciate the update @ZOS_RichLambert @ZOS_Kevin however I’m just typing this out because a lot of PSNA is suffering from some pretty severe symptoms the last half hour or so. MT disconnected mid pull and can’t log back in, nearly everyone in my trial group is having skipping and having severe barswap delay.

Just grabbed this from another discord to show it’s not just us, people are also talking about it in other guilds in game.

Yeah, PS NA is super laggy right now. One of my buddies got dropped just now, and a lot of my input is lagged.

Narvuntien

I have had some bad ping days, but as I live a very long way away from the servers (all servers) I expect it to some extent, usually, I start looking for broken connections. Today appears to be one of those days

AvalonRanger

2024/12/14 (Sat) PC-NA

After the maintenance, still very unstable. Server kick me out 3 times within 10 min.

ISCOLDSOUL

Whatever the issue ZOS ignored it for way to long and is bleeding players as a result. What changed in May this year to begin these problems is also a valid point that hasn't been answered (IMO). Thankyou regular posters on this page for continually drawing attention to these issues

Destai

Kicked and now cannot login. 216am Eastern. Ps-na

zeyva

After being kicked 2-6 times per hour all night, I was kicked at approximately 1:50am Eastern in my house at a transmute station, and now can't login. I really appreciate all the work that went into restoring the servers last night, as well as the extra rewards we are getting in Jan for our Endeavors (seriously that's awesome!!!!). But if this is an outside actor just let us know. If this is a continuation of the DC issue, that is something else too. but either way Please let us know.

I've disabled every add-on, per one of the earlier posts but I can't even login to a point where the game loads the add-ons and I'm sad.

ISCOLDSOUL

Destai wrote: »

Kicked and now cannot login. 216am Eastern. Ps-na

Same Same

ZOS_Kevin

Hey all, just wanted to note that we have a team looking into the current connection issues. So we’ll provide more info based on how those go. Just noting it might take a bit because of the current time on our end of things. Being almost 3am.

Destai

ZOS_Kevin wrote: »

Hey all, just wanted to note that we have a team looking into the current connection issues. So we’ll provide more info based on how those go. Just noting it might take a bit because of the current time on our end of things. Being almost 3am.

Sleep well, tired soldier!

Quackery

This is ridiculous, I keep getting booted from the server on Xbox NA-server constantly. What is going on?!?

Ardaghion

When my latency indicator goes red I get the usual delay in interacting with anything but I haven't seen any rubberbanding.

The weirdest thing is that if I'm sprinting, my toon runs really slow, like running through molasses.

LadyGP

Quackery wrote: »

This is ridiculous, I keep getting booted from the server on Xbox NA-server constantly. What is going on?!?

Given the datacenter power issues and the rebooting (I'm going to assume someone had to go one by one since the fail safes couldn't kick in) of the servers... give them a break. Probably some lingering issues from that still.

LadyGP

@React posted this video in another thread but wanted to drop it here as it's a perfect example of what I deal with when I play.

When the game works for me (no rubberbanding/high ping) this is the typical state. It's clunky at best. Difficult to target people because they are stuttering around so much, etc.

If I was able to stay connected to the game and this was my experience... well I'd take it. Unfortunately, I spend a great deal of time rubber banding or trying to log back in due to an "unexpected error".

https://www.youtube.com/watch?v=7s6zUdkv9A8

Linking the video to keep the conversation going.

KaironBlackbard

thaellin wrote: »

@sshogrin @KaironBlackbard

"The addons only get information from the ZOS servers, they don't travel to ZOS servers, then back to your computer, than back to ZOS, then back to them. The addons only get information like I stated directly from the servers that are part of the base game API. "

This is incorrect. KaironBlackbard has correctly observed the 'map ping backchannel communication' method. Map pings are used by addons which want to do data sharing outside the stuff provided by the game. In particular, any addon which has ultimate sharing (such as Hodors) or other realtime group data sharing (group DPS) will send map pings to the group which encode the data. It's a rather brilliant way to bypass the lack of any dedicated data passing API. Sometimes if you have your map up you can see the ping marker dancing around.. This is also (likely) why some group DPS trackers will sometimes show an absurd DPS score from users who have Bandit's UI installed: there is no firm protocol to prevent interfering with another addon's backchannel messaging.

I think there is an addon library which tries to standardize the process (might be internal to hodors), but it's optional and may not be written to minimize the bandwidth usage across a group. I have only looked at it in passing, thought "that is really clever" and realized I didn't need it for what I was writing at the time.

It might be possible for ZOS to limit the frequency of map pings before they are issued from the addon API, or de-prioritize the messages. At the API layer, an attempt to send a map ping could report failure if one had been sent too recently. That would prevent server spamming without kicking anyone and have no impact on regular users, so it might be a valid approach.

@LadyGP if you are not grouped, it should not be possible to get map ping backchannel stuff.

I had a feeling it was a backdoor.

Backdoor needs to be locked.

Varana

No it doesn't. :rolleyes: I have no idea what the reason for your personal crusade is, but just let it go.
The method enables a ton of very useful functionality. And works just fine.

LadyGP

Varana wrote: »

No it doesn't. :rolleyes: I have no idea what the reason for your personal crusade is, but just let it go.
The method enables a ton of very useful functionality. And works just fine.

I mean, if an add on someone in group is running is giving someone who doesn't have it performance issues... I would say it does not "work just fine".

I didn't spend a stupid amount of money on my pc to have someone in my group making my latency go through the roof and my system bog down because of the add-ons they decide to run.

manukartofanu

To begin with, this is not a backdoor.

YandereGirlfriend

manukartofanu wrote: »

To begin with, this is not a backdoor.

It really doesn't matter what the semantics of it are. It is singled-out in the official ZOS communication on the topic. So continue to deny if you like, I suppose. But it is being taken seriously by those in charge of actually running and maintaining the game.

If it is negatively affecting non-users of the add-on then it is a problem that needs to be resolved ASAP. If it cannot be resolved then that method needs to be patched out of the game. It that breaks some add-ons, oh well. The rights of add-on users end if/when they begin to negatively impact the game experience of others.

That said, it is important for everyone to understand that Bandits/sus add-ons aren't the only thing going on. That's a straw man argument that I see being erected quite often here. It is likely one part of the puzzle that most impacts a particular subset of players (those with limited internet connections, etc.), with others potentially being Akamai, issues with the codebase, misconfigured network infrastructure, etc..

In other words, all of these things can be true at the same time. They are not mutually exclusive of one another. There is no "One Weird Trick" that is going to magically resolve every issue. It will be a combination of different fixes.

manukartofanu

To begin with, this is not a backdoor.

YandereGirlfriend wrote: »

manukartofanu wrote: »

To begin with, this is not a backdoor.

It really doesn't matter what the semantics of it are. It is singled-out in the official ZOS communication on the topic. So continue to deny if you like, I suppose. But it is being taken seriously by those in charge of actually running and maintaining the game.

If it is negatively affecting non-users of the add-on then it is a problem that needs to be resolved ASAP. If it cannot be resolved then that method needs to be patched out of the game. It that breaks some add-ons, oh well. The rights of add-on users end if/when they begin to negatively impact the game experience of others.

That said, it is important for everyone to understand that Bandits/sus add-ons aren't the only thing going on. That's a straw man argument that I see being erected quite often here. It is likely one part of the puzzle that most impacts a particular subset of players (those with limited internet connections, etc.), with others potentially being Akamai, issues with the codebase, misconfigured network infrastructure, etc..

In other words, all of these things can be true at the same time. They are not mutually exclusive of one another. There is no "One Weird Trick" that is going to magically resolve every issue. It will be a combination of different fixes.

Don't confuse an investigation with a conclusion that there is a backdoor. Once again, the method used in Bandits is used in a large number of addons. Bandits, in particular, is under investigation because it was mentioned here. By the way, I was the first to bring it up in this thread months ago. IMHO, non-technical people got overly excited about its mention here because they didn't understand what exactly is being investigated, while already having formed an extremely biased opinion about the situation. And all of this completely ignores the fact that similar problems exist on consoles, where there are no addons at all. But you just want to assign blame.

meekmiko

Before everyone gets out their pitch forks they need to step back and reread what Rich said:

"Investigate the Bandits UI addon. (In the top 15 most used add-ons currently) There are lots of forum posts pointing to this add-on for potential latency issues. The suspicion is that the addon is using map pings as somewhat of a backdoor to send arbitrary data to clients, regardless of whether players have the add-on installed. The pings would get broadcast to everyone and cause game performance problems (latency & frame rate)."

He didn't specifically SAY that YES Bandits IS causing the problems. He says they're going to INVESTIGATE Bandits as there "are lots of forum posts pointing to this add-on for POTENTIAL latency issues". And he says "the suspicion" Besides the only "lots of forum posts" I've seen signaling out Bandits has been from Kairon.

And regardless HOW are console users having the same problems when they haven't ANY access to add-ons?

TX12001rwb17_ESO

And I crashed again, it was fine for 30 minutes.

manukartofanu

By the way, what tells me a lot about the situation is that no one has paid attention to the HeartbeatDisconnector. Its malfunctioning could very well explain the random disconnects with garbage errors happening anywhere and under any conditions. After all, it even disconnects during AFK in the middle of a desert. And even during a card game, where stable connectivity is not really necessary. If it is malfunctioning and does not accept all incoming packets as proof of connection, but only specific ones, a subset of them, or, worse, just its own heartbeats, and then rushes to disconnect, we end up with what is happening now: people with good internet and stable connections experiencing random disconnects. Why? Because minor packet losses are basically normal in today's internet.

But first, we were told that they are investigating. Secondly, these are just assumptions and speculations, and there’s no point in shouting that this is definitely the reason under certain conditions, so the HeartbeatDisconnector should be removed from the game immediately because it’s interfering with our ability to play! The investigation is ongoing, and the technical specialists with access to the code know better what the real problem is.

YandereGirlfriend

manukartofanu wrote: »

To begin with, this is not a backdoor.

YandereGirlfriend wrote: »

manukartofanu wrote: »

To begin with, this is not a backdoor.

It really doesn't matter what the semantics of it are. It is singled-out in the official ZOS communication on the topic. So continue to deny if you like, I suppose. But it is being taken seriously by those in charge of actually running and maintaining the game.

If it is negatively affecting non-users of the add-on then it is a problem that needs to be resolved ASAP. If it cannot be resolved then that method needs to be patched out of the game. It that breaks some add-ons, oh well. The rights of add-on users end if/when they begin to negatively impact the game experience of others.

That said, it is important for everyone to understand that Bandits/sus add-ons aren't the only thing going on. That's a straw man argument that I see being erected quite often here. It is likely one part of the puzzle that most impacts a particular subset of players (those with limited internet connections, etc.), with others potentially being Akamai, issues with the codebase, misconfigured network infrastructure, etc..

In other words, all of these things can be true at the same time. They are not mutually exclusive of one another. There is no "One Weird Trick" that is going to magically resolve every issue. It will be a combination of different fixes.

Don't confuse an investigation with a conclusion that there is a backdoor. Once again, the method used in Bandits is used in a large number of addons. Bandits, in particular, is under investigation because it was mentioned here. By the way, I was the first to bring it up in this thread months ago. IMHO, non-technical people got overly excited about its mention here because they didn't understand what exactly is being investigated, while already having formed an extremely biased opinion about the situation. And all of this completely ignores the fact that similar problems exist on consoles, where there are no addons at all. But you just want to assign blame.

Did you happen to notice how I qualified my statements with "if" as I am very aware that it is only under investigation at this point. And, of course, I'll accept whatever the outcome of that investigation ends up being. But it IS under investigation. If the case was so blindingly obvious that it could be dismissed out-of-hand and the effort of actually investigating it bypassed... it likely would have been.

You are also erecting the exact same straw man of "but the consoles" that I called-out earlier. Different players can disconnect or experience performance problems for different reasons. To think otherwise is naive. (Yes, this includes anyone trying to solely blame add-ons; though I do not believe that any such poster actually exists here.)

My own purely anecdotal evidence is that I run a 12-man ballgroup in Grey Host that was having enormous issues with random kicks and disconnections during our raids. Then the issue of Bandits was raised here and it turned out that nearly 3/4 of the raid was using it, along with the traditional RDK, as well as a whole raft of other state-sharing QoL add-ons. As an experiment, we pared everyone down to a very bare bones mod load-out, banished Bandits, etc. and, lo and behold, our incidence of random kicks and disconnects has plummeted. Correlation is not causation, of course; but, in our particular case, fewer state-sharing add-ons has led to much-improved stability for all of our raiders. Additional note: those most prone to kicks were also those playing on the least capable internet connections and hardware configurations.

DewiMorgan

KaironBlackbard wrote: »

I had a feeling it was a backdoor.
Backdoor needs to be locked.

You may be conflating "backchannel" (the word used in the post you responded to) with "backdoor".

Backdoor: an exploit to evade normal authentication procedures. Nobody's suggesting such a thing exists, nor that if it did, it would affect lag, ping times, or other people.

Backchannel: "out-of-band" communications - so, communicating without spamming chat, for example. This could include encoding meaning into anything that gets sent between players. So for example, you could add jitters to normal player movement, and others with the addon could interpret those jitters as data (though that particular one happens to be forbidden by the ToS for different reasons). In particular, map pings are decently reliable and lightweight, so tend to get used. There's even a well-behaved library called "libMapPing" to manage this channel of communication, which all addon developers use... except one.

Backchannel communications do not, in themselves, imply significantly higher bandwidth usage, though the evidence presented here suggests that Bandits can be obscenely bad at it. I wonder if this is a misconfiguration by users (perhaps there's a "send crazy debugging levels of data" checkbox or something?), as the levels of data being reported for Bandits sound kinda ludicrous for any reasonable form of metrics.

Then again, I lack much respect for Bandit's: I used it for years, but it struck me as bloatware that played very arrogantly and poorly with other addons, refusing to follow the standards agreed by all other addon developers: I found myself turning off and replacing most of its features with better alternatives (the pings, if I remember right, were the first thing I turned off), and this year I uninstalled it as I realized at this point it was doing nothing for me. Its design arrogance felt obvious at every level, from having its own settings menu on down. Conflicting with LibMapPing is just a particularly egregious example.

DewiMorgan

ZOS_RichLambert wrote: »

However, in the case of input lag or high ping, if it’s a result of an attack on the servers, we are able to turn adjust mitigation whenever we need.

I suspect that there're limits to what can be done there, and that this may be a factor in the lag over the last few days.

This is traditionally a time of seasonal DDoS attacks anyway, but the last few days in particular have got kinda silly: maybe a response to the recent multinational "Operation PowerOFF" that shut down 27 DDoS sites a few days ago (FBI page; Europol page)?

Other MMOs (eg FF XIV) have been struggling with this. Still, looks like things have quietened down, in the US at least, since last night... if it doesn't pick up again tonight, it might be over? Best to get my dailies done while I can, though, I think.

LadyGP

manukartofanu wrote: »

By the way, what tells me a lot about the situation is that no one has paid attention to the HeartbeatDisconnector. Its malfunctioning could very well explain the random disconnects with garbage errors happening anywhere and under any conditions. After all, it even disconnects during AFK in the middle of a desert. And even during a card game, where stable connectivity is not really necessary. If it is malfunctioning and does not accept all incoming packets as proof of connection, but only specific ones, a subset of them, or, worse, just its own heartbeats, and then rushes to disconnect, we end up with what is happening now: people with good internet and stable connections experiencing random disconnects. Why? Because minor packet losses are basically normal in today's internet.

But first, we were told that they are investigating. Secondly, these are just assumptions and speculations, and there’s no point in shouting that this is definitely the reason under certain conditions, so the HeartbeatDisconnector should be removed from the game immediately because it’s interfering with our ability to play! The investigation is ongoing, and the technical specialists with access to the code know better what the real problem is.

The heartbeat was the biggest take away that I took from the post. I've had this conversation with my partner but I always felt like in every game I have played if packets drop or connection stops for under a second... you hardly ever notice it.

In ESO I feel like if ANY packets get dropped or connection so much as "blips" then you get the "an error has occurred" message.


As to the Add on convo. There is clearly more than just Bandits possibly being an issue (and who knows if that even IS an issue). Doesn't hurt to investigate it. Who knows, maybe it's both hearbeat and add ons in general.

To your point, I'd be curious to see if they could just drop the heartbeat check for a patch cycle and just see what happens. All I'd ask is they communicate that they made that change so I can install the game again. I realllllly want to play this game (when everyone has told me to move on)... so... let us help you ZoS lol.

Sluggy

KaironBlackbard wrote: »

I see several faults with your assumption.

1. you assume that they store data on ZOS servers to be retrieved by the addon. They don't. They can't. Similar addons like TTC use a separate exe to send and receive such data to and from a server that will hold the data: their own.
2. you assume that they only see the stats of people using the same addon. They see my stats and that of non-addon users. Your explanation fails to explain that.
3. you claim they don't make calls, but it's clear they do, especially when they turn off their BanditsUI and I get pristine connection again.

1) I'm not entirely familiar with the API for addons but it's entirely possible they have some small amount of allowance for such things. Most of the data they are sharing is actually already being sent from your client to the server regardless of the presence of addons. There are parts of the API addon that allow your client to request some of that data from the server and that has a hardcap so as to not allow addons to dos the server itself. This is why it's usually a good idea to reduce or disable certain features of certain addons if you are going to be in a particularly hot conflict as it can result in hitting that cap and getting forcefully disconnected from the server. This is, however, no different than something like spamming the block key which will also cause a force disconnect as a protection by the server.

Regardless, the TTC .exe is solely for the purpose of posting externally to their own database on their own website.

2) See above... it's already data being shared anyway.

3) Placebo effect. There's plenty of us that never use addons and don't play in groups and we still get railed by these same issues. This means that even if it were in fact caused by an errant addon, then it's a fundamental issue with the server and the API itself and would need to be fixed regardless of the presence of addons or not (such as on consoles).


EDIT: Fixed the target of the quote.

Destai

Sluggy wrote: »

Destai wrote: »

I see several faults with your assumption.

1. you assume that they store data on ZOS servers to be retrieved by the addon. They don't. They can't. Similar addons like TTC use a separate exe to send and receive such data to and from a server that will hold the data: their own.
2. you assume that they only see the stats of people using the same addon. They see my stats and that of non-addon users. Your explanation fails to explain that.
3. you claim they don't make calls, but it's clear they do, especially when they turn off their BanditsUI and I get pristine connection again.

1) I'm not entirely familiar with the API for addons but it's entirely possible they have some small amount of allowance for such things. Most of the data they are sharing is actually already being sent from your client to the server regardless of the presence of addons. There are parts of the API addon that allow your client to request some of that data from the server and that has a hardcap so as to not allow addons to dos the server itself. This is why it's usually a good idea to reduce or disable certain features of certain addons if you are going to be in a particularly hot conflict as it can result in hitting that cap and getting forcefully disconnected from the server. This is, however, no different than something like spamming the block key which will also cause a force disconnect as a protection by the server.

Regardless, the TTC .exe is solely for the purpose of posting externally to their own database on their own website.

2) See above... it's already data being shared anyway.

3) Placebo effect. There's plenty of us that never use addons and don't play in groups and we still get railed by these same issues. This means that even if it were in fact caused by an errant addon, then it's a fundamental issue with the server and the API itself and would need to be fixed regardless of the presence of addons or not (such as on consoles).

The original quote there is not from me, FYI

Sluggy

YandereGirlfriend wrote: »

And to the courageous @KaironBlackbard for continuing to bang the add-ons/Bandits drum even in the face of intense opposition. So many told them, "But it couldn't possibly be add-ons!" and yet... here we are. And if Bandits is misbehaving it almost assuredly is not the only one to be doing so.

To be fair, if it is addon related then it's not a specific addon but rather the API design that needs to be addressed. The addon(s) in question would then have to adapt to any new changed or remove such functionality if it was no longer available. Such a weakness in the API could be exploited regardless of the presence of an addon.

Sluggy

Destai wrote: »

The original quote there is not from me, FYI

lol My mistake. It's fixed now.