ZOS, Massive Spike in Ping/LAG in Recent Days - What Gives?

blktauna · December 15

ZOS_RichLambert wrote: »

Hey folks - here's some answers to the main questions in this thread, as well as some insights to our process and what we're working on to help us better debug the issue(s). Thanks for the questions and please keep on providing feedback!

Thank you slashlurk, nice to see you again. This is a great start.

blktauna · December 15

code65536 wrote: »

@ZOS_RichLambert I'm surprised that there's no mention of connectivity in general and Akamai in particular. My game performance on PC/NA has been pretty consistently good, and the times that I've had issues were times when the problem was with the connection to ZOS (e.g., times when Akamai sends my packets on a tour around the world before delivering it to the datacenter). I know that there was a surge of comments in this thread a few months during a period when multiple people were having these kinds of "problem exists between you and the server" issues.

Very much this. Normally I have 3 akamai jumps, really bad performance has up to 7.
Normally I still have a lot of skill delay and bar switching issues on early prime time EU (im in NA) and early primetime NA is the same until I'm in cyrodil and the ball groups come in. I just exit at that point.

DewiMorgan · December 15

Sluggy wrote: »

To be fair, if it is addon related then it's not a specific addon but rather the API design that needs to be addressed. The addon(s) in question would then have to adapt to any new changed or remove such functionality if it was no longer available. Such a weakness in the API could be exploited regardless of the presence of an addon.

Sadly, that's happened in the past (guild stores, anything exploitable in PvP)

Thankfully, ZOS seem to get that, so long as a behavior isn't deliberate/malicious, it's better to work with the addon dev community to mitigate problems, than remove/restrict usage. Educating devs, documenting, even improving the scripting engine to help devs be good, rather than prevent devs being bad. They're startlingly generous about what can be done by addons, and it pays off in the creativity of the addon devs.

Cooperatively building sandcastles beats losing an arms race.

Coo_PnT · December 15

I believe this problem is happening on all platforms.

Is it safe to assume that ZoS suspects that the Bandits UI add-on for PC (and other add-ons) is causing an overload on the network and affecting the login servers that control all platforms NA and EU? The login servers are separate from the actual game servers, right? Would a virtual server be affected if they are co-located on the same hardware?

I'm sure there are multiple other factors (Akamai) involved, but can someone with knowledge explain it to me in a concise manner?

mook-eb16_ESO · December 15

I'm having a lot more lag than normal in pvp being disconnected which doesn't normally happen. are servers are broken still ?

KaironBlackbard · December 15

blktauna wrote: »

ZOS_RichLambert wrote: »

Hey folks - here's some answers to the main questions in this thread, as well as some insights to our process and what we're working on to help us better debug the issue(s). Thanks for the questions and please keep on providing feedback!

Thank you slashlurk, nice to see you again. This is a great start.

slashlurk?

LadyGP · December 15

KaironBlackbard wrote: »

blktauna wrote: »

ZOS_RichLambert wrote: »

Hey folks - here's some answers to the main questions in this thread, as well as some insights to our process and what we're working on to help us better debug the issue(s). Thanks for the questions and please keep on providing feedback!

Thank you slashlurk, nice to see you again. This is a great start.

slashlurk?

https://x.com/slashlurk

virtus753 · December 15

Coo_PnT wrote: »

I believe this problem is happening on all platforms.

Is it safe to assume that ZoS suspects that the Bandits UI add-on for PC (and other add-ons) is causing an overload on the network and affecting the login servers that control all platforms NA and EU? The login servers are separate from the actual game servers, right? Would a virtual server be affected if they are co-located on the same hardware?

I'm sure there are multiple other factors (Akamai) involved, but can someone with knowledge explain it to me in a concise manner?

I do not think it would be safe to assume that. Bandits was not mentioned as the sole or even primary cause of the game’s performance issues. It was listed at the end (literally last) as a potential source of some of the degraded performance and latency, not in relation to load on the login servers.

As Rich Lambert mentioned in his post, there are many factors at work for ZOS to figure out when it comes to the overall picture of why so many people are experiencing poor performance and disconnects. The Heartbeat issue is one that affects all players on all platforms, for example, and it was accordingly listed before Bandits in their list of things to investigate and address. Bandits primarily impacts those in group with people running the add-on, so PC.

As for other add-ons, check a few posts up to see why Bandits is unique here. It does not use the same library that other add-ons do to share data, so it is not safe to assume that other add-ons are causing the same symptoms.

LadyGP · December 15

Coo_PnT wrote: »

I believe this problem is happening on all platforms.

Is it safe to assume that ZoS suspects that the Bandits UI add-on for PC (and other add-ons) is causing an overload on the network and affecting the login servers that control all platforms NA and EU? The login servers are separate from the actual game servers, right? Would a virtual server be affected if they are co-located on the same hardware?

I'm sure there are multiple other factors (Akamai) involved, but can someone with knowledge explain it to me in a concise manner?

Not a network admin but took CCNA1-4 and am a Senior Software Engineer so here are my thoughts, based on my experience, as to what is going on.

TL:DR - A lot of smaller issues are impacting people in similar ways so it appears as one bigger issue.

Bandits: Clearly there is something there based on the number of comments on forums/social media. If there are any add-ons that can impact another players experience, who isn't even running the add-on, then that clearly is something ZoS has to look into and fix. It's highly possible that some users who are coming into this thread reporting sluggish-ness in the game, spikes, etc. are because someone in their group is using Bandits. Shoot, I think there was a comment a page or two ago confirming this. This doesn't explain the issue for console players but there is obviously smoke here - is there fire? Time will tell.

Akamai: Good ol' Akamai. There have been posts going back to 2017 (maybe even before) talking about Akamai and it causing hops to get dropped/hang and it impacting players experiences. I'm going to be honest, I don't know enough on the networking side of things to say this is a root cause or not but again there clearly is a lot of smoke here. I haven't been able to come across a post from ZoS talking about Akamai and their "view" on the topic to date - hopefully we can get something from @ZOS_JessicaFolsom / @ZOS_Kevin on this sometime. This could explain why console players are having issues as well.

Login Servers: Yes, login (authentication) servers are on different servers than what the game runs on. There is the NA data center and then EU. If one goes down the other shouldn't go down. It's best practice to have redundancies for things like authentication and I'm kind of surprised that there wasn't a "mirror" of the NA auth servers on EU that kicked in when NA went down. Maybe there are and there was an "issue" that prevented it from kicking in. So to your question about the virtual servers being affected if they are on the same hardware - yes. When the master power switch gets turned off... whatever is on that server... goes sleep sleep.

It's also entirely true that simply all authentication routes through NA first... which if that is the case I'm 100% positive some lower level Sys Admin said this is a single point of failure and a bad idea at some point in the process. Who knows why the decision (if this is in fact the case) was made to only keep it in NA. Clearly, if NA goes down all the other regions should be up.. that didn't happen. I'm confident ZoS is already bringing this point up internal in their postmortem.

HeartBeat: This is where all my attention is at right now. Reading what Rich said and trying to pair that with my personal experiences, it feels as though something here is forcing people to get dropped from the game. Something is happening where the packets/traffic coming in get bogged down and then the game thinks we closed the client (or something) and it kicks you.

It's almost as though playing ESO is like the water coming out of a hose - the hose is ZoS. Playing the game is fine.. water is coming out.. but step on it for a few seconds and the water starts to drop off (skills not firing... high ping). Take your foot off the hose and the water comes rushing out at a higher force until it catches back up (rubber banding). Keep your foot on the hose to long and .. well ESO kicks you "an unknown error has occurred."

My personal experience playing ESO as if someone is stepping on my hose every 3-5 minutes and I better not be needing water (stop moving/casting) when a foot is on the hose or else I get dropped.

what doesn't make sense to me with this situation is that it almost seems like it's region based. For a lot of people in this thread putting on a VPN "fixes" the issue.. for the most part. Sometimes I have to change my VPN to different parts of the country to get a different route before I land on a stable connection with ESO.

Additionally, I think the issue is more widespread than ZoS thinks/can see in the data. These are "blips" in network connection/server that happen in such a short time interval that I think it's easy for them to get masked in the overall data. How many people don't even realize they have 999+ ping when they are playing. Having to interact with something 2 or 3 times is fairly normal in ESO and until I turned on my latency indicator I just assumed I wasn't aligned correctly with the object - turns out I had 999+ EVERYTIME I was having to click a few times. I've also been in coms with people who had no idea they had 999+ until they got hit with rubberbanding or a hard crash. So, I'd venture to say the actual reports coming in on this issue are significantly less than they are in the wild. Just look at a few streamers playing and, if you know what to look for, you can clearly see them having similar issues that are being reported in this thread (and the streamer doesn't even have a clue about it.. or is just not mentioning it I guess)

It also sounded like (reading between the lines) that ZoS might not have their logging tools configured in a way to where they could see these kind of issues. How does the "An Error has occurred" get reported in the logging? Does it even show up as a crash/disconnect?

I don't know if this helped at all.. but this is just my... I've been around in this thread for like 7 months and personally have been having these issues and tried everything possible to get it fixed on my side aside from moving to a different part of the country... point of view.

StihlReign · December 15

Dump Akamai, move to an in-house Microsoft custom solution. Tune and replicate that system, license it, make billions, the end.

Varana · December 15

LadyGP wrote: »

Varana wrote: »

No it doesn't. :rolleyes: I have no idea what the reason for your personal crusade is, but just let it go.
The method enables a ton of very useful functionality. And works just fine.

I mean, if an add on someone in group is running is giving someone who doesn't have it performance issues... I would say it does not "work just fine".

I didn't spend a stupid amount of money on my pc to have someone in my group making my latency go through the roof and my system bog down because of the add-ons they decide to run.

Apart from what others have written, I was talking about the general method of using map pings to share information in the group. That works just fine, as numerous addons use it without causing issues.

If Bandits does something wonky with it (or another method) - which, again, hasn't actually been proven yet - that's another topic. But the guy was calling for the whole method to be shut down for whatever personal grudge, not just investigating the addon.

virtus753 · December 15

Varana wrote: »

LadyGP wrote: »

Varana wrote: »

No it doesn't. :rolleyes: I have no idea what the reason for your personal crusade is, but just let it go.
The method enables a ton of very useful functionality. And works just fine.

I mean, if an add on someone in group is running is giving someone who doesn't have it performance issues... I would say it does not "work just fine".

I didn't spend a stupid amount of money on my pc to have someone in my group making my latency go through the roof and my system bog down because of the add-ons they decide to run.

Apart from what others have written, I was talking about the general method of using map pings to share information in the group. That works just fine, as numerous addons use it without causing issues.

If Bandits does something wonky with it (or another method) - which, again, hasn't actually been proven yet - that's another topic. But the guy was calling for the whole method to be shut down for whatever personal grudge, not just investigating the addon.

If it wasn’t shut down when Hodor took it to the point of remotely controlling other players’ characters, I don’t foresee it getting shut down for this.

Aurielle · December 15

I find it interesting that separate threads keep popping up every day now about how people are getting disconnected, experiencing skill lag and rubberbanding, etc... Related to the issue those of us posting here have been experiencing since May, or separate issue? I mean, it sounds identical to what we’ve all been experiencing…

virtus753 · December 15

Aurielle wrote: »

I find it interesting that separate threads keep popping up every day now about how people are getting disconnected, experiencing skill lag and rubberbanding, etc... Related to the issue those of us posting here have been experiencing since May, or separate issue? I mean, it sounds identical to what we’ve all been experiencing…

Several of the recent threads from the past few days, especially Thursday and Friday, were related to the major power incident at the datacenter that hosts the NA servers (both the actual megaservers for NA and the login server for all platforms and regions). There seems to have been another period of major instability after that as well, though I’m not sure whether the two were related. I experienced the second myself - several disconnects just standing in an overland zone with two trash mobs, preceded by incoming and outgoing attacks being delayed and rubberbanding.

I don’t know how we can tell on our end what might be the aftereffects of the power failure or some other discrete episode as opposed to the “new normal” for performance that this thread brought attention to.

BasP · December 15

Time: 15:07 or 15:08 GMT+1
Server: PC-EU
Where: Infinite Archive
What: A ~20 second freeze mid-combat, followed by being disconnected. Afterwards I saw a generic error message when I landed in the lobby, and I got the following Login Error when I immediately tried to login again. I could login again about a minute later, but stopped playing.

The same thing happened in Vateshran Hollows on Friday evening for me, but I sadly didn't look at the time. I think it was around 23h.

Yesterday, and earlier today, I didn't have any issues while doing Overland content, a Battleground, multiple Dungeons and two Arena's though, and ESO's network latency indicator was in between 50 and 100 most of the time. So I certainly have far less issues than some other players in this thread. Still, it's annoying and a bit demotivating to be faced with random disconnects these days (this only started happening in December for me).

LadyGP · December 15

Aurielle wrote: »

I find it interesting that separate threads keep popping up every day now about how people are getting disconnected, experiencing skill lag and rubberbanding, etc... Related to the issue those of us posting here have been experiencing since May, or separate issue? I mean, it sounds identical to what we’ve all been experiencing…

Agreed. With the exception of the last few days (datacenter issue) there has been a steady stream of lag/rubberbanding/ping threads over the last few months.

This goes back to my point of - I think this is a much larger issue and is hitting a lot more people than we know (many don't even realize their ping spikes because they don't have the interface enabled or simply just say "I dc'ed logging back in now" and never report their experience).

I'd also love to know if these random error occurred messages are showing up in logs anywhere. If these aren't I hope there is a way ZoS can start tracking these. I'm sure that number would be massive compared to the other "crashes".

sshogrin · December 15

@thaellin
"This is incorrect. KaironBlackbard has correctly observed the 'map ping backchannel communication' method. Map pings are used by addons which want to do data sharing outside the stuff provided by the game. In particular, any addon which has ultimate sharing (such as Hodors) or other realtime group data sharing (group DPS) will send map pings to the group which encode the data. It's a rather brilliant way to bypass the lack of any dedicated data passing API. Sometimes if you have your map up you can see the ping marker dancing around.. This is also (likely) why some group DPS trackers will sometimes show an absurd DPS score from users who have Bandit's UI installed: there is no firm protocol to prevent interfering with another addon's backchannel messaging."
Not incorrect...the map pint backchannel communication method might possibly be an issue, but the addon is not actually doing a communication back to other player's computers, then back to zos, then back to other players, it is merely pinging on the zos servers themselves. That extra data call could be an issue...with that being said, it's also possible that having multiple addons that do ultimate/dps sharing using those options is actually causing the issue, ie. having ultimate/dps sharing turned on in Hodors, Bandits, etc. That could actually be throwing more data using the backchannel method mentioned by ZOS.

KaironBlackbard · December 15

virtus753 wrote: »

Varana wrote: »

LadyGP wrote: »

Varana wrote: »

No it doesn't. :rolleyes: I have no idea what the reason for your personal crusade is, but just let it go.
The method enables a ton of very useful functionality. And works just fine.

I mean, if an add on someone in group is running is giving someone who doesn't have it performance issues... I would say it does not "work just fine".

I didn't spend a stupid amount of money on my pc to have someone in my group making my latency go through the roof and my system bog down because of the add-ons they decide to run.

Apart from what others have written, I was talking about the general method of using map pings to share information in the group. That works just fine, as numerous addons use it without causing issues.

If Bandits does something wonky with it (or another method) - which, again, hasn't actually been proven yet - that's another topic. But the guy was calling for the whole method to be shut down for whatever personal grudge, not just investigating the addon.

If it wasn’t shut down when Hodor took it to the point of remotely controlling other players’ characters, I don’t foresee it getting shut down for this.

That's creepy

sshogrin · December 15

KaironBlackbard wrote: »

thaellin wrote: »

@sshogrin @KaironBlackbard

"The addons only get information from the ZOS servers, they don't travel to ZOS servers, then back to your computer, than back to ZOS, then back to them. The addons only get information like I stated directly from the servers that are part of the base game API. "

This is incorrect. KaironBlackbard has correctly observed the 'map ping backchannel communication' method. Map pings are used by addons which want to do data sharing outside the stuff provided by the game. In particular, any addon which has ultimate sharing (such as Hodors) or other realtime group data sharing (group DPS) will send map pings to the group which encode the data. It's a rather brilliant way to bypass the lack of any dedicated data passing API. Sometimes if you have your map up you can see the ping marker dancing around.. This is also (likely) why some group DPS trackers will sometimes show an absurd DPS score from users who have Bandit's UI installed: there is no firm protocol to prevent interfering with another addon's backchannel messaging.

I think there is an addon library which tries to standardize the process (might be internal to hodors), but it's optional and may not be written to minimize the bandwidth usage across a group. I have only looked at it in passing, thought "that is really clever" and realized I didn't need it for what I was writing at the time.

It might be possible for ZOS to limit the frequency of map pings before they are issued from the addon API, or de-prioritize the messages. At the API layer, an attempt to send a map ping could report failure if one had been sent too recently. That would prevent server spamming without kicking anyone and have no impact on regular users, so it might be a valid approach.

@LadyGP if you are not grouped, it should not be possible to get map ping backchannel stuff.

I had a feeling it was a backdoor.

Backdoor needs to be locked.

this is not a "back door", it's backchannel which is much different. Read a post of mine that I replied to @thaellin about this.
Now please explain to me why disconnects, latency, low frame rates, getting booted from server are happening on Xbox and PS when those are completely different servers and not even connected to the PC servers. They don't have any addons.

sshogrin · December 15

KaironBlackbard wrote: »

virtus753 wrote: »

Varana wrote: »

LadyGP wrote: »

Varana wrote: »

No it doesn't. :rolleyes: I have no idea what the reason for your personal crusade is, but just let it go.
The method enables a ton of very useful functionality. And works just fine.

I mean, if an add on someone in group is running is giving someone who doesn't have it performance issues... I would say it does not "work just fine".

I didn't spend a stupid amount of money on my pc to have someone in my group making my latency go through the roof and my system bog down because of the add-ons they decide to run.

Apart from what others have written, I was talking about the general method of using map pings to share information in the group. That works just fine, as numerous addons use it without causing issues.

If Bandits does something wonky with it (or another method) - which, again, hasn't actually been proven yet - that's another topic. But the guy was calling for the whole method to be shut down for whatever personal grudge, not just investigating the addon.

If it wasn’t shut down when Hodor took it to the point of remotely controlling other players’ characters, I don’t foresee it getting shut down for this.

That's creepy

It was a feature that is no longer in the addon that allowed the raid leader to keep other players in the group from attacking the boss or trash mob. It didn't "take control of other player's characters" like you think, it just prevented group members from starting a fight (using abilities and light/heavy attacks) until tanks/group were ready. You have a lot of people spamming abilities before a fight when the raid leader is explaining a fight or getting people assigned to a task, ie. portals, bridge, etc. As a raid leader myself, I have seen a lot of early pulls by impatient players spamming abilities, this is also distracting when trying to explain stuff, and a lot of times the people spamming stuff end up being the people that don't listen to the mechanics and end up being dead the most because they aren't paying attention.

Jaraal · December 15

StihlReign wrote: »

Dump Akamai, move to an in-house Microsoft custom solution. Tune and replicate that system, license it, make billions, the end.

Yeah, they could use Azure in-house for their security needs. Not sure why they prefer to pay Akamai, but I am sure that we will never find out.

sshogrin · December 15

Last night we ran vLC twice with absolutely no issues, at least one player was playing from the UK. The night before we had a full group of players doing vRG, all living in the US and we had 6 people DC at the same time during Bahsia. Main tank DCd during Oax. Before that run we ran vKA with nobody having any issues.
Some people had Bandits running, some player didn't in all these runs. Both nights the trials started around 7:30 pm CST. The trial that we had the most problems didn't start until 10:45 pm CST.

YandereGirlfriend · December 15

sshogrin wrote: »

KaironBlackbard wrote: »

virtus753 wrote: »

Varana wrote: »

LadyGP wrote: »

Varana wrote: »

No it doesn't. :rolleyes: I have no idea what the reason for your personal crusade is, but just let it go.
The method enables a ton of very useful functionality. And works just fine.

I mean, if an add on someone in group is running is giving someone who doesn't have it performance issues... I would say it does not "work just fine".

I didn't spend a stupid amount of money on my pc to have someone in my group making my latency go through the roof and my system bog down because of the add-ons they decide to run.

Apart from what others have written, I was talking about the general method of using map pings to share information in the group. That works just fine, as numerous addons use it without causing issues.

If Bandits does something wonky with it (or another method) - which, again, hasn't actually been proven yet - that's another topic. But the guy was calling for the whole method to be shut down for whatever personal grudge, not just investigating the addon.

If it wasn’t shut down when Hodor took it to the point of remotely controlling other players’ characters, I don’t foresee it getting shut down for this.

That's creepy

It was a feature that is no longer in the addon that allowed the raid leader to keep other players in the group from attacking the boss or trash mob. It didn't "take control of other player's characters" like you think, it just prevented group members from starting a fight (using abilities and light/heavy attacks) until tanks/group were ready. You have a lot of people spamming abilities before a fight when the raid leader is explaining a fight or getting people assigned to a task, ie. portals, bridge, etc. As a raid leader myself, I have seen a lot of early pulls by impatient players spamming abilities, this is also distracting when trying to explain stuff, and a lot of times the people spamming stuff end up being the people that don't listen to the mechanics and end up being dead the most because they aren't paying attention.

Was that feature on an opt-in basis? Or did it also impact non-users if a user was grouped with them? Goes without saying that the latter would be extremely problematic and invasive. Things like Loot Spy are already bad enough.

Because one of the issues that keeps surfacing here is non-users being (potentially) impacted by users of add-ons in ways that they 100% do not consent to. And that (potentially) impacting their performance/game experience. As I've said before, the rights of add-on users end when they start to touch other players.

Like, I categorically do NOT consent to being sent arbitrary data by other players via any mechanism, no matter how big or small it is or how efficiently or inefficiently it is supposedly being done. Full stop.

A large part of that is out of principle but also what an add-on dev considers to be trivial bandwidth to their Google Fiber connection in suburban LA might not be so trivial to someone on a rickety satellite connection from rural Idaho or someone on a fixed data plan.

Aldoss · December 16

@ZOS_RichLambert if what you need is footage, then fine. I'll start taking it to make you believe it.

BGs have never had so much lag as they do now. I main a necro and I think we have the class toolkit with the most number of ground targeted aoes in it:

- Colossus
- Boneyard
- Remote Totem
- Renewing Undeath
- Animate Blastbones

That's a lot!

I was happy when the feature to toggle auto-cast ground targeted aoes got added. It made playing necro a much more enjoyable experience.

However, this patch has been awful and I play 99% of my time in BGs.

Here's my build's set up:

Blighted Blastbones > bar swap > Boneyard > bar swap > Colossus

Never before have I had to sit and wait an extra GCD for boneyard to register. Never before have I witnessed boneyard fire AFTER my colossus fires. Never before have I witnessed my bar swap cancel my Boneyard placement. Never before have I witnessed the yellow placement circle pulse and then disappear without anything happening.

And this is happening in 4v4 BGs, 33% fewer players than the old 4v4v4 BG lobbies. Obviously, it also happens in 8v8 BGs.

This is Cyro levels of lag, which BGs used to be a reprieve from, that has now become a norm in BGs.

I'm sorry to say, your data is wrong. The lag is 100% real and it's gotten much worse in the last couple patches. The BG lag is new this patch. It never used to happen prior to U44.

I'll start recording my sessions for you.

sshogrin · December 16

YandereGirlfriend wrote: »

sshogrin wrote: »

KaironBlackbard wrote: »

virtus753 wrote: »

Varana wrote: »

LadyGP wrote: »

Varana wrote: »

No it doesn't. :rolleyes: I have no idea what the reason for your personal crusade is, but just let it go.
The method enables a ton of very useful functionality. And works just fine.

I mean, if an add on someone in group is running is giving someone who doesn't have it performance issues... I would say it does not "work just fine".

I didn't spend a stupid amount of money on my pc to have someone in my group making my latency go through the roof and my system bog down because of the add-ons they decide to run.

Apart from what others have written, I was talking about the general method of using map pings to share information in the group. That works just fine, as numerous addons use it without causing issues.

If Bandits does something wonky with it (or another method) - which, again, hasn't actually been proven yet - that's another topic. But the guy was calling for the whole method to be shut down for whatever personal grudge, not just investigating the addon.

If it wasn’t shut down when Hodor took it to the point of remotely controlling other players’ characters, I don’t foresee it getting shut down for this.

That's creepy

It was a feature that is no longer in the addon that allowed the raid leader to keep other players in the group from attacking the boss or trash mob. It didn't "take control of other player's characters" like you think, it just prevented group members from starting a fight (using abilities and light/heavy attacks) until tanks/group were ready. You have a lot of people spamming abilities before a fight when the raid leader is explaining a fight or getting people assigned to a task, ie. portals, bridge, etc. As a raid leader myself, I have seen a lot of early pulls by impatient players spamming abilities, this is also distracting when trying to explain stuff, and a lot of times the people spamming stuff end up being the people that don't listen to the mechanics and end up being dead the most because they aren't paying attention.

Was that feature on an opt-in basis? Or did it also impact non-users if a user was grouped with them? Goes without saying that the latter would be extremely problematic and invasive. Things like Loot Spy are already bad enough.

Because one of the issues that keeps surfacing here is non-users being (potentially) impacted by users of add-ons in ways that they 100% do not consent to. And that (potentially) impacting their performance/game experience. As I've said before, the rights of add-on users end when they start to touch other players.

Like, I categorically do NOT consent to being sent arbitrary data by other players via any mechanism, no matter how big or small it is or how efficiently or inefficiently it is supposedly being done. Full stop.

A large part of that is out of principle but also what an add-on dev considers to be trivial bandwidth to their Google Fiber connection in suburban LA might not be so trivial to someone on a rickety satellite connection from rural Idaho or someone on a fixed data plan.

that feature did not affect anyone not grouped up or that did not have the addon running. It was also not something that could be turned on or off by other players in the group, and was for the raid leader to control impatient players. Again, I raid lead, and there are many times that someone pulls that shouldn't pull because they're spamming abilities being impatient. You can tell people to not do it in voice, but they'll act like children and do it anyway. Even when you explain that what they're doing will pull the boss. Again, that feature was removed a very long time ago.
You aren't being sent anything, it's all on the server. It's like anything in base game, believe it or not, you send all types of data to the server without your knowledge, that's base game functions.
I live way out in the country, I used to have just a 10Mbs connection. with 1Mbs up, there are some addons that will cause issues, but those mainly affect just the end user and not other players.
If you actually watch your bandwidth, it doesn't really go above 250Kbs up or down (unless you're downloading a patch update). This game is more upload intensive than download intensive. You will see spikes when zoning somewhere, but it's always been like that, and it doesn't matter how great of connection (or poor) you have.
The biggest thing is to not have the same features running on multiple addons. While some devs think their addon contains "all you need", they don't.
There has been a comment on the BanditsUI ESOUI page made recently about all this, we'll see how long it takes Hoft to respond to it. I do think that Bandits should use the LibMapPing to be consistent with other addons so we aren't overlapping something.
With that being said, when you have the same problems going on on console or in groups that don't use BanditsUI, I'd have to say that pointing the finger at one addon is misguided considering console doesn't have addons, and when groups run without it and they have the same problems, it's not necessarily and addon issue and points to something else completely.
I have asked people that keep pointing their fingers at an addon to explain why this happens on console that doesn't even use the same server hardware as PC, and not one person has responded, totally ignoring the request to explain. So if somebody would like to explain why people get booted from game, have frame drops, latency spikes, login errors/issues on console that are the same as PC, I'm all ears. Console doesn't have access to addons, so if they're having issues, it's I don't see this as being addon related. Remember that consoles are more consistent as to the hardware the game runs on, less variation on both PS and Xbox, in reality, they should be a much more stable platform than PC.

Aurielle · December 16

LadyGP wrote: »

Aurielle wrote: »

I find it interesting that separate threads keep popping up every day now about how people are getting disconnected, experiencing skill lag and rubberbanding, etc... Related to the issue those of us posting here have been experiencing since May, or separate issue? I mean, it sounds identical to what we’ve all been experiencing…

Agreed. With the exception of the last few days (datacenter issue) there has been a steady stream of lag/rubberbanding/ping threads over the last few months.

This goes back to my point of - I think this is a much larger issue and is hitting a lot more people than we know (many don't even realize their ping spikes because they don't have the interface enabled or simply just say "I dc'ed logging back in now" and never report their experience).

I'd also love to know if these random error occurred messages are showing up in logs anywhere. If these aren't I hope there is a way ZoS can start tracking these. I'm sure that number would be massive compared to the other "crashes".

Yeah, I agree that it is likely underreported. Take this Reddit post as an example:

This person obviously doesn’t realize that what they are experiencing is latency. I can’t remember if the ping meter is available on console, but I expect they’d have good ol’ red 999+ at the bottom of their screen if they had it enabled… This person likely has not bothered to submit a report to support. At least we know it’s not addon-related.

zenonuk · December 16

In case it helps with anything ... every time I've had issues over recent days, I triggered Windows' own wait chain analysis... always it's stuck waiting on network I/O. Not sure this is helpful, may not help pin-point the root cause, but may be indicative of some dodgy code in the client that doesn't handle things too well?

It's only ESO that exhibits issues on my rig, no other game suffers with the latency problem, no other AAA title suffers from drops in frame rate - several of us often see fps drop below 10fps in some trials.

In case it was related to drivers, I've even refreshed and reinstalled Windows, drivers, etc. along with ESO itself... took a while... but still the issues exist with ESO. They've been apparent on my system since the most recent client update. No idea if that's just coincidence.

StihlReign · December 16

Over 20 boots from the servers and counting since the reboot. Tons of relog errors...200s, 300s take your pick. Fighting, not fighting, in dungeons, PvE or Cyrodiil...looking at a menu just now. ERROR

10:05 EST 12/16/2024

dk_dunkirk · December 16

YandereGirlfriend wrote: »

sshogrin wrote: »

KaironBlackbard wrote: »

virtus753 wrote: »

Varana wrote: »

LadyGP wrote: »

Varana wrote: »

No it doesn't. :rolleyes: I have no idea what the reason for your personal crusade is, but just let it go.
The method enables a ton of very useful functionality. And works just fine.

I mean, if an add on someone in group is running is giving someone who doesn't have it performance issues... I would say it does not "work just fine".

I didn't spend a stupid amount of money on my pc to have someone in my group making my latency go through the roof and my system bog down because of the add-ons they decide to run.

Apart from what others have written, I was talking about the general method of using map pings to share information in the group. That works just fine, as numerous addons use it without causing issues.

If Bandits does something wonky with it (or another method) - which, again, hasn't actually been proven yet - that's another topic. But the guy was calling for the whole method to be shut down for whatever personal grudge, not just investigating the addon.

If it wasn’t shut down when Hodor took it to the point of remotely controlling other players’ characters, I don’t foresee it getting shut down for this.

That's creepy

It was a feature that is no longer in the addon that allowed the raid leader to keep other players in the group from attacking the boss or trash mob. It didn't "take control of other player's characters" like you think, it just prevented group members from starting a fight (using abilities and light/heavy attacks) until tanks/group were ready. You have a lot of people spamming abilities before a fight when the raid leader is explaining a fight or getting people assigned to a task, ie. portals, bridge, etc. As a raid leader myself, I have seen a lot of early pulls by impatient players spamming abilities, this is also distracting when trying to explain stuff, and a lot of times the people spamming stuff end up being the people that don't listen to the mechanics and end up being dead the most because they aren't paying attention.

Was that feature on an opt-in basis? Or did it also impact non-users if a user was grouped with them? Goes without saying that the latter would be extremely problematic and invasive. Things like Loot Spy are already bad enough.

Because one of the issues that keeps surfacing here is non-users being (potentially) impacted by users of add-ons in ways that they 100% do not consent to. And that (potentially) impacting their performance/game experience. As I've said before, the rights of add-on users end when they start to touch other players.

Like, I categorically do NOT consent to being sent arbitrary data by other players via any mechanism, no matter how big or small it is or how efficiently or inefficiently it is supposedly being done. Full stop.

A large part of that is out of principle but also what an add-on dev considers to be trivial bandwidth to their Google Fiber connection in suburban LA might not be so trivial to someone on a rickety satellite connection from rural Idaho or someone on a fixed data plan.

Someone correct me if I'm wrong, but I don't think mods can do ANYTHING other than operate on what they receive from the servers. With Loot Log, the ESO servers are sending everyone's loot to everyone's client. If they have a mod to read that data, then they can look at it. If that's a problem, then that needs to be taken up with ZOS. I don't think there's any mechanism in mods to push data back to the servers (other than automate input that the user could have generated anyway), and there's certainly no way for them to send data to someone else's client.

LadyGP · December 16

Aurielle wrote: »

LadyGP wrote: »

Aurielle wrote: »

I find it interesting that separate threads keep popping up every day now about how people are getting disconnected, experiencing skill lag and rubberbanding, etc... Related to the issue those of us posting here have been experiencing since May, or separate issue? I mean, it sounds identical to what we’ve all been experiencing…

Agreed. With the exception of the last few days (datacenter issue) there has been a steady stream of lag/rubberbanding/ping threads over the last few months.

This goes back to my point of - I think this is a much larger issue and is hitting a lot more people than we know (many don't even realize their ping spikes because they don't have the interface enabled or simply just say "I dc'ed logging back in now" and never report their experience).

I'd also love to know if these random error occurred messages are showing up in logs anywhere. If these aren't I hope there is a way ZoS can start tracking these. I'm sure that number would be massive compared to the other "crashes".

Yeah, I agree that it is likely underreported. Take this Reddit post as an example:

This person obviously doesn’t realize that what they are experiencing is latency. I can’t remember if the ping meter is available on console, but I expect they’d have good ol’ red 999+ at the bottom of their screen if they had it enabled… This person likely has not bothered to submit a report to support. At least we know it’s not addon-related.

EXACTLY!

That's why when Rich said something to the effect of... they aren't seeing it as a widespread issue... I'm convinced it is. I don't think their logging tools are able to capture these "blips" in their current form. OR they have so much "all is good" data that it's masking the "this ain't right" data.

zenonuk wrote: »

In case it helps with anything ... every time I've had issues over recent days, I triggered Windows' own wait chain analysis... always it's stuck waiting on network I/O. Not sure this is helpful, may not help pin-point the root cause, but may be indicative of some dodgy code in the client that doesn't handle things too well?

It's only ESO that exhibits issues on my rig, no other game suffers with the latency problem, no other AAA title suffers from drops in frame rate - several of us often see fps drop below 10fps in some trials.

In case it was related to drivers, I've even refreshed and reinstalled Windows, drivers, etc. along with ESO itself... took a while... but still the issues exist with ESO. They've been apparent on my system since the most recent client update. No idea if that's just coincidence.

I second this post. I haven't ran the tool you're talking about but it 100% feels like ESO is waiting for network I/O (my water hose reference a few posts up). Also can confirm it's not a driver/hardware issue as I literally (I'm not exaggerating) have built 4 different machines, different brands of Mobo's/CPU/GPU, etc. and ALL had the issue with ESO ONLY. I even changed my ISP 3 times to see and it didn't change a thing.