ZOS, Massive Spike in Ping/LAG in Recent Days - What Gives?

Quackery · December 13

AAAAND...the power outage getting fixed didn't do anything for my game. It's still horrid with 999+ ping every 20-30 seconds.

LadyGP · December 13

Quackery wrote: »

AAAAND...the power outage getting fixed didn't do anything for my game. It's still horrid with 999+ ping every 20-30 seconds.

Honestly, I wouldn't have expected it to resolve anything. Rebooting all the servers isn't going to change routing/code. If anything it would release some of the "bloat" that bogs down the servers between maintenance but those effects dwindle after a day or so.

ZOS_RichLambert · December 13

Hey folks - here's some answers to the main questions in this thread, as well as some insights to our process and what we're working on to help us better debug the issue(s). Thanks for the questions and please keep on providing feedback!

What things do you look at when you get reports about in-game performance issues?
Performance issues and their related investigations are complex, and there are loads of things we look at. Just about anything could be a factor when it comes to performance issues. Generally, the big ones we look at are:

The server(s) where the issue is occurring

What time(s) the issue occurred

Machine specs for impacted players

What affected players were doing at the time the issue occurred

What addons affected players have installed. All these things combined help us get a full picture of most issues. And from there, we expand our list of what we look at if we need more information, as we have for this particular issue. This includes requesting additional information from affected players and also implementing new logging when necessary.

What do the in-game performance charts look like now?
The ones that I personally look at the most (crashes, long load times, and server frames) are all pretty solid overall. We have a few crashes we’re addressing, but they are rarely occurring. The long load times reports, especially on console are very small and server frames – how bogged down the server is also very stable, at least outside of prime time. During prime time, there are a few spikes that we’d like to figure out and are implementing more tooling (logging) in the next few incremental patches to try to get more info for our engineers.

What information is most helpful to get when investigating in-game performance issues?
Sharing with us the date and time the issue occurred (ideally to the minute) as well as what you were doing at the time is the most helpful. It’s also a huge bonus if you have a video/clip of it so we can see exactly what is being described. We have been watching the video clips you’ve been sharing with us for this issue – thank you!

What are you able to change when in-game performance issues occur?
It really depends on what the issue is. We are able to hotfix some things (hotfix = no server downtime) but most require an outage. Crashes, broken items/spawns and exploits are generally “easier” to find a fix and get something out quickly. “Input lag”, “high ping” or “low frame rate” type issues are much harder to pinpoint and often take a great deal of time to find and fix. However, in the case of input lag or high ping, if it’s a result of an attack on the servers, we are able to turn adjust mitigation whenever we need.

Why do investigations sometimes take a long time?
There’s lots of different reasons, some bugs are easy to repro, others are not. Load related bugs (in general the input lag or high ping type issues) are the most difficult and time consuming to track down. Remember ESO is a HUGE game – millions of lines of code (over 25mil) to sift through. Then you have countless different machine types, drivers, internet types/locations in the world, addons... etc. to make bug hunting more complex.
On top of that, massive servers with hundreds of thousands of players on them at any one time is really, really hard to replicate internally. It’s tough!

How many different groups have been involved in investigating this issue?
Quite a few people are involved in investigations for ongoing game performance issues, including people from our BI (Business Intelligence) group, Community, Customer Support, Game Design, Engineering, QA and Live Services teams.

Has anyone at ZOS been able to replicate this issue?
We have not been able to consistently replicate this issue on the live servers, or at all on our internal servers due to some live server factors such as load being involved. While some of our developers have experienced it on the live servers alongside our players, there’s been no clear and consistent “smoking gun” root cause we’ve been able to identify so far.

Can someone confirm the role of the AI chat monitoring here?
We’ve extensively investigated if this could have an impact on game performance. The way the tool works, it sifts through chat logs that are exported from the game. So, it isn’t integrated into ESO, and never actually touches the game or the chat servers.
Regarding the tool itself, it helps us identify at risk chat and areas of concern faster. But our agents make the determination on whether an account is suspended or banned. We do have some auto kick rules for chat spam, but anything automated is temporary. Our agents review every automatic action against account history before taking any permanent account action.
We have also been using the tool for a couple years and nothing has changed in how we log chat, so the timing doesn’t line up. The situation a couple months ago with increased actioning related to chat logs was more a case of changes to our processes and training up new personnel than with the tool itself. Ultimately, though, yes we did look at if the tool was having any noticeable impact to game performance and it’s not.

What is your process like when an in-game performance issue is reported?
The first step is to gather as much info as we can. Is this just one person running into it, or is it multiple? Is it region/location specific? Platform specific? Does this only happen during prime time? Once we have all the info available, we then start to work on internal repro. If we can find a way to repro it, we’re generally able to get a fix together quickly.

What work has been completed or is in progress so far?
Outside of the ongoing investigations based on the information all of you have provided, we’ve been working on lots of logging additions for the most part. None of these are going to solve any problems on their own, but they should provide us with better mechanisms for helping to better diagnose the problem.

Server logging - specifically collecting more metrics about a player's experience from within an instance/region. (number of messages per frame, avg and max bytes sent & received...etc.)

Client logging of alternate quit methods. Clients closed via Alt+F4 or the window 'X' button terminate instantly and the server wouldn't know it was intentional and would just be considered a client timeout. (This is to aid in false-positive detection.)

HeartbeatDisconnector (basically a message that checks to see if your client is still connected) should accept any message as traffic, not just heartbeats. If you get a ton of messages, it's rare but possible your heartbeat doesn't get processed in time and you get disconnected.

Present a more accurate ping in the game client. Our latency meter is not reflective of what the actual ping is. It's currently tied to framerate and load screens and what not.

Investigate the Bandits UI addon. (In the top 15 most used add-ons currently) There are lots of forum posts pointing to this add-on for potential latency issues. The suspicion is that the addon is using map pings as somewhat of a backdoor to send arbitrary data to clients, regardless of whether players have the add-on installed. The pings would get broadcast to everyone and cause game performance problems (latency & frame rate).

sshogrin · December 13

KaironBlackbard wrote: »

sshogrin wrote: »

KaironBlackbard wrote: »

sshogrin wrote: »

There is no "assuming" as you claim...there is no peer to peer communications with addons. The addons don't "store data" on the ZOS servers, it's information that the game itself transmits from client to server and server to client. .

Then why is it that when someone in my group has BanditsUI active, my system becomes sluggish when they don't have their map open, and at the same time they are spamming ping on the compass and on a random point on the map that they don't have open?

It's making calls, clear as day.

So what you're saying is that when someone is grouped with you and they open their map, it makes YOUR system sluggish if they don't have Bandits...do you have any out of date addons?

Opposite.
THEY have BanditsUI.
I don't.
I only have: Map Pins, Lorebooks, TTC, USPF. Nothing else.
When THEY are running around doing things, my system becomes sluggish, and there is an auto ping every second in their name.
While they have their map open, the pings stop, and my system returns to normal.
When they disable BanditsUI, the pings stop, and my system returns to normal.

My most memorable situation regarding this:
I'm on my psijic sorc trying to solo nFangLair, I'm thinking no big deal right?
I took a beating from the ad groups, but no deathed to the first boss.
Boss is just ad waves.
Ad wave 1, owned me. Ok, try again.
Made it into wave 2, then got owned.
Again. Made it to wave 3, then got owned.
Again. 3 again. Ok, need help.
Published a group finder.
about 10 minutes goes by and I get a dps.
Skipping common combat.
We make it halfway through and we get a tank.
When he enters our instance, his character starts pinging (ctrl+click default keybind) somewhere off the coast of Vvardenfell every second.
My system becomes sluggish.
Ping only lasts 2 seconds btw.
We make it to the Lich boss, and boy did I struggle to keep heals on them, practically 1 frame per second.
We defeat that boss, and I ask him what addons he's using, explaining that my system became sluggish when he joined.
He goes through one by one toggling his addons.
Ok, before that I explained to him about the pings and when he'd open his map to look the pings would stop and my system would return to normal.
Back to toggling.
He toggled at least 5 before the pings stopped.
He toggled a few more as I typed "there"
He had to backtrack toggling a couple times until he redisabled the one that cleared it.
"that one."
"which one is that?"
his reply: "BanditsUI"

Now for my brother's encounter:
He did a trial with some guildies around the beginning of the golden event.
at least 5 of them had auto pings going, and his system crashed every 15 seconds.
each time he logged in he tried to explain in half messages (because he'd get booted too often for a detailed message) that he's seeing auto pings and getting booted. I don't remember much about his conversation, except this next part:
He sent "Are any of y'all running BanditsUI?"
The next time he logged in, most of their auto pings had stopped, and he was able to play with them.
They disabled their BanditsUI for him.

That's how I know it makes calls, or something similar to that anyways.
I'm thinking the ping is a visualization of it making a request/call for information about another user/ally.
It requests more information than is sent to the server, or at least more often than is sent to the server thus calling on others machines.
Since my connection is pathetic, it causes sluggish connection and even loss of connection.

Addons do not make calls to another player's computer or client, that's not how this all works. The information about the class, race, dps, heals, death, etc. is part of the base game.
I had a 10Mbs down 1Mbs up connection thru Centurylink, worse than what you have. I had issues with 2 addons, those addons are no longer in my folder.
Again, I have ran in groups that nobody was using Bandits and people were getting kicked from game, bad latency and frame rate spikes as well. I've run in groups with a mix of Bandits, no UI addon, and other UI addons and haven't had any issues. I've been in groups where almost everybody had BanditsUI and no issues. It's not consistent. Slip's Sanity's Edge Assist is known for kicking people from game and giving the user bad latency/frame rate issues.
I'd have to look at the BanditsUI settings to see if there's some auto-ping, but honestly I don't remember an "auto-ping" in there. The addons only get information from the ZOS servers, they don't travel to ZOS servers, then back to your computer, than back to ZOS, then back to them. The addons only get information like I stated directly from the servers that are part of the base game API.

LadyGP · December 13

Massive thanks to @ZOS_Kevin @ZOS_JessicaFolsom and anyone else that had input in getting this typed up. Although there aren't a lot of answers to the "why" (and I'm sure people will get frustrated) it needs to be noted that this kind of communication is awesome and I appreciate the behind the scenes look into how the sausage is made - I hope this trend continues into 2025.

I will be honest, I expected things like Server Logging and HearBeatDisconnector to have been already an existing tool that was being utilized to learning that it isn't is kind of surprising.

Out of curiosity - is this kind of logging something that has existed and is just turned off/on as needed or is this something new that is being developed/implemented?

ZOS_RichLambert wrote: »

Server logging - specifically collecting more metrics about a player's experience from within an instance/region. (number of messages per frame, avg and max bytes sent & received...etc.)

Is this already up and running? If so, has there been anything that has jumped out in the data? Random thought, but if it would help I would be willing to play ESO, record to a private YouTube, and share it with engineers if there was a way they could put specific logging on my account.

Looking at the overall data it's going to be harder to nail down the affected users - maybe reach out to a few that are willing to help and see if we could do some closer 1 on 1 testing/logging? Just throwing that out there.

ZOS_RichLambert wrote: »

Client logging of alternate quit methods. Clients closed via Alt+F4 or the window 'X' button terminate instantly and the server wouldn't know it was intentional and would just be considered a client timeout. (This is to aid in false-positive detection.)

I know for me there are a lot of times the client just hard crashes without giving me an error or times it just hangs and the only way I can close it is with an Alt+F4.

ZOS_RichLambert wrote: »

HeartbeatDisconnector (basically a message that checks to see if your client is still connected) should accept any message as traffic, not just heartbeats. If you get a ton of messages, it's rare but possible your heartbeat doesn't get processed in time and you get disconnected.

Curious to see what this captures. I know for me when I'm playing and I "feel" the latency start to happen I have to take my hands off my mouse/keyboard and sit there. If I sit and wait for the "server to catch up".. so to speak... and then start playing again.. I don't crash. If things start to bog down and I keep moving/firing skills then there is a 100% chance my ping wont settle down and I crash.

ZOS_RichLambert wrote: »

Present a more accurate ping in the game client. Our latency meter is not reflective of what the actual ping is. It's currently tied to framerate and load screens and what not.

Interesting.

ZOS_RichLambert wrote: »

Investigate the Bandits UI addon. (In the top 15 most used add-ons currently) There are lots of forum posts pointing to this add-on for potential latency issues. The suspicion is that the addon is using map pings as somewhat of a backdoor to send arbitrary data to clients, regardless of whether players have the add-on installed. The pings would get broadcast to everyone and cause game performance problems (latency & frame rate).

Now this...this has my attention. So, I've never ran Bandits UI. If I'm tracking correctly you're saying there is reports that people who run this add on could unknowingly be giving people who don't run it high latency? As someone who still gets this issue when I'm not in a group - would Bandits still "hit" me? How does this work?

Aashiana · December 13

Thanks for the specific questions and the detailed answers from Rich. This kind of conversation is really helpful to other players observing.

Destai · December 13

Thanks Rich for the update! I can see a few of my questions there, happy they’ve been answered and meaningful to the conversation. Will need to think on those answers and will post anymore questions as time allows. Overall, this looks like a great reporting framework that hopefully can be used to report statuses going forward.

YandereGirlfriend · December 13

Gotta give a huge ovation to the MVP @LadyGP for starting this thread and for keeping it alive all this time!

And to the courageous @KaironBlackbard for continuing to bang the add-ons/Bandits drum even in the face of intense opposition. So many told them, "But it couldn't possibly be add-ons!" and yet... here we are. And if Bandits is misbehaving it almost assuredly is not the only one to be doing so.

Finally, to everyone else who has commented here and helped keep the thread active and alive. It definitely FeelsGoodMan to see some action on something so important. And to see the exchange of information happening between the community and the devs.

LadyGP · December 13

YandereGirlfriend wrote: »

Gotta give a huge ovation to the MVP @LadyGP for starting this thread and for keeping it alive all this time!

And to the courageous @KaironBlackbard for continuing to bang the add-ons/Bandits drum even in the face of intense opposition. So many told them, "But it couldn't possibly be add-ons!" and yet... here we are. And if Bandits is misbehaving it almost assuredly is not the only one to be doing so.

Finally, to everyone else who has commented here and helped keep the thread active and alive. It definitely FeelsGoodMan to see some action on something so important. And to see the exchange of information happening between the community and the devs.

Thanks for the kind words and thank you to everyone for coming in here and keeping up your experiences. I will say, as someone who said "it's not the add-ons" (because I've had this issue out of group with no add-ons installed on my machine) I'm a bit taken back by the fact that OTHER peoples add-ons could be giving me issues. I'm curious how that works since I'm out of group and there isn't a link between me and the other player (that I'm aware of anyways).

Also agree, I think having this kind of open dialogue between devs and the community is awesome. I'd love to see some Senior level devs/engineers get in here and solicit feedback once and a while (or have @ZOS_Kevin pass that on) but get technical with us... get in the weeds with us. Yes, a lot of people won't understand what is being said but not everyone needs to.

Side note - One thing I forgot to bring up in my reply to @ZOS_RichLambert was has there been any investigation/logging into if Akamai (sp) is causing any problems?

LadyGP · December 13

Aurielle wrote: »

Out of date addons are likely completely unrelated to this issue. I’ve never used BanditsUI. Ever. Here’s what happened: the game received an update in May, and I experienced frequent ping and connection issues from that point onwards, whether grouped or solo without ANY other players in my immediate vicinity.

The one thing almost all of us have in common is that the performance problems we’re experiencing in game all started around the same time this year in May. Pointing fingers at specific addons just unnecessarily muddies the waters, IMO, especially given that many console players are also having the same problems and literally cannot use addons.

This. So much this. Something happened around this time ( a few weeks after this thread was made) that made this issue pop up for the majority of us. I'm not sure "what" changed but something did around that time. Hopefully @ZOS_Kevin or someone can take that info back and the team can backtrack on updates. Maybe there will be an "AH HA" kinda moment?

sarahthes wrote: »

Aurielle wrote: »

Out of date addons are likely completely unrelated to this issue. I’ve never used BanditsUI. Ever. Here’s what happened: the game received an update in May, and I experienced frequent ping and connection issues from that point onwards, whether grouped or solo without ANY other players in my immediate vicinity.

The one thing almost all of us have in common is that the performance problems we’re experiencing in game all started around the same time this year in May. Pointing fingers at specific addons just unnecessarily muddies the waters, IMO, especially given that many console players are also having the same problems and literally cannot use addons.

Are you still having those issues? I'm genuinely curious, because in the NA endgame raiding scene at least, server/lag problems have died down a lot.

Yes

LadyGP · December 13

sshogrin wrote: »

KaironBlackbard wrote: »

sshogrin wrote: »

KaironBlackbard wrote: »

sshogrin wrote: »

There is no "assuming" as you claim...there is no peer to peer communications with addons. The addons don't "store data" on the ZOS servers, it's information that the game itself transmits from client to server and server to client. .

Then why is it that when someone in my group has BanditsUI active, my system becomes sluggish when they don't have their map open, and at the same time they are spamming ping on the compass and on a random point on the map that they don't have open?

It's making calls, clear as day.

So what you're saying is that when someone is grouped with you and they open their map, it makes YOUR system sluggish if they don't have Bandits...do you have any out of date addons?

Opposite.
THEY have BanditsUI.
I don't.
I only have: Map Pins, Lorebooks, TTC, USPF. Nothing else.
When THEY are running around doing things, my system becomes sluggish, and there is an auto ping every second in their name.
While they have their map open, the pings stop, and my system returns to normal.
When they disable BanditsUI, the pings stop, and my system returns to normal.

My most memorable situation regarding this:
I'm on my psijic sorc trying to solo nFangLair, I'm thinking no big deal right?
I took a beating from the ad groups, but no deathed to the first boss.
Boss is just ad waves.
Ad wave 1, owned me. Ok, try again.
Made it into wave 2, then got owned.
Again. Made it to wave 3, then got owned.
Again. 3 again. Ok, need help.
Published a group finder.
about 10 minutes goes by and I get a dps.
Skipping common combat.
We make it halfway through and we get a tank.
When he enters our instance, his character starts pinging (ctrl+click default keybind) somewhere off the coast of Vvardenfell every second.
My system becomes sluggish.
Ping only lasts 2 seconds btw.
We make it to the Lich boss, and boy did I struggle to keep heals on them, practically 1 frame per second.
We defeat that boss, and I ask him what addons he's using, explaining that my system became sluggish when he joined.
He goes through one by one toggling his addons.
Ok, before that I explained to him about the pings and when he'd open his map to look the pings would stop and my system would return to normal.
Back to toggling.
He toggled at least 5 before the pings stopped.
He toggled a few more as I typed "there"
He had to backtrack toggling a couple times until he redisabled the one that cleared it.
"that one."
"which one is that?"
his reply: "BanditsUI"

Now for my brother's encounter:
He did a trial with some guildies around the beginning of the golden event.
at least 5 of them had auto pings going, and his system crashed every 15 seconds.
each time he logged in he tried to explain in half messages (because he'd get booted too often for a detailed message) that he's seeing auto pings and getting booted. I don't remember much about his conversation, except this next part:
He sent "Are any of y'all running BanditsUI?"
The next time he logged in, most of their auto pings had stopped, and he was able to play with them.
They disabled their BanditsUI for him.

That's how I know it makes calls, or something similar to that anyways.
I'm thinking the ping is a visualization of it making a request/call for information about another user/ally.
It requests more information than is sent to the server, or at least more often than is sent to the server thus calling on others machines.
Since my connection is pathetic, it causes sluggish connection and even loss of connection.

Addons do not make calls to another player's computer or client, that's not how this all works. The information about the class, race, dps, heals, death, etc. is part of the base game.
I had a 10Mbs down 1Mbs up connection thru Centurylink, worse than what you have. I had issues with 2 addons, those addons are no longer in my folder.
Again, I have ran in groups that nobody was using Bandits and people were getting kicked from game, bad latency and frame rate spikes as well. I've run in groups with a mix of Bandits, no UI addon, and other UI addons and haven't had any issues. I've been in groups where almost everybody had BanditsUI and no issues. It's not consistent. Slip's Sanity's Edge Assist is known for kicking people from game and giving the user bad latency/frame rate issues.
I'd have to look at the BanditsUI settings to see if there's some auto-ping, but honestly I don't remember an "auto-ping" in there. The addons only get information from the ZOS servers, they don't travel to ZOS servers, then back to your computer, than back to ZOS, then back to them. The addons only get information like I stated directly from the servers that are part of the base game API.

Honestly, I'm starting to think there are multiple layers to this that are going on. Something with Bandits is screwing with peoples performance - theres a decent amount of evidence/complaints about that so that is one side of the coin. The other side of the coin is something around May happened that kicked this off.

code65536 · December 13

@ZOS_RichLambert I'm surprised that there's no mention of connectivity in general and Akamai in particular. My game performance on PC/NA has been pretty consistently good, and the times that I've had issues were times when the problem was with the connection to ZOS (e.g., times when Akamai sends my packets on a tour around the world before delivering it to the datacenter). I know that there was a surge of comments in this thread a few months during a period when multiple people were having these kinds of "problem exists between you and the server" issues.

sarahthes · December 13

code65536 wrote: »

@ZOS_RichLambert I'm surprised that there's no mention of connectivity in general and Akamai in particular. My game performance on PC/NA has been pretty consistently good, and the times that I've had issues were times when the problem was with the connection to ZOS (e.g., times when Akamai sends my packets on a tour around the world before delivering it to the datacenter). I know that there was a surge of comments in this thread a few months during a period when multiple people were having these kinds of "problem exists between you and the server" issues.

I have a suspicion (that may be completely wrong LOL) that when that happens there is a DDoS attack happening at the same time. And you're just getting caught up in it and the protective measures.

thaellin · December 13

@sshogrin @KaironBlackbard

"The addons only get information from the ZOS servers, they don't travel to ZOS servers, then back to your computer, than back to ZOS, then back to them. The addons only get information like I stated directly from the servers that are part of the base game API. "

This is incorrect. KaironBlackbard has correctly observed the 'map ping backchannel communication' method. Map pings are used by addons which want to do data sharing outside the stuff provided by the game. In particular, any addon which has ultimate sharing (such as Hodors) or other realtime group data sharing (group DPS) will send map pings to the group which encode the data. It's a rather brilliant way to bypass the lack of any dedicated data passing API. Sometimes if you have your map up you can see the ping marker dancing around.. This is also (likely) why some group DPS trackers will sometimes show an absurd DPS score from users who have Bandit's UI installed: there is no firm protocol to prevent interfering with another addon's backchannel messaging.

I think there is an addon library which tries to standardize the process (might be internal to hodors), but it's optional and may not be written to minimize the bandwidth usage across a group. I have only looked at it in passing, thought "that is really clever" and realized I didn't need it for what I was writing at the time.

It might be possible for ZOS to limit the frequency of map pings before they are issued from the addon API, or de-prioritize the messages. At the API layer, an attempt to send a map ping could report failure if one had been sent too recently. That would prevent server spamming without kicking anyone and have no impact on regular users, so it might be a valid approach.

@LadyGP if you are not grouped, it should not be possible to get map ping backchannel stuff.

LadyGP · December 13

thaellin wrote: »

@sshogrin @KaironBlackbard

"The addons only get information from the ZOS servers, they don't travel to ZOS servers, then back to your computer, than back to ZOS, then back to them. The addons only get information like I stated directly from the servers that are part of the base game API. "

This is incorrect. KaironBlackbard has correctly observed the 'map ping backchannel communication' method. Map pings are used by addons which want to do data sharing outside the stuff provided by the game. In particular, any addon which has ultimate sharing (such as Hodors) or other realtime group data sharing (group DPS) will send map pings to the group which encode the data. It's a rather brilliant way to bypass the lack of any dedicated data passing API. Sometimes if you have your map up you can see the ping marker dancing around.. This is also (likely) why some group DPS trackers will sometimes show an absurd DPS score from users who have Bandit's UI installed: there is no firm protocol to prevent interfering with another addon's backchannel messaging.

I think there is an addon library which tries to standardize the process (might be internal to hodors), but it's optional and may not be written to minimize the bandwidth usage across a group. I have only looked at it in passing, thought "that is really clever" and realized I didn't need it for what I was writing at the time.

It might be possible for ZOS to limit the frequency of map pings before they are issued from the addon API, or de-prioritize the messages. At the API layer, an attempt to send a map ping could report failure if one had been sent too recently. That would prevent server spamming without kicking anyone and have no impact on regular users, so it might be a valid approach.

@LadyGP if you are not grouped, it should not be possible to get map ping backchannel stuff.

Dang. I was hoping that this Bandits stuff would some how be my issue and there would be a way to fix it.

Also curious oh if they have any idea why VPNs tend to give some of us relief from these issues. I've had times where I put on a VPN to Kansas and was able to play all night with no issues at all. I've had other times where the VPN made it worse (which I'd expect a VPN to increase latency).

manukartofanu · December 14

KaironBlackbard wrote: »

manukartofanu wrote: »

KaironBlackbard wrote: »

sshogrin wrote: »

There is no "assuming" as you claim...there is no peer to peer communications with addons. The addons don't "store data" on the ZOS servers, it's information that the game itself transmits from client to server and server to client. .

Then why is it that when someone in my group has BanditsUI active, my system becomes sluggish when they don't have their map open, and at the same time they are spamming ping on the compass and on a random point on the map that they don't have open?

It's making calls, clear as day.

Once again: no, and we are absolutely certain about this. How could it happen that another player's requests cause your requests to enter a corrupted state or not be processed? There could be plenty of reasons, and they depend on the specifics of the implementation. If we need to come up with a general example of how this could happen, then sure, it's easy. For instance, a deadlock might occur.

15Mb connection, 1.5Mb upload speed. That's why. The extra calls it's requesting cannot be handled by my internet service.
ESO only uses 20000 packets on my router.
It jumps to over 60000 when someone has BanditsUI enabled in my group.
Explain that.
I can only think of that it's making calls.
It returns to normal when they disable their BanditsUI.
It's not a corrupted state, it's overload, which is causing my router or ISP to dump packets. Router never goes over 45% so I'm going to assume ISP is dumping. Definitely gets overloaded around hop 3 quite often. Qwest hop.

You lack the technical knowledge to evaluate the situation and have fixated on the idea that is not supported by ESO’s architecture. It’s easy to come up with an alternative explanation for the increase in traffic.
For example, the measurements might not have been conducted under the same conditions. Network traffic increases when events occur in the game and spikes significantly during combat.
It may be server or network congestion.
It’s also possible that some processes on server enter a fault state, delay responses, and clients, not receiving the necessary data, begin repeating requests, which increases traffic. And yes, this could happen because other group members are requesting additional data from the server using BanditsUI.
And many, many other possibilities, since we don’t even know what, when, and how you measured it.

Yes, and once again, ESO’s architecture is not up for speculation; we know exactly what addons can and cannot do. But I won’t write any more on this topic—I’m tired of it. If you don’t want to understand the situation and just want to argue without any knowledge, that’s your choice.

silky_soft · December 14

Has anyone at ZOS been able to replicate this issue?
We have not been able to consistently replicate this issue on the live servers, or at all on our internal servers due to some live server factors such as load being involved. While some of our developers have experienced it on the live servers alongside our players, there’s been no clear and consistent “smoking gun” root cause we’ve been able to identify so far.

So if it's not internal, it's akamai then. Still waiting for dev team to VPN outside US.

YandereGirlfriend · December 14

manukartofanu wrote: »

KaironBlackbard wrote: »

manukartofanu wrote: »

KaironBlackbard wrote: »

sshogrin wrote: »

There is no "assuming" as you claim...there is no peer to peer communications with addons. The addons don't "store data" on the ZOS servers, it's information that the game itself transmits from client to server and server to client. .

Then why is it that when someone in my group has BanditsUI active, my system becomes sluggish when they don't have their map open, and at the same time they are spamming ping on the compass and on a random point on the map that they don't have open?

It's making calls, clear as day.

Once again: no, and we are absolutely certain about this. How could it happen that another player's requests cause your requests to enter a corrupted state or not be processed? There could be plenty of reasons, and they depend on the specifics of the implementation. If we need to come up with a general example of how this could happen, then sure, it's easy. For instance, a deadlock might occur.

15Mb connection, 1.5Mb upload speed. That's why. The extra calls it's requesting cannot be handled by my internet service.
ESO only uses 20000 packets on my router.
It jumps to over 60000 when someone has BanditsUI enabled in my group.
Explain that.
I can only think of that it's making calls.
It returns to normal when they disable their BanditsUI.
It's not a corrupted state, it's overload, which is causing my router or ISP to dump packets. Router never goes over 45% so I'm going to assume ISP is dumping. Definitely gets overloaded around hop 3 quite often. Qwest hop.

You lack the technical knowledge to evaluate the situation and have fixated on the idea that is not supported by ESO’s architecture. It’s easy to come up with an alternative explanation for the increase in traffic.
For example, the measurements might not have been conducted under the same conditions. Network traffic increases when events occur in the game and spikes significantly during combat.
It may be server or network congestion.
It’s also possible that some processes on server enter a fault state, delay responses, and clients, not receiving the necessary data, begin repeating requests, which increases traffic. And yes, this could happen because other group members are requesting additional data from the server using BanditsUI.
And many, many other possibilities, since we don’t even know what, when, and how you measured it.

Yes, and once again, ESO’s architecture is not up for speculation; we know exactly what addons can and cannot do. But I won’t write any more on this topic—I’m tired of it. If you don’t want to understand the situation and just want to argue without any knowledge, that’s your choice.

Suggest that you read up a few posts.

Aurielle · December 14

Appreciate the update from Rich. Quite frankly, that’s a level of communication we should have had about this problem months ago, given how disruptive it has been to those of us experiencing it since it first started back in May. But hey, better late than never.

To be perfectly honest, I feel pretty dejected — even more so than I did when I uninstalled the game a couple months ago. It sounds like high latency and disconnects are actually only being reported by a relatively small number of players, and that a cause isn’t anywhere close to being identified. Since it’s not affecting a large number of players, and since it can’t be replicated, I fear that it’ll eventually be concluded that it’s a user-end issue that can’t be fixed. There’s simply nothing more on my end I could have done, short of trying to see if moving to another area of the country would help… and that’s obviously not going to happen for the sake of a video game of all things, haha. As I’ve mentioned multiple times in this thread, ESO is the only online game that gave me significant connection issues this year. I’ve been able to play other triple A MMORPGs and multiplayer PVP games with no issues whatsoever using the exact same computer setup and internet connection. High latency with skill input lag was, conversely, a daily issue for me in ESO starting in early May. Getting booted from the server wasn’t a daily issue, but it happened frequently enough (especially in Cyrodiil) to be a major source of frustration.

I’ll still check back in every so often to see if there’s anymore progress and try to keep my hopes up, but I can’t really say I’m all that optimistic at this point. It’s too bad, because ESO is still my favourite MMORPG and it was great to return to it last year.

Oh well.

Two things:

1. Why on earth does the in game latency meter not reflect our actual ping? Why even label it “ping” if that’s not what it’s measuring?

2. Can we get some clarification on whether or not the Bandits issue only affects other players grouped with someone running the addon? If an addon is able to increase the latency of ungrouped players who just so happen to be in the vicinity of another player running the addon, then that would go against virtually everything I thought I knew about how addons work in this game.

manukartofanu · December 14

YandereGirlfriend wrote: »

manukartofanu wrote: »

KaironBlackbard wrote: »

manukartofanu wrote: »

KaironBlackbard wrote: »

sshogrin wrote: »

There is no "assuming" as you claim...there is no peer to peer communications with addons. The addons don't "store data" on the ZOS servers, it's information that the game itself transmits from client to server and server to client. .

Then why is it that when someone in my group has BanditsUI active, my system becomes sluggish when they don't have their map open, and at the same time they are spamming ping on the compass and on a random point on the map that they don't have open?

It's making calls, clear as day.

Once again: no, and we are absolutely certain about this. How could it happen that another player's requests cause your requests to enter a corrupted state or not be processed? There could be plenty of reasons, and they depend on the specifics of the implementation. If we need to come up with a general example of how this could happen, then sure, it's easy. For instance, a deadlock might occur.

15Mb connection, 1.5Mb upload speed. That's why. The extra calls it's requesting cannot be handled by my internet service.
ESO only uses 20000 packets on my router.
It jumps to over 60000 when someone has BanditsUI enabled in my group.
Explain that.
I can only think of that it's making calls.
It returns to normal when they disable their BanditsUI.
It's not a corrupted state, it's overload, which is causing my router or ISP to dump packets. Router never goes over 45% so I'm going to assume ISP is dumping. Definitely gets overloaded around hop 3 quite often. Qwest hop.

You lack the technical knowledge to evaluate the situation and have fixated on the idea that is not supported by ESO’s architecture. It’s easy to come up with an alternative explanation for the increase in traffic.
For example, the measurements might not have been conducted under the same conditions. Network traffic increases when events occur in the game and spikes significantly during combat.
It may be server or network congestion.
It’s also possible that some processes on server enter a fault state, delay responses, and clients, not receiving the necessary data, begin repeating requests, which increases traffic. And yes, this could happen because other group members are requesting additional data from the server using BanditsUI.
And many, many other possibilities, since we don’t even know what, when, and how you measured it.

Yes, and once again, ESO’s architecture is not up for speculation; we know exactly what addons can and cannot do. But I won’t write any more on this topic—I’m tired of it. If you don’t want to understand the situation and just want to argue without any knowledge, that’s your choice.

Suggest that you read up a few posts.

I wrote my post after reading all the others. What exactly was written in them to make you doubt the architecture of how addons work? Yes, there is LibMapPing, and nearly everyone uses it—except for Bandits, who keep all their functionality within a single addon. They just use the same principle.

P.S. Incorrect DPS data isn’t displayed because someone has BanditsUI installed or because they’re using this "backdoor." Just look at how many addons use LibMapPing. If they couldn’t identify whose data is whose, this method of getting "extra" data wouldn’t make the slightest sense.

Just_Attivi · December 14

My question:

Is it only being reported by a small number of players NOW, since many of us have stopped playing altogether and therefore arent reporting it daily/weekly/whatever?

Quackery · December 14

@ZOS_RichLambert

Issue is happening on PC, both EU- and NA-server. I live in Sweden, so the high ping is pretty much consistent from the second I log into the NA-server. EU is more "playable", but it gets worse and worse the longer I play. Logging off and in again doesn't help - it's just as bad.

Time of day is irrelevant, the issue gets progressively worse the longer I play on EU. It gets unplayable after about 20-30 minutes of gameplay with 999+ ping every 20-30 seconds. NA-server is completely unplayable.

CPU: 11th Gen Intel Core i9-11900K
GPU: NVIDIA GeForce RTX 3080 GDDR6X 10GB
PCIe Gen 4 High speed M.2 SSD (1TB), M.2 Gen 3 SSD (2TB)
Memory: 32GB DDR4 RAM XMP4000MHz
2.5bE LAN, Intel Wi-Fi 6 AX200, Bluetooth V5.2
Realtek ALC1220-VB

I've played with both addons activated AND disabled - still same issue. I've played with antivirus/firewall disabled, and the issue still persists.

My problems all began when "Golden Pursuits" went live. Everything was just fine before that day. This sucks, because I play mostly on NA-server, and it's just impossible. The same issue happens on different computers, BUT not on my Xbox. I've contacted my ISP, and everything is fine on both their and my end. I don't know what else I can do except give up and uninstall the game. (Oh yes, reinstalling didn't help, already tried it.)

Edit: I have 1GB Internet speed, and I've tried both ethernet and wifi - still same issue.

manukartofanu · December 14

Great, that we finally got some details about what’s going on. The most important thing here is that the issue is acknowledged, and some work is being done to fix it.
@LadyGP, congratulations.

Jaraal · December 14

Bandits is a red herring, as it obviously doesn't effect people on console who are having identical issues to those on PC.

And yes, it's quite telling that the subject of documented latency and packet loss at Akamai was avoided.

Aurielle · December 14

Just_Attivi wrote: »

My question:

Is it only being reported by a small number of players NOW, since many of us have stopped playing altogether and therefore arent reporting it daily/weekly/whatever?

Good question. I know quite a few people in my Cyrodiil guilds who moved on months ago due to the performance issues. I still see them in Discord, and they’re all still playing other games.

Arunei · December 14

I will be blunt and say I doubt Bandits is a culprit in all this. Could it be making existing issues worse? Yes, if it does wonky things with the API. But if it was an addon causing this stuff, I'd have figured that to be one of the first things checked. Quite literally addons sending too many requests to the server has been an issue in the past, or addons doing things they weren't intended to be able to do. It's why the API had to be changed so those requests would be throttled or otherwise blocked. So I...don't really believe that addons wouldn't have been something they first looked at months ago, and I certainly haven't seen "numerous" people talking about it. I've seen no threads and only seen a small number of people here mentioning it. Perhaps I'm being pessimistic, but I feel the specific mention of Bandits is just to draw attention away from the other potential causes.

I don't use Bandits. None of the people I do stuff with use Bandits. I mostly RP or do Overworld stuff like questing, Event stuff, that sort of thing. If an addon I'm not touching and no one else I do stuff with is touching is somehow causing problems for ALL platforms, that's an addon that should have already been noticed as a problem. I'm certain ZOS can see what sorts of API calls/requests/whatever are being made, and can check those logs whenever they need to. And again, knowing addons HAVE caused issues for everyone in the past...I just find it hard to believe they wouldn't have checked the logs to make sure that wasn't an issue repeating itself with some new addon. Because it truly feels addons would be like...the easiest thing to check and rule out, given they SHOULD be able to access logs about what's going on with the API and if stuff is doing what it shouldn't be.

So yeah, I'm just not buying that they haven't already had time to rule out addons as a culprit. And even if Bandits or some other addon is causing the issue, how would that be impacting console? They have completely separate hardware and stuff, it's not like they're getting hit with PC/Steam server data. Iirc when Master Merchant I believe it was, and before that some PvP addon idr the name of now, were causing server issues, it was only happening on PC/Steam, NOT on console too.

LadyGP · December 14

Just_Attivi wrote: »

My question:

Is it only being reported by a small number of players NOW, since many of us have stopped playing altogether and therefore arent reporting it daily/weekly/whatever?

Very valid question.... I too have stopped playing and I know of at least 7 other people in my group that have, in part because of these problems.

LadyGP · December 14

manukartofanu wrote: »

Great, that we finally got some details about what’s going on. The most important thing here is that the issue is acknowledged, and some work is being done to fix it.
@LadyGP, congratulations.

Agreed!

Rkindaleft · December 14

Appreciate the update @ZOS_RichLambert @ZOS_Kevin however I’m just typing this out because a lot of PSNA is suffering from some pretty severe symptoms the last half hour or so. MT disconnected mid pull and can’t log back in, nearly everyone in my trial group is having skipping and having multiple second barswap delay. Not sure if it’s related to what has been going on previously but thought I should notify.

Just grabbed this from another discord to show it’s not just us and affecting multiple groups. There’s also also talk about it in other guilds in game chat for reference.

baconaura · December 14

Another wasted night. servers are just in a bad state still.

https://forums.elderscrollsonline.com/en/discussion/670509/booted-from-server-pcna#latest