Yeah, things felt a bit better. I just ended a few hours of Cyro and with multiple ball groups going at it fighting for a scroll I only ran into a few spikes. The biggest thing is i stayed connected during those spikes and didn't crash like usual. I can handle the spikes.. just keep me connected.
Really curious if @ZOS_Kevin or anyone can chime in and say if something was changed - it sure felt like it. Fingers crossed this is a start for some improvement.
ZOS_JessicaFolsom wrote: »Hey everyone, we've been seeing a recent increase in malicious activity targeting our servers, so we have made adjustments on our end to address this threat. Because of this, you may experience decreased game performance while these adjustments are in place. Please note that this is not related to the datacenter power failure last week.
We will continue to monitor for any malicious activity, and thank you for your patience and understanding.
Yeah, things felt a bit better. I just ended a few hours of Cyro and with multiple ball groups going at it fighting for a scroll I only ran into a few spikes. The biggest thing is i stayed connected during those spikes and didn't crash like usual. I can handle the spikes.. just keep me connected.
Really curious if @ZOS_Kevin or anyone can chime in and say if something was changed - it sure felt like it. Fingers crossed this is a start for some improvement.
So we have adjusted some things to deal with malicious activity. That is the main stuff we have done as of recent. Like during the weekend when we had those disconnect issues on Saturday.ZOS_JessicaFolsom wrote: »Hey everyone, we've been seeing a recent increase in malicious activity targeting our servers, so we have made adjustments on our end to address this threat. Because of this, you may experience decreased game performance while these adjustments are in place. Please note that this is not related to the datacenter power failure last week.
We will continue to monitor for any malicious activity, and thank you for your patience and understanding.
KaironBlackbard wrote: »KaironBlackbard wrote: »KaironBlackbard wrote: »@sshogrin @KaironBlackbard
"The addons only get information from the ZOS servers, they don't travel to ZOS servers, then back to your computer, than back to ZOS, then back to them. The addons only get information like I stated directly from the servers that are part of the base game API. "
This is incorrect. KaironBlackbard has correctly observed the 'map ping backchannel communication' method. Map pings are used by addons which want to do data sharing outside the stuff provided by the game. In particular, any addon which has ultimate sharing (such as Hodors) or other realtime group data sharing (group DPS) will send map pings to the group which encode the data. It's a rather brilliant way to bypass the lack of any dedicated data passing API. Sometimes if you have your map up you can see the ping marker dancing around.. This is also (likely) why some group DPS trackers will sometimes show an absurd DPS score from users who have Bandit's UI installed: there is no firm protocol to prevent interfering with another addon's backchannel messaging.
I think there is an addon library which tries to standardize the process (might be internal to hodors), but it's optional and may not be written to minimize the bandwidth usage across a group. I have only looked at it in passing, thought "that is really clever" and realized I didn't need it for what I was writing at the time.
It might be possible for ZOS to limit the frequency of map pings before they are issued from the addon API, or de-prioritize the messages. At the API layer, an attempt to send a map ping could report failure if one had been sent too recently. That would prevent server spamming without kicking anyone and have no impact on regular users, so it might be a valid approach.
@LadyGP if you are not grouped, it should not be possible to get map ping backchannel stuff.
I had a feeling it was a backdoor.
Backdoor needs to be locked.
this is not a "back door", it's backchannel which is much different. Read a post of mine that I replied to @thaellin about this.
Now please explain to me why disconnects, latency, low frame rates, getting booted from server are happening on Xbox and PS when those are completely different servers and not even connected to the PC servers. They don't have any addons.
Because it's a different issue.
It's not a one size fits all.
It's multiple problems, which needs multiple different solutions.
Here in Florida, we don't have those latency problems like you do in California.
Xbox works flawlessly.
However, it is quite evident when someone has BanditsUI (PC), and in dungeons it often snails or even crashes my game.
All you gotta do is see them auto pinging while their map isn't open, and every time I've asked they've responded BanditsUI.
Over 30 times in the past week.
Almost every group I get, someone has BanditsUI.
I'm tired of seeing those autopings, I'm tired of getting slowed with them in my team, I'm tired of it.
And there's nothing I can do to resist their influence. Nothing I can do to stop the auto pings. Nothing I can do to resist their calls and their info thefts.
I want options to allow us to resist, to allow us to decline their requests, to decline BanditsUI pulling from my system every second.
It's not affecting console, It's the only thing affecting me.
And I'm tired of it.
It's not affecting console? Really? I see posts in this thread from console users, I even see the posts on the ESO FB group page from console players...so it IS happening on console.
Again, I have run with groups that not one person was running Bandits and we had people booted from game, lags, frame rate drops, etc.
You need to take your blinders off and maybe look at what others are posting instead of your posts being "me, me, me".
Again, there is a comment to the ESOUI page for BanditsUI about this, we'll see if there's a response other than what is posted on the about page of the addon.
Again, you have failed to explain why this is happening on PS and XBox, which it IS happening on. The hardware is not the same on the server side. You being in denial of it happening on console doesn't help either.
Because the BanditsUI Issue is NOT the same issue.
It's one of a thousand issues.
You're telling me to take blinders off?
If you think it's a one problem for all thing, you're the one with blinders on.
There are a thousand problems, not all are related.
BanditsUI is just one of many problems.
One of many.
It doesn't affect console because console doesn't have addons. Can you prove otherwise?
Though I have myself said that if its just multiple virtual servers on one hardware, it could cause problems.
I'm just trying to get the others saying converse to chill.
Most are claiming the hardware is different.
You and I are saying but what if it's not.
They don't care.
And I don't like repeating myself.
Edit: I only get a 999+ ping when I swap zones. Otherwise no issues.
.
dk_dunkirk wrote: »YandereGirlfriend wrote: »KaironBlackbard wrote: »No it doesn't. :rolleyes: I have no idea what the reason for your personal crusade is, but just let it go.
The method enables a ton of very useful functionality. And works just fine.
I mean, if an add on someone in group is running is giving someone who doesn't have it performance issues... I would say it does not "work just fine".
I didn't spend a stupid amount of money on my pc to have someone in my group making my latency go through the roof and my system bog down because of the add-ons they decide to run.
Apart from what others have written, I was talking about the general method of using map pings to share information in the group. That works just fine, as numerous addons use it without causing issues.
If Bandits does something wonky with it (or another method) - which, again, hasn't actually been proven yet - that's another topic. But the guy was calling for the whole method to be shut down for whatever personal grudge, not just investigating the addon.
If it wasn’t shut down when Hodor took it to the point of remotely controlling other players’ characters, I don’t foresee it getting shut down for this.
That's creepy
It was a feature that is no longer in the addon that allowed the raid leader to keep other players in the group from attacking the boss or trash mob. It didn't "take control of other player's characters" like you think, it just prevented group members from starting a fight (using abilities and light/heavy attacks) until tanks/group were ready. You have a lot of people spamming abilities before a fight when the raid leader is explaining a fight or getting people assigned to a task, ie. portals, bridge, etc. As a raid leader myself, I have seen a lot of early pulls by impatient players spamming abilities, this is also distracting when trying to explain stuff, and a lot of times the people spamming stuff end up being the people that don't listen to the mechanics and end up being dead the most because they aren't paying attention.
Was that feature on an opt-in basis? Or did it also impact non-users if a user was grouped with them? Goes without saying that the latter would be extremely problematic and invasive. Things like Loot Spy are already bad enough.
Because one of the issues that keeps surfacing here is non-users being (potentially) impacted by users of add-ons in ways that they 100% do not consent to. And that (potentially) impacting their performance/game experience. As I've said before, the rights of add-on users end when they start to touch other players.
Like, I categorically do NOT consent to being sent arbitrary data by other players via any mechanism, no matter how big or small it is or how efficiently or inefficiently it is supposedly being done. Full stop.
A large part of that is out of principle but also what an add-on dev considers to be trivial bandwidth to their Google Fiber connection in suburban LA might not be so trivial to someone on a rickety satellite connection from rural Idaho or someone on a fixed data plan.
Besides, there are many posts in this thread that point out that people are having the same lag and disconnect problems on consoles. I don't understand why everyone is focused so hard on mods. One or more of them may exacerbate the problems for their users on PC, but the overall problem is clearly not related to mods.
My understanding (as someone who comes back to this thread mostly on a weekly basis) is that everyone is actually not focused on mods — less than a handful are, but they are being quite loud about it, so it just feels like mods are the focus. A number of us have pointed out that we don’t use Bandits and that console users can’t use addons period, and yet…. Bandits, Bandits, Bandits.
Well put.
Clearly there is smoke there with Bandits or add-ons in general or else ZoS wouldn't be investigating.
In parallel with that there is obviously other issues at play because people on ALL systems are having issues.
I am experiencing huge lag spikes, rubberbanding, mobs not processing visually and constant loading screens if I walk a little faster than baby steps. All in PvE, independently of the zone or amount of players on it, it happens constantly.
I am quite sure this has to do with the servers issue as in 10 years I was always able to play flawlessly.
I know my current setup is not the perfect one for this kind of game, but until now it was finely managing to run PvE without this much trouble even on dolmens and dungeons.
I have some hope, knowing how ZoS works, that this will be looked further (is this maintenance to stabilize the servers? Pretty please, with cherry on top?)
Thank you for everything guys!
edward_frigidhands wrote: »dk_dunkirk wrote: »YandereGirlfriend wrote: »KaironBlackbard wrote: »No it doesn't. :rolleyes: I have no idea what the reason for your personal crusade is, but just let it go.
The method enables a ton of very useful functionality. And works just fine.
I mean, if an add on someone in group is running is giving someone who doesn't have it performance issues... I would say it does not "work just fine".
I didn't spend a stupid amount of money on my pc to have someone in my group making my latency go through the roof and my system bog down because of the add-ons they decide to run.
Apart from what others have written, I was talking about the general method of using map pings to share information in the group. That works just fine, as numerous addons use it without causing issues.
If Bandits does something wonky with it (or another method) - which, again, hasn't actually been proven yet - that's another topic. But the guy was calling for the whole method to be shut down for whatever personal grudge, not just investigating the addon.
If it wasn’t shut down when Hodor took it to the point of remotely controlling other players’ characters, I don’t foresee it getting shut down for this.
That's creepy
It was a feature that is no longer in the addon that allowed the raid leader to keep other players in the group from attacking the boss or trash mob. It didn't "take control of other player's characters" like you think, it just prevented group members from starting a fight (using abilities and light/heavy attacks) until tanks/group were ready. You have a lot of people spamming abilities before a fight when the raid leader is explaining a fight or getting people assigned to a task, ie. portals, bridge, etc. As a raid leader myself, I have seen a lot of early pulls by impatient players spamming abilities, this is also distracting when trying to explain stuff, and a lot of times the people spamming stuff end up being the people that don't listen to the mechanics and end up being dead the most because they aren't paying attention.
Was that feature on an opt-in basis? Or did it also impact non-users if a user was grouped with them? Goes without saying that the latter would be extremely problematic and invasive. Things like Loot Spy are already bad enough.
Because one of the issues that keeps surfacing here is non-users being (potentially) impacted by users of add-ons in ways that they 100% do not consent to. And that (potentially) impacting their performance/game experience. As I've said before, the rights of add-on users end when they start to touch other players.
Like, I categorically do NOT consent to being sent arbitrary data by other players via any mechanism, no matter how big or small it is or how efficiently or inefficiently it is supposedly being done. Full stop.
A large part of that is out of principle but also what an add-on dev considers to be trivial bandwidth to their Google Fiber connection in suburban LA might not be so trivial to someone on a rickety satellite connection from rural Idaho or someone on a fixed data plan.
Besides, there are many posts in this thread that point out that people are having the same lag and disconnect problems on consoles. I don't understand why everyone is focused so hard on mods. One or more of them may exacerbate the problems for their users on PC, but the overall problem is clearly not related to mods.
My understanding (as someone who comes back to this thread mostly on a weekly basis) is that everyone is actually not focused on mods — less than a handful are, but they are being quite loud about it, so it just feels like mods are the focus. A number of us have pointed out that we don’t use Bandits and that console users can’t use addons period, and yet…. Bandits, Bandits, Bandits.
Well put.
Clearly there is smoke there with Bandits or add-ons in general or else ZoS wouldn't be investigating.
In parallel with that there is obviously other issues at play because people on ALL systems are having issues.
Can someone link me the post stating ZOS is investigating BanditUI?
ZOS_RichLambert wrote: »Hey folks - here's some answers to the main questions in this thread, as well as some insights to our process and what we're working on to help us better debug the issue(s). Thanks for the questions and please keep on providing feedback!
What things do you look at when you get reports about in-game performance issues?
Performance issues and their related investigations are complex, and there are loads of things we look at. Just about anything could be a factor when it comes to performance issues. Generally, the big ones we look at are:
- The server(s) where the issue is occurring
- What time(s) the issue occurred
- Machine specs for impacted players
- What affected players were doing at the time the issue occurred
- What addons affected players have installed. All these things combined help us get a full picture of most issues. And from there, we expand our list of what we look at if we need more information, as we have for this particular issue. This includes requesting additional information from affected players and also implementing new logging when necessary.
What do the in-game performance charts look like now?
The ones that I personally look at the most (crashes, long load times, and server frames) are all pretty solid overall. We have a few crashes we’re addressing, but they are rarely occurring. The long load times reports, especially on console are very small and server frames – how bogged down the server is also very stable, at least outside of prime time. During prime time, there are a few spikes that we’d like to figure out and are implementing more tooling (logging) in the next few incremental patches to try to get more info for our engineers.
What information is most helpful to get when investigating in-game performance issues?
Sharing with us the date and time the issue occurred (ideally to the minute) as well as what you were doing at the time is the most helpful. It’s also a huge bonus if you have a video/clip of it so we can see exactly what is being described. We have been watching the video clips you’ve been sharing with us for this issue – thank you!
What are you able to change when in-game performance issues occur?
It really depends on what the issue is. We are able to hotfix some things (hotfix = no server downtime) but most require an outage. Crashes, broken items/spawns and exploits are generally “easier” to find a fix and get something out quickly. “Input lag”, “high ping” or “low frame rate” type issues are much harder to pinpoint and often take a great deal of time to find and fix. However, in the case of input lag or high ping, if it’s a result of an attack on the servers, we are able to turn adjust mitigation whenever we need.
Why do investigations sometimes take a long time?
There’s lots of different reasons, some bugs are easy to repro, others are not. Load related bugs (in general the input lag or high ping type issues) are the most difficult and time consuming to track down. Remember ESO is a HUGE game – millions of lines of code (over 25mil) to sift through. Then you have countless different machine types, drivers, internet types/locations in the world, addons... etc. to make bug hunting more complex.
On top of that, massive servers with hundreds of thousands of players on them at any one time is really, really hard to replicate internally. It’s tough!
How many different groups have been involved in investigating this issue?
Quite a few people are involved in investigations for ongoing game performance issues, including people from our BI (Business Intelligence) group, Community, Customer Support, Game Design, Engineering, QA and Live Services teams.
Has anyone at ZOS been able to replicate this issue?
We have not been able to consistently replicate this issue on the live servers, or at all on our internal servers due to some live server factors such as load being involved. While some of our developers have experienced it on the live servers alongside our players, there’s been no clear and consistent “smoking gun” root cause we’ve been able to identify so far.
Can someone confirm the role of the AI chat monitoring here?
We’ve extensively investigated if this could have an impact on game performance. The way the tool works, it sifts through chat logs that are exported from the game. So, it isn’t integrated into ESO, and never actually touches the game or the chat servers.
Regarding the tool itself, it helps us identify at risk chat and areas of concern faster. But our agents make the determination on whether an account is suspended or banned. We do have some auto kick rules for chat spam, but anything automated is temporary. Our agents review every automatic action against account history before taking any permanent account action.
We have also been using the tool for a couple years and nothing has changed in how we log chat, so the timing doesn’t line up. The situation a couple months ago with increased actioning related to chat logs was more a case of changes to our processes and training up new personnel than with the tool itself. Ultimately, though, yes we did look at if the tool was having any noticeable impact to game performance and it’s not.
What is your process like when an in-game performance issue is reported?
The first step is to gather as much info as we can. Is this just one person running into it, or is it multiple? Is it region/location specific? Platform specific? Does this only happen during prime time? Once we have all the info available, we then start to work on internal repro. If we can find a way to repro it, we’re generally able to get a fix together quickly.
What work has been completed or is in progress so far?
Outside of the ongoing investigations based on the information all of you have provided, we’ve been working on lots of logging additions for the most part. None of these are going to solve any problems on their own, but they should provide us with better mechanisms for helping to better diagnose the problem.
- Server logging - specifically collecting more metrics about a player's experience from within an instance/region. (number of messages per frame, avg and max bytes sent & received...etc.)
- Client logging of alternate quit methods. Clients closed via Alt+F4 or the window 'X' button terminate instantly and the server wouldn't know it was intentional and would just be considered a client timeout. (This is to aid in false-positive detection.)
- HeartbeatDisconnector (basically a message that checks to see if your client is still connected) should accept any message as traffic, not just heartbeats. If you get a ton of messages, it's rare but possible your heartbeat doesn't get processed in time and you get disconnected.
- Present a more accurate ping in the game client. Our latency meter is not reflective of what the actual ping is. It's currently tied to framerate and load screens and what not.
- Investigate the Bandits UI addon. (In the top 15 most used add-ons currently) There are lots of forum posts pointing to this add-on for potential latency issues. The suspicion is that the addon is using map pings as somewhat of a backdoor to send arbitrary data to clients, regardless of whether players have the add-on installed. The pings would get broadcast to everyone and cause game performance problems (latency & frame rate).
Yes I can prove otherwise, go back thru this thread alone and look at the console players commenting on here. It's been said by ZOS the servers are different.
As far as the blinders, I've looked at your posts, you keep saying Bandits and your internet connection, even when others have explained about addons, you keep telling everybody it's Bandits.
There are multiple issues with what's going on, and the biggest problem was introduced earlier this year.
As far as repeating yourself, I think I've repeated myself a number of times about console having issues (both platforms, and they should be more consistent than PC since their hardware is more "uniform" on each platform than PC is)
I have messaged Hoft directly through ESOUI about this, and have made a comment on that addon page.
Again, I have ran with groups that don't use Bandits, ran with a mix, and ran with a full group of Bandits users...there's times we've had issues in every combination, including NO Bandits, then we've had absolutely no issues running with a mix of full Bandits users.
As an addon developer, I do know something about the addons, how they work, and what they can and can't do.
I have also posted on the BanditsUI page about not using Bandits for dps sharing or for notifications.
There are other addons that could potentially be causing issues, Bandits could be since it's not using LibMapPing, but my experience so far is it doesn't matter either way. I run trials in the "end game" community, doing vet HMs where you have a lot more going on, 12 players vs 4, etc. Trust me, there's a lot more visual things going on in HM trials, a lot more visuals from other players, a lot more data going on, if things are going to go wonky, that's where they will really show up. A lot of "end game" trial players reduce graphics settings, particles, distance, etc., to combat the issues of all the things going on in trials.
Just popping in to say that I consistently experience lag in the Wayrest Sewers dungeons. For some reason, this is particularly bad when I'm in Wayrest 1 solo (I don't solo 2). When I'm grouped in 1 or 2, there's some lag but not as bad. Today, fighting the final boss in 1, it was like I was in a disco with the strobing when it's action, freeze, action, freeze. And I was frozen for long periods of time. Pretty much unplayable. Fortunately I still managed to complete it without dying.
This isn't my internet connection. I run at least one dungeon a day without these problems. But Wayrest Sewers is always laggy.
Hard to know if this is because of the above issues Kevin mentioned or the stuff we have been highlighting in this thread since May. How long have you been experiencing those issues? That sounds like the typical "rubberbanding" that most of us deal with on the regular.
Hard to know if this is because of the above issues Kevin mentioned or the stuff we have been highlighting in this thread since May. How long have you been experiencing those issues? That sounds like the typical "rubberbanding" that most of us deal with on the regular.
Not rubberbanding. Freezing. I've posted in this thread several times already about other places I experience spikes, lags, etc. Kevin asked for specifics, so I'm providing them. I commonly experience problems in Wayrest Sewers.
KaironBlackbard wrote: »
Yes I can prove otherwise, go back thru this thread alone and look at the console players commenting on here. It's been said by ZOS the servers are different.
As far as the blinders, I've looked at your posts, you keep saying Bandits and your internet connection, even when others have explained about addons, you keep telling everybody it's Bandits.
There are multiple issues with what's going on, and the biggest problem was introduced earlier this year.
As far as repeating yourself, I think I've repeated myself a number of times about console having issues (both platforms, and they should be more consistent than PC since their hardware is more "uniform" on each platform than PC is)
I have messaged Hoft directly through ESOUI about this, and have made a comment on that addon page.
Again, I have ran with groups that don't use Bandits, ran with a mix, and ran with a full group of Bandits users...there's times we've had issues in every combination, including NO Bandits, then we've had absolutely no issues running with a mix of full Bandits users.
As an addon developer, I do know something about the addons, how they work, and what they can and can't do.
I have also posted on the BanditsUI page about not using Bandits for dps sharing or for notifications.
There are other addons that could potentially be causing issues, Bandits could be since it's not using LibMapPing, but my experience so far is it doesn't matter either way. I run trials in the "end game" community, doing vet HMs where you have a lot more going on, 12 players vs 4, etc. Trust me, there's a lot more visual things going on in HM trials, a lot more visuals from other players, a lot more data going on, if things are going to go wonky, that's where they will really show up. A lot of "end game" trial players reduce graphics settings, particles, distance, etc., to combat the issues of all the things going on in trials.
Your experience does not match mine.
I don't have any problems whatsoever unless someone is spam pinging my compass.
Every time, they are a BanditsUI user.
And I don't have any problems on Xbox at all.
You keep trying to claim it's not bandits, when clearly it is.
My problem lies with bandits.
It is not the consensus, but it is one of many problems.
It may not be your problem, but it definitely is mine.
And I have no control over who is using bandits.
Claim about console having issue have NO relevance to this specific issue of mine. They are different issues.
Yeah, it's the framerate of the actual gameplay simulation on the server side. Sometimes also called a tick or a heartbeat. Usually, singleplayer-only games that need to have very deterministic results are going to decouple their rendering from the actual game logic such that the game logic runs at a fixed rate of time per tick. This naturally extends to multiplayer games with a client and server, where there is no rendering on the server but there will be asynchronous housekeeping for processing incoming and outgoing batches of data. The gameplay simulation logic obviously isn't going to be tied to any of that but will run independently at it's own fixed rate of time.dk_dunkirk wrote: »I've been programming for 45 years, and I've been a full-stack developer for 30. Can someone tell me what a "server frame" is? The context seems to be networking- or load-related. But even with my background, I can't tell what this is referring to. Maybe it means the number of servers (frame/box/chassis) that need to be running to handle the load?
Well, the good news lasted for a day... Issues are back to normal... One day of fun seemed to be enough...