ZOS, Massive Spike in Ping/LAG in Recent Days - What Gives?
Maintenance for the week of March 24:
· [COMPLETE] PC/Mac: NA and EU megaservers for patch maintenance – March 24, 4:00AM EDT (8:00 UTC) - 10:00AM EDT (14:00 UTC)
· Xbox: NA and EU megaservers for patch maintenance – March 26, 6:00AM EDT (10:00 UTC) - 12:00PM EDT (16:00 UTC)
· Playstation®: NA and EU megaservers for patch maintenance – March 26, 6:00AM EDT (10:00 UTC) - 12:00PM EDT (16:00 UTC)
· [COMPLETE] PC/Mac: NA and EU megaservers for patch maintenance – March 24, 4:00AM EDT (8:00 UTC) - 10:00AM EDT (14:00 UTC)
· Xbox: NA and EU megaservers for patch maintenance – March 26, 6:00AM EDT (10:00 UTC) - 12:00PM EDT (16:00 UTC)
· Playstation®: NA and EU megaservers for patch maintenance – March 26, 6:00AM EDT (10:00 UTC) - 12:00PM EDT (16:00 UTC)
-
SolidBeast✭✭✭Just got kicked off the PC EU server while porting. Good thing it's late."Many fall, but one remains."0 -
LadyGP✭✭✭✭✭
✭Yeah, things felt a bit better. I just ended a few hours of Cyro and with multiple ball groups going at it fighting for a scroll I only ran into a few spikes. The biggest thing is i stayed connected during those spikes and didn't crash like usual. I can handle the spikes.. just keep me connected.
Really curious if @ZOS_Kevin or anyone can chime in and say if something was changed - it sure felt like it. Fingers crossed this is a start for some improvement.LadyGP/xCatGuy
PC/NA0 -
cyclonus11✭✭✭✭✭
✭✭Kicked twice so far this evening with a lot of lag/delays (like a second to open a container, etc.) - PC/NAEdited by cyclonus11 on December 17, 2024 1:12AM0 -
ZOS_KevinCommunity ManagerYeah, things felt a bit better. I just ended a few hours of Cyro and with multiple ball groups going at it fighting for a scroll I only ran into a few spikes. The biggest thing is i stayed connected during those spikes and didn't crash like usual. I can handle the spikes.. just keep me connected.
Really curious if @ZOS_Kevin or anyone can chime in and say if something was changed - it sure felt like it. Fingers crossed this is a start for some improvement.
So we have adjusted some things to deal with malicious activity. That is the main stuff we have done as of recent. Like during the weekend when we had those disconnect issues on Saturday.ZOS_JessicaFolsom wrote: »Hey everyone, we've been seeing a recent increase in malicious activity targeting our servers, so we have made adjustments on our end to address this threat. Because of this, you may experience decreased game performance while these adjustments are in place. Please note that this is not related to the datacenter power failure last week.
We will continue to monitor for any malicious activity, and thank you for your patience and understanding.Community Manager for ZeniMax Online Studio and Elder Scrolls OnlineDev Tracker | Service Alerts | ESO Twitter1 -
Quackery✭✭✭✭✭I remember why I stopped playing this game... Ball groups absolutely RUIN the performance!4 -
MagdaV✭✭
So we have adjusted some things to deal with malicious activity. That is the main stuff we have done as of recent. Like during the weekend when we had those disconnect issues on Saturday.
0 -
MagdaV✭✭So things were fine for me until tonight. I have been disconnected around ball groups about every 15 mins! Best part? all there is anymore is ball groups lol I am done for the night. I keep missing AP ticks and it is just not worth the stress. PLEASE @ZOS_Kevin, can we get a timeframe, plan.... something besides the same string of troubleshooting steps? (Which I have done btw)1
-
SaintJohnHM✭✭✭✭Many DCs and much lag in our DSR trifecta group tonight.
ESO is a super fun game when it actually works, but this is just frustrating.- Casual Roleplaying PVE player PC/NA
- Tank CP2400 'Sugar-Flame'
- I've completed all the dungeon trifectas. Godslayer, Gryphon Heart, Immortal Redeemer, Tick Tock Tormentor, Dawnbringer, and I'm looking for nice folks to complete more trial achieves with.
- I make music: http://www.moonghostband.com.
3 -
SaintJohnHM✭✭✭✭And then more big lag and DCs in our IR run tonight.
- Casual Roleplaying PVE player PC/NA
- Tank CP2400 'Sugar-Flame'
- I've completed all the dungeon trifectas. Godslayer, Gryphon Heart, Immortal Redeemer, Tick Tock Tormentor, Dawnbringer, and I'm looking for nice folks to complete more trial achieves with.
- I make music: http://www.moonghostband.com.
0 -
sshogrin✭✭✭KaironBlackbard wrote: »
Because the BanditsUI Issue is NOT the same issue.
It's one of a thousand issues.
You're telling me to take blinders off?
If you think it's a one problem for all thing, you're the one with blinders on.
There are a thousand problems, not all are related.
BanditsUI is just one of many problems.
One of many.
It doesn't affect console because console doesn't have addons. Can you prove otherwise?
Though I have myself said that if its just multiple virtual servers on one hardware, it could cause problems.
I'm just trying to get the others saying converse to chill.
Most are claiming the hardware is different.
You and I are saying but what if it's not.
They don't care.
And I don't like repeating myself.
Yes I can prove otherwise, go back thru this thread alone and look at the console players commenting on here. It's been said by ZOS the servers are different.
As far as the blinders, I've looked at your posts, you keep saying Bandits and your internet connection, even when others have explained about addons, you keep telling everybody it's Bandits.
There are multiple issues with what's going on, and the biggest problem was introduced earlier this year.
As far as repeating yourself, I think I've repeated myself a number of times about console having issues (both platforms, and they should be more consistent than PC since their hardware is more "uniform" on each platform than PC is)
I have messaged Hoft directly through ESOUI about this, and have made a comment on that addon page.
Again, I have ran with groups that don't use Bandits, ran with a mix, and ran with a full group of Bandits users...there's times we've had issues in every combination, including NO Bandits, then we've had absolutely no issues running with a mix of full Bandits users.
As an addon developer, I do know something about the addons, how they work, and what they can and can't do.
I have also posted on the BanditsUI page about not using Bandits for dps sharing or for notifications.
There are other addons that could potentially be causing issues, Bandits could be since it's not using LibMapPing, but my experience so far is it doesn't matter either way. I run trials in the "end game" community, doing vet HMs where you have a lot more going on, 12 players vs 4, etc. Trust me, there's a lot more visual things going on in HM trials, a lot more visuals from other players, a lot more data going on, if things are going to go wonky, that's where they will really show up. A lot of "end game" trial players reduce graphics settings, particles, distance, etc., to combat the issues of all the things going on in trials.
4 -
Renato90085✭✭✭yesterday my group run dsrhm ,when boss start mech(like open bridge) we all got 5sec stuck..
so many wipe from lag
but we finished, and we back training first boss
game run very fine0 -
Sluggy✭✭✭✭✭Edit: I only get a 999+ ping when I swap zones. Otherwise no issues.
.
For the record, this isn't a ping issue and can be safely ignored. It's just the result of averaging some numbers over time where some of that time happens to be during a zone loading - hence the numbers get skewed to the high end for a brief time.
The ping reported in the UI isn't live. It's averaged out over five or ten seconds (I forget which) which is the reason you'll often feel lag and then by the time it recovers you finally see it spike in the display. That's also why it's a good idea to stop everything once you feel the spike because by the time you see the display you're likely to push past the limit that the server allows the client to desync and you run the risk of disconnecting.
3 -
Sakiri✭✭✭✭✭
✭✭What is statshare? HP sharing? Default UI does that.
Ultimate sharing is needed by trial raid leads for calling specific ultimates.
What exactly is it about bandit's that's causing issues? Because I've NEVER had lag or ping issues with running it, or anyone in my group running it, until recently and that leads me to believe it's not bandit's.
And most of the addons you people are complaining about are default options in every other game I've played. It's no wonder they made addons for it here.
1 -
edward_frigidhands✭✭✭✭✭
Well put.
Clearly there is smoke there with Bandits or add-ons in general or else ZoS wouldn't be investigating.
In parallel with that there is obviously other issues at play because people on ALL systems are having issues.
Can someone link me the post stating ZOS is investigating BanditUI?0 -
Adinatha✭✭I am experiencing huge lag spikes, rubberbanding, mobs not processing visually and constant loading screens if I walk a little faster than baby steps. All in PvE, independently of the zone or amount of players on it, it happens constantly.
I am quite sure this has to do with the servers issue as in 10 years I was always able to play flawlessly.
I know my current setup is not the perfect one for this kind of game, but until now it was finely managing to run PvE without this much trouble even on dolmens and dungeons.
I have some hope, knowing how ZoS works, that this will be looked further (is this maintenance to stabilize the servers? Pretty please, with cherry on top?)
Thank you for everything guys!1 -
Sakiri✭✭✭✭✭
✭✭I am experiencing huge lag spikes, rubberbanding, mobs not processing visually and constant loading screens if I walk a little faster than baby steps. All in PvE, independently of the zone or amount of players on it, it happens constantly.
I am quite sure this has to do with the servers issue as in 10 years I was always able to play flawlessly.
I know my current setup is not the perfect one for this kind of game, but until now it was finely managing to run PvE without this much trouble even on dolmens and dungeons.
I have some hope, knowing how ZoS works, that this will be looked further (is this maintenance to stabilize the servers? Pretty please, with cherry on top?)
Thank you for everything guys!
It's maintenance for the winter event.
I had a desync last night though. Was running towards the Undaunted enclave in Mournhold only to get a load screen and teleported back a ways.0 -
Jaraal✭✭✭✭✭
✭✭✭✭✭edward_frigidhands wrote: »
Can someone link me the post stating ZOS is investigating BanditUI?
Last paragraph:
https://forums.elderscrollsonline.com/en/discussion/comment/8237075/#Comment_8237075
Edited by Jaraal on December 17, 2024 1:27PM1 -
dk_dunkirk✭✭✭✭✭ZOS_RichLambert wrote: »Hey folks - here's some answers to the main questions in this thread, as well as some insights to our process and what we're working on to help us better debug the issue(s). Thanks for the questions and please keep on providing feedback!
What things do you look at when you get reports about in-game performance issues?
Performance issues and their related investigations are complex, and there are loads of things we look at. Just about anything could be a factor when it comes to performance issues. Generally, the big ones we look at are:- The server(s) where the issue is occurring
- What time(s) the issue occurred
- Machine specs for impacted players
- What affected players were doing at the time the issue occurred
- What addons affected players have installed. All these things combined help us get a full picture of most issues. And from there, we expand our list of what we look at if we need more information, as we have for this particular issue. This includes requesting additional information from affected players and also implementing new logging when necessary.
What do the in-game performance charts look like now?
The ones that I personally look at the most (crashes, long load times, and server frames) are all pretty solid overall. We have a few crashes we’re addressing, but they are rarely occurring. The long load times reports, especially on console are very small and server frames – how bogged down the server is also very stable, at least outside of prime time. During prime time, there are a few spikes that we’d like to figure out and are implementing more tooling (logging) in the next few incremental patches to try to get more info for our engineers.
What information is most helpful to get when investigating in-game performance issues?
Sharing with us the date and time the issue occurred (ideally to the minute) as well as what you were doing at the time is the most helpful. It’s also a huge bonus if you have a video/clip of it so we can see exactly what is being described. We have been watching the video clips you’ve been sharing with us for this issue – thank you!
What are you able to change when in-game performance issues occur?
It really depends on what the issue is. We are able to hotfix some things (hotfix = no server downtime) but most require an outage. Crashes, broken items/spawns and exploits are generally “easier” to find a fix and get something out quickly. “Input lag”, “high ping” or “low frame rate” type issues are much harder to pinpoint and often take a great deal of time to find and fix. However, in the case of input lag or high ping, if it’s a result of an attack on the servers, we are able to turn adjust mitigation whenever we need.
Why do investigations sometimes take a long time?
There’s lots of different reasons, some bugs are easy to repro, others are not. Load related bugs (in general the input lag or high ping type issues) are the most difficult and time consuming to track down. Remember ESO is a HUGE game – millions of lines of code (over 25mil) to sift through. Then you have countless different machine types, drivers, internet types/locations in the world, addons... etc. to make bug hunting more complex.
On top of that, massive servers with hundreds of thousands of players on them at any one time is really, really hard to replicate internally. It’s tough!
How many different groups have been involved in investigating this issue?
Quite a few people are involved in investigations for ongoing game performance issues, including people from our BI (Business Intelligence) group, Community, Customer Support, Game Design, Engineering, QA and Live Services teams.
Has anyone at ZOS been able to replicate this issue?
We have not been able to consistently replicate this issue on the live servers, or at all on our internal servers due to some live server factors such as load being involved. While some of our developers have experienced it on the live servers alongside our players, there’s been no clear and consistent “smoking gun” root cause we’ve been able to identify so far.
Can someone confirm the role of the AI chat monitoring here?
We’ve extensively investigated if this could have an impact on game performance. The way the tool works, it sifts through chat logs that are exported from the game. So, it isn’t integrated into ESO, and never actually touches the game or the chat servers.
Regarding the tool itself, it helps us identify at risk chat and areas of concern faster. But our agents make the determination on whether an account is suspended or banned. We do have some auto kick rules for chat spam, but anything automated is temporary. Our agents review every automatic action against account history before taking any permanent account action.
We have also been using the tool for a couple years and nothing has changed in how we log chat, so the timing doesn’t line up. The situation a couple months ago with increased actioning related to chat logs was more a case of changes to our processes and training up new personnel than with the tool itself. Ultimately, though, yes we did look at if the tool was having any noticeable impact to game performance and it’s not.
What is your process like when an in-game performance issue is reported?
The first step is to gather as much info as we can. Is this just one person running into it, or is it multiple? Is it region/location specific? Platform specific? Does this only happen during prime time? Once we have all the info available, we then start to work on internal repro. If we can find a way to repro it, we’re generally able to get a fix together quickly.
What work has been completed or is in progress so far?
Outside of the ongoing investigations based on the information all of you have provided, we’ve been working on lots of logging additions for the most part. None of these are going to solve any problems on their own, but they should provide us with better mechanisms for helping to better diagnose the problem.- Server logging - specifically collecting more metrics about a player's experience from within an instance/region. (number of messages per frame, avg and max bytes sent & received...etc.)
- Client logging of alternate quit methods. Clients closed via Alt+F4 or the window 'X' button terminate instantly and the server wouldn't know it was intentional and would just be considered a client timeout. (This is to aid in false-positive detection.)
- HeartbeatDisconnector (basically a message that checks to see if your client is still connected) should accept any message as traffic, not just heartbeats. If you get a ton of messages, it's rare but possible your heartbeat doesn't get processed in time and you get disconnected.
- Present a more accurate ping in the game client. Our latency meter is not reflective of what the actual ping is. It's currently tied to framerate and load screens and what not.
- Investigate the Bandits UI addon. (In the top 15 most used add-ons currently) There are lots of forum posts pointing to this add-on for potential latency issues. The suspicion is that the addon is using map pings as somewhat of a backdoor to send arbitrary data to clients, regardless of whether players have the add-on installed. The pings would get broadcast to everyone and cause game performance problems (latency & frame rate).
I've been programming for 45 years, and I've been a full-stack developer for 30. Can someone tell me what a "server frame" is? The context seems to be networking- or load-related. But even with my background, I can't tell what this is referring to. Maybe it means the number of servers (frame/box/chassis) that need to be running to handle the load?
Also, "hundreds of thousands of players... at any time?" Even if Steam players represent only, say, 15% of all players, the Steam chart numbers suggest a total player count at most in the "tens of thousands." Has ZOS EVER released any official numbers on this?Edited by dk_dunkirk on December 17, 2024 3:04PM6 -
KaironBlackbard✭✭✭
Yes I can prove otherwise, go back thru this thread alone and look at the console players commenting on here. It's been said by ZOS the servers are different.
As far as the blinders, I've looked at your posts, you keep saying Bandits and your internet connection, even when others have explained about addons, you keep telling everybody it's Bandits.
There are multiple issues with what's going on, and the biggest problem was introduced earlier this year.
As far as repeating yourself, I think I've repeated myself a number of times about console having issues (both platforms, and they should be more consistent than PC since their hardware is more "uniform" on each platform than PC is)
I have messaged Hoft directly through ESOUI about this, and have made a comment on that addon page.
Again, I have ran with groups that don't use Bandits, ran with a mix, and ran with a full group of Bandits users...there's times we've had issues in every combination, including NO Bandits, then we've had absolutely no issues running with a mix of full Bandits users.
As an addon developer, I do know something about the addons, how they work, and what they can and can't do.
I have also posted on the BanditsUI page about not using Bandits for dps sharing or for notifications.
There are other addons that could potentially be causing issues, Bandits could be since it's not using LibMapPing, but my experience so far is it doesn't matter either way. I run trials in the "end game" community, doing vet HMs where you have a lot more going on, 12 players vs 4, etc. Trust me, there's a lot more visual things going on in HM trials, a lot more visuals from other players, a lot more data going on, if things are going to go wonky, that's where they will really show up. A lot of "end game" trial players reduce graphics settings, particles, distance, etc., to combat the issues of all the things going on in trials.
Your experience does not match mine.
I don't have any problems whatsoever unless someone is spam pinging my compass.
Every time, they are a BanditsUI user.
And I don't have any problems on Xbox at all.
You keep trying to claim it's not bandits, when clearly it is.
My problem lies with bandits.
It is not the consensus, but it is one of many problems.
It may not be your problem, but it definitely is mine.
And I have no control over who is using bandits.
Claim about console having issue have NO relevance to this specific issue of mine. They are different issues.
1 -
AzuraFan✭✭✭✭✭
✭✭Just popping in to say that I consistently experience lag in the Wayrest Sewers dungeons. For some reason, this is particularly bad when I'm in Wayrest 1 solo (I don't solo 2). When I'm grouped in 1 or 2, there's some lag but not as bad. Today, fighting the final boss in 1, it was like I was in a disco with the strobing when it's action, freeze, action, freeze. And I was frozen for long periods of time. Pretty much unplayable. Fortunately I still managed to complete it without dying.
This isn't my internet connection. I run at least one dungeon a day without these problems. But Wayrest Sewers is always laggy.0 -
LadyGP✭✭✭✭✭
✭Just popping in to say that I consistently experience lag in the Wayrest Sewers dungeons. For some reason, this is particularly bad when I'm in Wayrest 1 solo (I don't solo 2). When I'm grouped in 1 or 2, there's some lag but not as bad. Today, fighting the final boss in 1, it was like I was in a disco with the strobing when it's action, freeze, action, freeze. And I was frozen for long periods of time. Pretty much unplayable. Fortunately I still managed to complete it without dying.
This isn't my internet connection. I run at least one dungeon a day without these problems. But Wayrest Sewers is always laggy.
Hard to know if this is because of the above issues Kevin mentioned or the stuff we have been highlighting in this thread since May. How long have you been experiencing those issues? That sounds like the typical "rubberbanding" that most of us deal with on the regular.LadyGP/xCatGuy
PC/NA1 -
AzuraFan✭✭✭✭✭
✭✭Hard to know if this is because of the above issues Kevin mentioned or the stuff we have been highlighting in this thread since May. How long have you been experiencing those issues? That sounds like the typical "rubberbanding" that most of us deal with on the regular.
Not rubberbanding. Freezing. I've posted in this thread several times already about other places I experience spikes, lags, etc. Kevin asked for specifics, so I'm providing them. I commonly experience problems in Wayrest Sewers.1 -
LadyGP✭✭✭✭✭
✭
Not rubberbanding. Freezing. I've posted in this thread several times already about other places I experience spikes, lags, etc. Kevin asked for specifics, so I'm providing them. I commonly experience problems in Wayrest Sewers.
Ah, gotcha! All good, was just trying to help the conversation.
LadyGP/xCatGuy
PC/NA1 -
Sakiri✭✭✭✭✭
✭✭KaironBlackbard wrote: »
Your experience does not match mine.
I don't have any problems whatsoever unless someone is spam pinging my compass.
Every time, they are a BanditsUI user.
And I don't have any problems on Xbox at all.
You keep trying to claim it's not bandits, when clearly it is.
My problem lies with bandits.
It is not the consensus, but it is one of many problems.
It may not be your problem, but it definitely is mine.
And I have no control over who is using bandits.
Claim about console having issue have NO relevance to this specific issue of mine. They are different issues.
Spam... pinging... compass? The heck you on? My bandits isn't showing anything special on the compass?3 -
merevie✭✭✭✭✭Watching a live stream at the moment and there were several people watching other people play because...they cannot. We're all having dead crashes constantly while playing and people log in to do dailies, then just bail out because longer events like dungeons, BGs and pvp inevitably lead to being kicked.
Do you realise people are not just whiners, and there is something horribly unstable at the moment?
My friends who have never had connection issues (admitting here that I did laugh), needed help to know what to do now they are constantly crashing. It reminds me how used to it we are 'down under' (shouldn't be) and how upsetting and frustrating it is for people to experience it.Edited by merevie on December 17, 2024 6:29PM4 -
Downloading the recent update on my German internet connection is super slow. Like a few KB/s slow. If I use a VPN tunnel to the Netherlands, it updates super fast. Does it have something to do with the recent changes because of the malicious activity threat?0 -
Sluggy✭✭✭✭✭
Yeah, it's the framerate of the actual gameplay simulation on the server side. Sometimes also called a tick or a heartbeat. Usually, singleplayer-only games that need to have very deterministic results are going to decouple their rendering from the actual game logic such that the game logic runs at a fixed rate of time per tick. This naturally extends to multiplayer games with a client and server, where there is no rendering on the server but there will be asynchronous housekeeping for processing incoming and outgoing batches of data. The gameplay simulation logic obviously isn't going to be tied to any of that but will run independently at it's own fixed rate of time.dk_dunkirk wrote: »I've been programming for 45 years, and I've been a full-stack developer for 30. Can someone tell me what a "server frame" is? The context seems to be networking- or load-related. But even with my background, I can't tell what this is referring to. Maybe it means the number of servers (frame/box/chassis) that need to be running to handle the load?
The server frame time is something that has been mentioned in the past as something ZoS was looking in to. There were times where they saw spikes which obviously corresponded with massive lag and dropped inputs from the players as the server ended up having to ignore things in order to keep up in real time and avoid "the spiral of death". Curiously, there have also been times where they said that there were no apparent spikes in server frame time which means that it wasn't the server failing to keep the logic of the game itself running in real time - which likely points to either a problem in the aforementioned housekeeping portion of the game or some other networking infrastructure issue.
Edited by Sluggy on December 17, 2024 6:39PM3 -
Quackery✭✭✭✭✭Well, the good news lasted for a day... Issues are back to normal... One day of fun seemed to be enough...0