ZOS, Massive Spike in Ping/LAG in Recent Days - What Gives?

moderatelyfatman · December 19

For those who may consider me a gloomer, I want to refresh people's memories:

I'm going to ignore every comment that ZOS makes about improving performance. Talk is cheap, when ZOS fixes performance then I will start believing again.

LadyGP · December 19

moderatelyfatman wrote: »

For those who may consider me a gloomer, I want to refresh people's memories:

I'm going to ignore every comment that ZOS makes about improving performance. Talk is cheap, when ZOS fixes performance then I will start believing again.

I feel this in my bones. Maybe this time will be... different? One can hope.

moderatelyfatman · December 19

LadyGP wrote: »

moderatelyfatman wrote: »

For those who may consider me a gloomer, I want to refresh people's memories:

I'm going to ignore every comment that ZOS makes about improving performance. Talk is cheap, when ZOS fixes performance then I will start believing again.

I feel this in my bones. Maybe this time will be... different? One can hope.

Ok, I'm sure this won't happen then.

TX12001rwb17_ESO · December 19

This has to be fixed, I am crashing without any warning at all now, my ping can be fine and then within a matter of seconds it will boot me to the log in screen saying that an error had occurred.

Sakiri · December 19

silky_soft wrote: »

I don't know what you guys just did in the recent patch. But whatever it was it's horrible for customers 240ms. If I get CC now I'm dead, even with scroll wheel spam. Especially with a knock back or a pull.

*laughs in permanent 200 ms due to location*

It'll be ok. I feel ya.

dk_dunkirk · December 19

moderatelyfatman wrote: »

For those who may consider me a gloomer, I want to refresh people's memories:

I'm going to ignore every comment that ZOS makes about improving performance. Talk is cheap, when ZOS fixes performance then I will start believing again.

It seems like it would be an instructive exercise to go back over all of those old roadmaps, and take stock of how much or how little of it has actually landed.

Coo_PnT · December 19

I will return to ESO if innovative changes are made in 2025.

I am very interested to see what you do with this objectively numerous problematic game. If I were in management, I would keep the content as it is and focus on the crown store, mainly housing and costume related

sshogrin · December 19

@KaironBlackbard

BanditsUI has removed their map pings for now and will be investigating using LibMapPing like all addons should be using which shouldn't cause an overload of the MapPing function.

Coo_PnT · December 19

ZOS_RichLambert wrote: »

Hey folks - here's some answers to the main questions in this thread, as well as some insights to our process and what we're working on to help us better debug the issue(s). Thanks for the questions and please keep on providing feedback!

What things do you look at when you get reports about in-game performance issues?
Performance issues and their related investigations are complex, and there are loads of things we look at. Just about anything could be a factor when it comes to performance issues. Generally, the big ones we look at are:
The server(s) where the issue is occurring

What time(s) the issue occurred

Machine specs for impacted players

What affected players were doing at the time the issue occurred

What addons affected players have installed. All these things combined help us get a full picture of most issues. And from there, we expand our list of what we look at if we need more information, as we have for this particular issue. This includes requesting additional information from affected players and also implementing new logging when necessary.

What do the in-game performance charts look like now?
The ones that I personally look at the most (crashes, long load times, and server frames) are all pretty solid overall. We have a few crashes we’re addressing, but they are rarely occurring. The long load times reports, especially on console are very small and server frames – how bogged down the server is also very stable, at least outside of prime time. During prime time, there are a few spikes that we’d like to figure out and are implementing more tooling (logging) in the next few incremental patches to try to get more info for our engineers.

What information is most helpful to get when investigating in-game performance issues?
Sharing with us the date and time the issue occurred (ideally to the minute) as well as what you were doing at the time is the most helpful. It’s also a huge bonus if you have a video/clip of it so we can see exactly what is being described. We have been watching the video clips you’ve been sharing with us for this issue – thank you!

What are you able to change when in-game performance issues occur?
It really depends on what the issue is. We are able to hotfix some things (hotfix = no server downtime) but most require an outage. Crashes, broken items/spawns and exploits are generally “easier” to find a fix and get something out quickly. “Input lag”, “high ping” or “low frame rate” type issues are much harder to pinpoint and often take a great deal of time to find and fix. However, in the case of input lag or high ping, if it’s a result of an attack on the servers, we are able to turn adjust mitigation whenever we need.

Why do investigations sometimes take a long time?
There’s lots of different reasons, some bugs are easy to repro, others are not. Load related bugs (in general the input lag or high ping type issues) are the most difficult and time consuming to track down. Remember ESO is a HUGE game – millions of lines of code (over 25mil) to sift through. Then you have countless different machine types, drivers, internet types/locations in the world, addons... etc. to make bug hunting more complex.
On top of that, massive servers with hundreds of thousands of players on them at any one time is really, really hard to replicate internally. It’s tough!

How many different groups have been involved in investigating this issue?
Quite a few people are involved in investigations for ongoing game performance issues, including people from our BI (Business Intelligence) group, Community, Customer Support, Game Design, Engineering, QA and Live Services teams.

Has anyone at ZOS been able to replicate this issue?
We have not been able to consistently replicate this issue on the live servers, or at all on our internal servers due to some live server factors such as load being involved. While some of our developers have experienced it on the live servers alongside our players, there’s been no clear and consistent “smoking gun” root cause we’ve been able to identify so far.

Can someone confirm the role of the AI chat monitoring here?
We’ve extensively investigated if this could have an impact on game performance. The way the tool works, it sifts through chat logs that are exported from the game. So, it isn’t integrated into ESO, and never actually touches the game or the chat servers.
Regarding the tool itself, it helps us identify at risk chat and areas of concern faster. But our agents make the determination on whether an account is suspended or banned. We do have some auto kick rules for chat spam, but anything automated is temporary. Our agents review every automatic action against account history before taking any permanent account action.
We have also been using the tool for a couple years and nothing has changed in how we log chat, so the timing doesn’t line up. The situation a couple months ago with increased actioning related to chat logs was more a case of changes to our processes and training up new personnel than with the tool itself. Ultimately, though, yes we did look at if the tool was having any noticeable impact to game performance and it’s not.

What is your process like when an in-game performance issue is reported?
The first step is to gather as much info as we can. Is this just one person running into it, or is it multiple? Is it region/location specific? Platform specific? Does this only happen during prime time? Once we have all the info available, we then start to work on internal repro. If we can find a way to repro it, we’re generally able to get a fix together quickly.

What work has been completed or is in progress so far?
Outside of the ongoing investigations based on the information all of you have provided, we’ve been working on lots of logging additions for the most part. None of these are going to solve any problems on their own, but they should provide us with better mechanisms for helping to better diagnose the problem.
Server logging - specifically collecting more metrics about a player's experience from within an instance/region. (number of messages per frame, avg and max bytes sent & received...etc.)

Client logging of alternate quit methods. Clients closed via Alt+F4 or the window 'X' button terminate instantly and the server wouldn't know it was intentional and would just be considered a client timeout. (This is to aid in false-positive detection.)

HeartbeatDisconnector (basically a message that checks to see if your client is still connected) should accept any message as traffic, not just heartbeats. If you get a ton of messages, it's rare but possible your heartbeat doesn't get processed in time and you get disconnected.

Present a more accurate ping in the game client. Our latency meter is not reflective of what the actual ping is. It's currently tied to framerate and load screens and what not.

Investigate the Bandits UI addon. (In the top 15 most used add-ons currently) There are lots of forum posts pointing to this add-on for potential latency issues. The suspicion is that the addon is using map pings as somewhat of a backdoor to send arbitrary data to clients, regardless of whether players have the add-on installed. The pings would get broadcast to everyone and cause game performance problems (latency & frame rate).

Please come to a proper conclusion on the ZoS side.
There is a lot being said about this Bandits UI with various speculations.
Is it acceptable to use or not?

LadyGP · December 19

Coo_PnT wrote: »

ZOS_RichLambert wrote: »

Hey folks - here's some answers to the main questions in this thread, as well as some insights to our process and what we're working on to help us better debug the issue(s). Thanks for the questions and please keep on providing feedback!

What things do you look at when you get reports about in-game performance issues?
Performance issues and their related investigations are complex, and there are loads of things we look at. Just about anything could be a factor when it comes to performance issues. Generally, the big ones we look at are:
The server(s) where the issue is occurring

What time(s) the issue occurred

Machine specs for impacted players

What affected players were doing at the time the issue occurred

What addons affected players have installed. All these things combined help us get a full picture of most issues. And from there, we expand our list of what we look at if we need more information, as we have for this particular issue. This includes requesting additional information from affected players and also implementing new logging when necessary.

What do the in-game performance charts look like now?
The ones that I personally look at the most (crashes, long load times, and server frames) are all pretty solid overall. We have a few crashes we’re addressing, but they are rarely occurring. The long load times reports, especially on console are very small and server frames – how bogged down the server is also very stable, at least outside of prime time. During prime time, there are a few spikes that we’d like to figure out and are implementing more tooling (logging) in the next few incremental patches to try to get more info for our engineers.

What information is most helpful to get when investigating in-game performance issues?
Sharing with us the date and time the issue occurred (ideally to the minute) as well as what you were doing at the time is the most helpful. It’s also a huge bonus if you have a video/clip of it so we can see exactly what is being described. We have been watching the video clips you’ve been sharing with us for this issue – thank you!

What are you able to change when in-game performance issues occur?
It really depends on what the issue is. We are able to hotfix some things (hotfix = no server downtime) but most require an outage. Crashes, broken items/spawns and exploits are generally “easier” to find a fix and get something out quickly. “Input lag”, “high ping” or “low frame rate” type issues are much harder to pinpoint and often take a great deal of time to find and fix. However, in the case of input lag or high ping, if it’s a result of an attack on the servers, we are able to turn adjust mitigation whenever we need.

Why do investigations sometimes take a long time?
There’s lots of different reasons, some bugs are easy to repro, others are not. Load related bugs (in general the input lag or high ping type issues) are the most difficult and time consuming to track down. Remember ESO is a HUGE game – millions of lines of code (over 25mil) to sift through. Then you have countless different machine types, drivers, internet types/locations in the world, addons... etc. to make bug hunting more complex.
On top of that, massive servers with hundreds of thousands of players on them at any one time is really, really hard to replicate internally. It’s tough!

How many different groups have been involved in investigating this issue?
Quite a few people are involved in investigations for ongoing game performance issues, including people from our BI (Business Intelligence) group, Community, Customer Support, Game Design, Engineering, QA and Live Services teams.

Has anyone at ZOS been able to replicate this issue?
We have not been able to consistently replicate this issue on the live servers, or at all on our internal servers due to some live server factors such as load being involved. While some of our developers have experienced it on the live servers alongside our players, there’s been no clear and consistent “smoking gun” root cause we’ve been able to identify so far.

Can someone confirm the role of the AI chat monitoring here?
We’ve extensively investigated if this could have an impact on game performance. The way the tool works, it sifts through chat logs that are exported from the game. So, it isn’t integrated into ESO, and never actually touches the game or the chat servers.
Regarding the tool itself, it helps us identify at risk chat and areas of concern faster. But our agents make the determination on whether an account is suspended or banned. We do have some auto kick rules for chat spam, but anything automated is temporary. Our agents review every automatic action against account history before taking any permanent account action.
We have also been using the tool for a couple years and nothing has changed in how we log chat, so the timing doesn’t line up. The situation a couple months ago with increased actioning related to chat logs was more a case of changes to our processes and training up new personnel than with the tool itself. Ultimately, though, yes we did look at if the tool was having any noticeable impact to game performance and it’s not.

What is your process like when an in-game performance issue is reported?
The first step is to gather as much info as we can. Is this just one person running into it, or is it multiple? Is it region/location specific? Platform specific? Does this only happen during prime time? Once we have all the info available, we then start to work on internal repro. If we can find a way to repro it, we’re generally able to get a fix together quickly.

What work has been completed or is in progress so far?
Outside of the ongoing investigations based on the information all of you have provided, we’ve been working on lots of logging additions for the most part. None of these are going to solve any problems on their own, but they should provide us with better mechanisms for helping to better diagnose the problem.
Server logging - specifically collecting more metrics about a player's experience from within an instance/region. (number of messages per frame, avg and max bytes sent & received...etc.)

Client logging of alternate quit methods. Clients closed via Alt+F4 or the window 'X' button terminate instantly and the server wouldn't know it was intentional and would just be considered a client timeout. (This is to aid in false-positive detection.)

HeartbeatDisconnector (basically a message that checks to see if your client is still connected) should accept any message as traffic, not just heartbeats. If you get a ton of messages, it's rare but possible your heartbeat doesn't get processed in time and you get disconnected.

Present a more accurate ping in the game client. Our latency meter is not reflective of what the actual ping is. It's currently tied to framerate and load screens and what not.

Investigate the Bandits UI addon. (In the top 15 most used add-ons currently) There are lots of forum posts pointing to this add-on for potential latency issues. The suspicion is that the addon is using map pings as somewhat of a backdoor to send arbitrary data to clients, regardless of whether players have the add-on installed. The pings would get broadcast to everyone and cause game performance problems (latency & frame rate).

Please come to a proper conclusion on the ZoS side.
There is a lot being said about this Bandits UI with various speculations.
Is it acceptable to use or not? If you mention this issue, you should give us your concrete opinion as soon as possible.

Based on Bandits change long it looks like they removed map pings. I'd say it's safe to use. To be honest, ZoS probably isn't 100% sure yet on if map pings was an issue or not.

s3dulo · December 19

Any idea when PTS will be available again???

LadyGP · December 19

s3dulo wrote: »

Any idea when PTS will be available again???

I think @ZOS_Kevin said a week or two? I can't find the exact post.

Coo_PnT · December 19

LadyGP wrote: »

Coo_PnT wrote: »

ZOS_RichLambert wrote: »

Hey folks - here's some answers to the main questions in this thread, as well as some insights to our process and what we're working on to help us better debug the issue(s). Thanks for the questions and please keep on providing feedback!

What things do you look at when you get reports about in-game performance issues?
Performance issues and their related investigations are complex, and there are loads of things we look at. Just about anything could be a factor when it comes to performance issues. Generally, the big ones we look at are:
The server(s) where the issue is occurring

What time(s) the issue occurred

Machine specs for impacted players

What affected players were doing at the time the issue occurred

What addons affected players have installed. All these things combined help us get a full picture of most issues. And from there, we expand our list of what we look at if we need more information, as we have for this particular issue. This includes requesting additional information from affected players and also implementing new logging when necessary.

What do the in-game performance charts look like now?
The ones that I personally look at the most (crashes, long load times, and server frames) are all pretty solid overall. We have a few crashes we’re addressing, but they are rarely occurring. The long load times reports, especially on console are very small and server frames – how bogged down the server is also very stable, at least outside of prime time. During prime time, there are a few spikes that we’d like to figure out and are implementing more tooling (logging) in the next few incremental patches to try to get more info for our engineers.

What information is most helpful to get when investigating in-game performance issues?
Sharing with us the date and time the issue occurred (ideally to the minute) as well as what you were doing at the time is the most helpful. It’s also a huge bonus if you have a video/clip of it so we can see exactly what is being described. We have been watching the video clips you’ve been sharing with us for this issue – thank you!

What are you able to change when in-game performance issues occur?
It really depends on what the issue is. We are able to hotfix some things (hotfix = no server downtime) but most require an outage. Crashes, broken items/spawns and exploits are generally “easier” to find a fix and get something out quickly. “Input lag”, “high ping” or “low frame rate” type issues are much harder to pinpoint and often take a great deal of time to find and fix. However, in the case of input lag or high ping, if it’s a result of an attack on the servers, we are able to turn adjust mitigation whenever we need.

Why do investigations sometimes take a long time?
There’s lots of different reasons, some bugs are easy to repro, others are not. Load related bugs (in general the input lag or high ping type issues) are the most difficult and time consuming to track down. Remember ESO is a HUGE game – millions of lines of code (over 25mil) to sift through. Then you have countless different machine types, drivers, internet types/locations in the world, addons... etc. to make bug hunting more complex.
On top of that, massive servers with hundreds of thousands of players on them at any one time is really, really hard to replicate internally. It’s tough!

How many different groups have been involved in investigating this issue?
Quite a few people are involved in investigations for ongoing game performance issues, including people from our BI (Business Intelligence) group, Community, Customer Support, Game Design, Engineering, QA and Live Services teams.

Has anyone at ZOS been able to replicate this issue?
We have not been able to consistently replicate this issue on the live servers, or at all on our internal servers due to some live server factors such as load being involved. While some of our developers have experienced it on the live servers alongside our players, there’s been no clear and consistent “smoking gun” root cause we’ve been able to identify so far.

Can someone confirm the role of the AI chat monitoring here?
We’ve extensively investigated if this could have an impact on game performance. The way the tool works, it sifts through chat logs that are exported from the game. So, it isn’t integrated into ESO, and never actually touches the game or the chat servers.
Regarding the tool itself, it helps us identify at risk chat and areas of concern faster. But our agents make the determination on whether an account is suspended or banned. We do have some auto kick rules for chat spam, but anything automated is temporary. Our agents review every automatic action against account history before taking any permanent account action.
We have also been using the tool for a couple years and nothing has changed in how we log chat, so the timing doesn’t line up. The situation a couple months ago with increased actioning related to chat logs was more a case of changes to our processes and training up new personnel than with the tool itself. Ultimately, though, yes we did look at if the tool was having any noticeable impact to game performance and it’s not.

What is your process like when an in-game performance issue is reported?
The first step is to gather as much info as we can. Is this just one person running into it, or is it multiple? Is it region/location specific? Platform specific? Does this only happen during prime time? Once we have all the info available, we then start to work on internal repro. If we can find a way to repro it, we’re generally able to get a fix together quickly.

What work has been completed or is in progress so far?
Outside of the ongoing investigations based on the information all of you have provided, we’ve been working on lots of logging additions for the most part. None of these are going to solve any problems on their own, but they should provide us with better mechanisms for helping to better diagnose the problem.
Server logging - specifically collecting more metrics about a player's experience from within an instance/region. (number of messages per frame, avg and max bytes sent & received...etc.)

Client logging of alternate quit methods. Clients closed via Alt+F4 or the window 'X' button terminate instantly and the server wouldn't know it was intentional and would just be considered a client timeout. (This is to aid in false-positive detection.)

HeartbeatDisconnector (basically a message that checks to see if your client is still connected) should accept any message as traffic, not just heartbeats. If you get a ton of messages, it's rare but possible your heartbeat doesn't get processed in time and you get disconnected.

Present a more accurate ping in the game client. Our latency meter is not reflective of what the actual ping is. It's currently tied to framerate and load screens and what not.

Investigate the Bandits UI addon. (In the top 15 most used add-ons currently) There are lots of forum posts pointing to this add-on for potential latency issues. The suspicion is that the addon is using map pings as somewhat of a backdoor to send arbitrary data to clients, regardless of whether players have the add-on installed. The pings would get broadcast to everyone and cause game performance problems (latency & frame rate).

Please come to a proper conclusion on the ZoS side.
There is a lot being said about this Bandits UI with various speculations.
Is it acceptable to use or not? If you mention this issue, you should give us your concrete opinion as soon as possible.

Based on Bandits change long it looks like they removed map pings. I'd say it's safe to use. To be honest, ZoS probably isn't 100% sure yet on if map pings was an issue or not.

Thanks for the reply. I like the Bandits UI and find it a very functional add-on.
Kudos to all add-on developers. Thank you for everything.

sshogrin · December 19

Coo_PnT wrote: »

ZOS_RichLambert wrote: »

Hey folks - here's some answers to the main questions in this thread, as well as some insights to our process and what we're working on to help us better debug the issue(s). Thanks for the questions and please keep on providing feedback!

What things do you look at when you get reports about in-game performance issues?
Performance issues and their related investigations are complex, and there are loads of things we look at. Just about anything could be a factor when it comes to performance issues. Generally, the big ones we look at are:
The server(s) where the issue is occurring

What time(s) the issue occurred

Machine specs for impacted players

What affected players were doing at the time the issue occurred

What addons affected players have installed. All these things combined help us get a full picture of most issues. And from there, we expand our list of what we look at if we need more information, as we have for this particular issue. This includes requesting additional information from affected players and also implementing new logging when necessary.

What do the in-game performance charts look like now?
The ones that I personally look at the most (crashes, long load times, and server frames) are all pretty solid overall. We have a few crashes we’re addressing, but they are rarely occurring. The long load times reports, especially on console are very small and server frames – how bogged down the server is also very stable, at least outside of prime time. During prime time, there are a few spikes that we’d like to figure out and are implementing more tooling (logging) in the next few incremental patches to try to get more info for our engineers.

What information is most helpful to get when investigating in-game performance issues?
Sharing with us the date and time the issue occurred (ideally to the minute) as well as what you were doing at the time is the most helpful. It’s also a huge bonus if you have a video/clip of it so we can see exactly what is being described. We have been watching the video clips you’ve been sharing with us for this issue – thank you!

What are you able to change when in-game performance issues occur?
It really depends on what the issue is. We are able to hotfix some things (hotfix = no server downtime) but most require an outage. Crashes, broken items/spawns and exploits are generally “easier” to find a fix and get something out quickly. “Input lag”, “high ping” or “low frame rate” type issues are much harder to pinpoint and often take a great deal of time to find and fix. However, in the case of input lag or high ping, if it’s a result of an attack on the servers, we are able to turn adjust mitigation whenever we need.

Why do investigations sometimes take a long time?
There’s lots of different reasons, some bugs are easy to repro, others are not. Load related bugs (in general the input lag or high ping type issues) are the most difficult and time consuming to track down. Remember ESO is a HUGE game – millions of lines of code (over 25mil) to sift through. Then you have countless different machine types, drivers, internet types/locations in the world, addons... etc. to make bug hunting more complex.
On top of that, massive servers with hundreds of thousands of players on them at any one time is really, really hard to replicate internally. It’s tough!

How many different groups have been involved in investigating this issue?
Quite a few people are involved in investigations for ongoing game performance issues, including people from our BI (Business Intelligence) group, Community, Customer Support, Game Design, Engineering, QA and Live Services teams.

Has anyone at ZOS been able to replicate this issue?
We have not been able to consistently replicate this issue on the live servers, or at all on our internal servers due to some live server factors such as load being involved. While some of our developers have experienced it on the live servers alongside our players, there’s been no clear and consistent “smoking gun” root cause we’ve been able to identify so far.

Can someone confirm the role of the AI chat monitoring here?
We’ve extensively investigated if this could have an impact on game performance. The way the tool works, it sifts through chat logs that are exported from the game. So, it isn’t integrated into ESO, and never actually touches the game or the chat servers.
Regarding the tool itself, it helps us identify at risk chat and areas of concern faster. But our agents make the determination on whether an account is suspended or banned. We do have some auto kick rules for chat spam, but anything automated is temporary. Our agents review every automatic action against account history before taking any permanent account action.
We have also been using the tool for a couple years and nothing has changed in how we log chat, so the timing doesn’t line up. The situation a couple months ago with increased actioning related to chat logs was more a case of changes to our processes and training up new personnel than with the tool itself. Ultimately, though, yes we did look at if the tool was having any noticeable impact to game performance and it’s not.

What is your process like when an in-game performance issue is reported?
The first step is to gather as much info as we can. Is this just one person running into it, or is it multiple? Is it region/location specific? Platform specific? Does this only happen during prime time? Once we have all the info available, we then start to work on internal repro. If we can find a way to repro it, we’re generally able to get a fix together quickly.

What work has been completed or is in progress so far?
Outside of the ongoing investigations based on the information all of you have provided, we’ve been working on lots of logging additions for the most part. None of these are going to solve any problems on their own, but they should provide us with better mechanisms for helping to better diagnose the problem.
Server logging - specifically collecting more metrics about a player's experience from within an instance/region. (number of messages per frame, avg and max bytes sent & received...etc.)

Client logging of alternate quit methods. Clients closed via Alt+F4 or the window 'X' button terminate instantly and the server wouldn't know it was intentional and would just be considered a client timeout. (This is to aid in false-positive detection.)

HeartbeatDisconnector (basically a message that checks to see if your client is still connected) should accept any message as traffic, not just heartbeats. If you get a ton of messages, it's rare but possible your heartbeat doesn't get processed in time and you get disconnected.

Present a more accurate ping in the game client. Our latency meter is not reflective of what the actual ping is. It's currently tied to framerate and load screens and what not.

Investigate the Bandits UI addon. (In the top 15 most used add-ons currently) There are lots of forum posts pointing to this add-on for potential latency issues. The suspicion is that the addon is using map pings as somewhat of a backdoor to send arbitrary data to clients, regardless of whether players have the add-on installed. The pings would get broadcast to everyone and cause game performance problems (latency & frame rate).

Please come to a proper conclusion on the ZoS side.
There is a lot being said about this Bandits UI with various speculations.
Is it acceptable to use or not?

Yes, totally acceptable. Update it using the latest version, all map ping functions have been removed.
This is also not the issue that's going on. You will still get the issues without using it, running with a group that nobody's using it.
There are bigger issues going on on the server side of things.

drkfrontiers · December 19

Game is completely unplayable for me at the moment 650-999+ spike constantly, Been like this for a week now.

EU Server
TZ: GMT+2
Processor: AMD Ryzen 9 5950X 16-Core Processor 3.40 GHz
Installed RAM: 32,0 GB
GC: Radeon™ RX 6700 XT 12 GB GDDR6 memory

Coo_PnT · December 19

sshogrin wrote: »

Coo_PnT wrote: »

ZOS_RichLambert wrote: »

Hey folks - here's some answers to the main questions in this thread, as well as some insights to our process and what we're working on to help us better debug the issue(s). Thanks for the questions and please keep on providing feedback!

What things do you look at when you get reports about in-game performance issues?
Performance issues and their related investigations are complex, and there are loads of things we look at. Just about anything could be a factor when it comes to performance issues. Generally, the big ones we look at are:
The server(s) where the issue is occurring

What time(s) the issue occurred

Machine specs for impacted players

What affected players were doing at the time the issue occurred

What addons affected players have installed. All these things combined help us get a full picture of most issues. And from there, we expand our list of what we look at if we need more information, as we have for this particular issue. This includes requesting additional information from affected players and also implementing new logging when necessary.

What do the in-game performance charts look like now?
The ones that I personally look at the most (crashes, long load times, and server frames) are all pretty solid overall. We have a few crashes we’re addressing, but they are rarely occurring. The long load times reports, especially on console are very small and server frames – how bogged down the server is also very stable, at least outside of prime time. During prime time, there are a few spikes that we’d like to figure out and are implementing more tooling (logging) in the next few incremental patches to try to get more info for our engineers.

What information is most helpful to get when investigating in-game performance issues?
Sharing with us the date and time the issue occurred (ideally to the minute) as well as what you were doing at the time is the most helpful. It’s also a huge bonus if you have a video/clip of it so we can see exactly what is being described. We have been watching the video clips you’ve been sharing with us for this issue – thank you!

What are you able to change when in-game performance issues occur?
It really depends on what the issue is. We are able to hotfix some things (hotfix = no server downtime) but most require an outage. Crashes, broken items/spawns and exploits are generally “easier” to find a fix and get something out quickly. “Input lag”, “high ping” or “low frame rate” type issues are much harder to pinpoint and often take a great deal of time to find and fix. However, in the case of input lag or high ping, if it’s a result of an attack on the servers, we are able to turn adjust mitigation whenever we need.

Why do investigations sometimes take a long time?
There’s lots of different reasons, some bugs are easy to repro, others are not. Load related bugs (in general the input lag or high ping type issues) are the most difficult and time consuming to track down. Remember ESO is a HUGE game – millions of lines of code (over 25mil) to sift through. Then you have countless different machine types, drivers, internet types/locations in the world, addons... etc. to make bug hunting more complex.
On top of that, massive servers with hundreds of thousands of players on them at any one time is really, really hard to replicate internally. It’s tough!

How many different groups have been involved in investigating this issue?
Quite a few people are involved in investigations for ongoing game performance issues, including people from our BI (Business Intelligence) group, Community, Customer Support, Game Design, Engineering, QA and Live Services teams.

Has anyone at ZOS been able to replicate this issue?
We have not been able to consistently replicate this issue on the live servers, or at all on our internal servers due to some live server factors such as load being involved. While some of our developers have experienced it on the live servers alongside our players, there’s been no clear and consistent “smoking gun” root cause we’ve been able to identify so far.

Can someone confirm the role of the AI chat monitoring here?
We’ve extensively investigated if this could have an impact on game performance. The way the tool works, it sifts through chat logs that are exported from the game. So, it isn’t integrated into ESO, and never actually touches the game or the chat servers.
Regarding the tool itself, it helps us identify at risk chat and areas of concern faster. But our agents make the determination on whether an account is suspended or banned. We do have some auto kick rules for chat spam, but anything automated is temporary. Our agents review every automatic action against account history before taking any permanent account action.
We have also been using the tool for a couple years and nothing has changed in how we log chat, so the timing doesn’t line up. The situation a couple months ago with increased actioning related to chat logs was more a case of changes to our processes and training up new personnel than with the tool itself. Ultimately, though, yes we did look at if the tool was having any noticeable impact to game performance and it’s not.

What is your process like when an in-game performance issue is reported?
The first step is to gather as much info as we can. Is this just one person running into it, or is it multiple? Is it region/location specific? Platform specific? Does this only happen during prime time? Once we have all the info available, we then start to work on internal repro. If we can find a way to repro it, we’re generally able to get a fix together quickly.

What work has been completed or is in progress so far?
Outside of the ongoing investigations based on the information all of you have provided, we’ve been working on lots of logging additions for the most part. None of these are going to solve any problems on their own, but they should provide us with better mechanisms for helping to better diagnose the problem.
Server logging - specifically collecting more metrics about a player's experience from within an instance/region. (number of messages per frame, avg and max bytes sent & received...etc.)

Client logging of alternate quit methods. Clients closed via Alt+F4 or the window 'X' button terminate instantly and the server wouldn't know it was intentional and would just be considered a client timeout. (This is to aid in false-positive detection.)

HeartbeatDisconnector (basically a message that checks to see if your client is still connected) should accept any message as traffic, not just heartbeats. If you get a ton of messages, it's rare but possible your heartbeat doesn't get processed in time and you get disconnected.

Present a more accurate ping in the game client. Our latency meter is not reflective of what the actual ping is. It's currently tied to framerate and load screens and what not.

Investigate the Bandits UI addon. (In the top 15 most used add-ons currently) There are lots of forum posts pointing to this add-on for potential latency issues. The suspicion is that the addon is using map pings as somewhat of a backdoor to send arbitrary data to clients, regardless of whether players have the add-on installed. The pings would get broadcast to everyone and cause game performance problems (latency & frame rate).

Please come to a proper conclusion on the ZoS side.
There is a lot being said about this Bandits UI with various speculations.
Is it acceptable to use or not?

Yes, totally acceptable. Update it using the latest version, all map ping functions have been removed.
This is also not the issue that's going on. You will still get the issues without using it, running with a group that nobody's using it.
There are bigger issues going on on the server side of things.

Thank you for your reply.
I mentioned the name of this add-on in the problem description on the ZoS side, so if there is no problem, I would appreciate it if you could state that.

ZOS_Kevin · December 19

LadyGP wrote: »

s3dulo wrote: »

Any idea when PTS will be available again???

I think @ZOS_Kevin said a week or two? I can't find the exact post.

We don't have a firm date yet. The team started work to restore PTS yesterday. So timing is TBD at the moment.

s3dulo · December 19

LadyGP wrote: »

s3dulo wrote: »

Any idea when PTS will be available again???

I think @ZOS_Kevin said a week or two? I can't find the exact post.

The post I saw from last week said early this week but that has come and gone. I see the announcements at the bottom are displaying now and they weren't yesterday. The message after a login attempt is no longer an error and it now just says it's down for maintenance. Oh well. I guess it will be up when it's up.

sshogrin · December 19

Coo_PnT wrote: »

sshogrin wrote: »

Coo_PnT wrote: »

ZOS_RichLambert wrote: »

Hey folks - here's some answers to the main questions in this thread, as well as some insights to our process and what we're working on to help us better debug the issue(s). Thanks for the questions and please keep on providing feedback!

What things do you look at when you get reports about in-game performance issues?
Performance issues and their related investigations are complex, and there are loads of things we look at. Just about anything could be a factor when it comes to performance issues. Generally, the big ones we look at are:
The server(s) where the issue is occurring

What time(s) the issue occurred

Machine specs for impacted players

What affected players were doing at the time the issue occurred

What addons affected players have installed. All these things combined help us get a full picture of most issues. And from there, we expand our list of what we look at if we need more information, as we have for this particular issue. This includes requesting additional information from affected players and also implementing new logging when necessary.

What do the in-game performance charts look like now?
The ones that I personally look at the most (crashes, long load times, and server frames) are all pretty solid overall. We have a few crashes we’re addressing, but they are rarely occurring. The long load times reports, especially on console are very small and server frames – how bogged down the server is also very stable, at least outside of prime time. During prime time, there are a few spikes that we’d like to figure out and are implementing more tooling (logging) in the next few incremental patches to try to get more info for our engineers.

What information is most helpful to get when investigating in-game performance issues?
Sharing with us the date and time the issue occurred (ideally to the minute) as well as what you were doing at the time is the most helpful. It’s also a huge bonus if you have a video/clip of it so we can see exactly what is being described. We have been watching the video clips you’ve been sharing with us for this issue – thank you!

What are you able to change when in-game performance issues occur?
It really depends on what the issue is. We are able to hotfix some things (hotfix = no server downtime) but most require an outage. Crashes, broken items/spawns and exploits are generally “easier” to find a fix and get something out quickly. “Input lag”, “high ping” or “low frame rate” type issues are much harder to pinpoint and often take a great deal of time to find and fix. However, in the case of input lag or high ping, if it’s a result of an attack on the servers, we are able to turn adjust mitigation whenever we need.

Why do investigations sometimes take a long time?
There’s lots of different reasons, some bugs are easy to repro, others are not. Load related bugs (in general the input lag or high ping type issues) are the most difficult and time consuming to track down. Remember ESO is a HUGE game – millions of lines of code (over 25mil) to sift through. Then you have countless different machine types, drivers, internet types/locations in the world, addons... etc. to make bug hunting more complex.
On top of that, massive servers with hundreds of thousands of players on them at any one time is really, really hard to replicate internally. It’s tough!

How many different groups have been involved in investigating this issue?
Quite a few people are involved in investigations for ongoing game performance issues, including people from our BI (Business Intelligence) group, Community, Customer Support, Game Design, Engineering, QA and Live Services teams.

Has anyone at ZOS been able to replicate this issue?
We have not been able to consistently replicate this issue on the live servers, or at all on our internal servers due to some live server factors such as load being involved. While some of our developers have experienced it on the live servers alongside our players, there’s been no clear and consistent “smoking gun” root cause we’ve been able to identify so far.

Can someone confirm the role of the AI chat monitoring here?
We’ve extensively investigated if this could have an impact on game performance. The way the tool works, it sifts through chat logs that are exported from the game. So, it isn’t integrated into ESO, and never actually touches the game or the chat servers.
Regarding the tool itself, it helps us identify at risk chat and areas of concern faster. But our agents make the determination on whether an account is suspended or banned. We do have some auto kick rules for chat spam, but anything automated is temporary. Our agents review every automatic action against account history before taking any permanent account action.
We have also been using the tool for a couple years and nothing has changed in how we log chat, so the timing doesn’t line up. The situation a couple months ago with increased actioning related to chat logs was more a case of changes to our processes and training up new personnel than with the tool itself. Ultimately, though, yes we did look at if the tool was having any noticeable impact to game performance and it’s not.

What is your process like when an in-game performance issue is reported?
The first step is to gather as much info as we can. Is this just one person running into it, or is it multiple? Is it region/location specific? Platform specific? Does this only happen during prime time? Once we have all the info available, we then start to work on internal repro. If we can find a way to repro it, we’re generally able to get a fix together quickly.

What work has been completed or is in progress so far?
Outside of the ongoing investigations based on the information all of you have provided, we’ve been working on lots of logging additions for the most part. None of these are going to solve any problems on their own, but they should provide us with better mechanisms for helping to better diagnose the problem.
Server logging - specifically collecting more metrics about a player's experience from within an instance/region. (number of messages per frame, avg and max bytes sent & received...etc.)

Client logging of alternate quit methods. Clients closed via Alt+F4 or the window 'X' button terminate instantly and the server wouldn't know it was intentional and would just be considered a client timeout. (This is to aid in false-positive detection.)

HeartbeatDisconnector (basically a message that checks to see if your client is still connected) should accept any message as traffic, not just heartbeats. If you get a ton of messages, it's rare but possible your heartbeat doesn't get processed in time and you get disconnected.

Present a more accurate ping in the game client. Our latency meter is not reflective of what the actual ping is. It's currently tied to framerate and load screens and what not.

Investigate the Bandits UI addon. (In the top 15 most used add-ons currently) There are lots of forum posts pointing to this add-on for potential latency issues. The suspicion is that the addon is using map pings as somewhat of a backdoor to send arbitrary data to clients, regardless of whether players have the add-on installed. The pings would get broadcast to everyone and cause game performance problems (latency & frame rate).

Please come to a proper conclusion on the ZoS side.
There is a lot being said about this Bandits UI with various speculations.
Is it acceptable to use or not?

Yes, totally acceptable. Update it using the latest version, all map ping functions have been removed.
This is also not the issue that's going on. You will still get the issues without using it, running with a group that nobody's using it.
There are bigger issues going on on the server side of things.

Thank you for your reply.
I mentioned the name of this add-on in the problem description on the ZoS side, so if there is no problem, I would appreciate it if you could state that.

Some of the issues with addons like this is having the same type of data sharing turned on in all addons, like using Hodors and BanditsUI at the same time and turning on all the dps/ulitmate sharing. There is a Library that can be used to do that so it makes all addons just share "one instance" of each type of data, but there are addons that don't use that Library, and use their own type of data sharing, which can cause multiple issues using MapPing for data sharing.

Again, I've not had issues specific to BanditsUI. I've run in groups where none of us have had BanditsUI running, and half the trial group got kicked at the same time during a boss fight. I've run in groups where it was mixed on running Bandits or not running it, plus everybody using Hodors, and we had absolutely no problems with anything, no DCs, no frame drops, nothing. So I would have to say this is all unrelated to BanditsUI considering my experiences.

BagOfBadgers · December 19

Micro stutters and lag spikes around all the event activities. Never had this before in previous years on the NA. EU server from the UK and it's ONLY EVER ESO that I have issues with.

Coo_PnT · December 19

I am not familiar with these add-ons and programs, but I thought they could only work within some acceptable limits within ESO. (UI and surroundings).

So, if we use this MapPing and add various factors, the result would be a behavior similar to a DDoS attack?Is this unique type of add-on delivered with permission from ZoS? Also, LibMapPing has not been updated and is quite old (November 2021). This is a very interesting story. I am very sorry, but I would appreciate a simple explanation that is easy for the layman to understand.

KaironBlackbard · December 19

sshogrin wrote: »

@KaironBlackbard

BanditsUI has removed their map pings for now and will be investigating using LibMapPing like all addons should be using which shouldn't cause an overload of the MapPing function.

ok. Perhaps I'll be able to play with those users now.

sshogrin · December 19

KaironBlackbard wrote: »

sshogrin wrote: »

@KaironBlackbard

BanditsUI has removed their map pings for now and will be investigating using LibMapPing like all addons should be using which shouldn't cause an overload of the MapPing function.

ok. Perhaps I'll be able to play with those users now.

If you have issues, it's definitely NOT the addon.
From my experiences, it's not the addon causing issues...if half a trial group get kicked from the game and not one person is using BanditsUI, it's completely unrelated.

Like I said, I did contact the developer about this, one of the main people from ESOUI even chimed in in the addon comment section stating that all addons that share data like that use the LipMapPing Library which should unify any issues using the MapPing function in the API. LibMapPing has been tested and functions properly without causing any issues and keep all addons that use it to only share the same type of data once, instead of having multiple map pings for the same data...ie. dps share is only shared once instead of being shared for each addon.

sshogrin · December 19

Coo_PnT wrote: »

I am not familiar with these add-ons and programs, but I thought they could only work within some acceptable limits within ESO. (UI and surroundings).

So, if we use this MapPing and add various factors, the result would be a behavior similar to a DDoS attack?Is this unique type of add-on delivered with permission from ZoS? Also, LibMapPing has not been updated and is quite old (November 2021). This is a very interesting story. I am very sorry, but I would appreciate a simple explanation that is easy for the layman to understand.

some Libraries and addons don't need to be updated because the coding functions haven't been changed or updated to the latest LUA API. ESO doesn't use the current LUA API, it's behind, they do however every now and then update the API when they update the game...that doesn't mean they update the LUA API every time they update the game.

Part of the issue in general with addons is a developer that doesn't use any of the Libraries that have been created to support certain functions that might be used by other addons. A long time ago addons included the Libraries needed in the addon folder, this caused issues because you would have multiple copies of the same Library trying to add, and even have conflicting versions of the same Library because some were the latest versions, and some were old versions. At one point that all changed and Libraries weren't included with the addon install and you had to install those required Libraries, and shouldn't be nested in the addon folder, only the "Addons" folder. It was also recommended that users either delete all addons in that folder and reinstall, or go through their addons, pull out those Library folders and put them in the main Addon folder so there was only 1 copy of the Libraries in the Addons folder. Whether or not people actually did that is unknown, I know I have done that. There are also some Libraries that have been discontinued as they are not needed, and it's even been posted on the Wiki and ESOUI to not use these Libraries anymore, whether or not developers paid attention to those posts is unknown also, but most of us have. Libraries are supposed to consolidate the same function that is being used by different addons so there is only "1 call" or "share" for the same data type. What has happened in the case of data sharing like dps and ultimate sharing is that some of the developers haven't conformed to this programming "best practice", or even what's been stated on ESOUI and the ESO LUA Wiki website.
I don't think that developers using their own style of using MapPing is the same as a DDoS attack, but it can cause extra data to be sent using the MapPing function.
The main takeaway from this is that developers of addons need to use the tools provided from the Libraries, ESOUI (there's a developer section explaining the does and don'ts and Lib references to what's there) and also the ESO LUA Wiki. There's a lot of Libraries already created which make creating an addon so much easier because the coding has already been done for a lot of functions. Those libraries all get updated (or should get updated) when a change to the LUA API has been implemented by ZOS for the game. That will "fix" any LUA errors that would occur from an update to the LUA API that would make those functions either not work, or that the function call might have had a change made, so that function call needs to be renamed, updated, or deleted all together because that function call isn't needed as it's written.
I hope this helps.

cptscotty · December 19

Lag spikes have become a daily thing again. The last few days lag is spiking in the middle of a dungeon run during a fight. Multiple people reporting the same thing. People and enemies freeze, or do a looping non stop animation, then all of a sudden we are all rubber banded through and either people are dead or the enemies are dead.

Consistently dealing with lag now. Also noticing the ping is higher on average. The lowest ping I am getting is 100...where before it was at least in the 80s. Also getting weird ping numbers like 101 or 102...numbers that dont usually show up as it used to be pretty consistent going from 84->100->116 now its 100->101->102->133->999+ with lag spikes (sometimes the lag spikes still only show 100 ping)

Sakiri · December 19

sshogrin wrote: »

@KaironBlackbard

BanditsUI has removed their map pings for now and will be investigating using LibMapPing like all addons should be using which shouldn't cause an overload of the MapPing function.

What map pings dies it even use? Like what functionality. Because mines never pinged my map that I'm aware of.

TX12001rwb17_ESO · December 19

Another day another crash within minutes resulting in error messages, I can barely do anything in this event, I crashed before I even made it to the New Life shrine, I am crashing in my player home where there are no players around, I even crashed while previewing the Kethendral Deep Mines house yesterday where I was all by myself, I had sent in a ticket several days ago but nothing has worked.

I have never experienced this problem in 10 years of playing ESO, this started happening earlier this month, ESO is my favorite game and I am finding very difficult to come up with a reason to buy crowns, I was going to buy the new house but I am having second thoughts, what is the point if I cannot play?

@ZOS_Kevin

sshogrin · December 19

Sakiri wrote: »

sshogrin wrote: »

@KaironBlackbard

BanditsUI has removed their map pings for now and will be investigating using LibMapPing like all addons should be using which shouldn't cause an overload of the MapPing function.

What map pings dies it even use? Like what functionality. Because mines never pinged my map that I'm aware of.

There's a function in LUA that uses MapPing to share data.

silky_soft · December 19

First try into new life gateway. Disconnect.

Edit 1:
Did some of the event, teleport back to eastmarch, disconnect.
1a:
Disconnect second try.

sshogrin · December 19

Sakiri wrote: »

sshogrin wrote: »

@KaironBlackbard

BanditsUI has removed their map pings for now and will be investigating using LibMapPing like all addons should be using which shouldn't cause an overload of the MapPing function.

What map pings dies it even use? Like what functionality. Because mines never pinged my map that I'm aware of.

the addons that use data sharing for things like dps and ultimates use a function within LUA called MapPing.
There is a Library called LibMapPing that the developers of addons should be using so all those data shares and calls are handled by that Library. That Library would actually use just one call per data type for all addons you have installed that are wanting to share or use that data so you don't have 2 or more addons trying to make the call or share for the same thing.
This is something that was referred to in an earlier comment as a "back channel" use.
To equate this to graphics, in an RGB image, the user can only see those 3 channels R, G, B, if you add an alpha channel for something like masking, that alpha channel would be like a back channel, something going on behind the scenes. I hope I'm explaining that well enough.