FPS drop over time

ZOS_AlexTardif

Hi @sirinsidiator, thanks for the details! We do know that during big fights that effects sometimes do not appear because we can hit a cap on particle systems. Basically, when we reach the cap of particles allowed on screen at once, we stop rendering new effects until other effects expire. This helps to keep the amount of GPU/CPU/memory usage from getting out of hand. We do have some folks working on optimizing the bigger offending particle systems at the moment so this should get better over time.

The fix regarding particles that came out with Thieves Guild was more in line with particles disappearing and then never coming back until you zone or relog. For example, in Trials before TG, you may have noticed that by the end of the Trial, you would stop seeing particles almost altogether.

Unfortunate that Evil Hunter bugs during big fights, those images do help illustrate the problem, so thanks! We'll definitely take a look at that.

As for the permanent FPS drop in Cyrodiil, it appears to be unrelated to particles, and we are actively working on that issue as we speak. Thanks for your patience and for following up!

sirinsidiator

Hi @ZOS_AlexTardif, thanks for your answer.

I would like to see ESO utilizing my GPU/CPU/memory to its fullest, but this has never happened until now. On my PC (Intel i5-3570K, GeForce GTX 770 4GB, 16GB RAM, some SSD) it always stays somewhere at around 40% CPU and 50% GPU usage, even when the frame rate goes down. It also uses only ~3GB RAM and 900MB vRAM with the 64-bit client. I wouldn't mind it used 6GB RAM and 3GB vRAM if it reduced the loading times, especially these in Cyrodiil when a keep flips. Maybe you could add settings to the UserSettings.txt where I can specify how much memory it is allowed to eat before it unloads stuff?

The particle effect limit makes sense, but there might always be a situation where the cap could be reached, so shouldn't there be a reserved quota for player effects for these situations? In my opinion they should always be rendered regardless of what is going on. For example when I cast my Volcanic Rune and it does not show up on the ground, I do not know if the skill worked or not and might waste time and resources to cast it again (I assume this does happen due to the cap, but it could also be caused by the lag).

Another idea that could help with these problems - until you have a proper fix - is a slash command like /reloadui, just for the graphics. That way I could get my frame rate back and remove the stuck particle effects without having to restart the whole game.

I don't know if it is of any use to you, but if you want I can record and send you some xperf traces.

SirAndy

ZOS_AlexTardif wrote: »

The system itself has been completely rewritten and thoroughly vetted

@ZOS_AlexTardif
May i suggest you guys re-evaluate your "vetting" process.

FPS issues are consistently getting worse with every patch ...

Rune_Relic

@ZOS_AlexTardif

Would it not be better to allow new effects to start rather than old effects to linger ?
ie kill/flush old effects to allow new effects by sticking them in a pipe and pushing the old effects out of memory.
The only relevant effects are the current effects at the end of the day.
Perhaps 3 pipes with different prioity:
pipe 1. 1st party outbound (me attacking enemy).
pipe 2. 3rd party inbound (Enemy attacking me).
pipe 3. 3rd party local effects (local attacks/effects).
pipe 4. Atmoshpherics

ZOS_AlexTardif

sirinsidiator wrote: »

Hi @ZOS_AlexTardif, thanks for your answer.

I would like to see ESO utilizing my GPU/CPU/memory to its fullest, but this has never happened until now.

The particle effect limit makes sense, but there might always be a situation where the cap could be reached, so shouldn't there be a reserved quota for player effects for these situations?

Another idea that could help with these problems - until you have a proper fix - is a slash command like /reloadui, just for the graphics. That way I could get my frame rate back and remove the stuck particle effects without having to restart the whole game.
].

@sirinsidiator Thanks for your feedback. Indeed we're looking at all sorts of things we can improve to get better usage. Until Thieves Guild came out (with the 64-bit client and DirectX-11 requirement), we had to be able to run on much lower minimum spec machines. Now that our requirements are higher, we'll be able to do more. This includes a lot of hardware features that only DX11 allows, which will both improve visuals and increase framerate.

Your point about particle effect limits, and mirrored by ideas from @Rune_Relic are well received. The big issue first and foremost was just getting rid of the 3rd party software that was leaking effects and replacing it with our own, stable system. Now that we know it works, we can look at addressing better ways of managing priority of effects. To give you some insight, if you look at the "Maximum Particle Systems" bar in your graphics settings, that is the limit we look at for our caps (note that a Particle System can include multiple emitters and multiple particles). 75% of that budget goes specifically towards ability effects, 25% for everything else. The Particle Suppression Distance will suppress particles that appear past a certain distance from your character. If you have the CPU/GPU bandwidth, you'll want Maximum Particle Systems all the way up, and something you may want to consider trying would be to turn the Particle Suppression Distance down during PvP, which will limit effects that spawn to only those very close to you, which as you stated are the only ones you care about in PvP scenarios.

As far as allowing new effects and killing old ones, it's a tricky situation. In order to get the memory needed for a new effect, that would require immediately killing off an effect (it would just pop out of existence) which would not only be a bit jarring, but might not accurately reflect an effect that is still applied by you or against you. Speaking for myself specifically, I hadn't considered the idea that we could allow this kind of behavior as a togglable menu option so that PvP players can have it their way without it harming the PvE experience. It's a great idea, and I will pass this along to the team to evaluate.

SirAndy wrote: »

ZOS_AlexTardif wrote: »

The system itself has been completely rewritten and thoroughly vetted

@ZOS_AlexTardif
May i suggest you guys re-evaluate your "vetting" process.

FPS issues are consistently getting worse with every patch ...

@SirAndy stay tuned, we are actively working on it! However, by all accounts the particle system rework has been successful. The current FPS issues are unrelated to particle systems, keep an eye on patch notes for updates!

ZOS_AlexTardif

sirinsidiator wrote: »

I don't know if it is of any use to you, but if you want I can record and send you some xperf traces.

That would be extremely useful @sirinsidiator. This frame-rate drop doesn't happen for everyone, and we have so far been unable to reproduce this internally, so a perf trace of a time when you're getting this permanent frame drop could prove very helpful. Thank you for offering to profile that.

Rune_Relic

Thanks for the clarification and the heads up @ZOS_AlexTardif .... looking forward to see what you can knock up in the forthcoming updates.

Vaoh

@ZOS_AlexTardif PS4 player here! I want to let you know that the Thieves Guild Base Game Patch has caused FPS on consoles to take a hit as well during fights, especially when intensive. It even becomes noticeable when inside of group dungeons fighting or sprinting/using Bolt Escape. I'd estimate the FPS drops from the goal of 30 to the 20's range but can't tell since I'm on console. These FPS drops can also happen when I first start the game, and are not a result of some sort of memory leak over a long period of time.

Particle effects and ability visuals showing up has been a new detail I have noticed with this patch. Before, visual spell effects would disappear especially during PvP in order to help the performance stay as steady as possible. Now I can see everything going on but FPS is taking a very noticeable hit.

I have a video of myself in Veteran City of Ash which very clearly showcases this FPS issue if it helps.

zerosingularity

ZOS_AlexTardif wrote: »

sirinsidiator wrote: »

Hi @ZOS_AlexTardif, thanks for your answer.

...

Until Thieves Guild came out (with the 64-bit client and DirectX-11 requirement), we had to be able to run on much lower minimum spec machines. Now that our requirements are higher, we'll be able to do more. This includes a lot of hardware features that only DX11 allows, which will both improve visuals and increase framerate.

...

I for one am genuinely interested in these features and changes, and would love to hear about them as they relate to ESO. I am sure others would be as well, and it might make a neat segment on a future ESO live or a dev blog or something.

Xexpo

I am legitimately impressed with Zenimax's feedback in this thread.
If we could create more straightforward communication between players and game devs, similar to this, ESO:TU could have limitless potential.

Hats off to Mr. Tardif.

bravo!
encore!

Vaoh

kaer426 wrote: »

I am legitimately impressed with Zenimax's feedback in this thread.
If we could create more straightforward communication between players and game devs, similar to this, ESO:TU could have limitless potential.

Hats off to Mr. Tardif.

bravo!
encore!

Completely Agree! If communication was always this way, these forums would not be toxic. The "Positive Thread" would become every thread. I really appreciate this ZOS dev going out of his way to help. #PraiseAlexTardif

maboleth

Hats off to Alex Tardif! Thanks!

ZOS_AlexTardif

Thank you all for the feedback!

Doncellius wrote: »

@ZOS_AlexTardif PS4 player here! I want to let you know that the Thieves Guild Base Game Patch has caused FPS on consoles to take a hit as well during fights, especially when intensive. It even becomes noticeable when inside of group dungeons fighting or sprinting/using Bolt Escape. I'd estimate the FPS drops from the goal of 30 to the 20's range but can't tell since I'm on console. These FPS drops can also happen when I first start the game, and are not a result of some sort of memory leak over a long period of time.

Particle effects and ability visuals showing up has been a new detail I have noticed with this patch. Before, visual spell effects would disappear especially during PvP in order to help the performance stay as steady as possible. Now I can see everything going on but FPS is taking a very noticeable hit.

I have a video of myself in Veteran City of Ash which very clearly showcases this FPS issue if it helps.

@Doncellius thanks for the info! Your framerate issues in bigger fights are likely part of the same drops we're seeing on PC. Most of the time, that kind of thing affects all platforms. As for the FPS hit from particles, that brings up a good point. We may have had higher FPS before just as a result of particle effects leaking and not showing up, and it'd be a good idea for us to re-evaluate our limiter for better performance on consoles.

zerosingularity wrote: »

I for one am genuinely interested in these features and changes, and would love to hear about them as they relate to ESO. I am sure others would be as well, and it might make a neat segment on a future ESO live or a dev blog or something.

Sounds like a great idea! We have mentioned them in passing on a couple occasions, but to recap, the more immediate things on the horizon involve a movement toward compute shaders, thanks to us now being DX-11 specific. https://msdn.microsoft.com/en-us/library/windows/desktop/ff476331(v=vs.85).aspx

The short-and-sweet is that we'll be able to offload a lot of CPU-based logic onto the GPU. The bigger items on the menu for this would be character-compositing (the way we stitch together all the parts of a character), terrain tessellation, and GPU particles. Of course, these are subject to change as internal needs change, but between this and bug fixing, this is the kind of stuff we're looking at from the graphics side.

sirinsidiator

@ZOS_AlexTardif I made some traces today. I compressed it a lot but it's still 3GB of data. I will upload it over night and send you a pm with the download instructions tomorrow.

Tarrin

@ZOS_AlexTardif Would be CPUID Hardware Monitor logs helpfull?

Scraelos

I can not understand, have you fixed their issues with FPS drop?
Some people from our community are expierencing serious FPS stability issues. Technical support also replied them with link to this topic.

Seravi

I haven't played since the TG release due to the FPS issue. Reported the FPS issues on the PTS over and over again before the release.

sirinsidiator

@Scraelos It looks like they found the (or at least one) reason why the FPS would drop permanently. @ZOS_AlexTardif wrote in a pm that they noticed and fixed a leak in the animation system and once it is tested and proven to work there will be an official statement.

ZOS_AlexTardif

Thank you everyone for your patience and information, and willingness to help us out with logs, traces, images, and videos. Thanks in large part to you guys, we have finally been able to reproduce this internally, have pin-pointed the issue, and are testing fixes as we speak.

It was indeed an issue with some threaded code. We kick off multithreaded tasks in our game to do all sorts of things, but we had one type of task that was spammed such that it would eat a whole core and everything else would be waiting on it. This may have also created some perceived lag because the client would choke up from this and only be able to intermittently process other tasks.

We're still focusing on creating better CPU-usage spread, reducing spikes, and continually picking at server performance to make the overall experience better. However, the fix we're working on will solve the issue of one core choking up indefinitely.

Huge shout-outs to @sirinsidiator for the traces and videos, @Paulington, @amaniacub17_ESO, and everyone else here who has submitted information regarding this issue. It was invaluable.

Septimus_Magna

FPS is totally messed up in Cyrodiil, the only way I found to reduce fps drops and crashing was to disable Normal_Maps in the UserSettings.ini file. The details might have less depth but the fps stays above 20 in large scale fights.

4,0 GHz CPU, GTX970 GPU, 16GM RAM, ESO installed on SSD etc, still my fps drops to 1 or 2 in large fights with Normal_Maps enabled..

Taonnor

My best fix for FPS issues in Cyrodiil was to reduce the particle effects from 768 to 128 via UserSettings file. I see still all effects that i want and in large fights the screen is not overcharged with too many effects.

Xeven

ITT ZOS Graphics Engineer works with players over the course of three months to identify/squash bugs and improve performance.

WTB Combat System Designers with the same passion.

sirinsidiator

@ZOS_AlexTardif The most invaluable part about this whole situation is that you kept up a meaningful dialog, so a huge thank you to you too. This is a prime example how two way communication between developer and community can work. I just wish that other people at ZOS will follow suit and at the same time that players also learn to stay objective so we can see more like it in the future.

@Xeven To be fair, the combat/alliance war related threads are usually filled with a lot of ego and quickly become something where no meaningful discussion is possible. In many cases I can understand why nobody at ZOS wants to reply there.

ZOS_AlexTardif

Hi again everyone!

Our first attempt at fixing the permanent FPS drop will be in Monday's incremental patch (for PC). We believe there's more to this particular issue than what we discovered on the surface, so we're continuing to dive deeper into a number of systems to find any other potential root causes, as well as giving a hard look at certain systems that we know can perform better in general. We've also added some internal metrics so that we can better monitor issues like this one in the future before they go to Live.

After you guys get the patch, we'd love to know if it has improved or resolved the permanent FPS drop. We'll be checking this thread for any updates/information, and will be in Cyrodiil as well to see the effect of the changes. This is best tested by going into Cyrodiil and watching your Task Manager performance for one core maxing out constantly and never going back down, even after fights have completed. We still expect uneven distribution across cores with some doing more work than others (which is what we're working on now), but we're expecting improvement (or resolution) to the permanent drop.

zerosingularity

See, THIS is what we players want. You actually talk to us and describe how your helping.

There are still FPS issues in places like vMA where you will get massive FPS drops until you relog. I have not checked to see if one of my cores are ramping up though, so I am not sure it is related. That said, thank you for keeping us in the loop.

+1

Rune_Relic

Thanks for the constant updates and endeavour @ZOS_AlexTardif ....and your sidekicks help.
Even the slightest news of some progress removes the fear that nothing can be or is being done.
You've not backed away from engaging with the players and saying how it is ..walts and all.
I for one cant thank you enough.

Helluin

The issue I have since IC patch is a random drop to 8-15 fps for a few (5-20 sec), then fps are back to normal.
I hope these fixes will help also to solve this problem.

Thanks for this and for the updates @ZOS_AlexTardif .

OmniDevil

ZOS_AlexTardif wrote: »

Hi again everyone!

Our first attempt at fixing the permanent FPS drop will be in Monday's incremental patch (for PC). We believe there's more to this particular issue than what we discovered on the surface, so we're continuing to dive deeper into a number of systems to find any other potential root causes, as well as giving a hard look at certain systems that we know can perform better in general. We've also added some internal metrics so that we can better monitor issues like this one in the future before they go to Live.

After you guys get the patch, we'd love to know if it has improved or resolved the permanent FPS drop. We'll be checking this thread for any updates/information, and will be in Cyrodiil as well to see the effect of the changes. This is best tested by going into Cyrodiil and watching your Task Manager performance for one core maxing out constantly and never going back down, even after fights have completed. We still expect uneven distribution across cores with some doing more work than others (which is what we're working on now), but we're expecting improvement (or resolution) to the permanent drop.

Can we expect something similar for console?

Since Thieve Guild launch, on console, simply moving from Point A to Point B in a low populated town/city displays a noticeable drop rate in frames. If it's not a FPS issue, then maybe it's a processing lag problem. Either way, the screen freezes for a half second. Not long, but long enough to notice something isn't right. It has made combat in both PvE and PvP unplayable for me and frustrating for others.

- Tired hard resetting my Xbox One
- Restarting ESO

Nothing helps.

Darkstorne

ZOS_AlexTardif wrote: »

After you guys get the patch, we'd love to know if it has improved or resolved the permanent FPS drop. We'll be checking this thread for any updates/information, and will be in Cyrodiil as well to see the effect of the changes. This is best tested by going into Cyrodiil and watching your Task Manager performance for one core maxing out constantly and never going back down, even after fights have completed. We still expect uneven distribution across cores with some doing more work than others (which is what we're working on now), but we're expecting improvement (or resolution) to the permanent drop.

Has the patch gone live yet? Or did you mean next Monday (18th)? Because I'm still experiencing the issue in PVE. After an hour or two of play my FPS is around half what it should be and I need to restart the game to fix it.

sirinsidiator

@Darkstorne the patch has been delayed by one day as announced here.