Game Performance in Cyrodiil

cdc28preub18_ESO · November 2014

@ZOS_JessicaFolsom

As others have pointed out, the main problem the community is complaining about is very clearly not a FPS performance issue. I would describe it as a large scale client/server syncing issue.

For those with adequate and properly set up systems, no client-side changes we make affect this at all. No changes to our internet bandwidth affect this at all.

This is an issue where everyone in a battle, on both sides, are unable to get actions from the client through to the server. This also affects the surrounding area, even if out of sight of the majority of the combatants (outside a keep for instance. We hit abilities, try to change weapon bars, try to go through doors, etc, and experience anywhere from 10-30 second delays until the actions occur, if they ever do. During these periods we take very little damage as well. I see the log showing my abilities are doing damage, but no one takes significant damage or dies.

From my experience, this most often hits critical levels once too many people get stacked in the same physical space and hitting abilities. I've been at Bleakers with a large force on both sides and the attackers trickling in through a broken wall. As long as they don't bum-rush the hole, the game doesn't lag out too bad - it still does somewhat, but is manageable. Small delays of 1-2 seconds on abilities and such. But once a large enough group attacks at once and tries to force their way through the gap - bam, instant lag/sync issue for everyone. 95% of kills stop for both sides at this time.

From my limited knowledge of this sort of thing, imo the only way this can be fixed is to either:
- decrease the load on the server during these big fights
- if the servers appear to not be overloaded at these times, figure out WHY the servers are not effectively dealing with the client/server communication and syncing
- improve the server capability (hardware upgrades)
- improve the server's bandwidth

If you have other ideas that don't rely on telling us to try to fix it on our end individually then great!, but please, please understand that this is not an individual problem. This is a game design/server capability/server bandwidth problem that affects everyone.

Thanks!

GorraShatan · November 2014

Some feedback for you:

It's everyone. Any setting. Any add-ons or none. Seriously. Ask in zone during a lag spike. It's everyone. If you guys don't know whats going on that seriously troubling.

Seriously guys, even better communication on this would be nice.

Cherryblossom · November 2014

The most recent patch has made a significant improvement to performance in Cyrodill. Last night, not one crash. There was still lag, but not as bad as it has been.
The improvements you have made seem to be going in the right direction.

If the performance continues in this direction I may be able to convice some old guildies to come back.

sirston · November 2014

Rune_Relic wrote: »

sirston wrote: »

@Zos_jessicaFolsom
have your Team run with the big guilds of the servers as a wing-man or a camera crew and watch from a safe distances and view any lag first hand, im pretty sure the guilds would be more then happy to bring them along.

And why would they suffer server lag when they are playing over the LAN ?

Because No computer can escape hundreds of people at one keep, also it wouldn't matter because this game is not bias towards anyone's internet.

okraus · November 2014

I think there is not only one problem that is causing the bad "performance" in cyrodiil. There are several issues and some of those do interfere with each other which makes analyzing them even more difficult and the end-user has actually no chance to really help with a root-cause analysis because you would need system output (much more than a fps display) and log-files to evaluate the situations where performance drops significantly.
But besides this it all comes back to bad designed client-server infrastructure and bad network code!
Players can give the ZOS engineers only a "feeling" about the bad performance situations and what might have caused it. Every time ZOS claimed that they fixed or improved performance in cyrodiil i noticed a different lagg behavior. At the moment i would describe it like this:
In bigger fights in cyrodiil (not only large-scale battles with 100+ ppl involved) the server is now actually queuing your commands to trigger your abilities. Strangely it goes out of sync with the movement of your character. Which means that you run around as if there were no lagg at all but your abilities do not fire off. They actually do that with 2-20 seconds delay which results very often in very bad situations for the player: Impossible to use doors; impossible to react on triggers like crystal shard proc --> you want to fire the instant off and wonder 10 secs later that your character starts casting it; streaking of walls or inner keeps when you triggered the streak in a fighting situation in the keep yard, etc. That is really frustrating but it is totally different to the experience of lagg or bad performance in the past. Weeks (months?) ago it felt like the client-server communication was swallowing your commands and so they were lost and the abilities never triggered. I remember bigger fights where almost noone could do anything. Sometimes a few ppl came through with their commands and it could happen that you died out of nowhere. The situation got worse and worse ending in a zone crash, scroll reset, etc.
So i must give the ZOS engineers credit - they changed this but it did not get better and the situation is far far far from acceptable!
I love PvP but my patience got limits and i think it never happened before in MMO that i lost the fun in PvP and focused on the PvE content to bridge the time until PvP in ESO got fixed (and maybe some new content but i don't want to sound too brisky). What can i say... i currently level my DK twink. $:\$

Cheers!

ZOS_JessicaFolsom · November 2014

Thank you all very much for the replies and detailed information. It's helped immensely, and we're getting closer to pinpointing the cause of the lag and unresponsiveness in Cyrodiil that many of you have reported.

After further investigation, and as some of you surmised, we now believe this is an issue with client-to-server communication responsiveness during high volume combat situations. We would love more feedback on what specific abilities/actions stop working when this happens, and which continue working. For example, does chat still work? Mounting? Weapon swapping? Projectiles? Self-target abilities? Any details you can provide will help us better track down this specific issue and make improvements.

DovresMalven · November 2014

Chat still works, and so does my flame whip from time to time. Sometimes I can even sneak and feed on someone during the lag which is hilarious.

Orchish · November 2014

ZOS_JessicaFolsom wrote: »

Thank you all very much for the replies and detailed information. It's helped immensely, and we're getting closer to pinpointing the cause of the lag and unresponsiveness in Cyrodiil that many of you have reported.

After further investigation, and as some of you surmised, we now believe this is an issue with client-to-server communication responsiveness during high volume combat situations. We would love more feedback on what specific abilities/actions stop working when this happens, and which continue working. For example, does chat still work? Mounting? Weapon swapping? Projectiles? Self-target abilities? Any details you can provide will help us better track down this specific issue and make improvements.

Any charge ability during lag will cause you to end up stuck in the charge animation, yet you don't actually charge the target. This happens to me on a daily basis during prime time when the server has it's lag spikes during big battles. The same for snipe, you will get stuck in the bow raised animation, but it can take something like 5-6 seconds to fire off.

The worst thing about this is i can still be attacked and killed while stuck in the animation. Weapon swap works, but it's always delayed depending on how bad the server lag is. Usually it's a 2-4 second delay, which doesn't sound bad but it's still very noticeable. In worse situations it can take up to 10 seconds. Usually as the battles are so hectic people will press weapon swap and notice nothing happen, so they press it again and again, then once the server catches up your weapon will swap and swap back, basically just messing everything up which usually leads to your death or you throwing your mouse at the screen. Chat is exactly the same as in it works, but it is delayed. So if i type in zone chat "lag" it will appear a few seconds later. Same goes for mounting, pressing E to open a door etc, most of the time during lag they all still work, just delayed exactly as you would expect when a server or player is lagging. Although this isn't just 1 player with bad internet, as every one on the server at the same time will experience these issues.

There are a few videos posted on these very forums showing what happens during the lag. Many of them show large groups from opposing factions standing on the exact same spot spamming AoE, i am not saying this is the cause, but when this happens nobody seems to take any damage. Players like my self with single target abilities cannot do anything, while it does seem anyone using aoe such as impulse can still activate the ability. Although as i said, often when i charge into these blobs i take no damage at all. FTC addon doesn't report any damage and my health stays the same. Then after 30 seconds or so you get one of two things happen, either the server will crash or one of the groups starts dropping dead instantly as if all the damage suddenly catches up and takes effect half a minute later.

Izanagi.Xiiib16_ESO · November 2014

Let us know when you want to monitor lag on EU thornblade I'm sure we can demo the problems to you fairly quickly...

Also i missed the point about Camp removal it has actually worsened the lag. It's been worse since update 5.

eventide03b14a_ESO · November 2014

Your game forces us to use these addons because you flat out refuse to include some very basic and time saving additions to your own UI. I have heard nothing about any plans to improve the UI. I frequently go back and uninstall old addons that are either out dated and broken or simply no longer needed because I use a better one. Can you please look at the UI and really improve it's functionality so that we don't have to rely on addons? I am going to just point out right now that I am not talking about HUD I am mostly talking about the menu and it's lack of functionality. Although the HUD could certainly use some work, like an optional mini map, the menu is just awful. Some addons are also useful because you decided to make things a chore. There is no native way to check your recipes or crafting knowledge without taking a trip to a crafting station. Why? There is no native option to create multiple, instead you have to hit create 50 times. Why? It's not a minimalist style approach, it's just lazy. I'm sorry but no other MMO I have played do I have to go to a crafting station to see what I can make. Ridiculous. I need an addon to see what I have sold, because all I get is a lousy mail saying how much I have made but not what was sold and to whom. Seriously? Finish your UI because it's like someone started it and then just sort of gave up halfway through.

frosth.darkomenb16_ESO · November 2014

Izanagi.Xiiib16_ESO wrote: »

Let us know when you want to monitor lag on EU thornblade I'm sure we can demo the problems to you fairly quickly...

Also i missed the point about Camp removal it has actually worsened the lag. It's been worse since update 5.

Well it reduced the lag due to having fights last less long and reduce the strength of certain type of tactics.

Something else in the patch is the cause of this new found lag.
If the camps were not removed (or reduced, as they actually are) it would be even worse.

Thechemicals · November 2014

So addons cause crashes, yes. Ive only had 2 crashes in a month with like 7 out-of -date addons. The question is....do addons cause lag hell as well? The kind where im posing in snipe animation for 30 seconds? The kind where i cant even go near a battle?

ZOS_JessicaFolsom · November 2014

Thechemicals wrote: »

The question is....do addons cause lag hell as well? The kind where im posing in snipe animation for 30 seconds? The kind where i cant even go near a battle?

We don't have reason to believe that addons are causing the lag, no. Just the most common client crash.

GorraShatan · November 2014

ZOS_JessicaFolsom wrote: »

Thank you all very much for the replies and detailed information. It's helped immensely, and we're getting closer to pinpointing the cause of the lag and unresponsiveness in Cyrodiil that many of you have reported.

After further investigation, and as some of you surmised, we now believe this is an issue with client-to-server communication responsiveness during high volume combat situations. We would love more feedback on what specific abilities/actions stop working when this happens, and which continue working. For example, does chat still work? Mounting? Weapon swapping? Projectiles? Self-target abilities? Any details you can provide will help us better track down this specific issue and make improvements.

Chat works fine. Mounting I'm not sure and his hard to test, as you are generally going to be in combat when this happens. Charges get you stuck in the charge animation. Doors do not work (or I get frustrated and leave the door before it registers, not sure). Weapon swapping works. Any skill that self-targets seems to work (as does an AOE heal on yourself). Nothing that seems to affect anyone around you though, nor are you affected by people around you.

Heals and damage from other players seem to come in pulses. I'm not sure if it's only AOE spells that eventually register or not, just that they come in big waves. I'll try to see next time it lags.

Putting down siege weapons seems to not work.

Thechemicals · November 2014

ZOS_JessicaFolsom wrote: »

Thechemicals wrote: »

The question is....do addons cause lag hell as well? The kind where im posing in snipe animation for 30 seconds? The kind where i cant even go near a battle?

We don't have reason to believe that addons are causing the lag, no. Just the most common client crash.

Thank you. Knowing this really clears some things up for me.

Izanagi.Xiiib16_ESO · November 2014

ZOS_JessicaFolsom wrote: »

Thechemicals wrote: »

The question is....do addons cause lag hell as well? The kind where im posing in snipe animation for 30 seconds? The kind where i cant even go near a battle?

We don't have reason to believe that addons are causing the lag, no. Just the most common client crash.

Thanks for the new updates I'll provide as much information as possible in future posts for now could you also go have a look at the low population bonus issues reported in the Alliance war forum. AD (my faction) is now winning the campaign because of this 'bug' but its been responded to as 'intended' behaviour. Don't get me wrong I'm happy we are winning but I'd prefer we could do it on our own performance and not be boosted by somehow qualifying for a low pop bonus on the most active campaign.

cdc28preub18_ESO · November 2014

@ZOS_JessicaFolsom‌

Keeping in mind that sometimes your character (and everyone else's, as confirmed by voice-chat) can become entirely unresponsive temporarily, invalidating the below, these are the more normal observations.

-aoe's work (but lag badly)
-abilities that don't require a target work (bow shots, for instance) (but lag badly)
-entering/exiting a keep lags badly but works sometimes - my perception is that it lags so badly it fails at times
-weapon-swap sometimes works after a long lag, sometimes fails entirely

-ultimates, even untargetted, seem more difficult to activate than a normal untargetted ability. not sure why - maybe because they have to check for ultimate before activating in a different way than normal skills check for sta/mgk? I know your ultimate system is entirely different then normal skills, based on other things you've said in the past, so maybe...

-targetted abilities are hit and miss - some work, in that they activate eventually but the target is long gone (ambush empty air, for instance), and some seem to just not work. Those with long animations are particularly bad - snipe and charge for instance, and lock you out of any other actions for up to a minute at times.
-self-targetted abilities seem to work like aoes and non-targetted abilities.

-chat - I don't really know - most of us use voice chat programs, which function fine while the game is stuttering along

Thanks.

Rylana · November 2014

ZOS_JessicaFolsom wrote: »

Thank you all very much for the replies and detailed information. It's helped immensely, and we're getting closer to pinpointing the cause of the lag and unresponsiveness in Cyrodiil that many of you have reported.

After further investigation, and as some of you surmised, we now believe this is an issue with client-to-server communication responsiveness during high volume combat situations. We would love more feedback on what specific abilities/actions stop working when this happens, and which continue working. For example, does chat still work? Mounting? Weapon swapping? Projectiles? Self-target abilities? Any details you can provide will help us better track down this specific issue and make improvements.

You can use a keep wall masonry repair kit during these times to repair keep walls

But you cant drink a potion or drop a siege weapon.

I assume in the former youre applying an effect to a model already loaded into client memory allocation so can view the result without a server check. In the latter, both require server sync checks (both for stat change and for object placement for other client renders)

Also, the worst lagbombs have an interesting effect. Without character models running around in super high speed, it is plainly obvious at times that actual damage, effects, and siege animations spend a short period of time "catching up".

The effect would be like hitting the pause button on a movie, except the actors can still move but the environment or interactions do not move/work. Then all at once all those previous interactions "fire" all at the same time. (imagine a baseball game, where the batter has hit to deep center field, he starts running the bases, the outfielder moves to catch the ball, but the ball stops in midair. the baserunner makes it to home, and then after a while the ball falls from the sky and everything catches up in real time with all the players on the field scratching their heads whether its a home run or an out)

What I believe to be happening, is that we are actually overloading the megaservers ability to process data fast enough to render it in real time. We actually get ahead of what the system can handle, and it compensates by desyncing, piling up the data, and pushing it all through at one time. So to us, it looks like we are running around together, unable to attack each other or do damage, but the damage is just being saved up like in a balloon about to pop. In the worst cases, this desync lasts so long the server returns us to spawn (the so called crash) because it can no longer locate us and assumes we left the game/logged out.

Essentially data transfer packets are exceeding the throughput of the system, and even keep-alive packets are being lost in the shuffle, to use some more technical terms your devs might understand.

Running · November 2014

Yesterday Thornblade 09:00pm Bleakers Outpost: Red supergroup vs DC pug train
10 min lag and no winner
> Skills, ulti work, but lag badly and no damage
> Weapon swap, mount, emotes, chat work

When the fight was boring, and players gone, the server work fine.

frosth.darkomenb16_ESO · November 2014

A discussion that occured on another thread a while back had the theory that this was a "live lock" due to smart healing and, potentially, damage asignement.

As the game tries to decide which target to heal/hit, if it wanted to have accurate results, it would lock the states of all potential targets so that it doesn't change until it decided.

This shouldn't be an issue in a normal situation. However, the way people play, we spam spells even when not necessary and wih the stacking, that spam gets multiplied by a large factor.

But what happens if two abilities occur at the same time on a similar set of targets, and player 1 gets locked to ability A, and player 2 gets locked to ability B?
Both abilities would have to wait until all their targets become available to their calculation, but this will never occur. But everything else, not pertaining to health or some other effects applied on targets, will continue to be processed because the servers aren't actually having processing issues, just the logic is stuck.

So in the end, everyone's hp get's locked and effects keep on pilling up until at one point, the proper player decides to log off, and everything resolves at once.

This is a big "IF" though, perhaps there is no locking of the list and accurate results aren't attempted. But it could be a lead as to where the problem comes from. In my opinion, accurate results shouldn't be attempted as in most cases, they would be "good enough".
And of course, player behavior needs to be addressed.

Izanagi.Xiiib16_ESO · November 2014

Worst offenders during the 'lag' are.

Snipe - Animation freezes when the bow is pulled all the way back.
Critical Charge (and its morphs) - Animation freezes in the charging state
Shield Charge (and its morphs) - Animation freezes in the charging state
Ambush
Bolt Escape

Depending on how bad it is when you use the abilities your character will perform the animation but not move at all, then you can run around still in the animation and finally it might complete and teleport you somewhere where your original target was but generally for no damage.

AOE -
Abilities such as Impules, Sap Essence, Talons etc will play the animation but rarely damage - although they do give some damage as ultimate will go up slowly.
Healing aoe such as Healing springs appears to work.

Synergies -
Synergies will all generally work for example the lightning splash synergy if a sorc manages to eventually cast it will work.

Ultimates:
Bats - Appears to works fine during lag
Magma armor - Appears to work fine during lag (DK's can confirm).
Veil - Plays animation but doesn't drop on the floor will eventually place itself but often where you first used it - this can be 10m behind where you are now.
Negate - Same as veil doesn't work.

Siege can be placed but when you are putting it up it will be invisible, once this happens it will be lost once it completes (its there but you can't use it).
Also siege that is already up will lag so when you fire a shot it wont fire until just before the next shot is fired.

Grim13 · November 2014

Here's an example of the server abuse:

In this instance it's EP. Although, I've heard tales of all factions doing it.

I had just arrived in Cyro, transited to Fare, where I couldn't get any of the keep doors to allow me outside... so, I went to the top floor and jumped down to the outer wall... go to the eastside and what do I see?

A bunch of EP huddled in a ball spamming abilities... intentionally lagging the server.

It's the new (old) exploit, it would seem... just another in a long line of *** ridiculousness.

Skills take on massive delays, siege will drop but cannot be used,... walls repaired while still taking damage...

B S

Running · November 2014

Red zergball, and 10 min lag lag lag, no damage, laggy skills, etc...

Another day: (Yellow zergball, with pulsar, batswarm, healing spam)

LunaRae · November 2014

ZOS_JessicaFolsom wrote: »

Thank you all very much for the replies and detailed information. It's helped immensely, and we're getting closer to pinpointing the cause of the lag and unresponsiveness in Cyrodiil that many of you have reported.

After further investigation, and as some of you surmised, we now believe this is an issue with client-to-server communication responsiveness during high volume combat situations. We would love more feedback on what specific abilities/actions stop working when this happens, and which continue working. For example, does chat still work? Mounting? Weapon swapping? Projectiles? Self-target abilities? Any details you can provide will help us better track down this specific issue and make improvements.

Chat works (all forms, such as whisper/zone/group/guild) - hardly ever notice responsiveness issues.

Combat for me - Breathe of Life (Templar) almost ALWAYS works, even in the thickess of lag, immovable almost always works, blazing shield is usually responsive and so is radiant aura. Charges are a NO-NO (such as toppling charge or shield charge), weapon swapping is extremely bad but if you're patient it usually works after 2-7 seconds depending on lag severity. Supernova is a big no no, I have to spam the ultimate button about 40 times before I get it off, and then I see no on-screen indicator - just my ultimate points go to 0. Heavy attacks never work, but light attacks seem a lot more consistent. I don't recall ever having troubles dodge-rolling, so fairly responsive there - same with blocking. That's all the feedback I can think of for now.

ZOS_JessicaFolsom · November 2014

Thank you all for the helpful replies to our follow-up questions. Our engineers and programmers greatly appreciate it.

AbraXuSeXile · November 2014

Whatever the 1.2.3 patch was caused the lag, Even the heaviest combat before then wasn't laggy at all.

Since 1.2.3 patch this game has been horrendous.

Poxheart · November 2014

ZOS_JessicaFolsom wrote: »

We would love more feedback on what specific abilities/actions stop working when this happens, and which continue working. For example, does chat still work? Mounting? Weapon swapping? Projectiles? Self-target abilities? Any details you can provide will help us better track down this specific issue and make improvements.

Here are the things I notice when this happens:

Dual Wield Light/Heavy Attacks the attack animation will play but you'll do no damage.

Restoration Light/Heavy Attacks the animation plays intermittently and rarely does damage.

Ambush the skill's animation happens, but you do not teleport to your target, do no damage, can move around and your skill bar is not grayed out. After a time (5-25+ seconds depending on the severity of the lag) you will teleport toward where you would have landed had the skill worked properly when first triggered.

Shield Charge the classic Scooby Doo animation where your legs flail about but you can't do anything because your skill bar is grayed out. Lasts anywhere from 5-25+ seconds depending on severity of lag.

Talons the skill animation plays and sometimes the talons show up, but enemy models are not rooted and no damage occurs.

Cinderstorm the skill animation plays, but the ground effect does not show and no damage occurs.

Funnel Health the skill animation plays, but does no damage.

Elusive Mist this skill works, but there is often a delay (5+ seconds depending on severity of lag).

Shadow Cloak this skill effect is often delayed. What frequently happens is that you trigger it multiple times but the animation does not play, then after a delay the animation will play back to back for as many times as you had triggered the skill.

Crouch going into crouch is often delayed. After hitting your crouch key nothing will happen. If you hit it again, then at some point once the lag has died down you will go into and out of crouch. This can happen multiple times if you attempted to go into crouch multiple times.

Mounting this is typically impossible to do because you are stuck in combat.

Weapon Swapping this is typically delayed anywhere from 5-25+ seconds depending on the severity of the lag.

Chat have never really noticed delay in chat.

ewhite106b16_ESO · November 2014

Usually when severe lag happens I can move and see other players moving normally. Light/Heavy attacks can be done but I don't see the damage. Most of the time I can't get any abilities self targeted or otherwise to work, weapon swap is also disabled. Clicking on keep doors and teleporters also doesn't do anything. There is no problems with my personal FPS in these cases, usually if I"m with a group some or all of the other players are experiencing the same server lag.

At times this lag appeares to impact the entire Thornblade instance of Cyrodil, I become unable to open doors/teleports/use abilities or mount when nowhere near a large fight or zerg. More commonly the lag happens in close proximity to a large fight.

Lastly, when the lag ends and I've attempted to use several abilities and/or swap weapons repeatedly all the commands will execute in quick succession as though they were queued up server side. On the upside, I haven't had any crashes in Cyrodil since resubbing when update 5 came out.

Izanagi.Xiiib16_ESO · November 2014

ZOS_JessicaFolsom wrote: »

Thank you all for the helpful replies to our follow-up questions. Our engineers and programmers greatly appreciate it.

No problem, any further updates please let us know its been terrible over this weekend on Thornblade EU - various posts and videos about it available in pvp forum

Sensesfail13 · November 2014

Just STOP. STOP performing maintenance, it gets worse and worse every time you "fix" something and after this 6GB "fix" I have more lag and more problems weapon swapping than I EVER have before which is saying something.