Update on Cyrodiil Performance

Sylvyr

C0pp3rhead wrote: »

Update: PvP is still buggy and suffers from lag.

Thank you for that update. Will keep an eye out see if it gets better.

Sotha_Sil

Bouvin wrote: »

I have a friend beta testing Camelot Unchained and he says the PvP is sick .. well the whole game is pvp so it has to be.

http://camelotunchained.com/v2/

Wait until the game is released and the servers can't handle the masses of players....

glak

They should have a vet Campaign starting with no AoEs except siege and Templar healing AoEs whereby every couple days they introduce another AoE ability via hot fix.
This is a very old suggestion. May have to be done on Live for it to be fruitful. Because there will be no lag on such a campaign with 200 players on screen, players are sure to use it.

A lot of abilities are AoE and thus this suggestion translates to total BS. But we know that AoE abilities and feedback from their use causes the lag, so this is a starting point.

khaza

Lag is not gone, there is at the moment a group of reds causing alot of lag on Thornblade.
Feels kinda strange that it's almost always against one specific red train that the lag starts.
Makes me wonder if they are using some scripts or addons that are causing the lag.

I won't point any finger against anyone, just curious if this is what makes it lag.

Efficient

Bouvin wrote: »

McGeesus wrote: »

These are megaservers everyone. MEGA. There are no lag issues. I can't believe you would think otherwise, as Zenimax is clearly the best. I mean, as proof, just look at their quick responses, and they seem to be always diligent in implementing changes that the majority of the community suggests. You guys need to re-evaluate your hardware.

For those posting suggestions, why bother?

It's not a hardware issue.

It's a software issue.

Had they gone with Nvidia PhysX, this wouldn't be a problem.

Thought my sarcasm would have been obvious. Sorry.

khaza

Ok, it is confirmed...

They have not fixed the lag issues at all, had a fight 10 vs about 20 and lag spiked to 2000+ PR.

So the performance fix that was posted in news is all lies.

khaza

I invite all GM's to come play with us on Thornblade during primetime, and then tell us if they enjoyed playing in the lag...

Bouvin

Sotha_Sil wrote: »

Bouvin wrote: »

I have a friend beta testing Camelot Unchained and he says the PvP is sick .. well the whole game is pvp so it has to be.

http://camelotunchained.com/v2/

Wait until the game is released and the servers can't handle the masses of players....

I thought you were talking about ESO there for a second...

Bouvin

McGeesus wrote: »

Bouvin wrote: »

McGeesus wrote: »

These are megaservers everyone. MEGA. There are no lag issues. I can't believe you would think otherwise, as Zenimax is clearly the best. I mean, as proof, just look at their quick responses, and they seem to be always diligent in implementing changes that the majority of the community suggests. You guys need to re-evaluate your hardware.

For those posting suggestions, why bother?

It's not a hardware issue.

It's a software issue.

Had they gone with Nvidia PhysX, this wouldn't be a problem.

Thought my sarcasm would have been obvious. Sorry.

gard

Bouvin wrote: »

McGeesus wrote: »

These are megaservers everyone. MEGA. There are no lag issues. I can't believe you would think otherwise, as Zenimax is clearly the best. I mean, as proof, just look at their quick responses, and they seem to be always diligent in implementing changes that the majority of the community suggests. You guys need to re-evaluate your hardware.

For those posting suggestions, why bother?

It's not a hardware issue.

It's a software issue.

Had they gone with Nvidia PhysX, this wouldn't be a problem.

Silly me, I was under the impression that scene rendering was done client side.

PainfulFAFA

houimetub17_ESO wrote: »

There are some issues that could be adressed right now to at least help with lag.

1- AOE caps. Completely remove them. I'm pretty sure that all these calculations stresses out the server. Gameplay wise, its not going to to completely *** the gameplay.

2- Have a look at healing AOE abilities like healing spring, Combat prayers and morphs. Those abilities, when spammed, does create lag. Especially in huge numbers. Solutions like increasing cost and incrasing power of the heals could help.

3- Ressources Pools. The way regen currently works players have unlimited ressources pools to cast certains abilities that end up causing lag. A positive example would be non-vet campains. Although there are comparable numbers on this server than on Chillrend NA, there is less lag. Abilities are a bit less spammable.

4- animations and graphics of certain abilities (example. purge, impulse, wall of elements, sorc stormcalling armor buff, meteor, dk standarts). Some just seem to overload a players screen and cause lag. Some ult also seems to get spammable and have multiples anmiations of it (dk standarts and meteor are great examples). Maybe have an option to turn them down a little.

This. ZOS needs to step its game up with all this issues, unfortunately publicly revealed no new changes for ESO until after console (assuming after all the patches and bug fixes since its ZOS we're talking about)

ESO used to be fun without the AOE caps but 1.6 came around and changed the entire game and now the servers cant handle it. How is it that ESO wasn't the way it is now back then?

maryriv

FadedJeans wrote: »

What is the actual problem?

Is traffic bottlenecked somewhere?

Are hardware resources being tapped?

Are there clustering inefficiencies?

Are there application inefficiencies?

Surely someone is staring at a graph right now, looking at a spike and saying "well, THAT's not good", yes?

Personally I think the majority of it is client side. I rarely ever get these lag problems so that tells me it may be these peoples hardware configurations or internet.

Folkb

so like, when there's a bunch of people in one spot you lag alot, the end.

Kahrgan

March 13th post recognizing problem, month and a half later, nothing fixed. Stop saying "we know" and start saying " We fixed it"

Tyvarra

My PvP guild is falling apart and all players in zone chat are just arguing how stupid all is. Much frustration these days...
If you dont start to fix it you can soon close completely.

I fell in love with the PvP and I hate you sooooo much for destroying it >.<

Kupoking

after PTS event pvp against devs withpeople spamming healing spring/efficient purge/combat prayer to crash the server, the message was clear I guess.

I hope the next update on the case won't be to ask players to spread out in smaller groups. If they do, they should remove the first 'M' of MMORPG of their game description.

khaza

Still Meteor spammer lags with rubber-banding effects causing 60k+ damage...

Maybe look into why Meteors are causing so much lag that this happens?

PainfulFAFA

khaza wrote: »

Still Meteor spammer lags with rubber-banding effects causing 60k+ damage...

Maybe look into why Meteors are causing so much lag that this happens?

I got hit by a meteor and froze mid-air on an open field (server lag)
As a result, I got hit by a 42k fall damage....

NadiusMaximus

Trying to keep it positive.

Any new updates?

Arito

NadiusMaximus wrote: »

Trying to keep it positive.

Any new updates?

Funny.

Cherryblossom

ZOS_PaulSage wrote: »

Just to clarify. There seemed to be a misunderstanding that I was saying we are "giving up" or "blaming this on players." Absolutely not. We are looking at the issue from many angles. I certainly didn't mean to imply we couldn't or wouldn't change code to improve performance, but rather that the answer wasn't more hardware. (It is often suggested this is the best way to fix problems.) I also wasn't implying you shouldn't play naturally, but unfortunately it is true that more players right now in a smaller area causes the performance problems. I say this not to discourage behavior or try push off the issue, but rather to give completely frank information about the root cause.

Well Paul, it's coming up to 2 months since the last update on Cyrodiil Performance, is it likely we will get any further details or plans on improvement.

Or have you stopped looking at this from many different angles, or are you doing one angle a day and we can expect 360 days before any update!

Bouvin

khaza wrote: »

[snip]

Seems like the only communication we get is when the mods have to remove comments, close threads, and tell us to keep it civil.

I really wish the developers would pipe in more often and back up the poor community managers who are really just delivering the message and keeping things civil.

[edited to remove quote]

khaza

We have big fights and then all of a sudden there comes a lag-train spamming meteors that doesn't go off which creates lag and rubber-banding effects again which kills you for 60k+ damage.

Just remove the skill until it is fixed damnit.

Dan_Miathail

ZOS_PaulSage wrote: »

Hello everyone,

Our Cyrodiil performance is something we are very aware of. Performance drags when there are numerous players in the same place at the same time. This is why performance in Cyrodiil is fine for much of the day, but gets worse during more popular times. We are currently investigating ways in which we can reduce the spike of performance loss. We added in some features for Update 6 which we hoped would help, but ultimately did not. This is not a situation where we can just add more hardware. Player population in a given area hurts the performance and the more people that are in one area, the more performance is going to be hurt.

Actively, we are looking at changing the behavior of the players to remove incentives for large groups to stay in the same area. We want to do this by providing larger incentives for Alliances to split up and take on multiple-challenges in Cyrodiil. We’ll continue to work on this. We are also asked by players if there is anything they can do to help. In this situation, the best thing you can do is split off to different objectives when you notice performance going down. Cyrodiil is a big place with lots of different things to do. And thank you for asking.

Problem:

Victory relies on the capture of Scrolls and crowning of Emperors and the easiest way to do these things is with numbers. Whilst the map is divided relatively equally yes groups can break off and attack multiple targets but eventually it is going to come down to one or two keeps standing in the way of a claim which inevitably forces populations into a single point.

It takes numbers to capture a defended keep, the more that show up to attack the more show up to defend and on and on it goes until everyone in the campaign is fighting over a single target. When one faction has multiple scrolls they throw all their numbers into a single keep to prevent the other side taking that scroll back, this requires more numbers for offence and again everyone rocks up at the same location.

At the moment there is zero incentive for populations to split up, victory is quite literally a numbers game, the faction with the biggest zerg wins. If you truly want people to spread out across the maps then you need to offer a reason for them to do so, something that will reward or help the faction and make it worth while for multiple groups to do multiple things.



Suggestions:

I know you say you can't just add more hardware to fix it but wouldn't taking PvP off of the megaserver and giving each campaign its own independent server help?

From watching interviews with Paul we know that you guys have already considered things like supply conveys moving between keeps, if these convoys gave players siege equipment it would get players out on the roads, setting up ambushes and so forth. They could give attackers a wagon to loot once captured or defenders one once the convey reaches a keep.
A keep under attack could certainly use some free siege being escorted through the back door.

Have random artifact conveys pop up, the npcs of a faction find a special whatever that grants a one time amount of AP and Campaign points to what ever side captures or defends it.

Something like bonus keeps would also help, have one or two keeps randomly flag for triple AP when captured or defended and triple campaign points while held. These keeps could change every other hour during the campaign. This would force players to choose between bonus keeps, emp pushes and scroll caps.

Make use of all the towns and such scattered across Cyrodiil, set them up to be capture points like outposts that send extra workers to resources to give further hit point bonuses to the walls of nearby keeps.

Make bridges worth something more than troll farms for gankers. Have them as capture points too that also grant a small amount of campaign points like outposts, etc.


P.S:

Please try and remember that large scale, high population battles where an advertised thing for this game.

https://youtu.be/j9O4HqEp0cI

JTorus

Dan_Miathail wrote: »

-snip-
Suggestions:

I know you say you can't just add more hardware to fix it but wouldn't taking PvP off of the megaserver and giving each campaign its own independent server help?

-snip-

I think there's a lot of misconception out there revolving around the megaserver. The megaserver isn't one physical server, it's a clustered environment. Think of like crowd funding, or even the Seti at home project. One task can be spread out and delegated to multiple resources for processing.

As far as the megaserver is concerned, ESO is nothing more than a MUD, yup, remember those? So basically what you have is this thing replacing this guy's job. As far as the data is concerned, ESO is little more than a database. You're essentially connecting to a SQL Server, all the pretties are generated on your PC. So when they say adding more hardware won't solve the problem, I truly believe that it won't.

Allright, so maybe it isn't the hardware, maybe it's their internet connection?

Close! They've stated in an ESO live that they have a pretty robust internet connection. Think of it in terms of where you live. Lets say ESO is the Mall, and your PC is your house. The mall is located in an area that allows for a significant amount of traffic, whereas your home does not, but that's okay. Your home has just your car travelling back and forth to the mall, whereas the mall is right off the highway, with a big parking lot, it can handle lots of traffic. I'll venture that it's a safe bet ZOS's net connection isn't the bottleneck.

ESO's lag is latency, not bandwidth. When we all look up and see our ping rate jump up into the thousands, this doesn't mean data is being transmitted slower, it means it's getting to us late. Jumping back to the mall analogy, lets say you've got an appointment at the mall for 3:00. We know it takes 10 minutes to drive between your home and the mall. You leave at 2:50, however for whatever reason, your car can only move at 5mph. That's slow bandwidth. High latency would mean your car can go 100mph, but you leave at 3:05. You're just late.

Okay so how is that relevant?

ESO uses UDP, this is a one way transmission. The server sends information to your client, and if you don't get it all, then it's discarded, and the next transmission merely updates. This allows for more information to be transmitted and faster. Losing a couple packets isn't a big deal, since you'll just get updated with the next one. Think of it like you're playing Marco Polo with the mall. You say 'Marco', and it takes 5 minutes before you hear 'Polo'. High latency means the time-stamp on the Mall's (megaserver's) reply is essentially an expired packet of information. It is no longer an accurate representation of the current conditions.

So what's the solution?

Bad news. Remember the botting? Ore, alchemy ingredients, clothing flowers all disappearing? Lets not forget the dupe bugs, or the speed hacks.... UDP works both ways. In the same way the mega server transmits info, so does every player's client. Botters would just say 'I'm here, picking this flower' Megaserver says 'okay!' and a split second later 'I'm here! halfway across the map, picking this flower' and once more Megaserver says 'okay!'. There is no system in place that would check to ensure that sort of behavior is legally allowed within the game's mechanics. We were all on the honor system.

So to solve the botting problem, ZOS basically installed the TSA Every bit of information passed between your client and the server now has to go through a checkpoint. Once okayed it's allowed to go through. So.... Now you're playing Marco Polo with the mall, in a vehicle that can go 100mph, you've left your house at 2:45 and you reach the mall's entrance, early. Except now, there's a security guard there who needs you to remove your shoes, belt, and your dignity all while checking your ID before you can proceed. By the time the ordeal is done it's now 3:01.

So yes, when a group of people are spamming abilities, throwing calthrops, purging, etc. They're causing lag. Spreading players out merely addresses the symptoms of a problem. The issue isn't lag, latency, or bottlenecking. The problem is that ZOS never implemented a means in which information could be transported between client and server in a secure fashion. It's too easy to inject false information which now needs to be checked, server side, prior to acceptance. If you're looking for someone to blame, point your anger at the cheaters and botters. Though, it is poor engineering to design a system that could so easily be exploited. As for the solution... Blizzard had the right of it. I think ZOS needs to start thinking along those lines.

wraith808

JTorus wrote: »

<snip AWESOME reply />

That's the best and most understandable explanation of the problem that I've seen. I've been trying to explain it forever, but have never come close to being able to communicate these concepts as well as you've just done.

Kudos.

But to one of your points- I thought they'd already done something similar to Warden?

JTorus

wraith808 wrote: »

JTorus wrote: »

<snip AWESOME reply />

That's the best and most understandable explanation of the problem that I've seen. I've been trying to explain it forever, but have never come close to being able to communicate these concepts as well as you've just done.

Kudos.

But to one of your points- I thought they'd already done something similar to Warden?

Thank you,

I've been playing with the
SET MaxNetworkWriteBuffer & SET MaxNetworkReadBuffer lines in my settings, along with some QoS on my router to varying results. It's a hard thing to test because I need to change a setting and wait for a latency spike. At the end of the day though, the issue is definitely on ZOS's end.

As for the Warden thing, I don't believe ZOS has implemented anything like it. I would think that it'd be a requirement to run ESO in administrator mode in order to read what services and processes are running, and I figure programs and hooks like ESO Mod or ESO Launcher would be sending out red flags to ZOS. Not to mention all the people running Sweet FX, at the very least we would have seen some false positives by now. I also have some ethical issues with programs like Warden, it's like subscribing to HBO, and because I'm watching Game of Thrones Richard Plepler can now open my mail.

Of course, none of this applies to consoles. Which, if I recall correctly, will be located on independent server clusters, so we can be sure that they won't be subject to this anti-cheat system. (if that is the case)

Sylvyr

Is it fixed yet?

Bouvin

Dan_Miathail wrote: »

P.S:

Please try and remember that large scale, high population battles where an advertised thing for this game.

https://youtu.be/j9O4HqEp0cI

1. Where is the new content every 6-8 weeks that was promised? Seems like they fell through on their two biggest promises for this game.

2. PvP in that video is nothing like what's in game.. even disregarding lag. It's actually kind of hilarious... Had they showed what actual PvP is like.. giant zerg blobs rolling around spamming AoEs I'm wondering if people would have gotten as excited about the PvP haha..

Someone needs to do an "Honest Trailer" for the game.. like the honest trailers for movies on YouTube.

Nivzruo_ESO

Instanced pvp would help reduce cryodiil lag.

Bouvin wrote: »

Dan_Miathail wrote: »

P.S:

Please try and remember that large scale, high population battles where an advertised thing for this game.

https://youtu.be/j9O4HqEp0cI

1. Where is the new content every 6-8 weeks that was promised? Seems like they fell through on their two biggest promises for this game.

2. PvP in that video is nothing like what's in game.. even disregarding lag. It's actually kind of hilarious... Had they showed what actual PvP is like.. giant zerg blobs rolling around spamming AoEs I'm wondering if people would have gotten as excited about the PvP haha..

Someone needs to do an "Honest Trailer" for the game.. like the honest trailers for movies on YouTube.

Yea, looking at that vid again after playing ESO since beta... pretty funny.