Update on Guild History & Related Performance Issue

Daviiid_ESO

So this how they will say they fixed all the performance issues, while ignoring PC EU and Cyrodiil that will continue to be a lagfest?

ArenGesus

Daviiid_ESO wrote: »

So this how they will say they fixed all the performance issues, while ignoring PC EU and Cyrodiil that will continue to be a lagfest?

Yes, exactly - this is why I called it a distraction earlier. Now everybody can go off and debate with each other endlessly about the guild finder instead of creating complaint threads. And it looks like they've successfully drawn most people off. U23 will release with barely a tweak and all of the other problems are just left totally unaddressed. Last night the forums were disaster - this one "update" from Rich appears to have calmed everybody down to the degree that they can resume ignoring all of the problems, new ones included.

JHartEllis

I imagine one thing that would help with performance would be to reduce the sheer number of transactions that are processed through the guild traders (MM scan time is directly proportional to the number of sales records). I've gone through my guild trader history to see where the biggest impacts might be.

One thing that stands out is stack sizes. Generally, larger stack sizes would reduce the number of transactions. Materials make up ~14% of guild trader volume, and the vast majority of those are at the 200 item cap.

Other items that regularly sell at max stack sizes are foods, drinks, and AVA repair kits (these all weirdly stack to 100 instead of 200 like most everything else anyway). Potions and key fragments also regularly sell in volume at max stack size.

Intricate-traited gear makes up ~9% of guild trader volume. A way to reduce this would be to lower the drops rates while simultaneously making them more impactful (e.g. reduce the chance of getting intricates by 2/3 while also tripling the inspiration gained).

Glyphs make up ~8% of guild trader volume. This would be dramatically reduced if they were at all stackable.

Other ways to reduce volume would be to combine various collectibles. Lowering the drop rates of all motif pages while significantly increasing the drop rates of complete crafting style books would reduce the number of transactions. Cheap individual recipes could be largely replaced with combined recipe books. This could be done for various furnishing plans as well. Similarly, the collectible system of consuming 10 of the same item (e.g. Mummified Alfiq) helps a lot more in this regard than having 7 separate fragment items.

Similarly, lowering the drop rates for small-reward sealed writs and shifting to fewer, better sealed writs would reduce the number of sales without much impacting the writ voucher market overall.

Adding vendor values for Grease, Natural Water, provisioning ingredients, and trait items would keep some of these off guild traders. Generally, increasing vendor values for other items would help in the same way.

Some trader volume could be shifted to the mail system by lowering the C.O.D. mail fee.

There are also systemic ways to discourage listing low-value items on the guild traders. Setting a minimum listing price (e.g. 100g) or adding a small set transaction fee to every listing (e.g. 50g upfront fee on top of other fees), would discourage a large number of cheap listings. Alternatively, rewarding high-value or high-stack-size sales through lower fees would be more of a "carrot" solution.

Taken all together, these suggestions would reduce guild trader volume by approximately 30%, and many of these would also help to alleviate inventory management pains. These are mostly the low-hanging fruits--a lot of other changes could be done with things like gear drops, but that gets more complicated.

RMerlin

Cyrodiil was intended for massive, large-scale PvP, and it fails to work under the current load.

LFG group finder fails to work under the current load.

Guild logs fails to work under the current load.

Do I sense a trend there? Are ZOS's system engineers planning their performance projections based on a single-player game, or on a MMO?

Performance has been an issue for many years. I don't buy the excuse that it was caused by "a recent influx of new and returning players". We were told the exact same excuse a few months ago for another performance-related issue. I highly doubt that the player base has been exponentially increasing for the past 1-2 years. I certainly don't see more players in game than I did before.

Minno

Zathras wrote: »

ZOS_RichLambert wrote: »

Thanks for being patient with us while we work through this one.

-rich

TBH, this isn't the only serious performance issue. As you can read on the forums, your customer's patience is wearing thin.

agreed. its definitely not addons since in PVP I can turn off all addons and still receive crippling lag at 1am eastern time (which previously during summerset was mostly stable).

Its the animations and a host of new server calcs (new changes to vulnerabilties subtract before multiplying against dmg, block mitigation being additive, shields requring dmg to be mitigated before subtracting dmg, light from proc sets/skills bouncing around objects, etc.)

They need a performance audit and FAST.

Minno

LiraTaurwen wrote: »

Lol everyone and their theories let me tell you something, the "bug" that happened after that fatal maintenance has happened a few months ago on EU server (february I believe) and was solved without any of this mess.
Also how come it was working fine before the maintenace? Did alot of new players joined right after and broke the system?
They are just telling lies, wake up!

they let it slip in the reddit AMA that the animations in general result in more lag. It's my assumption the game engine can no longer support beautiful intensive animations (and why the game was boring at first)

gp1680

This dev “sticky” is just another in a long line of deflections by a dev team that seems to be caught off guard by their own product repeatedly. As another poster said earlier, this crap only pacifies those fanboys who will buy anything ZOS says. The history of the performance of this game says otherwise. It’s getting worse every time they release new content.

RMerlin

Minno wrote: »

LiraTaurwen wrote: »

Lol everyone and their theories let me tell you something, the "bug" that happened after that fatal maintenance has happened a few months ago on EU server (february I believe) and was solved without any of this mess.
Also how come it was working fine before the maintenace? Did alot of new players joined right after and broke the system?
They are just telling lies, wake up!

they let it slip in the reddit AMA that the animations in general result in more lag. It's my assumption the game engine can no longer support beautiful intensive animations (and why the game was boring at first)

And there are some serious bugs in the graphic engines that have been there for years (some ground textures randomly disappearing in Cyrodiil mainly, tho it`s less frequent lately). Flickering trees in Cyrodiil that started with Summerset. And the insane graphic flickering occurring mostly with keeps in Cyrodiil, which started with Elsweyr, and is still unresolved.

Despite looking gorgeous (thanks in large part to awesome work from the artists working on ESO), the mechanical portions of the graphics engine are also suffering from various technical issues, and things have been degrading these past few years.

karthrag_inak

What i can't understand is why does this mechanism seemingly use such an archaic client-server model? Why are these processes not event/call-back driven from the server, instead of allowing client queries to run rampant?

Why not allow for very rare db queries (one time per account per login?) coupled with event driven notification broadcasts from the servers to update client state? Especially with respect to relevant guild information, the events are going to happen substantially less frequently than the potential queries about them are (even guild-mate login/logoff and zone movement), so having the server manage a dispatcher driven by a future event loop shouldn't ever be a problem.

Client programs should never be able to actively impact server performance - total trollbait. All potential sources of client disruption should be throttled, especially from addons.

Elsonso

RMerlin wrote: »

Maybe consider exporting some of that data to a separate server, and provide a web-based RESTful API for people to query it. I've seen it done by another major MMO (I was one of the third party developers actually using their API in a web application that I had developed), it would prevent unnecessary load being put on the live servers, and moving those queries to an outside server. What they did was to proxy query results, so subsequent queries were answered by a caching server instead of being sent to the live servers.

According to Rich, above, they are not going to do that.

Instead, they look like they are going to come up with a system where the server pushes the guild information to the client, and the addons will get it from the client, rather than query the server directly. This has the benefit of them being able to control how often the data is pulled from the server, and prevents an addon from hammering the server.

Dont_do_drugs

There is a little problem. The much lower frequency and higher amount of data will most likely cause fps drops each time its processed by those addons. I guess thats the main reason why those addon had that high frequency. The game was smooth then, while the longer the interval between two requests and the higher the amount of new data, the bigger and longer those fps drops.

Minno

lordrichter wrote: »

RMerlin wrote: »

Maybe consider exporting some of that data to a separate server, and provide a web-based RESTful API for people to query it. I've seen it done by another major MMO (I was one of the third party developers actually using their API in a web application that I had developed), it would prevent unnecessary load being put on the live servers, and moving those queries to an outside server. What they did was to proxy query results, so subsequent queries were answered by a caching server instead of being sent to the live servers.

According to Rich, above, they are not going to do that.

Instead, they look like they are going to come up with a system where the server pushes the guild information to the client, and the addons will get it from the client, rather than query the server directly. This has the benefit of them being able to control how often the data is pulled from the server, and prevents an addon from hammering the server.

they honestly should lock some of these functions entirely so the game stops pulling data across a wide spectrum of performance/lag sucking commands.

how much info is there that are always present that addons only touch upon? think of miat's addon where it was pulling info on heavy attack cast times and players in stealth, that stuff is always there clugging up the server lol.

Zathras

silvereyes wrote: »

ZOS_RichLambert wrote: »

With significantly larger guilds, and more people using these add-ons, we’ve seen a huge amount of load on the servers that can’t be solved by throwing more hardware at the problem.

Hi Rich,

Thanks for the update. Could you perhaps explain why performance is really bad on console too, even though add-ons aren't used there? I think many of us are confused, since both the short-term and long-term plans don't seem like they would affect console at all.

Thanks!

It's the blame game. Us?? No, it's not us. It's them.

When, if you look around at the game as a whole, you can see that there are some fundamentally serious issues in many of the major systems.

Own it. Stop making PR promises. Fix it. Because people are just tired of it at this point.

Keldor

Another question: Have the merchants been completely deactivated?
I don't have an option in the bank to open the auction house with the banker. Even if I go to the merchants in front of the city, I can only choose Bye. No option to open the auction house.

Tested with serveral chars.

RMerlin

lordrichter wrote: »

RMerlin wrote: »

Maybe consider exporting some of that data to a separate server, and provide a web-based RESTful API for people to query it. I've seen it done by another major MMO (I was one of the third party developers actually using their API in a web application that I had developed), it would prevent unnecessary load being put on the live servers, and moving those queries to an outside server. What they did was to proxy query results, so subsequent queries were answered by a caching server instead of being sent to the live servers.

According to Rich, above, they are not going to do that.

Instead, they look like they are going to come up with a system where the server pushes the guild information to the client, and the addons will get it from the client, rather than query the server directly. This has the benefit of them being able to control how often the data is pulled from the server, and prevents an addon from hammering the server.

Remember a few months ago when people tracked down performance issues caused by the server pushing guild roster info to clients? I fear this would create just the very same issue. When you are in two busy trading guilds plus three other average guilds, you don't want the contant event info getting pushed to you while you are in the middle of a trial or PvP.

Dont_do_drugs

Keldor wrote: »

Another question: Have the merchants been completely deactivated?
I don't have an option in the bank to open the auction house with the banker. Even if I go to the merchants in front of the city, I can only choose Bye. No option to open the auction house.

Tested with serveral chars.

Did you try to chat with people in your guilds, Zone, private and so on? What you describe is part of a social ban usually. Something you were testing with addons or so might have triggered it. You should contact support for that.

RMerlin wrote: »

lordrichter wrote: »

RMerlin wrote: »

Maybe consider exporting some of that data to a separate server, and provide a web-based RESTful API for people to query it. I've seen it done by another major MMO (I was one of the third party developers actually using their API in a web application that I had developed), it would prevent unnecessary load being put on the live servers, and moving those queries to an outside server. What they did was to proxy query results, so subsequent queries were answered by a caching server instead of being sent to the live servers.

According to Rich, above, they are not going to do that.

Instead, they look like they are going to come up with a system where the server pushes the guild information to the client, and the addons will get it from the client, rather than query the server directly. This has the benefit of them being able to control how often the data is pulled from the server, and prevents an addon from hammering the server.

Remember a few months ago when people tracked down performance issues caused by the server pushing guild roster info to clients? I fear this would create just the very same issue. When you are in two busy trading guilds plus three other average guilds, you don't want the contant event info getting pushed to you while you are in the middle of a trial or PvP.

Such visualized infos can be turned off in such addons usually and come from addons, not from ZOS.

Keldor

Dont_do_drugs wrote: »

Keldor wrote: »

Another question: Have the merchants been completely deactivated?
I don't have an option in the bank to open the auction house with the banker. Even if I go to the merchants in front of the city, I can only choose Bye. No option to open the auction house.

Tested with serveral chars.

Did you try to chat with people in your guilds, Zone, private and so on? What you describe is part of a social ban usually. Something you were testing with addons or so might have triggered it. You should contact support for that.

The interaction with other players works very well. In my guild, other players have also reported this problem.

Dont_do_drugs

hmmmmmmmm that sounds weird. you guys should all message support then. i never heard of this outside of social ban and stuff :O

Androconium

ZOS_RichLambert wrote: »

<snip>

ESO has seen remarkable growth in the last few months. We’ve had a huge influx of both new and returning players - that combined with the newly added Guild Finder tool has resulted in a ton more players in guilds.

Really? More players? That isn't my experience in-game. Compared to 8 weeks ago, I very rarely see other players at 810+ out in regular zones any more. I find resources and drops that I have always had difficulty in obtaining; this suggest to me that a large number of 'power sellers' have left - at least the trading side of the game.

I can see new players around and they have no idea about what they are doing. They aren't getting meaningful responses to chat questions. A huge chunk of 'corporate memory' has walked out the door with the exodus of disgruntled players.

Meanwhile all the non-Elsweyr zones now have those existing and returning players that wanted the necro class, running about and lagging the place with even more fps-sapping effects.

I also find it impossible to believe that you introduced the new guild finder tool without assessing the potential impact.
I do not believe that you did not see this coming.

More players in guilds is a great thing… but this has also surfaced a nasty performance problem with guild history, that is exacerbated by user-created add-ons that constantly query it for data. With significantly larger guilds, and more people using these add-ons, we’ve seen a huge amount of load on the servers that can’t be solved by throwing more hardware at the problem.

"User Created". With the database changes released a few months ago now, it has become clear that the interaction between the 'users' creating theses addons and the Zenimax team that approves them is a lot closer that one would think. You do approve them first, yes? This is something that, with experience, should have been predictable.

In the short term, we are throttling how much and how often add-ons can request the guild history. We will be making tweaks to this throttling over the next few days to find an acceptable balance between performance and user experience. <snip>

This explains why turning off MM and AGS no longer stops the stuttering effect.

Merlight

ZOS_RichLambert wrote: »

Longer term, we’re working on changing the guild history to be more effectively queried by data being pushed to the client instead of allowing add-ons to poll the data on a whim.

There is a reason I disable any history polling add-ons outside towns. How do I prevent server pushing data I don't need because Cyrodiil is laggy enough even without guild history shoved in?

Grimm13

@ZOS_RichLambert With the amount of queries being part of the problem. Would it not make sense to add in the choice for guilds to either allow seeing into the guild bank or not to those that do not have access to remove items.
That in itself seems to me to remove a portion of a code created problem. Every portion adds up. I would suggest the default being that it is hidden so that guilds with inactive leadership would contribute more to lower server stress.

I know you don't want a single AH but would this not be the time reexamine your trader system. Perhaps have each zone be a AH with the Trader Locations being the access point for that Zone AH. Still limit the number of guilds that can sell through that ZAH, but triple the number of Guilds allowed.
So if there was 9 Traders before, the ZAH would have 27 winning guild bids. Still would be a blind bid process, so you have your gold sink still. With just a Zone pushing it's data to the addons, you could have them timed out like traffic lights so it does not tax your system. This tweak to the system would still have 32 ZAH's but significantly reduced load from the roughly 180+ Traders. (numbers might be slightly off.)
The ZAH system could also help with the Ghost Guild issue. You still have it so if not all slots could be filled by the bids, then you could purchase an available spot. If you did win and your guild disbands, then that slot is not resold for the remainder of the week. So there is no reason to pay a ghost guild to disband to free a spot since it does not work.

Any changes to the Trader system is not going to have everyone happy. Yet I can see some would be needed to manage the new influx of players and data streams. This hybrid proposal might be one of the better ones possible.

Toyed with the idea of allowing Guilds to place a secondary bid in a different zone for a 10% cost of the bid, non-refundable, still with only one trade zone available. It would just be a backup plan.
So if you win the primary bid you get back your secondary bid minus the 10%.
Lose the primary get 100% of that bid back but possibly still had won in that secondary zone.
Lose both bids, get 100% of the primary bid but only 90% of the secondary.
Timing would be tricky so would need the Primary bids resolved then secondary bids for the unsold slots, then if any open slots those could be bought as first come basis. Even if all slots sold in Primary bids, then any secondaries still have that 10% fee, Guilds gamble on a backup plan at a cost so takes some thought.

Best leave off now, might have stressed too many people already.

Dont_do_drugs

U can already change rights about seeing Bank history, most trade guilds also use that option.

Varana

The systems in place were maybe not really good from the start (as per the article linked by FrancisCrawford).
But they were certainly not designed for the guild trader update, guild finder, many guild and sales management addons, all constantly requesting data, and maybe an increased player base.

So they try to refactor the code base to better deal with the issue, and that takes time.

I fail to see why completely changing the whole system to an auction house or something similar, i.e. re-programming the trading part of the game from the ground up, would be in any way faster than what they do now. "Just change to an auction house" - yeah, "just".

Also, can someone please find a way to get "nerf what killed me" into this topic? Just for completeness' sake. ;D

kitsune_beth

luton0watford4 wrote: »

Perhaps the best solution to to remove the guild finder function. If this is the root of the problem, then why don't you just remove it?

@ZOS_RichLambert

I agree. Why do people even need to look at a guilds history? Surely people can find other ways to find a guild too. There's a guild forum on here for example.

If it's causing so much of a problem, simply remove it.

Dont_do_drugs

kitsune_beth wrote: »

luton0watford4 wrote: »

Perhaps the best solution to to remove the guild finder function. If this is the root of the problem, then why don't you just remove it?

@ZOS_RichLambert

I agree. Why do people even need to look at a guilds history? Surely people can find other ways to find a guild too. There's a guild forum on here for example.

If it's causing so much of a problem, simply remove it.

Do u even know what the guild history is and which data it contains?

kitsune_beth

Dont_do_drugs wrote: »

kitsune_beth wrote: »

luton0watford4 wrote: »

Perhaps the best solution to to remove the guild finder function. If this is the root of the problem, then why don't you just remove it?

@ZOS_RichLambert

I agree. Why do people even need to look at a guilds history? Surely people can find other ways to find a guild too. There's a guild forum on here for example.

If it's causing so much of a problem, simply remove it.

Do u even know what the guild history is and which data it contains?

No, which is why I asked - sort of.

Dont_do_drugs

kitsune_beth wrote: »

Dont_do_drugs wrote: »

kitsune_beth wrote: »

luton0watford4 wrote: »

Perhaps the best solution to to remove the guild finder function. If this is the root of the problem, then why don't you just remove it?

@ZOS_RichLambert

I agree. Why do people even need to look at a guilds history? Surely people can find other ways to find a guild too. There's a guild forum on here for example.

If it's causing so much of a problem, simply remove it.

Do u even know what the guild history is and which data it contains?

No, which is why I asked - sort of.

The guild history is a section of a guild which includes information as follows:
* Sales/purchases
* bank movements, donations, items which get taken out of bank or moved into bank
* bid amount
* ownership of keeps in cyrodiil
* joining, leaving members, rank changes

Sales Infos in that history are necessary fir addons like mm and att so people have a price tracker. Donations, bank movements as well as membership logs are necessary for gm, as well as the bid logs.

Elsonso

kitsune_beth wrote: »

Dont_do_drugs wrote: »

kitsune_beth wrote: »

luton0watford4 wrote: »

Perhaps the best solution to to remove the guild finder function. If this is the root of the problem, then why don't you just remove it?

@ZOS_RichLambert

I agree. Why do people even need to look at a guilds history? Surely people can find other ways to find a guild too. There's a guild forum on here for example.

If it's causing so much of a problem, simply remove it.

Do u even know what the guild history is and which data it contains?

No, which is why I asked - sort of.

It contains a list of recent activity in the guild. Members added and removed, gold deposits, bank activity, trader activity (sales). It is useful for both people and addons to see what has been happening recently in the guild. Without it, guild management becomes harder, and it is not possible to get sales information from the trader. The latter is what I use to figure out what to sell stuff for (vs what people are asking).

Mancombe_Nosehair

lordrichter wrote: »

kitsune_beth wrote: »

Dont_do_drugs wrote: »

kitsune_beth wrote: »

luton0watford4 wrote: »

Perhaps the best solution to to remove the guild finder function. If this is the root of the problem, then why don't you just remove it?

@ZOS_RichLambert

I agree. Why do people even need to look at a guilds history? Surely people can find other ways to find a guild too. There's a guild forum on here for example.

If it's causing so much of a problem, simply remove it.

Do u even know what the guild history is and which data it contains?

No, which is why I asked - sort of.

It contains a list of recent activity in the guild. Members added and removed, gold deposits, bank activity, trader activity (sales). It is useful for both people and addons to see what has been happening recently in the guild. Without it, guild management becomes harder, and it is not possible to get sales information from the trader. The latter is what I use to figure out what to sell stuff for (vs what people are asking).

It's not the guild history I am suggesting being removed. It is the new finder I want gone.

Dont_do_drugs

luton0watford4 wrote: »

lordrichter wrote: »

kitsune_beth wrote: »

Dont_do_drugs wrote: »

kitsune_beth wrote: »

luton0watford4 wrote: »

Perhaps the best solution to to remove the guild finder function. If this is the root of the problem, then why don't you just remove it?

@ZOS_RichLambert

I agree. Why do people even need to look at a guilds history? Surely people can find other ways to find a guild too. There's a guild forum on here for example.

If it's causing so much of a problem, simply remove it.

Do u even know what the guild history is and which data it contains?

No, which is why I asked - sort of.

It contains a list of recent activity in the guild. Members added and removed, gold deposits, bank activity, trader activity (sales). It is useful for both people and addons to see what has been happening recently in the guild. Without it, guild management becomes harder, and it is not possible to get sales information from the trader. The latter is what I use to figure out what to sell stuff for (vs what people are asking).

It's not the guild history I am suggesting being removed. It is the new finder I want gone.

Were not talking and explaining to u but kitsune. 🤔