Update on Guild History & Related Performance Issue

ZOS_RichLambert

Hi All!

Just wanted to give everyone a quick update on the guild history issue, its impact on server performance and what our plan of attack to address the issue is.

ESO has seen remarkable growth in the last few months. We’ve had a huge influx of both new and returning players - that combined with the newly added Guild Finder tool has resulted in a ton more players in guilds. More players in guilds is a great thing… but this has also surfaced a nasty performance problem with guild history, that is exacerbated by user-created add-ons that constantly query it for data. With significantly larger guilds, and more people using these add-ons, we’ve seen a huge amount of load on the servers that can’t be solved by throwing more hardware at the problem.

In the short term, we are throttling how much and how often add-ons can request the guild history. We will be making tweaks to this throttling over the next few days to find an acceptable balance between performance and user experience. Longer term, we’re working on changing the guild history to be more effectively queried by data being pushed to the client instead of allowing add-ons to poll the data on a whim. This work is being done now, and should be ready for testing with U23 PTS. (which is only a few weeks away!)

Thanks for being patient with us while we work through this one. It’s been a particularly difficult issue to nail down.

-rich

Zathras

ZOS_RichLambert wrote: »

Thanks for being patient with us while we work through this one.

-rich

TBH, this isn't the only serious performance issue. As you can read on the forums, your customer's patience is wearing thin.

reoskit

Thank you for the update.

One potential concern - if load is an issue, can this be properly tested in PTS?

Edit: PTS feedback here.

iris56

Thank you for the update. It means a lot to know that issues are being actively worked on. A little communication goes a long way.

MyNameIsElias

Appreciate the communication Rich, i wish the team all the best of luck in working out a solution for this problem

ArenGesus

I don't see how this addresses the intense performance problems on console, where add-ons have been denied to us. Seems like a convenient excuse - don't acknowledge a bug with the api and instead blame performance on the add-ons querying it. Two issues, neatly cast aside because they have caused each other, neither of which could ever be a core issue. If only console was treated equally to pc, it might be believable. Still inexcusable that the infrastructure isn't in place to handle the demand the player-base paid for, but believable.

But the problems can't be waved away when they occur on all servers/platforms if the cause is something that can only happen on one of those platforms. Either way, the answer is as it always is - reduce functionality instead of addressing the problem. And of course, no answer for console whatsoever. What an absolute joke.

thermatico

Why not add the functionality of these add-ons into the guild interface. This would avoid users having to query your database. The information is already there. Parse the data and make it available for players. I just want to be able to see what I sold in a guild store, how much it sold for, a total of my sales over month/week/day, and what is selling at what prices.

Thanks for your consideration.

therift

ArenGesus wrote: »

I don't see how this addresses the intense performance problems on console, where add-ons have been denied to us. Seems like a convenient excuse - don't acknowledge a bug with the api and instead blame performance on the add-ons querying it. Two issues, neatly cast aside because they have caused each other, neither of which could ever be a core issue. If only console was treated equally to pc, it might be believable. Still inexcusable that the infrastructure isn't in place to handle the demand the player-base paid for, but believable.

But the problems can't be waved away when they occur on all servers/platforms if the cause is something that can only happen on one of those platforms. Either way, the answer is as it always is - reduce functionality instead of addressing the problem. And of course, no answer for console whatsoever. What an absolute joke.

Consoles have the new Guild Finder tool, which is what Rich stated is the root of the problem

PC add-ons exacerbate the problem. Perhaps that is why PC has endured much worse issues than console.

I know you want to paint ZoS and Rich as deceptive and incompetent, but you're going to have to improve your reading comprehension. You can't base your complaint on what you think was said... you must first understand what was actually said.

Re-read and try again.

Varana

Also, he never said that this was the only reason for performance issues.

This game has severe problems with the amount of players currently, and this is only one of them. So restricting guild history access is not going to solve everything at once.
Obviously.

saf227_ESO

Hey Rich,
Thanks for updating us on this. The explanation and planned fix (along with a timeline!!) is much appreciated.

chexor

TBH i think this year is by far worst for ESO players. LFG system is almost disabled, God knows how will this impact scaning in large trading guilds, bugs you said were fixed are still present such as "cone bug", necromancer abilitys are still buggy, and it has been almost 1 month since Elsweyr went live.

Mancombe_Nosehair

Perhaps the best solution to to remove the guild finder function. If this is the root of the problem, then why don't you just remove it?

@ZOS_RichLambert

adirondack

Rich -

Acknowledgement and status are critical components of customer complaints. Very glad to see you addressing both while working on the issue. Keep the communication flowing and you'll build that customer loyalty. No one will ever be happy but at least they will know what is or is not being done.

And no one will ever be happy.

Ray

reoskit

adirondack wrote: »

And no one will ever be happy.

Pssh. If I'm provided a way to see my guild's consolidated, sortable, and filterable sales data, I'll be Tom-Cruise-jumping-on-a-couch happy.

Dont_do_drugs

u knew about the "influx" of new players, you knew about the problems with increasing requests on history before elsweyr, you knew it most likely pre guild store interface update. u still had to build in one after another, while not communicating with the addon devs of mm and att mainly about solutions before those problems the recent weeks occured. how are you guys supposed to solve the spaghetti code thingy, when you still dont use, what you should have learned about stacking problems on problems?

StabbityDoom

ZOS_RichLambert wrote: »

Hi All!

Just wanted to give everyone a quick update on the guild history issue, its impact on server performance and what our plan of attack to address the issue is.

ESO has seen remarkable growth in the last few months. We’ve had a huge influx of both new and returning players - that combined with the newly added Guild Finder tool has resulted in a ton more players in guilds. More players in guilds is a great thing… but this has also surfaced a nasty performance problem with guild history, that is exacerbated by user-created add-ons that constantly query it for data. With significantly larger guilds, and more people using these add-ons, we’ve seen a huge amount of load on the servers that can’t be solved by throwing more hardware at the problem.

In the short term, we are throttling how much and how often add-ons can request the guild history. We will be making tweaks to this throttling over the next few days to find an acceptable balance between performance and user experience. Longer term, we’re working on changing the guild history to be more effectively queried by data being pushed to the client instead of allowing add-ons to poll the data on a whim. This work is being done now, and should be ready for testing with U23 PTS. (which is only a few weeks away!)

Thanks for being patient with us while we work through this one. It’s been a particularly difficult issue to nail down.

-rich

Thank you for letting us know. I will stop pestering for now

gepe87

Seems a nice start about performance

Elsonso

Awesome. Thanks for the update, Rich.

ArenGesus

therift wrote: »

ArenGesus wrote: »

I don't see how this addresses the intense performance problems on console, where add-ons have been denied to us. Seems like a convenient excuse - don't acknowledge a bug with the api and instead blame performance on the add-ons querying it. Two issues, neatly cast aside because they have caused each other, neither of which could ever be a core issue. If only console was treated equally to pc, it might be believable. Still inexcusable that the infrastructure isn't in place to handle the demand the player-base paid for, but believable.

But the problems can't be waved away when they occur on all servers/platforms if the cause is something that can only happen on one of those platforms. Either way, the answer is as it always is - reduce functionality instead of addressing the problem. And of course, no answer for console whatsoever. What an absolute joke.

Consoles have the new Guild Finder tool, which is what Rich stated is the root of the problem

PC add-ons exacerbate the problem. Perhaps that is why PC has endured much worse issues than console.

I know you want to paint ZoS and Rich as deceptive and incompetent, but you're going to have to improve your reading comprehension. You can't base your complaint on what you think was said... you must first understand what was actually said.

Re-read and try again.

In the short term, we are throttling how much and how often add-ons can request the guild history.

How about you try instead. This is the fix that was promised. To throttle add-ons. Yeah, maybe there is some extra load due to the game's own querying, but I happened to have off on release day for Elsweyr and loaded up as soon as my console patched. The performance was garbage from that very moment. Nobody will convince me that everybody else who managed to also have off that day immediately joined enough guilds to make this be a day zero problem. It simply isn't feasible.

Maybe the guild finder is the root of the problem, but the blame is being put on add-ons and the only communication and planned fix is to limit add-ons - that is a non-fix and ignores console entirely. I'm not painting anybody - they're painting themselves. You wish to be fooled and so you are - I'm not. This is insulting.

Mancombe_Nosehair

Ok, these are my suggestions:

1 Remove guild finder
2 Remove inactive players from guild's for more than a month or something
3 Shut down any guild which has been completely inactive for at least a month
4 Restore the guild store back to the previous format from the ridiculous lagfest it now is
5 Ask the person who wrote awesome guild store for their coding. Pay them a fee if you like.

This should resolve a lot of the problems the guild activity function is having.

@ZOS_RichLambert

Dusk_Coven

ZOS_RichLambert wrote: »

In the short term, we are throttling how much and how often add-ons can request the guild history. We will be making tweaks to this throttling over the next few days to find an acceptable balance between performance and user experience.

Would having a global auction house -- thereby making these addons obsolete -- be a better solution?
More data to search certainly but MMOs with global auction house simply throttle how often players can repeatedly search.

ArenGesus

saf227_ESO wrote: »

Hey Rich,
Thanks for updating us on this. The explanation and planned fix (along with a timeline!!) is much appreciated.

What? There is no fix - they plan to limit functionality that add-ons provide on their behalf because the don't do it themselves. This isn't a fix. They will attempt to limit the impact to the degree necessary for people to stop complaining on a single platform. You guys will eat anything they're serving.

SpiderKnight

So what you're saying is that it's actually 2 1/2 months until console will see any "possible fix" for this crap. Can you at least suspend some eso+ for like a month so I can take a break and not get pissed off at this shite all the time?

ManwithBeard9

Dusk_Coven wrote: »

ZOS_RichLambert wrote: »

In the short term, we are throttling how much and how often add-ons can request the guild history. We will be making tweaks to this throttling over the next few days to find an acceptable balance between performance and user experience.

Would having a global auction house -- thereby making these addons obsolete -- be a better solution?
More data to search certainly but MMOs with global auction house simply throttle how often players can repeatedly search.

No. An auction house wouldnt solve any issue caused by add ons. Master Merchant would still be extremely useful in tracking the sold price of goods.Tamriel Trade Center could still be used for something similar. And an auction house has nothing to do with the other none selling add ons that help track donations, raffle purchases, and things taken and put into the guild banks.

MyKillv2.0

I Am slightly confused. Does this mean performance issues since Elsweyr has to do with Add Ons? Specifically, ones like Master Merchant?

Not trolling but I seeing if I read that correctly or if I misunderstood what Rich said

Dusk_Coven

ManwithBeard9 wrote: »

Dusk_Coven wrote: »

ZOS_RichLambert wrote: »

In the short term, we are throttling how much and how often add-ons can request the guild history. We will be making tweaks to this throttling over the next few days to find an acceptable balance between performance and user experience.

Would having a global auction house -- thereby making these addons obsolete -- be a better solution?
More data to search certainly but MMOs with global auction house simply throttle how often players can repeatedly search.

No. An auction house wouldnt solve any issue caused by add ons. Master Merchant would still be extremely useful in tracking the sold price of goods.Tamriel Trade Center could still be used for something similar. And an auction house has nothing to do with the other none selling add ons that help track donations, raffle purchases, and things taken and put into the guild banks.

Ah good point. All sorts of Add-Ons poll the data. However, a good chunk of polling would be gone as well as the side effects of the current system, such as raffles and donations to fund securing a trader. I haven't seen any other MMO having actual performance issues related to trading. Their guilds don't run raffles, ask for weekly donations/fees; neither do their games need (or allow) add-ons that eat up so much performance.

As for Master Merchant, FFXIV has a global market that includes an MM type function where sales are presumably captured at point of sale and the most recent sales can be viewed when you are browsing an item. That data need not be guild-specific by any means.

ArenGesus

SpiderKnight wrote: »

So what you're saying is that it's actually 2 1/2 months until console will see any "possible fix" for this crap. Can you at least suspend some eso+ for like a month so I can take a break and not get pissed off at this shite all the time?

You actually believe there is ever a fix coming for this? I won't believe anything I read from them until they follow up on the promises they've been making for years. Here is my prediction - pc add-ons will remain limited to the point of being useless for basically ever. U23 will release with no change and we'll not hear another update on this for several more years - just like lfg finder. Just like stuck in combat. Just like cyrodil performance. Just like, just like, just like.

Performance sucks. It has sucked before this tool was released and it's going to continue to suck forever. I have to try and fail like 15 times to login on xbox/na every single time I want to play - that has been going on for half a year before Elsweyr and this guild tool that is supposedly at the core of all performance issues. Even if they disable add-ons entirely, it's going to be a small drop in a big bucket and is going to have zero impact on the console experience regardless.

You guys believe what you want, but I'm setting my expectations right were zos has said over and over they should be. The only thing that will happen will be exactly what is needed for them to weather this storm and calm the forums/reddit down so they can go back to business as usual.

redspecter23

Removing guild finder is like shooting a soldier after he's lobbed a grenade at you. It doesn't stop the grenade at all. The impact of guild finder is the large amount of players in guilds. Removing guild finder won't help. The players are already set up in those guilds now.

In theory, what would help is reducing the number of guilds that players could be a part of or reducing the capacity of guilds. I'm not saying this would necessarily help, but following the logic in the first post, it makes more sense as a solution than just turning off guild finder. The problem would likely just resurface as more players join the game.

Dusk_Coven

luton0watford4 wrote: »

2 Remove inactive players from guild's for more than a month or something
3 Shut down any guild which has been completely inactive for at least a month

These dead guilds aren't contributing to Add Ons polling data.
Removing dead guilds from the Guild Finder might work to relieve stress specifically on Guild Finder, which I suspect is the least of their worries -- but you're already advocating removing Guild Finder altogether.

Dusk_Coven

redspecter23 wrote: »

Removing guild finder is like shooting a soldier after he's lobbed a grenade at you. It doesn't stop the grenade at all. The impact of guild finder is the large amount of players in guilds. Removing guild finder won't help. The players are already set up in those guilds now.

In theory, what would help is reducing the number of guilds that players could be a part of or reducing the capacity of guilds. I'm not saying this would necessarily help, but following the logic in the first post, it makes more sense as a solution than just turning off guild finder. The problem would likely just resurface as more players join the game.

I think it's a tricky situation that's proposed. Guild Finder is precisely good for the new players that they are seeing to get them settled in and find people to ask the questions that arise from not reading the tutorial or tooltips or ZOS event announcements and such so they don't get ignored when spamming zone chat with them.

code65536

@ZOS_RichLambert @ZOS_Finn

What about the trials leaderboard issues? Are they similar/related? When can we expect a fix for that?