Update on Guild History & Related Performance Issue

ArenGesus · June 2019

code65536 wrote: »

@ZOS_RichLambert @ZOS_Finn

What about the trials leaderboard issues? Are they similar/related? When can we expect a fix for that?

You can never expect a fix. Cyrodil-like performance is now the new normal for all of Tamriel. Learn to love it.

ManwithBeard9 · June 2019

Dusk_Coven wrote: »

ManwithBeard9 wrote: »

Dusk_Coven wrote: »

ZOS_RichLambert wrote: »

In the short term, we are throttling how much and how often add-ons can request the guild history. We will be making tweaks to this throttling over the next few days to find an acceptable balance between performance and user experience.

Would having a global auction house -- thereby making these addons obsolete -- be a better solution?
More data to search certainly but MMOs with global auction house simply throttle how often players can repeatedly search.

No. An auction house wouldnt solve any issue caused by add ons. Master Merchant would still be extremely useful in tracking the sold price of goods.Tamriel Trade Center could still be used for something similar. And an auction house has nothing to do with the other none selling add ons that help track donations, raffle purchases, and things taken and put into the guild banks.

Ah good point. All sorts of Add-Ons poll the data. However, a good chunk of polling would be gone as well as the side effects of the current system, such as raffles and donations to fund securing a trader. I haven't seen any other MMO having actual performance issues related to trading. Their guilds don't run raffles, ask for weekly donations/fees; neither do their games need (or allow) add-ons that eat up so much performance.

As for Master Merchant, FFXIV has a global market that includes an MM type function where sales are presumably captured at point of sale and the most recent sales can be viewed when you are browsing an item. That data need not be guild-specific by any means.

Guilds that aren't trade guilds still do raffles, giveaways, and all that. Even if ZOS moves MM and like add ons in game, they are still going to be retrieving info. Even more so with an auction house. Likely it will make sever stability worse judging from the issues that are arising already.

misfitmanic · June 2019

Providing guild info to a client side log file that can be parsed via ESOLogs would be neat

silvereyes · June 2019

ZOS_RichLambert wrote: »

With significantly larger guilds, and more people using these add-ons, we’ve seen a huge amount of load on the servers that can’t be solved by throwing more hardware at the problem.

Hi Rich,

Thanks for the update. Could you perhaps explain why performance is really bad on console too, even though add-ons aren't used there? I think many of us are confused, since both the short-term and long-term plans don't seem like they would affect console at all.

Thanks!

ArenGesus · June 2019

silvereyes wrote: »

ZOS_RichLambert wrote: »

With significantly larger guilds, and more people using these add-ons, we’ve seen a huge amount of load on the servers that can’t be solved by throwing more hardware at the problem.

Hi Rich,

Thanks for the update. Could you perhaps explain why performance is really bad on console too, even though add-ons aren't used there? I think many of us are confused, since both the short-term and long-term plans don't seem like they would affect console at all.

Thanks!

I'd also like to know why some of us had issues as a result of the guild finder prior to it being released. And then, hours after release, performance plummeted substantially further and has remained consistently at rock bottom. Everybody has been searching guild finder on xbox since the moment Elsweyr released, even during off hours - and haven't stopped yet because they just haven't found the right guild? Seems totally plausible, lol.

This is a distraction, plane and simple. The problems were there before and are just worse now to the degree that more people are complaining than ever before. This little forum revolt going on right now will eventually calm down - and that's the last anybody will ever hear of this "little non game-breaking" issue.

Invest in your hardware zos. More eyes are on ESO right now than ever before, given the current state of wow and the press that recent releases have drawn. I'm for damn sure doing my part to make sure those eyes are seeing the full scope of the game experience. I hope everybody (other than the shills, who are too busy washing your car right now) does the same.

FrancisCrawford · June 2019

The database engineering of ESO has been catastrophic. I get the impression that certain bits are being totally rewritten, and I'd guess strongly that that's the only good alternative. Indeed, I suspect that the rewriting isn't thorough enough.

Technical debt can be hell to pay off.

http://www.dbms2.com/2014/04/16/the-worst-database-developers-in-the-world/

Dusk_Coven · June 2019

ZOS_RichLambert wrote: »

we’ve seen a huge amount of load on the servers that can’t be solved by throwing more hardware at the problem.

@ZOS_RichLambert Is there a connection between this and some of the constant lag and frequent disconnects that some players are experiencing or are the causes of those issues still not definitely identified?

eso_lags · June 2019

ZOS_RichLambert wrote: »

Hi All!

Just wanted to give everyone a quick update on the guild history issue, its impact on server performance and what our plan of attack to address the issue is.

ESO has seen remarkable growth in the last few months. We’ve had a huge influx of both new and returning players - that combined with the newly added Guild Finder tool has resulted in a ton more players in guilds. More players in guilds is a great thing… but this has also surfaced a nasty performance problem with guild history, that is exacerbated by user-created add-ons that constantly query it for data. With significantly larger guilds, and more people using these add-ons, we’ve seen a huge amount of load on the servers that can’t be solved by throwing more hardware at the problem.

In the short term, we are throttling how much and how often add-ons can request the guild history. We will be making tweaks to this throttling over the next few days to find an acceptable balance between performance and user experience. Longer term, we’re working on changing the guild history to be more effectively queried by data being pushed to the client instead of allowing add-ons to poll the data on a whim. This work is being done now, and should be ready for testing with U23 PTS. (which is only a few weeks away!)

Thanks for being patient with us while we work through this one. It’s been a particularly difficult issue to nail down.

-rich

Its not the influx of players causing this. Any zone in your game, where alot of people group up, will have terrible performance. This has happened in the capitol cities at certain times. This has happened in certain zones during events. But this truly shines most in pvp and has for a long time. Sure, now with the influx of people everyone is seeing it regularly in pve as well. Im sure there are many other factors to it but you have to be aware of that fact.

And add ons? I hope you mean something else by add ons, like your own in game add ons on all platforms, because this issue is on console as well. Performance is terrible in pve, and pvp as usual.

Rich, this is unacceptable. Truly. And some of us are done with being patient, we just dont care. Some of us dont think you will ever fix this. Personally, after seeing the decline of pvp into the absolutely UNPLAYABLE state that the main campaign on xbox is today (and has been for the last year), i have given up any hope that you guys are capable or willing to fix this mess. Good luck.

Dont_do_drugs · June 2019

This is a distraction, plane and simple. The problems were there before and are just worse now to the degree that more people are complaining than ever before. This little forum revolt going on right now will eventually calm down - and that's the last anybody will ever hear of this "little non game-breaking" issue.

It's no distraction. There is a problem for sure. The core lies in the database structure which cant be solved that fast. The second problem produced by ZOS was coding more stuff into the game which increases the amount of requests drastically in the last 2 Updates next to that "influx" of new players.

BUT they had a clue how to at least remedy the more and more upcoming problems since ZOS blocked and shortened guild history not the first time. The first time was uncommnuicated in April. From Early April till Elsweyr Release they also had time to contact Phil (MM) and Arka (ATT) via esoui (ZOS has min. one staff member there registered) and leave them a message/mail about the upcoming problem to find a shortterm solution before the server goes rip and the guilds are left with nothing for a while.

It's been also in April when I sensed that the history problem wont be solved that fast, even if ZOS was able to re-enable the 10 days again, thats why Arka and I had been discussing options how to lower requests on server, but we didnt went on discussing this, with agreeing, that we doubt, that MM and ATT are the core problem or that it would have a greater impact on the server situation. I mean, ZOS didnt communicate anything to anyone about requests, about the deeper reason of shortening the history and so on. It's been also in April when I pointed in a PTS-discussion about guild search on the history problem and that the guild search will add new requests and a lot of new communication work for the server in that social sector and will pile up on the already existing problem, that ZOS already tried to "solve" in a short term solution by shortening the history of the guilds. I am pretty sure, the devs of MM and ATT are willing to make changes and communicate options to help and support the situation, but it would also have been nice to communicate the need of this maybe a bit earlier, before setting up one after another dev thread throwing shame on those addons and their authors as if they were the root cause of those server issues.

I dislike how all of this happened, as a GM who just wants to do his work, and did that for years, playing and paying your game and also using my free time to organize things in and around ESO instead of gaming then, because ZOS ran into the server issues and into the need of removing/shortening the history again with open eyes and knowingly.

shaielzafine · June 2019

code65536 wrote: »

@ZOS_RichLambert @ZOS_Finn

What about the trials leaderboard issues? Are they similar/related? When can we expect a fix for that?

It's probably never going to actually be fixed or they'll say it's "fixed" (but not really). Much like Cyrodiil and LFG tool lol

Dusk_Coven · June 2019

ManwithBeard9 wrote: »

Even if ZOS moves MM and like add ons in game, they are still going to be retrieving info. Even more so with an auction house. Likely it will make sever stability worse judging from the issues that are arising already.

"Even more so" -- explain your reasoning please since the core services would be in-house and they could further throttle repeated rapid searches just like every other MMO with a global market system.

Imza · June 2019

I suspect that the database system they have serves all the game types, not just PC, but also the consoles, and that that is why even consoles are experiencing all these problems with lag....

I also suspect that the database system is inundated with requests all the time. Everytime you look at any information - you are querying the database.....

EDIT:

I also suspect that the only true solution is to ban add-ons....

maybe we should be thinking about that.....

Dont_do_drugs · June 2019

Imza wrote: »

I suspect that the database system they have serves all the game types, not just PC, but also the consoles, and that that is why even consoles are experiencing all these problems with lag....

I also suspect that the database system is inundated with requests all the time. Everytime you look at any information - you are querying the database.....

EDIT:

I also suspect that the only true solution is to ban add-ons....

maybe we should be thinking about that.....

Why? Zos supports addons, addons support their staff and make deeper ui changes unnecessary for zos, especially since it would tale much more time to make adjustments on that elements. Addons provide functions to the user, so zos doesn't need to write that qol improvements.

Imza · June 2019

Dont_do_drugs wrote: »

Imza wrote: »

I suspect that the database system they have serves all the game types, not just PC, but also the consoles, and that that is why even consoles are experiencing all these problems with lag....

I also suspect that the database system is inundated with requests all the time. Everytime you look at any information - you are querying the database.....

EDIT:

I also suspect that the only true solution is to ban add-ons....

maybe we should be thinking about that.....

Why? Zos supports addons, addons support their staff and make deeper ui changes unnecessary for zos, especially since it would tale much more time to make adjustments on that elements. Addons provide functions to the user, so zos doesn't need to write that qol improvements.

I think half the problem was that ZoS tried to keep the Elder Scrolls culture of mods and add-ons alive - but this is an MMO not a single player game... and I think that now that it is popular..... the flaws are showing and they truly are just discovering what those flaws may be.....

Dont_do_drugs · June 2019

Imza wrote: »

Dont_do_drugs wrote: »

Imza wrote: »

I suspect that the database system they have serves all the game types, not just PC, but also the consoles, and that that is why even consoles are experiencing all these problems with lag....

I also suspect that the database system is inundated with requests all the time. Everytime you look at any information - you are querying the database.....

EDIT:

I also suspect that the only true solution is to ban add-ons....

maybe we should be thinking about that.....

Why? Zos supports addons, addons support their staff and make deeper ui changes unnecessary for zos, especially since it would tale much more time to make adjustments on that elements. Addons provide functions to the user, so zos doesn't need to write that qol improvements.

I think half the problem was that ZoS tried to keep the Elder Scrolls culture of mods and add-ons alive - but this is an MMO not a single player game... and I think that now that it is popular..... the flaws are showing and they truly are just discovering what those flaws may be.....

It's a while ago and I won't search it now, but I do remember, that zos openly said, that addon creators are able to make ui changes and improvements faster than zos, also are able to fix and change those changes faster, while zos would have to do the loop of idea - approval - coding - approval for update and so on over and over again. The addon authors are able to do that faster and also safe time so devs can focus other topics. This is definitely not only bcs of some gaming culture but also economic for zos.

Turelus · June 2019

Thanks for the update and additional information Rich.

Posts like this, especially when they give more open insight into what, why and how go a long way to appeasing some of the community.

Imza · June 2019

Imza wrote: »

...
I also suspect that the only true solution is to ban add-ons....

maybe we should be thinking about that.....

With this statement I meant to draw everyone into the discussion of is it possible for ZoS to do SOME modifications to the the UI and for us THE CUSTOMER to do without add-ons...

Do you think it is possible?

Dont_do_drugs · June 2019

Imza wrote: »

Imza wrote: »

...
I also suspect that the only true solution is to ban add-ons....

maybe we should be thinking about that.....

With this statement I meant to draw everyone into the discussion of is it possible for ZoS to do SOME modifications to the the UI and for us THE CUSTOMER to do without add-ons...

Do you think it is possible?

It is lol, but they won't.

Zorgon_The_Revenged · June 2019

So no guild and market place companion app for mobile anytime soon? Oh well.

NupidStoob · June 2019

therift wrote: »

ArenGesus wrote: »

I don't see how this addresses the intense performance problems on console, where add-ons have been denied to us. Seems like a convenient excuse - don't acknowledge a bug with the api and instead blame performance on the add-ons querying it. Two issues, neatly cast aside because they have caused each other, neither of which could ever be a core issue. If only console was treated equally to pc, it might be believable. Still inexcusable that the infrastructure isn't in place to handle the demand the player-base paid for, but believable.

But the problems can't be waved away when they occur on all servers/platforms if the cause is something that can only happen on one of those platforms. Either way, the answer is as it always is - reduce functionality instead of addressing the problem. And of course, no answer for console whatsoever. What an absolute joke.

Consoles have the new Guild Finder tool, which is what Rich stated is the root of the problem

PC add-ons exacerbate the problem. Perhaps that is why PC has endured much worse issues than console.

I know you want to paint ZoS and Rich as deceptive and incompetent, but you're going to have to improve your reading comprehension. You can't base your complaint on what you think was said... you must first understand what was actually said.

Re-read and try again.

"Perhaps that is why PC has endured much worse issues than console.*

Thanks for the new meme.

Nobody needs to paint them as incompetent they do that well enough on their own. At this point this game is an embarrassment. All they care about is to keep the game running with absolute minimum investment while milking as much out of the players as possible. The only motivation they have at this time is money. If ZoS ever had principles they already lost them long ago. Just look at the blatant pay to win that came with Cloudrest sets, Necromancer and Sunspire sets.
This game has tons of performance issues and bugs since it's release and there has never been a serious attempt to resolve this. Now that recent patches added even more to the mess they finally want to do something about it and will again only do the bare minimum to appease the anger of the majority.

FierceSam · June 2019

Saddest thing I see is that ZOS clearly made no plans for success..

The whole point of the Season of the Dragon must have been to draw in new players (otherwise why bother). Yet it’s clear that all of the issues that have arisen this year are down to a complete inability to actually accommodate even a relatively low number of additional players.

None of these issues are new and ZOS’s preferred solution appears to be reducing services and restricting access rather than scaling their systems to cope.. nothing that ZOS have said in any of their pinned threads suggests otherwise. Nothing suggests they will be scaling up their infrastructure to deal with this..

It will be interesting to see just how painful the mid year meltdown becomes..

Elsonso · June 2019

FrancisCrawford wrote: »

The database engineering of ESO has been catastrophic. I get the impression that certain bits are being totally rewritten, and I'd guess strongly that that's the only good alternative. Indeed, I suspect that the rewriting isn't thorough enough.

Pretty much the impression I have, as well. In a growing environment, adding hardware processing power only postpones the problem for later. Guess we have arrived at "later", eh?

marius_buys · June 2019

@ZOS_RichLambert

What is the current app listener ping interval?

DragonRacer · June 2019

Dusk_Coven wrote: »

luton0watford4 wrote: »

2 Remove inactive players from guild's for more than a month or something
3 Shut down any guild which has been completely inactive for at least a month

These dead guilds aren't contributing to Add Ons polling data.
Removing dead guilds from the Guild Finder might work to relieve stress specifically on Guild Finder, which I suspect is the least of their worries -- but you're already advocating removing Guild Finder altogether.

Just as a point, dead guilds are most likely not even listed in Guild Finder. You have to have an active GM actually create a listing and checkmark to make the listing public before the guild shows up in Guild Finder. If a guild is dead or has a dead GM, they're not gonna be listed. At least, that has been the console experience. We had to poke an inactive GM to log in and pass Crown for a guild I am in so that the remaining officers still active could create a listing and manage applications... otherwise, no Guild Finder listing.

Thoragaal · June 2019

ZOS_RichLambert wrote: »

Hi All!

Just wanted to give everyone a quick update on the guild history issue, its impact on server performance and what our plan of attack to address the issue is.

ESO has seen remarkable growth in the last few months. We’ve had a huge influx of both new and returning players - that combined with the newly added Guild Finder tool has resulted in a ton more players in guilds. More players in guilds is a great thing… but this has also surfaced a nasty performance problem with guild history, that is exacerbated by user-created add-ons that constantly query it for data. With significantly larger guilds, and more people using these add-ons, we’ve seen a huge amount of load on the servers that can’t be solved by throwing more hardware at the problem.

In the short term, we are throttling how much and how often add-ons can request the guild history. We will be making tweaks to this throttling over the next few days to find an acceptable balance between performance and user experience. Longer term, we’re working on changing the guild history to be more effectively queried by data being pushed to the client instead of allowing add-ons to poll the data on a whim. This work is being done now, and should be ready for testing with U23 PTS. (which is only a few weeks away!)

Thanks for being patient with us while we work through this one. It’s been a particularly difficult issue to nail down.

-rich

It feels like a breath of fresh air when given the recognition of the issue, the explanation and a better insight into why it might take a bit of time to resolve.
I only wish we could get more explanations like this regarding other aspects of the game too. They're few and too far between.
Thank you Rich!

Mix · June 2019

Thank you for the information and update on what is being done!

Reading this thread though it is easy to see why you wouldn't want to post updates like this, so much negativity and references to other issues!

ArenGesus · June 2019

Mix wrote: »

Thank you for the information and update on what is being done!

Reading this thread though it is easy to see why you wouldn't want to post updates like this, so much negativity and references to other issues!

"negativity" is one way to look at it. Another way to look at it is that these posts have happened in the past, but nothing has ever come of any of them. They "fixed" pc-eu performance. They "fixed" lfg. They "fixed" cyrodil lag. This is just another of a series of "calm down it will be fine, even though we're doing nothing to give you the game experience you paid for" posts.

It's embarrassing to be taken advantage of. Some people like to pretend that didn't happen. Others try to prevent it from happening again.

VaranisArano · June 2019

MyKillv2.0 wrote: »

I Am slightly confused. Does this mean performance issues since Elsweyr has to do with Add Ons? Specifically, ones like Master Merchant?

Not trolling but I seeing if I read that correctly or if I misunderstood what Rich said

Not exactly.

The root of the problems is that ESO has a lot more players than ever. Just like having a ton of players on at the same time strained the EU servers and breaks Groupfinder, it places a ton of pressure on the Guild History, which records the activities of all those players. ESO's code and servers simply don't handle large numbers of players at the same time as well as we'd like.

Then ZOS introduces Guild Finder, which added a lot more players to guilds, further straining the Guild History.

On PC, players have a additional pressure on the Guild History, which is addons (including but not limited to Master Merchant) which can apparently query the Guild History for info as often as the player wants. So that's even worse for the already burdened Guild History.

So ZOS current solution is to turn off the API that addons use to query the Guild History, working on a compromise between user experience (on PC) and the server performance. This is not ideal for PC, since we've always had these addons and trading guilds are built accordingly, but its better than losing our Guild History entirely or having Guild bugs.

So are the performance issues related to Master Merchant?

If you were seeing the Guild-related performance issues on PC, yes, Master Merchant is one of the mods that queries Guild History, thus contributing to the increased load on the servers. But its only one factor out of several that contributed to this mess.

RMerlin · June 2019

Maybe consider exporting some of that data to a separate server, and provide a web-based RESTful API for people to query it. I've seen it done by another major MMO (I was one of the third party developers actually using their API in a web application that I had developed), it would prevent unnecessary load being put on the live servers, and moving those queries to an outside server. What they did was to proxy query results, so subsequent queries were answered by a caching server instead of being sent to the live servers.

Application developers had to apply to obtain an API key, so misbehaving applications could be blocked server-side by rejecting their API key.

While not as convenient as having it in-game, it could still greatly help reducing the load on the live servers.

Otherwise, I agree that throttling queries would make sense.

Dont_do_drugs · June 2019

I am pretty sure, the trade guild interface update contributed as well. Each time store gets opened, especially with full text search and additional functions through ags, the server also hast to do more than before. A lot of stuff gets automatically loaded, and it's a fine idea, but not healthy for performance as well.

In guild finder I recognize a lot of more fluctuation in guilds. A lot of people do not read and just spam inv requests, which leads to a much faster rotation of "low rank members". But this is also something in people behavior. Even if we sort out "no message" requests, we still don't know, wether a message is ctrl+c/ctrl+v as example. It's a whole lot of a mess at moment. Also that the finder is pretty unmoderated and leads into a lot of wrong listing and categorization. But that's another topic.

There are a lot of problems, and I wish the affected authors had been given the chance to support and beeing able to discuss what's needed and what zos needs from them.