Guild History Changes in U23

sylviermoone · July 2019

Is there more information forthcoming about how this will work? Currently, addon access to guild history is throttled in that we can query for old records every 30 seconds per 100 records, while new information can only be queried once every 30 MINUTES per 100 records. Will this new system be limited to pushing out data in batches of 100 records? What is the distinction between old and new data, and how long will it take for the server to push out "old" data to the client?

Thanks to some hard work and lobbying by the trade community, sales records have a unique ID in the system, though sometimes are missed through the process of querying for records. How will we be able to ensure no records are missed with the new system? Are bank withdraws and deposits subject to this change, or is it purely sales history? What about roster changes?

This is again a change to a crucial system that is IMPOSSIBLE to fully test on PTS. There is basically ZERO trade volume on PTS, which makes it incredibly difficult to test how long it takes for the new changes to scan back and load the sales data. Are there changes to the API calls? Are there changes to what is returned in the current API calls? Are those API calls going to be published to the API patch notes? The current PTS is 22 days old; without access to the current volume of records found on the live client, there are not significant records to fully vet this change.

The changes to how add-ons that access guild history function are already confusing to the player base at large. People don't understand why it takes 10 times as long for their addons to populate data, and are at times downright hostile in addressing developers with these concerns, though we can't do anything about the speed at which ZOS allows us to query the data. It is our concern that this will create even longer data population times, leading to more confusion and hostility from the player base towards people that are just trying to make a usable interface through which to view the sales history already in the game. If the answer to maintaining a working interface through which to view this data means a significant rework on our end, it would be nice to know that before this change is fully live and we're inundated with hate mail because "your addon doesn't work".

@ZOS_GinaBruno @ZOS_JessicaFolsom @ZOS_RichLambert @ZOS_PhilipDraven @ZOS_ChipHilseberg

silvereyes · July 2019

I’m in two large trade guilds, and it takes nearly two hours without relogging to populate the data on live. I usually play only an hour or so at a time, switching between characters, so my data is pretty stale.

If the new push api doesn’t resolve the situation, there’s a good chance that I’ll just give up on trading, which is a real shame. Trading has been one of the most fun aspects of the game to me, but it’s turned into a chore in Elsweyr.

reoskit · July 2019

I second this request for more information. As noted here by Sylvie and in my PTS feedback, it's impossible to tell what these changes will look like when processing huge amounts of data.

Access to the guild history/sales data has changed several times in the past couple months, leading to lots of confusion - even for those of us who are paying close attention to official communications.

With the upcoming changes, it would be nice to be able to explain the situation fully and accurately to our guildies, especially if there is going to be impact to the performance of MM/ATT/etc.

silvereyes · July 2019

From the esoui gitter channel:

ZOS_ChipHilseberg wrote:

basically RequestGuildHistoryCategoryNewest is gone and RequestGuildHistoryCategoryOlder is now RequestMoreGuildHistoryCategoryEvents
and new events are sent to you on a timer

--- D:\Downloads\ESOUIDocumentationP23.txt	2019-07-28 14:13:17.000000000 -0500
+++ D:\Downloads\ESOUIDocumentationP23-2.txt	2019-07-28 13:57:13.000000000 -0500
@@ -8327,16 +8338,19 @@
-* RequestGuildHistoryCategoryNewest(*integer* _guildId_, *[GuildHistoryCategory|#GuildHistoryCategory]* _category_)
-** _Returns:_ *bool* _requested_
+* HasGuildHistoryCategoryEverBeenRequested(*integer* _guildId_, *[GuildHistoryCategory|#GuildHistoryCategory]* _category_)
+** _Returns:_ *bool* _hasEverBeenRequested_
 
+* HasOutstandingGuildHistoryRequest()
+** _Returns:_ *bool* _hasOutstandingGuildHistoryRequest_
-* RequestGuildHistoryCategoryOlder(*integer* _guildId_, *[GuildHistoryCategory|#GuildHistoryCategory]* _category_)
+* RequestMoreGuildHistoryCategoryEvents(*integer* _guildId_, *[GuildHistoryCategory|#GuildHistoryCategory]* _category_)
 ** _Returns:_ *bool* _requested_

No insight here on what the timers are, but I'm sure any addon dev that is in a big enough guild with enough listings still in the store from the character copy would be able to create some history. With gold from a template char, it's easy to generate a ton of sales.

silvereyes · July 2019

And here is the actual file: https://dropbox.com/s/7bnsogs80p3cczu/ESOUIDocumentationP23-2.txt?dl=0

sylviermoone · July 2019

silvereyes wrote: »

From the esoui gitter channel:

ZOS_ChipHilseberg wrote:

basically RequestGuildHistoryCategoryNewest is gone and RequestGuildHistoryCategoryOlder is now RequestMoreGuildHistoryCategoryEvents
and new events are sent to you on a timer

Baertram wrote:

Diff from esoui p23 and p23_2:

--Added
* SCT_EVENT_TYPE_HEAL_ABSORBED
** ITEM_COMBINATION_RESULT_IS_IN_HIDEY_HOLE
* CRAFTING_RESULT_NO_RECIPE
* NUM_ALCHEMY_TRAITS_PER_REAGENT
h5. StoreDefaultSortField
* STORE_DEFAULT_SORT_FIELD_NAME
* STORE_DEFAULT_SORT_FIELD_VALUE
5. Globals
* MAX_STORE_WINDOW_STACK_QUANTITY
* HasOutstandingGuildHistoryRequest()
** _Returns:_ *bool* _hasOutstandingGuildHistoryRequest_
* HasAllianceLockPendingNotification()
** _Returns:_ *bool* _hasNotification_

* GetAllianceLockPendingNotificationInfo()
** _Returns:_ *integer:nilable* _campaignId_, *[Alliance|#Alliance]:nilable* _alliance_, *integer:nilable* _timeLeftS_

* MarkAllianceLockPendingNotificationSeen()

* IsAlchemyItemTraitKnown(*[Bag|#Bag]* _reagentBagId_, *integer* _reagentSlotIndex_, *luaindex* _traitIndex_)
** _Returns:_ *bool* _isKnown_

* IsSmithingTraitKnownForPattern(*luaindex* _patternIndex_, *[ItemTraitType|#ItemTraitType]* _traitType_)
** _Returns:_ *bool* _known_
* CanCombinationFragmentBeUnlocked(*integer* _collectibleId_)
** _Returns:_ *bool* _canBeUnlocked_

* EVENT_CAMPAIGN_ALLIANCE_LOCK_PENDING (*integer* _campaignId_, *[Alliance|#Alliance]* _lockedToAlliance_, *integer* _timeLeftS_)
* EVENT_CAMPAIGN_QUEUE_JOINED (*integer* _campaignId_, *bool* _isGroupMember_, *[Alliance|#Alliance]* _willLockToAlliance_)



--Removed
* HasGuildHistoryCategoryEverBeenRequested(*integer* _guildId_, *[GuildHistoryCategory|#GuildHistoryCategory]* _category_)
** _Returns:_ *bool* _hasEverBeenRequested_
* RequestMoreGuildHistoryCategoryEvents(*integer* _guildId_, *[GuildHistoryCategory|#GuildHistoryCategory]* _category_)
** _Returns:_ *bool* _requested_

* EVENT_CAMPAIGN_ALLIANCE_LOCK_ACTIVATED (*integer* _campaignId_, *[Alliance|#Alliance]* _wasLockedToAlliance_)

No insight here on what the timers are, but I'm sure any addon dev that is in a big enough guild with enough listings still in the store from the character copy would be able to create some history. With gold from a template char, it's easy to generate a ton of sales.

I appreciate you posting this here. While I understand that the gold from a template character is enough to generate some sales, I'm not convinced that it's sufficient to fully mimic the volume of live.

Right now, we have no idea how the devs envision this will function. Because we don't know how this is intended to function, we can not test that it is functioning as intended. It would still be nice to have clarification from the developers on this so that we can begin to test whether or not the system is actually working.

If I recall correctly, EU copy should be on PTS starting tomorrow, which will hopefully mean a newer volume of records. If there are any high volume trading guilds out there that want to shoot PTS invites to @sylviermoone and @Philgo68, it would help us greatly in testing this new feature.

*edits for clarification and requesting EU PTS guild invites.

silvereyes · July 2019

sylviermoone wrote: »

While I understand that the gold from a template character is enough to generate some sales, I'm not convinced that it's sufficient to fully mimic the volume of live.

It would still be nice to have clarification from the developers on this.

Oh, for sure. I’m not saying this would stress the system at all, but it could answer the questions about batch size and timing.

I think it’s safe to say that if batch size doesn’t increase, or frequency increase, MM will be in no better situation after the patch.

sylviermoone · July 2019

silvereyes wrote: »

I think it’s safe to say that if batch size doesn’t increase, or frequency increase, MM will be in no better situation after the patch.

According to this patch note, the frequency is set to actually DECREASE. Without a change to the batch size, not only will MM (and other addons that scan the guild sales history) be in no better situation, they will be in a far worse one.

silvereyes · July 2019

I did some really quick testing with the following spiked addon, and polling once a second seems to load in batches of slightly less than 100 sales each.

https://www.esoui.com/downloads/info2432-GuildHistoryTest.html

I fail to see how this is better for server load, since I am basically just polling the server still. (Edit: technically, I'm polling the game client, not the server. The game client used to send history requests directly on to the server, but I'm not actually sure that's the case any longer.)

One thing I did notice, though, is that sales history appears to be cached to the local client for the duration of the session, even between character changes. It doesn't clear until after a full quit and relog.

silvereyes · July 2019

One thing that I don’t understand is why the data needs to be loaded so piecemeal. The historical data isn’t changing, so why not cache it to a static file every so often on the server side? The clients could download tons of data that way in large batches without straining the server in the least. Web servers do it all the time, and it’s highly scalable.

The small batches don’t even seem to help with the problems that the in-game UI has with displaying thousands of historical guild event rows at once. The small batch size would make more sense if those lists were paginated like guild store listings, but they aren’t.

Edit: from PTS 5.1.3 patch notes:

ZOS_GinaBruno wrote: »

Improved the performance of the guild history window when looking at a large number of events.

That was fast.

Thanks!

silvereyes · July 2019

After adjusting the polling interval to 1/10th of a second, just to see the fastest that the PTS server would load data for me, I was able to achieve a throughput of about 170 sales downloaded per second.

For my two large trade guilds, I can easily see upwards of 80-100k transactions per two-week period (the longest I keep data for). At 170 sales per second, that would take 7-10 minutes to download. It's not going to break any speed records (the MM files for all that data are <5MB, zipped, so you can imagine how much faster it would be to download the same data if it were transferred as static, zipped files), but it is totally acceptable for my purposes.

Take all this with a very large grain of salt, though. PTS has practically no load, and I have no clue if the same throttles are applied to it as the ones that will be used for production.

reoskit · July 2019

silvereyes wrote: »

Take all this with a very large grain of salt, though. PTS has practically no load, and I have no clue if the same throttles are applied to it as the ones that will be used for production.

reoskit wrote: »

there isn't much data on the PTS with which to test, so I cannot speak to load.

<snip>
I would *love* to see the PTS copied from live with a real week's worth of sales data.

Some lingering questions/concerns:
Is there impact to the client side, especially for users who don't care? Think back to the issue where the updating status of each guild member brought lag crashing down on everyone and we had to start filtering guild lists to only show ourselves. Same thing here - is there behind-the-scenes rendering of this data client-side which may introduce lag to everyone? Hard to tell with limited data.

Stress test - Is there any impact on the server having to push large amounts of data to large numbers of people? Again, with limited data and very few people on PTS, it's hard to tell. Is there extra load when people log in and need to get the most recent push? Is there extra load during primetime when LOTS of sales are happening every minute, etc... There are lots of different parameters that ought to be tested. Is all this data being stored in a central repository and then it needs to sort out which data to push to which people? I've no idea what the architecture of this system looks like, so... *shrug* Hard to tell what to ask.

<snip> we need to be able to test load. We can't do that without the full glut of data we usually work with and without lots of live sales occurring.

Amen, @silvereyes. The lack of testability gives me pause.

StabbityDoom · July 2019

silvereyes wrote: »

One thing that I don’t understand is why the data needs to be loaded so piecemeal. The historical data isn’t changing, so why not cache it to a static file every so often on the server side? The clients could download tons of data that way in large batches without straining the server in the least. Web servers do it all the time, and it’s highly scalable.

The small batches don’t even seem to help with the problems that the in-game UI has with displaying thousands of historical guild event rows at once. The small batch size would make more sense if those lists were paginated like guild store listings, but they aren’t.

I was wondering that myself. Why limit it so much? The slowdown may not affect us as badly if we get the info in large batches but less often. @ZOS_GinaBruno could you perhaps direct a dev to this thread? It's super important.

silvereyes · July 2019

Given that the patch notes say new data will be pushed once a minute, I wonder if the data actually is cached server side and sent out in larger batches to the clients and stored in some local data store. In that case, all I'm doing by quickly polling is loading data from the local data store into the UI / addon layer, not from the server directly.

That would make a lot more sense as to why it allows me to get batches every 500-600 ms or so. It would also explain why the batch size is smaller, since they need to be small enough to not block the UI for more than a tiny fraction of a second. It would also make more sense as to why they think it would fix performance problems.

This is all still just speculation, though. I'd love to get an official explanation from the devs.

reoskit · July 2019

Speculation, since I don't understand this system:

Unless client-side is actually indicating "yes, please, I'd like the data" you're potentially going to have lots of data pushing to people who don't necessarily want/need it, or at a time when it will impact their personal performance experience. The last thing we need is PvP/PvE folks pointing fingers at trade.

While server-side performance may be enhanced, because there isn't that ad hoc request from x # of clients requesting different snippets of data, what does this do for/to client-side performance?

silvereyes · July 2019

So, I just tried running Guild History Test on PTS today with a fresh history from my biggest trade guilds, and got kicked to login after about 1000 records. I'll continue testing to see if I can find a sweet spot for throughput without getting kicked.

chess1ukb16_ESO · July 2019

Hi @sylviermoone and @Philgo68

Happy to help with Guild invites to my busiest trade guilds once the data flips to EU.

silvereyes · July 2019

It looks like polling any faster than once per second (1000ms) ends up kicking me to the login screen. Still, it's not terrible:

Number of sales: 21583
Throughput: 79.06 sales/sec
Total time: 273 sec

Number of sales: 23105
Throughput: 92.42 sales/sec
Total time: 250 sec

Around 8-9 minutes to load the last 10 days of sales for my two largest trade guilds. YMMV depending on number of sales, server load, and how much work your addon does to process each batch.

Edit: I've updated https://www.esoui.com/downloads/info2432-GuildHistoryTest.html with the same code I used to see the above metrics, so others can test with their guilds as well.

Philgo68 · July 2019

Looks very promising!!

Is this right after login, or had you been logged in for some amount of time?

silvereyes · July 2019

Philgo68 wrote: »

Looks very promising!!

Is this right after login, or had you been logged in for some amount of time?

The first results were for a test performed right after login. The second test was performed after the first ran.

silvereyes · August 2019

So, bad news. I asked @ZOS_ChipHilseberg the following question on the addon developers chat:

silvereyes wrote:

I've been testing the guild history changes on PTS using https://www.esoui.com/downloads/info2432-GuildHistoryTest.html, and I'm seeing batches of slightly less than 100 records come in every 500-600ms or so.
Is that expected? The patch notes indicated that records would be pushed once per minute..

... and he replied today:

ZOS-ChipHilseberg wrote:

The behavior of the request for older info should be unchanged. PTS probably has 0s for the query cooldown.

So it sounds like the only changes to guild history this patch are for updating the list with new sales that come in after you log in. If no changes have been made to queries for historical data, then that means we are still stuck with the abysmal 100 records every 30 seconds.

gg. I'm out.

P.S. No, you cannot have my stuff.

reoskit · August 2019

-.-