U41 Guild History problems

Sharlikran · April 29

Sachsen71 wrote: »

Can anyone help me understand what I need to do to fix this? I get this popup every time I log out now. I have tried setting my categories to Automatic and also to Force On but the popup is there either way.

Reading the information in the window how would you summarize that?

I have been using LibHistoire since it was released and I used the developmental version prior to its first release. Therefore, what I feel it says may not be what you see.

What does that notification say to you?

If you open your guild history and you have all green bars then you have a different issue and you should post a screen shot of your LibHistoire UI on the ESOUI forums here.

If you don't know what guild history is and you don't know what green bars means then you are simply ignoring the entire process. Read how to obtain history here.

Sachsen71 · April 30

hiyde wrote: »

Sachsen71 wrote: »

So is there ANY tool, add-on, or setting that works now, that allows us to see when guild members joined our guild? Shissu's used to provide that but now it's gone. I used to be able to see the join date and how many days they had been a member, which was indispensable since we promoted based on time in guild. Now I don't know what to do.

U41 was such a disaster, I'm so incredibly frustrated trying to be a GM with one hand tied behind my back.

ITTsRosterBot still seems to work for me (though not 100% of the time). I do see some members (including brand new ones) where there's no info in the tooltip.

Thank you - can you tell me how to access that screen in your screenshot? I have ITTSRosterBot installed but not sure how to access.

erdbeerheld · April 30

Coming back to the loading screen topic. I did some more tests on two different PCs. All tests were performed without any addons active (the old PC didnt even have addons installed). All tests were simple transits between wayshrines in the Blackwood zone (only "open field" wayshrines, no wayshrines inside a city with a minimap). While there is no linear increase in loading screen time based on cache file size, there is definitely a pattern and relation between loading screen duration and cache rewrite time.

I was using a simple powershell script with copied code from a blog, that is based on an event handler using System.IO.FileSystemWatcher object and watching the cache file directory for .dat file changes. The duration of the loading screen is more or less equal to the duration of the cache files being written (+1/2 seconds).

PC1 results (old PC)

PC2 results (somewhat less old PC)

The threshold when things become noticable vs. annoying vs. painful does vary. And rewrite times are not always identical/similar/consistent based on file size. So there other factors. However, the duration of the loading screen depends primarily (almost exclusively) on the rewrite of the cache files (IF a rewrite happens).

I would be surprised if ZoS is not able to reproduce this behavior.

hiyde · April 30

Sachsen71 wrote: »

hiyde wrote: »

Sachsen71 wrote: »

So is there ANY tool, add-on, or setting that works now, that allows us to see when guild members joined our guild? Shissu's used to provide that but now it's gone. I used to be able to see the join date and how many days they had been a member, which was indispensable since we promoted based on time in guild. Now I don't know what to do.

U41 was such a disaster, I'm so incredibly frustrated trying to be a GM with one hand tied behind my back.

ITTsRosterBot still seems to work for me (though not 100% of the time). I do see some members (including brand new ones) where there's no info in the tooltip.

Thank you - can you tell me how to access that screen in your screenshot? I have ITTSRosterBot installed but not sure how to access.

@Sachsen

Mouse over usernames in the guild roster.

Sachsen71 · April 30

hiyde wrote: »

Sachsen71 wrote: »

hiyde wrote: »

Sachsen71 wrote: »

So is there ANY tool, add-on, or setting that works now, that allows us to see when guild members joined our guild? Shissu's used to provide that but now it's gone. I used to be able to see the join date and how many days they had been a member, which was indispensable since we promoted based on time in guild. Now I don't know what to do.

U41 was such a disaster, I'm so incredibly frustrated trying to be a GM with one hand tied behind my back.

ITTsRosterBot still seems to work for me (though not 100% of the time). I do see some members (including brand new ones) where there's no info in the tooltip.

Thank you - can you tell me how to access that screen in your screenshot? I have ITTSRosterBot installed but not sure how to access.

@Sachsen

Mouse over usernames in the guild roster.

Thank you. Sadly, when I mouse over usernames in the guild roster, I only get the character name in the popup. I have LibHistoire and ITT's Roster Bot both installed and running -- am I missing some other add-on or library or dependency? I looked through the Settings for Roster Bot and didn't see anything that would help.

Thank you for any assistance.

erdbeerheld · May 1

Looks like there is a bug with the cache file rewrite and the SET GuildHistoryCacheMaxNumberOfDays_trader setting. I am currently running with SET GuildHistoryCacheMaxNumberOfDays_trader "5", and the files have decreased accordingly. I can also see this in the Guild History itself, as I can click through the history directly, and once I hit the configured range, I get the "Show More" button.

However, when retrieving data beyond that range, this data is still written to the cache files, as long as ESO is not closed. Only after closing and restarting ESO, the files are shrinked accordingly based on what is configured in UserSettings.txt

I did a test and retrieved 2 months, which resulted in a 18MB cache file. Once I restarted ESO, only the data for the configured range was written to the cache file and shrinked to its intended size.

Techwolf_Lupindo · May 1

I found a workaround for most of the issues here.

ZOS managed to fix one bug that was causing history to get truncated in display, but not in the files. I usually had to restore from an earlier backup of the cache files to fix. That BUG was fixed.

My daily routine is to log into a minor character that does not have LibHistre or any addon that depend on it. Go though history and manually update/download until there is no more to download. Big trade guilds will have several pages and "get more" before catching up to what in the cache files.

Log into each of my daily writs characters and do my crafting writs.

Log into my main character that has LibHistre, MM and all the other addons active. MM updates fine and Libhistory should be all green.

What messes things up is when I log into the main character before updating the history causes the hangs and other issues. The fix is to log out and log into a character without Libhisre active and update the history manually.

From my experience, cache files are not saved until you log out of the entire game itself. Not just the character. This is different from the save lua files that are saved to disk upon logging out of that character or /reloadui.

erdbeerheld · May 1

Techwolf_Lupindo wrote: »

From my experience, cache files are not saved until you log out of the entire game itself. Not just the character. This is different from the save lua files that are saved to disk upon logging out of that character or /reloadui.

Cache files are saved/rewritten when both
a) new data has been retrieved
b) wayshrine usage, door usage, portal usage, logout

Each cache file is only saved/rewritten, when new data has been retrieved for that specific guild

Sharlikran · May 1

Techwolf_Lupindo wrote: »

I found a workaround for most of the issues here.

My daily routine is to log into a minor character that does not have LibHistre or any addon that depend on it. Go though history and manually update/download until there is no more to download. Big trade guilds will have several pages and "get more" before catching up to what in the cache files.

https://www.esoui.com/forums/showthread.php?t=10879

On page 2 toward the bottom I list programs using LibHistoire and those that don't. Mods that don't use LibHistoire and instead access the new API directory may stumble on any issues with the API.

If I limit my mods to MM and AMT then I have no issues and I don't have to worry about the guild history. Although I don't really have any issues with TTC either except that it may cause issues when the sales tracking is enabled because that uses the new API directory and doesn't go through LibHistoire.

Gabriel_H · May 6

erdbeerheld wrote: »

The threshold when things become noticable vs. annoying vs. painful does vary. And rewrite times are not always identical/similar/consistent based on file size. So there other factors. However, the duration of the loading screen depends primarily (almost exclusively) on the rewrite of the cache files (IF a rewrite happens).

I would be surprised if ZoS is not able to reproduce this behavior.

I would add to your "noticable vs. annoying vs. painful". When my cached data size exceeds 1MB (a tiny amount of data by today's standards) my loadscreen times begin to increase from a couple of seconds to upwards of 30 seconds. At 5MB, being about 30 days worth of sales data in an active trading guild the loadscreens become UNPLAYABLE.

An example: Ruins of Mazzatun. A dungeon with lots of short hops to doors that then trigger a load screen. There are about 10 load screens in all during a single run, at 5MB of cached data I'm stuck on a loadscreen for 1 to 2 minutes per door, being 10 to 20 minutes in total - making a speed run impossible.

Having removed LibHIstoire from my addons has of course helped, but even without that the cached data begins to slowly build in size, forcing me to have to delete the cache before a play session to avoid the loadscreen increased length.

This is obviously not was intended by ZOS, [snip]

[edited for bashing]

Sharlikran · May 6

erdbeerheld wrote: »

Having removed LibHIstoire from my addons has of course helped, but even without that the cached data begins to slowly build in size, forcing me to have to delete the cache before a play session to avoid the loadscreen increased length.

Some users still think LibHistoire is storing data, it does not. ZOS stores the data and LibHistoire is simply a central library to retrieve and send the ZOS data to mods such as ATT or MM.

Without any mods, for example if you renamed your Addons folder, then the ZOS cache would still be there because it's part of the game itself. The ZOS cache will always be present and load whether or not you have any mods.

Gabriel_H · May 7

Sharlikran wrote: »

erdbeerheld wrote: »

Having removed LibHIstoire from my addons has of course helped, but even without that the cached data begins to slowly build in size, forcing me to have to delete the cache before a play session to avoid the loadscreen increased length.

Some users still think LibHistoire is storing data, it does not. ZOS stores the data and LibHistoire is simply a central library to retrieve and send the ZOS data to mods such as ATT or MM.

Yes, but LibHistoire retrieves that data by looking back at the cached data and growing it.

Sakiri · May 7

Sharlikran wrote: »

Sachsen71 wrote: »

Can anyone help me understand what I need to do to fix this? I get this popup every time I log out now. I have tried setting my categories to Automatic and also to Force On but the popup is there either way.

Reading the information in the window how would you summarize that?

I have been using LibHistoire since it was released and I used the developmental version prior to its first release. Therefore, what I feel it says may not be what you see.

What does that notification say to you?

If you open your guild history and you have all green bars then you have a different issue and you should post a screen shot of your LibHistoire UI on the ESOUI forums here.

If you don't know what guild history is and you don't know what green bars means then you are simply ignoring the entire process. Read how to obtain history here.

I also get that error message when I log out, if I've logged out too soon before the history has caught up. as a result, the characters I quick hop on in the mornings I just disabled the addon and libraries for.

If I'm thinking correctly, it's because the only category I have it collecting is my trader, everything else is forced OFF. It's probably complaining it can't index that which I don't have it collecting.

erdbeerheld · May 7

@Sharlikran I guess something got messed up on your quote. The quoted statement is from @Gabriel_H , not me

(I assume you wanted to quote my earlier post, at least thats where the link points).

While the game-native caching is a good idea, I think there are a few design/coding issues.

Cache retention setting (SET GuildHistoryCacheMaxNumberOfDays_* in UserSettings.txt)

I still think this is the best/cleanest way to mitigate the loading screen issue in general.
While the game uses that setting to truncate events older than the configured setting on initial load and start of ESO, it still writes retrieved data beyond that range to the cache files while the game is running. So theoretically you could end up with an increased size during a very long ESO session.

Events that trigger a cache rewrite

This should not be done for simple door/portal usage inside an instance. Removing that part would greatly improve the experience esp. for group content.
Even doing this for wayshrine transits within the same zone is at least debatable.

Sharlikran · May 7

Sakiri wrote: »

If I'm thinking correctly, it's because the only category I have it collecting is my trader, everything else is forced OFF. It's probably complaining it can't index that which I don't have it collecting.

The notification that you have missing events will not show up for categories set to Force Off.

Sharlikran · May 7

Gabriel_H wrote: »

Yes, but LibHistoire retrieves that data by looking back at the cached data and growing it.

It does not request more information when set to Automatic and no mods are enabled.

If you set everything to force off and trader to Automatic as suggested, then disable any trading mods then with LibHistoire active it wouldn't request any older sales. It's easier to have LibHistoire disabled but that is the behavior.

With that setup or without mods you will always have the new sales occurring after you log in and there is no way to disable that.

Sachsen71 · May 10

hiyde wrote: »

Sachsen71 wrote: »

hiyde wrote: »

Sachsen71 wrote: »

So is there ANY tool, add-on, or setting that works now, that allows us to see when guild members joined our guild? Shissu's used to provide that but now it's gone. I used to be able to see the join date and how many days they had been a member, which was indispensable since we promoted based on time in guild. Now I don't know what to do.

U41 was such a disaster, I'm so incredibly frustrated trying to be a GM with one hand tied behind my back.

ITTsRosterBot still seems to work for me (though not 100% of the time). I do see some members (including brand new ones) where there's no info in the tooltip.

Thank you - can you tell me how to access that screen in your screenshot? I have ITTSRosterBot installed but not sure how to access.

@Sachsen

Mouse over usernames in the guild roster.

Hey all, sorry to be a bother. I would really love it if I could figure out how to make this function for me to see when members joined my guild. I'm not even sure exactly which add-on displays this window. If you could at least give me the add-on name, I can make sure I have it installed and then check esoui for some additional help. Thank you everyone!

Gabriel_H · May 10

Sharlikran wrote: »

Gabriel_H wrote: »

Yes, but LibHistoire retrieves that data by looking back at the cached data and growing it.

It does not request more information when set to Automatic and no mods are enabled.

If you set everything to force off and trader to Automatic as suggested, then disable any trading mods then with LibHistoire active it wouldn't request any older sales. It's easier to have LibHistoire disabled but that is the behavior.

With that setup or without mods you will always have the new sales occurring after you log in and there is no way to disable that.

Doesn't solve the problem though - being the cache file, which grows regardless of LibHis, but the latter when active causes it grow quicker. Hence why I removed it.

Gabriel_H · May 10

erdbeerheld wrote: »

While the game-native caching is a good idea, I think there are a few design/coding issues.

Cache retention setting (SET GuildHistoryCacheMaxNumberOfDays_* in UserSettings.txt)
I still think this is the best/cleanest way to mitigate the loading screen issue in general.

While the game uses that setting to truncate events older than the configured setting on initial load and start of ESO, it still writes retrieved data beyond that range to the cache files while the game is running. So theoretically you could end up with an increased size during a very long ESO session.

Events that trigger a cache rewrite
This should not be done for simple door/portal usage inside an instance. Removing that part would greatly improve the experience esp. for group content.

Even doing this for wayshrine transits within the same zone is at least debatable.

After exploring this further, you are right and wrong. Changing the retention setting does indeed work, but it is NOT truncated at the initial load - it is however re-written on the events you mentioned.

I manually edited the settings to 3 days for all cache settings.

This is 5 minutes after load, stood in Necrom Bank: https://ibb.co/Tq66Gsw

This is 5 seconds after I exit Necrom Bank: https://ibb.co/bBmMQPc

So a long play session should not impact as every time you port/door the older than x days are removed and any new events are added. It's also worth mentioning that logging out does not appear to update the cache either.

I have also noted that in the addon settings for LibHistoire, it gives the ability to change the number of days each type of cache is stored - this is picked up from the UserSettings.txt, and any changes here are saved back to that file. The problem here is altering those days in LibHistoire are subjected to the defaults set by LibHistoire.

i.e. If you only want to cache 3 days of data, you have to change the UserSettings.txt manually, as LibHistoire has a hard-coded minimum 30 days.

EDIT: Added this more complete explanation.

Lets say you just installed Master Merchant (or ATT) and LibHistoire - you have to manually go into Guild History and refresh with the E-Key. This generates the cache, which MM then picks up and stores in its own saved variable files.

The larger that cache file gets, the slower the load screen because it is being re-written every time you open a door or port, but oddly not at load or quit. The size of the MM saved variables does not cause any issues.

If you go to Addon Settings for LibHistoire you'll see a bunch of sliders, these control the number of days data saved in the cache - they have hard coded minimums. Trader for example is 30 days. These setting come from the UserSetting.txt file in "C:\Users****\Documents\Elder Scrolls Online\live". Changing them in LIbHistoire change them in the settings file.

Opening the UserSettings.txt file with any text editor and scroll down to the lines highlighted in the picture. https://ibb.co/7JZpv6R

You can see I have manually changed them to 3 days and then save the file. Open the game, and go through a door/port and the game will automatically cut the cache file size - no more extended load screen time.

If you go to Addon Settings for LibHistoire you'll see in the boxes the number of days you manually entered in UserSettings.txt BUT if you touch the slider or edit the boxes the default number will be restored.

Now lets say you go offline for a week. You come back, go to Guild History and refresh the data manually to pull through the missing sales data, the cache file will get big, the data will be pulled out by MM and stored in its own variable files, and the moment you go through a door/port the cache file will reduce in size back to the number of days you manually set in UserSettings.txt

Sakiri · May 10

Sharlikran wrote: »

Sakiri wrote: »

If I'm thinking correctly, it's because the only category I have it collecting is my trader, everything else is forced OFF. It's probably complaining it can't index that which I don't have it collecting.

The notification that you have missing events will not show up for categories set to Force Off.

Then why on earth does it show up intermittently?

I had it every time I logged out yesterday, even after sitting in mournhold for hours bsing with folks, but I had weeks of it not popping up before?

I don't understand. And I wish I could turn it off without another addon. Because yes, one exists to disable this message.

Sharlikran · May 10

Sakiri wrote: »

Then why on earth does it show up intermittently?

I had it every time I logged out yesterday, even after sitting in mournhold for hours bsing with folks.

I don't understand how people have these issues. The first thing I do when I play is manually get all my data. I never sit and wait talking to other players because I know the data is server controlled and I won't get the data fast enough.

Mods cannot change the server cooldown and I don't understand why people don't want to manually get the data when they know it's server controlled and will take hours. It depends on how busy your guilds are. A small casual guild will work. A busy guild with billions per week in sales, forget it.

A guild with billions in sales, the server will never give you the data fast enough not even for one days worth of sales. If the server needs to give you more than a day in sales you won't be online long enough to get that because the server will restrict it too mich. This has been this way for over 4 years now.

Sakiri · May 11

I never had to manually fetch data before. Expecting me to do so now just is a bad idea. They didn't need to change anything, it worked fine before.

I have better things to do than click "show more" 400 times before getting on with my play session.

hiyde · May 11

Sakiri wrote: »

I never had to manually fetch data before. Expecting me to do so now just is a bad idea. They didn't need to change anything, it worked fine before.

I have better things to do than click "show more" 400 times before getting on with my play session.

For years now, as has been said, people in large trade guilds / multiple trade guilds / multiple large trade guilds have had to mash that "E" key to pull in records 100 at a time for each guild.

It's actually faster now than it used to be (thankfully we get more records each pull), and I'm finding that I'm able to catch up my data within a minute or two. Remember, you don't have to load allllllll the way back every time. You just need to load far enough back for Master Merchant to pick up where it left off. If you're logging in daily, this should take as little as a minute. If you're logging in and updating less often, it'll proportionally take longer.

And yeah, it would be terrific if none of us had to do this.
It would be even more awesome if the functionality of trading addons like MM was built into the game, especially for console players. ZOS set up a system of 100s of small businesses and provides zero reporting on super basic things like how much the guild sold, what each member is contributing, etc. Gods bless console guilds. I could never...

Sharlikran · May 12

Sakiri wrote: »

I never had to manually fetch data before. Expecting me to do so now just is a bad idea. They didn't need to change anything, it worked fine before.

I have better things to do than click "show more" 400 times before getting on with my play session.

You and many other people don't realize this, but it has been that way for 4 years now. Even before LibHistoire.

Prior to Update 41, people that had your opinion of not manually fetching data blamed LibHistoire for missing data when it was the users neglect. I had people with sometimes 90 days of missing data even though the server only had 10 days prior to the update.

Once the notification was added by the author so LibHistoire would remind people they had missing data, people didn't like it. Some people simply uninstalled LibHistoire. I don't know why but for the most part, people seemed to think LibHistoire was using an inferior method of obtaining data. No matter how many times I explained it wasn't mods, that it was server controlled, nobody understood that. People's expectations versus what I have been saying for 4 years didn't align so people believed whatever they wanted.

image.png?ex=6641d324&is=664081a4&hm=164c62a87c9dc3a1aa67385cc68ddbec1760577b3f54e22cff9c860e7a1252e1&

image.png?ex=6641d324&is=664081a4&hm=164c62a87c9dc3a1aa67385cc68ddbec1760577b3f54e22cff9c860e7a1252e1&

That is what I have usually.

image.png?ex=6641caf0&is=66407970&hm=8759ebb763c223b9b22d382dbe99813078f7468c9e25bad63827ec76c11838d5&

image.png?ex=6641d3a8&is=66408228&hm=e3d1cd1c401a49ff6e1b0e802851676562b9d3739cf54d579849782a83286ef8&

Your bar may look something like those. Although I use Dark UI.

Whatever you see for your Trader category at the moment is exactly what has happened for you in the past. You have had gaps in data and simply didn't know it. You may feel it worked before and ZOS didn't need to change it but, it was changed.

As mentioned, unless you delete your cache, you only need the data you are missing since you were last online. Each time you press E you receive 500 events or 5 pages, unless your data is more fragmented. Depending on how busy a guild is, 3000 sales in a day you would need to press E Show More 6 times, not 400.

LibHistoire uses the new API or the new Guild History functionality to provide a user interface similar to what it had before using the new functions ZOS provided. You can see what data you are missing, you can disable automatic requests for categories mods are not using, you can clear the cache, you can choose how much data you retain, all from the LibHistoire UI. The only difference is that now all the data is saved in the ZOS binary cache. ZOS was, and still is in total control of the cooldown and how often you get data.

It's been discussed and planned for months. I think I have probably known for 6 to 8 months about this. They aren't going to change it.

The server controlled rate of data has been slow for years. That's nothing new and it wasn't part of Update 41. Update 41 simply added new functionality for mod authors.

erdbeerheld · May 13

Gabriel_H wrote: »

After exploring this further, you are right and wrong. Changing the retention setting does indeed work, but it is NOT truncated at the initial load - it is however re-written on the events you mentioned.

So a long play session should not impact as every time you port/door the older than x days are removed and any new events are added. It's also worth mentioning that logging out does not appear to update the cache either.

I still think they are not truncated while you are playing and the game is rewriting the cache files. The data is only truncated according to retention settings when initially loaded from the cache file. My guess is, after that initial load from the cache files, the data is only saved back to the files on the events mentioned before, but not adhering to the retention settings.

You can check/simulate this, if you intentionally load several days of data manually AFTER the initial load and save (including truncate) has been done. The files will get significantly larger, depending on how many additional days you load.

Gabriel_H wrote: »

I have also noted that in the addon settings for LibHistoire, it gives the ability to change the number of days each type of cache is stored - this is picked up from the UserSettings.txt, and any changes here are saved back to that file. The problem here is altering those days in LibHistoire are subjected to the defaults set by LibHistoire.

i.e. If you only want to cache 3 days of data, you have to change the UserSettings.txt manually, as LibHistoire has a hard-coded minimum 30 days.

Agreed, thats why I provided the manual step-by-step description.

Gabriel_H wrote: »

It's also worth mentioning that logging out does not appear to update the cache either.

If it has retrieved additional data since last save, it will also rewrite on logout.