Saved variables file size limits

Slick_007

so iv been using price tracker addon and it stores pricing data that i see on traders into the savedvariables folder.

The file has hit 76.8MB - yeh theres a lot of traders in the game, and ESO doesnt load any of it now.

Is there a file size limit for the stuff in this folder?

how does mm or ttc compare for instance for its variables? dont tell me to simply use them, id rather not. im asking about the savedvariables not what you prefer as a pricing addon.

Ayantir

there is several limits. But I could say that 10MB+ can lead to issues.
76MB file size is dangerous and can create data corruption

Slick_007

at around 50MB it still seemed reasonably stable. im editing out the ridiculus pricing entries by people obviously posting to mess with average pricing, slowly cutting it down. i can see a way to improve on how much data each entry takes up, but the question is can i rewrite the file to do it heh.

76M is the trade vendors in order of zones, from alik'r to grahtwood. there is so much pricing data to collect.

InvitationNotFound

Ayantir wrote: »

there is several limits. But I could say that 10MB+ can lead to issues.
76MB file size is dangerous and can create data corruption

source for this statement please. i haven't seen any restrictions nor issues.

Shinni

InvitationNotFound wrote: »

Ayantir wrote: »

there is several limits. But I could say that 10MB+ can lead to issues.
76MB file size is dangerous and can create data corruption

source for this statement please. i haven't seen any restrictions nor issues.

I guess in this case, the post you quoted is the source. If you are not aware, Ayantir is the developer of Skyshards, Lorebooks, Destinations etc.
If you want another addon dev to confirm the save file issue: I can also confirm from bug reports on my addon HarvestMap (another very data heavy addon like PriceTracker), that the game sometimes just stops writing to the save file halfway through, if it gets too large.

InvitationNotFound

Shinni wrote: »

InvitationNotFound wrote: »

Ayantir wrote: »

there is several limits. But I could say that 10MB+ can lead to issues.
76MB file size is dangerous and can create data corruption

source for this statement please. i haven't seen any restrictions nor issues.

I guess in this case, the post you quoted is the source. If you are not aware, Ayantir is the developer of Skyshards, Lorebooks, Destinations etc.
If you want another addon dev to confirm the save file issue: I can also confirm from bug reports on my addon HarvestMap (another very data heavy addon like PriceTracker), that the game sometimes just stops writing to the save file halfway through, if it gets too large.

i don't consider this a valid source. it could be possible, of course. but just because two addon developer had an issue with the file size, doesn't mean it is a general issue, right? it could be something else, game did crash, whatever. a lot of information is missing imho.

Have any bug reports been sent? has anyone at ZOS given an answer to this? is it only the size of a single file or instead the size of all files?

Slick_007

Shinni wrote: »

If you want another addon dev to confirm the save file issue: I can also confirm from bug reports on my addon HarvestMap (another very data heavy addon like PriceTracker), that the game sometimes just stops writing to the save file halfway through, if it gets too large.

in this case it appears to be a reading problem not a writing problem. so what happens is, the folder contains the large file when i login. when you login, its loads this information. no pricing data loads at all.
i removed some entries, and missed a bracket. when i logged in, i got an error. cleared the bracket - no more error. so it sees the file.

now the fun begins when you logout. this is when it writes to the variables files. all of them. because price tracker has zero pricing data for your character, it removes all the data in the file when you logout, and the 78MB is now a 1KB file of nothing.

it also writes to the price tracker variable if you disable the addon. i dont recall if it did the others then too. but then i put the big file back in and re-enable, no data loads.

SirAndy

All SavedVariables are loaded into LUA memory when you log into a character.
If you don't have enough memory, you will get data corruption.

With a file that big, you need to bump up your LUA memory, which you can do from the general ESO settings file, can't remember the exact name of the entry (at work right now) but it says something about LUA and memory.

Again, whenever you log into a character, all AddOns enabled for this character will be loaded into memory along with their respective SavedVariables files and the SavedVariables will be written back to file when you log out to the character select (or /reloadui).

That means that if you run out of LUA memory and you log out, your SavedVariables file *will* be corrupt!
At that point, it would be better to kill the ESO process and not log out.

Ayantir

InvitationNotFound wrote: »

Ayantir wrote: »

there is several limits. But I could say that 10MB+ can lead to issues.
76MB file size is dangerous and can create data corruption

source for this statement please. i haven't seen any restrictions nor issues.

Me, I and Myself. And I know more on ESO SavedVars than anyone here. If you don't believe me, no problem, I'm accustomed.

SirAndy wrote: »

All SavedVariables are loaded into LUA memory when you log into a character.
If you don't have enough memory, you will get data corruption.

That means that if you run out of LUA memory and you log out, your SavedVariables file *will* be corrupt!
At that point, it would be better to kill the ESO process and not log out.

It's a bit more complicated than this.

You can have a savedvar corrupted without any Lua Memory issues.

And I would say that 64bits client and few internal reports almost fixed all the cases. But it can still be possible.

--

The guy who played with thoses issues 2 years ago.

InvitationNotFound

SirAndy wrote: »

All SavedVariables are loaded into LUA memory when you log into a character.
If you don't have enough memory, you will get data corruption.

With a file that big, you need to bump up your LUA memory, which you can do from the general ESO settings file, can't remember the exact name of the entry (at work right now) but it says something about LUA and memory.

Again, whenever you log into a character, all AddOns enabled for this character will be loaded into memory along with their respective SavedVariables files and the SavedVariables will be written back to file when you log out to the character select (or /reloadui).

That means that if you run out of LUA memory and you log out, your SavedVariables file *will* be corrupt!
At that point, it would be better to kill the ESO process and not log out.

i don't think it works like that.

you don't run out of lua memory just like that. they had a warning a while ago about lua memory and stuff, which you could increase through changing some settings. later this option has been removed and the developers stated that the lua memory was increased in that case and no memory issues or corruptions have occurred. it simply was an error message, nothing else.
if it is what you are referring to. if you refer to memory in general, no that's not an issue as well. in such a case many applications couldn't run at all.

Ayantir wrote: »

InvitationNotFound wrote: »

Ayantir wrote: »

there is several limits. But I could say that 10MB+ can lead to issues.
76MB file size is dangerous and can create data corruption

source for this statement please. i haven't seen any restrictions nor issues.

Me, I and Myself. And I know more on ESO SavedVars than anyone here. If you don't believe me, no problem, I'm accustomed.

SirAndy wrote: »

All SavedVariables are loaded into LUA memory when you log into a character.
If you don't have enough memory, you will get data corruption.

That means that if you run out of LUA memory and you log out, your SavedVariables file *will* be corrupt!
At that point, it would be better to kill the ESO process and not log out.

It's a bit more complicated than this.

You can have a savedvar corrupted without any Lua Memory issues.

And I would say that 64bits client and few internal reports almost fixed all the cases. But it can still be possible.

--

The guy who played with thoses issues 2 years ago.

would you mind to provide more than just claims? on a technical level please. don't get me wrong here, i'm not saying that you're telling bs or so, but i'd simply like to understand why you came to that conclusion. and no, "i had corrupted files" isn't enough. so please go ahead and share your knowledge with us.

SirAndy

InvitationNotFound wrote: »

i don't think it works like that.

I'm the author of several AddOns, so yeah, it pretty much works that way.

The LUA memory in ESO is separate from the normal application memory, so yes, you can easily run out of LUA memory with the default settings if you have AddOns that load large SavedVariables files or are otherwise memory intensive.

Philgo68

I will echo Ayantir's findings.

The LUA memory setting, or "running out" of LUA memory, is not a thing. I would venture to say that I hold more addon data than most everyone in the game and I've never "run out" while in the game. Nor have I seen data get corrupted during a gaming session. The issue seems to be an occasional bug that is related to saving large LUA Saved variable files. The file is then corrupted on disk. Bad data, or no data, is then read the next time you login. When MM only stored in one file it would end up getting corrupted around 64mb (maybe 32mb back then with the 32bit client.) This happen very consistently when the file got around that size. As soon as I broke it into multiple files, it could store much more data safely and consistently on the same computer with the same LUA setting. The MM processing code was not really adjusted, just the way the data was saved to disk, via the MMxxData mule addons.

PriceTracker could probably be adjusted to use Mule addons like MM does to hold more information.

And with how much sales data people are trying to save, I may have to adjust MM to use more files, as some people are hitting the "limit" with some of the mule addon files.

--Philgo

SirAndy

@Philgo68

So ZOS did away with the LUA memory setting? Because i still have it in my settings file ...

SET LuaMemoryLimitMB "512"

sylviermoone

SirAndy wrote: »

@Philgo68

So ZOS did away with the LUA memory setting? Because i still have it in my settings file ...

SET LuaMemoryLimitMB "512"

Per Chip Hilseberg, the error was purely informative. It also never needed to be updated, as it's my understanding that LUA memory always auto-expanded to the user's need. It was removed with the Morrowind update.

http://www.esoui.com/forums/showthread.php?t=6985

SirAndy

sylviermoone wrote: »

SirAndy wrote: »

@Philgo68

So ZOS did away with the LUA memory setting? Because i still have it in my settings file ...

SET LuaMemoryLimitMB "512"

Per Chip Hilseberg, the error was purely informative. It also never needed to be updated, as it's my understanding that LUA memory always auto-expanded to the user's need. It was removed with the Morrowind update.
http://www.esoui.com/forums/showthread.php?t=6985

Hmmm, that is interesting, however, i did experiment with SavedVariables file sizes quite a bit when i started working on my main AddOn and i could reproduce file corruption when running out of LUA memory.
Granted, that was some 2 years ago and things obviously have changed since then.

Now i wonder if they were segmenting the memory into distinct pages per AddOn, that would explain something like the 64MB size limit Philgo mentioned above.

InvitationNotFound

SirAndy wrote: »

InvitationNotFound wrote: »

i don't think it works like that.

I'm the author of several AddOns, so yeah, it pretty much works that way.

The LUA memory in ESO is separate from the normal application memory, so yes, you can easily run out of LUA memory with the default settings if you have AddOns that load large SavedVariables files or are otherwise memory intensive.

Well, i guess someone was faster providing the link and the ZOS statement.

I do not see a reason why there should be a difference between 32 / 64 bit clients, they both should have plenty of addressable memory (2gb userspace for 32 bit besides what's already in the ram) for a 100MB lua file. And even if that would be the case, where's the difference between a 100mb file and 20 5mb files?

Anyway, I'm still waiting for some technical background information instead of some wild assumptions, which didn't really explain anything at all or were simply wrong.

SirAndy

InvitationNotFound wrote: »

Anyway, I'm still waiting for some technical background information instead of some wild assumptions, which didn't really explain anything at all or were simply wrong.

In that case, you came to the wrong place. The only people who can give you that information work for ZOS.

We're just lowly AddOn authors so please pay no attention to us, carry on, nothing to see here ...

Reorx_Holybeard

Based on my experiences with large saved variable files the issue is not the overall size of the file but the structure of the Lua data in the file and how the data is distributed. I've had 150MB files work fine because the data is distributed over 100s/1000s of variables and arrays while smaller 10-20MB cause data corruption and game crashes as most of the data is in one array.

The one thing I know causes issues is having more than 60k elements in a single Lua table. There doesn't seem to be a set number (perhaps the 64k) but almost all my corruption/crash issues are a result of having more than 60k elements. It may also be related to the size of the elements themselves. For uespLog I have checks that prevent logged data from generating more than 60k entries in a table (for the most part anyways).

Slick_007

this is how price tracker stores it:
PriceTrackerSettings =
{
["Default"] =
{
["<account name goes here>"] =
{
["$AccountWide"] =
{
["showSeen"] = true,
["historyDays"] = 30,
["ignoreFewItems"] = false,
["keyPress"] = "None",
["version"] = 0.3000000000,
["showMinMax"] = true,
["playSound"] = "Book_Acquired",
["algorithm"] = "Weighted Average",
["isPlaySound"] = true,
["itemList"] =
{
["88996"] = <-each item falls under one of these
{
[210] = <-unsure what this is
{
[1512770236] = <-thought this was unique id but appears to be the expiry timer. every single item has this array
{
["guildName"] = "Azura's Guidance",
["guildId"] = 0,
["quality"] = 3,
["purchasePrice"] = 800,
["name"] = "Girdle of the Juggernaut",
["expiry"] = 1512770236,
["stackCount"] = 1,
},
},
},
},
},
},
},
}

Reorx_Holybeard

Its hard/impossible to say from just a short file snippet although it looks like the file is reasonably structured. When you say it "doesn't load it" what exactly do you mean? Do you get a UI error, game crash, the addon doesn't work? If it was truly a corrupt saved variable file then I would expect the large file to be overwritten by a small file when you logout. If that doesn't happen it may be another issue.

The PriceTracker addon hasn't been updated in a year and a half so I would guess it is just broken with the last update which changed some of the API calls breaking MM and our price tracker built into uespLog.

Slick_007

Reorx_Holybeard wrote: »

Its hard/impossible to say from just a short file snippet although it looks like the file is reasonably structured. When you say it "doesn't load it" what exactly do you mean? Do you get a UI error, game crash, the addon doesn't work? If it was truly a corrupt saved variable file then I would expect the large file to be overwritten by a small file when you logout. If that doesn't happen it may be another issue.

The PriceTracker addon hasn't been updated in a year and a half so I would guess it is just broken with the last update which changed some of the API calls breaking MM and our price tracker built into uespLog.

no data loads for any item. thats it. no errors, no nothing. i login, check an item i know has data and no pricing data displays. when i logout, the file is reset to 1KB.

iv been checking it recently, while its working (15MB or so), i have noticed it rewrites the whole file as the arrays are changing the order in which the data is stored. I cant see a difference though in the way its presented now, and the 80MB file that wasnt working. i dont believe its corruption.

Reorx_Holybeard

The symptoms you mention does sound like a corrupt saved variable file though. When a file is corrupt in any way the game just fails to load it (I don't think you get a UI error message) and then the default data is saved when you logout. So the game tries to load the 80MB file, fails somehow and you get the empty 1KB file when you logout.

When I say "corrupt" I mean any saved variable file that the game won't load. This could be actual corruption in the form of an invalid Lua file or just too much data in the file that the loader fails to load it somehow. I've seen both instances happen with the game.

Saved variable files are valid Lua files so you could just install a version of Lua on your computer and try to manually load the 80MB file. This would at least tell you if the file is actually valid or not. If not you could try to fix whatever the corruption is and perhaps load it in ESO...at least once as you would likely get a corrupt file again when you logout.

Slick_007

time to build up the file again and test it.

fyi luaedit 2012 (it says 2010 when you load it) does not work. it wont even load the files that work.
got notepad++, its extremely fast to load a 20MB file. we'll see how that goes once the file is bigger

Elsonso

SirAndy wrote: »

With a file that big, you need to bump up your LUA memory, which you can do from the general ESO settings file, can't remember the exact name of the entry (at work right now) but it says something about LUA and memory.

This is no longer necessary.

Slick_007 wrote: »

so iv been using price tracker addon and it stores pricing data that i see on traders into the savedvariables folder.

The file has hit 76.8MB - yeh theres a lot of traders in the game, and ESO doesnt load any of it now.

Is there a file size limit for the stuff in this folder?

how does mm or ttc compare for instance for its variables? dont tell me to simply use them, id rather not. im asking about the savedvariables not what you prefer as a pricing addon.

I did some testing a while back to find a maximum, but I suspect that it is somewhat system dependent. I ran into trouble only when files exceeded 50MB.

I try to keep saved variable files less than 30MB, but my game can handle larger and remain stable. I have had two between 30 and 35 MB for quite a few months with no ill effects.

I certainly would like to find out from ZOS (ie @ZOS_ChipHilseberg ) if there are design limitations for individual or cumulative data sizes, other than available memory.

Ayantir

If author is aware of his data being up to 20 MB HE_MUST_DO_SOMETHING And with 20MB i'm kind, it should be 10MB.

It's not to ZOS to handle the *** of our code.
There is tons of tips and tricks to reduce data length of our addons.

Encode them, Reduce column keys, remove data, drop ZO_SavedVars, it's not hard.


["88996"] = -- BAD TYPE : It should be a number.
{
[210] = -- BAD CONCEPTION If the ItemId cannot determine your final item, it should be at the same level of the itemId
{
[1512770236] = -- BAD CONCEPTION They key is a timestamp ? It should be an ipaired data.
{
["guildName"] = "Azura's Guidance", -- BAD CONCEPTION, use a table and add a key.
["guildId"] = 0, -- UNNEEDED if guildname is set. guildName and guildId should be 1 and only 1 column. 1-5 for user guild 6 to n for non user guilds.
["quality"] = 3,
["purchasePrice"] = 800,
["name"] = "Girdle of the Juggernaut", -- UNNEEDED, you should be able to build the item name from the itemid/link.
["expiry"] = 1512770236, -- BAD CONCEPTION , it's the key. and this is a timestamp. I agree this should be here and the value is OK but we have a redundancy here.
["stackCount"] = 1, -- UNNEEDED. This should only go in sv when qty is > 1
},


I never used this addon, I never looked to its code and I can divide by 50% the length of its savedvar.

Slick_007

Ayantir wrote: »

If author is aware of his data being up to 20 MB HE_MUST_DO_SOMETHING And with 20MB i'm kind, it should be 10MB.

It's not to ZOS to handle the *** of our code.
There is tons of tips and tricks to reduce data length of our addons.

Encode them, Reduce column keys, remove data, drop ZO_SavedVars, it's not hard.


["88996"] = -- BAD TYPE : It should be a number.
{
[210] = -- BAD CONCEPTION If the ItemId cannot determine your final item, it should be at the same level of the itemId
{
[1512770236] = -- BAD CONCEPTION They key is a timestamp ? It should be an ipaired data.
{
["guildName"] = "Azura's Guidance", -- BAD CONCEPTION, use a table and add a key.
["guildId"] = 0, -- UNNEEDED if guildname is set. guildName and guildId should be 1 and only 1 column. 1-5 for user guild 6 to n for non user guilds.
["quality"] = 3,
["purchasePrice"] = 800,
["name"] = "Girdle of the Juggernaut", -- UNNEEDED, you should be able to build the item name from the itemid/link.
["expiry"] = 1512770236, -- BAD CONCEPTION , it's the key. and this is a timestamp. I agree this should be here and the value is OK but we have a redundancy here.
["stackCount"] = 1, -- UNNEEDED. This should only go in sv when qty is > 1
},


I never used this addon, I never looked to its code and I can divide by 50% the length of its savedvar.

the guild id is not related to the guild name i dont think. i believe its to do with you and your guilds. but i got more stuff to go through yet. i was considering removing it if possible.
1 zone of traders is a couple of MB. i think alikr was like 8MB. its impossible to keep this under 20MB and have a comprehensive list of items for sale.

it has a lot of spaces. it looks in the text file properly formatted for code. but using spaces not tabs. that might cut down a lot. its one thing i havent checked before. it might not support tab though.

Slick_007

68MB so far. still working. it cut 15MB out with expired sales, so building it back up with more trader info.
with all new info iv cut out the guildid line too, still appears to work fine.

still working at 74.1MB. just got an error about the weight average, while scanning a trader. /reloadui and the file is back to 48MB. but it doesnt seem to be related to the earlier edit to remove guilid. i still have entries from before and after. i ran the mouse over the page of items that was displaying when i got the error and nothing showed. but it does have a housekeeping option that removes unparsable data.

74MB is alikr to greenshade, with some data expiring and removing itself.

Solinur

I'll probably try and see if I can find out about the limits with a few tests. It doesn't seem to be filesize but may be number of elements in a table.

Slick_007

68.1MB, game crashed. reloaded and since i lost all the trader data id just gathered, after a couple, did a reloadui.
this corrupted the file, saving item data outside out of the item array.
cleared that out, it still wouldnt load the data. so i created a new file and copied the item list into it and its loaded ok again

so yes, corruption of these files does exist.

Dolgubon

Slick_007 wrote: »

Ayantir wrote: »

If author is aware of his data being up to 20 MB HE_MUST_DO_SOMETHING And with 20MB i'm kind, it should be 10MB.

It's not to ZOS to handle the *** of our code.
There is tons of tips and tricks to reduce data length of our addons.

Encode them, Reduce column keys, remove data, drop ZO_SavedVars, it's not hard.


["88996"] = -- BAD TYPE : It should be a number.
{
[210] = -- BAD CONCEPTION If the ItemId cannot determine your final item, it should be at the same level of the itemId
{
[1512770236] = -- BAD CONCEPTION They key is a timestamp ? It should be an ipaired data.
{
["guildName"] = "Azura's Guidance", -- BAD CONCEPTION, use a table and add a key.
["guildId"] = 0, -- UNNEEDED if guildname is set. guildName and guildId should be 1 and only 1 column. 1-5 for user guild 6 to n for non user guilds.
["quality"] = 3,
["purchasePrice"] = 800,
["name"] = "Girdle of the Juggernaut", -- UNNEEDED, you should be able to build the item name from the itemid/link.
["expiry"] = 1512770236, -- BAD CONCEPTION , it's the key. and this is a timestamp. I agree this should be here and the value is OK but we have a redundancy here.
["stackCount"] = 1, -- UNNEEDED. This should only go in sv when qty is > 1
},


I never used this addon, I never looked to its code and I can divide by 50% the length of its savedvar.

the guild id is not related to the guild name i dont think. i believe its to do with you and your guilds. but i got more stuff to go through yet. i was considering removing it if possible.
1 zone of traders is a couple of MB. i think alikr was like 8MB. its impossible to keep this under 20MB and have a comprehensive list of items for sale.

it has a lot of spaces. it looks in the text file properly formatted for code. but using spaces not tabs. that might cut down a lot. its one thing i havent checked before. it might not support tab though.

The guild id is related to the guild name. It is different for everyone but with the guild name you can determine the guild ID.

While currently alikr might be 8 MB, if you changed stuff, maybe it would be 2-4. There would still be a large saved vars file but it would be smaller.

Addons have no effect on the format of the saved variable tables.

Slick_007

Dolgubon wrote: »

The guild id is related to the guild name. It is different for everyone but with the guild name you can determine the guild ID.

While currently alikr might be 8 MB, if you changed stuff, maybe it would be 2-4. There would still be a large saved vars file but it would be smaller.

Addons have no effect on the format of the saved variable tables.

why do you think the guild id is related to the guild name? they are all zeros in my file. i havent scanned my own guilds trader lately so will do that and see if that changes it.

i put the guild id back in as i was getting errors that i havent seen before, but found the item it was bugging on. removed that from the file. it had some entries with no guild id. but so did a few of the others that did not bug. its math.util error for calculating weighted average thats failing. maybe i missed something on those items when the file went bad the other day.
iv got a bit of work to do to see if i can edit it, but with the event currently, not likely to do much work on it heh.