PTS Update 33 - Feedback Thread for Account Wide Achievements

tmbrinks

DarcyMardin wrote: »

code65536 wrote: »

For people who are looking for a reliable way to back up and save for posterity your character-specific data before Monday:

LibAchievementsArchive

@code65536 , thanks so much for making this library. One question -- I have more than one account. Can I use this library to store the achievement data for all my characters across all 4 accounts, or should I somehow separate them? Thanks!

At quick glance into the saved variables file, it saves Server/account/character name, so you can use it for all your accounts on either server.

(But Code can confirm)

After logging through all 18 characters... the stored data (in this addon)



131 KB... I use like 50x that data to stream a single song to my phone over a wireless network

I struggle mightily to understand how this is going to save the game and fix performance issues. But I'm not a database engineer, just somebody who understands numbers and relative size.

code65536

@DarcyMardin I personally play on multiple accounts, on multiple servers, so you can rest assured that the library can handle multiple accounts.

SilverBride

The closer this gets the less I am playing. I tried to come up with ways to adapt and make this work but I don't think it's possible. I will install LibAchievementsArchive in hope that it can help some day, but for now I don't even want to play any more.

In light of all the negative feedback I hope they decide to pull this and work on the issue of not being able to track our character's progress before releasing it.

I will not purchase another chapter, dlc, or any crowns unless there are some changes made.

Kesstryl

code65536 wrote: »

For people who are looking for a reliable way to back up and save for posterity your character-specific data before Monday:

LibAchievementsArchive

I love you and thank you! Well done!

DarcyMardin

code65536 wrote: »

@DarcyMardin I personally play on multiple accounts, on multiple servers, so you can rest assured that the library can handle multiple accounts.

Thanks so much for your reply...and of course for your awesome coding!

kringled_1

I was happy to see that both the mini zone guide and full version worked OK for skyshards for me. It's a little odd being able to see the achievement list completed and the guide list barely touched, but I can work with it. Not sure if skyshards addon can be made as functional as it is now (not sure if the addon Api allows querying the zone status field rather than the achievement).
Tracking of wb and delve completion is borked but I knew that would be the case.

silvereyes

code65536 wrote: »

For people who are looking for a reliable way to back up and save for posterity your character-specific data before Monday:

LibAchievementsArchive

A million thanks to @code65536 for this library.

I whipped up a quick integration into the Quest Journal for that library into Update 33, so that any completed achievements in the archive are displayed as character-specific, even achievement links.

Character Achievements addon

It's backwards compatible with live, if you want to load it up now, but it won't do anything until Update 33 drops.

Elsonso

code65536 wrote: »

For people who are looking for a reliable way to back up and save for posterity your character-specific data before Monday:

LibAchievementsArchive

Awesome! Thanks for doing this.

SilverBride wrote: »

The closer this gets the less I am playing. I tried to come up with ways to adapt and make this work but I don't think it's possible. I will install LibAchievementsArchive in hope that it can help some day, but for now I don't even want to play any more.

I will be making a rare appearance in game this weekend just to run this addon and clean up before Update 33 hits.

silvereyes

Also just released.

Character Zone Tracker addon

Install that on live and log in to each of your characters in order to preserve character completion for delves, world bosses and world events (e.g. dolmens, geysers).

Make sure you install the LibSavedVars dependency, or the addon won't load.

Features

• Backs up zone completion for delves, world bosses and world events (dolmens, geysers, etc.) on live (Update 32).
• Tracks character zone completion for delves, world bosses and world events once Update 33 is released.
• Shows completion popup announcements for delves, world bosses and world events on individual characters once Update 33 is released.
• Adds tools to reset delves, world bosses or world events. (Keyboard only)
• Adds tools to set a character to load delve, world boss and world event completion from the account (Keyboard only)

Known Issues / Planned Fixes

• Compass shows account-wide icons. This is not intentional.
• No text localization for non-English languages, yet.
• Multi-boss delves in Cyrodiil, Craglorn and Hew's Bane report delve completion after only a single boss kill. This is not intentional.

Limitations

• A game client crash will wipe your progress. To ensure your progress is saved, either relog, or do a /reloadui periodically.
• Progress is tracked in the local SavedVariables folder. Back up your files, preferably to the cloud, to prevent data loss if your storage device fails.

Jaraal

Console folks will still be out of luck, but thanks for picking up the ball that an entire development team dropped! Not all heroes wear capes!

I also find it amusing that instead of data being eliminated, it will simply be transferred to addons. Will that negate any potential performance benefits? I’m not a professional coder, so I’m not sure how that works. Does it only impact the client, or do the data calls add to the server load?

Kesstryl

So many good addons being made, thank you guys!

Now if ZOS just fixes the bugs with AwA, I could make this work, but not until the NPC and quest bugs are fixed and decoupled from achievements.

tomofhyrule

silvereyes wrote: »

Also just released.

Character Zone Tracker addon

Install that on live and log in to each of your characters in order to preserve character completion for delves, world bosses and world events (e.g. dolmens, geysers).

Make sure you install the LibSavedVars dependency, or the addon won't load.

This is beautiful!

Thank you, thank you, thank you for allowing us to keep this functionality at least!

code65536

Jaraal wrote: »

I also find it amusing that instead of data being eliminated, it will simply be transferred to addons. Will that negate any potential performance benefits? I’m not a professional coder, so I’m not sure how that works. Does it only impact the client, or do the data calls add to the server load?

It's effectively transferring the work from the server to the client. That said, any performance impact is negligible. While writing LibAchievementsArchive, I had taken some measurements to see how long it took to fully back up a character's achievement data...

The entire process of reading, processing, and encoding all of the data for all 3219 achievements for a character (which includes associated progress data; e.g., "Humanoid Slayer" is a single achievement with three separate requirements of killing 1000 humanoids, 500 goblins, and 100 giants) took something on the order of 30ms (0.03s).

On the server side of things, each achievement costs ZOS just 128 bits (16 bytes): their APIs reveal that they have a 64-bit field into which the various progress data is packed, and another 64-bit field for the timestamp. Multiply that by 3219 achievements (the highest valid achievement ID this patch), and we're looking at around 50KB of achievement data per character.

The impact on the client is trivial, but that also suggests a somewhat limited impact for leaving it on the server.

Jaraal

Thanks for the explanation!

And their servers must be on life support if scavenging just 50kb per character is going to make or break performant-ness!

Mycelius

code65536 wrote: »

For people who are looking for a reliable way to back up and save for posterity your character-specific data before Monday:

LibAchievementsArchive

Many many thanks for this.
Very much appreciated!

Mycelius

silvereyes wrote: »

Also just released.

Character Zone Tracker addon

Install that on live and log in to each of your characters in order to preserve character completion for delves, world bosses and world events (e.g. dolmens, geysers).

Make sure you install the LibSavedVars dependency, or the addon won't load.

Features

• Backs up zone completion for delves, world bosses and world events (dolmens, geysers, etc.) on live (Update 32).
• Tracks character zone completion for delves, world bosses and world events once Update 33 is released.
• Shows completion popup announcements for delves, world bosses and world events on individual characters once Update 33 is released.
• Adds tools to reset delves, world bosses or world events. (Keyboard only)
• Adds tools to set a character to load delve, world boss and world event completion from the account (Keyboard only)

Known Issues / Planned Fixes

• Compass shows account-wide icons. This is not intentional.
• No text localization for non-English languages, yet.
• Multi-boss delves in Cyrodiil, Craglorn and Hew's Bane report delve completion after only a single boss kill. This is not intentional.

Limitations

• A game client crash will wipe your progress. To ensure your progress is saved, either relog, or do a /reloadui periodically.
• Progress is tracked in the local SavedVariables folder. Back up your files, preferably to the cloud, to prevent data loss if your storage device fails.

Thank you, thank you, many blesings of cheese and sweetrolls upon you!
I just breathed a sigh of relief, now there's some way for me to preserve some form of zone completion tracking for my characters.

jecks33

@code65536 @silvereyes

Addons guys, we love you!!!!!!!!!!!

remosito

aaaa

code65536 wrote: »

Jaraal wrote: »

I also find it amusing that instead of data being eliminated, it will simply be transferred to addons. Will that negate any potential performance benefits? I’m not a professional coder, so I’m not sure how that works. Does it only impact the client, or do the data calls add to the server load?

It's effectively transferring the work from the server to the client. That said, any performance impact is negligible. While writing LibAchievementsArchive, I had taken some measurements to see how long it took to fully back up a character's achievement data...

The entire process of reading, processing, and encoding all of the data for all 3219 achievements for a character (which includes associated progress data; e.g., "Humanoid Slayer" is a single achievement with three separate requirements of killing 1000 humanoids, 500 goblins, and 100 giants) took something on the order of 30ms (0.03s).

On the server side of things, each achievement costs ZOS just 128 bits (16 bytes): their APIs reveal that they have a 64-bit field into which the various progress data is packed, and another 64-bit field for the timestamp. Multiply that by 3219 achievements (the highest valid achievement ID this patch), and we're looking at around 50KB of achievement data per character.

The impact on the client is trivial, but that also suggests a somewhat limited impact for leaving it on the server.

you are neglecting it's database stuff.

One 4k movie is a lot less of an issue than equivalent byte database table with super small data per entry...

a 10GB database with 300M 32byte entries will behave quite different than a 10GB one with 300k 32kbyte entries.

and you forgot to count charid plus achieveid in your space sumup.

code65536

remosito wrote: »

aaaa

code65536 wrote: »

Jaraal wrote: »

I also find it amusing that instead of data being eliminated, it will simply be transferred to addons. Will that negate any potential performance benefits? I’m not a professional coder, so I’m not sure how that works. Does it only impact the client, or do the data calls add to the server load?

It's effectively transferring the work from the server to the client. That said, any performance impact is negligible. While writing LibAchievementsArchive, I had taken some measurements to see how long it took to fully back up a character's achievement data...

The entire process of reading, processing, and encoding all of the data for all 3219 achievements for a character (which includes associated progress data; e.g., "Humanoid Slayer" is a single achievement with three separate requirements of killing 1000 humanoids, 500 goblins, and 100 giants) took something on the order of 30ms (0.03s).

On the server side of things, each achievement costs ZOS just 128 bits (16 bytes): their APIs reveal that they have a 64-bit field into which the various progress data is packed, and another 64-bit field for the timestamp. Multiply that by 3219 achievements (the highest valid achievement ID this patch), and we're looking at around 50KB of achievement data per character.

The impact on the client is trivial, but that also suggests a somewhat limited impact for leaving it on the server.

you are neglecting it's database stuff.

One 4k movie is a lot less of an issue than equivalent byte database table with super small data per entry...

a 10GB database with 300M 32byte entries will behave quite different than a 10GB one with 300k 32kbyte entries.

and you forgot to count charid plus achieveid in your space sumup.

We don't know how things work exactly on the backend, but there's no evidence to suggest that all of the characters' achievements in the game are just dumped into a flat database. Instead, it seems more likely that each character has a chunk of data that is loaded when that character is loaded.

As for the achievements themselves, it makes zero sense from an engineering perspective for achievements to be stored on the server as anything other than a single binary blob, rather than as actual individual database entries. Why on Earth would you give it the full database treatment when you're only ever accessing it by the ID, which are sequential integers, not by timestamp or the amorphous progress bitfield?

So the database shouldn't have 3219 achievement entries. What gets stored in the database for a character should just be a single binary blob, representing an array of fixed-length 128-bit entries; e.g., achievement ID 42 would just be the 16 bytes located at the 672-byte offset.

Okay, maybe ZOS didn't implement it this way. But I find it hard to believe that they'd be so brutally efficient with how each individual achievement is packed and then so completely drop the ball on the obviously correct way for how achievements as a whole should be stored and accessed. And if they did drop the ball, then fixing that would yield far more dividends than whatever they're doing in this patch.

And there is evidence to suggest that ZOS did implement this correctly. Accessing achievement information does not incur any server hits (it would simply be impossible for me to query all 3219 achievements in a fraction of my ping time if any part of that required a server hit), which means that all of the achievement data is transferred to the client when the character is loaded, and all achievement queries happen locally on the client. Which would suggest that achievements are stored as a blob and not individual database entries.

I do believe that, at the end of the day, they are merely concerned about size. 50KB for 10 million characters is half a TB of data to stuff in a database. Nevertheless, I have been skeptical--and remain so--of any purported performance benefits of AwA. After all, if there really was a significant gain, wouldn't ZOS have stated that from the get-go instead of pulling the performance card late in the game only in the face of controversy? That's because all the evidence suggests that the achievement system is already pretty performant and thus any gains would just be overall space gains and nothing more.


PS: I would say that the fact that ZOS's customer support is unable to grant achievements is another point of evidence in favor. If achievements were individual database entries, writing a tool to manually grant a player an achievement would be relatively easy. But if it exists as a blob, a tool to manually edit that would be quite a bit more difficult to make, which is probably why all those people begging ZOS to grant them the bugged Planesbreaker achievement got "sorry, it's impossible" responses.

Troodon80

code65536 wrote: »

On the server side of things, each achievement costs ZOS just 128 bits (16 bytes): their APIs reveal that they have a 64-bit field into which the various progress data is packed, and another 64-bit field for the timestamp. Multiply that by 3219 achievements (the highest valid achievement ID this patch), and we're looking at around 50kB of achievement data per character.

Just to go into this in further detail, because I also wanted to address this:

Jaraal wrote: »

Thanks for the explanation!

And their servers must be on life support if scavenging just 50kb per character is going to make or break performant-ness!

50kB should be considered a worst-case scenario. If what is being said is an accurate representation of how ZOS is dealing with the data on the server side (and… honestly, who knows)… then this is less than 1MB per account (assuming maximum of 18 characters being 900kB), and if we assume 1MB (rounding up)… now multiply that by all the accounts and work out an average. ZOS's last statement about accounts was somewhere just over 19 million, collectively, if I recall.

This is… 19TB (again, rounding up here, 900kB would end up being more around 17TB). That's immense and, quite frankly, nonsensical. 50kB per character is the absolute worst-case scenario. It's split over three platforms, divided separately over North America and EU: PC-EU/NA, PS-EU/NA, Xbox-EU/NA. So, divide by six. Each platform and region would therefore have 3.2TB. This obviously isn't entirely accurate as certain platforms/servers will have varying numbers of players and you can't divide them evenly, and not everyone has 18 characters. But we don't have those numbers, so we can only work with what we have.

The (purely theoretical) reduction in total data stored⁠—not forgetting that achievement data is only a portion of the data actually being stored in total, given data such as inventory, journal/quests, homes, furniture positions, etc.⁠ is in addition to this—would mean the database goes from ~19TB down to 950GB, split over regions and platforms, making it an average of 158GB per platform and region, down from 3.2TB per region/platform… Purely in theory, like I said, that shouldn't be negligible. Again, this is the worst-case scenario and assumes ZOS has made some epic blunder in their database/storage management when they first set out.

Now, even this isn't accurate given the… hybridisation, for lack of a better word and seems to be the buzz-word of the year, of their database to allow for character-specific achievements mingled along with the account-wide achievements. It's higher than 158GB, but I'm not spending any more time on the worst-case scenario.

Realistically, not everyone has 18 characters. I would hazard a guess and say that a very small percentage of players out of the 19m that ZOS has mentioned have more than 8, and most probably don't even have more than 4. From a data storage perspective, having all achievement data stored regardless of completed status wouldn't make sense. The game engine and server should be able to fill in the blanks in terms of what is complete or incomplete.

Possibly the best data we have on this, and this also assumes ZOS is using the most efficient data storage on the server side, ZOS have previously said they've given out "over 1 billion achievements." Wow, that's a lot of achievements! However, when we divide by the total number of players… it works out at an average of 52.63 achievements handed out per account (not character) assuming their own numbers work out at "over 1 billion" (which is a bit vague, but doesn't really change the average significantly enough to care; we can safely assume it's less than 2 billion, otherwise they'd be saying that instead) divided by their stated number of players being 19m+. This somewhat checks out when you look at the achievement/trophy data on various tracking sites which seems to suggest only a small percentage of accounts even reach level 50. Obviously averages only go so far. There are those with 48k+ achievement points (3200+ achievements) and there are those with none.

If they're only storing data for those completed achievements and only using 16 bytes per record, that means… 16~32GB, with 16GB being 1 billion exactly, and 32GB being 2 billion, so it's probably somewhere between the two. This also gets divided by six, over platform and region, meaning 2.67~5.3GB on average per region and platform. We can probably also assume that there are character IDs or names which take up more bytes, but, again, we don't know in what way / the way ZOS is storing it / the schema being used on the server.

16~32GB seems much more reasonable than 19TB, but for anyone to see more than a negligible increase in performance, it would have to assume that there are a lot of players with lots of achievements duplicated over various characters, so that they can be merged. In terms of client/server performance, you'd need to bring down the size of the database to considerably less than half (i.e. into the MB instead of GB), and bear in mind that it will continue to grow as more players join. Which is good in terms of future-proofing it, but won't really see a noticeable performance increase on the client-side in the interim.

Unless they're dealing with the worst-case scenario above, which I honestly couldn't even fathom, client side will see very little in terms of a performance increase when the patch hits. Marginally faster loading times. Maybe. Potentially. Possibly. Also depending on your system specs.

peacenote

Ty to @code65536 and @silvereyes for the great add-ons!!

On another note...

Jaraal wrote: »

Isn't it interesting that people who are are against this change are for the most part passionate about it, writing detailed, multi-paragraphed explanations as to why it disrupts how they approach the game, yet those in favor usually write brief, vague remarks supporting it?

I'm genuinely curious why that is.

Well, I have a theory.

I have seen countless threads where someone says "no, I don't want xyz" and the responses are "you don't have to use it!" And to a certain extent, there's some validity to those responses. While some of us think bigger, about limited resources, unintended consequences, etc., if new functionality can be avoided without too much effort it really can't be that controversial of a change.

But for AwA, because it is a destructive change, folks can't say "just don't use it." So people fully in support of the current implementation are left with either just voicing their personal opinion and saying "well, I'm happy" or going down the road of debating the "right" way to play the game, which never goes well. Which means really there isn't much to say. The implementation can't be defended in the way a non destructive addition or reversible balance change could be.

I do wish the community was more united in insisting that we don't lose something to gain something, and I'm extremely appreciative of the folks who have said "Well, this doesn't impact the way I play but I understand why it is a problem for others and I wish ZOS would fix it." THOSE are the people who get how, as a community, we're more effective if we advocate for consistent game logic and decisions, and don't prescribe to the "I play like this and if I'm happy who cares about everyone else" thinking. This is an MMO which succeeds by having lots of players in its community. We all don't play the game the same way and it is short-sighted to only care about one's own view. This is true in RL as well as in a gaming community, actually.

Elsonso

silvereyes wrote: »

I whipped up a quick integration into the Quest Journal for that library into Update 33, so that any completed achievements in the archive are displayed as character-specific, even achievement links.

Character Achievements addon

silvereyes wrote: »

Also just released.

Character Zone Tracker addon

Thanks! This is great! I will add this to my cleanup this weekend.

Troodon80 wrote: »

code65536 wrote: »

On the server side of things, each achievement costs ZOS just 128 bits (16 bytes): their APIs reveal that they have a 64-bit field into which the various progress data is packed, and another 64-bit field for the timestamp. Multiply that by 3219 achievements (the highest valid achievement ID this patch), and we're looking at around 50kB of achievement data per character.

50kB should be considered a worst-case scenario. If what is being said is an accurate representation of how ZOS is dealing with the data on the server side (and… honestly, who knows)… then this is less than 1MB per account (assuming maximum of 18 characters being 900kB), and if we assume 1MB (rounding up)… now multiply that by all the accounts and work out an average. ZOS's last statement about accounts was somewhere just over 19 million, collectively, if I recall.

This is… 19TB (again, rounding up here, 900kB would end up being more around 17TB). That's immense and, quite frankly, nonsensical. 50kB per character is the absolute worst-case scenario. It's split over three platforms, divided separately over North America and EU: PC-EU/NA, PS-EU/NA, Xbox-EU/NA. So, divide by six. Each platform and region would therefore have 3.2TB. This obviously isn't entirely accurate as certain platforms/servers will have varying numbers of players and you can't divide them evenly, and not everyone has 18 characters. But we don't have those numbers, so we can only work with what we have.

[... much snippage...]

I don't think that the "performance issue" is related to the size of the database used to store this data.

I agree. This isn't about how much data there is. The amount of space being talked about for storage isn't that large. We are only talking double digit Terabytes, which is nothing. I mean, it is certainly large by 1980's standards, when no one needed more than 640k, but not so much today.

I think it is more the burden that this data represents that is the problem they are trying to solve with this misguided effort. These days, the storing of data is not the issue, it is the using of the data that can be done wrong. We have made it possible to store massive amounts of data, but our ability to manage and use that data and remain performant has not kept up at the same rate. Their own comments reveal that they simply did not plan for the game to be as large as what they ended up with.

I said "misguided", not just because I dislike it, but due to the fact that if this was the best that they could come up with, then "interesting times" are in store for the players of this game. This reveals a level of "acceptable problem solving" that I am not comfortable with, and is likely to be repeated.

Troodon80 wrote: »

Now, even this isn't accurate given the… hybridisation, for lack of a better word

The better word is "homogenization". It appears to be for more than combat skills. I am thinking that this word represents the shining light leading them from the darkness they are in. It is the theme for 2021 and 2022. This light might be salvation, or they may end up wishing Carol Anne was there to warn them from going into the light.

silvereyes

Troodon80 wrote: »

code65536 wrote: »

On the server side of things, each achievement costs ZOS just 128 bits (16 bytes): their APIs reveal that they have a 64-bit field into which the various progress data is packed, and another 64-bit field for the timestamp. Multiply that by 3219 achievements (the highest valid achievement ID this patch), and we're looking at around 50kB of achievement data per character.

Just to go into this in further detail, because I also wanted to address this:

Jaraal wrote: »

Thanks for the explanation!

And their servers must be on life support if scavenging just 50kb per character is going to make or break performant-ness!

50kB should be considered a worst-case scenario. If what is being said is an accurate representation of how ZOS is dealing with the data on the server side (and… honestly, who knows)… then this is less than 1MB per account (assuming maximum of 18 characters being 900kB), and if we assume 1MB (rounding up)… now multiply that by all the accounts and work out an average. ZOS's last statement about accounts was somewhere just over 19 million, collectively, if I recall.

This is… 19TB (again, rounding up here, 900kB would end up being more around 17TB). That's immense and, quite frankly, nonsensical.

It really isn't. Not for a data-driven software service catering to a theoretical 19 million people (which, let's be honest, most are going to be in cold storage)

code65536 wrote: »

remosito wrote: »

aaaa

code65536 wrote: »

Jaraal wrote: »

I also find it amusing that instead of data being eliminated, it will simply be transferred to addons. Will that negate any potential performance benefits? I’m not a professional coder, so I’m not sure how that works. Does it only impact the client, or do the data calls add to the server load?

It's effectively transferring the work from the server to the client. That said, any performance impact is negligible. While writing LibAchievementsArchive, I had taken some measurements to see how long it took to fully back up a character's achievement data...

The entire process of reading, processing, and encoding all of the data for all 3219 achievements for a character (which includes associated progress data; e.g., "Humanoid Slayer" is a single achievement with three separate requirements of killing 1000 humanoids, 500 goblins, and 100 giants) took something on the order of 30ms (0.03s).

On the server side of things, each achievement costs ZOS just 128 bits (16 bytes): their APIs reveal that they have a 64-bit field into which the various progress data is packed, and another 64-bit field for the timestamp. Multiply that by 3219 achievements (the highest valid achievement ID this patch), and we're looking at around 50KB of achievement data per character.

The impact on the client is trivial, but that also suggests a somewhat limited impact for leaving it on the server.

you are neglecting it's database stuff.

One 4k movie is a lot less of an issue than equivalent byte database table with super small data per entry...

a 10GB database with 300M 32byte entries will behave quite different than a 10GB one with 300k 32kbyte entries.

and you forgot to count charid plus achieveid in your space sumup.

We don't know how things work exactly on the backend, but there's no evidence to suggest that all of the characters' achievements in the game are just dumped into a flat database.

Perhaps not a flat database, but character loads do seem to take longer on older characters with more achievements (and other progression) than brand new character loads. If both had a fixed width binary object with all the data connected to the main character record, they should theoretically take the same amount of time to load.

Also, remember what they said: "By reducing the amount of data we have to store, the database doesn’t have to work as hard so the time it takes to search...." (emphasis mine).

This means that achievements must be in a separate data store that, at the very least, has to do a lookup. Even if it's just by character id and not achievement id too, switching to a lookup by account id could still cut the index size significantly.

code65536 wrote: »

And if they did drop the ball, then fixing that would yield far more dividends than whatever they're doing in this patch.

Yes, but could fixing that be done within the time constraints they seem to be operating under?

code65536 wrote: »

Accessing achievement information does not incur any server hits (it would simply be impossible for me to query all 3219 achievements in a fraction of my ping time if any part of that required a server hit), which means that all of the achievement data is transferred to the client when the character is loaded

Agreed. The client is interacting with a cache.

code65536 wrote: »

all achievement queries happen locally on the client. Which would suggest that achievements are stored as a blob and not individual database entries.

Not sure I follow. How is the fact that there is a cache indicative in any way of the backing database architecture?

code65536 wrote: »

I have been skeptical--and remain so--of any purported performance benefits of AwA. After all, if there really was a significant gain, wouldn't ZOS have stated that from the get-go instead of pulling the performance card late in the game only in the face of controversy?

They never promised any performance benefits to the player. They threw in some theoretical "may" verbiage about load screens and database-heavy operations, but almost as a side-effect, not their motivation for the change. They called out that the "most important" bit was preserving their ability to add "more content and achievements in the future".

Saieden

tmbrinks wrote: »

DarcyMardin wrote: »

code65536 wrote: »

For people who are looking for a reliable way to back up and save for posterity your character-specific data before Monday:

LibAchievementsArchive

@code65536 , thanks so much for making this library. One question -- I have more than one account. Can I use this library to store the achievement data for all my characters across all 4 accounts, or should I somehow separate them? Thanks!

At quick glance into the saved variables file, it saves Server/account/character name, so you can use it for all your accounts on either server.

(But Code can confirm)

After logging through all 18 characters... the stored data (in this addon)



131 KB... I use like 50x that data to stream a single song to my phone over a wireless network

I struggle mightily to understand how this is going to save the game and fix performance issues. But I'm not a database engineer, just somebody who understands numbers and relative size.

Also this is in lua, which is basically plain text. A database can store this in a fraction of the space, with redundancy.

silvereyes wrote: »

Troodon80 wrote: »

code65536 wrote: »

On the server side of things, each achievement costs ZOS just 128 bits (16 bytes): their APIs reveal that they have a 64-bit field into which the various progress data is packed, and another 64-bit field for the timestamp. Multiply that by 3219 achievements (the highest valid achievement ID this patch), and we're looking at around 50kB of achievement data per character.

Just to go into this in further detail, because I also wanted to address this:

Jaraal wrote: »

Thanks for the explanation!

And their servers must be on life support if scavenging just 50kb per character is going to make or break performant-ness!

50kB should be considered a worst-case scenario. If what is being said is an accurate representation of how ZOS is dealing with the data on the server side (and… honestly, who knows)… then this is less than 1MB per account (assuming maximum of 18 characters being 900kB), and if we assume 1MB (rounding up)… now multiply that by all the accounts and work out an average. ZOS's last statement about accounts was somewhere just over 19 million, collectively, if I recall.

This is… 19TB (again, rounding up here, 900kB would end up being more around 17TB). That's immense and, quite frankly, nonsensical.

It really isn't. Not for a data-driven software service catering to a theoretical 19 million people (which, let's be honest, most are going to be in cold storage)

code65536 wrote: »

remosito wrote: »

aaaa

code65536 wrote: »

Jaraal wrote: »

I also find it amusing that instead of data being eliminated, it will simply be transferred to addons. Will that negate any potential performance benefits? I’m not a professional coder, so I’m not sure how that works. Does it only impact the client, or do the data calls add to the server load?

It's effectively transferring the work from the server to the client. That said, any performance impact is negligible. While writing LibAchievementsArchive, I had taken some measurements to see how long it took to fully back up a character's achievement data...

The entire process of reading, processing, and encoding all of the data for all 3219 achievements for a character (which includes associated progress data; e.g., "Humanoid Slayer" is a single achievement with three separate requirements of killing 1000 humanoids, 500 goblins, and 100 giants) took something on the order of 30ms (0.03s).

On the server side of things, each achievement costs ZOS just 128 bits (16 bytes): their APIs reveal that they have a 64-bit field into which the various progress data is packed, and another 64-bit field for the timestamp. Multiply that by 3219 achievements (the highest valid achievement ID this patch), and we're looking at around 50KB of achievement data per character.

The impact on the client is trivial, but that also suggests a somewhat limited impact for leaving it on the server.

you are neglecting it's database stuff.

One 4k movie is a lot less of an issue than equivalent byte database table with super small data per entry...

a 10GB database with 300M 32byte entries will behave quite different than a 10GB one with 300k 32kbyte entries.

and you forgot to count charid plus achieveid in your space sumup.

We don't know how things work exactly on the backend, but there's no evidence to suggest that all of the characters' achievements in the game are just dumped into a flat database.

Perhaps not a flat database, but character loads do seem to take longer on older characters with more achievements (and other progression) than brand new character loads. If both had a fixed width binary object with all the data connected to the main character record, they should theoretically take the same amount of time to load.

Also, remember what they said: "By reducing the amount of data we have to store, the database doesn’t have to work as hard so the time it takes to search...." (emphasis mine).

This means that achievements must be in a separate data store that, at the very least, has to do a lookup. Even if it's just by character id and not achievement id too, switching to a lookup by account id could still cut the index size significantly.

code65536 wrote: »

And if they did drop the ball, then fixing that would yield far more dividends than whatever they're doing in this patch.

Yes, but could fixing that be done within the time constraints they seem to be operating under?

code65536 wrote: »

Accessing achievement information does not incur any server hits (it would simply be impossible for me to query all 3219 achievements in a fraction of my ping time if any part of that required a server hit), which means that all of the achievement data is transferred to the client when the character is loaded

Agreed. The client is interacting with a cache.

code65536 wrote: »

all achievement queries happen locally on the client. Which would suggest that achievements are stored as a blob and not individual database entries.

Not sure I follow. How is the fact that there is a cache indicative in any way of the backing database architecture?

code65536 wrote: »

I have been skeptical--and remain so--of any purported performance benefits of AwA. After all, if there really was a significant gain, wouldn't ZOS have stated that from the get-go instead of pulling the performance card late in the game only in the face of controversy?

They never promised any performance benefits to the player. They threw in some theoretical "may" verbiage about load screens and database-heavy operations, but almost as a side-effect, not their motivation for the change. They called out that the "most important" bit was preserving their ability to add "more content and achievements in the future".

They're probably trying to improve performance impacts for themselves, not for us. Like, for example, players logging in and out of toons for daily writs and random normal dungeons. If this loading is causing a bandwidth issue, I understand what they are trying to do, but it's really a fundamental game design issue that encourages this behavior in first place, and their "solution" is just awful decision making.

As for indexing, they could solve this simply with better partitioning of data to reduce the depth of the B-trees. Still, the bandwidth load can be reduced tremendously by saving the data client-side and verifying flat file with a checksum. The more progress a character makes, the less likely they are to get achievements and have to refresh client data. In fact, pretty much the only time there should be a cache miss is if the game is forcibly closed after an achievement is updated but before the client persistent store is fully synced, which would be exceedingly rare if implemented even half decently.

SilverBride

@code65536 @silvereyes

I had about given up entirely on being able to ever enjoy my characters again. Your addons saved the day and I can't thank you enough. You are like superheroes to me. 🦸‍♂️

silvereyes

Saieden wrote: »

tmbrinks wrote: »

DarcyMardin wrote: »

code65536 wrote: »

For people who are looking for a reliable way to back up and save for posterity your character-specific data before Monday:

LibAchievementsArchive

@code65536 , thanks so much for making this library. One question -- I have more than one account. Can I use this library to store the achievement data for all my characters across all 4 accounts, or should I somehow separate them? Thanks!

At quick glance into the saved variables file, it saves Server/account/character name, so you can use it for all your accounts on either server.

(But Code can confirm)

After logging through all 18 characters... the stored data (in this addon)



131 KB... I use like 50x that data to stream a single song to my phone over a wireless network

I struggle mightily to understand how this is going to save the game and fix performance issues. But I'm not a database engineer, just somebody who understands numbers and relative size.

Also this is in lua, which is basically plain text. A database can store this in a fraction of the space, with redundancy.

silvereyes wrote: »

Troodon80 wrote: »

code65536 wrote: »

On the server side of things, each achievement costs ZOS just 128 bits (16 bytes): their APIs reveal that they have a 64-bit field into which the various progress data is packed, and another 64-bit field for the timestamp. Multiply that by 3219 achievements (the highest valid achievement ID this patch), and we're looking at around 50kB of achievement data per character.

Just to go into this in further detail, because I also wanted to address this:

Jaraal wrote: »

Thanks for the explanation!

And their servers must be on life support if scavenging just 50kb per character is going to make or break performant-ness!

50kB should be considered a worst-case scenario. If what is being said is an accurate representation of how ZOS is dealing with the data on the server side (and… honestly, who knows)… then this is less than 1MB per account (assuming maximum of 18 characters being 900kB), and if we assume 1MB (rounding up)… now multiply that by all the accounts and work out an average. ZOS's last statement about accounts was somewhere just over 19 million, collectively, if I recall.

This is… 19TB (again, rounding up here, 900kB would end up being more around 17TB). That's immense and, quite frankly, nonsensical.

It really isn't. Not for a data-driven software service catering to a theoretical 19 million people (which, let's be honest, most are going to be in cold storage)

code65536 wrote: »

remosito wrote: »

aaaa

code65536 wrote: »

Jaraal wrote: »

I also find it amusing that instead of data being eliminated, it will simply be transferred to addons. Will that negate any potential performance benefits? I’m not a professional coder, so I’m not sure how that works. Does it only impact the client, or do the data calls add to the server load?

It's effectively transferring the work from the server to the client. That said, any performance impact is negligible. While writing LibAchievementsArchive, I had taken some measurements to see how long it took to fully back up a character's achievement data...

The entire process of reading, processing, and encoding all of the data for all 3219 achievements for a character (which includes associated progress data; e.g., "Humanoid Slayer" is a single achievement with three separate requirements of killing 1000 humanoids, 500 goblins, and 100 giants) took something on the order of 30ms (0.03s).

On the server side of things, each achievement costs ZOS just 128 bits (16 bytes): their APIs reveal that they have a 64-bit field into which the various progress data is packed, and another 64-bit field for the timestamp. Multiply that by 3219 achievements (the highest valid achievement ID this patch), and we're looking at around 50KB of achievement data per character.

The impact on the client is trivial, but that also suggests a somewhat limited impact for leaving it on the server.

you are neglecting it's database stuff.

One 4k movie is a lot less of an issue than equivalent byte database table with super small data per entry...

a 10GB database with 300M 32byte entries will behave quite different than a 10GB one with 300k 32kbyte entries.

and you forgot to count charid plus achieveid in your space sumup.

We don't know how things work exactly on the backend, but there's no evidence to suggest that all of the characters' achievements in the game are just dumped into a flat database.

Perhaps not a flat database, but character loads do seem to take longer on older characters with more achievements (and other progression) than brand new character loads. If both had a fixed width binary object with all the data connected to the main character record, they should theoretically take the same amount of time to load.

Also, remember what they said: "By reducing the amount of data we have to store, the database doesn’t have to work as hard so the time it takes to search...." (emphasis mine).

This means that achievements must be in a separate data store that, at the very least, has to do a lookup. Even if it's just by character id and not achievement id too, switching to a lookup by account id could still cut the index size significantly.

code65536 wrote: »

And if they did drop the ball, then fixing that would yield far more dividends than whatever they're doing in this patch.

Yes, but could fixing that be done within the time constraints they seem to be operating under?

code65536 wrote: »

Accessing achievement information does not incur any server hits (it would simply be impossible for me to query all 3219 achievements in a fraction of my ping time if any part of that required a server hit), which means that all of the achievement data is transferred to the client when the character is loaded

Agreed. The client is interacting with a cache.

code65536 wrote: »

all achievement queries happen locally on the client. Which would suggest that achievements are stored as a blob and not individual database entries.

Not sure I follow. How is the fact that there is a cache indicative in any way of the backing database architecture?

code65536 wrote: »

I have been skeptical--and remain so--of any purported performance benefits of AwA. After all, if there really was a significant gain, wouldn't ZOS have stated that from the get-go instead of pulling the performance card late in the game only in the face of controversy?

They never promised any performance benefits to the player. They threw in some theoretical "may" verbiage about load screens and database-heavy operations, but almost as a side-effect, not their motivation for the change. They called out that the "most important" bit was preserving their ability to add "more content and achievements in the future".

They're probably trying to improve performance impacts for themselves, not for us. Like, for example, players logging in and out of toons for daily writs and random normal dungeons. If this loading is causing a bandwidth issue, I understand what they are trying to do, but it's really a fundamental game design issue that encourages this behavior in first place, and their "solution" is just awful decision making.

I'm not going to white-night their solution, but there are a lot of factors that go into such decisions, and we are missing most of that information. Without all the facts, it's really hard to say that this wasn't the best of many bad options. I reserve judgement on the technical side of things, even if I can think of many ways to address the technical side differently.

Still, the end result on the community is heartbreaking, and it deserves some introspection from ZOS to figure out how they arrived at this point. Even if all their decisions were the best they could have made under the circumstances, perhaps there's something they can do as a company to ensure similar circumstances don't lead to future no-win scenarios like this.

I really wish that there had been someone in power that had insisted that ripping out character progression tracking and journal was a red line not to cross - for the identity of the game if nothing else - and they needed to shift project timelines or content releases or whatever else was needed to regroup and find a different technical solution. If that's the sort of thing that just can't happen for whatever reason, maybe the best question for them to ask at this point is, why not?

Jaraal

I don't have an EU account, so I'm wondering if any EU folks have checked to see what effect the two day PTS maintenance has on account wide achievements? Did they roll it back, fix the bugged quests, fix the map autocompletion issues, etc?

Tandor

silvereyes wrote: »

<snip> Still, the end result on the community is heartbreaking, and it deserves some introspection from ZOS to figure out how they arrived at this point. Even if all their decisions were the best they could have made under the circumstances, perhaps there's something they can do as a company to ensure similar circumstances don't lead to future no-win scenarios like this.

I really wish that there had been someone in power that had insisted that ripping out character progression tracking and journal was a red line not to cross - for the identity of the game if nothing else - and they needed to shift project timelines or content releases or whatever else was needed to regroup and find a different technical solution. If that's the sort of thing that just can't happen for whatever reason, maybe the best question for them to ask at this point is, why not? <snip>

Agreed, What I also hope a period of introspection by ZOS might cover is the wisdom, should such circumstances unfortunately ever arise again, of battening down the hatches of communication while the community's unrest grew exponentially with an inevitable mixture of bewilderment and frustration.

tmbrinks

silvereyes wrote: »

I'm not going to white-knight the decisions that have been made. I will just say that I don't personally see any reason to second-guess

Saieden wrote: »

tmbrinks wrote: »

DarcyMardin wrote: »

code65536 wrote: »

For people who are looking for a reliable way to back up and save for posterity your character-specific data before Monday:

LibAchievementsArchive

@code65536 , thanks so much for making this library. One question -- I have more than one account. Can I use this library to store the achievement data for all my characters across all 4 accounts, or should I somehow separate them? Thanks!

At quick glance into the saved variables file, it saves Server/account/character name, so you can use it for all your accounts on either server.

(But Code can confirm)

After logging through all 18 characters... the stored data (in this addon)



131 KB... I use like 50x that data to stream a single song to my phone over a wireless network

I struggle mightily to understand how this is going to save the game and fix performance issues. But I'm not a database engineer, just somebody who understands numbers and relative size.

Also this is in lua, which is basically plain text. A database can store this in a fraction of the space, with redundancy.

silvereyes wrote: »

Troodon80 wrote: »

code65536 wrote: »

On the server side of things, each achievement costs ZOS just 128 bits (16 bytes): their APIs reveal that they have a 64-bit field into which the various progress data is packed, and another 64-bit field for the timestamp. Multiply that by 3219 achievements (the highest valid achievement ID this patch), and we're looking at around 50kB of achievement data per character.

Just to go into this in further detail, because I also wanted to address this:

Jaraal wrote: »

Thanks for the explanation!

And their servers must be on life support if scavenging just 50kb per character is going to make or break performant-ness!

50kB should be considered a worst-case scenario. If what is being said is an accurate representation of how ZOS is dealing with the data on the server side (and… honestly, who knows)… then this is less than 1MB per account (assuming maximum of 18 characters being 900kB), and if we assume 1MB (rounding up)… now multiply that by all the accounts and work out an average. ZOS's last statement about accounts was somewhere just over 19 million, collectively, if I recall.

This is… 19TB (again, rounding up here, 900kB would end up being more around 17TB). That's immense and, quite frankly, nonsensical.

It really isn't. Not for a data-driven software service catering to a theoretical 19 million people (which, let's be honest, most are going to be in cold storage)

code65536 wrote: »

remosito wrote: »

aaaa

code65536 wrote: »

Jaraal wrote: »

I also find it amusing that instead of data being eliminated, it will simply be transferred to addons. Will that negate any potential performance benefits? I’m not a professional coder, so I’m not sure how that works. Does it only impact the client, or do the data calls add to the server load?

It's effectively transferring the work from the server to the client. That said, any performance impact is negligible. While writing LibAchievementsArchive, I had taken some measurements to see how long it took to fully back up a character's achievement data...

The entire process of reading, processing, and encoding all of the data for all 3219 achievements for a character (which includes associated progress data; e.g., "Humanoid Slayer" is a single achievement with three separate requirements of killing 1000 humanoids, 500 goblins, and 100 giants) took something on the order of 30ms (0.03s).

On the server side of things, each achievement costs ZOS just 128 bits (16 bytes): their APIs reveal that they have a 64-bit field into which the various progress data is packed, and another 64-bit field for the timestamp. Multiply that by 3219 achievements (the highest valid achievement ID this patch), and we're looking at around 50KB of achievement data per character.

The impact on the client is trivial, but that also suggests a somewhat limited impact for leaving it on the server.

you are neglecting it's database stuff.

One 4k movie is a lot less of an issue than equivalent byte database table with super small data per entry...

a 10GB database with 300M 32byte entries will behave quite different than a 10GB one with 300k 32kbyte entries.

and you forgot to count charid plus achieveid in your space sumup.

We don't know how things work exactly on the backend, but there's no evidence to suggest that all of the characters' achievements in the game are just dumped into a flat database.

Perhaps not a flat database, but character loads do seem to take longer on older characters with more achievements (and other progression) than brand new character loads. If both had a fixed width binary object with all the data connected to the main character record, they should theoretically take the same amount of time to load.

Also, remember what they said: "By reducing the amount of data we have to store, the database doesn’t have to work as hard so the time it takes to search...." (emphasis mine).

This means that achievements must be in a separate data store that, at the very least, has to do a lookup. Even if it's just by character id and not achievement id too, switching to a lookup by account id could still cut the index size significantly.

code65536 wrote: »

And if they did drop the ball, then fixing that would yield far more dividends than whatever they're doing in this patch.

Yes, but could fixing that be done within the time constraints they seem to be operating under?

code65536 wrote: »

Accessing achievement information does not incur any server hits (it would simply be impossible for me to query all 3219 achievements in a fraction of my ping time if any part of that required a server hit), which means that all of the achievement data is transferred to the client when the character is loaded

Agreed. The client is interacting with a cache.

code65536 wrote: »

all achievement queries happen locally on the client. Which would suggest that achievements are stored as a blob and not individual database entries.

Not sure I follow. How is the fact that there is a cache indicative in any way of the backing database architecture?

code65536 wrote: »

I have been skeptical--and remain so--of any purported performance benefits of AwA. After all, if there really was a significant gain, wouldn't ZOS have stated that from the get-go instead of pulling the performance card late in the game only in the face of controversy?

They never promised any performance benefits to the player. They threw in some theoretical "may" verbiage about load screens and database-heavy operations, but almost as a side-effect, not their motivation for the change. They called out that the "most important" bit was preserving their ability to add "more content and achievements in the future".

They're probably trying to improve performance impacts for themselves, not for us. Like, for example, players logging in and out of toons for daily writs and random normal dungeons. If this loading is causing a bandwidth issue, I understand what they are trying to do, but it's really a fundamental game design issue that encourages this behavior in first place, and their "solution" is just awful decision making.

I'm not going to white-night their solution, but there are a lot of factors that go into such decisions, and we are missing most of that information. Without all the facts, it's really hard to say that this wasn't the best of many bad options. I reserve judgement on the technical side of things, even if I can think of many ways to address the technical side differently.

Still, the end result on the community is heartbreaking, and it deserves some introspection from ZOS to figure out how they arrived at this point. Even if all their decisions were the best they could have made under the circumstances, perhaps there's something they can do as a company to ensure similar circumstances don't lead to future no-win scenarios like this.

I really wish that there had been someone in power that had insisted that ripping out character progression tracking and journal was a red line not to cross - for the identity of the game if nothing else - and they needed to shift project timelines or content releases or whatever else was needed to regroup and find a different technical solution. If that's the sort of thing that just can't happen for whatever reason, maybe the best question for them to ask at this point is, why not?

That's why we've asked (repeatedly) for that information. For a justification. For somebody to explain why.

We got one tiny squeak (from the Q&A), then nothing. Radio silence.

And then to still be told "we're working on our communication"...

It's proof that the word "communication" was used as buzzword and there was not really any intention on doing so. And it's why I've pulled my financial support from this game after supporting it continuously (and on multiple accounts) for years.

I'm not naïve, so I know that I'm nothing more than a line in a database (irony?) with an account number and a credit card number attached to it to a large company like this. It that I feel like we were given meaningless lip service, that has left this bitter taste in my mouth. Because they never really cared about us, the players.

RaddlemanNumber7

Jaraal wrote: »

I don't have an EU account, so I'm wondering if any EU folks have checked to see what effect the two day PTS maintenance has on account wide achievements? Did they roll it back, fix the bugged quests, fix the map autocompletion issues, etc?

I tried logging in. On my first attempt it took so long my connection timed out "due to inactivity".

Once I got in I had a brief look around but couldn't see anything that had changed. Characters that have done Kari's Hit List on live are still shown as not having done it on PTS. I gave up at that point.