Question about developer sampling

tinythinker

A few questions for ZOS that might help players better understand developer decisions such as those coming up with the next big update:

1. Do you ever look at subsets and sub-sub-sets when trying to gauge the impact of changes?

In other words, is it just aggregate by instance (average time to completion for a dungeons for all groups, average DPS in a trial for all groups, average numbers of deaths for all players, etc.) or by large categories (average number of deaths and kills per hour by class in Cyrodiil, average heals per second or damage per second by class in all dungeons/trials, etc.)?

Or is their a finer grain? Such as looking at the kinds of things above divided up by something like average number of hours players per week (1-10, 11-20, 30+) or by key stats (unbuffed highest power, spell or weapon, say from 1-500, 501-1000, 1001-1500, 1501-2000, 2001-2500, 2500-3000, 3000+)? In other words, when you say you are satisfied players are adapting or successful by talking generically about looking at your data, how much detail is involved? If you do use these kinds of subsets, are there key subsets you target for specific content, such as those at 2000 and above for weapon/spellpower in trials?


2. Do you ever focus on a specific sampling of individual players meeting certain demographic categories?

Kind of like the first question, but tracking several individual players for a period of time to get beyond "averages" and see direct, individual impact? If so, is this limited to people who stream or post videos and if not, how does that work? Other than posting on the forums or reddit, how can individual players help you get a better sense of what is working? Do you have an over-represented or under-represented population in such sampling (i.e. skill level, gear/stat level, number of hours per week played, etc.)?


3. What are your biggest limitations in your ability to track trends and assess how well changes are working?

And, is there anything you are working on to get better at it?

@ZOS_JessicaFolsom @ZOS_GinaBruno, do you know who might be able to talk about this? If so, might you be able to grab them for an ESO Live?