Multicraft Refining - Small Batches Test, 10K mats

VaranisArano

Yes, its another Multicraft Refining thread, but hey, at least this one has data!

Goal
To record data related to refining Small Batches with Multicraft, and to encourage any others who want to contribute.

Context
The general debate on the forum is whether or not Multicraft lowered the drop rate for improvement mats since Scalebreaker launched. Since launch, players have complained of lower drop rates. On the other hand, the players who tested it on PTS and Live in large quanities have found no significant change.

I'm particularly indebted to @tmbrinks for their wonderful data set refining materials in bulk. This is an excellent resource with a huge data set, that supports the claim that ZOS has not changed the drop rate for refining.
tmbrinks' data: https://docs.google.com/spreadsheets/d/1yQDUjYNQVsIFl0ktkbkSlfYPkzP6pgCOTMQ-qCuzfaI/edit#gid=1078240196

What has not been tested and recorded is Multicraft refining in smaller batches. This is something players do regularly, but usually don't record or aggregate their data. That means the results of "Small Batch Refining" is very vulnerable to extremes of RNG, confirmation bias, and the human tendency to remember the bad results instead of the average or good results. The majority of complaints about Multicrafting have focused on refining small batches of a few thousand or less, so I decided to focus my testing on small batches, rather than replicate the above data for large bulk refinements. (Also, I was out of raw mats and don't have unlimited funds to buy tons and tons of mats for an experiment, so that's why I went with the limited sample size I did.)

Hypothesis
The larger the sample size, the closer the results should be to the statistical mean.
Therefore, if Multicraft did not change the drop rate in small batches, my results from bulk refining should be consistent with the accepted means for various improvement mats, i.e. 1 in 200 for gold quality mats.
However, if Multicraft did lower the drop rate in small batches, my results should be significantly lower than the accepted mean.

Procedure & Caveats
My Procedure

1. I bought 10,000 raw ruby ash.
2. I refined it in 10 batches of 1000 at a time, recording mats, Rosins, Mastic, Turpen, and Pitch for each batch.

Caveats
10 batches of 1,000 ruby ash seems like a lot, but in reality its a drop in the bucket compared to the amount of refining that goes on throughout all the servers. Even compared to tmbrinks' excellent data above, you can tell its a small number. This is a start of a data set, not something that's meant to be fully conclusive. As you see, my results are NOT going to be representative of everyone who's used Multicraft to refine small batches.

Data
Total Raw Mats: 10,000 ruby ash
Rosin: 62 (0.62%)
Mastic: 77 (0.77%)
Turpen: 115 (1.15%)
Pitch: 155 (1.55%)
Sanded Ruby Ash: 8,448

Full data sets for each batch of 1,000 mats.
Batch 1: 5R, 10M, 15T, 11P, 835 sanded
Batch 2: 3R, 9M, 11T, 12P, 856 sanded
Batch 3: 3R, 9M, 13T, 11P, 835 sanded
Batch 4: 8R, 7M, 6T, 18P, 835 sanded
Batch 5: 6R, 4M, 12T, 20P, 840 sanded
Batch 6: 7R, 6M, 7T, 17P, 847 sanded
Batch 7: 8R, 11M, 12T, 17P, 847 sanded
Batch 8: 7R, 10M, 17T, 15P, 848 sanded
Batch 9: 8R, 8M, 11T, 10P, 832 sanded
Batch 10: 7R, 3M, 11T, 24P, 858 sanded

Results
I'm going to focus in on Rosin because that's the most valuable item most players focus on.

1. The Mean for Rosin
Expected Mean: 0.5% or 1 in 200
tmbrinks' bulk refining mean: 0.49%
My mean: 0.62%

2. Range for Rosin
In 10 batches of 1,000 mats, my range was 5, going from 3 to 8 rosin per patch.

3. Median for Rosin
In 10 batches of 1,000 mats, my median was 7 rosin.

Conclusions
In 10 batches of 1,000 raw mats, my drop rate for gold quality Rosin exceeded the expected drop rate of 1 in 200. The expected result was 50 rosin; I received 62 rosin. From that result I draw several conclusions:

1. I did not see a significant lowering of the drop rate for any mat in my data set that would be expected given complaints on the forums. My Turpen was a little low, but the more valuable Rosin was significantly higher than the expected value. That's not terribly surprising given the small data set, but it argues against the hypothesis that ZOS significantly lowered the drop rate on gold improvement mats for players refining in small batches. Rather, it makes it more likely that small batch refinements that fall lower than the expected mean are still within the accepted RNG for refinement. (That doesn't make it suck less for disappointed players, but at least you can hopefully be a little more confident that ZOS isn't messing with the numbers.)

2. I got lucky with my higher than expected number of Rosin, and that's not surprising given the small sample size. If I continued to buy and refine more mats in small batches of 1000, over time my results would continue to approach the expected mean of 1 in 200. Remember, even 10,000 mats is a small amount, statistically speaking. So over time, I'd expect to see a wider Range of results, and for my results to skew a little lower as I got closer to the expected mean.

3. Due to my small sample size of only 10 batches, my Range is particularly limited! From other threads, we know its entirely possible to get less than 3 gold improvement mats per 1000 raw mats. That's entirely predictable and if I had a larger sample size, I'm confident that I would have a bigger range...and that as with #2, the results would approach the expected mean as the sample got larger.

What to do now?
Having bought 10K mats to test, I'm not sure I'm going to continue adding to this. I'm also well aware that other players aren't going to have the same results as me - certainly there are enough complaint threads to prove that. If you want to post your results to this thread, I'd be happy to hear them...with some instructions.

So here's my challenge: I welcome your own small batch testing! But there's a procedure to follow!
1. Please record your total mats refined along with your results.
2. Please record ALL your refining batches. Don't just post your bad ones, or your good ones. We can't get a good average if you cherrypick!

TLDR
I refined 10k mats in batches of 1000 mats each with Multicraft, and got 62 gold improvement mats, when the expected value was only 50 Rosin. I didn't see any evidence to support the complaints that ZOS lowered the drop rates of gold improvement mats with Multicraft.

Lylith

i wanted to see how close to your results i would come doing the same procedure.

i'm not going to list all the raw data, but the bottom lines are as follows.

rosin - 43

masti - 81

turpi - 123

pitch -139

wood - 8487

range 1 - 7

median 4

tmbrinks

Both data sets are indicative that nothing has changed, especially when combined

Lylith

hopefully, others' experiences will be less representative of the low end of zo$ rng.

tmbrinks

Lylith wrote: »

hopefully, others' experiences will be less representative of the low end of zo$ rng.

I agree. Your results puts you at the 18th percentile in terms of what you'd expect (If we assume the drop rate of 0.5%). Unlucky, yes, but not horrendous.

I'm actually at the 2nd percentile in my refining of tempering alloys from blacksmithing materials across the nearly 250k refines I've done, so I've had some really, really poor luck

(Varanis' was on the opposite end, his results were in the 96th percentile!)

SpAEkus

Hey I am coming up on some bulk refining after catching up on some surveys.

You didn't mention crafting passives.

Are you at max on Deconstruction Passives? Any others turned on (even if they don't affect Refining rewards)?

Just so we also all use the same parameters as yours.

VaranisArano

SpAEkus wrote: »

Hey I am coming up on some bulk refining after catching up on some surveys.

You didn't mention crafting passives.

Are you at max on Deconstruction Passives? Any others turned on (even if they don't affect Refining rewards)?

Just so we also all use the same parameters as yours.

Oops! I forgot. I'm doing this on my main crafter, a fully leveled character at max crafting skill and maxed refining passives; the same character I used to refine before Scalebreaker. The other passives don't impact refining, but the only ones I don't have are the research ones because I finished all woodworking research.

If you were doing it with less refining passives, you would naturally see a much lower drop rate for improvement mats than the expected value.

However, I think its reasonable to assume that the vast majority of folks who had complained about seeing a lower drop rate were also refining with full refining passives, and on the same character they used prior to the Scalebreaker patch. Otherwise, it would seen foolish to complain.

SpAEkus

Thanks, I figured as much.

Also as long as we do 1000 refine batches, do they need to be consecutive for 10000?

Does the RNG get too far apart if we don't do 10000 in a single back-to back to match your own run?

VaranisArano

SpAEkus wrote: »

Thanks, I figured as much.

Also as long as we do 1000 refine batches, do they need to be consecutive for 10000?

Does the RNG get too far apart if we don't do 10000 in a single back-to back to match your own run?

The RNG on one batch doesn't somehow roll over and impact the next one, so it doesnt matter if you do the tests back to back or you spread them out. Timing shouldn't matter because the RNG drop rate doesn't change, just the result. (I realize there are some snake oil ideas about "refine time" and such, but I've never seen any evidence those work.) so timing doesnt need to be controlled for because timing doesn't impact your RNG. That would be a whole different experiment if we thought it actually did.

Even 1000 mats was somewhat arbitrarily picked, largely because its an even number, the expected drop rate is easily to figure out, and I've seen others complaining about the low drop rate give examples of 1k mats.

Similarly, 10k was just the number of woodworking mats I decided to buy. Large enough to give a good starting data set, but not large enough to break my bank. The purpose isnt to test the drop rate for 10k mats because that's a pretty small sample size. Rather, I hope to keep adding "small batches" to that total.

Its great to see more data, because now we're up to 105 rosin / 18,487 mats in small batches = 0.57% (down from my absurdly lucky result of 0.62%, compared to the expected mean of 0.5%). That's still well over the expected mean, but we can already see how more mats refined means we're getting closer to the expected mean of 0.5%.

The number of batches of 1000 you decide to refine doesnt matter as much because through this thread, we can aggregate from the new total to check the drop rate. My #2 point was more because its fine if someone wants to only refine like 2k mats and post that result, but not okay if someone refines 4k worth and only reports the worst batch here because that would skew the data set.

The variable being tested was the "small batches" to better replicate how the folks complaining about the lowered drop rates were refining their mats. We already knew that the drop rate didn't change for large amounts of refining, thanks to tmbrinks' data. Stuff like timing and how many batches you do won't change the drop rate, since timing doesnt impact it at all and how many batches only changes the size of the sample size.

tmbrinks

VaranisArano wrote: »

Its great to see more data, because now we're up to 105 rosin / 18,487 mats in small batches = 0.57% (down from my absurdly lucky result of 0.62%, compared to the expected mean of 0.5%). That's still well over the expected mean, but we can already see how more mats refined means we're getting closer to the expected mean of 0.5%.

Not to nitpick.. pretty sure they refined 10,000 materials and got 8,487 ruby ash out of it. So we're at 105/20,000 for these samples. 0.525%

VaranisArano

tmbrinks wrote: »

VaranisArano wrote: »

Its great to see more data, because now we're up to 105 rosin / 18,487 mats in small batches = 0.57% (down from my absurdly lucky result of 0.62%, compared to the expected mean of 0.5%). That's still well over the expected mean, but we can already see how more mats refined means we're getting closer to the expected mean of 0.5%.

Not to nitpick.. pretty sure they refined 10,000 materials and got 8,487 ruby ash out of it. So we're at 105/20,000 for these samples. 0.525%

Oh, then I misread! Thanks for the correction

SpAEkus

Okay finally caught up on Surveys to have some 1000 batches (added some CopperDust just because):

Sorry About the formatting, my pers webpage is down, can't just post a pic.

All Tempers are Gold-Purple-Blue-Green

PDUST - CH ZI IR TI oz
BATCH - 2 10 12 17 840
BATCH - 6 8 13 13 837
BATCH - 6 6 9 12 859
BATCH - 5 6 11 14 864

COPPERD - CH ZI IR TI oz
210 - 2 3 1 4 183

ASILK - W E EM H SILK
BATCH - 4 10 14 15 850
BATCH - 6 5 6 13 851
BATCH - 4 7 9 18 828

RHIDE - W E E H HIDE
BATCH - 6 11 18 14 851
BATCH - 6 6 13 17 840

RASH - R M T P ASH
BATCH - 3 8 8 20 834
BATCH - 3 9 11 9 848
BATCH - 6 5 8 16 830
BATCH - 5 10 13 11 857
BATCH - 8 6 16 20 845

RORE - A S O S INGOT
BATCH - 2 7 10 21 843
BATCH - 4 11 10 22 838
BATCH - 2 10 15 15 855
BATCH - 8 5 11 15 856
BATCH - 2 1 10 18 851
BATCH - 5 6 12 16 832