Heat without light

Hypothesis testing pertaining to game mechanics is mostly a waste of time because the kinds of questions asked are mostly a waste of time.

For example, on the BG forums I found a proposal to determine whether the critical hit rate bonus on Senjuinrikio increases the critical hit rate of the first hit of Blade: Jin, involving testing on "too weak" mobs so that the mob dies on the first hit of the weapon skill. Two sets of data, one using Mamushito +1 (DMG 38), the other using Senjuinrikio (also DMG 38), are to be produced.

Never mind how tedious it would be to carry out such an experiment. Never mind the widely accepted conventional wisdom that Senjuinrikio's crit bonus does affect Blade: Jin. Let's pretend a formal statistical test is actually worth using.

Then the concern here is what sample size (the same for each set of data) is "sufficient" enough to be able to detect the effect of Senjuinrikio on Blade: Jin? To answer that question, it might help to do some prospective power computations for a test of two independent proportions (under the Neyman-Pearson paradigm of hypothesis testing). In other words, given a sample size that is the same for each group, what is the probability of rejecting the null hypothesis (Senjuinrikio has no effect) when the null hypothesis is false (so that you are inclined instead to accept that Senji has an effect)?

When determining a "suitable" sample size, one practical concern is that the variance of a (known) binomial proportion is a function of both the actual value of the proportion and n, the number of trials, and that when the number of trials is fixed, the variance is maximized for p = .5.

So not only can you control the sample size for your experiment, you can also attempt to keep your "base" critical hit rate as far away from 50% as possible in your attempt to achieve adequate statistical power. If you want to try to keep your sample size low (relatively speaking), and you know people willing to be your bitch on demand, you could try to conduct these tests in Salvage and, hell, throw in some Stumbling Sandals, too. (I honestly don't know what kind of critical hit rates you can expect in Salvage, though.)

Assuming that the Senji does actually have an effect on Blade: Jin, power curves of this two-sample test (Fisher's exact test in this case) for various null proportions (base critical hit rates) show that the power is lower for higher null proportions:


Let's say you'll accept a power of .80 given a Type I error of .05 ("false positive"). If your base critical hit rate is 20%, you'll need a sample size of 639 for each sample proportion. But if your base critical hit rate is 5%, you'll need a sample size of 278.

Of course, you can always accept a higher Type I error in exchange for lower sample sizes for a given level of power:


Given a Type I error of .10, if your base critical hit rate is 20%, you'll need a sample size of 475 for each sample proportion to achieve a power of .80. But, if your base critical hit rate is 5%, you'll need a sample size of "just" 211.

Again, this is all under the assumption that Senji does have an effect on Blade: Jin in the way that we expect.

Let us suppose it is possible to achieve a base critical hit rate of 5% and, therefore, assume that the critical hit rate with Senji is 11%. Assuming this, here is some R code that estimates the power of the test of two proportions from 1,000 simulated experiments (as expected, Fisher's exact test will correctly reject the null hypothesis about 80% of the time), using n = 211 for each group:

p_value = rep(0,1000)
n = 211

for (i in 1:1000) {
  a = rbinom(1,n,.05)
  b = rbinom(1,n,.11)
  c_table = matrix(c(a,n-a,b,n-b),nrow=2)
  p_value[i] = fisher.test(c_table,alternative="less")$p.value

}

power = sum(p_value < .1)/1000
power