Give me data or give me...

I will consider this my last post on the topic of magic accuracy/magic hit rates. Again, most of the credit should go to lodeguy for all of his time-consuming experiments and insights, much of which I elided in the interest of saving (my) time. I don't fancy myself a gatekeeper of knowledge (anyone who knows Japanese may want to review his posts in their entirety, since I cannot read Japanese for the most part) but two years on it's about time someone English-speaking talked about this stuff in some detail. Yet I thought writing about what someone else already figured out would be fairly straightforward...

You may review my first four posts on this topic:

• On magic resist rates (Dec 17)
• More on magic resist rates (Dec 18)
• Describing "magic hit rate" symbolically (Dec 19)
• Even more on magic resist rates (Dec 29)

First off, I'd like to address my use of the terminology "magic hit rate" to make a distinction between "effective magic accuracy" (output) and "magic accuracy" the attribute (input). I am not a terribly big fan of the term "magic hit rate" that I've been using the last few weeks, especially since in the Japanese language, the term 魔法命中率 could be translated as either "magic hit rate" or "magic accuracy," so there is no distinction between "hit rate" and "accuracy" using the Japanese term. Instead of using "magic hit rate" as "short-hand" for "the probability of landing an unresisted magic spell," I could have used one of the following:

• effective magic accuracy
  
• rate of landing magic unresisted
  
• resist rate (as 1 minus the probability of landing an unresisted magic spell)
  

The first term seems the best since 1 point of magic accuracy input does not always yield 1 point of magic accuracy output, as shown previously, so I will use that term from here on. lodeguy himself used the term ヒット (hit) for an unresisted spell, so that is one reason I just adopted the terminology "magic hit rate" to start.

The second term is more awkward than "magic hit rate" and the third necessitates the use of negative language ("reducing resist rate") and making a distinction between different levels of resists (so that "resist rate" is understood as a catch-all for all types of resists). So that was supposed to be some kind of excuse for me using the term "magic hit rate" throughout.

Anyway, I mainly want to address whether a 1-point increase in elemental magic skill is equivalent to a 0.9% increase in effective magic accuracy above the 200 magic skill level.

This claim has endured as long as it has because partly because of the intuitive appeal inherent in the notion that magic accuracy is supposed to be analogous to melee accuracy. Perhaps it was to trump up the value of pure magic accuracy as opposed to specific magic skill. (All things being equal, which rarely occurs, magic accuracy does have appeal as a general attribute, a catch-all for all types of magic.)

Yet with no way to verify easily what one's effective magic accuracy is, there was no convenient way to refute or confirm that claim (among many, many other claims). But lodeguy did all the inconvenient work for you, and it was sitting under my nose. And I can provide some additional cover for lodeguy.

First, let's re-examine one of lodeguy's data sets.

Casting Water magic (103 INT, Neptune's Staff) on a level 78 Earth Elemental at various levels of elemental magic skill, he obtained the following results (11,934 trials):

Skill	No resist	1/2 resist	1/4 resist	1/8 resist
235	1960 (.532)	967 (.262)	409 (.111)	348 (.094)
240	1294 (.582)	563 (.253)	229 (.103)	136 (.061)
250	1390 (.694)	407 (.203)	142 (.071)	65 (.032)
262	1585 (.821)	287 (.149)	43 (.022)	15 (.008)
270	1858 (.887)	204 (.097)	30 (.014)	2 (.001)

When I looked at this data set, it was to establish the increase in effective magic accuracy, above 50% effective magic accuracy, for every 1-point increase in elemental magic skill. For some inexplicable reason, I used only the bottom three rows of the above table when fitting the linear probability model, obtaining the following results:

           Criteria For Assessing Goodness Of Fit

Criterion                 DF           Value        Value/DF

Deviance                   1          1.1684          1.1684
Scaled Deviance            1          1.1684          1.1684
Pearson Chi-Square         1          1.1610          1.1610
Scaled Pearson X2          1          1.1610          1.1610
Log Likelihood                    -2878.9070


Algorithm converged.


                      Analysis Of Parameter Estimates

                         Standard     Wald 95% Confidence       Chi-
Parameter    DF    Estimate       Error           Limits            Square    Pr > ChiSq

Intercept     1     -1.7053      0.1604     -2.0197     -1.3909     113.03        <.0001
skill         1      0.0096      0.0006      0.0084      0.0108     249.35        <.0001

What I should've done instead was use all the data, in which case I obtain the following results:

           Criteria For Assessing Goodness Of Fit

Criterion                 DF           Value        Value/DF

Deviance                   3          2.4983          0.8328
Scaled Deviance            3          2.4983          0.8328
Pearson Chi-Square         3          2.4871          0.8290
Scaled Pearson X2          3          2.4871          0.8290
Log Likelihood                    -6935.4539


Algorithm converged.


                      Analysis Of Parameter Estimates

                         Standard     Wald 95% Confidence       Chi-
Parameter    DF    Estimate       Error           Limits            Square    Pr > ChiSq

Intercept     1     -1.8719      0.0683     -2.0056     -1.7381     751.90        <.0001
skill         1      0.0102      0.0003      0.0097      0.0108    1458.65        <.0001

While the first 95% confidence interval covered .009, the last 95% confidence interval, which was generated considering all the data at hand, does not cover .009 (effective magic accuracy increase of 0.9% for every one-point increase in elemental magic skill), so it's a pretty safe bet that above 200 elemental magic skill, 1 point of elemental magic skill is equivalent to effective magic accuracy higher than 0.9%.

To visualize how good the model fit is, here's some graph-junk for you:



Conclusion: There is scant reason to believe that 1 point of elemental magic skill above the 200 level yields only a 0.9% increase in effective magic accuracy (unless lodeguy and I were extremely unlucky). You might as well treat it as a 1% increase! Perhaps this is not the case for other types of magic skill, and perhaps there is some funny business above the 300 level, but at least here is some conclusive evidence for the range of elemental magic skill considered.

The following is supposed to be the extra "cover" for lodeguy's results (they can stand on their own though), and again is mainly for my own amusement.

Finally, using the exact same sample-size allocation and levels of elemental magic skill that lodeguy used for this particular experiment, I can generate approximate sampling distributions for the mean change in effective magic accuracy (magic hit rate) per one-point increase in elemental magic skill.

First off, I generated a sampling distribution assuming +0.9% effective magic accuracy per +1 elemental magic skill. Here, I assumed that the effective accuracy at 240 elemental magic skill was exactly 53%, but it is the changes in elemental magic skill that really matter:

(The approximate normal distribution is drawn with a red curve, and the histogram uses the data generated from simulation.)

As you can see, under the assumption of +0.9% effective magic accuracy per +1 elemental magic skill, observing (as a point estimate) an increase in effective magic accuracy of 1% or greater for any one experiment (given 11,934 trials...) is pretty rare (in the right tail). If you treat this assumption as a straw man to knock down (otherwise known as the null hypothesis), you will knock down the straw man (reject the null) with an approximate probability of .93 (given Type I error of .05) if the real (not estimated) accuracy increase is 1% for every 1-point increase in elemental skill. Of course, if the real increase is just 0.9%, the null will be rejected "only 5% of the time" (the Type I error of .05 that was fixed in advance of frequentist inference).

It may also be interesting to see what an approximate sampling distribution for the mean change in effective magic accuracy, assuming +1.0% effective magic accuracy per +1 elemental magic skill, would look like:

If the assumption (+1.0% effective m.acc per +1 elemental skill) is actually true, then observing an increase in effective magic accuracy of 0.9% (or less) should be extremely rare (see left tail), given 11,000+ samples.

That's a wrap for this topic.