(
Edit - Dec. 30: First image was fixed.)
(
Edit - 7:00 PM PST: I wrote the last section in a muddle and it makes no sense. It was amended.)
(
Edit - 5:00 AM PST:
Summary added.)
This post is a continuation of my discussion of
extensive data that a Japanese blogger collected for the purposes of investigating the relationship between "
magic hit rate"--
defined as a "lack of resist" rate for the purposes of my discussion unless otherwise stated--and each of several factors that are known to affect the accuracy of magic spells.
So far, I have gone over the possible relationship between magic hit rate and elemental staves and the relationship between hit rate and elemental magic skill.
You may view the "tentative" conclusions so far. (I say "tentative" because I will be the first to acknowledge the limited scope of the binary regression models that are the basis for making any conclusions.) I will continue to focus exclusively on the accuracy of direct-damage magic ("nukes") as opposed to other magic types (but I may get around to discussing enfeebling magic later).
The importance of checking for linearityFirst off, I just want to make a few comments regarding the (apparently) piecewise-linear relationship (which is plausible because it fits the data well, even if the author's procedure was more of an ad hoc one... not sure) between magic hit rate and elemental magic skill that the blogger described.
Yes, in the past I have said it may be feasible to estimate changes in "magic hit rate" with elemental magic skill by choosing two levels of elemental magic skill that are very far apart (and hope that your magic hit rate isn't capped before your higher level), and then perform some "regression" procedure, which is basically drawing a line through the two observed rates (sample proportions). If you fix the number of observations you will set out to collect, allocating your number of observations equally between the two levels will be the most efficient way to detect an effect (a statistical power rationale). Obviously, though, you can't even check for the linearity assumption (hence the term linear regression) since a line through two observed values is a perfect fit, and if the trend is
not linear overall, the validity of your point estimate is highly suspect.
As an example, I return to this data set (experimental conditions: Water magic on a lv78 Earth Elemental, using 78 INT and varying levels of elemental magic skill with a Neptune's Staff):
| Skill | No resist | 1/2 resist | 1/4 resist | 1/8 resist |
| 230 | 1233
(.380)
| 768
(.237)
| 499
(.154)
| 746
(.230)
|
| 240 | 832
(.434)
| 469
(.245)
| 245
(.128)
| 369
(.193)
|
| 250 | 1536
(.476)
| 826
(.256)
| 399
(.124)
| 468
(.145)
|
| 262 | 1000
(.598)
| 373
(.223)
| 163
(.097)
| 137
(.082)
|
| 270 | 1780
(.667)
| 600
(.225)
| 188
(.070)
| 99
(.037)
|
Performing a regression on this data yields the following results:
Criteria For Assessing Goodness Of Fit
Criterion DF Value Value/DF
Deviance 3 19.8838 6.6279
Pearson Chi-Square 3 19.8790 6.6263
Analysis Of Parameter Estimates
Standard Wald 95% Confidence
Parameter Estimate Error Limits
Intercept -1.2832 0.0719 -1.4242 -1.1422
Skill 0.0072 0.0003 0.0066 0.0077
As I said previously, obviously this model, which assumes a linear relationship between hit rate and skill
over the entire range of skill, is a poor fit to the data. The Japanese blogger was aware of this and proposed piecewise linearity. I suspect a failure to check for lack of fit is behind the
estimated hit rate increases described on wiki.ffo.jp for 1 point of elemental magic skill (.064), 1 point of INT/MND/CHR attribute (.074), and 1 point of magic accuracy (.072), although there is no source cited.
For your convenience, I have furnished a graph plotting the observed magic hit rates (sample proportions) versus elemental magic skill for the above data set, and plotted the linear probability model fit to show poorness of fit. I also drew 95% (exact) confidence intervals for the point estimates:
I did include a loglinear model fit mainly for my own amusement (not as bad a fit), but there is no reason to think "the dev team" would really use some kind of explicit loglinear relationship (much less some general logistic one) for anything in FFXI. So I lean toward piecewise linearity because that would be simple to implement, I think.
I must admit, however, that I'd like to see any trends beyond 270 elemental magic skill and below 230 elemental skill, but this thought only comes about because the blogger was perceptive enough to propose piecewise linearity. Is there even enough elemental skill equipment out there to test across a range of elemental skill broader than a 40-point range?
That said, we can proceed to examine the relationship between magic hit rate and INT, keeping in mind the perils of assuming "global" linearity.
Magic hit rate versus INTRefer to
the original post for specifics. (
There is more commentary, but the table doesn't make any sense to me.) The author endeavored to examine the relationship between magic hit rate versus INT for a wide range of ΔINT (his INT minus the target's INT). Did he suspect "global" nonlinearity with ΔINT to begin with, or did these results lead him to suspect a similar trend with elemental magic skill? You'll have to ask him.
Level 78 Earth Elementals appear to have 73 INT. He used some Water nuke with a Neptune's Staff with 262 elemental magic skill. The data is summarized as follows:
| ΔINT | No resist | 1/2 resist | 1/4 resist | 1/8 resist |
| -20 | 957
(.545)
| 439
(.250)
| 183
(.104)
| 176
(.100)
|
| -15 | 1000
(.598)
| 373
(.223)
| 163
(.097)
| 137
(.082)
|
| -10 | 637
(.653)
| 230
(.236)
| 73
(.075)
| 36
(.037)
|
| -5 | 870
(.678)
| 270
(.210)
| 101
(.078)
| 42
(.033)
|
| 0 | 886
(.721)
| 242
(.197)
| 71
(.058)
| 30
(.024)
|
| +10 | 1585
(.821)
| 287
(.149)
| 43
(.022)
| 15
(.008)
|
| +20 | 884
(.854)
| 129
(.125)
| 19
(.018)
| 3
(.003)
|
| +30 | 1387
(.909)
| 127
(.083)
| 9
(.006)
| 3
(.002)
|
I don't really feel like replicating graphs that the original author already created, so I will just show you the one he created plotting the data (with 68% confidence intervals) and his piecewise linear regression:
Perhaps the presence of the piecewise regression model fit influences your perception of the the trend. Still, it seems that INT appears somewhat less effective at high levels of ΔINT.
Thus, it is pretty obvious that assuming "global" linearity would yield a poor fit to the data, so the piecewise linear regression (whether the cutoff point is intuited or rigorously chosen) approach seems reasonable in order to estimate precisely the effect of a 1-point change in INT on magic hit rate (depending on the range of ΔINT).
And whether or not you use ordinary least-squares regression (which assumes a normal response, which a marginally binomial proportion is not) or a MLE method (GLM), the point and interval estimates of the slopes are pretty close anyway. The following uses MLE estimation for ΔINT between -20 and 10:
Criteria For Assessing Goodness Of Fit
Criterion DF Value Value/DF
Deviance 4 1.6641 0.4160
Pearson Chi-Square 4 1.6626 0.4157
Analysis Of Parameter Estimates
Standard Wald 95% Confidence
Parameter Estimate Error Limits
Intercept 0.7291 0.0051 0.7190 0.7391
dint 0.0090 0.0004 0.0082 0.0098
It seems that between ΔINT -20 and ΔINT 10, a 1-point increase in INT is expected to result in a 0.9% increase in magic hit rate. (
Aside: I dislike expressing changes in proportions--hit rate is a proportion--as percentages because they are often interpreted as increases by a factor of (1+[percent]/100), which is not what I mean. So that is why I usually lean toward expression of rates in decimal form... not that it really helps understanding all that much.)
Is .009 (0.9%) significantly different (statistically) than .01 (1%)? The 95% confidence interval bounding the true rate of change in magic hit rate happens to exclude .01 (1%), so yes. But of course, it's possible Type I error has manifested.
Finally, considering the range of ΔINT between 10 and 30:
Criteria For Assessing Goodness Of Fit
Criterion DF Value Value/DF
Deviance 1 0.8059 0.8059
Pearson Chi-Square 1 0.8156 0.8156
Analysis Of Parameter Estimates
Standard Wald 95% Confidence
Parameter Estimate Error Limits
Intercept 0.7741 0.0133 0.7482 0.8001
dint 0.0044 0.0006 0.0033 0.0056
It seems that between ΔINT 10 and ΔINT 30, a 1-point increase in INT is expected to result in a .0044 increase in magic hit rate. But we cannot distinguish between .0045 and .005 given the 95% confidence interval.
Observations: I am interested in what happens to the effect of INT below 50% magic hit rate. This could be achieved by removing the Neptune's Staff and repeating the experiment. (Have fun collecting 11,000 observations!) But
this data set (not mine) does not show any effect of INT+30 (is ΔINT after INT+30 still below 0 for an Elvaan mage versus an Ebony Pudding?) at 242 elemental magic skill. Is this because the "base" magic hit rate (whatever it is) is well below 50%?
Conclusion: Above 50% "base" magic hit rate (whatever it is), it appears that below ΔINT 10, 1 point of INT gives about a .01 increase (or .009 if you are a stickler for statistical significance) in magic hit rate for direct-magic damage, and above ΔINT 10, a .005 increase in magic hit rate.
I hope this result can be generalized to any kind of mob and also to MND and CHR.
Magic hit rate versus magic accuracyI don't see an in-depth examination of the effect of magic accuracy (from equipment). Early on, he seemed to have been trying to get a feel for things (see
the original post). I just see the following data pertaining specifically to magic accuracy:
For level 75 Qiqirn Archaeologists (Aydeewa Subterrane), using Stone magic, 82 INT, and 230 elemental magic skill, and no elemental staff (1,365 observations):
| Condition | No resist | 1/2 resist | 1/4 resist | 1/8 resist |
baseline
| 379
(.420)
| 194
(.215)
| 133
(.147)
| 196
(.217)
|
+10 m. acc
| 205
(.443)
| 124
(.268)
| 58
(.125)
| 76
(.164)
|
For level 75 Steelshells (The Boyahda Tree), using Stone magic, 82 INT, and 230 elemental magic skill, and no elemental staff (1,142 observations):
| Condition | No resist | 1/2 resist | 1/4 resist | 1/8 resist |
baseline
| 580
(.744)
| 141
(.181)
| 49
(.063)
| 10
(.013)
|
+10 m. acc
| 303
(.837)
| 49
(.135)
| 8
(.022)
| 2
(.006) |
Since Qiqirn are resistant to earth magic, there is a huge discrepancy in the magic hit rate of Stone between the two sets of trials.
For the Qiqirn trial, the magic accuracy effect is not
statistically significant, but that may just be a consequence of "small" sample sizes (poor statistical power to detect an effect size so small):
Analysis Of Parameter Estimates
Standard Wald 95% Confidence Chi-
Parameter Estimate Error Limits Square Pr > ChiSq
Intercept 0.4202 0.0164 0.3880 0.4524 653.65 <.0001
macc 0.0023 0.0028 -0.0033 0.0078 0.64 0.4254
For the Steelshell trial, the magic accuracy effect is highly, statistically significant, but the interval estimate is rather wide:
Analysis Of Parameter Estimates
Standard Wald 95% Confidence Chi-
Parameter Estimate Error Limits Square Pr > ChiSq
Intercept 0.7436 0.0156 0.7129 0.7742 2262.00 <.0001
macc 0.0093 0.0025 0.0045 0.0142 14.05 0.0002
Still, in light of what we know about the relationship between magic hit rate and each of the factors that have been investigated well above 50% magic hit rate and well below 50% magic hit rate (elemental staff and elemental magic skill), it seems reasonable to infer that 1 point of magic accuracy is equivalent to about 0.5% magic hit rate below 50% "base" magic hit rate and about 1.0% magic hit rate, at best, above 50% "base" magic hit rate. (The confidence interval is duly noted, but common sense dictates that the 1-point magic accuracy bonus is 1% hit rate at best.)
What the heck is "base" magic hit rate, and what evidence supports such an idea?I am speculating that "base" magic hit rate is the result of a calculation that compares your "magic accuracy" score before equipment and buffs (debuffs) to a mob's "magic evasion" score, which may be comprised of elemental resistance factors.
So far, the main purpose of making a distinction between a "base" magic hit rate and magic hit rate bonuses from equipment (and possibly buffs/debuffs) is that the bonuses from staves, elemental magic skill, and magic accuracy (and probably INT) are conditional on magic hit rate, based on the extensive data provided. And how do you go about determining the bonuses from equipment if the bonuses from equipment determine the "base" hit rate?
My initial thought was that if a "base" hit rate is below 50%, then any bonuses from equipment will be as I described previously, even if the actual hit rate ends up being above 50%. Again, speculation.
As far as evidence goes, here is one that contradicts what I just wrote.
Yet another post from our highly esteemed Japanese blogger illustrates the magic hit rate bonus from using a staff that is the same element as that of the magic being used. The data are summarized as follows:
For level 75 Qiqirn Archaeologists (Aydeewa Subterrane), using Stone magic, 82 INT, and 230 elemental magic skill, and no elemental staff (1,307 observations):
| Staff | No resist | 1/2 resist | 1/4 resist | 1/8 resist |
None
| 379
(.420)
| 194
(.215)
| 133
(.147)
| 196
(.217)
|
Terra's Staff
| 262
(.647)
| 81
(.200)
| 29
(.072)
| 33
(.081)
|
Note that the interval estimate of the magic hit rate
without staff (not shown) does not cover .50, so I am 95% confident the real hit rate is below .50. Furthermore, the interval estimate of the magic hit rate
with Terra's Staff (also not shown) does not cover .50 either, so I am 95% confident that the magic hit rate with Terra's Staff is well above .50.
Previously it was shown that a 95% confidence interval for the HQ staff effect "well"
below 50% hit rate was (.1359, .1665). Here, the point estimate for the staff bonus appears to be .227, but how precise is this estimate?
Analysis Of Parameter Estimates
Standard Wald 95% Confidence
Parameter Estimate Error Limits
Intercept 0.4202 0.0164 0.3880 0.4524
staff HQ 0.2267 0.0289 0.1701 0.2833
staff None 0.0000 0.0000 0.0000 0.0000
What's going on? First, one set of data showed that for magic hit rates above 50%, a HQ staff seemed to confer (what is thought to be) a constant 30% magic hit rate bonus (estimated). Then, another set of data showed that for magic hit rates below 50%, a HQ staff seemed to confer (what is thought to be) a constant 15% magic hit rate bonus (estimated). But the above 95% CI covers neither .15 nor .30.
So this data seems to undermine the idea of the "base" hit rate check I speculated about, unless a transition below 50% magic hit rate to above 50% magic hit rate (and vice versa) is handled by the game in a way that is difficult to observe. (Well, I have gone delirious at this point, so let me revisit this later.)
SummaryThe conclusions inferred from the data so far (
see my last post as well for a summary) rest on a few ideas and concessions that really warrant further examination:
- There are two distinct "regimes" of magic hit rate before any bonuses from equipment (and probably buffs/debuffs and food, etc.) that determine the magnitude of the accuracy bonuses from elemental magic skill, elemental staves, and magic accuracy (all from equipment).
- One region is below 50% magic hit rate
- The other region is above 50% magic hit rate
- We are assuming piecewise linearity to model the existence of the above phenomenon. Otherwise, some nonlinear relation (e.g. logistic) will result in more complex interpretations
- I acknowledge that only direct-magic damage ("nukes") was investigated. "Further examination" here means that we should look at other types of magic (enfeebling) to see if the conclusions for direct-magic damage can be generalized.
- I concede the possibility of weather/day possibly confounding the results. But these effects do not process 100% for the magic damage calculation, so if they also apply to magic accuracy, the effect is probably not 100% either (without obis). The effect may also be weak and hard to detect, if it even exists at all; if this is the case, it is not a serious confounding threat. (I don't see any data to corroborate this though. You can perform some regression diagnostics to check for omitted explanatory variables, too.)
However, even if the model described above is not exactly as SE designed magic accuracy/magic hit rate to work, it still is a model that seems to approximate well the "reality" of the situation (for nukes). It's not like I have a vested interest in promoting this view of magic hit rate bonuses. It is well within the realm of possibility that the data provide only a limited view of the whole situation.
That said, so far it appears (and I do emphasize that these are estimates) that, given the data so far:
- If the initial and final magic hit rates are both below 50%, then
- An HQ staff of the correct element gives a constant increase of 15% magic hit rate
- A NQ staff of the correct element gives a constant increase of 10% magic hit rate
- 1 point of elemental magic skill gives a constant increase of 0.5% magic hit rate
- 1 point of magic accuracy gives a constant increase of 0.5% magic hit rate (caveat being the evidence is not that strong)
- If the initial and final magic hit rates are both above 50%, then
- An HQ staff of the correct element gives a constant increase of 30% magic hit rate
- A NQ staff of the correct element gives a constant increase of 20% magic hit rate
- 1 point of elemental magic skill gives a constant increase of 1% magic hit rate
- 1 point of magic accuracy gives a constant increase of 1% magic hit rate (caveat being the evidence is not that strong)
"Open question": If 50% magic hit rate (before equipment or "base") really is a critical point, what happens to accuracy bonuses (or penalties) that cross this critical point?
Finally, we also saw that for INT, specifically ΔINT, the difference between your INT and your target's INT:
- If the initial and final magic hit rates are both above 50%, then
- Between ΔINT -20 and ΔINT 10, 1 point of INT gives a constant increase of 1% magic hit rate
- Between ΔINT 10 and ΔINT 30, 1 point of INT gives a constant increase of 0.5% magic hit rate
- There is no information for initial and final magic hit rates both below 50% magic hit rate
"Open question": Suppose ΔINT 10 really is a critical point. Then what happens to INT bonuses (or penalties) that cross this critical point? (The Burn experiment that the Japanese blogger described, which I did not address, seems inconclusive on this point.)
Finally, there appears to be a level correction/penalty to magic hit rate when targeting something higher level than you.
The temptation now (at least for me) is to seek out existing data sets and see if they are consistent with the model just described, but I will try to look at the "open questions" I just identified in a later post.
(The approximate normal distribution is drawn with a red curve, and the histogram uses the data generated from simulation.)
If the assumption (+1.0% effective m.acc per +1 elemental skill) is actually true, then observing an increase in effective magic accuracy of 0.9% (or less) should be extremely rare (see left tail), given 11,000+ samples.
