Fates RN Regression Project

[Minternets]

I can brute a couple hundred more. Is 75 the only anomaly, or is there a range I can look for (such as 75-79)? Finding castles with exact hit rates is a chore.

[Crimean Archivist]

I can brute a couple hundred more. Is 75 the only anomaly, or is there a range I can look for (such as 75-79)? Finding castles with exact hit rates is a chore.

I've got 75 covered because of a Normal save at Ch 2 abusing Kaze. You can brute force any value less than 90 with a statistically significant sample size and it will make a difference overall -- above 90 it would take too many tries to get any meaning on its own and is only useful when fitting a broader model. Anywhere between 60 and 80 is ideal, or 20 to 40 on the other side if you want to look for more evidence that values < 50 may use just one RN. We currently have the best sample size between 74 and 85, so those are a little less pressing than other values.

Edited March 9, 2016 by BlenD

[Twiggy]

I wonder what should we call this kind of Hit Rate system... it's both 1RN and 2RN at the same time! Makes for a bit of a fun thing since, well, it does reduce the "miss at 90%+" probabilities, while still giving you more probably to land a hit at low accuracies... and having your dodgers to be more dangerous since it's more likely to hit now.

Digging apart the hit rates seem to be a nice thing, really, and I appreciate how beautiful the resulting data looks like.

[eclipse]

I have a Conquest save where I can test out 19% hits consistently. Of the five or so times I did it, all of them missed.

If you want more data points, let me know how many.

[Verile]

I can brute a couple hundred more. Is 75 the only anomaly, or is there a range I can look for (such as 75-79)? Finding castles with exact hit rates is a chore.

I find it's easier just to modify my own defenders and then "Check Defenses" under the Streetpass settings.

[Crimean Archivist]

I have a Conquest save where I can test out 19% hits consistently. Of the five or so times I did it, all of them missed.

If you want more data points, let me know how many.

19 would be a good value to farm at, since there's a 10% difference between 1-RN and 2-RN and will help ascertain one way or another if we have a split system.

For testing raw vs weighted 2-RN, 66 is the absolute best spot, as there's a 5% difference between the two models. It'll take about 300 points at some value between 63 and 70 to make any kind of conclusion. I'd suggest 67 because it's the one with the most points so far, but we're acknowledging that our value for 67 is anomalously high at present and might confound results. Testing another nearby value may be enough to ascertain whether or not to discard that outcome.

[Vascela]

I currently have about ~550 data points in an excel spreadsheet. Once I get to about ~1000, what would be the easiest way to convert that into forum acceptable format?

edit: and those points are all over percentages, hard and lunatic, friend and foe (if that was relevant)

Edited March 10, 2016 by Vascela

[Crimean Archivist]

Honestly it looks like the best way is to copy-paste the values since you can't upload spreadsheets here. To ensure that the formatting stays okay, I'd recommend exporting the data to a .csv or a .txt file first. In Excel: File > Export > Change File Type.

[Crimean Archivist]

~10000 data point update:

I tested several different hit rates in Check Defenses. We now have quite a few points <50 with ~200 data points or greater.

9 -- 200 points, 7.5% hits

23 -- 179 points, 21.23% hits

36 -- 312 points, 33.65% hits

41 -- 189 points, 42.86% hits

49 -- 249 points, 51.41% hits

And above 50:

56 -- 282 points, 57.09% hits

63 -- 270 points, 71.48% hits

66 -- 171 points, 74.27% hits

67 -- 284 points, 83.45% hits (anomaly?)

74 -- 240 points, 84.58% hits

75 -- 395 points, 82.28% hits

76 -- 217 points, 84.79% hits

82 -- 420 points, 90.71% hits

84 -- 190 points, 92.63% hits

85 -- 213 points, 96.23% hits

87 -- 200 points, 94% hits

89 -- 202 points, 96.53% hits

94 -- 430 points, 99.07% hits

95 -- 183 points, 97.81% hits

This is where some educated guessing comes in.

Raw 2-RN almost universally overestimates measured hit rates. Ideally, if the system was a raw 2-RN one, we would expect a near-equal number of values to be represented above the line as below it (taking into account that misses disproportionately affect the upper edge case). Taking that a step further, the average error for high-confidence values for the (3A+B)/4 model is -1.01872 as opposed to the Split model's 1.679617. When only the magnitude of the errors is considered, the (3A+B)/4 model rests at 2.38098 for high-confidence values while the Split model is at 2.69778. Here's the graph, for reference:

I don't think our current data is conclusive for (3A+B)/4, but I do think it is conclusive to say that the system has changed. Also, the value for (3A+B)/4 and raw 2-RN fall outside the 99% confidence interval at Hit=9, so that's pretty substantial additional evidence for 1-RN < 50.

[Twiggy]

I'm thinking the model might actually be a Split Weighted RN. As in, the upper 50% hit rates are probably weighted somewhere instead of strictly averaged 2RN, while the lower 50% go back to the FE1-5 way?

Edited March 10, 2016 by Twiggy

[Crimean Archivist]

That's what it looks like. It's about time to start playing with different combinations of (M*A + N*B)/(M+N) just to make sure we're looking at the right number combination, although I doubt either M or N is greater than 5 (really, I doubt anything greater than 3, but quality assurance is good). I'm going to calculate out a mesh with (2A+B)/3, (4A+B)/5, (5A+B)/6, (3A+2B)/5, (4A+2B)/6, (5A+2B)/7, (4A+3B)/7, (5A+4B)/9, starting with the bolded values and (3A+B)/4.

[XeKr]

Another complicated possibility is if there are dynamic hit rates (like FE11 dynamic growth rates), that result in an increased probability for the next attack to hit if the current one missed. This could mean that data collected for repeated trials without resets (like I think the normal playthrough and MyCastle stuff is?) follows different rules from data collected from resets of the same situations (C2 Kaze, C6 Hoshido partially)

If it only applies for misses, for example, this could also explain why low hit rates are skewed differently from high ones.

I’m not sure if you already have working code (it's simple), but it’s completely trivial for me to switch a few variables and calculate all those various (or any) weighted models if you want.

Edited March 10, 2016 by XeKr

[Crimean Archivist]

I was going to nix a lot of the tables I have set up in Excel and replace them with the values for each model so I can compare them side-by-side in real-time, so I'll take care of it.

I am not prepared to deal with dynamic hit rates so let's pray that's not the case.

[GoXDS]

I'm really hoping it's nothing like a split system or dynamic system. it's not elegant and a lot more deceptive than non-1 RN already is

[Crimean Archivist]

I used a meshgrid and added the above bolded models and their errors to the spreadsheet. Haven't recorded any new values since last night. Only thing to report really is that if the raw 2-RN is out for the upper range, so is (3A+2B)/5.

Edited March 11, 2016 by BlenD

[Crimean Archivist]

How did we find dynamic growths in Shadow Dragon anyway? I've looked for the credited author on the main SF page (Nitrodon) and that person has been inactive since 2011. If there's something that needs to be tracked in order to be sure I want to be keeping tabs on it.

Edit: Thinking it over, and I don't think a dynamic system is likely. Unlike growth rates, which can be fixed in an array and modified easily, the function that calls the RN never "knows" what it's getting. It would have to make modifications to RNs as they came in because a 99 value array would be a big waste of space. That would mean that a result at 41 could affect the result at 99 after enough trials, and that would spell all kinds of disaster.

Edited March 11, 2016 by BlenD

[eclipse]

How did we find dynamic growths in Shadow Dragon anyway? I've looked for the credited author on the main SF page (Nitrodon) and that person has been inactive since 2011. If there's something that needs to be tracked in order to be sure I want to be keeping tabs on it.

Edit: Thinking it over, and I don't think a dynamic system is likely. Unlike growth rates, which can be fixed in an array and modified easily, the function that calls the RN never "knows" what it's getting. It would have to make modifications to RNs as they came in because a 99 value array would be a big waste of space. That would mean that a result at 41 could affect the result at 99 after enough trials, and that would spell all kinds of disaster.

IIRC people leveled Navarre on Chapter 3, didn't proc HP, then reclassed him to Cavalier on Chapter 4, gained a level, and noticed that he gained 2 HP (which shouldn't be possible under normal circumstances).

Anyway, a dynamic system would be something like this:

- Check hit rate against random numbers

- If an attack hits, increment some counter

- If an attack misses, decrement the counter

- Modify the next hit rate by the counter before rolling the next set of numbers

THIS IS JUST A THEORY, NO PROOF. It's an example of how such a system could be implemented.

19 would be a good value to farm at, since there's a 10% difference between 1-RN and 2-RN and will help ascertain one way or another if we have a split system.

Unfortunately, this percentage occurs at the end of the chapter, so it may be influenced by other hit/miss values. For consistency's sake, I'll note down. . .well, everything. But I can't guarantee as many trials, due to lack of patience/my back being stupid. I'll get to this on a day when I can sit up for longer than a couple of hours.

[XeKr]

The reason I brought up dynamic hit rates is there was some precedent in FE (dynamic growth rates). A split model seems like an rather random way to program things, especially if one side ends up being weighted anyway. And the hypothesis only really showed up after we got a bunch of MyCastle data (ofc, we also didn't have much low rn data before), which made me think a bit about the method of data collection.

I obviously think it would be preferable to go with as simple a model as possible (ideally neither dynamic or split), especially as there are any number of models it could be, but things are a bit confusing right now.

It might be rather straightforward to check for though. If it's possible to find 75% in MyCastle somewhere, just test that value without reseting for a few hundred trials, and compare to our earlier results. If the number differs significantly, it would suggest a dynamic model. I'll note, it's somewhat interesting that 67% (which I think was mostly tested repeatedly in MyCastle) had many more hits than was expected, even by raw 2RN, which could be consistent with a model that dynamically buffs hit rates after misses (and not vice versa. This is important because if both occurred it should tend more to the "true" hit rate, not skew).

edit: I don't actually know how many of those >50% in post #59 you tested in MyCastle, BlenD. I read that post as you tested some points <50% and the other data was from before. That might already be enough data to kill this dynamic idea, depending on how it was collected.

Edited March 11, 2016 by XeKr

[Verile]

500 trials at a single value.

[Hit Rate], [Outcome H/M], [Actual]

27, 149/351, 29.8%

Edited March 12, 2016 by Verile

[Crimean Archivist]

Verile's results at 27 places both (3A+B)/4 and raw 2-RN outside the confidence interval for 27, providing more evidence for a 1-RN system at low values.

Looking at the math, I don't think a dynamic system explains our anomalies. If we had a dynamic system, then what we would effectively have is a chance P to move a fixed point of reference up or down to either P+V in the case of a miss or P-V in the case of a hit. The probability would then oscillate up and down around that point. At high values, the expectation would be to undershoot the given value, as every time the probability returned to its "normal" value, it would be more likely than not to hit, adjusting down to P-V and reducing the likelihood of success the next trial. So it would be more likely to oscillate down than up. Apply that to all values, and a dynamic system would be middle-shifted, if that makes sense.

A dynamic system could be how to explain why our measured values undercut raw 2-RN values, but modelling that would be a pain so hopefully we won't have to.

Edited March 12, 2016 by BlenD

[Verile]

500 trials at a single value.

[Hit Rate], [Outcome H/M], [Actual]
66%, 358/142, 71.6%

About 5.6% lower than what raw 2RN anticipates (77.22%), so may help rule out that system in favor of weighted (for >50 hit, at least)

Edited March 12, 2016 by Verile

[Crimean Archivist]

edit: I don't actually know how many of those >50% in post #59 you tested in MyCastle, BlenD. I read that post as you tested some points <50% and the other data was from before. That might already be enough data to kill this dynamic idea, depending on how it was collected.

The other night I tested values at 71, 66, 56, 82, 41, and 49. As far as I know, all of the low values we have large samples for were collected in My Castle. Everything else is fairly mixed and includes both My Castle and chapter data.

Thanks to Verile's latest data, 66 is now the point with the largest sample size, at 680 points. Aggregate hit rate is 72.36%. Raw 2-RN expected value (77.22%) is ~0.5% outside the 99% confidence interval. (3A+B)/4 is nearly in agreement with results with an expected value of 71.83%.

[Crimean Archivist]

I tried to make a generalized integral in the vein of what Dark Holy Elf did for the function Ax + By = (A+B)z (where x is RN1, y is RN2, A and B are their coefficients, and z is the nominal probability), but all of my attempts have resulted in unsolvable equations. Basically, I expected the result of the integral = P (measured probability) to make some kind of concrete relationship between A and B when applied to different hit values and measured results. I'm pretty certain it's possible, but there's probably something wrong in my methodology here.

I tried:

From x = 0 to (A+B)*z / A; y = 0 to [(A+B)*z - A*x] / B,

int( A*x + B*y dy dx) = P

The result I got (checked with WolframAlpha) is z^3(A+B)^3 / (3*A*B) = P. Unfortunately this isn't even remotely close to a viable solution, as (A+B)^3 / (3*A*B) would be a constant, since A and B are constants, making P a linear function of z^3. It's very possible my bounds are wrong; it's been a while since I've done this kind of integral.

If anyone comes up with an integral that works, let me know. It would allow me to make a single model that fits to our best data and not worry about checking against multiple different models. Then I would be able to extract A and B dynamically as we continue forward, which should hopefully expedite things.

[XeKr]

Thinking of it in terms of how Dark Holy Elf wrote it, below a certain line on the A-B plane (x-y plane), I think we only need an integral in terms of x ("A" in Dark Holy Elf's equation), with y ("B") as a "function" of x. For example, in the "(3*A+B)/4" or (3x+1y)/(3+1) weighted case, it's roughly a single line of the form...

int((200/3-100)/(100-0) * (x-0) + 100) from x = 0 to 100

Which gives 8333.3333.../10000

(the 0s are important, they denote shifts that are relevant for other weights, see later)

But it should be apparent that in certain cases, like the raw 2RN (A and B = 1, the weights A and B. Yeah it's confusing), the sloping line only begins around x = 50. From x = 0 to 50, the integral should be summed separately, with y = 100. From x = 50 to 100, the integral is for a shifted line.

int(100) from x = 0 to 50

int((50-100)/(100-50) * (x-50) + 100) from x = 50 to 100

For a total of 8750/10000, which I think is the (continuous) raw 2RN hit for 75% disp?

More generally, these are...

int(100) from x = 0 to (z*(A+B)-A*100)/B

int(((z*(A+B)-B*100)/A-100)/(100-(z*(A+B)-A*100)/B)*(x-(z*(A+B)-A*100)/B)+100) from x = (z*(A+B)-A*100)/B to 100

Note that

(z*(A+B)-A*100)/B for z = 75 A = 3 B = 1

is equal to 0 for these weights, which determines the shift and integral bounds.

Note that

(z*(A+B)-B*100)/A for z = 75 A = 3 B = 1

is a slightly different formula, and is 200/3 or 66.666... , which determines the slope.

Note that both are 50 for A = B = 1, z = 75.

I used 100 instead of 99 for simplicity. All should work in Wolfram. Math check later b/c I rushed this...

Edit:

afaict math looks fine for (z*(A+B)-A*100)/B > 0. Forgot to account for cases where the line shifts left/downward. Try later, a similar approach should work, though it gets even more piecewise.

Also, it’s only happy for weight ratios that aren’t too far from each other. I suppose very high or low ratios probably aren’t that relevant, as they look very similar to raw 1RN. But the practical problem modeling that in this method is it gets close to divide by zero and vertical lines that are not happy y(x) functions and rounding (and other) errors can occur.

A different approach may be easier for a more generalized formula. Is there an analytical approach without summing separate integrals?

and finally, I haven’t really investigated but I think models of the form (Ax+By)/(A+B) only results in symmetric systems, however? And it seems (3x+y)/4 is already pretty close for >50.

Edited March 14, 2016 by XeKr

[Crimean Archivist]

Can we rewrite that integral in a way that I can calculate A and B given Z and P across multiple points using a system of equations?