Error Target vs. Iterations

Serious discussion Simulations
1

For those of you who have been using simulators for a while, you have probably seen terms like “error target” and “iterations” used, and they generally impact how “accurate” your simulation will be and how long it will run. But most people aren’t that clear on exactly what these mean and what they should care about. Here we explain!

Background

WoW simulators work on a very basic principle: the game is too complex to come up with a mathematical formula to calculate how much DPS a given player will do, so we create a model of the game that is as close to the real thing as possible, but waaaay faster. Then we simulate e.g. 1000 fights in a few seconds, and average the DPS from all 1000 of those. This average DPS is what you can expect to do on any given fight.

The number of fights that we average will determine how accurate that average is. A simple analogy would be measuring the length of a unicorn’s horn: there are 1 million unicorns in the known universe. If we find 10 of them and measure their horns and average, it probably won’t be a very good average. Maybe we found the 10 unicorns with the puniest horns. If we instead spend a bit more time and measure 1000 unicorn horns, the average will be much more likely to be accurate for any given unicorn.

Error Target

We use an Error Target to figure out how many fights to simulate to get your average DPS. The idea is simple: I want to know my average DPS with a known amount of certainty, e.g. within 0.25% of the true average. I don’t really care how many fights I need to simulate to get that accuracy, just stop once it is that accurate.

This is by far the best way to tell a simulator how long to run, and it is the only way to do it in the AMR simulator.

Iterations

“Iterations” is simply another word for the number of fights to simulate. Back in the day, most people would run a fixed number of iterations, e.g. 10,000. The problem with this is that some specs take a lot more iterations than others to get the same amount of certainty – their damage is prone to more randomness or big swings. So you really don’t know how accurate your result will be if you run a fixed number of iterations.

We no longer display the number of iterations on the AMR simulator because it is a useless number. It is a terrible way to tell the simulator how long to run. You want to get a specific amount of accuracy, not a variable amount of accuracy.

Iterations are an implementation detail, not something that users of a simulator should care about. It was a mistake to ever make it something “visible” to the end user, and we are correcting that mistake.

If you want more accurate simulations, set a smaller Error Target! That will necessarily run more iterations behind the scenes, and it will stop once your target is reached. The margin of error is calculated via well-known statistical formulas – there’s no way to “cheat” it and reduce error with less iterations. Some specs will just take longer than others.

2

This is a very good explanation, but I wanted to quibble with one statement at the end:

The margin of error is calculated via well-known statistical formulas – there’s no way to “cheat” it and reduce error with less iterations.

Actually, there are ways. Maybe not easy ways, but there are simulation techniques that fall into a general category called “variance reduction”. The variance of the average computed by a simulation is (sigma)^2/n, where sigma is the standard deviation of simulation outcomes, and n is the number of trials. Rather than (or in addition to) increasing n, you can reduce sigma by simulating a different process that has the same average, but smaller variance.

In the context of a WoW simulation, one way you could likely gain would be by coercing critical and other random proc rates closer to their expected values (or by re-weighting to reduce the effect of the deviations from expectation).

There is something to be said for simplicity and transparency, too, of course, even at the expense of waiting a little longer for the answer. As the saying goes “brute force isn’t always the answer, but it’s usually the answer”…

3

Yeah, you’re right – there are certainly tricks you could use in the simulator that should produce the same average but with lower variance.

We only do that in a couple of cases though: one is the slightly random range on damage for many effects, e.g. melee attacks will hit for +/-20% of your weapon damage I believe. Many other spells have a similar damage range. The simulator will always hit for your average weapon damage though – to my knowledge, there is nothing conditional on the amount of a weapon damage hit (by that I mean the fact that you hit for 1.2 million vs. 1.6 million does not cause something different to proc or a different rotation decision to be made, etc.) So the final result should be exactly the same.

We stop there though instead of trying to get clever and reduce other sources of variance. Might be fun to play around with it in the future for fast simulations where all you care about is e.g. the final DPS number.

edit: I guess I should have added to be more precise: if you change nothing about the simulation then there is no way to get a smaller margin of error with less iterations. The kind of tricks mentioned above would require changing the simulation itself.