[Ganchrow]

In reference to this thread a couple of people have asked how to properly calculate the variance on ternary outcomes events, specifically those that may end only in a win, lose, or push (so this wouldn't apply to Asian handicap).

Without further ado:

Let p_w = win probability
Let p_l = loss probability
Let p_d = 1 - p_w - p_l = draw probability
Let f = fractional odds
Let x = bet size
Let E = % edge

E = p_w * f - p_l
^σ2/_x² = (1 - p_d + E) * (f - E) - p_d * E


You'll note that for a push probability of zero (i.e., for p_d = 0) this reduces to the familiar formulation of variance:

^σ2/_x² {no push} = (1 + E) * (f - E)

Another question, of particular relevance when back testing a strategy, would be how to calculate the standard deviation in expectation of an event observed and bet over n occurrences.

This is actually a less general question than the one answered above (insofar as it constrains payout odds and bet size to be identical for each wager), but because some are most accustomed to the n*p*(1-p) formulation of binomial variance I'll provide the trinomial win/loss/push analogue here. (I'll leave it an exercise to the reader to demonstrate that the formulation in post# 1 is just a generalization of the formulation following.)

Let N = # of trials
Let p_w = observed win frequency over N trials
Let p_l = observed loss frequency over N trials
Let p_d = 1 - p_w - p_l = observed draw frequency over N trials

The variances and covariances for # of wins, # of losses, and # of pushes are then given by:

Var[# of wins] = N * p_w * (1-p_w)
Var[# of losses] = N * p_l * (1-p_l)
Var[# of pushes] = N * p_d * (1-p_d)
Covar[# of wins, # of losses] = -N * p_w * p_l
Covar[# of wins, # of pushes] = -N * p_w * p_d
Vovar[# of losses, # of pushes] = -N * p_l * p_d

If we further define:

Let f = fractional odds
Let x = bet size

Then the total variance in win amount would be given by:

^σ2/_x² = f² * Var[# of wins] + Var[# of losses] - 2 * f * Covar[# of wins, # of losses]
^σ2/_x² = N * ( f²*p_w*(1-p_w) + p_l*(1-p_l) + 2*f*p_w*p_l )

[Ganchrow]

So here's an example of how to practically use the second formulation above to test for significance.

Observed:

260 observations (N = 260)
all bets at -110 (f = ¹⁰/₁₁)
wins = 143 (p_w = 55%)
losses = 104 (p_l = 40%)
pushes = 13 (p_d = 5%)

Assuming results were properly obtained out-of-sample, what would be our 95% confidence lower bound for true strategy ROI (using the Central Limit Theorem)?

Calculating ROI and ROI variance we find:

absolute ROI = 143 * ¹⁰/₁₁ - 104 = +26 units (+10%)
σ² = 260 * (¹⁰⁰/₁₂₁*55%*45% + 40%*60% + 2*¹⁰/₁₁*55%*40%) ≈ 219.58 units²

or σ = 14.8183 units.

So taking into account the ternary outcome set, our (frequentist) 95% confidence lower bound on ROI would be 26/260 - 1.6261*14.8183/260 ≈ +0.6254%.

Let's compare this to the results we'd find using conditional probability with a binary outcome set.

Let p* = win frequency conditioned on not pushing = ¹⁴⁰/_(260-13) ≈ 57.895%

Which would give us a conditional ROI and variance of conditional ROI of:

conditional ROI = +26 units (that's 26 / (260-13) = +10.526%)
σ² = (260-13)*(¹⁰⁰/₁₂₁ + 1 + 2*¹⁰/₁₁) * 57.895%*(1-57.895%) ≈ 14.8137 units²

So taking into account the binary outcome set, our (frequentist) 95% confidence lower bound on conditional returns would be 26/(260-13) - 1.6261*14.8137/(260-13) ≈ +0.6614%.

Removing the conditioning would yield a lower bound of +0.6614% * (260-13)/260 ≈ 0.6283% on absolute returns. Pretty close to what we had found above (and the error is certainly less than that introduced by using the Central Limit Theorem in the first place).

But here's real issue:

Taking into account the ternary outcome set, how would one go about testing the null hypothesis that the strategy is in reality no better than breakeven?

Anyone?

This is straightforward if we condition the results and use binomial variance, but not quite so otherwise. It's exactly this difficulty that leads most practitioners to simply condition out pushes rather than deal with the vagaries of the trinomial distribution.

[square1]

Originally Posted by Ganchrow

But here's real issue:

Taking into account the ternary outcome set, how would one go about testing the null hypothesis that the strategy is in reality no better than breakeven?

Anyone?

This is straightforward if we condition the results and use binomial variance, but not quite so otherwise. It's exactly this difficulty that leads most practitioners to simply condition out pushes rather than deal with the vagaries of the trinomial distribution.

Well, there are an infinite number of breakeven distributions: choose p(d) on (0,1), and then set p(l)/p(w) = f. I don't believe the problem as stated is well-defined without a prior on the bivariate. So to test, you'd need an additional structural assumption of some kind. If I were doing it I think I'd fix p(d) at its MLE, which is 13/260 in your example, and go from there. Rigorously justifiable? Of course not, but I'd have minor faith in the result.

[Ganchrow]

Originally Posted by square1

Well, there are an infinite number of breakeven distributions: choose p(d) on (0,1), and then set p(l)/p(w) = f. I don't believe the problem as stated is well-defined without a prior on the bivariate. So to test, you'd need an additional structural assumption of some kind. If I were doing it I think I'd fix p(d) at its MLE, which is 13/260 in your example, and go from there. Rigorously justifiable? Of course not, but I'd have minor faith in the result.

Why restrict yourself by fixing the p_d parameter? Why not treat it as a random variable with a uniform prior distribution (say)?

But yeah, you got the idea.

It's a pain in the ass.

[Ganchrow]

This spreadsheet illustrates what I had in mind. It implicitly assumes a uniform prior distribution of p_d and uses the t-distribution to impute the final p-value (in both the binomial and trinomial cases).

Assuming I didn't make an error, the trinomial p-value of the null-hypothesis (that population EV ≤ 0%) actually turns out to be slightly lower than the binomial p-value (4.168% vs. 4.199%), and coming up with the latter is a whole lot easier.

[square1]

Originally Posted by Ganchrow

Why restrict yourself by fixing the p_d parameter? Why not treat it as a random variable with a uniform prior distribution (say)?

But yeah, you got the idea.

It's a pain in the ass.

There's nothing wrong with the uniform, and it's probably not going to have a huge impact on the result.

I like the MLE because the essence of hypo-testing is to ask, "How objectionable does our data find these probabilities?". Since we are presumably a lot more interested in the win/loss ratio than the draw probability, I chose the draw probability the data finds least "objectionable", which is the essence of the MLE. So if we reject, we know the data has no issues with our p_d parameter, and thus is telling us the null-hypo win/loss ratio is the part of the null that is causing our result to be improbable under the null.

[Ganchrow]

Originally Posted by square1

I like the MLE because the essence of hypo-testing is to ask, "How objectionable does our data find these probabilities?". Since we are presumably a lot more interested in the win/loss ratio than the draw probability, I chose the draw probability the data finds least "objectionable", which is the essence of the MLE. So if we reject, we know the data has no issues with our p_d parameter, and thus is telling us the null-hypo win/loss ratio is the part of the null that is causing our result to be improbable under the null.

Well I'd say we're less concerned with the win/loss ratio per se, and more concerned simply with the model return. Win/loss ratio is just so ... binomial.

Regardless, I think we can both readily agree that it's not going to change the results much at all one way or the other and that the best easiest solution is probably clearly to go with binomial modeling coupled with conditional-form model parameters.