[The HG]

I assume that the amount bet on a game that winds up as a push counts when calculating ROI, but what about a game that gets rained out?

The money was in use to make the bet, but since the game was never "official", that money was never actually at risk.

Do you count money wagered on a rained out game when calculating ROI?

[Data]

If I were to calculate ROI I would use neither but the amounts posted up and withdrawn.

[Arilou]

If you're calculating how much you make when you place a wager, the wagers you placed should count whether the game was played or not, or so I would think.

[Wheell]

I disagree with Arilou. If a bet pushes due to a tie (A 3 point favorite winning by 3), then yes, that counts, but a game that could never win nor lose should not count.

I'll give an example: when calculating my hold on baseball I remove games that were nullified due to rain or pitching changes.

I understand the possibility of ties does affect your return on investment (think of an arb on a .5 vs. an arb on a non-zero whole number), but I find that hard to reconcile with one's hold in a game that simply never reached the point of grading.

[MrX]

I'm with Wheel. The event has to happen for an investment to have been made.

[VideoReview]

I have never read a serious academic paper on sports gambling that includes "rained out" or games postponed or cancelled for any reason and would definitely not count these.

As far as pushes go, virtually all of the recent (last 10-15 years) papers I have read specifically say that the results are statistically more "robust" when the pushes are excluded. In other words, when a correlation is found, they found that future predicted results were more accurately made when the pushes were excluded completely from the analysis.

I have also had this method confirmed and the best alternative by a well informed moderator here at SBR who I suspect will be chiming in on this thread soon.

So, my recommendation to you is not to use either.

[Data]

Originally Posted by Wheell

I disagree with Arilou.

I could not understand neither of you. What is the meaning or significance of calculating ROI this way? From economical standpoint, you have to count all the money you post up no matter whether you bet them or not.

[Ganchrow]

I think this really begs the question "Counts towards ROI to what end?" as I'm personally aware of no Platonic Ideal of ROI.

ROI is just one possible metric of many and the way to best calculate it is going to be a function of how one plans to use it.

On the one hand, from an investment perspective I think it makes sense to include no-actioned games insofar as funds were at risk (even briefly) and could potentially have been lost. I see this as true regardless of whether the game was no actioned due to a push or due to it having been physically canceled.

The rationale for this is that an investor attempting to choose between alternative strategies would need to take into account each outcome set in its entirety in order to make a truly optimal decision. Imagine, for example, a hypothetical strategy taking advantage of the observed fact that a particular style of pitcher tends to outperform market expectations given sufficiently rainy weather. While this strategy might correspond to a conditional ROI of let's say 5%, we'd still need to consider that a relatively high proportion of games would be rained out, and as such an n-Kelly investor would never choose this strategy in preference to one with identical conditional edge but a lower push probability (although a sufficiently risk averse Markowitz investor might -- but this is only indicative of some nasty peculiarities of Markowitz.) Of course in reality an investor's choices aren't usually so stark as strategies aren't usually all-or-none propositions. Nevertheless when allocating between strategies the fact that one is more likely to be no-actioned than another needs to be a part of the allocation optimization.

Of course, whether we consider win/loss probabilities in absolute or conditional terms (i.e., conditioned on not pushing) won't impact optimal single bet n-Kelly staking. What it will impact, however, is expected n-Kelly cardinal utility (but not ordinal utility, meaning that the relative preference rankings of various stake sizes won't change regardless of how probability is construed). This means that even though single-bet n-Kelly stakes won't change, simultaneous bet n-Kelly stakes will change. In practice, however, this will generally be a relatively minor effect.

On the other hand, when back testing a strategy in isolation there's compelling practical rationale for factoring out pushes and focusing solely on conditional probabilities and edges. Specifically, when one includes pushes in one's analysis it's no longer correct to model results solely using the binomial distribution. Rather one would need to use the trinomial distribution insofar as there exist three possible outcomes for each event. This adds a not insubstantial additional layer of complexity. However, because the difference in sample Sharpe ratios (that’s the sample mean divided by the sample deviation) can be shown to be minimal for edges and push rates reasonably close to zero, most academic papers on sports betting ignore no actioned bets by dealing solely with conditional probabilities and edges. This is very much standard practice although it should be noted that this will tend to slightly overestimate the sample Sharpe ratio of events that may push. (To give an idea of scale, a bet on an event paying out at even odds is observed over 300 trials to win, lose, and push 55%, 40%, and 5% of the time respectively. The correct trinomial Sharpe ratio would be 2.6977, while the approximated binomial Sharpe ratio would be 2.6994, corresponding to Gaussian p-values of 99.651% and 99.653%, respectively.)

But yes, if you really wanted to be completely anal about it you should really factor in all no actioned bets (and hence use the trinomial distribution) when calculating significance.

So when comparing classes of strategy a practitioner is going to be best served factoring in the impact of pushes. If there also exists a substantial discrepancy between the relative frequencies of physically canceled games, then this should be taken into account as well. For example, assuming he could only bet one, an n-Kelly bettor would choose to bet an NBA game rather than an MLB game with identical conditional payout characteristics due to the higher probability of the latter game being no-actioned. (Academic? Maybe a bit.)

When backtesting single strategies, however, a bettor will find nearly identical p-values (for reasonable push frequencies) whether he looks at conditional or absolute probabilities. Because using conditional probabilities allows the bettor to use the binomial distribution, this generally represents a more tractable solution.

Lastly, Data's interpretation of ROI is probably the most theoretically sensible when looking at the overall macro success of a particular bettor, but probably isn't as useful for evaluating the statistical properties of an individual strategy or when comparing multiple strategies that generally put into use no more than a fairly small percentage of bankroll at any given time. Certainly ROI by itself as construed above tells us relatively little but that's why it's usually looked at in conjunction with odds ranges, frequency of bets, and variance.

I'd certainly agree with Data that strategy capacity/bankroll utilization should ultimately be considered, but this is more of a concern in "big picture" scenarios such as when trying to determine whether or not one should quit one's job to become a professional gambler, or how much one should allocate to one's betting portfolio versus one's stock market portfolio. (Another time this might come into play would be when trying to determine optimal allocations amongst difficult-to-transfer-between sportsbooks that house different strategies.)

Still, this interpretation of ROI+variance doesn't preclude either of the two others insofar as nothing would prevent an investor from calculating expectation and standard deviation in terms of percent of total bankroll. While doing this does circumvent the issue of how to treat pushes in the case of calculating ROI, it still leaves open the fundamentally linked issue of how to treat pushes when calculating variance.

[Data]

I am trying to understand how ROI can be used while developing, comparing and backtesting betting strategies.

Originally Posted by Ganchrow

Certainly ROI by itself as construed above tells us relatively little

That is my impression as well to the extent that I can only extract a bit of useful information judging by ROI being either positive or negative. What I do not see is the "bridge" between ROI and the weighting odds (linemaking), it just feels somewhat being "backwards" to use ROI for this. So, I never thought about using ROI this way but you guys certainly know more about this, so I hope you can explain.

Also, I think that at best ROI shows by how much your lines were better than the bookmakers lines in the past. It does not show how good your lines are. Because of that, its predictive value is very questionable. What do you think?

[VideoReview]

Originally Posted by Ganchrow

Lastly, Data's interpretation of ROI is probably the most theoretically sensible when looking at the overall macro success of a particular bettor, but probably isn't as useful for evaluating the statistical properties of an individual strategy or when comparing multiple strategies that generally put into use no more than a fairly small percentage of bankroll at any given time. Certainly ROI by itself as construed above tells us relatively little but that's why it's usually looked at in conjunction with odds ranges, frequency of bets, and variance.

BINGO.

In regard to Data's question, as a single stake Kelly bettor (I do bet multiple games but my fractional Kelly is certainly low enough to compensate and my BR is relatively low enough for me not to worry about it at the moment), I can personally confirm for you that ROI can be predictive in a substantial (not just a very small edge either, e.g. 1%) and statistically significant way in the way that Ganchrow has advised. Ask yourself "what if" the market/bookmaker wanted the ROI to be that way on purpose (i.e. true odds were known by the bookmaker/market and well informed shading has occured). Ganchrow's explanation above gives the blueprint for this.

Originally Posted by Ganchrow

I'd certainly agree with Data that strategy capacity/bankroll utilization should ultimately be considered, but this is more of a concern in "big picture" scenarios such as when trying to determine whether or not one should quit one's job to become a professional gambler, or how much one should allocate to one's betting portfolio versus one's stock market portfolio.

Your answer was not directed to me but it's like you read my mind. I was aware that I would have to eventually consider the push/cancel frequency and came to the conclusion that my time was best spent working on the systems rather than factoring this in, at least until my BR became large enough to have to make the determination you suggested above.

[RickySteve]

Originally Posted by Data

If I were to calculate ROI I would use neither but the amounts posted up and withdrawn.

So then you're willing to buy a CD that matures in 50 years and pays you a robust 20% of principal?

[Data]

Originally Posted by VideoReview

In regard to Data's question, as a single stake Kelly bettor (I do bet multiple games but my fractional Kelly is certainly low enough to compensate and my BR is relatively low enough for me not to worry about it at the moment), I can personally confirm for you that ROI can be predictive in a substantial (not just a very small edge either, e.g. 1%) and statistically significant way in the way that Ganchrow has advised.

How come it can be predictive if all it shows is the advantage over arbitrary number (the line) in the past? What if the bookmakers line gets sharper due to personnel changes and technology advancements?

Ask yourself "what if" the market/bookmaker wanted the ROI to be that way on purpose (i.e. true odds were known by the bookmaker/market and well informed shading has occured).

I did not get this, please explain.

[MrX]

Originally Posted by Data

How come it can be predictive if all it shows is the advantage over arbitrary number (the line) in the past?

Just as an athlete's past performance is predictive of his future performance, so can the bookmakers' past behavior be predictive of future behavior.

Originally Posted by Data

What if the bookmakers line gets sharper due to personnel changes and technology advancements?

Of course that is going to happen, but it doesn't mean that there is no predictive value there.

[Ganchrow]

Originally Posted by Data

How come it can be predictive if all it shows is the advantage over arbitrary number (the line) in the past? What if the bookmakers line gets sharper due to personnel changes and technology advancements?

Whenever you work off of any any sort statistically modeled data, regime change always needs to be a consideration. This is just as true in quantitative sports betting as it is quantitative finance or weather prediction or non-gambling related sports modeling. But we don't want to be tossing the baby with the bathwater, either.

I'll agree that because of this very issue one should be exceedingly cautious when dealing with predictive models that base forecasts solely off of market microstructure without regard to concrete measures of future probability. For example earlier this year I posted to this thread regarding the performance of large NCAA football dogs ATS in early bowl games.

Over the data set, there's been a very clear statistically significant outperfomance on the part of these dogs, making them excellent bets historically. This trend continued last year (out-of-sample), to the extent that it was actually the strongest year to date.

Nevertheless, while I have no problem performing this type of analysis in my capacity on this forum, I personally abhor strategies such as this and would be exceedingly hesitant to bet into them as a professional player (and would certainly never recommend them to others without massive qualification) for the exact reason you've indicated -- even if this does represent a real phenomenon, we'd have no way of knowing ex ante if and when the bookmakers (or "the market" if you prefer) were to "catch up" to reality and modify pricing habits.

What I much prefer are strategies that first make concrete predictions of prior outcome probabilities without any regard to the current market line, and then only after that use Bayesian inference (if necessary) to temper prior beliefs, creating posterior fore