Over the final three weeks of the regular season I’ve been comparing the predictiveness of StarsDraft.com’s salaries against the projections of CBS fantasy football experts Dave Richard and Jamey Eisenberg. So far StarsDraft.com is 2 for 2 for having more accurately ranked and projected all fantasy athletes, on average, in both weeks 15 and 16. For round 3, rather than focusing on either expert individually, we will take the average ranking of the two as a baseline against which StarsDraft.com salaries are compared.
When dealing with expert opinions in statistics, averaging is a common technique that tends to improve overall performance. The so-called ‘wisdom of crowds’ explanation is that experts tend to systematically error one way or another, but when you average multiple experts together these errors offset each other resulting in a generally more tempered and more accurate approach. The idea is that the mean of the projections is more predictive than any of the individual projections.
Again, the same caveats apply to this analysis (SPOILER ALERT: this week we will see why they matter so much)! StarsDraft.com salaries are published with our contests on Monday afternoon. The projections on which the salaries are based are made Monday morning and do not change throughout the week. Most analysts, including the CBS crew, have the added benefit of an entire week of injury updates, breaking news, or Twitter buzz to perfect their projections. More importantly, going into week 17 experts have the added benefit of hearing about all the deactivations or other personnel moves that make week 17 an exceptionally tough fantasy week to predict.
Perhaps the most straightforward way to compare fantasy football predictions, and by far the easiest to understand, is to compare the absolute rankings of two rankers against the actual ranks of each player. Treating both StarsDraft.com’s projections and those from CBS as ranks, we can see the following (I’ve restricted the output to athletes that actually scored 1 or more fantasy points):
In summary: Of the 267 players analyzed, StarsDraft.com ranks still managed to outperform CBS 55.4% of the time. StarsDraft.com ranks were superior to the CBS consensus at every position, with the smallest edge at the QB position where StarsDraft.com’s rank was closer to actual only 51.4% of the time (CBS ranks were superior 42.9% of the time with 5.7% of the cases being identical rankings). The RB, WR and TE positions were ranked closer to actual 57.7%, 54.2% and 57.8% of the time, respectively.
Here’s where things get interesting. I’ve been reiterating how difficult it is for a set of projections made Monday afternoon to outperform experts who have the benefit of an entire week’s worth of research and breaking news. This may have fallen on deaf ears since, generally speaking, StarsDraft.com salaries have dominated both Eisenberg and Richard in predictive performance despite this disadvantage. This week we see an interesting twist:
Check that chart out closely. StarsDraft.com projections were more accurate than CBS in 142 out of 267 fantasy relevant players, but the average error of the StarsDraft.com projections was greater than that of CBS. How can that be possible?
Check out the QB projections:
QB accounts for the bulk of the errors. The QBs most inaccurately projected by StarsDraft.com were, in order: Tom Brady, Brock Osweiler, and Jimmy Garappolo. To offset the uncertainty of going into week 17 without knowing which team would sit their starters, StarsDraft.com made a ‘best guess’ and bumped Osweiler and Garappolo up to fairly hearty projections while maintaining the standard high projections for their respective starting QBs. Over the course of the week it became obvious that Bellicheck intended to sit or decrease the roll of many of his starting players, and the Denver loss on Monday night in week 16 ensured that Osweiler – rather than getting a start, or at least a huge chunk of playing time – would be relegated to only garbage time points as Denver would need to start Peyton in week 17 against the Raiders to try and improve their playoff picture.
CBS analysts had all of this data at their fingertips and responded accordingly. As a result, they were much more accurate in situations where this sort of special knowledge played a key role. For the majority of athletes that were playing in situations where the mathematics and algorithms that StarsDraft.com uses to set player prices still applied, StarsDraft.com outperformed CBS. But when StarsDraft.com was wrong on account of incomplete information, it was exceptionally wrong – and those outsized errors served to offset the fractionally better predictions in all the other cases.
That being said, StarsDraft.com’s projections still outperformed CBS at 3 of the 4 positions, more accurately projecting 55.1% of RBs, 51.4% of WRs and – continuing the theme of industry wide over-estimation of the TE position – 68.9% of tight ends.
Information matters. Mathematics and advanced statistics can take you a long way towards predicting the outcomes of future events, but these methods are only as good as the data they are given. To some degree the QB projections exception here proves the rule: had CBS been forced to put out week 17 projections on Monday prior to Bellicheck deactivating Gronk and Edelman, and Denver getting dusted on Monday Night Football, the QB projection data would likely tell a very different story.
In any event, I’d still say StarsDraft.com generally outperformed the CBS duo, at most positions and based on most metrics. As I mentioned last week, predicting fantasy football performance is one of the more challenging prediction problems I’ve faced as a data scientist. The fact that we can arrive at any algorithm that outperforms human experts who are employed full time to analyze, dissect, and prognosticate on fantasy football speaks to the merits of an unbiased statistical approach to the problem.