StarsDraft released its daily fantasy hockey product about a month ago and it’s been well received. Games are filling and people seem to be enjoying playing hockey on our platform. The only complaint has been that we score fantasy hockey differently than other sites — in particular, we do not award a bonus to goalies if their team gets a win. This is counter to most other sites in the industry, which award a 3-point bonus to the goalie of the winning team, regardless of the goalies’ other stats.
We do a large amount of analytics each time we launch a new sport, and our aim is to capture the purest form of fantasy sports possible. Fantasy sports is about drafting players whose individual contribution is superior to other players at that same position — and a fantasy points scoring system should capture exactly that. Early on we made the decision not to award an arbitrary 100+ yards rushing or 300+ yards passing 3-point bonus in the NFL, for instance. In our eyes, the 100th yard rushing shouldn’t count for 30 times as much as the 99th yard, and the 300th passing yard certainly shouldn’t count for 75 times as much as all the other yards before it. These bonuses act to artificially increase variance and decrease the importance of the skill aspect of the game.
So what did we find with goalies that made us leery of the win- bonus? Let’s get into the numbers. The first thing to observe is the distribution of actual goalie points scored using the old method of awarding a win bonus:
What does this show? This is what statisticians would call a ‘bi-modal’ distribution, or two distributions — one including goalies whose teams have won and one of goalies whose teams have lost. In this system, goalies on losing teams average 1.7 points; goalies on teams that have won average 7.2 points. There is no meaningful notion of a goalie having an ‘average day’ in the old scoring system — there are goalies on winning teams and goalies on losing teams, but there is no meaningful concept of a middle-ground goalie.
It’s possible that this is as it should be. Perhaps it’s the case that goalies are so pivotal to team performance that a team win might as well be attributed to the goalie and the goalie alone. If this were true, it would imply that a ‘win’ in the NHL is really just a handy proxy for excellent goalie play — we could preserve the win bonus happily knowing that we are awarding the most points to the goalies who do the most to help their teams perform.
To check this in the data, we need to compare known measures of goalie performance — things like save percentage, which is the percentage of shots on goal that a goalie manages to save — to see if these measures explain the majority of team wins / losses. If so, we could conclude that goalies have a large degree of control and responsibility over team level outcomes. If, on the other hand, measures of goalie performance do not neatly translate into wins and losses for teams, it implies that the win bonus isn’t a good measure of goalie performance, and awarding the bonus points to a goalie based on team wins is more or less an arbitrary scoring decision.
So we know that in the old system winning goalies score roughly 420% more fantasy points than losing goalies, on average (1.7 for goalies on losing teams; 7.2 for goalies on teams that win). How do these two groups compare in saves, goals allowed, and save percentage? Is there a 420% difference in other measures of their performance as a goalie?
Since 2012, winning goalies have had the following average stat-line:
27.9 saves, 1.7 goals allowed, 94.2% save percentage
Losing goalies, on the other hand, have averaged:
24.8 saves, 3.2 goals allowed, for an 87.3% save percentage
For reference, if we ignore wins and try to get at how the ‘average goalie’ might perform, it’s something like:
26.4 saves, 2.4 goals allowed, for a 90.7% save percentage
As we expect, goalies on winning teams do, in fact, outperform goalies on losing teams. But not by 420% on average; not even close. This is the first strong evidence that the win-bonus is either arbitrary or, at best, inflated. What seems to be going on here, at least in part, is what’s known as a ‘third-variable problem’ tying goalie performance to a team win. In a perfect third-variable problem, two totally unrelated things are correlated to one another because of a third thing that’s related to both. For example, cities with more churches also have more murders each year — but a third variable explains both: population. In the case of goalies and team wins, the situation isn’t so dire. But we have to ask exactly how causal the relationship is and if there are other factors at play. Are the two related? Yes, obviously. Is team performance 100% caused by goalie performance (or vice versa)? That’s a very different question. The old system of scoring just assumes that such a strong tie exists. But if that goalie performance isn’t 100% (or even 50% +) causally tied to teams’ wins or losses, then attributing goalie points for a team’s win/loss would, at best, muddy up goalie selection, and, at worst, bring about some insane realities for selecting top-scoring goalies each night.
For example: what is one of the most strongly correlated statistical categories related to goalie scoring in the old system? If you are following me so far, hopefully you guessed it – the goalie’s team’s score! A goalie’s team’s score explains 30% of the variation in a goalie’s fantasy points production in the old win-bonus-based scoring system. Does that make sense? The implication is that a major consideration when you select your goalie ought to be the quality of the offense on the team for which he plays. In terms of causality, this is quite a stretch.
Following this train of thought further leads to all sorts of absurdities. For instance, in games using the old system, if the goalie’s team’s offense puts up more than 3 goals, goalies are awarded an average of 5.71 points. If their team’s offense puts up less than 2 goals, the goalie scored an average of 2.8 points. In other words, the difference of nearly 3 fantasy points for the goalie (coincidentally, roughly the size of the traditional win-bonus) is accounted for if you can predict whether a team’s offense will have a good or bad night. The distribution below separately shows the fantasy points produced by these two populations — fantasy points for goalies on teams that scored less than 2 goals in a game are in orange, and fantasy points on teams that scored more than 3 goals in a game are in blue.
Notice that the overlap of these two distributions is very small — the bulk of the blue distribution is to the right of the bulk of the orange distribution. Translated into English, this implies that in the old scoring system, goalies on teams whose offenses score more than 3 goals almost always outperform goalies on teams whose offenses score less than 2 goals, ignoring all goalie-specific stats. And things get even weirder…
Since we know that one team’s good night will depend on poor goalie play by the opposition, the old win-bonus-driven system rewards goalies that play against weaker opposing goalies. Think about that for a second. Playing an opponent with a weak goalie is just as significant in the calculus of whether or not your team will win a game as is having a terrible goalie on your own team (or, in the case of fantasy goalies, being a terrible goalie yourself). Daily fantasy hockey players who were looking to accurately predict goalie points in the old system would rationally want to look at the strength of the opponent’s goalie to predict how the goalie of the other team would fare. I have no idea why the opponent’s goalie’s performance would have anything to do with a fantasy goalie’s performance, but these are the types of things that happen when you arbitrarily award such a huge chunk of points to goalies for their team’s wins.
If you aren’t convinced that the win-bonus is arbitrary based on the high-level data, it might help to look at the details of some of this year’s actual games. The highest scoring goalie performance on a losing team this year (as of Nov. 1) belongs to Seymon Varlamov who, on Oct. 29, allowed only 2 goals on 49 shots against in the Avalanche’s loss to the Sharks (96% save percentage). In the old scoring system, this earned Varlamov 7.8 points. This gave him .2 points less than, for instance, Jaraslov Halak’s Oct. 14 performance when he allowed 3 goals on 40 shots against (93% save percentage) in the Islanders’ win over the Rangers. Is Halak’s 40 save, 3 goals allowed really a better individual hockey performance than Varlamov’s 49 save, 2 allowed?
At the other end of the spectrum, things get even uglier. The lowest scoring winning goalie this year is Marc-Andre Fleury of the Penguins whose offense pulled out a win against the Devils despite the fact that Fleury allowed 3 goals on only 16 shots on goal (a putrid 84.2% save percentage). His 3.2 fantasy points for that win puts him above all of the “not-that-bad” goalie performances; goaltenders who allowed fewer goals on more shots against, but happened to be on losing teams. A few examples are Pekka Rinne on Oct. 15 against the Flames (26 saves, 2 goals against, 92.9%) and again Varlamov on Oct. 26 against the Jets (also, 26 saves, 2 goals against, 92.9% in the loss). The list goes on.
In between these two maximums for this year are all sorts of terrible fantasy-point attributions in the old system. Like the 3 times so far this year that a goalie has won despite allowing 5 goals in less than 35 shots on goal, ending up with above average fantasy numbers. Or the 7 times this year when a goalie has allowed one goal against and lost — sometimes going perfect in regulation and losing in a shutout — to end up with below-average fantasy numbers. These were mostly above average to excellent goalie performances that were scored, for fantasy purposes, as cut-rate below-average nights.
In light of all of this analysis, StarsDraft released an NHL product awarding an increased .3 (up from .2 to compensate for a lack of win-bonus) points per save to goalies, preserved the shutout bonus, and decided not to award a win-bonus. Under this new scoring system, the distribution of goalie fantasy performances looks like this:
Notice how smooth this distribution is compared to the lumpy bi-modal win-bonus distribution. This distribution looks a lot closer to what one expects of an athlete’s performance: there are a lot of average days, and the more extraordinary performances are less likely (both good and bad).
No matter how much sense this makes, people will always take umbrage to such a fundamental change in the way any daily fantasy sport is played. Let me anticipate more questions that will come from those that insist the win-bonus is not arbitrary, and that by getting rid of it, StarsDraft is ruining NHL daily fantasy sports.
Issue #1 – Without a win bonus, goalies won’t score any points! How can StarsDraft have them priced so high in their salaries?!
Answer — We award additional .1 points per save to the goalie. As a result, using StarsDraft’s scoring, the average goalie, without a win bonus, scored 5.65 points compared to 4.45 fantasy points on average with the old win-bonus system. Like QBs in the NFL, NHL goalies are on average dramatically more consistent and higher scoring than the other positions. This fantasy points difference appropriately translates to the increased importance the goalie has on the ice. The higher salaries are a natural result of the goalie position’s lower variance, higher average fantasy output.
Issue #2 – Without a win bonus, losing goalies on teams who get tons of shots against will be worth so much more than winning goalies! Why wouldn’t I just roster goalies on terrible teams every night?!
Answer — Go for it. But before you do, you might want to stop and think about whether your assumptions actually play out in the data. We can help. Below is a distribution, using StarsDraft’s goalie scoring system, of fantasy performances by goalies on winning teams (blue) vs. those on losing teams (orange):
In the StarsDraft scoring system, losing goalies scored 4.25 while winning goalies scored 7.07 fantasy points (on average). It’s still much better to have a goalie on a winning team in the StarsDraft scoring system — just not 420% better. This much more accurately reflects the importance of the goalie’s performance to the win/loss performance of the team. It’s highly significant (on average, it’s worth about 3 fantasy points, depending on the goalie’s actual performance stats for saves and goals allowed), but it is still possible to play well at the goalie position and be on a losing team.
Issue #3 – Yeah, ok, so this makes sense. Why do you care? Fantasy hockey has been around for a long time — why mess with it now?
Answer — This is probably the only legitimate gripe on our approach to NHL. This complaint falls into the general bucket of users who dislike any sort of change in DFS. Why would we allow 6 forwards instead of sticking to the more restrictive, traditional 2LW, 2RW, 2C setup? Why would we not award points for penalty minutes? Why would we change anything at all about daily fantasy hockey? For people who think the old ways are always best, there’s not a lot StarsDraft can do to lure them away from wherever it is they currently play. Our response to these types of complaints is always the same: if we thought that DFS had already been perfected, we never would have entered the space. Of all the daily sports, NHL has been perhaps the most neglected, receiving the least deep thought in terms of how it ought to translate from a season-long format. In daily, the game is different and requires a different thought paradigm.
We will always be responsive to the voice of our users. If, at the end of a full NHL season, the users still insist that the lack of a win-bonus detracts dramatically from their experience, we will have to reconsider our position. From our perspective, the daily fantasy sports industry is still very much in its infancy, and that includes how the game itself is played. Adjusting the way that the goalie position is scored in hockey is peanuts compared to the changes StarsDraft sees coming, and intends to drive, in the months and years to come.