Goaltender Perseverance: a meaningless stat
Copyright Iain Fyffe, 2002
This essay examines Chapter 8 of Klein and Reif's Hockey Compendium, which is all about goaltending. This chapter makes up pages 131 to 159, and fully 22 of these pages (76%) is taken up by the discussion of a Klein and Reif (KR) statistical creation, Goaltender Perseverance.
Unfortunately, all of these 22 pages are wasted. Goaltender Perseverance is a meaningless and useless concept. But don't take my word for it; read on and you'll discover why.
KR begin their discussion with the appropriate point that traditional goaltending statistics (that is, Goals-Against Average and Won-Lost-Tied records) are deceptive a best:
"The statistical method hockey has used since the very beginning of the game for determining who is a good goalie and who is a bad one is all wrong. Goals-against averages, they're called, and as we all know, you figure out a goalie's GAA by taking the number of goals he allows and dividing it by the number of full games he plays. This is patently ridiculous…" (p.134)
This statement is true; GAA is of very limited meaning for evaluating goalies. However, KR have a problem with it because it evaluates goalies based on something that is more a measure of team defence (goals against), rather than the individual goalie's performance. This is rather ironic, because Goaltender Perseverance also evaluates goalies based upon something that is more a reflection of team defence (shots faced), rather than individual goalie performance. I will demonstrate this, but first, we will examine KR's rationale.
KR move on to Save Percentage, which is a much better number for evaluating goalies. Unfortunately, the NHL has officially tracked this figure since 1982-83, even though it had been invented decades earlier.
But KR are not content with Save Percentage as it is. Their reasoning is as follows:
"The more shots you face, the less likely the chance that you have time to get set for each shot, to be in position for each shot, to see each shot." (p.137)
That is, if you face a larger number of shots per game, your save percentage will be lower, on average. This is simply a statement KR make, and it forms the entire basis for their Perseverance index. They make no attempt whatsoever to prove that the statement is true. They present no evidence that the fiftieth shot in a game is more likely to produce a goal than the fifth shot. In fact, their claim is entirely false.
To disprove their claim, I will use only the data KR themselves had: the official NHL shots figures for 1982-83 to 1986-87, and the 1981-82 numbers they compiled themselves. Their argument is that as a goalie's shots per game increases, his save percentage decreases. This is easy to test. I took the shots per game and save percentage figures for these six years, using the top tier of goalies as identified by KR (that is, those playing 1600 or more minutes in a season). I then calculated the correlation coefficients between shots per game and save percentage. If KR's premise is correct, then a higher number of shots per game should produce a lower save percentage. Thus, the correlation coefficient should be a significant negative one (say -0.40 at a minimum). The results are as follows:
| Year | Correlation |
| 1981-82 | 0.17 |
| 1982-83 | -0.32 |
| 1983-84 | 0.15 |
| 1984-85 | 0.14 |
| 1985-86 | 0.25 |
| 1986-87 | 0.08 |
In none of the years is there any sort of strong negative relationship. Which is to say, the number of shots you face does not have a negative effect on your save percentage. Five of the six years have a slight positive relationship, meaning that as shots increase, save percentage increases. I have a theory as to why this might be, but have not yet tested it; the positive relationship is quite small at any rate. In summary, KR's premise is entirely false.
There is further evidence that the Perseverance Index is a meaningless concept. As an index, it is designed to rank players, but the actual number itself has no meaning. There is nothing inherently wrong with indexes. However, an index must be based upon solid premises and logical reasoning. Perseverance is not; it is a meaningless, arbitrary mishmash of numbers. Here is the formula (p.138):
Perseverance = (6 x (Saves/Shots x 100) + Shots per 60 minutes)/.6
Why 6? Why .6?
"It [the formula] is based on our perceptions, and in that sense it is subjective to some degree." (p.138)
This last statement is not true. The formula is entirely subjective. It was admittedly arrived at by trial and error, with the end result affecting the formulation until a result that "looked right" was found. Arriving at any metric, even an index, by trial and error opens the door for personal biases to enter after examining the results, but this is beside the point.
The entire point of Perseverance is to reward goalers for facing a lot of shots (or, conversely, penalizing them for facing few shots). This makes the rating meaningless, because the number of shots faced by a goalie is beyond his control; it is a function of his team.
To prove this, I again used only the data KR had access to. In this case, I could not use 1981-82, because the splits for traded goalies are not provided by KR. I calculated the correlation coefficients between teams' starting goalies and backups for each year. If there is a high degree of correlation (0.60 or more), then there is evidence that shots faced per game are a function of team. The results:
| Year | Correlation |
| 1982-83 | 0.77 |
| 1983-84 | 0.67 |
| 1984-85 | 0.89 |
| 1985-86 | 0.71 |
| 1986-87 | 0.70 |
Each and every year, there is strong evidence that shots are a function of team; meaning that the number of shots a goalie faces depends on which team he plays for. Thus, KR are actually evaluating goaltenders partly based on which team they play for. Talk about patently ridiculous!
KR spend 22 pages gushing over their creation, which (1) is based on a false premise, and (2) evaluates players based on something that is beyond their control. In short, Perseverance is useless and meaningless. This is a noble cause, trying to find new ways to evaluate goalies. However, this failed attempt shows that more work needs to be done.
If you have further interest in the subject of evaluating goaltenders, there are other method available besides save percentage; for example, Neutral Winning Percentage and the Point Allocation system.
Reference
Klein, J. and K.-E. Reif. The Klein and Reif Hockey Compendium. Toronto: McClelland and Stewart, 1987.
No comments:
Post a Comment