Individual offense rating - no solution yet
During the weekend I decided to write down the definite answer for individual offensive ratings. After spending way too much time without being able to formulate even one paragraph for that post, I was forced to downgrade my objective. Now I just hope to start some discussion on this topic. Quite lame, I have to admit.
Different formulas have been suggested in many places (see for example FrisbeeStats and rec.sport.disc for a sample). I should probably dig all different examples in here to provide background info… maybe later.
[Update: forget the multiplier 2 in the following paragraps]
The simplest scenario: only scoring information (assists and goals) and turnovers are recorded. The first shot would be to use a) “goals + assists - 2 * turnovers”. The “2″ multiplier in front of turnovers comes from a fact that there are two ways to end a possession: scoring a goal or turning the disc over. As two points are rewarded for each goal (goal + assist), also each turnover is worth 2 points. (This is way different from Bill Mill’s suggestion of using a multiplier .8 -although Bill’s multiplier is based on opponent scoring efficiency… or close to it).
a) is most likely too simplistic. I think it would be better to add together percentages, leading us to b) “goals/total_goals + assists/total_goals - 2 * turnovers / total_turnovers”. In here “total_goals” represents the total number of goals scored by the team and “total_turnovers” represents the total number of turnovers by the team. Note, that if the team turns the disc over many times, committing one turnover does not affect your rating very much. However if your team commits only 5 turnovers in a game and you commit one of those, you are set back 40 percentage points.
The good point in b) is that with this formula one can compare the relative value of players from different teams: who is most valuable to her team. The bad point is that the real point value of one’s efforts is unknown. Of course this could be achieved by multiplying the whole formula with total_goals, giving us c) “goals + assists - 2 * turnovers *total_goals / total_turnovers”. (This is starting to look nasty without proper formatting)
c) probably looks more familiar than b) to many (measuring the value of a player with a scale close to “points” instead of being a arbitrary number between -200 and +100 much like football’s passer rating). Again there is a downside; players in good teams (scoring many goals) tend to have higher ratings than players in bad teams.
By adding points played, one gains a little more perspective. a) and c) can be “normalized” by dividing the formula with it. b) is a little more complicated.
Another step forward would be jotting down the intended receiver for all turnovers. For point blocks - subtract two points from the thrower (remember the multiplier 2 in the formula a). For drops - subtract two points from the receiver. Other types of turnovers are more hairy. Block usually is caused by a bad throw, therefore most of the 2 points should be subtracted from the thrower, but usually also the receiver is at fault, so -1.5 / -0.5 from thrower/receiver might be better estimate. Stalled disc - arguably all throwers fault. However if nobody is open, how can one pass to others; should all the players on the field get a deduction.
September 21st, 2005 at 7:59 pm
Assume that for the 3 points I’ve been in, I threw 2 scores and turned the disc once. Why should I get a zero rating? Haven’t I been a net positive for the team? We’re up 2-1 on my points. This is the problem I have with the multiplier of 2 - why doesn’t this argument make sense?
September 21st, 2005 at 10:48 pm
Here’s what I did with some stats from the Purdue women:
1a) For a given game, calculate team scoring efficiency with the disc. Since every posession ends in a score or a turnover, this is just scores/(scores+turnovers). If you want to separate out O and D points (effectively turn it into two seperate games), you can do that, although I didn’t bother.
1b) Do the same thing for the other team.
2a) Using sums of geometric series, calculate the chance of scoring a point eventually if you have the disc:
Team_eff + Team_eff(1-Team_eff)(1-Opp_eff) + Team_eff((1-Team_eff)(1-Opp_eff))^2 + …
= Team_eff / (1 - (1-Team_eff)(1-Opp_eff)
2b) Same thing for when you don’t have the disc. Ends up just being (1-Opp_eff) times the above result.
3) Now, assign “true” fantasy values for scores, assists, and blocks. A score and assist take the chance of scoring from (2a) to 1, so the value (which can be split up between the thrower and the receiver as appropriate) is 1 - (2a). Similarly, a block’s value is (2a)-(2b), and a turnover’s is (2b)-(2a).
The result of all this work is that blocks and turnovers are the dominant factor in clean games, while scores and assists are the dominant factor in turnover-fests.
September 22nd, 2005 at 1:09 am
Now I am embarrassed. Should I change the blog title to “Correcting the Finn”? or “Clueless Finn”?
Bill, you are right that in this form the multiplier of two does not really work. I was too eager to get it into writing. I was not thinking from the perspective of an individual player. One player should not get two points for one turnover. Max one for individual. But shouldn’t there be (almost) always more than one player getting the credit?
I guess my point is that we should also keep score of the intended receiver of the pass (if that’s possible) and make her a little responsible for the turnover too. Or if it’s a perfect pass and one drops it, give only the receiver a turnover point and thrower is in the clear. (Well, that’s the common practice nowadays, so no news here)
I know that in the Team USA play-by-play stats there are some half-turnovers recorded (a bad pass and catch attempt did not succeed, I guess), but some kind division of a turnover point(s) should be there in case of a block too. Throwaways, mostly thrower. Marker block, only thrower. Stall, thrower 1 and rest of the line-up 1/6th? And so on.
And thanks Tarr for a math lesson.
September 22nd, 2005 at 3:32 am
Tarr, that looks beautiful, I’m going to try it out as soon as I get home.
Hartti, don’t be so hard on yourself. 2 is the number I started with, as I said in my comment on one of your old entries. It was only after that number didn’t jive with what I saw that I changed my thought on the number. It appears Tarr took what I was doing and brought it to its logical end.
Did I mention that looks beautiful, Tarr?
September 22nd, 2005 at 4:36 am
My team was getting into hardcore stats and analyzing every aspect of them. Then we discovered stats made the team care to much about individual performance and not about the team…(Young team filled with egos.. maybe on a veteran team it would be different.) It also might have led to blaming, scapegoats, and playing time issues.
September 23rd, 2005 at 6:42 am
I think it’s important to remember that you don’t actually have to show the team all their individual player stats. More often than not, stats are most useful for a coach or strategists crew to mull over and devise plans to improve performance based on what the stats illuminate. If applicable, the leaders that are privy to the stats can discuss something from the stats with the individual involved.
September 23rd, 2005 at 6:53 am
see here
September 27th, 2005 at 8:17 am
I hate to rain on this parade, but I haven’t really seen anything here which measures an individual’s impact on the offense. There are a few problems. For one, in the original post the tail is wagging the dog. You have to first decide what it is the mark of an individual’s offensive contribution — *then* try to find a measure which correlates well with the concept. You will likely find that there is no satisfactory overall measure, but that some are more meaningful than others. Second, ultimate stats and games are almost always “small N,” meaning the application of probabilities is suspect, and even more so when you apply them to successive turnovers as a geometric series.
How important is a goal pass? Was it a one-yard pass or a sixty-yard huck? Did the cutter bust free of his defender, or was he left uncovered in a zone? The only thing we can say for certain is that a goal pass traveled forward some number of yards (or yard). Note the difference from a hit in baseball (which can’t result from an error) or a TD pass from a quarterback (who at least marshaled his team down the field).
The situation is a bit like trying to define a Dow Jones average without having an idea of the stock price. I think we should first aim for a passing rating (something like yards thrown with a bonus for goals) and a receiver rating (something like yards caught with a bonus for goals), then define an individual’s offensive contribution as some combo (or sum) of these two. (Of course, we don’t have yard stats in ultimate yet, but we might soon be able to give a rough rating of short, medium, long for 1, 2, or 3 points.)
One could get more sophisticated by trying to quantify how an individual adds to the overall offense through “intangible” contributions. One way might be to find whether the team scores a greater percentage of the time when a given player is in than when he is out (i.e., you could multiply the previous index by the ratio of scoring percentages). This number might come closer to what you’re looking for.
To test, you can take videotapes of games and calculate the index for various players. If the obvious offensive stars come out on top nearly every time, then maybe it makes sense.
Hmm… this is an old thread. Maybe I’ll expand on this idea elsewhere….
September 27th, 2005 at 11:49 am
It will be tough to quantify the effect of a single player on a team - there are many ways to contribute without touching the disc.
When i was in college, our coach had a few stats he kept - offensive scoring percentage (ie, scores per posession) offensive power percentage (% of scores per first offensive possesion per point), defensive osp (%of goals allowed per opponent possession) and defensive power percentage (not allowing the opponent to score on their first possession).
it’s then easy to compare stats between players on the same time and also possible to create coefficient to compare players on different teams to each other (something similar to ‘park effects’ in baseball)
I think it’s a good overall system because most teams have goal scorers and goal throwers, but players who’s job it is to be a threat, work the disc through the middle, draw coverage, etc, are just as responsible for creating the goal as anyone else*.
September 27th, 2005 at 12:46 pm
Zaz, this pic tries to give some kind of idea of one’s value as a thrower and a receiver. http://www.hartti.com/IndividualOffenseRating2.jpg (Finland national women’s team in WUGC 2004 final against Canada)
The x-asis is based on “yardage”, y-axis on how one contributed on end of the possession. Dotted ends are players as receivers.
As an example Jenni scored a lot of goals and gained a lot of yardage as a receiver. Hence the blue end of the line is in the upper right hand corner. However her passes did not advance the disc as much. Hence the green end is lower. She could be categorized as a deep. Kossu and Reija are examples of handlers contributing positively (scoring and passing more goals than turning the disc over).
But still it gives no relative valuation of yardage vs. goals/turnovers.
September 28th, 2005 at 8:34 pm
“One way might be to find whether the team scores a greater percentage of the time when a given player is in than when he is out (i.e., you could multiply the previous index by the ratio of scoring percentages). This number might come closer to what you’re looking for.”
I have been trying to use something similar to this for years, but the problem that it always comes up against is the quality of the competition. That is, your second level offensive players come in against a weaker team and score 100% of the time vs. your top line scoring far less successfully against superior competition.
Specifically, if there is a role-player on the top offensive line who plays a bigger role on the second string, his scoring percentage will be inflated. this is in addition to the obvious second-string only player who will have a great scoring percentage.
Perhaps categorizing stats by quality of opponent? Difficult to be consistent though… Plus, the stats wouldn’t be relevant from team to team…
(I wish I had enough knowledge to truly follow all of the info you work with here)
September 30th, 2005 at 12:25 pm
Zaz, my primary objection to your approach is that it does not seem representative of anything in particular. You’re talking about a bunch of things that correlate positively with good offense, but when you’re multiplying a sum of yardages plus bonus points for scores times a ratio of team scores to scoring opportunities without any sort of theorietical model that supports it, it becomes clear that you’re not really measuring any thing. I suppose NFL passing rating is the same way, but then again I’ve always thought passing rating is an awfully stupid stat. A much better stat would be something like footballoutsiders.com’s defense-adjusted points above replacement player level. (Developed by a Brown grad, of course.)
I’ve developed two statistical tools so far in ultimate:
1) The adjusted fantasy numbers from above. I’m attempting to aapproximate every fantasy stat’s impact on the probability of scoring a point. Once I’ve done that, I total those up. Your score is the expected increase in goals scored you caused through your presence on the field.
The nice thing about this measure is that it works regardless of who is on the field with who. It’s also very easy to get probabilistic approximations here, because every event I’m considering ends a posession (by turnover or score). Its weakness is that it fails to measure things like clogging or letter your player get open long (or conversely, shutting your man down or creating space).
You could expand this approach to include all the statistics you mention in your blog entry, but this would require developing a reasonable answer to questions like, “how much does the completion of a 20 yard pass affect the probability of scoring on a given posession/point”? Not an easy question, obviously, but unless we have some answer there then including passing yardage in some sort of aggregated statistic amounts to a bunch of hand-waving (IMNSHO).
2) Most of my efforts have been developing another statistical tool, what I call adjusted plus-minus (or, when I’m working in rate form in stead of in counting form, relative efficiency). You can probably guess the basics, although I’ve mentioned it before on Idris’s blog, as well as the ultistats yahoogroup. The only things you consider are who is on the field, whether you start on O or D, who your opponent is, and whether you score. If I get around to it, I’d like to add adjustments for how close the game is at any given point, so that blowout points receive a smaller weight. All the times I’ve used this stat so far, I have not corrected for which teammates are on the field. Ideally, I’d like to adjust for that, which makes the calculation of relative efficiencies a overdetermined linear system of equations.
I think that this path of analysis is more meaningful, at least until we have a LOT of play-by-play data to work with, so we can really measure the value of a swing or the cost of a covered in-cut. It’s biggest weakness is that two players can only be distinguished by the points where they are not in together. If Jim and Count are in all the same points, then they will appear identical by this measure even if one plays terribly and one plays great.
September 30th, 2005 at 12:42 pm
DDT,
You are absolutely right that adjusting for opponents’ strength is crucial to any good analysis. Both of my approaches do this.
Here’s an example: I computed the Purdue Women’s team statistics for Great Lakes regionals using the method I posted on the 21st.
Lucy threw 6 goals with no turnovers in Purdue’s 11-1 win over Case Western. For that game, Lucy’s fantasy score by my system was +.91. In other words, despite throwing SIX goals, she didn’t manage to even increase our expected score by ONE goal. Why? Because if those are all throwaways, we probably get the disc back anyway.
By contrast, Kelsey caught two scores and had 3 blocks (with no turns) in the 10-9 win over Illinois (our closest of the weekend). That performance was worth a score of +4.6, demonstrating how important every one of those plays was to the final outcome.
The adjustment for opponent strength in the adjusted plus/minus calculations is a little more explicit — a player’s relative efficiency for a given game is just the team efficiency while he’s on the field, divided by the team efficiency overall. If you do well in a blowout, well, that’s nothing special.
September 30th, 2005 at 9:42 pm
I just wanted to chime in here, since it’s a stats discussion.
We really ought to start up the stats groups again.
I wanted to add that yardage, possession, position, and scoring goals are all important in offense.
One little flaw in Tarr’s method is that it assumes the team plays equally well in all games, so a 13-7 win against a good team in which the starters play well gives as much total credit as a 13-7 win against a bad team in which the backups play like backups. Weren’t you going to try to incorporate an a priori estimate of the other team’s strength?
October 1st, 2005 at 12:47 am
Yes, I was going to do that. I haven’t actually run any of these stats in a while, so I haven’t really refined my techniquess much lately. I should get the Purdue guys to keep feeding me their stats - they use the palm stats pretty consistently so the data is fairly good.
There’s a few obvious approaches to doing this estimate of how good the team SHOULD do against an opponent:
1) Past scores against that team. This has some value, but if you consistently coast against some bad teams and give up more scores than you should, this won’t correct for it.
2) Use the predicted score from RRI. Works a lot better late in the season.
The best approach would probably be some sort of weighted combination of these, along with the actual score of the game.