Lies, Damn Lies, and…Torin Koos?

Ok, so I lied in the title. This post has almost nothing to do with Torin Koos.

The whole reason I started this blog (or, at least, one of them) was that it seemed to me that folks in the XC skiing world had all sorts of weird beliefs about their sport that ought to be easily checked if someone would simply collect the necessary data.

Not that these beliefs floating around would always be wrong, but that they always seemed based on personal experience or on a very limited sample. And despite this, everyone seemed very certain about themselves.

Case in point: like everyone else in the US skiing world, I have been following the comments at FasterSkier on all the posts related to Torin Koos’s disqualification at US Nationals. I obviously wasn’t at US Nationals and have no opinion on whether the DSQ was justified or not. (Although, major props to FIS for working the word ‘automatism’ into their official report.) But as with any good controversy on FasterSkier, I pulled up my chair, made some popcorn, and have been enjoying the comments mainly for their entertainment value.

Until this Monday morning, that is, when Marty Hall, a coach that I do not know, but have a lot of respect for, said this:

I can guarantee you if you were to go and check a ton of US and Canada results from sprints in Europe 80 to 90 % of the time our kids go backward fron their qualifying postiion [sic].

Now, I want to be very clear that my aim here isn’t to call out Marty, who I’m sure knows plenty more than me about skiing and coaching. I certainly wouldn’t know if European racing is “rougher” or “tougher” than racing the US, so I have to take the word of the folks who have experience with it.

But as it happens, I can actually check the results Marty refers to. And he’s wrong.

Let’s stipulate that the only results we’re interested in for the purposes of verifying Marty’s claim are those where a US or Canadian skier qualified between 29th and 2nd in a race in Europe. Why 29th and 2nd? Only those folks are capable of either moving up or down from their qualifying position. (We’re restricting ourselves to results since 2005-2006, when the switch to 30 people in the heats took place.)

So we grab those results, and subtract their final position from their qualifying position. A negative value means they moved backwards. What do we get?

For all such results, North Americans moved backward 45.3% of the time, stayed the same 11.1% of the time and moved up 43.5% of the time.
For only the WC, OWG, WSC and TdS results, North Americans moved backward 50.0% of the time, stayed the same 5.5% of the time and moved up 44.4% of the time.
For non-WC/OWG/WSC/TdS results, North Americans moved backward 40.4% of the time, stayed the same 17.0% of the time and moved up 42.6% of the time.

Values may not add to 100% due to rounding. In general, US and Canadian skiers seem to move up or down from their qualification place more or less the same amount. The biggest discrepancy is for the WC level results, but even then we’re nowhere close to 80-90%. We can’t even get over 60% by combining the moved backward and stayed the same categories.

But maybe these numbers are still far worse than what the Scandinavian skiers do?

Nope.

I reran this same analysis for Norway, Sweden and Finland and the numbers were basically indistinguishable. They all move down ~40-45% of the time, stay the same 5-15% of the time and move up 40-45% of the time. Even the general pattern is the same, with the WC level Scandinavian skiers moving down closer to 50% of the time and a lower percentage staying the same (~7%). So even if you considered the numbers above for US and Canadian skiers to be bad, they aren’t materially different than those for Scandinavian skiers in the same races.

But wait, there more!

A sophisticated reader may object that we should be using a relative measure, not an absolute one. Skiers who qualify 2nd are much more likely to finish further down the results, simply because there’s only one option for improvement. Likewise, those qualifying in 29th are more likely to do better.

So instead of simply counting how often North American skiers move up or down after qualification, maybe we should compare them directly to Scandinavian skiers, but control for the qualification result. So now our question is, are North American sprinters more or less likely than Scandinavian skiers to move up or down from their qualification result when racing in Europe?

This calls for a full on regression model. There are plenty of options, but I went with a multinomial model on movement (up, down, stay the same) versus continent (North America vs Norway/Finland/Sweden) while controlling for the qualification result itself. (I did some due diligence as well, checking a handful of other variables to see if they had an impact, but none of the obvious things seemed to effect our question of interest.)

The following table summarizes the fitted probabilities of moving in each direction for someone who qualified in 15th (the general pattern is the same if you focus on 5th, 10th, 25th, etc.).

	Qualification	Worse	Same	Better
North American	15th	0.462	0.120	0.418
Scandinavian	15th	0.384	0.185	0.432

The results in this case are considerably more subtle to interpret. Note that the probability of improving for both the North Americans and Scandinavians is basically the same, ~42%. The Scandinavians are modestly less likely to do worse after qualification (38% vs 46%) while they are more likely to finish in exactly the same spot (19% vs 12%).

First, its important to note that we’re still not even in the same solar system as the 80-90% numbers we started out checking. Here though, there might be some evidence of a small difference between the North Americans and the Europeans, depending on how you view the “Same” category.

If we lump “Same” in with “Worse”, the North Americans and Scandinavians are nearly identical. Lumping “Same” in with “Better” would suggest a modest advantage to the Scandinavians in their ability to do as well or better than their qualification result.

In my “expert” opinion, I’d call this some evidence for a modest difference in the ability to ski heats in Europe between North Americans and Scandinavians. However, it is clearly not the case that North Americans almost always do worse in the heats, as suggested above.

Again, my point here isn’t to criticize Marty in particular. I see this sort of thing regularly, and heck, we ended up doing some reasonably complicated maths.

But if you find yourself professing a strong belief about XC skiing that is reasonably quantifiable, stop and ask yourself: why do I think that? Is it just my impression, or do I have data to back it up?