I saw this survey a while back, but had forgotten about it. Now I see they’ve written up a little summary that has been published both on SkiTrax and FasterSkier. The actual survey can be found here.
Why am I writing about this? Well, the summary includes some basic statistics. One of the things that is required (upon pain of death!) of people who have pieces of paper hanging on their wall claiming some sort of expertise in statistics is that we must always push people to do a good job of explaining data to people, and I think that in this case some major improvements could be made.
If you think I’m gearing up to “debunk” this survey, or slam Dr.’s Hoenig, Brams and Stock for doing shoddy work, you couldn’t be further from the truth! These guys boldly went out and collected some data on a medical/training issue that’s of growing importance to xc skiers. For that they should be commended. I am neither a physician nor an expert in physiology or training. But I am very good (I hope!) at explaining data to people in ways that are clear and avoid misunderstandings, and in this case I’m worried that their summary could use some editing to avoid confusion.
Let’s start with how the purpose and goals of this survey were originally described:
No one really knows how big or little a deal [compartment syndrome] is. So we are conducting a prevalence study. We want to know how many xc athletes have CECS in the USA and Canada, whether there is a particular geographical distribution, as association with gender, body mass index, use of creatine, association with training and stretching pattern, etc,” explained Hoenig. While he admits that the survey will likely reveal more questions than answers, he’s hopeful that they’ll be the right questions.
“With good data we will hopefully be able to answer some of these questions with pretty reliable numbers. These numbers will help us to define the problem, follow the problem over years (with repeat surveys) and further motivate us to look for prevention and treatment modalities.”
(Italics mine.) I think this is excellent. We’re being clear that this endeavor is exploratory, and that the primary goal is to identify possible hypotheses to investigate more formally down the road.
Ok, now let’s turn to the summary that I read over at SkiTrax. They run through some basic descriptive statistics (geography, gender, etc.) and then near the end we get two statements that feel an awful lot like “conclusions”:
The teams that trained at >500 hours/year had a statistically significant (p value 0.06) higher prevalence of CECS, than the teams that trained < 500 hours/year. Those teams with a higher average vertical gain achieved during skate roller-ski workouts had a statistically significant (p value 0.04) increase in CECS. (The survey did not look at vertical gain during classic roller skiing.)
I’d like to re-emphasize that I’m talking mostly about presentation here, not substance. Note that the original description of the purpose and goals of this study suggested (correctly) that this kind of survey is best used to identify/refine questions, not to establish firm conclusions. Yet these two results are presented with no qualifiers or discussion of their possible strength or relevance.
To people unfamiliar with statistics, these sorts of statements can contain a distinct air of finality and official-dom that I doubt that the authors would really claim. Let’s try to unpack these two conclusions a little bit and see if we can’t re-write those two sentences in a way that’s less likely to lead to confusion.
First: The teams that trained at >500 hours/year had a statistically significant (p value 0.06) higher prevalence of CECS, than the teams that trained < 500 hours/year.
I’m not going to quibble (as some might) about a claim of statistical significance with a p-value of 0.06, when values under 0.05 is usually what people consider significant. Like I said, I’m going for style here, not substance. To unpack what this means in the context of the survey, we need to go back and look at what the survey actually asked.
The first alarm bell for me is the formulation that described “teams that trained >500 hours” rather than individuals. Wouldn’t it make more sense to relate the prevalence of CECS to an individual’s training load rather than to their team’s? Well, it turns out that the only question in the survey regarding total training hours was phrased thusly:
-What is the range of hours per year that your top 10% (most committed) athletes put into training?
There were five possible responses (<100,100-300,…,>700). So it turns out that the relationship they are testing here is actually between the number of hours trained by a team’s 10% most committed athletes and the proportion of athletes on the team who developed CECS. This seems very different to me than what a casual reader would take from this conclusion, namely that individuals who train more than 500 hours are more likely to develop CECS.
Additionally, I would want to make the following concepts clear,
- As a voluntary response survey, no causal relations can be gleaned safely from these data.
- Further, even if the data allowed us to make a causal claim, it is unlikely that higher training hours alone causes CECS. A more reasonable hypothesis would be that there is a specific training activity or behavior that is causing or exacerbating the problem, and so we would naturally expect athletes who train more hours are also engaging in this specific activity more often.
How could we incorporate these qualifiers into that conclusion to make things more clear? How about this:
The teams whose top athletes trained at >500 hours/year had a statistically significant (p=0.06) higher prevalence of CECS than the teams whose top athletes trained <500 hours/year. Naturally, this sheds little light on any potential relationship between an individual’s training load and the risk of CECS, but our data do not allow us to examine this question precisely or to make causal claims. However, this result is consistent with what we would expect from injuries associated with high volume training (increased activity leads to increased risk of injury) and suggests that it may be fruitful to focus on investigating what specific activities or training levels are associated with CECS on an individual level.
I’m moving a bit beyond my realm of expertise with the language about overuse injuries, but I feel like the spirit of that is mostly correct and could be cleaned up by a physician if they object to CECS being referred to as an “overuse injury”. Still, I think this simple addition of 2-3 sentences makes it much less likely that people will read their conclusion and go “ZOMG! Must not train more than 500 hours!”. Which, believe me, if you’ve tried to explain statistics to as many people as I have, someone will do that. Trust me.
Second: Those teams with a higher average vertical gain achieved during skate roller-ski workouts had a statistically significant (p value 0.04) increase in CECS. (The survey did not look at vertical gain during classic roller skiing.)
Once again we have the confusing formulation about teams, rather than individuals. The reason is the same, as the only question about vertical gain in the survey was phrased as follows:
-If your athletes are skate roller-skiing what is the approximate vertical gain that your top 10% (most committed) athletes will achieve in a training session?
with six possible responses (0-100ft, 100-300ft, etc.). So again, we need to be careful relating a quantity measured on a particular subset of the athletes (the top 10%) with the prevalence among all the athletes on the team and the same issues about causal relationships apply here as well.
One last issue I want to highlight on this conclusion (it applies to the entire survey really) is the question of whether the respondents were able to accurately estimate the average vertical gain by their top 10% of athletes in skating rollerski workouts. Is that a number that you feel like you have a good intuitive handle on? I’d be willing to bet that most people (even athletes/coaches) would have a tough time estimating vertical gain if I physically put them on a road, had them ski on it and asked what the vertical gain was. I’m willing to let this slide for estimating total hours trained; it seems plausible that coaches would have at least a good sense of this, one would hope. But I seriously doubt that many coaches are meticulously tracking the vertical gain traversed by their athletes.
How might I rephrase this one? How about this:
Those teams whose top athletes had a higher average vertical gain achieved during skate roller-ski workouts had a statistically significant (p value 0.04) increase in CECS. (The survey did not look at vertical gain during classic roller skiing.) Again, given the nature of our data we cannot make any causal claims about this relationship. Additionally, we should note that it is reasonable to question the ability of our survey respondents to accurately estimate average vertical gain. However, uphill skating has been suggested as a possible risk factor for CECS, and these data are broadly consistent with that hypothesis. This reaffirms that we should investigate more closely the relationship between uphill skating and CECS on and individual level.
Ok, personally, I think both of these rewrites are huge improvements. And not in the sense that Dr.’s Hoenig, Brams and Stock were wrong and I’ve corrected them. I’d be willing to bet that they actually agree with most of my clarifications here. The part that I hope I helped with is effectively communicating these results to the general public. Which, when it comes to statistics, is a lot harder than it looks!
Related posts:
- <Most Improved: Women’s Sprint
- <What Does It Take To Win A Medal? (con’t)
- <The Quick Way To Stay On Top Of FasterSkier Comments
