Jump to comment:

• Response to: “Potentially Biased Results on Mouthguard Use and Reduction of Concussion Risk”
  Brent E. Hagel, Carolyn A. Emery, Luz Palacios-Derflingher, Kathryn J. Schneider, Amanda M. Black, Paul H. Eliason and Dirk A. Chisholm
  Published on: 1 July 2020
• Potentially Biased Results on Mouthguard Use and Reduction of Concussion Risk
  Nathan D Schilaty, Nathaniel A Bates, Aynsley M Smith and Michael J Stuart
  Published on: 1 July 2020

• Published on: 1 July 2020

  Response to: “Potentially Biased Results on Mouthguard Use and Reduction of Concussion Risk”

  • Brent E. Hagel, Professor Departments of Paediatrics and Community Health Sciences, Cumming School of Medicine, University of Calgary, Canada
  • Other Contributors:
    • Carolyn A. Emery, Professor
    • Luz Palacios-Derflingher, Senior Research Associate, Biostatistician
    • Kathryn J. Schneider, Associate Professor
    • Amanda M. Black, Assistant Professor
    • Paul H. Eliason, PhD Candidate
    • Dirk A. Chisholm, Medical Student

  We read with interest, and concern, the letter submitted by Schilaty et al arguing bias in our analysis examining the association between concussion and mouthguard use. Schilaty et al argue that a nested case-control study was not optimal and that “Based on a relatively small cohort, a complete case-control study would have been more appropriate than a nested case-control study.” They then go on to argue that “selection criteria of the non-concussion group biased the study as a random sample was not selected from the remaining cohort (n=2,040)” eliminating “from the analysis all non-injured players who wore mouthguards.” Finally, Schilaty et al contend that our study did not “properly compare the incidence of concussion between wearers or non-wearers of mouthguards.” There are multiple concerning statements and assertions made by the authors of the letter, Schilaty et al., that we will address below.

  Shilaty et al discuss the desire to compare “incidence of concussion between wearers and non-wearers of mouthguards.” Incidence cannot truly be estimated from a case-control study, given that the number of cases and controls is fixed from the design. Rather, we are after the odds ratio based on the ratio of the odds of exposure in cases relative to controls (the odds ratio of exposure is mathematically the same as the odds ratio of being a case). Modern conceptualizations of the case-control study invoke the idea of pseudo frequencies or quasi-rates related to construc...

  Show More

  We read with interest, and concern, the letter submitted by Schilaty et al arguing bias in our analysis examining the association between concussion and mouthguard use. Schilaty et al argue that a nested case-control study was not optimal and that “Based on a relatively small cohort, a complete case-control study would have been more appropriate than a nested case-control study.” They then go on to argue that “selection criteria of the non-concussion group biased the study as a random sample was not selected from the remaining cohort (n=2,040)” eliminating “from the analysis all non-injured players who wore mouthguards.” Finally, Schilaty et al contend that our study did not “properly compare the incidence of concussion between wearers or non-wearers of mouthguards.” There are multiple concerning statements and assertions made by the authors of the letter, Schilaty et al., that we will address below.

  Shilaty et al discuss the desire to compare “incidence of concussion between wearers and non-wearers of mouthguards.” Incidence cannot truly be estimated from a case-control study, given that the number of cases and controls is fixed from the design. Rather, we are after the odds ratio based on the ratio of the odds of exposure in cases relative to controls (the odds ratio of exposure is mathematically the same as the odds ratio of being a case). Modern conceptualizations of the case-control study invoke the idea of pseudo frequencies or quasi-rates related to construction of the odds ratio and what it means.[1,2,3] Regardless, the control group in a case-control study is intended to reflect the exposure (e.g., mouthguard) use in the source population that produced the cases (e.g., concussions).[1] We feel strongly that our control group of non-head injuries reflects well the mouthguard use experience in this source population and that it was sampled independently of exposure status.

  Schilaty et al argue that non-random selection of controls resulted in bias because it “eliminated from the analysis all non-injured players who wore mouthguards.” We agree that if all non-concussed players were wearing mouthguards, then this would have biased our results. However, Schilaty et al don’t seem to appreciate that some of those non-injured players who would have been selected in a random sample would have been using mouthguards and importantly, some would not. There is nothing inherently biased in sampling non-concussion, but still injured controls who may more accurately capture the mouthguard experience of the source population that produced the cases. It is this split of the proportion in the mouthguard exposed and unexposed groups we are after to tell us what the expectation should be for the cases, under the null hypothesis of no effect of mouthguards. That is, “we want the control group to provide estimates of the relative size of the denominators of the incidence proportions or incidence rates for the compared groups.”[1] Also, it is what allows us to calculate the odds ratio as the measure of effect in a case-control study. Ironically, if the bias Schilaty et al propose did in fact exist (i.e., that all or even a larger percentage of non-selected non-injured controls were using mouthguards), it would have resulted in an even more protective mouthguard effect than what we found-no additional analysis required!

  It is very interesting that Schilaty et al are asking us "to estimate mouthguard compliance among youth players and use the estimated trends for all control subjects”. We discussed in the paper that our investigation was conducted within cohorts assembled for other reasons. As such, we had no opportunity to sample controls in a random fashion as Schilaty et al proposed. The very reason we wanted to use the methodologically rigorous approach we chose was to avoid making assumptions about mouthguard use for the entire cohort (given that mouthguard use was the main exposure) and address the potentially important issue of confounding by other factors (e.g., mechanism of injury). The analysis the authors of the letter are arguing for would likely result in an effect more biased toward the null (no effect). This would not be because of a superior sampling approach, but because of misclassification bias of mouthguard use among controls. A serious issue to be sure, and one that Schilaty et al even acknowledge as adding “an additional limitation to the study.”

  We are concerned that Schilaty et al are mixing up the issue of random error and systematic error. More subjects in what they describe as a “complete case-control study” would potentially reduce random (i.e., statistical) error, but in isolation, would have no effect on the systematic error of the estimates. That is, a larger sample may provide a more precise estimate (narrower confidence limits), but it does not guarantee less bias.

  Regarding the issue of causality, we agree more work is required to understand the primary mechanisms of concussion and how mouthguards may reduce the risk. However, this was not the focus of our study and we were simply outlining potential explanations. We do agree that there continue to be gaps in the evidence base that need to be addressed from a causal mechanism perspective.

  In summary, we clearly outlined our methodological and conceptual rationale for the rigorous case-control approach we used in our paper and acknowledged limitations. Our approach was feasible and produced what we argue is strong evidence on the mouthguard-concussion relationship. Schilaty’s approach would involve artificially estimating mouthguard use for thousands of children and sampling from that distribution at, presumably, the time of a concussion. This would add significant limitations to the analysis in terms of capturing mouthguard use and key covariates essential for addressing potential confounding. We respectfully disagree with the approach suggested by Schilaty et al and feel it would be much less methodologically and analytically defensible than what we outlined in our paper.

  REFERENCES
  1. Rothman, Greenland, Lash. Chapter 8: Case-control studies. Modern Epidemiology. Third Edition. Lippincott Williams & Wilkins, 2008.

  2. Miettinen OS. Etiologic research: Needed revisions of concepts and principles. Scand J Work Environ Health 1999; 25 (6, special issue):484-490.

  3. Miettinen OS. Epidemiological research: Terms and concepts. Springer, 2011.

  Show Less

  Conflict of Interest:
  None declared.

  • Back to top
• Published on: 1 July 2020

  Potentially Biased Results on Mouthguard Use and Reduction of Concussion Risk

  • Nathan D Schilaty, Assistant Professor of Physiology; Physical Medicine & Rehabilitation Mayo Clinic
  • Other Contributors:
    • Nathaniel A Bates, Assistant Professor of Biomedical Engineering
    • Aynsley M Smith, Associate Professor of Orthopedics; Physical Medicine & Rehabilitation
    • Michael J Stuart, Orthopedic Surgeon; Professor of Orthopedics

  We read the referenced article by Chisholm et al.1 with keen interest. Concussions present a significant injury burden on the athletic community, especially among youth athletes who are more susceptible to potential long-term consequences.3,7,9 Concussion diagnosis and treatment are important, but prevention is key. Chisholm and colleagues present data on young athletes that supports a reduction in the risk of concussion with the use of a mouthguard. However, the authors admit that the current literature on mouthguards has methodological limitations and high risk of bias. The primary objective of their study was to examine the association between concussion and mouthguard use in youth ice hockey.

  We agree with the benefit players derive from wearing mouthguards to protect dentition and possibly reduce the incidence and/or severity of concussion during contact sports. However, we question the statistical methodology performed and the resultant conclusions of the manuscript. The authors utilized a nested case-control design to determine the risk of concussion with mouthguard use. Due to this design utilization, the results potentially present a high risk of bias that the authors were attempting to avoid. A nested case-control design compares incident cases nested in a cohort study with controls drawn at random from the rest of the cohort.2,6 Further, a nested case-control is useful for summarizing the trends observed in a large population when study of the e...

  Show More

  We read the referenced article by Chisholm et al.1 with keen interest. Concussions present a significant injury burden on the athletic community, especially among youth athletes who are more susceptible to potential long-term consequences.3,7,9 Concussion diagnosis and treatment are important, but prevention is key. Chisholm and colleagues present data on young athletes that supports a reduction in the risk of concussion with the use of a mouthguard. However, the authors admit that the current literature on mouthguards has methodological limitations and high risk of bias. The primary objective of their study was to examine the association between concussion and mouthguard use in youth ice hockey.

  We agree with the benefit players derive from wearing mouthguards to protect dentition and possibly reduce the incidence and/or severity of concussion during contact sports. However, we question the statistical methodology performed and the resultant conclusions of the manuscript. The authors utilized a nested case-control design to determine the risk of concussion with mouthguard use. Due to this design utilization, the results potentially present a high risk of bias that the authors were attempting to avoid. A nested case-control design compares incident cases nested in a cohort study with controls drawn at random from the rest of the cohort.2,6 Further, a nested case-control is useful for summarizing the trends observed in a large population when study of the entire population would be too expensive or burdensome with relatively minor loss in statistical efficiency.6

  The authors utilized two databases that total 2,355 youth hockey players. Based on a relatively small cohort, a complete case-control study would have been more appropriate than a nested case-control study. Table 1 shows that only n=270 players were used as controls and n=315 were used as cases. This represents only 25% of the cohort, since 1,770 players were excluded. The authors stated, “Cases were defined as those who sustained a suspected concussion during a game or practice. Controls were players who sustained a non-concussion (e.g. trunk or extremity orthopaedic) injury.” Consequently, this selection criteria of the non-concussion group biased the study as a random sample was not selected from the remaining cohort (n=2,040). In effect, the authors eliminated from the analysis all non-injured players who wore mouthguards. These non-injured controls are a factor in the non-concussed group, regardless of whether they wore a mouthguard. Thus, the reported odds ratio is very favorable (64% less likely). Although the primary objective was to examine the association between concussion and mouthguard use, the analysis compared concussion to other orthopaedic injuries. It did not properly compare the incidence of concussion between wearers or non-wearers of mouthguards. Clearly, many of the excluded participants (n=1,770) wore a mouthguard who did not suffer a concussion or any other orthopaedic injury. Consequently, there is likely significant error in the results reported. Furthermore, common practice with controls in a case-control or even a nested case-control is to use a subject ratio of 2:1 or 4:12,5 to allow for adequate statistical power and elimination of bias. The study in question did not adhere to these ratio paradigms.

  Perhaps the authors did not have accurate data on the cohort of non-injured players (n= 1,770) that did or did not wear a mouthguard, especially due to the retrospective study design. However, it may be possible to estimate mouthguard compliance among youth players and use the estimated trends for all control subjects. This adds an additional limitation to the study, but would permit a sampling of the entire population (n=2,355) and more appropriately address the original objective of the manuscript.

  To properly establish evidence of causality (in this case, wearing a mouthguard to reduce concussion incidence), results from various study designs must demonstrate: 1) consistency, 2) strength of association, and 3) biologic plausibility.6 Based on the results (biased in our opinion), the authors suggested three potential mechanisms of how mouthguards could reduce concussion incidence among young hockey players to establish biologic plausibility. Two of three mechanisms involve mandibular impacts. Although impacts to the mandible occur, they are not the most common mechanism of injury for concussion.3,4,8,10–12 Even a video analysis of Taekwondo (in which mandibular contact would be more common), only 12.1% of concussions were caused from contact to the mandible.8 Chisholm et al. stated that biomechanical modelling research is needed to better inform the mechanism by which mouthguards protect players from concussion, but the authors subsequently cited an article that used an instrumented artificial mandible skull model tested with and without a mouthguard that demonstrated no differences in head injury criterion.13 Thus, the theory they posited for mandible directed impacts is not sufficiently supported with consistency or biologic plausibility.

  We ask the authors, who are recognized leaders in epidemiology of injury research, to address the significant bias inherent in their current nested case-control. A corrigendum would be appropriate which performs a full case-control analysis on the data to include the remaining n=1,770 individuals. A report of these outcomes on the full, unbiased cohort would be ideal for the published literature.
   

  REFERENCES

  1. Chisholm DA, Black AM, Palacios-Derflingher L, et al. Mouthguard use in youth ice hockey and the risk of concussion: Nested case-control study of 315 cases. Br J Sports Med. 2020:1-6. doi:10.1136/bjsports-2019-101011.
  2. Ernster VL. Nested Case-Control Studies. Prev Med (Baltim). 1994;23(5):587-590. doi:10.1006/pmed.1994.1093.
  3. Haarbauer-Krupa J, Arbogast KB, Metzger KB, et al. Variations in Mechanisms of Injury for Children with Concussion. J Pediatr. 2018;197:241-248.e1. doi:10.1016/j.jpeds.2018.01.075.
  4. Hendricks S, O’Connor S, Lambert M, et al. Video analysis of concussion injury mechanism in under-18 rugby. BMJ Open Sport Exerc Med. 2016;2(1):e000053. doi:10.1136/bmjsem-2015-000053.
  5. Hennessy S, Bilker WB, Berlin JA, Strom BL. Factors influencing the optimal control-to-case ratio in matched case- control studies. Am J Epidemiol. 1999;149(2):195-197.
  6. Hulley SB, Commings SR, Browner WS, Grady DG, Newman TB. Designing Clinical Research. 4th ed. Philadelphia: Lippincott Williams & Wilkins; 2013.
  7. Kamins J, Bigler E, Covassin T, et al. What is the physiological time to recovery after concussion? A systematic review. Br J Sports Med. 2017;51(12):935-940. doi:10.1136/bjsports-2016-097464.
  8. Koh JO, Watkinson EJ, Yoon YJ. Video analysis of head blows leading to concussion in competition Taekwondo. Brain Inj. 2004;18(12):1287-1296. doi:10.1080/02699050410001719907.
  9. Langlois, Jean A. Scd M, Wesley Rytland-Brown M, Marlena M Wald, MLS M. The Epidemiology and Impact of Traumatic Brain Injury. J Head Trauma Rehabil. 2006;21(5):375-378.
  10. McIntosh AS, McCrory P, Comerford J. The dynamics of concussive head impacts in rugby and Australian rules football. Med Sci Sports Exerc. 2000;32(12):1980-1984. doi:10.1097/00005768-200012000-00002.
  11. Rowson S, Duma SM, Stemper BD, et al. Correlation of Concussion Symptom Profile with Head Impact Biomechanics: A Case for Individual-Specific Injury Tolerance. J Neurotrauma. 2018;35(4):681-690. doi:10.1089/neu.2017.5169.
  12. Sobin L, Kopp R, Walsh R, Kellman RM, Harris T. Incidence of concussion in patients with isolated mandible fractures. JAMA Facial Plast Surg. 2016;18(1):15-18. doi:10.1001/jamafacial.2015.1339.
  13. Viano DC, Withnall C, Wonnacott M. Effect of mouthguards on head responses and mandible forces in football helmet impacts. Ann Biomed Eng. 2012;40(1):47-69. doi:10.1007/s10439-011-0399-x.

  Show Less

  Conflict of Interest:
  None declared.

  • Back to top