Article Text
PDF
Abstract
Objectives Assess the validity and feasibility of current instrumented mouthguards (iMGs) and associated systems.
Methods Phase I; four iMG systems (Biocore-Football Research Inc (FRI), HitIQ, ORB, Prevent) were compared against dummy headform laboratory criterion standards (25, 50, 75, 100 g). Phase II; four iMG systems were evaluated for on-field validity of iMG-triggered events against video-verification to determine true-positives, false-positives and false-negatives (20±9 player matches per iMG). Phase III; four iMG systems were evaluated by 18 rugby players, for perceptions of fit, comfort and function. Phase IV; three iMG systems (Biocore-FRI, HitIQ, Prevent) were evaluated for practical feasibility (System Usability Scale (SUS)) by four practitioners.
Results Phase I; total concordance correlation coefficients were 0.986, 0.965, 0.525 and 0.984 for Biocore-FRI, HitIQ, ORB and Prevent. Phase II; different on-field kinematics were observed between iMGs. Positive predictive values were 0.98, 0.90, 0.53 and 0.94 for Biocore-FRI, HitIQ, ORB and Prevent. Sensitivity values were 0.51, 0.40, 0.71 and 0.75 for Biocore-FRI, HitIQ, ORB and Prevent. Phase III; player perceptions of fit, comfort and function were 77%, 6/10, 55% for Biocore-FRI, 88%, 8/10, 61% for HitIQ, 65%, 5/10, 43% for ORB and 85%, 8/10, 67% for Prevent. Phase IV; SUS (preparation-management) was 51.3–50.6/100, 71.3–78.8/100 and 83.8–80.0/100 for Biocore-FRI, HitIQ and Prevent.
Conclusion This study shows differences between current iMG systems exist. Sporting organisations can use these findings when evaluating which iMG system is most appropriate to monitor head acceleration events in athletes, supporting player welfare initiatives related to concussion and head acceleration exposure.
- brain concussion
- rugby
- dentists
- sporting injuries
Data availability statement
All data relevant to the study are included in the article or uploaded as supplementary information.
Statistics from Altmetric.com
WHAT IS ALREADY KNOWN ON THIS TOPIC
Concussive and subconcussive head acceleration events, from direct head impacts or inertial head loading (ie, head acceleration from impacts to the body), are a priority for sports to ensure player welfare is optimised.
A proactive approach to player welfare is to develop mitigation strategies to reduce the magnitudes of head acceleration events, and reduce the frequency of high magnitude head acceleration events without compromising the dynamics of the sport.
Various sensor systems have been used in sport, in an attempt to monitor the head accelerations of athletes.
Instrumented mouthguards (iMGs) are a relatively new technology and have the potential to accurately quantify head acceleration events in sport due to superior skull coupling in comparison to helmet or skin patch sensor systems.
Despite the potential and emerging use of iMGs in research and practice, limited validation studies on iMG systems exist.
WHAT THIS STUDY ADDS
Based on the evaluation of four iMG systems (Biocore-Football Research Inc (FRI), HitIQ, ORB, Prevent) for in-laboratory (linear and angular (rotational) kinematics) and on-field validity (false-positives and false-negatives), and player and practitioner perceptions of feasibility, differences between current iMG systems exist.
In-laboratory validity was similar for Biocore-FRI, HitIQ and Prevent, followed by ORB.
Considering false-positives and false-negatives collectively, on-field validity was highest for Prevent. Biocore-FRI and HitIQ were similar, followed by ORB.
Player perceptions of fit, comfort and function were similar for HitIQ and Prevent, followed by Biocore-FRI and ORB.
Practitioner perceptions of feasibility were highest for Prevent, followed by HitIQ and Biocore-FRI.
HOW THIS STUDY MIGHT AFFECT RESEARCH, PRACTICE OR POLICY
The results of this study can be used when determining which iMG system is most appropriate to monitor head acceleration events in athletes.
Instrumented mouthguard systems have the potential to monitor the frequency and magnitude of head acceleration events, but clinicians should be aware of the different on-field linear and angular kinematic data reported by different iMGs.
Thus, currently kinematic data in isolation may not be appropriate to support clinical decision making.
When interpreting head acceleration event data on athletes from iMGs, video-verification may be required to ensure data are not inclusive of false-positives, and false-negatives are considered.
It is anticipated that this study will be repeated to re-evaluate iMGs and associated systems, given the rapid development of existing iMGs and emergence of new iMG technology in the future.
Introduction
Instrumented mouthguards (iMGs) have the potential to quantify head acceleration events (HAEs) in sport. Direct impacts to the head or inertial head loading from impacts to the body can cause HAEs. Head acceleration events can be monitored acutely and cumulatively using iMGs.1 Limited validation and feasibility studies exist that evaluate iMGs and their associated systems,2 3 thus their appropriateness for both research and practice is unknown. Prior to the application and adoption of iMGs in sport, their validity and feasibility require investigation to enable sporting organisations, clinicians and the scientific community to be aware of the strengths and limitations of iMGs and associated systems (eg, system and data management, and visualisation). Valid iMG data could enable researchers to further understand the biomechanical mechanisms of concussion injuries and repetitive HAE in sport, the importance of which has been detailed in a recent review article.1 Ultimately, iMG data could provide essential information to support player welfare, safety and medical initiatives in sport.4
This four-phase study4 aimed to determine: phase I—the criterion validity of iMG kinematic magnitude measures against laboratory criterion standards; phase II—the on-field construct validity of iMGs via video-verification; phase III—iMG feasibility evaluated via player perceptions of fit, function and comfort and phase IV—practical feasibility of iMG systems from a practitioner perspective.
Methods
Study design
This four-phase study evaluated four currently available iMG systems, consistent with the published protocol.4 Phase I validated the linear and angular (rotational) kinematic measures of iMGs against laboratory criterion standards.2 3 Phase II evaluated the on-field validity of iMG-triggered events by identifying true-positives, false-positives and false-negatives via video-verification from 80 rugby league player matches (20±9 player matches per iMG company). Phase III evaluated the fit, comfort and function 5 of iMGs from a player’s perspective. Phase IV evaluated the practical feasibility (preparation and management) of the iMG system from a practitioner’s perspective6 7 using questionnaires. Given some of the research team had used, and were familiar with specific iMGs prior to the design and undertaking of this study (ie, Prevent), the protocol for the study was published4 to ensure there was transparency on the study design, analysis and interpretation for all iMG companies. Phase I design could have advantaged iMGs previously tested in this laboratory. Phase II design could have advantaged iMGs previously used in rugby league. Phase III and IV design could have advantaged iMG companies if they had worked with similar cohorts, although the players (phase III) and practitioners (phase IV) who participated in this study had no prior experience with iMG use. At the end of each phase, the results were provided back to each iMG company with an explanation, although companies did not contribute in any way to the analysis or interpretation of results.
Instrumented mouthguard recruitment
Each iMG company was initially invited to provide 43 iMGs (n=3 phase I, n=20 phase II, n=20 phase III) and associated systems free of charge. The protocol4 was shared, and companies were informed that the findings would help determine the appropriateness of iMGs for the Tackle and Contact Kinematic, Load and Exposure project.8 Six iMG companies were initially approached, one declined and one withdrew prior to data collection. Four companies provided iMGs and associated systems (table 1).
Hardware provided for the study, and product specification of iMGs and associated systems
Phase I—laboratory validation of kinematic measures
In-laboratory testing to evaluate the criterion validity of iMG kinematic magnitudes was conducted at the Virginia Tech Helmet Lab (USA),2 using a bareheaded dummy headform (figure 1A)9 10 impacted at various locations, impact magnitudes (target linear head accelerations 25, 50, 75 and 100 g) and durations (rigid; nylon, 25 mm thickness and padded; vinyl-nitrile foam, 40 mm thickness) using a pendulum impactor (127 mm diameter) (figure 1B) as previously described.4 Phase I impact conditions were selected to encompass the range of impact locations, impact energies and loading durations that were expected on-field.11 Three replicates of each condition were evaluated to assess within condition variance. Three iMGs were tested for each iMG company (figure 1C). Reference kinematics were measured at the headform centre of gravity with an instrumentation package consisting of three linear accelerometers (Endevco 7264b-2000; Meggitt Orange County, Irvine, California) and a tri-axial angular rate sensor (DTS ARS3 Pro 18k; Diversified Technical Systems, Seal Beach, California), recording at 20 kHz, and filtered using a CFC 1000 and CFC 155, respectively. Custom-fit iMGs were mounted inside the headform as previously described.4
(A) Experimental set-up of pendulum impactor to simulate bareheaded impacts to the dummy headform for Phase I. (B) Padded (vinyl-nitrile) and rigid (nylon) impactor to the bareheaded dummy headform at the front, front boss, rear boss and rear locations of the headform. (B) The custom-fit instrumented mouthguard (iMG) mounted inside the headform with detachable three-dimensional printed detention.
Biocore-FRI conducted three additional tests at the lowest speed test configuration. During the first round of testing, the Biocore-FRI iMG device did not record these three impacts due to their trigger mechanism (table 1), which had to be adjusted for the additional tests. HitIQ repeated testing, due to an issue with one test configuration producing high errors during the first round. HitIQ believed this was due to the use of their non-research data portal and/or experimental error. HitIQ implemented a firmware update and provided access to a research portal for the second round of testing. ORB updated the impact data analysis to account for an issue involving the impact detection algorithm and the iMGs two onboard accelerometers. ORB noted their algorithm sometimes saved data from the iMGs low-range instead of high-range accelerometer (±16 vs ±200 g). Prevent had previously been tested in this laboratory using the same methodology.2 HitIQ undertook pilot testing, independent of the research team in this laboratory prior to the study. Peak resultant linear acceleration (PLA), peak resultant angular velocity (PAV) and peak resultant angular acceleration (PAA) were recorded.4
Phase II—on-field validity
Fifty-one male rugby league players from five Super League academies (ie, teams) underwent three-dimensional dental scans, performed by an experienced dentist and reviewed by a third-party dentist with experience in iMG manufacture. Each company manufactured the custom-fit iMGs based on dental details received via a standard language tessellation file. HitIQ provided 11 additional iMGs at the request of the research team to increase match observations. Team 1 was provided with iMGs by HitIQ (n=10) and Prevent (n=10). Team 2 was provided with iMGs by Biocore-FRI (n=10) and HitIQ (n=10). Team 3 was provided with iMGs by ORB (n=10) and Prevent (n=10). Team 4 was provided with iMGs by ORB (n=10). Team 5 was provided with iMGs by HitIQ (n=11) and Biocore-FRI (n=10).
Eighty player match observations were included from a total of nine matches. This was influenced by injuries, suspensions and non-compliance of iMG wearing. The total player match observations (mean±SD observations per player) for each iMG was Biocore-FRI=17 from 10 players (1.7±0.8 observations per player), HitIQ=28 from 18 players (1.6±0.5 observations per player), ORB=9 from 7 players (1.1±0.4 observations per player) and Prevent=27 from 11 players (2.4±0.9 observations per player). For ORB, manufacturing delays reduced data collection, and battery issues resulted in 6/9 player matches having partial data collected (mean duration=30:22±09:51 min). One player removed their Prevent iMG 5 min into a match, so data were only partially captured for this match.
All data collection and management were undertaken by the research team as per manufacturer instructions. Postmatch, events recorded by iMGs were time-synchronised to high-quality video footage of match-play recorded by the home team to verify whether each iMG-triggered event (postapplication of companies’ recording threshold and/or HAE detection algorithm, if applicable) was associated with an HAE for the instrumented player. Time-synchronisation involved converting iMG timestamps to the time of iMG event in video. This was achieved by using the real-world time of a clear reference event in the video (eg, kick-off of each half) to get the real-world time at the start of the video. Real-world time at the start of the video was subtracted from the timestamp of each iMG event to give the time in video (ie, synchronised time). This process was repeated for each device using a true positive iMG event as a reference event to refine the time-synchronisation and to account for differences in internal clocks of the iMGs.
Companies determined the iMG recording and trigger thresholds which they deemed appropriate for optimal false-positive and false-negative performance (table 1), as previously described.4 Instrumented mouthguard data were trimmed to synchronise with kick-off and 10 min postmatch, resulting in data being recorded during the half-time period.
To determine true-positives or false-positives, a guided analysis was undertaken to quantify the construct validity for each iMG company. Instrumented mouthguard-triggered events were classified by a trained video analyst (professional sports video analyst with 9 years’ experience) as true-positives, false-positives or assumed false-positives (table 2). To determine false-negatives, an unguided analysis was undertaken. When a player who was wearing an iMG during a match was identified as a tackler in a one-on-one shoulder tackle, the event was labelled by the video analyst.12 The timestamp of each labelled one-on-one shoulder tackle event was cross-referenced with the iMG-triggered dataset. True-positives or false-negatives were based on whether the one-on-one shoulder tackle event timestamp matched an iMG-triggered timestamp (table 2). Only periods of play where iMGs were collecting data were used to avoid false-negatives being labelled when iMGs were not worn or active. The collection, analysis and reporting is consistent with the published protocol.4 Additional analysis and reporting of variables included quantification of player activities during identified false-positives, a breakdown of false-negatives and iMG kinematic magnitudes during match-play.
True-positive, false-positive, assumed false-positive, false-negative definitions for the guided and unguided analysis (phase II)
Phase III—player feasibility (perception of fit, comfort and function)
Eighteen male rugby league players with no prior experience of iMGs, from four Super League clubs, received one iMG from each company (n=4 iMGs), following dental scans as per phase II. These players were a different cohort to those in phase II. Using a randomised cross-over design between clubs, each iMG was worn during two field-based training sessions (>45 min). Within 1 hour following the second training session, players rated the fit, comfort and function of iMGs5 using an online questionnaire (Qualtrics, Washington, USA). This questionnaire has previously demonstrated an ability to discriminate between ideal and non-ideal mouthguard designs (eg, extent of labial extension and occlusal adjustment).5 This resulted in the completion of four questionnaires for each iMG per player (ie, n=18 completed questionnaires per iMG company). The research team supervised players to ensure the questionnaires were completed independent of other players. Fit, comfort and function were evaluated using eight, six and four questions, answered on binary, 10-point and 3-point Likert scales.5 Results were reported consistent with the protocol.4
Phase IV—practitioner feasibility (perception of usability)
Four physiotherapy/sports science practitioners with no prior experience of the iMG systems, from four different Super League clubs (same as phase III) evaluated three iMG systems (HitIQ, Prevent, Biocore-FRI) in a randomised cross-over design. ORB did not participate in phase IV, at their request, due to the software requiring their interpretation and involvement. Usability was evaluated for (1) preparing iMG systems (eg, setup, charging and deployment of the iMGs to participants) and (2) managing iMG data (eg, extracting data to software interface, accessing information and feedback mechanisms within system). iMG companies provided an online ‘on-boarding’ session (observed by the research team) to familiarise practitioners with the operating procedures 1 day before initial use. Practitioners had access to operating procedure documentation, and support from research team and/or iMG companies if required. Practitioners completed the preparation and management of iMG systems on two training days. Within 1 hour of downloading data from the second training session, practitioners completed two online System Usability Scale (SUS) questionnaires to evaluate the preparation and management of the iMG system (Qualtrics) each including 10 questionnaire items combined into a single score out of 100.6 7
Data analysis
Phase I—laboratory validation of kinematic measures
Concordance correlation coefficients (CCC) with 95% CIs that account for replicate test conditions were calculated using variance components.2 4 13 14 The iMGs output the peak resultant values for linear (ie, PLA) and angular (ie, PAA) kinematic measures. After the sensor and reference measurements were recorded, both were normalised relative to the maximum reference measurement. The CCC values were computed for the linear and angular kinematic measure(s), and the combination of linear and angular measures. The combined CCC value that accounts for peak linear and angular acceleration represented the overall iMG in-laboratory validity. However, the ORB iMG system was unable to output PAA, and thus PAV was used in their combined CCC value. In addition to CCC, Bland-Altman mean bias and 95% limits of agreement (LOA) were computed for each iMG. Differences between iMG and headform measurements were normalised by headform measurement to account for heteroscedasticity and express agreement as a percentage. Variance components were used when computing the SD that the 95% agreement values are dependent to account for replicate testing.15 Bland-Altman analysis was undertaken in addition to the analysis described in the published protocol,4 at the request of three reviewers.
Phase II—on-field validity
Positive predictive values (PPV) with 95% CI (Eq. 1) were calculated for true-positives and false-positives, as well as combinations of assumed false-positives (ie, false-positives summed).2
[1]
Sensitivity scores with 95% CI were calculated (Eq. 2) from the true-positive and false-negative counts for each iMG.
[2]
Phase III—player feasibility (perception of fit, comfort and function)
Mean±SD (range) percentage of positive responses (ie, ‘no’) for individual questionnaire items, evaluating fit and function were calculated. Median and IQRs were calculated from the 10-point Likert questionnaire items evaluating comfort.
Phase IV—practitioner feasibility (perception of usability)
The mean±SD (range) of the two SUS scores (ie, preparation and management) were calculated.
Results
Phase I—laboratory validation of kinematic measures
Table 3 presents linear, angular and total CCC for in-laboratory testing (phase I). The linear and angular kinematic values for each iMG from phase I are shown in figure 2 and Bland-Altman plots are shown in figure 3.
Linear (a) Biocore-FRI, (b) HitIQ, (c) ORB, (d) Prevent and angular (e) Biocore-FRI, (f) HitIQ, (g) ORB, (h) Prevent kinematic values for each iMG and reference measurements during laboratory bareheaded dummy headform impacts (Phase I). FRI, Football Research Inc; iMG, instrumented mouthguard; PAA, peak resultant angular acceleration; PAV, peak resultant angular velocity; PLA, peak resultant linear acceleration.
Bland-Altman plots visualising agreement between the iMGs and headform reference measurements (Phase I). The x-axis on each plot is the average between paired iMG and headform measurements. The y-axis on each plot was transformed to per cent difference ((iMG–headform)/headform) to minimise heteroskedasticity that was observed in the raw differences. The solid line represents the mean bias and the dashed lines bound the limits of agreement, defined as the range of difference that 95% of measurements are expected to fall in. FRI, Football Research Inc; iMG, instrumented mouthguard.
Laboratory (phase I) and on-field (phase II) validity and player (phase III) and practitioner (phase IV) feasibility results
Supplemental material
Phase II—on-field validity
The linear and angular kinematic values for each iMG from phase II are shown in figure 4. Table 3 also presents the total player matches, iMG triggered events, the number of true-positives, false-positives and PPV for the on-field (phase II) validity. Inactive assumed false-positives (ie, during half-time and 10 min postmatch) and all assumed false-positives (ie, inactive assumed false-positives and off-camera assumed false-positives) are also shown in table 3. Online supplemental table 1 shows off-camera assumed false-positives and associated PPV, and a breakdown of false-positives and false-negatives. Online supplemental table 2 shows true-positives, false-positives and associated PPV for each player from phase II. Online supplemental figure 1 shows the linear and angular kinematic values, associated PPVs, true-positives, false-positives and combinations of false-positives and assumed false-positives for each iMG from phase II.
Linear and angular on-field kinematic values from true-positives and false-positives for each iMG from phase II. ORB did not provide PAA, thus not presented. Data are presented as individual iMG-triggered events and median (IQRs) for true-positives and false-positives. An outlier was defined as any value that is >1.5 times the IQR or <75th and <25th percentile, respectively. FRI, Football Research Inc; iMG, instrumented mouthguard; PAA, peak resultant angular acceleration; PLA, peak resultant linear acceleration.
Phases III and IV—player and practitioner feasibility
Player evaluations of fit, comfort and function (phase III) and practitioner SUS (phase IV) are shown in table 3, and online supplemental table 3 shows the fit, comfort and function for each questionnaire item from phase III.
Discussion
This study was the first to determine the criterion and construct validity and feasibility of current iMGs and associated systems, which is vital to consider the error associated with HAE data and strengths and limitations of iMG systems to maximise the adoption in sport. In phase I, Biocore-FRI, HitIQ, ORB and Prevent had total CCC (95% CI) of 0.986 (0.980 to 0.990), 0.965 (0.949 to 0.976), 0.525 (0.420 to 0.617) and 0.984 (0.977 to 0.989) when compared with dummy headform reference measurements.2 The PLA mean bias (LOA) for Biocore-FRI, HitIQ, ORB and Prevent was −1.2% (−17.1% to 14.8%), −4.0% (−33.7% to 25.8%), −36.4% (−78.3% to 5.6%) and −1.0% (−16.9% to 14.8%). The PAA mean bias (LOA) for Biocore-FRI, HitIQ and Prevent was −2.3% (−17.0% to 12.4%), −4.2% (−24.5% to 16.2%) and −6.8% (−21.7% to 8.0%). The PAV mean bias (LOA) for ORB was −51.0% (−96.1% to −6.0%) (figure 3). False-positive (PPV; 95% CI) and false-negative performance (sensitivity; 95% CI) should be considered simultaneously in phase II. Overall, Prevent had the highest performance with a PPV of 0.94 (0.92 to 0.95) and sensitivity of 0.75 (0.67 to 0.83). For Biocore-FRI and HitIQ, false-positive performance was higher (PPV=0.98; 0.95 to 0.99 and 0.90; 0.86 to 0.93, respectively) than false-negative performance (sensitivity=0.51; 0.41 to 0.61) and 0.40; 0.31 to 0.49, respectively). ORB had a higher false-negative (sensitivity=0.71; 0.54 to 0.88), but lower false-positive (PPV 0.53; 0.44 to 0.62) performance. To understand likely iMG adoption, the feasibility of current iMG systems were determined. In phase III, players reported HitIQ and Prevent to have similar, and Biocore-FRI and ORB lower ratings for fit, comfort and function. Practitioners reported the SUS to be highest for Prevent (>80), followed by HitIQ (>70) and Biocore-FRI (>50). This study demonstrates differences between current iMG systems exist. These findings can be used by sporting organisations to determine the appropriateness of iMG systems, and provides reference data to improve the validity and feasibility of iMG systems, optimising their adoption in sport, supporting player welfare initiatives.
Phase I—laboratory validation of kinematic measures
For phase I, Biocore-FRI, HitIQ and Prevent had total CCC values >95%, higher than other wearable head sensors previously evaluated.2 All iMGs achieved higher CCC for PAA than PLA (table 3). Varying in-laboratory iMG performance has previously been observed (eg, mean error of 32.4% for PLA3), which was attributed to a lack of filtering of iMG kinematic data.16 Biocore-FRI, HitIQ and Prevent had a mean bias of <7% for PLA and PAA. Biocore-FRI and Prevent had similar LOA for PLA, and Biocore-FRI, HitIQ and Prevent had similar LOA for PAA. This highlights the importance of ongoing hardware/firmware/data-processing development to improve the validity of kinematic data, and poses a challenge for researchers and practitioners if proprietary algorithms are frequently updated exclusive of independent validation.17 As such, this study should be repeated, given the rapid development of existing, and emergence of new iMG systems.
Phase II—on-field validity
Phase II evaluated the on-field validity of iMGs (PPV; true-positives and false-positives and sensitivity; true-positives and false-negatives). Prevent had the highest concurrent false-positive and false-negative performance. Both Biocore-FRI and HitIQ demonstrated high false-positive, but lower false-negative performance, and ORB demonstrated relatively high false-negative, but lower false-positive performance. The different performance by iMGs may be due to the trigger and recording thresholds set by iMG companies, and/or HAE detection algorithms. Prevent had the lowest trigger threshold (8 g), vs Biocore-FRI, ORB (both 10 g) and HitIQ (13 g). The lower trigger threshold selected by Prevent likely improved false-negative performance when compared with the other iMGs. Lower trigger thresholds could result in more false-positives (due to lower-magnitude iMG-triggered non-HAE being captured) and fewer false-negatives. Most false-positives observed for Prevent (78%) occurred while players were running (online supplemental table 1). These may represent true head accelerations from inertial loading during non-contact events,18 19 necessitating HAE detection algorithms to differentiate contact from non-contact events.1 This study defined true-positives as HAE during contact events, however HAE from non-contact events (eg, running and jumping) may also be desirable in future studies. Further analysis of false-negatives for Biocore-FRI revealed that the iMG algorithm incorrectly binned 17.4% of false-negatives. Biocore-FRI algorithms were trained on American Football data,20 therefore optimising algorithms using non-helmeted sports would likely improve false-negative performance. HitIQ false-negative performance was likely due to the 13 g recording threshold, which was different to the trigger threshold (table 1). Further analysis of HitIQ data revealed that 42.5% of false-negatives were collected as iMG-triggered events, but subsequently binned as they fell below the 13 g recording threshold, and 1.4% of false-negatives were greater than the recording threshold but were misclassified by the iMG algorithm (online supplemental table 1). Decreasing the 13 g recording threshold (or removing it entirely) may improve HitIQ’s false-negative performance, however the effect this would have on false-positive performance is unknown. Increasing the ability to remove false-positives would improve the overall performance of ORB.
When off-camera and inactive assumed false-positives are considered, false-positive performance reduced for all iMGs (Biocore-FRI, PPV=0.98 (95% CI 0.94 to 0.99) to 0.81 (0.74 to 0.86); HitIQ, PPV=0.90 (0.86 to 0.93) to 0.60 (0.56 to 0.65); ORB, PPV=0.53 (0.44 to 0.62) to 0.37 (0.30 to 0.44); Prevent, PPV=0.94 (0.92 to 0.95) to 0.89 (0.87 to 0.92); online supplemental figure 1). No inactive assumed false-positives were observed for Prevent (online supplemental table 1), which may be due to proximity sensors embedded within Prevent iMGs, preventing the iMG from triggering when not on the players’ teeth. HitIQ and ORB accumulated more off-camera assumed false-positives, and HitIQ and Biocore-FRI accumulated more assumed false-positives during inactive periods (93.5% of these for Biocore-FRI came from a single iMG, which may have been faulty; online supplemental tables 1 and 2). Therefore, video-verification or alignment to another data-source (eg, time motion analysis of match events21) may be required prior to use in research and/or practice.
Validating the kinematics of on-field HAE is beyond the scope of this study, due to a lack of an accurate on-field reference measurement. While Biocore-FRI, HitIQ and Prevent all had similar kinematics CCC in phase I (table 3 and figure 4), the kinematics between iMGs from phase II varied and some exceeded the 100 g magnitude used in phase I. The median kinematic data are influenced by different thresholds, therefore comparison between iMGs is not possible, although the number and range of high magnitude true-positives differ (figure 4). The number of true-positives with a PLA above 100 and 200 g, were 0 and 0 for Biocore-FRI, 14 and 5 for HitIQ, 1 and 0 for Prevent and 1 and 0 for ORB. Similarly, for true-positives which had a PAA above 10 and 20 krad/s2, HitIQ reported 14 and 5, while Biocore-FRI, ORB and Prevent all reported none. Measuring the kinematics of on-field HAE is associated with numerous challenges. For example, direct contact to the iMG, mandible interaction (eg, biting), adherence to teeth and vocalisation, which may affect the magnitude of peak kinematics by introducing noise into iMG signals. It is outside the scope of this study to evaluate the on-field HAE kinematics, however future studies should investigate the kinematic frequency content to evaluate potential signal noise of on-field events.
Phases III and IV—player and practitioner feasibility
To optimise iMG adoption, phases III and IV evaluated the feasibility from a player and practitioner perspective. Most players perceived no issues with the iMGs for fit (mean response of ‘no’=62%–88%). Individual questionnaire items revealed players perceived Biocore-FRI and ORB ‘too bulky’ (84% and 79% of players), with fewer expressing this perception for HitIQ and Prevent (37% and 33% of players; online supplemental table 3). HitIQ and Prevent (both 8/10) had higher median scores for comfort, than Biocore-FRI and ORB (6/10, 5/10; table 2). In a previous study,5 a mouthguard intentionally containing common design faults had a median comfort rating of 6/10, which can be used as a comparison. For Biocore-FRI, HitIQ and Prevent, most players (55%–67%) perceived no issues with the function of iMGs. This was lower for ORB (43%; online supplemental table 3), where 50% of players perceived it interfered with speech ‘a lot’ in comparison to Biocore-FRI (26%), HitIQ (11%) and Prevent (17%). As such, the design of iMGs (including the housing and positioning of component parts) is a key consideration for companies when considering the adoption of iMGs within sport.
Phase IV evaluated the usability for Biocore-FRI, HitIQ and Prevent system preparation and data management. The industry standard of average SUS for internet-based web pages and applications is 68, which both HitIQ (>70) and Prevent (>80) surpassed. Biocore-FRI was lower (>50), which could influence adoption by practitioners. Both the system preparation and data management scores were similar within iMGs, which may suggest companies have prioritised these areas within their product development or indicate practitioners do not differentiate between these two aspects. The findings of phases III and IV provide comparative data for iMG evaluation and development, and potential system adoption.
Clinical implications
In practice, iMGs have the potential to enhance clinical care of athletes via the live (where possible) and retrospective monitoring of HAEs. Both the frequency and magnitude of HAEs that athletes experience within a match, training week, over a season and career can be used to inform player welfare initiatives, and could be linked to the long-term brain health of athletes.1 In addition to other systems (eg, pitch-side video replay), iMGs have the potential to contribute to concussion detection, contact load monitoring and return to play practices, which are all priorities in sport.
Limitations
The findings of this study are specific to the cohort (eg, male rugby players and unhelmeted collision sports) and may not be applicable to other sports with different HAE characteristics (eg, soccer heading, helmeted impacts). While peak resultant kinematic values are important, so are other signal characteristics such as pulse-duration and frequency content, not assessed in phase I or II. Instrumented mouthguard companies determined their own recording or trigger thresholds, which influenced phase II performance. It is unclear if false-negatives associated with no iMG-triggered event were due to one-on-one shoulder tackles not exceeding trigger thresholds or whether iMGs did not capture HAEs for another reason. Until the clinical relevance of the accumulation of low magnitude HAE is understood, the aim of using iMGs should be to record data from all contact events, to gain a robust understanding of player HAEs. Only one camera view was available for phase II, thus some iMG-triggered events could not be verified. Given the camera followed the ball, and therefore every tackle, these iMG-triggered events were assumed false-positives and not the result of contact events. Assumed false-positives and inactive PPV did not take into account player activities during these periods. Future studies should use multiple camera angles to capture all iMG-triggered events during match-play and control for player activities (eg, off-camera collisions with players on own team). Not all iMG companies used automatic data processing pipelines for phase I and II, which may be advantageous for larger-scale project given the amount of data produced (table 1), which was not assessed in phase IV. The field-based training session duration, type or intensity was not standardised in phase III, which may have influenced player perceptions of fit, comfort and function.
This study should be repeated for a number of reasons. First, to overcome any potential advantages an iMG company may have had through their previous experience both with similar validity methods or cohorts, and also working with members of the research team. Second, to allow iMG companies who participated in this study the opportunity to optimise algorithms and/or use different trigger and/or recording thresholds, as well as optimising the fit, comfort and function for players and feasibility for practitioners, re-evaluating iMG and associated system performance. Third, to include other available iMGs for comparison. Specifically in this study phase I may have advantaged Prevent, who had previously tested in the Virginia Tech Helmet Lab,2 although similar performance was observed for Biocore-FRI and HitIQ iMGs. In addition, phase II may have advantaged Prevent, who had previously provided iMGs for use in a research study in rugby league, undertaken by some of the research team.22 The research team were aware of their conflicts of interest and previous experiences, therefore the interpretation of the results was undertaken collectively, to reduce potential bias.
Conclusion
In conclusion, for the first time this study evaluated the criterion and construct validity and feasibility of current iMG systems, which have potential to provide data on HAEs, and inform player welfare initiatives, which could be linked to the long-term brain health of athletes. This study showed that when compared with reference measurements from a dummy headform, between iMG systems, Biocore-FRI, HitIQ, Prevent had lower and ORB had higher measurement error. When video-verification was used to determine on-field validity, Prevent had the highest false-negative performance and second-highest false-positive performance. Biocore-FRI and HitIQ had higher false-positive performance than false-negative performance, and ORB had higher false-negative performance than false-positive performance. To support the adoption of the iMG systems by players and practitioners, feasibility ratings were provided. HitIQ and Prevent achieved similar high ratings for fit, comfort and function, and practitioners reported the feasibility of iMG systems to be highest for Prevent, followed by HitIQ, then Biocore-FRI. For the first time, this study provides data on the strengths and limitations of iMG systems. This can be used by sporting organisations and provides reference data for iMG systems to optimise hardware and software, to improve both the validity of iMG data and feasibility of iMG systems from a player and practitioner perspective, optimising the adoption of iMGs in sport.
Data availability statement
All data relevant to the study are included in the article or uploaded as supplementary information.
Ethics statements
Patient consent for publication
Ethics approval
This study was approved by Leeds Beckett University (85551). Participants gave informed consent to participate in the study before taking part.
Acknowledgments
The authors would like to thank all players and staff at Hull KR RLFC, Leeds Rhinos RLFC, St Helens RLFC, Wakefield Trinity RLFC, Warrington Wolves RLFC and Wigan Warriors RLFC for participating in the study. The authors would like to acknowledge the funding (dental scans, project expenses and Virginia Tech lab costs) provided by Leeds Beckett University, Carnegie School of Sport. The authors would like to thank all Super League CEOs, Laura Fairbank and Karen Moorhouse (Rugby Football League) for supporting the project. The authors acknowledge Brianna Mulhern and Mily Spiegelhalter for supporting data collection and Matt Halkier for supporting the video analysis. The authors would like to thank Dr Anthony Lovat (OPRO) for reviewing dental scans. Finally, the authors would like to acknowledge their sincere gratitude to Dr Nate Dau and Dr Lee Gabler (Biocore-Football Research Inc (FRI)), Damien Hawes and Tom Laudenbach (HitIQ), Robert Paterson and Thomas Quinn (ORB innovations) and Drew Goodger and Dr Adam Bartsch (Prevent Biometrics) for their support and cooperation during the study.
References
Supplementary materials
Supplementary Data
This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.
Footnotes
Twitter @23benjones, @danweaving, @camowen94, @drkeithstokes, @Sharief_H, @GTBiomech
Contributors BJ, DW, GT conceptualised the research project and designed the study. BJ, JT, DW, KT, CO, MB, SR, GT collected data for the study. BJ, JT, DW, KT, CO, MB, KAS, SR, GP, ÉCF, GT were responsible for analysis and interpretation of the results. BJ, JT, DW, KT, CO, GT drafted the manuscript. All authors critically reviewed and edited the manuscript prior to submission. BJ is the guarantor for the study and the content.
Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.
Competing interests GT, JT and DW have received Prevent Biometrics iMGs free of charge for use with Leeds Rhinos RLFC for a research project. BJ, MAD, KT are employed by Leeds Rhinos in a consultancy capacity. GT, JT and ÉCF use Prevent Biometrics iMG for research projects funded by World Rugby. GT submitted a funded research proposal for a different study to Prevent Biometrics during the review process of this article, which was subsequently successful. ÉCF is employed by World Rugby. BJ, JT, DW, KT, CO, KAS, GP, SH, MAD, GT are involved in the Rugby Football League Tackle and Contact Kinematic Load & Exposure (TaCKLE) project. BJ and KT will be involved in the procurement of iMGs for the TaCKLE project. BJ, GP are employed in a consultancy capacity by the Rugby Football League. KAS is employed by the Rugby Football Union. MB and SR lead the Virginia Tech Helmet Lab. Leeds Rhinos, World Rugby and Virginia Tech are research partners of Prevent Biometrics.
Patient and public involvement Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research.
Provenance and peer review Not commissioned; externally peer reviewed.
Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.