Article Text
PDF
Abstract
Objective To compare Orchard Sports Injury Classification System (OSICS-10) sports medicine diagnoses assigned by a clinical and non-clinical coder.
Design Assessment of intercoder agreement.
Setting Community Australian football.
Participants 1082 standardised injury surveillance records.
Main outcome measurements Direct comparison of the four-character hierarchical OSICS-10 codes assigned by two independent coders (a sports physician and an epidemiologist). Adjudication by a third coder (biomechanist).
Results The coders agreed on the first character 95% of the time and on the first two characters 86% of the time. They assigned the same four-digit OSICS-10 code for only 46% of the 1082 injuries. The majority of disagreements occurred for the third character; 85% were because one coder assigned a non-specific ‘X’ code. The sports physician code was deemed correct in 53% of cases and the epidemiologist in 44%. Reasons for disagreement included the physician not using all of the collected information and the epidemiologist lacking specific anatomical knowledge.
Conclusions Sports injury research requires accurate identification and classification of specific injuries and this study found an overall high level of agreement in coding according to OSICS-10. The fact that the majority of the disagreements occurred for the third OSICS character highlights the fact that increasing complexity and diagnostic specificity in injury coding can result in a loss of reliability and demands a high level of anatomical knowledge. Injury report form details need to reflect this level of complexity and data management teams need to include a broad range of expertise.
- Epidemiology
- Measurement
- Sporting injuries
This is an Open Access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 3.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/3.0/
Statistics from Altmetric.com
Introduction
Injury data collection or surveillance is the foundation of most sports injury prevention research.1 ,2 Injury data collected in a routine, systematic manner, when coupled with appropriate exposure data, can (1) enable estimation of injury incidence in both experimental and observational studies, (2) be used to monitor temporal and spatial trends in injury rates and (3) be used to evaluate the success of injury prevention strategies.3 The amount and type of data that are collected in an injury surveillance system is largely dependent on the needs of its end-users (eg, clinicians, epidemiologists, etc). However, it is also recommended that a core set of variables should always be collected, regardless of the system's purpose.4 ,5
The quality of data collected for sports medicine is dependent on how well key data items, such as injury diagnoses, nature and body part are recorded and coded. Use of standardised coding systems enables cross-study comparisons. Unfortunately, many important sports medicine diagnoses are completely absent from the International Classification of Diseases (ICD).6
Several diagnosis coding systems have been developed to meet the specific needs of sports injury research. The Sport Medicine Diagnostic Coding System (SMDCS), first developed in 1991 for use in the Canadian Intercollegiate Sport Injury Registry, has since been incorporated into the National Collegiate Athletics Association Injury Surveillance System, the National Hockey League and the Canadian Athlete Monitoring Program.7 To date, only one paper has been published on the SMDCS itself7 and the system's reliability and validity are unknown.
The Orchard Sports Injury Classification System (OSICS) which is a hierarchical four-character system for coding sports medicine diagnoses has been more widely applied.8 Originally developed in 1992 for use in elite Australian football, it is now in its 10th version (OSICS-10, http://www.johnorchard.com/osics.html). It is recommended by international sports injury consensus statements,9–13 and has been incorporated into the injury surveillance systems of many international professional sporting bodies.14–19 It has been suggested that the high level of detail required in the new four-character version of OSICS-10 could potentially increase the capacity for disagreements between users and that the system is too complex for users without a clinical background.18
As with any outcome measure, injury diagnosis coding schema must have good reliability to ensure comparability across studies and clinicians/data coders. In many studies, the person who diagnoses or records the injury is not the same as the person who codes the injury for database management purposes. This introduces potential error because a coder could either misinterpret the written diagnosis or select the wrong code.6 The likelihood of this can be increased for coders with different backgrounds or levels of expertise. While there is some evidence that having a non-clinical background may affect the accuracy of coding sports injuries using the ICD,6 this has not previously been evaluated for OSICS. Therefore, this study aimed to evaluate intercoder agreement in OSICS coding between individuals with and without a clinical background.
Methods
The OSICS-10 is structured hierarchically with four characters assigned to each injury.20 The first (leftmost) character relates to the anatomical location; the second character to the specific injured tissue or the pathology of the injury; the third and fourth characters further describe the pathology or broaden the diagnosis.19 ,20 The letter X is used as a character when some aspect of the injury is unspecified,20 or as the fourth character if sufficient detail is provided by the previous three characters (eg, the code AFAX is used to denote a fracture of the tibia and fibula at the ankle joint, because this is fully specified by the first three characters). The letter Z is used to denote an absence of injury or illness details or to represent some ‘other’ injury not listed by OSICS (eg, the code AFTZ indicates a fracture to the talus of the ankle, not further specified).19
The injury data used in this study were taken from the Preventing Australian Football Injuries through eXercise (PAFIX) randomised controlled trial (RCT). Full details of the study protocol are published elsewhere21 and the injury data collection procedures shown to be reliable.22 The study was approved by the University of Ballarat Ethics Committee. In summary, 40 trained primary data collectors (PDCs) with an exercise and sports science background used a standardised data collection form (figure 1) to collect details of injuries sustained during all games and training sessions during the 2007 and 2008 playing seasons in 18 community-level Australian football clubs.
Injury data collection form used in the Preventing Australian Football Injuries through eXercise study.
Overall, 1082 injury record forms were completed by the PDCs. No attempt was made to confirm injury details with a medical practitioner as many injuries would only have been treated by the first-aid staff at the game.
The full injury records were checked and edited before exporting to a Microsoft Excel spreadsheet. This was then provided to two independent coders for assigning of an OSICS code to each injury record. One coder was a sports physician, with extensive experience in the use of OSICS and the diagnosis of sports injuries. The second coder was an epidemiologist with no clinical background, but with considerable experience in the analysis and reporting of sports injury data, including with OSICS. Each coder entered the OSICS-10 codes they allocated into separate versions of the Excel spreadsheets.
Both sets of allocated codes were compared to identify exact matches and discrepancies. As OSICS-10 is hierarchical, it was considered only relevant to compare disagreements at a given character level, if coders agreed fully on all preceding characters. For example, it was not relevant to determine how many codes disagreed at the second character level if they did not agree at the first character level. A third coder, with a background in injury biomechanics but no previous experience in OSICS coding, adjudicated when the two main coders differed. The adjudicator, who was blinded to the coder backgrounds, decided on the final OSICS code to be assigned. The nature of the systematic discrepancies were documented and summarised by the adjudicator; they were then interpreted in the light of the coder background, but only after the adjudication was finalised.
All data are presented as the percentage (%) of agreements or disagreements. Patterns in the reasons for discrepancies were noted by the adjudicator and are presented in the results section.
Results
The two coders provided exactly the same four-digit OSICS-10 code for 46.3% (n=501) of the 1082 recorded injuries. The 53.7% of injuries (n=581) with coding discrepancies underwent adjudication. In just over half (54.4%, n=316), the initial four-character code assigned by the sports physician was deemed correct; in 44.2% (n=257) of cases, the initial code assigned by the epidemiologist was considered correct. The adjudicator differed from both coders on eight cases.
Table 1 shows the level of coding agreements and disagreements at each hierarchical level of OSICS-10. There was very high agreement at the first character level and 84% of all cases agreed to the second (given the first was correct) character level. The majority of disagreements occurred for the third character (given agreement on the first two characters), with only 49% agreement. However, when the first three characters agreed it was most likely that the fourth would also agree.
Summary of the hierarchical Orchard Sports Injury Classification System OSICS-10 coding agreements (n-1082 injuries)
Anatomical location (first character) disagreements
Overall, there were 50 cases in which the two coders disagreed on the first character. The adjudicator judged the sports physician's coding to be correct in 80% of these cases and, in one case, the adjudicator disagreed with both coders. Of these 50 disagreements, 32 were because one coder used an X rather than a more specific body code. For example, for 30 discrepancy cases, the text read ‘back’; the epidemiologist assigned a general code but the physician coded the body region to the lumbar spine based on other information given in the written record. In five cases, the epidemiologist coded a writs/hand injury but the physician correctly coded them to forearm.
Injured tissue/pathology (second character) disagreements
After agreeing on the first character, the coders disagreed on the second OSICS character in 102 injuries; 5 instances were due to one of the coders assigning an X and 40 cases due to assigning of a Z. According to the adjudicator, the epidemiologist was incorrect in 38 cases, compared to 66 cases for the sports physician; the adjudicator disagreed with both in two cases. Generally, when the sports physician was deemed incorrect, it appeared that additional information present in the injury records had not been used. For example, the major body region data field category was upper leg but information from a separate data field needed to be consulted to indicate the hamstrings or quadriceps muscle. There were 15 injuries for which the epidemiologist assigned a non-specific injury code, but the sports physician correctly coded them as a sprain or haematoma.
Specific diagnosis (third and/or fourth character) differences
After agreeing on both first and second characters, the coders disagreed on 399 injury diagnoses at the third character level; the vast majority (85%) of these was because one coder assigned an X and 6% were due to one code assigning a Z. This was the character level where the major level of discrepancy between the two coders was reported. The adjudicator confirmed that the sports physician correctly coded 56.1% (n=224) of cases, compared with 43.1% (n=172) for the epidemiologist; they disagreed with both coders for three cases. The major reasons for the discrepancies appeared to be non-use of additional information present in the injury records by the sports physician and a lack of detailed anatomical or clinical knowledge on the part of the epidemiologist. Correct coding of haematomas and ankle sprains, in particular, was more common for the sports physician.
The two coders only disagreed on the fourth character in 30 injuries (after agreeing on the first three characters); in two cases, the adjudicator disagreed with both coders. The sports physician was deemed correct in 17 cases and the epidemiologist in 11 cases. In all cases, the discrepancies occurred because an X was used by only one coder; in four cases, one coder also used a Z.
Use of X and Z codes
The OSICS-10 allows for un-needed characters (eg, when the diagnosis is fully described by three characters and so a fourth is not needed) to be indicated by an X. In the final consensus diagnoses of the 1082 injuries, an X was assigned to just the fourth character in 959 (or 88.6%) of cases. In 406 cases, the last two characters were assigned as XX and in only three cases were the last three characters an XXX. When there is insufficient detail a Z code can be used. In the final consensus codes, no first character had a Z, 21 second characters were a Z, 15 cases had a Z in the third character and only three cases had a Z in the fourth.
Discussion
Sports injury research requires accurate injury classification to ensure that it is useful for identifying patterns in injury occurrences and opportunities for prevention. The accuracy of this coding depends on both the quality of the initial information provided (eg, text narrative descriptions of injury events can be useful for providing information about sports injuries, but can vary in quality23) and the translation of this information into an appropriate code. In many epidemiological field-based data collections in non-professional sport, responsibility for data collection, coding and/or analysis often rests with a database manager/epidemiologist without formal sports medicine clinical knowledge.
To our knowledge, this is the first study to compare how different coders allocate OSICS-10 sports medicine codes to a large-scale epidemiological database of injuries reported in community sport. The injury data collectors were formally trained in the data collection procedures and previous research using the same sorts of data collectors has been shown to be reliable.22 ,24 The information they recorded was then provided to two ‘expert’ coders from different backgrounds and adjudicated by a third.
There was very high agreement at the first character level (95%) across 1082 injuries, and 86% of injury codes also agreed to the two character OSICS-10 level. While the OSICS-10 has four characters to enable a high level of diagnostic specificity, this level of detail is generally not needed in non-clinical settings.25 The majority of coding disagreements occurred for the third OSICS character (given agreement on the first two) with only 49% of all cases agreeing at the third character hierarchical level. This highlights the fact that increasing complexity and diagnostic specificity in injury coding can result in a loss of accuracy.18 ,25 ,26 In epidemiological studies it is rare to code injuries to the highest level of diagnostic specificity. Importantly, given the hierarchical nature of OSICS-10, even where the third or fourth characters disagree, the final injury code will map to the same two-character category (eg, ankle sprain), which is more commonly reported. It should be pointed out, however, that the data collection form was designed to maximise information for the first and second character levels because the data collectors were often non-clinical people associated with community football teams. There were no specific prompts to record additional information that may have assisted in assigning more correct codes, such as the type of treatment provided. It could be expected that these data recording limitations could be overcome in studies in professional sport.
The level of three character coding agreement was lower in this study than in previous studies which have evaluated intercoder agreement among clinicians. One study had at least two of three clinically trained coders assigning three-character OSICS-8 codes to 300 sports medicine diagnoses and found a coding agreement of 84%.6 In another study,18 20 written injury diagnoses from professional football, cricket and rugby were used to assess intercoder agreement among eight clinicians (orthopaedic surgeons, physiotherapists and specialist general practitioners). Similar to our findings, the level of agreement declined with increasing diagnostic specificity with kappa values ranging from 0.95 (almost perfect agreement) for agreement on the first character to 0.56 (moderate agreement) on the fourth character. Reasons for the lower agreement in this study could include the smaller number of raters and their disparate backgrounds.
Both coders were provided with exactly the same information to code from in the form of an Excel spreadsheet which contained all information reported on the initial paper-based collection form. However, the adjudication of coding disagreements by a biomechanist identified some apparent systematic biases in how the two coders assigned OSICS codes. First, the physician did not always use the additional information in other parts of the injury record. This emphasises the importance of ensuring that coders consider the full text fields when assigning codes, since the text may contain valuable information to enhance the injury classification. This issue has also been noted as being important for coding injury-related hospital admissions.27 Second, the epidemiologist appeared to assign incorrect codes based on a lack of detailed anatomical or medical knowledge. The fact that different professional backgrounds of coders influence the quality of diagnostic coding allocation has also been shown in hospital settings.28 ,29
One advantage of OSICS-10 over previous versions is that its hierarchical arrangement of characters allows mapping back to broader categories of anatomical location and body structure. Even when there is poor coder agreement on the third and fourth characters, classifying injuries according to their first and second characters can still minimise errors.18 This study demonstrated very high agreement on both the first and second characters. It is recommended that when coders have non-clinical backgrounds that only the first two characters are used to ensure the injury is correctly identified. In most epidemiological applications, this two-character level of coding would be sufficient for injury description and trend monitoring.
Coders with non-clinical backgrounds generally have a lack of specific anatomic or diagnostic knowledge and may benefit from receiving formal training in some aspects of this to improve their coding accuracy. In the Australian Football League (AFL) Injury Surveillance System coordinated by author JO, sports medicine staff at each professional AFL teams prospectively record player injury status and the full medical diagnosis of each injury. An OSICS code is then assigned centrally and sent back to the club for confirmation and corrected, if necessary. While this approach may be suitable for professional sport with medical support teams, it is unlikely to be appropriate for all settings, including community sport.
The major limitation of this study is that only two coders were involved, with a third adjudicating when discrepancies occurred. However, the non-clinical coder backgrounds are typical of the researchers who would most commonly code injuries in large-scale epidemiological studies, particularly with the frequent absence of medical or allied health professionals in non-professional sport. There could be scope to develop a list of injury categories that could be used by non-clinicians to aid better coding across several sports. The strength of the study was the large number of injuries coded and these covered a spectrum of severities from epistaxis to suspected fractures. As all data were collected on the field where the injuries occur, it is likely that many overuse injuries were not captured and so this evaluation is probably most relevant for the coding of acute or traumatic injuries.
Conclusions and recommendations
In summary, this study has found an overall high level of agreement in hierarchical OSICS-10 coding to the two character level, which would be appropriate for most epidemiological applications. In studies requiring more diagnostic specificity (ie, to the third or fourth character level), one beneficial strategy could be to ensure that a panel of experts reviews identified cases to obtain consensus on their coding. Irrespective of the post-data collection coding, it is strongly recommended that future injury data collections ensure the recording of high-quality text-fields for recording additional detail, over and above what might be requested with prespecified tick box options. Such information could provide additional information about the nature of injury, or the actual treatment received, to improve specific diagnoses. This is consistent with narrative text descriptions being recommended as core-items of any injury surveillance system.4 ,5
What this study adds
-
This study emphasises the importance of the background expertise of data coders in ensuring sports medicine diagnoses are fully accurate and with a high degree of diagnostic specificity.
-
Where possible, teams of coders with varying expertise should be used to agree on sports medicine diagnoses in epidemiological studies.
-
Data coders with no clinical background or limited anatomical knowledge can accurately use the OSICS codes to the two digit level, which would be appropriate for most epidemiological applications.
Acknowledgments
Dr Tim Doyle is thanked for his contributions to the data collection phase.
References
Footnotes
-
Contributors CFF conceived the study, coordinated its conduct and had major responsibility for the paper writing, assisted with coding of data. JO assisted with coding of data and contributed to the writing of the paper. DMT assisted with coding of data and contributed to the writing of the paper. MSS undertook database management and analysis and contributed to the writing of the paper. CLE contributed to the writing of the paper. DGL and BCE both contributed to the design of the study that the data came from and provided editorial input into the paper writing.
-
Funding The PAFIX study was funded by a nationally competitive research grant from the (Australian) National Health and Medical Research Council (NH&MRC)—Project ID 400937. CFF was supported by an NHMRC Principal Research Fellowship (ID: 565900). The Australian Centre for Research into Injury in Sport and its Prevention (ACRISP) is one of the International Research Centres for Prevention of Injury and Protection of Athlete Health supported by the International Olympic Committee (IOC).
-
Competing interests None.
-
Ethics approval University of Ballarat.
-
Provenance and peer review Not commissioned; externally peer reviewed.