Epidemiological monitoring studies, and formal evaluations of injury prevention measures, in community sports rely on the collection of reliable data.^1,2 Unlike studies in more professional or elite athletes, community based sporting teams rarely have medically trained personnel associated with teams or in attendance at games or training sessions. This means that relevant data need to be collected by either paid research staff or by volunteers associated with the teams themselves. For logistic and funding reasons, the use of team based primary data collectors (PDCs) is often the most feasible option. However, the reliability of data collection by PDCs has not previously been reported. The aim of this paper is to describe the reliability of using PDCs to collect exposure and protective equipment use data during a community based sports injury randomised controlled trial.

METHODS

The Australian Football Injury Prevention Project (AFIPP) was a randomised controlled trial to assess the effectiveness of headgear and mouthguards in community level Australian Rules Football. Nine metropolitan football clubs in Victoria volunteered 23 senior/junior teams for the project. Over the duration of one playing season, every player involved in AFIPP was monitored during all games and training sessions for their exposure (that is, amount of participation time) and their protective equipment behaviours.

Each club nominated a PDC to be responsible for the collection of exposure and protective equipment use data. The PDCs were formally trained and received a procedures manual and data collection forms to ensure standardised data collection processes. Exposure data forms required PDCs to place a tick next to each player’s name during training and games, if they were present, and a cross if they were absent. Similarly, compliance data forms were used to indicate whether players wore headgear and/or mouthguards. Overall, there were nine trained data collectors (one from each club) who were responsible for collecting exposure and protective equipment use data over the regular playing season. All PDCs were parents of players, team managers, or club presidents. All data collectors were instructed to be present for the entirety of training and games.

A random audit of the collection of the exposure and protective equipment use data by the PDCs was undertaken by the researchers. Each team was audited twice at training and twice during competition (that is, 23 teams were each audited four times for a total of 92 audits). Exposure and protective equipment use information was collected by the auditors using the same protocol and data collection forms as those used by the PDCs. All auditors and PDCs worked from the sideline of the playing field. The audits were conducted to assess the reliability of primary data collection in the field and did not consider the validity of the information collected.

The average percent agreement between the audits and the PDCs for both exposure and protective equipment use during both training and games was calculated. The two exposure measurements were compared by the kappa statistic and McNemar tests were used to compare protective equipment use (%) data. Kappa values were interpreted using the strength of agreement scale as reported by Feinstein.³

Ethics approval was obtained from the Deakin University and Monash University Human Research Ethics Committees.

RESULTS

The overall average percent agreement, across all teams, for the exposure and protective equipment use data is shown in Table 1.

Generally, the percent agreement in exposure data was highest for games with the overall strength of agreement falling in the “substantial” category. There was also a high level of agreement in the protective equipment use data. The McNemar tests indicated no significant differences between the PDCs and the random audits for either mouthguard or headgear use.

DISCUSSION

Reliability of measurement is important for the interpretation and generalisation of research findings.⁴ This is the first community based Australian Football injury study to assess the reliability of its field based data collection procedures for the collection of exposure and protective equipment use data. Overall, there was a high level of agreement between the team based PDCs and the independent audits. This suggests that the use of PDCs provides a reliable method for collecting this data.

The level of agreement in the exposure data was higher during competition than training. Competitive games occur in a structured session and this may make it easier to define players as either in attendance or not in attendance. In contrast, during training sessions, there are often more players on the ground doing many different tasks at any one time. The sessions are also less structured and not all players may be present for the whole time.

Although average agreement for exposure was high during a match (90.9%), one reason for it not being 100% was that players, although allocated playing numbers, may not necessarily remain in their numbered jumper for the entirety of the match or season (for reasons such as injury or lack of player jumpers). Identification, therefore, may be difficult as jumper number is the most convenient way, from the sideline, to identify players.

The reliability of the protective equipment use data was (slightly) higher during training than games. Previous studies have found that community level Australian Footballers are less likely to wear mouthguards/headgear during training.⁵ From a data collector’s perspective, it may be easier to record compliance data during training, as studies show that fewer players are likely to use protective equipment during this time.⁵ Unlike headgear, mouthguards can be difficult to see from a distance and, during competition, accurate monitoring of mouthguard use can only take place during the quarter time breaks. Even then, visibility of mouthguards can be hard to monitor as players often take their mouthguards out of their mouth and place them down their socks or in their shorts during the breaks. This could be one of the reasons why data agreement was higher during training for this piece of equipment.

Although there was some variability in the measurements recorded by the PDCs and the independent auditors, there is substantial agreement in exposure and protective equipment use data from both sources. This means that trained PDCs based in teams from community sports clubs can collect reliable exposure and protective equipment use data. They are therefore a reliable alternative source of data collection in community based studies of sports safety behaviours.