Article Text
PDF
Abstract
Background/aim The contemporary electronic media is regarded as a practical tool in the dissemination of preventive measures and interventions. For this purpose an App (free of charge) was developed including an efficacious programme for the prevention of ankle sprain recurrences. This study evaluated the implementation effectiveness of this ‘Versterk je Enkel’ App.
Methods The App was evaluated within its practical context using the Reach Effectiveness Adoption Implementation Maintenance (RE-AIM) Framework. The launch of the App was accompanied by a press release, website banners, as well as online and offline advertisements. Data for the evaluation of the App were objectively registered through Google analytics. Data were obtained in February 2013 based on 25 781 users resulting in follow-ups of 18 months (iOS version) and 15 months (Android version), respectively. Users questionnaires provided a qualitative view of the objectively assessed measures (n=82) to gain insight into the demographics of users, reasons to download, user experience and how the information was used.
Results The App reached only 2.6% of the projected target population. User ratings for the App's relevancy, clarity, usefulness, appeal, information and reliability were high. App usage indicates that compliance with the embedded programme was low.
Conclusions Although the App was well received by the users, targeted efforts are required to ensure proper uptake and usage of the App by the target population. This also holds true for eHealth and mHealth efforts aimed at athlete care and injury prevention in general.
- Ankle injuries
- Implementation
- Injury Prevention
Statistics from Altmetric.com
Background
Ankle sprains continue to pose a significant burden to the individual athlete as well as to society as a whole. Ankle sprains are the most common athletic injury,1–3 carry high costs from a societal perspective,4–6 and may result in long-term symptoms.7 ,8 As such, prevention of these injuries continues to be an important goal for sports medicine and injury prevention. Research has shown that externally applied supports and neuromuscular training programmes are very successful in preventing recurrent cases of ankle sprains, from the effectiveness and cost perspectives.5 ,6 ,9 Therefore, preventive measures, preferably through continued neuromuscular training, are recommended after rehabilitation in evidence-based treatment guidelines.10
Despite ankle sprains being prevalent, and despite an active approach to implement epidemiological evidence on (cost)effective measures, a wide-scale uptake of effective preventive measures, and thus actual prevention of ankle sprains under real-life conditions, is lagging behind. The previously mentioned neuromuscular training programme for the secondary prevention of ankle sprains had poor compliance.11 On the level of the individual athlete, secondary analysis of data from a randomised controlled trial on the effectiveness of this programme indicated that the established preventive effect had been achieved in a subsample of compliant participants resulting in significant population effects.12 Although analyses considered an intention-to-treat approach, this study showed there is a lot to gain at the individual as well as the population level by increasing the uptake of and compliance with these simple effective preventive exercises. In general, this arguably also holds true for the wide-scale uptake of effective preventive measures under real-life conditions in sports medicine.13 ,14
In an attempt to bridge the implementation gap between research and practice there is an alleged role of eHealth in general and mHealth in particular.15–17 Generically, eHealth can be defined as the practice of medicine and public health supported by electronic processes and communication. In line with this definition, mHealth is the practice of eHealth using mobile devices and applications. In order to employ this potentially important communication channel to support the dissemination and implementation of preventive measures, the Dutch Consumer Safety Institute (VeiligheidNL) developed a freely available, evidence-based mobile application (App) to prevent recurrent cases of ankle sprains: ‘Versterk je Enkel’ (‘Strengthen your ankle’).18 ,19 An evaluation was undertaken to describe the implementation effectiveness of the ‘Versterk je Enkel’ App.
Methods
Intervention
The App contains an 8-week neuromuscular training programme described and evaluated for preventive effectiveness in a previous randomised controlled trial.6 ,11 ,20 The training programme offers a set of six exercises for the prevention of ankle sprain recurrences and has been linked to a 50% reduction in the recurrence risk, if used and complied with by the athlete.11 For the original trial offline printed materials were produced.20 The App contained a full translation of these materials into an interactive package, including videos of individual exercises and an interactive training schedule providing feedback through push messaging (figure 1). The original programme and materials were developed in cooperation with the Netherlands Association of Sports Medicine and Dutch organisations for physical therapy and sports physical therapy, the EMGO+ Institute for Health and Care Research and participating Dutch sports associations (soccer, volleyball). Volleyball trainers, coaches and athletes were involved in a pilot intervention.
Screenshots of the App.
The App was launched for iOS in September 2011, and for Android in November 2011. The App is free to download through the iTunes App store19 and Android Market.18 The launch of both Apps was accompanied by a press release, resulting in notable exposure in national online and offline media. In addition, advertisements and banners to guide potential users to the App were placed in the print media and on relevant sports and sports medical websites.
Design
We evaluated the App within its practical context using the Reach Effectiveness Adoption Implementation Maintenance (RE-AIM) Framework.21 ,22 The RE-AIM Framework designed by Glasgow et al21 is originally developed to evaluate the public health impact of health promotion interventions. This framework describes five interacting dimensions to evaluate the public health impact of interventions, that is, ‘Reach’, ‘Effectiveness’, ‘Adoption’, ‘Implementation’ and ‘Maintenance’. The operational definitions of these dimensions for our study are given below.
Data for the evaluation of the App were acquired through objective data of 25 781 users registered through Google Analytics,23 Android Market24 and Appfigures.25 These data were obtained in February 2013 resulting in follow-up periods between 18 and 15 months. In addition, App users were invited from within the App to complete an online questionnaire. This questionnaire provided additional and subjective insight into the demographics of users, how the users became aware of the existence of the App, reason(s) for downloading the App, frequency of App use, knowledge about the App content, user experience and App evaluation. A total of 90 questionnaires were completed from the launch of the App to May 2012 included, of which 82 contained complete data usable for analyses. Additional information on user experience was gathered through reviews in Android Market24 and iTunes App Store.25
Outcome measures
Reach
Reach was defined as the number of App downloads as a proportion of the total number of ankle sprains sustained in sports in the Netherlands over the follow-up period. From App launch to 28 February 2013, the number of downloads of the App have been obtained through the Android Market24 and iTunes App Store.25 The total number of ankle sprains, medically and non-medically treated, was estimated based on registry data of VeiligheidNL.3 Data were obtained from the national survey ‘Injuries and Physical Activity in the Netherlands’26 including demographic characteristics of the athletes, details of the sports injury (type of injury, anatomical region), medical treatment, cause of injury and injury mechanism. This survey interviews 10 000 respondents (4–80 years of age) through telephone or Internet. Data used in the current study covered 6 years (2006–2011) of continuous registration.
Effectiveness
Within the RE-AIM Framework effectiveness is defined as the success rate of the product to achieve its intended goal(s). In relation to the App, the overall goal is to reduce the number of recurrent ankle sprains through completion of the preventive exercise programme. The embedded exercises have been previously evaluated for effectiveness in a randomised controlled trial.11 Therefore, it can be argued that effective usage of the App will reduce ankle sprain recurrence risk. In addition, adoption and implementation of the App and its embedded exercises are described in other RE-AIM dimensions. Therefore, we chose to define effectiveness for the current evaluation as the user experience of the App, while the rationale to translate an effective preventive programme into an App was to increase compliance through enhanced user friendliness and experience.
User reviews and responses were obtained from the users questionnaires, as well as via the iTunes App store (n=84)25 and Android Market (n=18).24 In addition, the App's Appsfire score was noted.27 The Appsfire score is based on the rating and ranking performance over time, developer's track record and online reviews for an App, and provides an objective score between 0 and 100; a score higher than 75 corresponds to a ‘good’ rating.
Adoption
Adoption was described in terms of usage of the App after download. Data have been obtained through a combination of data registered through Google Analytics25 and users questionnaires. App usage data were obtained from the launch of the App onwards, providing objective insight into the actual use of the App after installing. Registered data included the number of unique users and frequency of use, number of page visits within the App, usage time per App visit, as well as new and recurrent use of the App.
Implementation
Implementation is the extent to which the intervention is implemented as intended in the real world. In relation to the current evaluation this was defined as the percentage of users that followed the embedded exercise programme as intended. Data from Google Analytics and questionnaires were used to describe implementation.
Maintenance
Maintenance is the extent to which the intervention is sustained over time. This is a difficult dimension to describe in relation to the current study, while the App implements an 8-week training programme for the prevention of ankle sprain recurrences. Exercises are no longer required after 8 weeks of training and prolonged use was not the goal of the intervention. In addition, App usage during the 8-week training period was described in the dimension adoption and implementation. Therefore, maintenance was not addressed in this evaluation.
Results
Reach
The annual number of all ankle sprains in the Netherlands due to sport participation is estimated at 650 000.3 Over a period of 18 months, this corresponds to an estimated 975 000 ankle sprains that make up the target population. Over the follow-up period the iPhone version was downloaded 20 262 times (of which 19 292 times in the Netherlands) and the Android V.5519 times by unique users, with an average number of 27 (iPhone) and 10 (Android) downloads per day. These totals correspond to a reach of 2.6%, which is a low percentage in light of the attention given to the App. Peak number of downloads was reached soon after the launch, and stabilised thereafter at a low level for the entire follow-up period.
Questionnaire data provided additional insight into the demographics of the users that downloaded the App. Of all respondents 72% (n=59 of 82) reported a previous ankle sprain, which was self-treated with 42% (n=34 of 82) of all cases. Reasons for download are depicted in figure 2 and indicate that the App was also downloaded out of interest rather than an actual intent to use the App for its content. As such, reach of the App into the target population is lower than 2.6%.
Reasons for App download (% of users, questionnaire data; n=82).
Effectiveness
Questionnaire respondents scored high on given statements regarding the App's relevancy, clarity, usefulness, appeal, information and reliability (figure 3). Respondents rated the App with a score of 8.1 of 10. This rating corresponds to the ratings from the Android Market (4 of 5) and iTunes App store (4 of 5). The Appsfire score is with a score of 62 of 100, distinctly lower than user reviews.
User ratings of App characteristics (% of users, questionnaire data; n=82).
In written iTunes and Android Market reviews users appreciated the clarity and ease of use of the App. The only written negative review (Android Market) indicated that the size of the App (16 MB) was too big for its content and functionality. A notable share of questionnaire respondents (19%; n=16 of 82) asked about the future availability of similar Apps for other injuries and other anatomical locations, which may be regarded as an approval of the current App by the users.
Adoption
Of all questionnaire respondents 38% (n=31 of 82) did not actively use the App, whereas 33% (n=27 of 82) used the App frequently (ie, multiple times per week). Others report infrequent use of the App, which corresponds to Google Analytics data reporting that over the course of the follow-up period 24 360 unique users have accessed the App 80 670 times in total; 3.3 App sessions per user. Of all users 32.8% were new and 67.2% accessed the App more than once.
The App consists of 65 pages in total. Per session users viewed 9.4 pages within the App. However, most often 2 pages were visited (21.6%), followed by 20 pages or more (13.9%) or 1 page (10.4%). The bounce rate (ie, the percentage of users that only access 1 page within the App) was 10. This indicates a skewed distribution of users who actually use the App and those that do not. This skewness is also apparent in usage time per visit (figure 4).
Usage time per App visit (n=25 781).
Implementation
Google Analytics showed a mean of 3.3 App sessions per unique user over the follow-up period. This number lies far below the 24 prescribed exercise sessions embedded in the 8-week training programme (3 sets of exercises per week). Mean usage time per App visit was 16 : 25 min. One set of exercises will take 15–20 min on average. Only 32% (n=26 of 82) of all questionnaire respondents indicated to have followed the entire programme and 59% (n=48 of 82) stated to have followed part of the programme. However, it is unknown how long the latter respondents had been using the App at the time of questionnaire completion. It is likely that part of this group was still following the programme at the time of completion. Of all respondents who had not yet followed the entire programme the majority (65%; n=53 of 82) stated that they intended to do so in the future; only 4% (n=3 of 82) had no further intent to conduct the exercises and 26% (n=21 of 82) intended to complete more exercises.
Discussion
Until now, only a few Apps are publicly available containing a claim to prevent sports and physical activity-related injuries. A recent review of this topic found that of 18 iPhone and iPads Apps only four incorporate evidence on efficacious preventive measures or treatment protocols.15 This study evaluated the implementation effectiveness of the ‘Versterk je enkel’-App aimed at the prevention of recurrent ankle sprains through the advocacy of an evidence-based neuromuscular training programme. The dissemination of this App seemed encouraging, based on the total number of downloads since its release. However, these totals correspond to a low percentage of potential users, that is, athletes who sustained an ankle sprain over that period. In addition, actual App usage did not indicate that users conducted number of required exercise sessions for the embedded programme to be effective.
The objectively assessed information on App usage did not reveal what information was actually viewed within the App or how this information was used. As such, questionnaires were included in the App to provide a qualitative view of these objectively assessed measures. Owing to the low users questionnaire response (n=82), probably as a result of the non-prominent location of the questionnaire within the App, responses are most likely not representative of the entire population of App users. Unfortunately, no information is available on demographics of non-responders to make comparisons. It is likely that responders are those with positive experiences and therefore are motivated to complete the questionnaire. This argument arguably also applies to those who took the effort to review the App within the iTunes Store or Android Market. As such, we believe that the actual user experiences and ratings may be lower than described.
Recent reviews and content analyses regarding the use of social networking websites (SNSs) with regard to concussion management in sports have indicated that SNSs have the potential to be a viable adjunct to traditional concussion management programmes.16 ,17 ,28 General practitioners’ opinion towards their use in concussion management has also been shown to be positive.16 Overall, these studies indicate that SNSs have a high potential for the sharing and dissemination of knowledge regarding prevention and management of specific injuries in sports. Owing to the widespread use of smartphones within the community,29 similar uses may arguably be viable for Apps as well. The current evaluation, however, indicated that many issues need to be addressed to realise this potential for this App and may apply to other health-related Apps as well.
In light of the above the title of the current manuscript may be misleading, while no structured implementation plan was employed. The implementation of the App followed an ecological approach. Regular dissemination channels of the Dutch Consumer Safety Institute were followed, resulting not so much in an ‘implementation’ strategy but more so in a ‘marketing’ strategy to promote the App within the target population. Unfortunately we have not been able to track the successes and failures of this approach, and have only been able to establish how effective the strategy was in reaching the target population through the outcomes described in the current paper. The current study has indicated that a ‘marketing’ type of strategy may not be the optimal method of implementing an evidence-based App. Consequently, future similar efforts are recommended to follow a structured and well-described implementation scheme. We also need to know the target population better before venturing in future implementation of evidence-based prevention Apps. What can be read between the lines of the current results is that the target population may not be intrinsically interested in the use of Apps for injury prevention. From the results of the current study one could argue that the exposure of the App to the public may not have been optimal, but if the target population was interested in Apps for injury prevention they would have come across the App in the iTunes store, Android Market or any search engine.
The reach and adoption of this App may be improved following Rogers’ Diffusion of Innovations Theory.30 This theory is considered to be one of the most successful approaches for understanding the uptake and adoption of interventions,31 and defines relative advantage, compatibility with needs, simplicity and trialability as relevant intrinsic characteristics of innovations that influence an individual's decision to adopt or reject an innovation. These factors may be influenced in a positive manner if the App is consistent with an athlete's training regime, treatment and rehabilitation, and is personally recommended by professionals in sports and medicine so-called influential opinion leaders. In addition to our efforts to promote the App through online and offline media, it might therefore be valuable to have general practitioners, physical therapists, trainers and coaches promote proper usage of the App by the target population. User reviews in our study appear consistent with this need. It is recommended to further study various contexts by which these tools can be effectively employed as a communication channel to support implementation for athlete care.
Conclusions
Despite widespread marketing of the App through a variety of channels, reach and adoption of the App within the target population was found to be low. In general, results of the current study indicate that targeted efforts are required to ensure proper uptake and usage of the App by the target population. We believe the latter holds true for eHealth and mHealth efforts targeted at athlete care and injury prevention in general.
What are the new findings?
-
Providing athletes with an mHealth tool, supported with exposure in national online and offline media, is insufficient to disseminate effective interventions to the field.
-
Although eHealth and mHealth are considered important tools for the sharing and dissemination of knowledge on prevention and management of specific injuries in sports, targeted efforts are required to ensure proper uptake and usage by the target population.
-
Further studies are required on contexts by which eHealth and mHealth tools can effectively be employed as a communication channel to support the dissemination and implementation of preventive measures for athlete care.
How might it impact on clinical practice in the near future?
-
eHealth and mHealth, and specifically the use of Apps and Social Media, are likely to become integrated parts of clinical practice.
-
Targeted implementation efforts are required to ensure proper uptake and usage of eHealth and mHealth tools by the target population.
-
Well-designed and planned evaluations must be conducted to describe the effectiveness and implementation success of eHealth and mHealth tools.
References
Footnotes
-
Contributors EV, the guarantor of this manuscript, conceived the idea and formulated the study hypothesis. IC and IV conducted the evaluation. All authors interpreted the data, discussed core ideas and participated in writing the paper. All authors had full access to the data (including statistical reports and tables) and take responsibility for the integrity of the data and the accuracy of the data analyses.
-
Competing interests None.
-
Provenance and peer review Not commissioned; externally peer reviewed.
-
Data sharing statement Data are available on reasonable request from the authors. Full dataset available from the corresponding author at e.verhagen@vumc.nl.