1 Updated Findings from an Environmental Scan and EvaluationAssessing Online Systematic Review Training: Updated Findings from an Environmental Scan and Evaluation Leah Boulos, Sarah Visintini, Robin Parker, Krista Ritchie, Jill Hayden Presented by Leah Boulos CHLA/ABSC Conference 2017 May 17, 2017
2 Objective To conduct an environmental scan and assessment of online systematic review training resources in order to: Describe available resources Evaluate whether they follow current best practices for online instruction
3 Methods: Environmental scanBroad Google search Exhaustive YouTube search Targeted website search Google search string: ((“systematic review” OR “scoping review” OR “evidence review” OR “knowledge synthesis” OR “evidence synthesis”) online teaching OR course OR workshop OR seminar OR education OR training OR module)
4 Selection criteria Six systematic review steps: Content FormatAvailability Language At least three of six systematic review steps Online courses, videos, web tutorials or modules Available to the public or a group to which membership is open English Six systematic review steps: Defining a research question and/or creating a protocol Conducting a rigorous search Determining selection criteria Critical appraisal and/or risk of bias assessment Data extraction Analysis and/or creation of an in-depth report
5 Methods: Evaluation Evaluation framework based directly on Foster, Shurtz, & Pepper (2014) In turn based on QuADEM approach Four categories, 26 questions, 37 possible points Content Design Interactivity Usability
6 Methods: Evaluation Content Design Interactivity UsabilityCredibility, relevance, currency, organization, ease of understanding, focus, appropriateness of language Levels of Bloom’s Taxonomy, learning objectives, learning styles Interactivity Usability Level of interactivity, variety of tasks, appropriateness and difficulty of tasks, opportunities for reflection and feedback Layout, navigation, compliance with Americans with Disabilities Act
7 Methods: Analysis All resources separately and top five (top quartile)Produced descriptive statistics Repeated measures ANOVA between scores in each category
8 Results: Environmental scanResources identified through environmental scan (n = 55) Multi-part resources combined to make new total (n = 48) Resources screened for eligibility after duplicates removed (n = 41) Resources excluded (n = 14): Did not meet content criteria (n = 10) Did not meet format criteria (n = 4) Resources identified as candidates for evaluation (n = 27): Creators contacted (n = 13) Resources excluded (n = 7): No contact information (n = 2) No response (n = 5) Resources included for evaluation (n = 20)
9 Results: Environmental scanAudiences: researchers, health care professionals, and students Creators: universities, research organizations, and government agencies
10 Results: Environmental scanTime to complete ranges from < 1 hour to over 200 hours 14/20 resources available free of charge Prices range from $15 USD to over $3,000 USD
11 Results: Evaluation
12 Results: Top five resourcesTop five resources (top quartile) Resource # Resource name Creator R01 Comprehensive Systematic Review Training Program (CSRTP) Joanna Briggs Institute R02 Introduction to Systematic Review and Meta-Analysis Johns Hopkins University (through Coursera) – FREE R03 Online Learning Modules for Cochrane Authors Cochrane Training – FREE R04 Introduction to Systematic Review and Meta-Analysis Course Dalla Lana School of Public Health, University of Toronto / Knowledge Translation Program, St. Michael's Hospital R05 Systematic Reviews: diversity, design and debate EPPI-Centre
13 High cost did not always correlate with high scorePrincipal finding #1: High cost did not always correlate with high score
14 Results: Content Coverage of systematic review steps% (n) of resources covering step Defining a research question and/or creating a protocol 90% (18) Conducting a rigorous search Determining selection criteria 80% (16) Critical appraisal and/or risk of bias assessment Data extraction 85% (17) Analysis and/or creation of an in-depth report 95% (19) 12 resources covered all 6 steps Five resources covered 5/6 steps Remaining three resources covered 3/6 steps
15 Results: Content
16 Audience was frequently undefined, leading to lower scoresPrincipal finding #2: Audience was frequently undefined, leading to lower scores
17 Results: Usability
18 Results: Design
19 Results: Interactivity
20 Low interactivity scores were related to deficiencies in designPrincipal finding #3: Low interactivity scores were related to deficiencies in design
21 Principal finding #3 In the top five, average content–design difference is 9%; average content–interactivity difference is 1% In the remaining 15 resources, the differences are 38% and 45%, respectively
22 Format correlated with score; online courses performed bestPrincipal finding #4: Format correlated with score; online courses performed best
23 Principal finding #4 As a reminder, here are the overall scores by format:
24 Summary: Principal findingsHigh cost did not always correlate with high score Audiences were frequently undefined Low interactivity scores were related to deficiencies in design Format correlated with score; online courses performed best
25 Discussion: RecommendationsInclude measurable objectives and increase interactivity to cover more levels of Bloom’s Taxonomy Improve video resources Recommend appropriate resources
26 Discussion: Future researchWhich users would benefit from different resource types and formats? User testing of high-quality resources Evaluating learner satisfaction Assessing rate of completion and quality of resulting reviews
27 Discussion: LimitationsEnvironmental scan currency Systematic review step coverage Access to complete resources
28 Conclusions Reflect on the material currently being used and developedBe aware of common limitations of online resources Keep in mind important elements of content, design, interactivity, and usability
29 Thank You Contact Leah Boulos References [email protected]Evidence Synthesis Coordinator Maritime SPOR SUPPORT Unit References Foster M. Evaluation of best practices in the design of online evidence-based practice instructional modules. JMLA. 2014;102(1):31-40. Galipeau J, Moher D. Repository of ongoing training opportunities in journalology [Internet]. [Winnetka (IL)]: World Association of Medical Editors; c2017 [cited 22 Mar 2017]. Available from:
30 Appendix: Bloom’s TaxonomyArmstrong P. Bloom's taxonomy: the revised taxonomy (2001) [Internet]. Nashville (TN): Vanderbilt University, Center for Teaching; c2017 [cited 10 May 2017]. Available from: