1 A Randomized Controlled Study of Face-to-Face and Web-based COMPASS ConsultationAn Example of an Evidence Based Implementation and Intervention Practice in the Schools Lisa Ruble University of Kentucky John McGrew Indiana University-Purdue University Michael Toland Thank you for being here today. I’m going to describe our research in teacher consultation comparing a web-based vs face-to-face coaching for teachers of students with autism.

2 Why Schools? Only public funded service provider for children with disabilities May be the sole provider for children of low income, minority, or less educated mothers More than 500% increase in students served High burnout…. National shortage teachers Three times higher costs for education Less than 10% of research supported practices used in classrooms __________________________________ Schools are highly significant as an area of services research because they are the only public funded service provider for children with disabilities They may also be the sole provider for children of low income, minority, or of less educated mothers There has been at least a 500% increase in students served But there is High burnout and a national shortage of special education teachers The costs for educating students with Autism is 3 to 5 times higher compared to costs for educating other students Teachers instructional strategies lack research support and less than 10% of research supported practices are used in classrooms Hess et al., 2008; Morrier, et al., 2011; Ruble, et al., 2010; Simpson et al., 2011; Stahmer et al., 2005

3 Clearly, we need to address the growing research to practice gap

4 Implementation ScienceThe processes and procedures that help or hinder the transfer, adoption, and use of evidence-based practices. _________________________________________________________ Dunst (2012). Framework for Conceptualizing the Relationship Between Evidence-Based Implementation and Intervention Practices. Kelly, B., & Perkins, D.F., (Eds.). (2012). Handbook of implementation science for psychology in education. Cambridge, England: Cambridge University Press. Implementation practice Intervention practice Practice outcome Clearly, we need to address the growing research to practice gap. Implementation science provides a framework; here is one suggested by Dunst. Implementation practices include methods and procedures used by implementation agents (trainers, coaches, instructors, supervisors, etc.) to promote interventionists’ use of evidence-based intervention practices. Intervention practices include methods and strategies used by intervention agents (teachers, therapists, clinicians, parents, etc.) to affect changes or produce desired outcomes in a targeted population or group of recipients (e.g., infants and toddlers with disabilities). •

5 Evidence Based Interventions“Focused treatments” National Professional Development Center OCALI – Autism Modules National Autism Center Intervention practices based on research are numerous. Dr. Sam Odom and his colleagues identified 24 practices. These represent focused intervention strategies because they target a specific skill. You can find more information about each of these methods on their website at the National Professional Development Center for ASD. These websites provide helpful reviews of teaching methods, checklists, including implementation checklists, and video examples. Also, OCALI and the National Autism Center have online sources too. The challenge is that teachers are unlikely to be able to discern which practice to use for which child under that condition. This leads to the need for the study of an implementation practice.

6 Consultation Consultation is effective and has a “multiplier effect”By supporting teachers, we support an even larger number of students ____________________________________ Busse et al., 1995; Medway & Updyke, 1985; Sheridan et al., 1996 Consultation is the selected intervention practice studied. IT is well suited for meeting the high needs of better trained and supported teachers. By increasing the skills of teachers, we can serve a larger number of students. Research reviews indicate consultation is effective. Many models of consultation have been studied.

7 Consultation As implementation & intervention practiceQuality of the procedures as delivered by the implementation agent (Consultant) Quality of the strategies as delivered by the intervention agent (Teacher) Implementation Practice COMPASS Intervention Practice Instructional Quality Practice Outcome Child Goal Attainment Consultation can include methods and procedures to promote interventionists’ use of evidence-based intervention practices. In our case, we examined quality of instruction measured by IEP quality and teacher adherence for implementing teaching plans and their association with child goal attainment change over the school year.

8 Overview of COMPASS (Collaborative Model for Promoting Competence and Success)Decision-making framework Based on assumptions of child- environment interaction as critical – ecological framework Proactive problem solving Research-supported practices Teaching plan is specific to autism Forms are specific to autism Teaching strategies are linked to each specific skill __________________________ The consultation model we have been working with over the past several years is called COMPASS. It was specifically developed as an autism – specific program planning, decision making process for gaining consensus on the child’s targeted goals through understanding the child and the child’s environment at school and at home. It is also based on the assumption that all children have the ability to learn and learning is influenced by environmental in addition to individual factors Proactive problem solving that can be done on an ongoing basis is demonstrated and encouraged Provides a link to focused research supported practices Ruble, Dalrymple, & McGrew, 2012

9 Research Questions Can we replicate findings from a previous RCT of COMPASS and TAU (d = 1.5) Does COMPASS work as well when delivered via Web based technologies? Child goal attainment outcome Fidelity of intervention practice Teacher satisfaction ______________________________ Ruble, Dalrymple, & McGrew, NIMH RC1MH089760 With funding from the challenge grants, we were able to purse two primary questions: The first one was whether or not we could replicate the findings that COMPASS is effective and has a very large effect size. We also wanted to know how well COMPASS works when we vary the delivery of the coaching sessions using web based technologies compared to a face-to-face approach.

10 Design Teachers randomized to TAU+, FF, or WEB group (N=44)TX: FF COMPASS consultation at start of school year (parents and teachers) Similar to our prior study, we randomized teachers, but this time we used three groups. One third of the teachers did not receive anything directly from the research team but did receive online instruction on three research supported methods through the OCALI AIMS modules. The other two consultation groups received the initial compass consultation, but the coaching varied. One group received web based coaching, the other received face to face coaching. Half received 4 FF coaching sessions (n = 15) Half received 4 WEB coaching sessions (n = 14) FF = face-to-face; WEB = web-based

11 Group Comparison Intervention Groups TAU Group 3 hour consultationAssessment of baseline skills Services as usual + Online training Final evaluation Intervention Groups 3 hour consultation (parent & teacher) 3 IEP objectives Measurable Teaching plans Goal attainment scales 4 teacher coaching sessions (FF or WEB) ( / 4-6 weeks) Final evaluation More specifically, the intervention group, on the right, received a 3 hour COMPASS consultation that included the teacher and caregiver. Teachers were asked to update the child’s IEP after the consultation and include the goals. Teaching plans for each skill were developed using the information shared about the child’s personal and environmental challenges and supports as part of program development. After the consultation, four teacher coaching sessions were provided, about every 4-6 weeks. Each was between 60 and 90 minutes. At the end of the school year, the final evaluation focused on measurement using goal attainment scaling of the three IEP objectives developed during the consultation for the experimental group and the three IEP objectives evaluated at the start of the year for the control group. The final observer was done by an independent observer unaware of group assignment.

12 WEB Group: Teacher EquipmentTeachers in the web group received a laptop, webcam, headphones, and a flip mino mini digital camera. Teachers in the FF condition also received the flip mino so that we didn’t have to interrupt the classroom to observe the child-teacher instructional situation for the coaching session.

13 Adobe Connect Session Here is an example of the actual adobe meeting captured with a picture. We used ADOBE connect as the videoconferencing software.

14 Enhanced Services As Usual (n = 15)Time 1 Comparisons Enhanced Services As Usual (n = 15) Face-to-Face (n = 16) Web-Based (n = 18) Variable M SD F(2, 46) p ADOS (S&C) 17.9 3.7 17.8 4.0 18.6 0.2 .84 DAS1 61.3 24.6 60.9 17.0 44.6 20.6 3.5 .03 OWLS1 53.8 13.7 57.3 14.7 49.6 10.7 1.5 .23 Vineland (TR)1 58.6 12.8 62.0 13.5 58.3 13.8 0.4 .67 Child age (years) 5.6 6.4 1.6 5.9 1.7 1.0 .61 Years teachinga 1.2 2.2 0.9 3.0 2.3 3.6 1.9 .15 Students taught 4.5 9.0 7.3 7.0 6.9 2.8 .06 Num services2 1.4 1.1 .32 Hrs services2 12.3 20.8 6.8 .34 Family incomeb 26.5 21.4 26.9 .51 We used an intent to treat approach and analyzed all data on pretreatment variabales at Time 1. All variables, except child IQ were similar between groups. Child IQ lowest for the Web Based group. As a result it was used as a covariate in our final analysis.

15 Conceptual Framework Implementation Practice COMPASSIntervention Practice Instructional Quality Practice Outcome Child Goal Attainment Recall Dunst’s conceptual framework we applied. Next we’ll describe the practice outcomes and the intervention practice of instructional quality.

16 Practice Outcome Planned Comparisons* WEB FF TAU d = 1.12 d = 1.41We replicated our findings that GAS scores from FF COMPASS was significantly higher compared to TAU scores. Further, the WEB condition was significantly higher compared to the TAU condition, but similar to the FF condition. Planned Comparisons* WEB FF TAU d = 1.12 d = 1.41 d = 0.27 *adjusted for DAS scores

17 Implementation and Intervention Practice FidelityImplementation Practice Fidelity – What the Consultant Did Initial Consult: 80-90% of features implemented Coaching: 3.8 / 4.0 No diff FF and WEB Intervention Practice Fidelity by Coaching Session- What the Teacher Did Group 1 2 3 4 FF 3.6 3.4 4.0 4.2 WEB 3.7 4.1 Analysis of the quality of the implementation practice – COMPASS indicated that the consultation was implemented with good fidelity as reported by parents and teachers. Coaching sessions were also implemented with good fidelity and no difference between the WEB or FF group was noted in fidelity. Analysis of the quality of the intervention practice which was how much of the teaching plans were implemented during each coaching session, indicated higher adherence over time when analyzed overall and no difference between the WEB and FF conditions. 11-5 Likert Scale 1 ‘0%’; 5 ‘100%’ No diff FF and WEB. Significant difference in adherence ratings across coaching sessions, 2(3) = 12.39, p = .006, Kendall’s W = .15

18 Satisfaction Median = 3.7 / 4 Initial Consult: Coaching:No difference between FF and WEB groups for teachers, z = -0.07, p = .95, r = .01, and parents, z = -0.98, p = .33, r = .19. Coaching: No difference between the WEB (M = 3.2, Median= 3.3, SD = 0.62) and FF groups (M = 3.2, Median = 3.3, SD = 0.44), z = -0.48, p = .63, r = .09. Both parent and teacher satisfaction for the initiation Consultation was high and no difference between parents and teacher report was observed. Satisfaction in coaching was still high but somewhat lower than the initial consult (3.2 vs 3.7) and no difference between WEB or FF group was observed.

19 COMPASS Active IngredientsChild Outcome COMPASS Consultation Personalized Goals IEP Quality Personalized Teaching Plans Coaching Due to time, I am unable to describe more about the active ingredients in much detail, but we believe IEP quality (determined by having measureable goals and social, communication, and learning goals) is important as well as teacher adherence to implementation of teaching plans. Active Ingredients

20 Active ingredients IEP quality Teacher adherencer = .61, p< .001 (replicated from study 1) Teacher adherence r = .23, p = .11 (did not replicate) Restricted range of scores Need to examine teacher competence, not just adherence IEP quality correlated with child goal attainment outcomes. This is a replication form our first study. BUT teacher adherence did not correlate with child outcome. A finding not replicated. Further analysis showed that adherence scores were limited to the top three possible responses, unlike scores in the first study that had a higher range. Also, we need to look at competence, not just adherence – which is a focus of future research.

21 Likely Features of Effective Consultation ModelsCollaborative vs expert approach with teachers, families & therapists Personalized goals & teaching plans Measurable goals/objectives Reflective practice & feedback Progress monitoring & data keeping Cultural sensitivity of family values Although we studied a specific consultation model, there are likely key ingredients that can be applied in other models. In other words, one may not necessarily have to use our model. We believe these are some of the essential components: Collaborative vs expert approach with teachers, families & therapists Personalized goals & teaching plans Measurable goals/objectives Progress monitoring & data keeping Cultural sensitivity of family values Reflective practice & feedback

22 Conclusions COMPASS replicated in 2 RCTsWeb based coaching is a promising approach for improving outcomes Fidelity equal to FF Satisfaction equal to FF Child outcomes equal to FF COMPASS needs to be evaluated when implemented by school-based practitioners COMPASS is now replicated in 2 RCTs Web based coaching is a promising approach for improving outcomes and access to services Fidelity, satisfaction, and child outcomes are similar between the WEB and FF conditions Next step is effectiveness research to test COMPASS when implemented by school-based practitioners

23 Acknowledgements Teachers, Indiana & Kentucky Families and ChildrenNancy Dalrymple, Co-investigator Jennifer Grisham-Brown, Co-investigator Research team, UK RAs: Ryan Johnson & Lauren Feltner GRAs: Rachel Aiello, Jessie Birdwhistell, Jennifer Hoffman, Rachel Wagner Research was supported by Grants No. R34MH and RC1MH from the National Institute of Mental Health This presentation is based on a group effort and I want to acknowledge the school administrators who allowed us to work with their teachers; the teachers and parents who agreed to participate in our work, and Nancy Dalrymple, co-developer of COMPASS, as well as other team members and our funding source, NIMH.