1 Easing the Transition to College: When and How Is Transition Coaching Effective?March 4, 2017 Tamara Linkow*, Beth Gamse*, Fatih Unlu**, Erin Bumgarner*, Hayley Didriksen*, Jane Furey* * Abt Associates; ** RAND

2 Policy Context Access to jobs—and to the middle class— increasingly requires postsecondary credentials Students from low-income backgrounds and racial/ethnic minority groups fail to enroll in, persist in, and graduate from college at rates similar to their White, Asian, and more advantaged peers Specifically, such students face limited support: socially and academically (Arnold et al. 2009; Roderick et al. 2008; Scott-Clayton 2011), and to manage key deadlines (Avery and Kane 2004; Castleman and Page 2015) Introduction

3 Improving College Outcomes for Disadvantaged Students: Previous ResearchSeveral studies demonstrate that purposeful pre- and early enrollment support can improve outcomes for low- income and minority students, including using one or more of the following promising supports and practices: knowledge of the college meeting students where they are relationship building time management skills support with logistics developing self-sufficiency nudges and reminders to complete time-sensitive tasks providing social-emotional supports proactive outreach (See: Avery, Howell, and Page 2014; Bettinger and Baker 2014; Carrell and Sacerdote 2013; Castleman, Arnold, and Wartman 2012; Castleman, Page, and Schooley 2014; Scrivener and Weiss 2009; Stephan and Rosenbaum 2013) Background

4 Citywide Coalition for Coaching (CCC)Launched in 2009, CCC provides one-on-one transition coaching to a sample of the city’s high school graduates who enroll in local 2-year and 4-year colleges The goal of CCC coaching is to help newly enrolling college students become independent and capable of navigating through to college graduation successfully   CCC Coaching incorporates many supports and practices found effective; it is sustained, proactive, responsive to student needs, and covers academic, financial, life, and career topics Full-time coaches from nonprofit organizations provide the coaching About the Program Discussion

5 Citywide Coalition for Coaching ModelAbout the Program Discussion

6 Study Overview Examines transition coaching – how is it implemented? Does it improve student outcomes? Focuses on one city’s concerted effort to improve college outcomes for its high school graduates Purposeful longitudinal lens: 2014 – 2020 Implementation data [student surveys, interviews with coaches, program records] Impact data [extant data from City School District, State, NSC, local colleges, program records] Introduction

7 CCC Recruitment AvenuesStudents are recruited through multiple avenues: referrals from high school guidance counselors and other community organizations nonprofit organizations’ middle school and high school programming pipelines word of mouth college referrals However, recruited students may differ vis-à-vis motivation, attachment to the nonprofit organizations, and willingness to seek support Recruitment

8 4 hours average of one-on-one coaching per yearCoaching Intensity One-on-one coach-student interactions in Average number: 8 Number of interactions vary by student—some students had more than 20 one- on-one interactions over the year Typical length: 30 minutes One-on-one interactions tended to last between 25 and 45 minutes 4 hours average of one-on-one coaching per year Common Practices

9 Examining Impacts of CCCOverall Program Impacts: What is the effect of coaching on early college outcomes, including: persistence, achievement, and FAFSA renewal? Variation in Program Impacts: How do outcomes for coached students vary, and are certain features of the coaching or characteristics of students associated with impacts? Impact Questions

10 Analytic Approach: Local and Focal MatchingQuasi-experimental, local and focal matching design Local: Comparison cases drawn from the same setting as the treatment cases, to the extent possible CCC students are matched with non-CCC students from the same cohort, postsecondary institution, and similar high schools Focal: Matching to ensure that treatment and comparison students are statistically equivalent on baseline characteristics believed to predict both selection into the CCC program and outcomes of interest Impact Design

11 Local Matching: Eligible Student SampleThe sample is drawn from: 2013 and 2014 city high school graduates Similar districts near the city, and then Who enrolled in college in the fall after graduation Coached Student Group Coached students from the city school district Non-Coached Student Group High school students from the city and surrounding districts who did not receive coaching Impact Sample

12 Focal Matching: Baseline EquivalenceMatched on over 32 baseline characteristics gender, race/ethnicity, high school academic achievement (GPA, SAT, and HS test scores), socioeconomic status, behavioral indicators, and knowledge of and motivation about college Assessed balance between treatment and matched comparison students using the standardized difference in means of the matching characteristics between treatment and comparison students (e.g., Steiner et al ; Rubin 2001) Required the difference to be less than 15 percent (0.15) of a standard deviation in absolute value Impact Sample

13 Radius Matching Each treatment student is matched with all potential comparison students with propensity scores within the pre-specified caliper of his/her score (±0.2 of the standard deviation of the propensity scores) in his/her block Treatment students with no potential comparison students within their propensity score caliper were unmatched and excluded from estimation of impacts This matching method balances two key features: closeness of the matches size of the matched groups Impact Sample

14 Focal Matching: Baseline Equivalence (T vs C) for Full SampleBefore Matching After Matching Treatment Mean Comparison Mean Std. Diff STUDENT CHARACTERISTICS Female 0.60 0.53 0.17 0.61 -0.02 Free or reduced-price lunch status 0.87 0.71 0.36 0.84 0.08 High school GPA 2.82 0.02 2.81 2.80 0.01 Took an advanced course 0.57 0.50 0.22 0.55 0.00 SAT score 1,233.07 1,363.81 -0.29 1,240.11 1,245.24 -0.01 10th grade Math test score -0.08 0.06 -0.04 School days present (percentage) 79.86 80.79 0.03 79.10 79.03 Number of suspensions 0.04 -0.07 0.05 Number of extracurricular years 3.08 2.95 0.18 3.04 Race/Ethnicity Black 0.41 0.14 0.42 -0.03 White 0.19 -0.38 0.07 -0.05 Asian/Pacific Islander 0.16 0.15 0.11 0.12 Hispanic 0.27 0.38 Native American Other/Multiracial HIGH SCHOOL CHARACTERISTICS College-going rate 0.59 0.62 -0.12 Average Math Test Score 47.83 50.30 -0.27 47.88 48.31 -0.06 Average high school-level GPA 2.41 2.54 -0.28 2.42 Impact Sample

15 Estimation Model 𝑌 𝑖𝑗 = 𝜋 0 + π 1 𝑇 𝑖𝑗 + 𝑏=1 𝐵−1 π (1+𝑏) 𝐼 𝑖𝑗 𝑏 + 𝑛=1 𝑁 π (𝐵+𝑛) 𝑋 𝑖𝑗 𝑛 + 𝜀 𝑖𝑗 where: 𝑌 𝑖𝑗 = outcome measure for student i in matching block j 𝑇 𝑖𝑗 = treatment indicator for student i in block j, which equals one if student i is an CCC student and zero otherwise 𝐼 𝑖𝑗 𝑏 =indicator variable for the b-th matching block for student i. It equals one if student i is a member of the b-th block and zero otherwise 𝑋 𝑖𝑗 𝑛 =n-th matching characteristic or covariate for student i in block j 𝜀 𝑖𝑗 = random error term for student i in school j, which is assumed to be normally distributed with mean zero and variance of 𝜎 𝜀 2 Impact Analysis

16 Primary Outcomes Persistence Achievement FAFSA RenewalCollege persistence into 2nd year College persistence into 3rd year Achievement College GPA Good Academic Standing FAFSA Renewal Impact Outcomes

17 Limitations Data availability varies for some outcomes limited to a subset of colleges; therefore selected impacts are based on ~70 percent of the sample Matching students across high schools and school districts: too few students to match same students from a given high school >> same college Quasi-experimental design rather than an experimental design: the local and focal matching approach may not sufficiently control for potentially confounding factors that an experimental design could Impact Limitations

18 Persistence into 2nd and 3rd Years} 13* } 8* Impact Results * Impact is significant at the 5 percent level; + indicates that statistical significance was determined with the Benjamini-Hochberg correction

19 Academic Achievement: Cumulative GPAImpact Results * Impact is significant at the 5 percent level; + indicates that statistical significance was determined with the Benjamini-Hochberg correction

20 } 7* FAFSA Renewal Impact Results* Impact is significant at the 5 percent level; + indicates that statistical significance was determined with the Benjamini-Hochberg correction

21 Impacts by Frequency of Coaching InteractionsPersistence † GPA FAFSA Renewal Impact Results *Impact is significant at the 5 percent level; † indicates statistically significant difference between high and low groups

22 Summary of Results CCC has consistent, statistically significant, and positive impacts on students across three domains: persistence, achievement, and financial aid Compared to non-coached students, CCC students: 11% more likely to persist into the 2nd year of college 21% more likely to persist into the 3rd year of college Achieved GPAs that are 8% higher 9% more likely to renewal their FAFSAs Impact Results

23 Discussion The impact results suggest that the CCC model has meaningful and significant promise CCC coaches provided students with an average of 4 hours of one-on-one coaching across the year, and had an average of 13 interactions with students A model in which professional coaches deliver sustained, proactive, and responsive support may not necessarily need to be “high touch” to help move more students move more quickly and effectively along the pathway to college completion Discussion

24 What Next? Annual impact updates until 2020, when we can obtain data for 6 years from the high school graduation for the 2013 cohort (5 years for 2014, etc.) Meanwhile … CCC has expanded to serve nearly three times as many students in the City (meaning likely challenges for finding comparison students) Implementation-focused research suggests that local colleges are expanding the types of support they provide Will effects decay? Discussion

25 Questions, comments? Questions? Contact:Tamara Linkow: Hayley Didriksen: Beth Gamse: Coached students are persisting longer, achieving higher academic progress, and taking the necessary steps to maintain funding for college—all good signs of progress toward college completion. However, the low number of credits accumulated toward graduation, particularly for students at two-year colleges, may be cause for concern.