1 Quality By Design Statewide Human Services Workforce Evaluation Using an Integrated Framework American Evaluation Association Conference San Antonio, TX November 2010 California Social Work Education Center (CalSWEC) UC Berkeley, School of Social Welfare

2 CalSWEC Created in 1990, the California Social Work Education Center (CalSWEC) is a consortium of California’s 20 accredited social work graduate schools, the 58 county departments of social service and mental health, the California Departments of Social Services (CDSS) and Mental Health (CDMH), the California Chapter of the National Association of Social Workers, professional associations, and foundations. It is the nation's largest coalition of its kind working together to provide professional education, student financial aid, in-service training, and workforce CalSWEC facilitates the integration of education and practice to assure effective, culturally competent service delivery in the human services.

3 Goals The partnership’s goals are to:re-professionalize public human service through a specialized education program for public human services, develop a continuum that connects pre service education to in service training, engage in research and evaluation to develop evidence based practices advocate for responsive policies and resources to support practice improvement and client outcomes.

4 Today’s Discussion Review of Evaluations thus far:Title IV-E Stipend Program—MSW/BASW program that provides specialized preparation for work in Public Child Welfare Agencies In-Service Training Program for Child Welfare—develops common and specialized in-service curriculum for Public Child Welfare Agencies Mental Health Program—MSW program that provides specialist preparation for work in Mental Health Agencies Discussion of where we are headed: A conceptual model for integrating the evaluations using theoretical constructs and longitudinal design as guiding principles Evaluation goal to improve our ability to better assess the impact of these programs on practice and client outcomes. Developmental process. We are in the process of integrating program and methodology 3 parallel programs with common goals and have developed evaluation efforts that mirror one another.

5 Title IV-E Stipend Program

6 Overall Program Goal To recruit and prepare a diverse group of social workers for professional careers in public human services, with child welfare emphasis.

7 Title IV-E Research QuestionsIs the curriculum being delivered as intended? Are the students learning the curriculum? To what extent are the graduates able to practice what they learned in the program within the child welfare agencies? Do CalSWEC IV-E graduates remain in public child welfare? What effects, if any, has the project had on the public child welfare agency and workforce? Does the IV-E program have effects on child and family outcomes?

8 Are we meeting our overall goal?Entry/Graduation Survey 10 cohorts panel study of all MSWs in participating CalSWEC Universities Designed to compare IV-E MSWs with Non-MSWs. Measured diversity, attitudes toward poverty, motivations and career goals, preferences for field of practice and client population. Significant results: Compared to non-IVE, the IV-E students were: More diverse racially, and had more prior experience working in child welfare. More oriented to service & less to private practice More inclined toward helping the poor Preferred direct services, but not clinical work Preferred at risk or impoverished clients, but not medical or clinical clients These attitudes and preferences increased from entry to graduation

9 Is the curriculum being delivered as intended?Curriculum Snapshot Every two years, the Snapshot captures classroom & field program elements at the school level for 20 member programs Curriculum competency infusion & partnership activities with county and nonprofit agencies are included Schools also report on curriculum delivery methods & program structure

10 Are the Students Learning the Curriculum?New Graduate Survey Annual survey of recent graduates, completed six months into their employment Asked how well their MSW programs prepared them for public CW social work Graduates rank the quality of a variety classroom & field experiences Also asked what content they believe is most important for beginning SWs to know

11 Findings at 6 months to 1.5 years Post GraduationThe New Graduate Survey is done each year as the MSW classes graduate and begin their first jobs. It is delivered as a Web survey. Currently this component of the evaluation is an extension of the Curriculum Snapshot (In School Evaluation). The classroom experiences that best prepared graduates for their work in public child welfare were practical techniques (91%), followed closely by clinical education (85%); then vignettes/role-playing (79%) and diversity education (69%). The field experiences that best prepared them were hands-on direct practice experience (97%), then shadowing/observing (93%), followed by mentoring/guidance (84%) and supervision and prior experience (tied at 83%). Topics graduates most often noted their programs covered especially well were diversity/sensitivity education, child welfare policy and advocacy/social justice, followed by making assessments and relationship building and gaining client/family participation. Respondents noted that some MSW program components could have been done differently or were handled less than adequately by their programs: Field Placement Quality at nearly 85%, an increase of 15% over last year’s survey. Next noted was Working with the court system (82%), similar to the prior year’s responses. Also similar were the frequently cited Case management (76%), and Substance abuse education (nearly 79%). Clinical skills for assessment (73%) and selecting appropriate intervention models (71%).

12 To what extent are the graduates able to practice what they learned in school within the public child welfare agencies? and Do CalSWEC IV-E graduates remain in public child welfare?

13 3-Year Post Graduate Survey & InterviewOngoing study of CalSWEC IV-E grads 3 years post graduation. Measures Retention Status Work conditions Work experiences Work attitudes Demographics A large majority stay after completing their work obligation The major predictor of retention is Supervision. They often feel that the culture and stress of the agencies inhibits the ability to practice as they were taught.

14 Findings at 3 years Post Work ObligationThose who stay in child welfare differ significantly from those who leave after their work obligation is completed. Worker characteristics (being of mixed ethnicity and cohort) and previous county employment (except in the last model when county differences were controlled for) predicted retention. In addition to worker characteristics, at least one variable from each of these categories predicts retention: extrinsic job factors (salary, hours and supervisor support), intrinsic job factors (level of success), and response-to-job factors (client-related stress). One worker characteristic (cohort) and two response-to-job factors (burnout-emotional exhaustion and visit-related stress) were significantly associated with leavers. Supervisor support significantly increases the odds of several types of job satisfaction, except client-related job satisfaction. Peer support increased the odds of client-, growth-, office-, and salary-related job satisfaction.

15 5-Year Post Graduate Survey & InterviewOngoing study of CalSWEC IV-E grads 5-6 years post graduation Examines the graduates’ career paths with measures of: Retention Status Jobs, roles, and responsibilities Work experiences Demographics Many of the grads have stayed in CW for more than 5 years. Some have moved within the system to up level positions or work as instructors to interns. (Sherrill will say more in the Integrated piece)

16 Findings from the Retrospective Study on the 5 – 6-Year-Post Grads154 of the Retrospective respondents fell into this category 98 were county employees while IV-E 132 were women 51 were Caucasian 70 were bilingual Due to missing data we could not isolate these respondents and perform a separate analysis. However, we are now conducting a separate survey just for those who graduated 5-6 years ago, based on findings from the Retrospective study.

17 Findings: Retrospective Study—Training & LicensureHaving access to training more than twice a year is associated with longer retention. Getting agency support for licensure is associated with longer retention. Kaplan Meier analysis

18 Findings: 8 to 10+ years (The Retrospective Study)The data showed distinct drop-offs in retention of Title IV-E graduates at 3 & 10 years. Not surprisingly, length of service is longer in the field of child welfare than in the first position.

19 Findings: Retrospective Study—Supervisory PositionHaving at least one supervisory position is associated with longer retention. (Note how the distance between the curves gets larger at about 5-6 years.) However, holding an administrative or managerial position is not. What about those who are still in direct service positions? Difference between promotion and recognition.

20 What effect has the Title IV-E program had on public child welfare agencies?Workforce study Census survey of the California child welfare workforce conducted every 3-5 years How do agencies organize workers and assignments? What are the education levels of child welfare workers? Where are unmet needs in workforce diversity and language capabilities? What agencies are underserved by master’s level social workers? How many current workers are participated in and are interested in IV-E? statewide child welfare workforce survey. Every three to five years, our state contracts with the university to conduct a survey of administrators and individual workers to estimate progress towards supplying child welfare agencies with professionally educated, transcultural workers using the Title IV-E stipend program Results: IV-E MSWs in supervisor positions have increase over time. IV-E workers are diverse and many bilingual. Many workers express the desire to go back for a BASW or MSW.

21 Findings—Education LevelWhen comparing the workforce studies with regard to educational attainment we are able to better understand how the state is progressing in raising the educational level of the child welfare workforce. The 1992, 1995, 1998 surveys estimated the percentage of MSWs using administrative data. The 1998, 2004, and 2008 surveys used the Individual Worker Surveys to estimate the percentage of MSWs. Since different methods (different definitions of workers, different number of counties responding, and different categories combined) were used, the comparisons between years may not be reliable. However, The percentage of MSWs in the case-carrying social worker population appears to have grown from 1998 to 2008.

22 Findings-Language Of the survey respondents, more than one in four speaks a language other than English, primarily Spanish Los Angeles County has the highest proportion of Spanish-speaking staff By comparison of the 510 Title IV-E participants, 68.4% (349) state they speak another language besides English on the job

23 The unanswered question.Does the IV-E Program have effects on child and family outcomes? We sure hope so…… What effects does the IV-E Program have? The unanswered question.

24 Child Welfare In Service Training Evaluation in California:Where We’ve Been and Where We’re Going 24

25 Why evaluate child welfare training?Almost every practice improvement initiative involves training Very few public child welfare agencies systematically evaluate their training “The tail wags the dog” – The evaluation process informs the curriculum, which in turn informs practice. 1. Most practice improvement initiatives involve training. 2. Very few of them systematically assess the impact of training on trainees’ knowledge, skills, or ability to transfer the skills to the job. 3. “The tail wags the dog” – Evaluation forces the entire system to focus on what specific knowledge, skills and values are most essential to effective practice. 25

26 Key Partners in California’s Child Welfare In-Service Training Evaluation26

27 Key Steps 2001 - Macro Evaluation Team convenes, begins planning.PIP mandates development & Implementation of Framework for Training Evaluation [1] 2004 – Framework Completed and Adopted 2004 to 2009 – Partners implement Framework Late 2008/Early 2009 – Begin Strategic Planning process for next 3 year period Fall 2009 – Implementation of next strategic plan commences [1] Parry, C. & Berdie, J. (2004). Training evaluation framework report. Berkeley, CA: California Social Work Education Center. 27

28 Timeline of Activities28

29 How did we design evaluation efforts to determine if training had an effect?Framework for Evaluation: Levels of Training Eval in CA Level 1: Tracking training (Demographics) Level 2: Formative evaluation of training courses (course level) Level 3: Satisfaction Level 4: Trainee knowledge acquisition Level 5: Skills acquisition (as demonstrated in class) Level 6: Transfer of learning (TOL: use of knowledge and skill on the job) Level 7: Agency/client outcomes Resources: It requires a great deal of time and effort to plan and implement training evaluation efforts. What is important is that training evaluation efforts help us determine whether or not training is effective, especially given the amount of funding that is used for training. 29

30 Why Evaluate at these Levels?Chain of Evidence: Establishes a linkage between training and desired outcomes for the participant, the agency, and the client such that a reasonable person would agree that training played a part in producing the desired outcome. Is a way to assess the extent to which training has had an impact on trainees. As one moves from Level 1 to Level 7, there are more intervening variables that can affect trainees beyond training. These variables make it difficult to know if changes in trainees’ behavior are the result of training. The Chain of Evidence links findings from earlier levels of training evaluation to show a logical connection of what has been trained to behavior on the job or impacts on the agency or families. 30

31 Let’s see how far we’ve come…31

32 What do we know now? (Summary of Progress/Results by Level)Level 1: Demographic data – 4,608 new line worker hires from mid-2005 thru 6/30/10 540 new supervisors Level 2: Formative evaluation data – Comprehensive observation-based data used to revise 7 standardized statewide curricula Level 3: Satisfaction data – collected at regional level only Level 1: Demographic data captured for nearly all new child welfare social workers and supervisors (4608 trainees) since formal evaluations began in 2005 (data thru 12/31/08.) Level 2: Formative Evaluation dataCollected and analyzed data regarding training content and delivery, resulting in improvements to the Common Core. Level 3: This level of evaluation is completed at a regional level, and not on a statewide basis. Level 4: Knowledge Tests For topics in which knowledge pre- and post-tests were administered (Child & Youth Dev; Case Planning & Case Management, and Placement & Permanency): Trainees (new CWWs) improved at a statistically significant level in their scores from pre-to post-test. IV-E Effects: In some years and for some curricula, IV-E trainees achieved significantly higher scores at pre-test and at post-test than non IV-E trainees. However, they also achieved statistically significant gains from pre-test to post-test, indicating that they gained knowledge as a result of the training. Level 4, continued: For the topic in which a knowledge post-test only has been administered (Critical Thinking in Child Welfare Assessment: Safety, Risk & Protective Capacity): -Although no formal standard has been established that serves as a yardstick of mastery, the data indicates that trainees leave the classroom with a substantial level of knowledge related to the learning objectives for the course. 32

33 What do we know now? (Summary of Progress/Results by Level)Level 4: Knowledge test data – Pre & Post (3 multi-day curricula) Post-only (1 single-day curriculum) Significant learning from pre- to post-test (See sample report) IV-E Effects Level 1: Demographic data captured for nearly all new child welfare social workers and supervisors (4608 trainees) since formal evaluations began in 2005 (data thru 12/31/08.) Level 2: Formative Evaluation dataCollected and analyzed data regarding training content and delivery, resulting in improvements to the Common Core. Level 3: This level of evaluation is completed at a regional level, and not on a statewide basis. Level 4: Knowledge Tests For topics in which knowledge pre- and post-tests were administered (Child & Youth Dev; Case Planning & Case Management, and Placement & Permanency): Trainees (new CWWs) improved at a statistically significant level in their scores from pre-to post-test. IV-E Effects: In some years and for some curricula, IV-E trainees achieved significantly higher scores at pre-test and at post-test than non IV-E trainees. However, they also achieved statistically significant gains from pre-test to post-test, indicating that they gained knowledge as a result of the training. Level 4, continued: For the topic in which a knowledge post-test only has been administered (Critical Thinking in Child Welfare Assessment: Safety, Risk & Protective Capacity): -Although no formal standard has been established that serves as a yardstick of mastery, the data indicates that trainees leave the classroom with a substantial level of knowledge related to the learning objectives for the course. 33

34 Examples of Findings: Training Evaluation of IV-E vs. non IV-EFamily Engagement & Case Planning Test Versions: 1.25, 1.26. Date Range: Classes conducted between January 2008 and December 2008. N=743 complete pairs of pre‐ and post‐tests. Gains from pre‐ to post‐test are statistically significant for both Title IV‐E and non‐IV‐E participants. Title IV‐E participants scored higher at pre‐test and post‐test than non‐Title IV‐E participants. These differences are statistically significant.

35 Summary of Progress/Results, cont’dLevel 5: Skill assessment in classroom (2 Curricula with embedded skills assessment) – Majority of new CWWs made 3 out of 4 correct decisions when asked to indicate whether or not child maltreatment occurred. Level 6: Transfer of Learning – Completed regional studies on Transfer and Field Training. (See 2009 White Paper, noted at end of this PPT.) Level 7: Outcomes level – Focused on building blocks/chain of evidence Level 5: For topics in which skill is assessed in the classroom (e.g., embedded evaluation) that pertains to identification of physical abuse and sexual abuse (Child Maltreatment Identification, Parts 1 and 2): At least 80% (and in most years 90% or more) of new CWWs made 3 out of 4 correct decisions when asked to indicate whether or not child maltreatment occurred in a given case scenario. Level 6: The original Framework planned for two principal activities related to Level 6. This evaluation occurred in two phases: Phase 1 focused on self-report of transfer and could be implemented in the short term; Phase 2 examined the relationship between skill development in the classroom and demonstration of that skill in a work product. Phase 1: Central and Northern RTA participated Assessed the extent to which the provision of mentoring services: 1) increased perceived transfer (by workers and their supervisors) of Common Core knowledge and skills; 2) increased worker satisfaction with the job and feelings of efficacy; and 3) contributed to improved relationships with supervisors.*Had to drop data collection on this Phase due to staff turnover and the need to re-work design. Phase 2: Retrospective case study of an intervention in one Academy Focus group, interview, and survey data have indicated improvements in timeliness of court reports and increased use by supervisors of organizers such as a standard Court Control Log format in the unit and daily calendars for workers. Feedback was mixed with respect to report quality. Also, both workers and supervisors felt that the tools provided by the mentors had helped make the supervisors’ review process more consistent and made it possible for one supervisor to pick up where another had left off when necessary. Outside of Framework: IUC (CSULB & LA DCFS) “Portfolio and Training Guide for New CSWs” is in initial stages of evaluation. Level 7: Under the Framework, efforts have focused on developing the building blocks at the lower levels in a rigorous manner (as part of developing a chain of evidence). Overarching goal is to link training interventions to outcomes for children and families served by CWS. 35

36 Where are We Going? 36

37 Where Are We Going (by Level)?Level 1: Demographic data – continue collection Analyze lineworker, supervisor, IV-E trainee test data, added section on expectations for the work Level 2: Formative evaluation data – continue collection & integration. Revise forms for observers/trainers; develop forms for e-learning platform. Level 3: Satisfaction data – continue regional/county collection Level 1: -Lineworker Core: Demographic profiles and related analyses of lineworker core test data will continue. -Supervisor Core: Demographic profiles and related analyses of supervisor core test data will commence. -Analyses of IV-E trainee test data. Level 2: -Formative evaluations for observers (and separate ones for trainers) will be divided into assessments of content and assessments of delivery. -Formative evaluation materials also will be developed for a new statewide venture: the e-learning platform. Level 3: These efforts will continue solely at the regional and county levels. 37

38 Where Are We Going? (cont’d…)Level 4: Knowledge tests – continue Move toward more diagnostic testing (focused on key knowledge) Continue to analyze differential functioning of items based on demographic characteristics Pilot study to assess possible effect of stereotype threat in trainee test performance (SDSU site) Explore trainer-level differences in test item performance Compare/monitor Title IV-E vs. non-IV-E performance Level 4: Knowledge tests – continue -Continue knowledge tests (consists of multiple choice test questions, aka “test items”) for the curricula currently evaluated at this level. -Move toward diagnostic testing: Focus on key learning objectives, Make targeted revisions to training based on evaluation data. -Continue analysis of differential functioning by demographic groups. -Explore trainer-level differences in test item performance to provide feedback on fidelity of curriculum delivery. Differential Analysis: Using the demographic data linked by a unique identifier code, CalSWEC analyzes the validity of test items to determine if particular demographic characteristics (such as race or gender) were associated with a correct or incorrect answer. Test-takers who perform well on the tests as a whole should perform well on individual test items, and should perform well regardless of differences in demographics or the area of the state where they are trained and where they practice. If performance on a test item is unrelated to general performance, or is interpreted and answered differently by subgroups of trainees, then that item is functioning poorly. Test items with poor performance were either modified and re-administered or removed on subsequent tests. Differential Item Functioning (DIF): This term refers to a situation where a disproportionate number of trainees from a focal group (e.g., based on race or gender) miss the test item in comparison to other trainees of similar overall ability. 2. Definition of Stereotype Threat: “Is being at risk of confirming, as self-characteristic, a negative stereotype about one’s group” (Steele & Aronson, 1995). This is a situational threat, meaning that it is not dependent on an internalized belief in the stereotype, just a fear that one will be judged by others through the lens of a given negative stereotype. 38

39 Where Are We Going? (cont’d…)Level 5: Skill assessment in classroom - continue Continue to analyze differential performance: Physical Abuse and Sexual Abuse Identification Pilot SDMTM version of the Critical Thinking in Child Welfare Assessment: Safety, Risk & Protective Capacity curriculum (assessment curriculum specific to CA Assessment tool) Revise and continue Casework Supervision module Pilot neglect scenario as part of an embedded eval (SDSU site) 39

40 Where Are We Going? (cont’d…)Level 6: Transfer of Learning – Feasibility study Level 7: Outcomes – Feasibility study, and Continue building Chain of Evidence to link training to outcomes. Other training evaluation projects, outside of levels 1-7: Attitudes/Values Evaluation re CMI 1 (physical abuse): pilot by SDSU site. Attitudes/Values Evaluation re CMI 2 (Sexual Abuse): Collaboration with UNC School of Medicine. Trainer Evaluation Quality Assurance Level 6: Conduct a feasibility study of Transfer of Learning evaluations as applied at a statewide level, based on findings and lessons learned from initial TOL evaluations. Level 7: Continue building Chain of Evidence to link training to outcomes, AND -Conduct a feasibility study of linking training to outcomes evaluation as applied at a statewide level. (May link this to program evaluation efforts, or to research related to the Statewide Research Agenda for CWS.) Other training evaluation projects, outside of Levels 1-7: Attitudes/Values Evaluation re: CMI 1: The Public Child Welfare Training Academy (PCWTA) plans to pilot a neglect scenario as part of an embedded evaluation and study of the effects of demographics and other trainee background differences with respect to CMI 1 decisions (on whether or not maltreatment occurred in a given scenario). Attitudes/Values Evaluation re: CMI 2: In collaboration with the University of North Carolina (UNC) School of Medicine, CalSWEC and several of the regions hope to link the CMI 2 data to the Child Forensic Attitude Scale (CFAS), a measure developed by UNC that assesses social workers’ attitudes toward child sexual abuse disclosures. Trainer Evaluation: Identify trainer-related differences in test item difficulty. Develop & obtain feedback on model of trainer evaluation. Quality Assurance: Convene small group of representatives from around the state. The same small group observes one Phase 1 training (curriculum up for revision) and one Phase 2 training (LOs up for revision) in each region. Analyze feedback from observers and provide information to Content Development Oversight Group and Statewide Training & Education Committee as part of curriculum/LO revisions and simultaneous trainer development activities. 40

41 MENTAL HEALTH PROGRAM

42 MENTAL HEALTH WORKFORCE SHORTAGES2001: Approx. 4 million adults in California needed MH services1 : 600,000 clients received county MH services2 2008: Vacancy rate for MH providers in California = 20-25%; higher in rural areas.3

43 CALIFORNIA MENTAL HEALTH SOCIAL WORK WORKFORCE TRENDS2000 – 2010: # of social work graduate programs increased from 13 to 20 Council on Social Work accredited programs or accreditation candidates. 2007: 15,560 LCSWs represent 21.5% of all licensed MH professionals in California. This is the second largest group after Marriage and Family Therapists (37%), followed by psychologists (19%), psych techs (13%), psychiatrists (9%), and psychiatric MH nurses (.5%). 66% of LCSW’s reside in urban communities.

44 CalSWEC MENTAL HEALTH INITIATIVE1993 – Mental health directors, social work educators, and practitioners began to explore creating a program modeled on the Title IV-E program. CalSWEC Board members re-ignited efforts to: Develop a curriculum to train graduate students for careers in public mental health services Create partnerships between education and provider communities to recruit and deploy more social workers into the mental health system Search for resources to support stipends and program development.

45 MENTAL HEALTH SERVICES ACT (MHSA or PROP. 63) - 20041% tax on income >$1 million to transform the public mental health system 2005 revenues = $900 million Expands mental health services in five program areas: (1) Children’s systems of care; (2) Adult and Older Adult systems of care (3) Prevention and early intervention (4) Innovative programs (5) Education and Training

46 MHSA WORKFORCE, EDUCATION, AND TRAINING FUNDS (WET) GOALSAddress critical MH workforce shortages Retool the existing workforce to create and sustain system transformation Create/strengthen career pathways for consumers and family members.

47 MENTAL HEALTH PROGRAM CalSWEC receives MHSA WET funding of $5.8 million annually for stipends and program activities that is distributed to Schools of Social Work. Stipends ($18,500) for up to 196 final-year MSW students Related program and indirect costs at each school. Schools develop and implement curricula that teach a set of core competencies identified by CalSWEC stakeholders. Stipend students have a one-year payback obligation to work in a county or contract mental health agency or to pay the loan back in cash with interest.

48 MENTAL HEALTH PROGRAM CURRICULUM COMPETENCIESCulturally and Linguistically Competent Generalist and Advanced CLC MH Practice Social Work Practice / MH Practice Human Behavior and the Social Environment / MH Practice General Social Work / Mental Health Management, Policy, Planning and Administration

49 MHP EVALUATION UC Berkeley School of Social Work Outcome Study:Cohort characteristics Payback obligation performance Post-payback employment Methods: - Analysis of administrative data - Paper and interview surveys of project coordinators Loma Linda University, Dept. of Social Work & Social Ecology: Implementation of the curriculum competencies Preparation of MHESP graduates to work in recovery-oriented mental health systems Methods: - Self-report surveys of schools - Content analysis of syllabi - Survey of graduates (IRB approved) - Survey of employment supervisors

50 FINDINGS The program contributes to the diversity of the mental health workforce. 56% of the 2005 – 2010 cohorts are ethnically and culturally diverse; Latino is the largest population 57% of the cohorts speak at least one language in addition to English; Spanish is spoken most often. Graduates are meeting their payback obligations 92% of the graduates of the cohorts completed their employment obligations; 51% worked in county-operated mental health agencies, and 41% worked in contract agencies. Graduates are continuing their careers in public mental health. 69% of the cohorts were still at their payback agency in 2010. 54% were in county agencies and 46% were employed in contract agencies.

51 FINDINGS Continued… Curricula in schools of social work are changing to address mental health competencies A survey of the MHP schools of social work identified a number of strategies being used to implement the mental health competencies with a particular focus on content about recovery, and inclusion of consumers and family members on advisory boards and as participants in classroom presentations. Schools are creating pathways for greater collaboration among faculty and with agencies Schools have developed specialized seminar courses for MH stipend students, increased involvement of MH Stipend Project Coordinators in their school/department curriculum committee meetings and increased interaction with local county mental health agencies. Analysis of data from the graduate surveys and supervisor interviews is in process.

52 EVALUATION AUDIENCES or Who cares/ should care?Key stakeholders Funders – current and potential Policymakers Elected Appointed Advocates Researchers Taxpayers

53 NEXT STEPS Build consensus about central role of program evaluationStrengthen capacity for more sophisticated and rigorous evaluation strategies within the CalSWEC framework. Broaden dissemination of our story and evaluation results. Find resources to support expanded evaluation efforts.

54 Where we are headed: Integrated Evaluation Framework

55 Main Assumptions and Points of EvaluationThe curriculum, delivered as intended, will prepare social work students for work in public human services. Graduates are better prepared for public human services practice than others. The number of MSWs is increased in the state’s public human services staff population. Graduates are staying in public human services agencies after their work obligation is completed. Graduates who stay in public human services have been able to affect practice, program, & policy through experience and promotion. Graduates are involved in the educational process through becoming field instructors, sitting on admissions panels etc. Excluding clerical, there were case-carrying, non case-carrying, supervisors, managers, & administrators as of 2008.

56 Theoretical Frameworks & LiteratureIn School 6 months to 1.5 year post graduation (Core Training) 3 years 5 to 6 years 8 years to 10 + years Reactivity in research design Effects of feedback Adult learning theories Education Models of the schools. CalSWEC mission & program objectives Control theory (Buckley, Fedor, Veres, Wiese, & Carraher, 1998) Occupational sociology (Landsman, 2001) Role theory Expectancy theory Locus of Control Theory (P. Spector) Social Exchange Theory Perceived Organizational Commitment (Rhoades & Eisenberger, 2002) Organizational Social Context (Glisson) Integrated Turnover Model (Mowday, Porter, & Steers, 1982) Intrinsic/Extrinsic Job Factor Theory (Benton, in progress) Locus of Control Theory (P. Spector) Professionalism Bureaucracy Organizational citizenship behavior Goodness of fit Social Exchange Theory Psychological Theories: Locus of Control (Spector and Michaels (1986) Learning Theory (Miller, 1996) Organizational Turnover Theory (Hom et al., 1992). Sociological Theories: Social Comparison Theory (Geurts et al., 1998) Social Exchange Theory Social Ecological Theory (Moos, 1979). Economic theoretical: Human Capital, Dual Labor Market Models, (Miller, 1996).

57 Research Questions—The ModelStudy Interval In School 6 months to 1.5 years (Core Training) 3 years 5 to 6 years 8 years to 10 + years RESEARCH QUESTIONS Stem: Are the competencies infused into the MSW curriculum? Q2. Does use of the student competency assessment raise degree of infusion of competencies in school curriculum? Q3. To what extent do the schools models reflect adherence to CalSWEC’s program requirements? Is graduate still working in public agency? Stem: To what extent are the graduates prepared for work in public agency? Q2. Are they able to practice what they learned? (core) Q3. What are the graduates’ expectations for the job? Q4. What are the graduates’ career goals? A. Do CalSWEC trainees’ knowledge levels skill acquisition differ from non-CalSWEC trainees? (core) Is graduate still working in public agency? Stem: What factors contribute to retention of public sector workers? Q2. Are they able to practice social work skills, knowledge & values? Q3. Are job expectations being met? Q4. Does agency/organization support graduates career path/goals? A. Do CalSWEC graduates stay longer than non CalSWEC? Why or why not? Is graduate still working in public agency? Stem: What are the career paths of public sector workers? Q2. Are they able to practice social work skills, knowledge & values? Q3. Are job expectations being met? Q4. Does agency/organization support graduates career path/goals? A. Do CalSWEC graduates stay longer than non CalSWEC? Why or why not? Is graduate still working in public agency? Stem: To what extent do public sector workers perceive an influence on policy, organization and/or program development/evaluation? Q2. To what extent have graduates career goals been realized in public sector employment? Q3. What are the graduate’s future goals? A. Do CalSWEC graduates stay longer than non CalSWEC? Why or why not?

58 Organizational VariablesIn School 6 months to 1.5 year (Core Training) 3 years 5 to 6 years 8 years to 10 + years Demographics: size, age, model CalSWEC instituted curricula (workshops, field class) vs. pre-existing curricula (core classes, field placement) Extent to which the students are exposed to the curriculum competencies Attitudes and preparation for training and professional development Exchange Relationship Job stressors & job satisfiers Org responsiveness Role conflict Available resources Organizational culture & climate Supervisor support/ quality supervision Professional development Salary Exchange Relationship Promotion/Professional development Organizational culture and climate Organizational politics Available resources Resources Mentorship Organizational responsiveness Supervisor/Management support Exchange Relationship Organizational culture and climate Organizational politics Work schedules Mentorship Organizational responsiveness Supervisor/Management support Professional development

59 6 months to 1.5 year post graduationIndividual Variables In School 6 months to 1.5 year post graduation (Core Training ) 3 years 5 years 8 years to 10 + years Demographic: previous experience in CW, race, gender, etc Knowledge level /acquisition Skill mastery Demographic characteristics Job expectations Commitment to a service profession Career Goals Individual factors: Demographic characteristics Commitment to a service profession Career Goals Work Locus of Control Response to job factors: Job satisfaction Met expectations Attempts to change conditions/office Demographics Self-efficacy Work locus of control Autonomy Commitment to service profession Career Goals Survival time to leave Survival time to promotion Compensation Demographic Self-efficacy Work locus of control Autonomy

60 Integration: IVE Stipend Program and In-Service Program EvaluationAll newly hired public child welfare workers are required to complete Core Training within two years of hire. (Legislation) Demographic information is collected at the time newly hired child welfare workers take Core Training as part of In-service Evaluation. As of 2010, we have added questions about career goals, job expectations and social commitment to the Demographic Questionnaire which will allow us to compare IV-E with non IV-E child welfare workers and follow them over time. [1] Findings from CalSWEC Career Path Study of Title IV-E MSW graduates [1] Landsman, M.J. (2001). Commitment in public child welfare. Social Service Review, 75(3), [1] Adapted from Buckley, M.R., Fedor, D.B., Veres, J.G., & Wiese, D.S. (1998). Investigating newcomer expectations and job-related outcomes. Journal of Applied Psychology, 83(3),

61 Integration: IVE Stipend Program and In-Service Program EvaluationThe inservice evaluators are already able to compare IV-E with non IV-E on certain knowledge tests. Pre- Post-Evaluations are done with Child & Youth Development, Placement & Permanency, Family Engagement in Case Planning & Case Management. Embedded Evaluations are done for Child maltreatment Identification & Neglect and Sexual Abuse. [1] Findings from CalSWEC Career Path Study of Title IV-E MSW graduates [1] Landsman, M.J. (2001). Commitment in public child welfare. Social Service Review, 75(3), [1] Adapted from Buckley, M.R., Fedor, D.B., Veres, J.G., & Wiese, D.S. (1998). Investigating newcomer expectations and job-related outcomes. Journal of Applied Psychology, 83(3),

62 Integration: Mental Health Program and Child Welfare ProgramsOngoing development of evaluation capacity to use the framework. Database development Identifying other applicable theories Resource development Curriculum evaluation

63 Key points Leverage resources of multiple evaluation effortsUtilize common evaluation framework to tie evaluation efforts together Dissemination of findings Contractual obligation Stakeholders Policy

64 Title IV-E Stipend ProgramContact Information Title IV-E Stipend Program Chris Mathias, Director Sherrill Clark, Research Specialist Susan Jacquet, Research Specialist Elizabeth Gilman, Curriculum Specialist

65 In-Service Training ProgramContact Information In-Service Training Program Barrett Johnson, Director Leslie Zeitler, Training & Evaluation Specialist

66 Contact Information Mental Health Program Gwen Foster, Director Beverly Buckles, Primary Investigator, Dean, Social Work & Social Ecology, Loma Linda University James Midgley, Harry & Riva Specht Professor, UC Berkeley School of Social Welfare

67 Title IV-E Stipend Program ReferencesReference for CalSWEC Goal Reference for CalSWEC Research Questions Grossman, B., Laughlin, S., & Specht., H. (1992). Building the commitment of social work education to publicly supported social services: The California model. In K.H. Briar, V.H. Hansen, & N. Harris (Eds.), New Partnerships: Proceedings from the National Public Child Welfare Symposium, Florida International University, North Miami Institute on Children and Families at Risk. For more CalSWEC publications based on CalSWEC studies see:

68 In-Service Training ReferencesMultiple reports and other information on CA’s In-Service Training Evaluation efforts are on the CalSWEC website, including: Evaluation of the California Common Core for Child Welfare Training: Implementation Status, Results and Future Directions (December 2009), Original Framework for Child Welfare Evaluation report (2004) 12 Years of Proceedings from the National Human Services Training Evaluation Symposia Many other concept papers and reports CalSWEC also posts child welfare Common Core curriculum online, and can share associated training evaluation materials (validated test items, demographic forms, code books, etc.) 68

69 CalSWEC Mental Health Program ReferencesAll references are from Lok, V. and Chapman, S. Lund, L. (2005). Mental Health Care in California Counties: Perceived Need and Barriers to Access, Center for Health Statistics Jew-Lochman (2008). Health Data Summaries for California Counties, Department of Health Services California Department of Mental Health. (2008). Mental Health Services Act Five-Year Workforce Education and Training Development Plan. Retrieved April 23, 2008 from NASW Center for Workforce Studies. (2006). NASW National Study of Licensed Social Workers. Washington, DC: National Association of Social Workers. Lok, V., and Chapman, S. (2009). The Mental Health Workforce in California: Trends in Employment, Education, and Diversity. Center for the Health Professions