1 Evaluating the Quality and Impact of Community Benefit ProgramsSeptember 27, 2016 Julie Trocchio Senior Director, Community Benefit and Continuing Care The Catholic Health Association Kimberly Peeren Senior Public Health Consultant Healthy Communities Institute, a Xerox Company

2 Learning Objectives Understand the mission and legal basis for evaluating the quality and impact of community benefit programs. Learn how the CDC’s Framework for Program Evaluation can be used to evaluate community health improvement activities. Become familiar with tools and resources for evaluating and reporting the impact of community health improvement programs to help meet new IRS requirements for tax-exempt hospitals.

3 Resource: CHA Evaluation Guide

4 Why Evaluate? To improve programs To assess the organization’s impactTo ensure resources are being used wisely To meet federal requirements

5 Implementation Strategy Community Health Needs AssessmentFederal Requirements Implementation Strategy Community Health Needs Assessment Include anticipated impact of activities taken in response to CHNA Report impact Source: https://www.federalregister.gov/articles/2014/12/31/ /additional-requirements-for-charitable-hospitals-community-health-needs-assessments-for-charitable

6 Community Benefit CycleCommunity Health Needs Assessment Implementation / Program Planning Implementation and Evaluation REPORT EVALUATION FINDINGS EVALUATE PROGRESS DESIGN EVALUATION IDENTIFY PRIORITIES

7 CDC Evaluation FrameworkPLAN Engage stakeholders Describe the program Focus evaluation design IMPLEMENT 4. Gather credible evidence ANALYZE AND USE FINDINGS 5. Justify conclusions 6. Ensure use and share lessons learned 1. Engage Stakeholders 2. Describe the Program 3. Focus Evaluation Design 4. Gather Credible Evidence 5. Justify Conclusions 6. Ensure Use and Share Lessons Learned Standards Utility Feasibility Propriety Accuracy Source:

8 Resource: CDC Guide Source:

9 CDC Evaluation FrameworkPLAN Engage stakeholders Describe the program Focus evaluation design IMPLEMENT 4. Gather credible evidence ANALYZE AND USE FINDINGS 5. Justify conclusions 6. Ensure use and share lessons learned 1. Engage Stakeholders 2. Describe the Program 3. Focus Evaluation Design 4. Gather Credible Evidence 5. Justify Conclusions 6. Ensure Use and Share Lessons Learned Standards Utility Feasibility Propriety Accuracy Source:

10 Step 1. Engage StakeholdersWho is involved in program operations? Who is served? Who will use findings?

11 Resource: HPOE Guide Source:

12 Who Will Use the Findings?Decision-makers Program planners Individuals reporting to regulators Senior staff Board members Funders

13 CDC Evaluation FrameworkPLAN Engage stakeholders Describe the program Focus evaluation design IMPLEMENT 4. Gather credible evidence ANALYZE AND USE FINDINGS 5. Justify conclusions 6. Ensure use and share lessons learned 1. Engage Stakeholders 2. Describe the Program 3. Focus Evaluation Design 4. Gather Credible Evidence 5. Justify Conclusions 6. Ensure Use and Share Lessons Learned Standards Utility Feasibility Propriety Accuracy Source:

14 Step 2. Describe the ProgramGoal Resources to be used Activities Anticipated outcomes

15 Activities + ParticipantsProgram Mapping Inputs Outputs Outcomes Resources What we invest in our programs Activities + Participants What we do and whom we reach through our programs Results What happens as a result of our programs

16 Backpacking gear & suppliesSimple Logic Model INPUTS OUTPUTS OUTCOMES Family & friends Budget Time Backpacking gear & supplies Prepare and pack Drive to destination Set up camp Learn about nature; develop backpacking skills; increase bonding among family and friends! Backpack & explore

17 Resource: UWEX Logic Models

18 CDC Evaluation FrameworkPLAN Engage stakeholders Describe the program Focus evaluation design IMPLEMENT 4. Gather credible evidence ANALYZE AND USE FINDINGS 5. Justify conclusions 6. Ensure use and share lessons learned 1. Engage Stakeholders 2. Describe the Program 3. Focus Evaluation Design 4. Gather Credible Evidence 5. Justify Conclusions 6. Ensure Use and Share Lessons Learned Standards Utility Feasibility Propriety Accuracy Source:

19 Step 3. Focus Evaluation DesignDetermine evaluation purpose Program expansion/replication Funding Objectives Identify what program aspects to evaluate Identify what questions to ask Identify indicators for evaluation questions Design the evaluation

20 Step 3. Focus Evaluation DesignDetermine evaluation purpose Identify what program aspects to evaluate Short, medium, long-term outcomes Cost effectiveness, etc. Identify what questions to ask Identify indicators for evaluation questions Design the evaluation

21 Step 3. Focus Evaluation DesignDetermine evaluation purpose Identify what program aspects to evaluate Identify what questions to ask Identify indicators for evaluation questions Design the evaluation

22 Process Questions How was the program carried out?Who did the program reach? How involved were participants in program activities? Were activities implemented as planned? How satisfied were participants? What else happened?

23 Resource: Evaluation Briefs

24 Impact Questions What changed?Did participants increase their knowledge or awareness? Did participants acquire new skills? Did participants change their behavior? Did health status improve? Did program achieve anticipated impact?

25 Step 3. Focus Evaluation DesignDetermine evaluation purpose Identify what program aspects to evaluate Identify what questions to ask Identify indicators for evaluation questions Design the evaluation

26 Evaluation Designs Post-test onlyPre-test and post-test (single group) Pre-test and post-test with two or more groups Pre-test and post-test with control group

27 Post-Test Only AFTER PROGRAMSimple design; many factors can influence results and no other results to compare with

28 Pre-Test Post-Test (one group)BEFORE AFTER PROGRAM Can compare pre/post results of one group; must consider factors that can influence participants and results (e.g., participant selection, external events)

29 NO PROGRAM OR DIFFERENT PROGRAMMore than One Group BEFORE AFTER PROGRAM BEFORE AFTER NO PROGRAM OR DIFFERENT PROGRAM Can compare different groups’ results; must consider factors that can influence participants, groups and results

30 NO PROGRAM OR DIFFERENT PROGRAMControl Group BEFORE Groups with similar characteristics AFTER PROGRAM AFTER NO PROGRAM OR DIFFERENT PROGRAM Can compare results of similar groups receiving different interventions; consider group comparability and other factors that can influence results

31 Resource: Community Toolbox

32 CDC Evaluation FrameworkPLAN Engage stakeholders Describe the program Focus evaluation design IMPLEMENT 4. Gather credible evidence ANALYZE AND USE FINDINGS 5. Justify conclusions 6. Ensure use and share lessons learned 1. Engage Stakeholders 2. Describe the Program 3. Focus Evaluation Design 4. Gather Credible Evidence 5. Justify Conclusions 6. Ensure Use and Share Lessons Learned Source:

33 Types of Data Quantitative data = numbers Qualitative data = narrativeCounts, ratings, scores E.g. # of participants, survey scores Qualitative data = narrative Descriptions E.g. key informant interviews, observation studies

34 Types of Data PRIMARY DATA SECONDARY DATAOriginal data collected for your own purposes Data collected by someone else Surveys, questionnaires Focus group discussions Key informant interviews Program records, existing databases Observation County Health Rankings State and local health departments CMS Small Area Health Insurance Estimates USDA Food Environment Atlas, etc.

35 Step 4. Gather Credible EvidenceHave a plan for: Questions to be answered Data sources for answers Method of collecting data Timing Who is responsible

36 Data Collection Plan Source:

37 Resource: WISEWOMAN Toolkit

38 CDC Evaluation FrameworkPLAN Engage stakeholders Describe the program Focus evaluation design IMPLEMENT 4. Gather credible evidence ANALYZE AND USE FINDINGS 5. Justify conclusions 6. Ensure use and share lessons learned 1. Engage Stakeholders 2. Describe the Program 3. Focus Evaluation Design 4. Gather Credible Evidence 5. Justify Conclusions 6. Ensure Use and Share Lessons Learned Standards Utility Feasibility Propriety Accuracy Source:

39 Step 5. Justify ConclusionsAnalyze and interpret results Examine patterns Consider factors that may have influenced results (e.g., external events)

40 Step 5. Justify ConclusionsProcess: Was the strategy implemented correctly? Did any external events intervene? If program was evidence-based, was it faithful to the model?

41 Step 5. Justify ConclusionsImpact: Did program participants change? Were the changes measurable? Were the changes likely because of the program?

42 Resources: Data AnalysisCDC’s WISEWOMAN Evaluation Toolkit CDC Evaluation Briefs: Analyzing Qualitative Data: /pdf/brief19.pdf Analyzing Quantitative Data: /pdf/brief20.pdf

43 Look At Your Evaluation FindingsWhat does information suggest about program? Was program carried out as planned? Any surprising information? What lessons were learned? Positive findings? Negative findings?

44 CDC Evaluation FrameworkPLAN Engage stakeholders Describe the program Focus evaluation design IMPLEMENT 4. Gather credible evidence ANALYZE AND USE FINDINGS 5. Justify conclusions 6. Ensure use and share lessons learned 1. Engage Stakeholders 2. Describe the Program 3. Focus Evaluation Design 4. Gather Credible Evidence 5. Justify Conclusions 6. Ensure Use and Share Lessons Learned Standards Utility Feasibility Propriety Accuracy Source:

45 Audiences to Share Findings WithProgram participants Program managers, staff Funders Hospital leaders Community members Policy makers

46 Implementation Strategy Community Health Needs AssessmentFederal Requirements Implementation Strategy Community Health Needs Assessment Include anticipated impact of activities taken in response to CHNA Report impact Source: https://www.federalregister.gov/articles/2014/12/31/ /additional-requirements-for-charitable-hospitals-community-health-needs-assessments-for-charitable

47 Step 6. Ensure Use Improve program Identify resources neededMake decisions about program Replicate/expand program Disseminate information to those who might be interested in adopting program

48 Resource: BetterEvaluationSource: BetterEvaluation - Report and Support Use of Findings (May 2013)

49 Resource: HPOE Guide Source:

50 Resource: CDC CHI Navigator

51 Resource: CHA Evaluation Guide

52 Thanks for participating in this webinar!Questions? Thanks for participating in this webinar!