1 Evaluating Human Resources for Health Systems Strengthening: Experiences from USAID’s Capacity Project Laura Gibney, Director of M&E and & Research, IntraHealth International, Inc. Linda Fogarty, Director of Results and Knowledge Management, Capacity Project (Jhpiego) Danny DeVries, Monitoring, Evaluation & Research Manager, Capacity Project (IntraHealth)

2 Why the Global Focus on HRH?Insufficient # of qualified health workers Maldistribution of health workers Inadequate support systems for HCWs  Low access to health services Poor quality health services Poor health outcomes

3 Health Impact of Provider ShortageThere is a direct relationship between the ratio of health workers to population and survival of women during childbirth and children in early infancy. As the number of health workers declines, survival declines proportionately. WHO 2006

4 Global Health Workforce Shortage57 countries with critical shortages WHO estimated at least 4, health workers needed to fill the gap (providers, managers) Worst shortage is in Sub-Saharan Africa 11% of the world’s population 24% of global disease burden 3% of the world’s health workers 1% of world health expenditure. WHO 2006 See recent national geographic (India article) for related info.

5 Demands are increasing with HIVDemand for care is increasing e.g., from in Kenya, the number of HIV+ patients in hospitals more than doubled Supply of health care providers is dwindling e.g., Botswana lost 17% of its health workforce to AIDS between 1999 and Will reach 40% by 2010. Sick workers are less productive In an HIV+ worker’s final year of work, likely to miss on average every other day of work. Capacity to supply health providers is decreasing Faculty/trainers are dying of AIDS, so there are fewer to train new providers Where HCW density is low Infant mortality rate is higher Child mortality rate is higher Maternal mortality rate is higher Immunization coverage is lower

6 The Capacity Project USAID-funded project led by IntraHealth International, with IMA World Health, Johns Hopkins, Liverpool Associates for Tropical Health, Management Sciences for Health, PATH and Training Resources Group. Technical leadership in global human resources for health (HRH) Worked in 25 countries, with a focus on: Improving workforce planning and leadership Developing better education and training programs Strengthening systems to support workforce performance Human resources, when pertaining to health care, can be defined as the different kinds of clinical and non-clinical staff responsible for public and individual health intervention

7 IntraHealth International’s HRH Capacity Building ProjectsHuman Resources Development Capacity Project (5 year project. End date: 12/09) Human Resources for Health and Quality Services (HRHQS) Project (5 year project. Start date: 10/09) The Southern Africa Human Capacity Development Coalition (Botswana, Lesotho, Malawi, Namibia, and Swaziland) HRH-related Associate Awards: Uganda, Tanzania, Namibia, Kenya, G-CAP (Guatemala, Panama, El Salvador, Costa Rica, Belize)

8 Challenges in Evaluating HRH System StrengtheningRelatively new area of assessment. No compendium of standardized indicators. Dearth of data needed as most countries do not have functional, populated HRIS systems. Increasing expectation that we demonstrate impact of HRH system strengthening on health service delivery & health outcomes

9 Multipaper Session Presentations1. Context and Evaluating Human Resources for Health Capacity Building (Danny DeVries) 2. Indicators for Measuring Human Resources for Health Capacity Building (Linda Fogarty)

10 Context and Evaluating Human Resources for Health Capacity BuildingDanny de Vries, Monitoring, Evaluation and Research Manager, IntraHealth International, Inc. Linda Fogarty, Director, Results and Knowledge Management, Jhpiego Erik Reavely, Monitoring and Evaluation Advisor, IntraHealth International, Inc. Elizabeth Bunch, Manager, Knowledge Sharing, IntraHealth International, Inc.

11 Questions How does the context of HRH capacity building influence the technical approach? What evaluation strategies have been suggested to incorporate context in evaluation of program success? What indicators are suggested to incorporate context, if needed?

12 Context in Capacity BuildingTraditional “boxed” system approach where context is seen as a boundary in a closed system (the “external”). Relational approach that emphasizes an “enabling environment.” Ecological theory of complex adaptive systems, seeing “context” as an internal dynamic landscape embedding the program. “Contextual factors such as political or economic stability can also play a dominant but poorly understood role in ensuring system capacity.” conceptual continuation of an “external/internal” boundary distinguishing between the inside and outside of the relevant sphere of influence a range of indicators to assess the “interconnectedness of capacity issues between the targeted level(s) and the enabling environment” (2005: 3). In this view of context, though limited to “enabling” factors, there is an openness to relevant conditions and relationships of exchange rather than boundaries and closed systems. the enabling environment is conceived of in terms of a range of potentially relevant conditions, substantive relationships, and access to resources that may cross-cut sectors and systems. Potential for developing outcome indicators that recognize broad relationships and unanticipated outcomes is left open for discovery, rather than limited to the performance of a perceived “interior.” This dynamic, system oriented approach is pivotal in their conceptualization of context, which they define as: “the political, social, and institutional landscape within which actors struggled to make their way”. The concept of context here is described as “the complex web of … pressures and relationships that produces, influences, sustains, destroys and makes use of it [capacity]” (101). context is described as a dynamic of situational and interpenetrating relationships.

13 HRH Action Framework, developed (WHO 2006; Dal Poz et al. 2006)Generally limited to numeric measures, precisely bypassing the contextually driven processual issues, such as evolving relationship outcomes or behavioral changes needed to sustainably drive the HRH agenda (for example see Hornby et al. 1997). HRH Action Framework, developed by the Global Health Workers Alliance collaboration with the World Health Organization and Capacity Project (WHO 2006; Dal Poz et al. 2006) HRH Action Framework, developed (WHO 2006; Dal Poz et al. 2006)

14 Health Workforce Outcome Indicators% health facilities with staffing levels that meet the requirements of the HRH plan Numbers of patients: professional staff numbers Bed utilization: number of staff Vacancy rate Turnover rate Retention rate Absenteeism Rural/urban distribution Staff satisfaction (based on attitude surveys) % of total budget spent on staff salaries and allowances. From: Hornby P, Forte P. Human Resource Indicators and Health Service Performance. Human Resource Development Journal. 1997;1(2):

15 Intermediate Action Field IndicatorsSuggested Indicators HR Management Systems HRM units in place and strategically located at national and local levels HR information system in place and used for HRH planning at all levels Performance management system in place. Policy HRH policies in place Appropriate scopes of practice defined for all cadres. Leadership Evidence of high-level advocacy to promote the implementation of the HRH plan Leadership development program established for managers at all levels Involvement of wide range of stakeholders in policy and decision-making processes (professional associations, researchers, unions, private sector, FBOs). Partnership Mechanisms in place for coordination (e.g., other key ministries, donors, service providers including private sector providers, professional associations, etc.) Agreements in place between MOH and other health providers Mechanisms in place to involve communities in health services. Education Pre-service education tied to health needs In-service training (e.g., distance and blended, continuing education) Capacity of training institutions Training of community health workers and non-formal care providers. Finance Salaries and allowances competitive in local labor market Salaries and allowances equitable between cadres of health workers National health accounts routinely collect data on HRH expenditure. HRH Action Framework, developed (WHO 2006; Dal Poz et al. 2006)

16 HRH Action Framework, developed (WHO 2006; Dal Poz et al. 2006)Context Indicators? Generally limited to numeric measures, precisely bypassing the contextually driven processual issues, such as evolving relationship outcomes or behavioral changes needed to sustainably drive the HRH agenda (for example see Hornby et al. 1997). HRH Action Framework, developed by the Global Health Workers Alliance collaboration with the World Health Organization and Capacity Project (WHO 2006; Dal Poz et al. 2006) HRH Action Framework, developed (WHO 2006; Dal Poz et al. 2006)

17 HRH Action Framework, developed (WHO 2006; Dal Poz et al. 2006)Context Indicators? Generally limited to numeric measures, precisely bypassing the contextually driven processual issues, such as evolving relationship outcomes or behavioral changes needed to sustainably drive the HRH agenda (for example see Hornby et al. 1997). HRH Action Framework, developed by the Global Health Workers Alliance collaboration with the World Health Organization and Capacity Project (WHO 2006; Dal Poz et al. 2006) HRH Action Framework, developed (WHO 2006; Dal Poz et al. 2006)

18 HRH Action Framework, developed (WHO 2006; Dal Poz et al. 2006)Context Indicators? Generally limited to numeric measures, precisely bypassing the contextually driven processual issues, such as evolving relationship outcomes or behavioral changes needed to sustainably drive the HRH agenda (for example see Hornby et al. 1997). HRH Action Framework, developed by the Global Health Workers Alliance collaboration with the World Health Organization and Capacity Project (WHO 2006; Dal Poz et al. 2006) HRH Action Framework, developed (WHO 2006; Dal Poz et al. 2006)

19 Study: Role of context in HRH M&ETaking advantage of technical staff experience at end of the project to inform next steps for M&E in HRH systems strengthening A semi-structured, open-ended interview to document project’s history and technical evolution and explore implications for M&E. 29 key informants: senior Capacity Project leadership and technical staff, USAID/OPRH stakeholders, and country directors. One hour interviews transcribed and analyzed using qualitative data software and inductive coding strategy.

20 Findings: Contexts of HRH Capacity BuildingHistorical-Cultural Situation Political & Leadership Context Individual growth of HRH Champions Health Systemic Context (Action Fields, physical capital and consumables) Multi-Sectoral Context Global Context Temporal Context

21 Historical & Cultural SituationUnderstanding what actually drives change within that context and what sorts of enabling environment exists or not exist.… It’s something new in the capacity building world. I think it’s really important because if we are able to have sufficient understanding of what we are trying to change, then we are likely to engage in those contexts better. Participation and Partnership of Key Stakeholders in order to Enlist and Incorporate Local Interests and Needs I think that one of our key approaches has been really to involve stake holders from the very beginning and throughout to really try to understand what the needs of our in-country partners, especially the Ministry of Health, what their needs were before we move forward.

22 Political and Leadership ContextWorking with Key Central Leaders to ensure buy-in, policy change, and sustained support for regional and system-wide holistic HRH interventions It matters where you start. The reason is that the center will always be in charge overall … If you start from down, if the center does not approve or does not buy in, then it will be unsustainable. And also the issue is that the center gives you a framework or a platform on which to build your system… you must understand the policies, you must form partnerships with key people that are interested in those things happening down there… Well it depends I think on how autonomously the districts or health centers hospitals work from the central level. In some countries each hospital is almost its own entity and they can make their own decision and others you go to the central first you introduce your ideas Even in the most decentralized place, if the central level of a highly decentralized country that manages policy doesn’t make the intervention a policy or not willing to accept as a policy, or as a practice, then you can work with the decentralization all the way along and they’re still not going to have that support that is necessary.

23 Individual Context HRH ChampionsValue of Coaching and Mentoring HRH champions in the field And so as I understand it and as I’ve experienced it, it’s very iterative, so you know, you find out what people need to know and what they think of themselves as leaders and then you work with that. Benefit of utilizing internal and in-country secondments as part of an overall strategy of direct involvement and support

24 Health Systemic ContextSystems level scope demands technical approach has had to adjust towards a comprehensive approach We realized with training is that you can do training really, really well, but if you don’t have these other things in the working environment for health workers to go back and to support them then you know, it’s just a waste of money pouring it into training. HRH Action Framework Fields But there’s a bigger framework back there that says if you’re going to change something, you’ve got to be looking at all these different action areas, kind of what we’ve called action fields

25 Multi-sectoral contextAdopting a Cross-Cutting & Multi-Sectored Approach in order to close the gaps, get ministries and other organizations working together on interrelated issues To deal with a problem like workplace violence and discrimination these are things that cross-cut sectors so we adopted a multi-sectoral approach to it. And you need to have intervention at the level of the health sector, but also at the policy level, ministries of labor and gender. You also need to have intervention at the individual service provider level and you need to have different kinds of interventions. So it’s best to take a multi-sectoral approach.

26 Global Context Health worker migration as a global issueUSAID donor global context - PEPFAR “bean counting” (number of trainings says nothing about quality of trainings) Project has international list of partners Global ethical problems health worker distribution issues, including brain-drain and post-colonial power dynamics.

27 Temporal Context Long-term, strategic framework demands “strategic patience” Complexity of stakeholder relationships The time it takes for structural change in an organizational or core level Resistance inherent to HR Delay in expression of that change in service outcomes. Novelty of CB itself: took time for stakeholder interest to develop.

28 Expanding measurement boundariesGlobal Political & Leadership Individual Generally limited to numeric measures, precisely bypassing the contextually driven processual issues, such as evolving relationship outcomes or behavioral changes needed to sustainably drive the HRH agenda (for example see Hornby et al. 1997). HRH Action Framework, developed by the Global Health Workers Alliance collaboration with the World Health Organization and Capacity Project (WHO 2006; Dal Poz et al. 2006) Temporal Other System Components

29 Cultural Contexts of HRH Capacity BuildingContextual influence Technical Response Evaluation/Indicator needs Historical-Cultural Situation Situational assessments Partnerships & Stakeholder groups Secondments In-country offices Historically informed baseline studies Participatory evaluation to negotiate meaning “success” Measure quality of partnerships. Measure cultural sensitivity of technical approach, including adaptation of systems and products to fit local culture Political Context Partnerships Strategic patience Adaptive management Evidence of ownership & commitment Changes in discourse (adoption of terms) Government actions taken such as policy changes Integration HRH in routine government operations Measure quality of partnerships Illustrate changes in relationships and coordination mechanisms Measures of transparency & corruption

30 Cultural Contexts of HRH Capacity BuildingContextual Technical Response Evaluation/Indicator needs Individual context of HRH champions and key leaders Coaching Mentoring Training Measures of coaching, mentoring, training quality Evidence of Ownership Champions take the lead Changes in discourse Professionalization Levels of advocacy Operate HRH management functions without technical assistance Increase in HRH system thinking Increase in data driven decision making Measures of innovation Health Systemic & Non-Health Systemic (multisector) Contexts Holistic, systems thinking Adaptive management Multiplier effects (TOT, ripple effects) Mixed-methods for data collection Measures of systemic resilience (crisis response) Functional redundancy, diversity, reversibility Level of internal learning through self-organization, innovation, etc. Levels of internal communication & fragmentation of information Buy-in in program by other funders Acknowledge multi-causality

31 Cultural Contexts of HRH Capacity BuildingContextual influence Technical Response Evaluation/Indicator needs Global Context Global partnerships Managing expectations Development of global HRH system indicators Ethical and moral indicators (e.g. fair regional distribution) Level of ethical concerns and standards within morally conscious framework Fairness in relationship to recruitment (transparency) Equitable distribution of the workforce Measure quality of partnerships Building capacity of local M&E staff to balance and align global/local priorities Global cost-savings because of local adoption of tools Level to which model is replicated elsewhere Frequency / extent of technical assistance given to other projects Temporal Context Benchmarking Adaptive Management Measurement of changes in system efficiency Development expectation of strategic patience for donor and clients Benchmark indicators that recognize key steps along the way Early warning indicators to know when intervention is going off track (e.g. loss of interest)

32 Conclusion: the enabling environmentIn response to the complexity of the HRH system, the Capacity Project technical approach has emphasized a flexible, adaptive management approach to explicitly incorporate “context” as strategy for long-term success. Sustainability of the HRH capacity building approach seems strongly linked to seeing context as an enabling environment, or even dynamic landscape for HR strengthening. A multiscalar HRH M&E system is recommended to include HRH Workforce Outcome Indicators, HAF Action Field indicators as approximate measures, and enabling environment indicators to monitor sustainability. Special attention should be given to indicators that measure levels of ownership, the quality of relationships, and ethics as missing elements.

33 Indicators for Measuring Human Resources for Health Capacity BuildingLinda Fogarty, Director, Results and Knowledge Management, Jhpiego Danny de Vries, Monitoring, Evaluation and Research Manager, IntraHealth International, Inc. Erik Reavely, Monitoring and Evaluation Advisor, IntraHealth International, Inc. Elizabeth Bunch, Manager, Knowledge Sharing, IntraHealth International, Inc.

34 The Capacity Project - AcknowledgementsUSAID, Office of Population and Reproductive Health USAID, Office of HIV/AIDS 7 Partners: IntraHealth International, IMA World Health, Jhpiego, Liverpool Associates in Tropical Health (LATH), Management Sciences for Health (MSH), Program for Appropriate Technology in Health (PATH), Training Resources Group, Inc. (TRG) Informants (Senior Technical Advisors, USAID colleagues)

35 Outline What did Capacity Project do? How were results tracked?How did technical advisors define success? How can we refine our indicators?

36 Examples of technical workPlan Southern Sudan: assessment, stakeholder group development, policy, costed plan HRH information systems in 10 countries Develop Rwanda: HIV, FP in PSE, IST Support G-CAP (Guatemala, Belize, Costa Rica, Nicaragua, El Salvador, Panama) performance support to decentralize HIV services

37 Outline What did Capacity Project do? How were results tracked?How did technical advisors define success? How can we refine our indicators?

38 Capacity Project Results FrameworkDevelop Support Plan

39 Monitoring Method Project PMP and targets established with USAID approval (12 core-, 40 sub-indicators) Country-specific PMP developed Year one Core indicators appropriate for country strategy Baseline “country profile” established Quarterly discussions with country “point-person” Annual indicator assessment and reporting

40 Most indicators track outputNumber of HRH systems strengthened Pre-service education systems, in-service training systems, human resources for health information systems, retention systems, supervision systems… HRH strategic plan, policies in place

41 Some indicators track outcomesImproved use of data systems Improved health worker retention Improved health care access (special studies only)

42 Capacity Project Indicator Areas: HRH System StrengtheningExamples Have the necessary information, tools, policies, “building blocks” to improve capacity HR information resources, job descriptions, approved workforce plan, workforce assessments Improved infrastructure clinical practice sites with updated providers, necessary equipment Improved systems in place HRIS, training, HRM, employee recognition, retention Improved individual skills HR management skills, business acumen Improved collaboration between groups stakeholders working together to address HCD issues, key global partnering mechanisms

43 Outline What did Capacity Project do? How were results tracked?How did technical advisors define success? How can we refine our indicators?

44 Findings: New HRH Outcome IndicatorsFrom To Realigning the workforce HRH plans in place HRM systems strengthened PSE strengthened Better HRM & Retention sys n/a (small scale evals) Needs-based deployment HRH plans implemented Reducing vacancy rates PSE tutor/student balance HRM sys and retention links Quality of services Links to service delivery Note:

45 Findings: New HRH Process and Output Indicators, “Critical Success Factors”From To na Key people are now in key places making key decisions Changes in system efficiency Systems, products are used, integrated into existing, users innovate Quality of relationships Program ownership Note:

46 Implications Need more sophisticated immediate and intermediate outcomes reflecting more sophisticated TA and country context Include wide range of process, or “critical success factors” indicators Link HRH system changes with service delivery outcomes Bring together HRH measurement experts for “next generation indicator development”