1 Sharing information about choosing and using evaluation optionsPatricia Rogers – RMIT University Canberra Evaluation Forum 20 June 2013 Canberra

2 It is hard to know how to implement them wellChallenges It is hard to choose the right combination of evaluation methods and proceses There is increasing recognition of the importance of evaluation. But when NGOs or government agencies start to plan an evaluation, or to engage an evaluator, it is difficult to find accessible, comprehensive and credible information about how to choose the right combination of evaluation methods, and how to implement them well – or how to manage these issues with a potential evaluator. Many guides to evaluation focus on only a few types of data collection methods and a few types of research designs. Few guides cover newer types of data collection (such as GIS logging on mobile phones, or tracking social media mentions), designs and strategies that can be used when experimental or quasi-experimental research designs are not feasible or appropriate, and methods to address issues such as clarifying and negotiating the values that should underpin the evaluation. There is information about some of these available on various web sites, but they can be difficult to locate, of varying quality, and not co-ordinated. And much of the information is partisan for particular methods, or does not include examples from practice. It is hard to know how to implement them well

3 Lots of material on evaluation methods (some not maintained)Opportunities Lots of material on evaluation methods (some not maintained) At the same time there are opportunities. There is good material on many evaluation methods available online, if you know where to look. Some of it has been produced by projects that have ended and is at risk of disappearing. There are also many people willing to share their experience and expertise in using evaluation methods or designing or conducting evaluations in particular situations. A networked approach to knowledge generation and translation can meet the challenge and use the opportunities. An international collaboration has been seeking to do just that. And today I’m delighted to be able to introduce it to you. Many people willing to share their experience and expertise

4 COMMUNITY WEBSITE Documenting Sharing R & D Events DescriptionsComments The BetterEvaluation project provides a platform for sharing information about evaluation methods to improve evaluation practice and theory. It consists of a website that provides information about how to choose and implement the appropriate combination of evaluation methods and a community that generates information about evaluation methods by sharing existing knowledge, documenting tacit knowledge, providing feedback, and undertaking research and development on evaluation methods. This project is taking a very different approach from the usual build-test-refine-launch process of most websites. Instead, we recognise that most of the knowledge, expertise and ideas that are required to make this project a success reside outside of the core group within a distributed network of practitioners. So rather than build a website and ‘let them come’, we will offer a platform for and facilitate a process of co-creation which will result in a resource that will be owned, used and valued by a community of users. A prototype of the platform has been developed with funding from the Rockefeller Foundation, the International Fund for Agricultural Development, Pact, the Royal Melbourne Institute of Technology, and the Institutional Learning And Change initiative of the Consultative Group on International Agriculture. The next phase of the project will involve further development of the platform before it goes live, more development while it is in open beta-phase until a formal launch, and then ongoing development. This presentation shows the current stage of development of the platform and the overall project, and discusses the developments currently in train in preparation for going live. Examples Guides Tools

5 Financial Supporters Founding Partners

6 Contributors and collaborators include …Quality and authenticity: Content is subject to scrutiny by experts to maintain technical accuracy and methodological authenticity. Reviewed and/or authenticated content will be labelled clearly to give users confidence in the information provided.

7 Co-creation by practitioners, researchers, sector experts, method experts Co-creation of content: BetterEvaluation is a facilitated platform for knowledge exchange which relies on the contributions of its users. Content is not created in a vacuum but through the interaction of users – their challenges and experiences – and content . The project has developed a process for identifying and engaging individuals to act as stewards for a particular method, evaluation task, thematic area or issue , who foster content in a particular area of the site.

8  Reflection, processes for quality and authenticityQuality and authenticity: Content is subject to scrutiny by experts to maintain technical accuracy and methodological authenticity. Reviewed and/or authenticated content will be labelled clearly to give users confidence in the information provided.

9 Rooted in practical experience Quality and authenticity: Content is subject to scrutiny by experts to maintain technical accuracy and methodological authenticity. Reviewed and/or authenticated content will be labelled clearly to give users confidence in the information provided. Rooted in practitioner experience: Rather than just replicating text books and method guides, BetterEvaluation enriches this with advice from practitioners to help other choose methods and use methods. Rooted in practical experience

10 Advice from experienced practitioners

11  Task oriented taxonomy: Support to navigate optionsTask oriented taxonomy: When people talk about evaluation methods they more often than not refer to methods for data collection or for causal analysis. BetterEvaluation, on the other hand, had a much broader definition of methods that covers a broad range of evaluation tasks; from the planning stages right through to reporting and supporting use of findings. This task oriented taxonomy is a central feature of the site and acts the primary navigation. Photo: David Masters

12 Over 200 methods/options related to 35 tasks in 7 clustersFRAMEWORK Over 200 methods/options related to 35 tasks in 7 clusters

13

14

15 Methodological pluralism Methodological pluralism: BetterEvaluation doesn’t favour one method over another but rather seeks to support diverse perspectives. We are firmly of the opinion that no single method or approach will suffice in all situations but that a choice should be made from a wide range of methods based on the appropriateness to the given situation Simon Kneebone ‘Show Me The Change”

16 Spotlight on invisible evaluation tasks and less well known methods

17  Respectful, inclusive and constructive discussion

18 Scenario 1- describing a situationCommunity renewal project Have existing social indicator profile Want to learn how the residents see their community – what are its strengths and problems? Intended users of the evaluation – local authorities for planning and advocacy What methods might you use – or suggest in a ToR? Under what circumstances might you use them? How could you oversee the quality of their use?

19 Information from individualsDeliberative Opinion Polls Logs and Diaries Mobile Phone Logging Questionnaires: Peer/Expert Reviews Face to Face Questionnaires Photolanguage Postcards Global Assessment Scales Projective Techniques Questionnaires Goal Attainment Scales (GAS) Seasonal Calendars Sketch Mapping Internet Questionnaires Stories (Anecdote) Interviews Telephone Questionnaires Key Informant

20 Information from groupsAfter Action Review Focus Groups Future Search Conference Brainstorming Card Visualization: Hierarchical Card Sorting Concept Mapping Keypad technology Convergent Interviewing Mural Delphi Study ORID Dotmocracy SWOT Analysis Fishbowl Technique World Cafe

21 Scenario 1- PhotoVoice Participatory photography methodParticularly useful for giving a voice to marginalized people Community members : Take photos Select the most significant photos  Tell stories about what they mean  Identify key themes What questions would you ask?

22

23 Scenario 2(a)- causal inferencePilot project of service delivery to individual clients Intended users – department managers to inform decision about future funding What methods might you use or suggest in a ToR? Under what circumstances might you use them? How could you ensure quality?

24 Scenario 2 (b) - causal inferenceCompleted health intervention to reduce early childhood obesity Series of evening information sessions Ten participating families Data (from service records) show children’s weight was high at the beginning and lower afterwards Intended users – management of health service to inform decision about future funding What methods might you use – or suggest in a ToR? Under what circumstances might you use them? How would you ensure quality?

25 Source: Hick, P. (1988) An Evaluation of a Nutrition Program for Young Turkish Mothers. Unpublished paper for the Graduate Diploma in Adult and Continuing Education, Melbourne College of Advanced Education. Presented as a case study in Owen, J. (1999) Program Evaluation: Forms and Approaches (2nd edition). Sydney: Allen & Unwin

26 Weights of subsequent childrenSource: Hick, P. (1988) An Evaluation of a Nutrition Program for Young Turkish Mothers. Unpublished paper for the Graduate Diploma in Adult and Continuing Education, Melbourne College of Advanced Education. Presented as a case study in Owen, J. (1999) Program Evaluation: Forms and Approaches (2nd edition). Sydney: Allen & Unwin

27 Scenario 2 - Check timing is consistent with program theoryUse time series data to go beyond just ‘Before’ & ‘After’ See if the change trajectory is consistent with the theory that the program has caused the observed results

28 Scenario 2 - General Elimination MethodIdentify possible alternative explanations Gather and analyze data to see if these can be ruled out

29 Scenario 2 - methods for causal inferenceWhich methods were suggested by the group? Which methods were not suggested? What new methods sound potentially useful? What information would you need to be able to use (or oversee someone using) a new method?

30 Content Over 200 options pages Over 500 resource pages16 approaches (package of methods) 8 thematic pages – sectors or cross-cutting issues

31 Choose the right option Use it wellInformation on methods : Description Examples Advice for choosing the method Advice for using the method Short list of recommended resources A full list of all resources, ordered by user rating This information helps users determine if the method suits their needs and requirements

32

33

34 How could you benefit from BetterEvaluation?Advice for choosing appropriate methods Advice on applying methods effectively A framework for thinking about evaluation design Discover new methods Learn from other practitioners and experts Library of resources, case study, links and people

35 How could you contribute to BetterEvaluation?Suggest methods Comment on content Improve the description Suggest resources Rate resources Curate a section Host a writeshop Financially support BetterEval Spread the word SUGGEST A METHOD: Is there a method missing? Is it there but with a different label? good example that’s not yet on the site COMMENT: Got some advice on using a method? Add a comment here IMPROVE THE DESCRIPTION: Got some specific suggestions to improve the description? Or able to expand a stub? SUGGEST A RESOURCE: Know a good guide, or overview, or a specific tool for using a method, or a RATE A RESOURCE: Give us your feedback on the resources on the site. Rate them on a 5 star scale and add your comments. CURATE A SECTION: Are you interested in a particular method or task or thematic areas and could share new resources as they emerge? HOST AN EVENT: Interested in hosting a writeshop to document examples, or a design clinic to provide face to face support to design evaluations? RESOURCE BETTEREVALUATION: Able to contribute to BetterEvaluation’s core functions or development of content in a specific area? SPREAD THE WORD: Tweet, bookmark, and engage others.

36 Partnering with BetterEvaluationAn AusAID perspective

37 Context Pressures to demonstrate effectiveness of growing aid programIndependent Review of Aid Effectiveness recommendations Strengthen our evaluation approach (evidence) More rigorous reporting on aid program effectiveness Standards for whole of government aid initiatives All projects (>$3m) to undertake an evaluation Challenges lack of baselines/data, range of programs and partners, constraints of working in developing countries diverse staff training needs

38

39 Why partner with BetterEvaluation?Because it’s GREAT! Generous – free access Rigorous –promotes rigor and good quality resource materials Easy – to navigate and interactive options Accessible & Active – housed in Australia but often travelling! Timely – coincides with increasing demand for monitoring and evaluation tools by aid program and other managers

40 What do we get out of it? Helpful – practical advice & insightsEncouraging – opportunity for feedback / discuss ideas Leveraging Opportunities – access to new networks & opportunity for us to influence direction Promotion – opportunities to share our evaluation products and learning

41 What’s next? To GROW the PartnershipGo places – BE is expanding relationships with other organisations and agencies/greater reach Reach out Overseas – to our partners in Asia-Pacific, development of local case study materials in local languages & translations Work more collaboratively – stronger focus on developing Canberra and Post skills & increasing participation on the site

42 Questions/Comments?

43 Further Information Comprehensive Aid Policy Framework 2012 Annual Review of Aid Effectiveness 2013 OECD DAC Peer Review of Australia ODE Website: