1 Greater Manchester Primary Care Patient Safety Translational Research Centre INVOLVE 2014 Conference: Changing landscapes 27th November 2014 Doing PPI at scale: What is the impact of public involvement in the NIHR Greater Manchester Primary Care Patient Safety Translational Research Centre? Interim findings of a formative evaluation Jonathan Boote, Reader in Patient Experience and Public Involvement, CRIPACC, University of Hertfordshire Sally Giles, Research Fellow, Greater Manchester Primary Care Patient Safety Translational Research Centre Carolyn Gamble, RUG member, Greater Manchester Primary Care Patient Safety Translational Research Centre Hello everyone. My name’s Jonathan Boote, and I’m presenting today with two colleagues Sally Giles and Carolyn Gamble. We are here to talk about an evaluation of the impact of public involvement that is currently place at an organisation in Manchester, which is known as the NIHR Greater Manchester Primary Care Patient Safety Translational Research Centre. The NIHR Greater Manchester Primary Care Patient Safety Translational Research Centre is funded by the National Institute for Health Research (NIHR) and is a partnership between the University of Manchester and Salford Royal NHS Foundation Trust

2 Outline of presentationTo describe the NIHR Greater Manchester PSTRC To set out PPI in the work of the NIHR Greater Manchester PSTRC To describe the PPI evaluation method To discuss some challenges of doing the evaluation To present some preliminary findings To present a Research User Group (RUG) perspective. So this is an outline of what we’re going to cover in the next 15 minutes. Sally is going to start off by describing the work of the Greater Manchester PSTRC, and to set out how the public is involved in the work of the Centre. Then I’m going to outline the method that we are using to evaluate the impact of PPI in the Centre, to discuss some of the challenges in doing the evaluation, and to present some preliminary findings. Finally, Carolyn is going to present her perspective of being involved in the Centre and in the evaluation as a member of the public, through being on the Centre’s Research User Group. So over to Sally… NIHR Greater Manchester Primary Care Patient Safety Translational Research Centre

3 Setting for the evaluationThe NIHR Greater Manchester Primary Care Patient Safety Translational Research Centre is funded by The National Institute for Health Research (NIHR) and is a partnership between Salford Royal NHS Foundation Trust and the University of Manchester SG NIHR Greater Manchester Primary Care Patient Safety Translational Research Centre

4 Aims of Greater Manchester PSTRCGreater Manchester PSTRC aims to improve patient safety in primary care through high quality translational research through the achievement of three objectives: 1.  Develop evidence-based innovative approaches to keep patients safe in their interactions with primary care; 2.  Develop capacity in primary care patient safety research; 3.  Develop and test interventions aimed at both patients and practitioners to improve patient safety. SG NIHR Greater Manchester Primary Care Patient Safety Translational Research Centre

5 Research Themes Core Theme (Prof Stephen Campbell)Governance/Finance, EMB, SAG, PPI-PE Medication safety (Prof Darren Ashcroft / Prof Tony Avery) Multimorbidity (Prof Peter Bower) General practice (Prof Stephen Campbell) Interface / informatics (Prof Iain Buchan) SG NIHR Greater Manchester Primary Care Patient Safety Translational Research Centre

6 8% of our overall budget is allocated to PPIPPI Structure 8% of our overall budget is allocated to PPI SG NIHR Greater Manchester Primary Care Patient Safety Translational Research Centre

7 Where can I find out more?If you are interested in finding out more about the Centre, do visit its website. The website includes videos from Prof Stephen Campbell, the Centre’s principal investigator, as well as from the Centre’s research fellows, which set out the aims and objectives of each theme.

8 Evaluating the impact of PPI in the Greater Manchester PSTRCEvaluation of ‘partnership working’, not PPI The Centre wishes to investigate: The impact of the partnership working on research processes, outcomes and stakeholders RUG members’ perceived quality of their involvement Evaluation team made of internal and external investigators: Dr Sally Giles and Dr Jill Stocks (Greater Manchester PSTRC) Dr Jonathan Boote (UoH) and Dr Jill Thompson (UoS) Moving on to the evaluation itself, it’s important to state that the evaluation is not evaluating PPI in isolation; it is very much an evaluation of the partnership working between the RUG and the Centre’s researchers The Centre wishes to evaluate two key issues. Firstly, the impact of the partnership working on research process and outcomes in the Centre, and on key stakeholders Secondly, RUG members’ perceived quality of their involvement. The evaluation is made up both internal and external investigators – there is Sally Giles and Jill Stocks from the Centre’s core research theme, and then there is myself and Dr Jill Thompson from the University of Sheffield as the external investigators. NIHR Greater Manchester Primary Care Patient Safety Translational Research Centre

9 Evaluation method The evaluation employs a formative, participatory, longitudinal, mixed methods design Main methods of data collection include: Yearly stakeholder interviews Non-participant observation of RUG meetings Documentary analysis of key documents A regular structured questionnaire to RUG members Informed consent from all stakeholders Evaluation advisory group (PEAG) B1 – The evaluation employs what we call a formative, participatory, longitudinal mixed methods design and I’ll explain what I mean by that now. The evaluation is formative as we will be feeding back quarterly on our findings, so that the Centre can act on the recommendations of the evaluation on a regular basis. The evaluation is participatory because stakeholders are involved as advisory group members, and by having the opportunity in yearly reflection cycles to suggest key issues that the evaluation team may wish to address. The evaluation is longitudinal as we will be collecting data throughout the lifetime of the Centre, and the evaluation is mixed methods because we are collecting different types of qualitative and quantitative data. B2 – we have been contracted to deliver 25 stakeholder interviews per year B2 – RUG meetings are held every 6 weeks and the meetings are observed and audio-recorded by the evaluation team B2 – documents include a 6 weekly partnership working impact capture form completed separately by the RUG members affiliated to the theme and the theme’s researchers B2 – a structured questionnaire is sent out every 6 weeks after the RUG meeting asking RUG members to reflect on the perceived quality of their involvement NIHR Greater Manchester Primary Care Patient Safety Translational Research Centre

10 Challenges in undertaking the evaluationSelecting an appropriate theoretical framework Developing a questionnaire to capture perceived quality of involvement Evaluation has to be done in an atmosphere of honesty and trust – ‘uncomfortable findings’ Just to highlight a few challenges in trying to robustly evaluate the impact of PPI in a complex patient safety research organisation. Firstly, its important that any evaluation is guided by a theoretical framework. In our case, we are using normalisation process theory or NPT, which looks at the dynamics of implementing, embedding, and integrating some new technology or way of working in the workplace. Secondly, we are having to develop our own questionnaire to evaluate RUG members’ perceived quality of their involvement, because no valid and reliable measure has been developed previously. Our questionnaire is based on a paper by Elizabeth Morrow and colleagues, who proposed that quality public involvement in research can be measured across a number of key domains . These domains comprise: feeling valued, achieving goals, feeling empowered, participation in research, and impact of previous experience. But perhaps the biggest challenge to doing the type of evaluation that I have described is that it has to be conducted within an atmosphere of honesty and trust, because such evaluations may uncover things that are going wrong, and which may be uncomfortable for certain stakeholders. Stakeholders need to be honest not only with each other but also with the evaluation team, as we require honest and frank reflections of impact for us to do our work properly. We need to know when things aren’t working, as well as when they are, so that we can feed this back to the Centre, for improvements to be made. Stakeholders need also to trust one another because feedback on impact can potentially be identifiable in themes with a relatively small number of stakeholders. NIHR Greater Manchester Primary Care Patient Safety Translational Research Centre

11 Contributions made by RUG membersThe RUG has helped the Greater Manchester PSTRC by: Helping the Centre think through how PPI can make a difference in each theme Contributing actual research ideas Contributing to ethics applications/amendments Helping to design publicity and patient information material for studies Helping to run focus group discussions with patients Undertaking secondary research Planning PPI in individual primary research studies Just to highlight how the RUG is making a difference so far to the work of the Centre. It’s important to make the point that many of the studies in the Centre are just getting underway, so most of these impacts relate to the earlier stages of the research cycle. So RUG members have helped theme researchers think through how patients and the public can be involved in individual studies within each theme. They have contributed specific research ideas. For example, RUG members in the information and informatics theme have suggested a study to look at capturing data from patients via wearable technology and sensors, allowing patients to determine what data are collected about them and what happens once that data are collected. RUG members have also helped to draft submissions to research ethics committees, they have commented on patient information material and drafts of interview schedules. And they have helped to run focus group discussions of patients The Centre has also involved the RUG in undertaking a small number of systematic reviews. One was a review off published qualitative research about continuity of care experiences of patients with multiple long-term conditions. RUG members received training about how to perform qualitative data extraction and performed a series of data extractions during the review process NIHR Greater Manchester Primary Care Patient Safety Translational Research Centre

12 Partnership working in one theme: multi-morbidityLongitudinal qualitative study of patient safety issues for people with MM RUG members have commented on the Participant Information Sheet, tested the data collection tool and guidance notes Just to highlight the impact of the partnership working in one particular theme, which is multi-morbidity. This theme is beginning a longitudinal qualitative study of patient safety issues in a small cohort of people with MM. Participants will be asked to keep diaries on a tablet device. Here RUG members have fed back on the wording of the proposed participant information sheet, and the draft guidance notes for using the tablet device. They have suggested changes to post-consultation in-depth interviews that will be conducted with participants in this study. The RUG members were also named as authors on the recently published protocol paper for this study, as highlighted on this slide NIHR Greater Manchester Primary Care Patient Safety Translational Research Centre

13 Concerns raised by stakeholdersWhat can the RUG do as an entity and how can it be used as entity? I suppose I know what the members are doing [at theme level] but I’m not so sure what the RUG meetings are doing. I’m not entirely sure of the purpose of them (RUG member 3) I'm starting up a new project that I think would benefit from a lot of input from the RUG…it's not clear what you can and can't say to them… (Centre employee 7) I just want to highlight two key concerns that have so far been raised in our evaluate. Firstly, there was concern about what PPI panels can do as an individual entity aside from the contribution of individual members within individual projects. This concern was raised by both RUG members and centre employees [read text] , NIHR Greater Manchester Primary Care Patient Safety Translational Research Centre

14 Concerns raised by stakeholdersHow to get the ‘PPI structure’ right within a complex organisation I suppose it [PPI] always has been done at the project level…we have siloed these people into narrow little sections of the work, and they should have opportunities to do project-by-project work across the different themes (Centre employee 7) I wanted to be involved more with the research…As I say, I feel it’s probably more an associate type of role, but I’m perfectly happy doing that sort of thing, in some respects more than sitting on a committee (RUG member 12) The second key issue is how to get the PPI structure right in a complex organisation so that it will please the majority of key stakeholders. In the PTSRC, the allocation of RUG members to individual thematic areas was questioned by a number of stakeholders. Some RUG members didn’t particularly want to serve on the RYG itself – they just wanted to be involved in individual projects. NIHR Greater Manchester Primary Care Patient Safety Translational Research Centre

15 Some interim quantitative dataThis slide shows some interim data on how RUG members’ perceived quality of their involvement is changing over time. As you can hopefully see, this diagram shows how the RUG members’ perceptions have changed since baseline, when the first questionnaire was administered, and then through 5 successive administrations of the questionnaire. The red line is overall perception of involvement quality, and the other lines focus on the different dimensions of involvement quality… [Go through them, but also highlight that the dark blue line is impact of previous research experience. As you can see, overall perceptions of involvement quality have declined by a small amount over the first six iterations of the questionnaire, but it does show a fall RUG members feeling valued. It will be interesting to see whether this trend is reversed over the lifetime of the Centre. NIHR Greater Manchester Primary Care Patient Safety Translational Research Centre

16 Improving the Greater Manchester PSTRC’s partnership workingThe evaluation has suggested a number of areas for improvement, which are currently being addressed by the Greater Manchester PSTRC: Clearer expectations for how RUG members can contribute at strategic, theme and project levels Each theme and project to be encouraged to draft a PPI plan, with RUG input Consider how RUG members can work across themes if they wish Consider how feedback is given to RUG members about the value (or otherwise) of their input These issues are all being addressed in a revised PPI strategy Our interim findings about the partnership working have been fed back to the Centre, and our key recommendations were that The centre should have clearer expectations about how RUG members can contribute at strategic, theme and project levels. We have suggested that each theme in the centre and also each project draft a plan for how patients and the public can contribute We have suggested that RUG members could work across themes if they so wished, and we have suggested that the Centre considers how best to feed back to RUG members about the value of their advice and input to specific projects. These issues are currently being addressed by the Centre, which is currently developing a revised PPI strategy. I’m now going to hand over to Carolyn who will be giving her perspective as a member of the Centre’s Research User Group NIHR Greater Manchester Primary Care Patient Safety Translational Research Centre

17 Research User Group (RUG) perspectiveCG

18 Greater Manchester Primary Care Patient Safety Translational Research Centre A closer look at the RUG CG the Centre website section on public engagement, you will be able to open a link to see the profiles of all RUG members. I haven’t been able to show you them all in this screen shot, which just displays the RUG Chair, Ailsa Donnelly. If this link works, I’ll play you a short video Ailsa has recorded for the Centre, which describes the RUG and where she offers her thoughts on the value of public involvement in the Centre

19 Research User Group (RUG) PerspectiveActivities and impact – personal experiences Importance of combining professional skills and personal patient experience Reviewed patient forum websites Public engagement event – for example chose the title to make it more accessible to the general public – “What should never happen at your GP practice?” Involvement in a systematic review Co-presenting at conferences. CG NIHR Greater Manchester Primary Care Patient Safety Translational Research Centre

20 Research User Group (RUG) PerspectivePurpose of the PPI Evaluation Advisory Group (PEAG) Membership – researchers, RUG members, PPI experts, external PPI evaluation team Advise the evaluation of the Research User Group (RUG) Support assessment of impact Advise how best to encourage other stakeholders to take part in the evaluation Agree how best to communicate findings both internally and externally. CG NIHR Greater Manchester Primary Care Patient Safety Translational Research Centre

21 Research User Group (RUG) Perspective Case StudyPPI Evaluation Advisory Group (PEAG) Involvement of RUG members. Why? Our involvement? Why is it important? CG NIHR Greater Manchester Primary Care Patient Safety Translational Research Centre

22 Research User Group (RUG) PerspectiveThe future…….? Engaging the “easy to ignore” groups RUG working more as a group Engaging Associate RUG There’s work to be done…..input into the revised PPI strategy Scope for RUG involvement in the future. primary care CG NIHR Greater Manchester Primary Care Patient Safety Translational Research Centre

23 For more information or to get involved visit: www.gmpstrc.nihr.ac.ukHow to get involved? For more information or to get involved visit: CG

24 Acknowledgements Evaluation team: Drs Jill Stocks, Sally Giles, Jill Thompson, Jonathan Boote RUG: Ailsa Donnelly, Faith Mann, Brian Minor, Cara Afzal, Jo Beresford, Carolyn Gamble, Henri Rahman, Jackie Nightingale, Tony Barlow, Bella Starling PEAG: Prof Stephen Campbell, Jo Beresford, Carolyn Gamble, Dr Sian Taylor, Kristina Staley, Mandy Wearne, Dr Sudeh Cheraghi-sohi , Dr Sally Giles, Dr Jill Stocks Centre employees: Prof Stephen Campbell, Prof Peter Bower, Prof Darren Ashcroft, Prof Iain Buchan, Dr Denham Phipps, Dr Sally Giles, Dr Jill Stocks, Dr Gavin Daker-White, Dr Sudeh Cheraghi-sohi, Philip Hammond JB The National Institute for Health Research (NIHR) Greater Manchester Primary Care Patient Safety Translational Research Centre (PSTRC) is a partnership between the University of Manchester and Salford Royal NHS Foundation Trust This presentation summarises independent research funded by the NIHR Greater Manchester Primary Care PSTRC. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health.

25 Thank you Any questions? JB