1 Evidence-Based Medicine Author: dr. Martin RusnákImproving Health Care with Evidence-Based Medicine Author: dr. Martin Rusnák
2 Guideline Development MethodologyInternational perspective What is being done and why? Steps in guideline methodology Any evidence for what we do? What are the challenges ? What could each of us do ? Add in AGREE stuff, what do guideline users think? Look up SIGn and NICE methodology
3 Medical Mistakes National Institute of Medicine found that medical mistakes kill somewhere between 44,000 and 98,000 people (average: 71,000) in hospitals in the U.S. each year on average, one out of every 500 people admitted to a hospital in the U.S. is killed by mistake the chance of being killed in a commercial airline accident is one per 8 million flights
4 Healthcare Quality is the degree to which health services for individuals and populations increase the likelihood of desired health outcomes and are consistent with current professional knowledge Lohr KN, Harris-Wehling J. Medicare: a strategy for quality assurance. Quarterly Review Bulletin 1991;17,(1):6-9.
5 Improving Quality of HCCreativity and motivation among healthcare workers of all kinds; Leadership is an essential ingredient of success: senior managers feel personally responsible for each error; The problem is not fundamentally due to lack of knowledge; we already know far more than we put into practice. Based on Lucian Leape and Donald Berwick: Safe health care: are we up to it? We have to be. Editorials BMJ 2000;320: ( 18 March )
6 History of Guideline DevelopmentEarly guidelines Consensus methods Literature reviews not always systematic Not many systematic reviews Prior to 1990 consensus methods were commonly used, Mcglyn noted that improvements in the methods for synthesising the evidence was needed and then Eddy introduced the explicit evidence based approach. Not sure which country was first to develop guidelines but certainly the Dutch Coolege of GPs and the US AHCPR were among the first
7 MY EXPERT OPINION
8
9 History of Guideline DevelopmentFirst evidence based guidelines Searching for all the evidence Systematic reviews Recommendations linked to evidence Explicit evidence based guidelines Benefits, harms and costs are presented Prior to 1990 consensus methods were commonly used, Mcglyn noted that improvements in the methods for synthesising the evidence was needed and then Eddy introduced the explicit evidence based approach. Not sure which country was first to develop guidelines but certainly the Dutch Coolege of GPs and the US AHCPR were among the first
10 Randomized controlled trial
11 Where does EBM come from?Evidence-based medicine developed from a practical application of clinical epidemiology to a philosophy of rational clinical decision making. Epidemiology is concerned with the quantitative research of distribution, determinants & risk factors of health & disease in populations (population groups) & the application of this evidence to control (prevent & treat) disease.
12 Where does it all come from?Modern epidemiology found out i.e. that: There are huge geographic variations in the treatment of patients with the same health problems. There is great uncertainty about the effects of medical therapies (only 10-20% of all medical interventions are scientifically sound, e.g. proved by controlled studies); In the 90’s: a group around David Sackett & Gordon Guyatt, McMaster University, Canada developed a concept, which does not only qualify scientists to understand and interpret research literature.
13 What is Evidence-Based Medicine?Conscientious, explicit and rational use of current best evidence in making decisions about the care of individual patients.
14 Evidence Based Medicine – PracticeThe practice of evidence-based medicine means integrating individual clinical expertise and patient values with the best available external clinical evidence from systematic research. Sackett DL et al. Evidence-based medicine: What it is and what it isn‘t, BMJ 312 (1996) 71-72
15 Work-experience Time of work-experience External evidenceInternal evidence – Intuition Time of work-experience Doctors, Nurses, other
16 EBM is the conscientious, explicit and rational use of current best evidence in making decisions about the care of individual patients The patient Evaluation of performance Critical appraisal und clinical applicability Search for best external evidence Problems, questions Individual experience & external evidence
17 Spread of the idea of evidence Evidence-belt Eminence-belt
18 Steps in Guideline DevelopmentTopic Identification Suitability screen Form a multidisciplinary working party Formulate clinical questions Identify evidence (internal and external) Evaluate evidence Develop balance sheet Develop recommendations Implementation and dissemination Update (evaluate and improve) And this is what we do. Our methodology has evolved over the years from a variety of sources – group health puget sound and SIGN group in particular
19 1. Topic Identification What are the areas where there is a gap between the evidence and current practice ? Health Status Patient/Provider Satisfaction Cost/Utilization The topic is complex and there is debate Implementation is feasible Is there an evidence base for doing this?
20 2. Suitability Screens Does the project have a driver/owner?Is there evidence of a gap? Can we measure the proposed change? Is there a suitable guideline that could be identified? Is there adequate literature to make an evidence based decision about practice? How much effort would it take to close the gap? Is there a reasonable likelihood that we could implement the change? Add an example
21 3. Multidisciplinary teamClinicians Primary and secondary care Allied health care workers Consumers, patient representatives Epidemiologists Information experts Health economists Health managers Cultural
22 4. Developing clinical questionsWell developed questions form the basis of the evidence-based guideline structure Focuses guideline team on important issues & the most relevant evidence Requires a structured approach PICO PICO: Patients, Intervention, Control, Outcome Easy to get side-tracked
23 P.I.C.O. Model for Clinical QuestionsPatient, Population, or Problem How would I describe a group of patients similar to mine? I Intervention, Prognostic Factor, or Exposure Which main intervention, prognostic factor, or exposure am I considering? C Comparison or Intervention (if appropriate) What is the main alternative to compare with the intervention? O Outcome you would like to measure or achieve What can I hope to accomplish, measure, improve, or affect? What type of question are you asking? Diagnosis, Etiology/Harm, Therapy, Prognosis, Prevention Type of study you want to find What would be the best study design/methodology?
24 When to use the P.I.C.O. model Patients, Intervention, Control, OutcomeBackground questions concern general knowledge. A question root (who, what, when, where, how, why), and A disorder, test, treatment, or other aspect of health care. Often these questions can best be answered by using a textbook or consulting a clinical database. Foreground questions are specific knowledge questions that affect clinical decisions, including a broad range of biologic, psychological, and sociologic issues. These are the questions that generally require a search of the primary medical literature and that are best suited to the PICO format.
25 Well verbalized question leads to well formulated topicDoes mammography-screening I In women over 50 years P Compared with no screening C change mortality O Q: By e.g. 10% ?
26 Framing a PICO questionPatients Intervention (eg. Cause, factor, Rx, disease) Control Outcome How would I describe a group of patients like mine? Which main exposure am I considering? What is the main alternative to compare with the exposure? What can I hope to accomplish? What does this exposure affect?
27 5. Identifying the evidenceComprehensive searching Avoid applying limitations to reduce publication bias Non English studies included Unpublished data sought Use PICO framework to drive searching Also includes internal data
28 Clinical Search FiltersA method for improving the retrieval of high quality studies applicable to clinical practice is to include search terms that select studies at advanced stages of testing for clinical application. Including one, or a combination of these terms, in a MEDLINE search strategy will selectively retrieve evidence-based literature that is more likely to answer clinical questions.
29 PubMed with Clinical Queries three types of filtersClinical Study Categories filter searches on questions of therapy, diagnosis, etiology, prognosis, or clinical prediction guides by looking for the highest levels of evidence in the literature. These filters can also be limited to a broad or narrow scope. Clinical filters are imposed behind the scenes by the search engine based on the work of B. Haynes, et al, on the best strategies for retrieving clinically relevant information from MEDLINE.
30 Clinical Queries FiltersSystematic Reviews filter searches for citations identified as systematic reviews, meta-analyses, reviews of clinical trials, evidence-based medicine, consensus development conferences, guidelines, and citations to articles from journals specializing in review studies of value to clinicians.
31 Clinical Queries FiltersMedical Genetics filter searches for citations on seven different topics in medical genetics (diagnosis, differential diagnosis, clinical description, management, genetic counseling, molecular genetics, or genetic testing) or a combination of all topics in medical genetics.
32 6. Evaluate the evidence Critical appraisal Develop evidence tablesStudy quality checklists Develop evidence tables Quantification Summarise outcomes STUDY QUALITY scoring systems – LOOK UP – more than 25 of them and generally agreed at least for systematic reviews that they are not of much value
33 Evidence Table An Example
34 Scales for Quality Assessment?35+ scales of composite scores published Generally agreed that they are not useful in differentiating high and low quality studies Better approach is to analyse the individual components of study quality Blinding Concealment of allocation Intention to treat analysis
35 7. Balance sheets Benefits, harms (and costs) consideredfor the current situation and if the guideline was implemented Not a full economic analysis Simple analysis of projected costs if apply guideline Delivers the ‘value’ of the Guideline Resource utilisation The ‘final chapter’ of an explicit evidence-based guideline Use some examples
36 8. Developing the Recommendations and AlgorithmProbably the part of the guideline most often read Considers applicability: for whom will the intervention do more harm than good ? to whom should the recommended intervention be offered ? Each recommendation should advise a course of action, followed by an indication of the strength of the recommendation
37 Considered judgment formFor each clinical question: Volume of evidence Consistency of evidence Applicability of evidence Clinical impact of evidence Evidence Summary with levels/scores Recommendation with grade
38 [email protected] 3. 12. 2017 Considered judgementConsidered judgement Key question: A: Quality of evidence 1. How reliable are the studies in the body of evidence? (see SIGN 50, section 5.3.1, 5.3.4) If there is insufficient evidence to answer the key question go to section 9. Comment here on any issues concerning the quantity of evidence available on this topic and its methodological quality. Please include citations and evidence levels. Evidence level 2. Are the studies consistent in their conclusions? (see SIGN 50, section 5.3.2) Comment here on the degree of consistency demonstrated by the evidence. Where there are conflicting results, indicate how the group formed a judgement as to the overall direction of the evidence. 3. Are the studies relevant to our target population? (see SIGN 50, section 5.3.3) For example, do the studies: include similar target populations, interventions, comparators or outcomes to the key question under consideration? report on any comorbidities relevant to the target population? use indirect (surrogate) outcomes use indirect rather than direct comparison of outcomes 4. Are there concerns about publication bias? (see SIGN 50, section 5.3.5) Comment here on concerns about all studies coming from the same research group, funded by industry etc
39 [email protected] 3. 12. 2017 B: Evidence to recommendations5. Balancing benefits and harms (see SIGN 50, section 6.2.2, 6.2.3) Comment here on the potential clinical impact of the intervention/action – eg magnitude of effect; balance of risk and benefit. What benefit will the proposed intervention/action have? Describe the benefits. Highlight specific outcomes if appropriate. What harm might the proposed intervention/action do? 6. Impact on patients (see SIGN 50, section 6.2.4, 6.2.5) Is the intervention/action acceptable to patients and carers compared to comparison? Consider benefits vs harms, quality of life, other patient preferences (refer to patient issues search if appropriate). Are there any common comorbidities that could have an impact on the efficacy of the intervention? 7. Feasibility (see SIGN 50, section 6.2.6) Is the intervention/action implementable in the Scottish context? Consider existing SMC advice, cost effectiveness, financial, human and other resource implications.
40 8. Recommendation (see SIGN 50, section 6.3)What recommendation(s) does the guideline development group agree are appropriate based on this evidence? ‘Strong’ recommendations should be made where there is confidence that, for the vast majority of people, the intervention/action will do more good than harm (or more harm than good). The recommendation should be clearly directive and include ‘should/ should not’ in the wording. ‘Conditional’ recommendations, should be made where the intervention/action will do more good than harm, for most patients, but may include caveats eg on the quality or size of the evidence base, or patient preferences. Conditional recommendations should include ‘should be considered’ in the wording. strong/conditional Briefly justify the strength of the recommendation 9. Recommendations for research List any aspects of the question that have not been answered and should therefore be highlighted as an area in need of further research.
41 9. Implementation & DisseminationDissemination/Implementation Increasingly electronic Target Clinicians Patients Policy makers
42 10. Update Update Evaluation first (quality indicators?)
43 Steps in guideline developmentSystematic Pragmatic Evidence Topic selection/SS M-D working party Clinical questions ? Literature search Critical appraisal Balance sheets Recommendations Implementation Update ?
44 Methodological Challenges in Guideline DevelopmentIncorporating patient preferences Dealing with uncertainty Grading levels of evidence/recommendations Adaptation of guidelines Updating of guidelines Incorporating quality indicators Electronic clinical decision support
45 Methodological Challenges in Guideline DevelopmentIncorporating patient preferences Dealing with uncertainty? Grading levels of evidence/recommendations Adaptation of guidelines Updating of guidelines Incorporating quality indicators Electronic clinical decision support
46 Patient preferences Research into barriers to successful guideline implementation Patients don’t want the recommended treatments Patients want treatments not recommended by guidelines 30% of patients didn’t want Research into barriers to successful guideline implementation: Patients don’t want the recommended treatments OR Patients want treatments not recommended by guidelines
47 How to get to patient preferences?Inclusion of consumer representatives on guideline development teams Formal surveys of patients opinions Focus groups to seek patient opinions on the clinical questions and implementation Decision analytic methods Qualitative approach
48 Example of need for more information on patient preferencesCaesarean section: obstetricians report that pregnant women are asking for elective caesarean sections Survey data of women who had just given birth: 50% of women felt that c-section was more convenient that normal birth 30% felt that c-section was safer than normal birth 15% were planning to ask for c-section in the future (Australian data)
49 Methodological Challenges in Guideline DevelopmentUnderstanding patient preferences Dealing with uncertainty Grading levels of evidence/recommendations Adaptation of guidelines Updating of guidelines Incorporating quality indicators Electronic clinical decision support
50 Dealing with uncertaintyIf you need to make a recommendation when there is no evidence: Consensus approach Wide consultation Be explicit about what we don’t know Advise need for more research Caution
51 Methodological Challenges in Guideline DevelopmentUnderstanding patient preferences Dealing with uncertainty Grading levels of evidence/recommendations Adaptation of guidelines Updating of guidelines Incorporating quality indicators Electronic clinical decision support
52 How to go from individual study components to a grade or level?Consistent and systematic approach needed Checklists, scores, individual study components?
53 Grading systems SIGN NICE ICSI AHRQ USPTF NHMRC NZGGI wont go into the detail of all the grading systems used. However all seem to be based on a systematic scoring of trial design, qulality ( including risk of bias, confounding, etc.) Other presentations this week have indicated some of the difficulties here NICE National Institute of Clinical Excellence Oxford (Centre for Evidence Based Medicine) SIGN (Scottish Intercollegiate Guidelines Network) NHMRC (National Health Medical Research Council - Australia) ICSI (Institute for Clinical Systems Improvement) US Preventive Services Task Force NICE ICSI AHRQ USPTF NHMRC NZGG
54 Study design Best treatment? Diagnosis Aetiology/risk factorsClinical Question Best treatment? Diagnosis Aetiology/risk factors Prediction and prognosis Optimal Study Design RCT or SR Comparative study Cohort or case control Cohort or survival study
55 The GRADE Project International group working on grading levels of evidence: 4 levels of quality High Moderate Low Very low Lead by Andy Oxman and Gordon Guyatt
56 Grading systems for recommendations2 tier approach: Grade the individual studies with the study design (level of evidence - RCT+, SR or COH+, 1++, 1+, 2++, 2+ etc) Grade the recommendations based on the body of evidence (A, B, C etc)
57 NZGG Grading System Recommendation supported by GOOD (strong) evidence Recommendation supported by FAIR evidence Recommendation supported by EXPERT OPINION only No recommendation can be made – evidence is INSUFFICIENT A B C I
58 Taking evidence and turning it into recommendationsLeast systematic part of the process One persons evidence is another persons rubbish Group process requires skilled facilitation
59 Methodological Challenges in Guideline DevelopmentUnderstanding patient preferences Dealing with uncertainty Grading levels of evidence/recommendations Adaptation of guidelines Updating of guidelines Incorporating quality indicators Electronic clinical decision support
60 Potential Steps in Adapting Guidelines1. Find guidelines NICE/SIGN websites GIN 2. Appraise guideline - using the AGREE instrument The steps: 1. Find the guidelines – suggest an organisation such as NZHTA or a medical school library should be asked to conduct a search and identify these. 2. Appraise the guidelines (using AGREE instrument) for quality and process 3. Analyse the content for scope and applicability · Same health settings, professional groups? · Same patients, consumers? · Same interventions? · Same outcomes? 4. Get permission to use the relevant parts (evidence tables and search strategies) from guidelines teams and writers. NZGG is in contact with most overseas guideline organisations so can be asked to facilitate this. 5. Look at the gaps in the issues covered in the overseas guidelines · Any clinical questions not covered? 6. Look at the sources of evidence · Is the search strategy available? · Are there any evidence tables? · Are the evidence statements and recommendations referenced? A number of other guideline organisations are prepared to make these available. 7. Look at the quality of recommendations o Are the recommendations valid and is the grading correct? o Pick a few controversial topics to review how these were dealt with 8. Re-run the search strategy to include the questions selected · To include literature at least one year prior to the date of publishing · Check if any large study would radically change the recommendations 9. Rewrite and regrade recommendations if necessary · What are the cultural and consumer issues particular to New Zealand? · Are the practice points or I grades appropriate? 10. Implementation planning Redesign the implementation plan to meet local circumstances
61 Steps in guideline adaptation3. Analyse the content for scope and applicability - same health settings, professional groups - same patients, consumers - same interventions - same outcomes 3. Analyse the content for scope and applicability · Same health settings, professional groups? · Same patients, consumers? · Same interventions? · Same outcomes?
62 Steps in adapting guidelinesPermission to use relevant parts Look for the gaps - any clinical questions not covered Look at sources of evidence - is the search strategy available? - any evidence tables? - links from evidence statements to references 4. Get permission to use the relevant parts (evidence tables and search strategies) from guidelines teams and writers. NZGG is in contact with most overseas guideline organisations so can be asked to facilitate this. 5. Look at the gaps in the issues covered in the overseas guidelines · Any clinical questions not covered? 6. Look at the sources of evidence · Is the search strategy available? · Are there any evidence tables? · Are the evidence statements and recommendations referenced? A number of other guideline organisations are prepared to make these available.
63 Steps in guideline adaptationRecommendations - are the recommendations valid and is the grading correct? - look at the controversial topics in particular 8. Re-run the search strategy - to include literature at least one year prior to the date of publishing - check if any large study would radically change the recommendations 7. Look at the quality of recommendations o Are the recommendations valid and is the grading correct? o Pick a few controversial topics to review how these were dealt with 8. Re-run the search strategy to include the questions selected · To include literature at least one year prior to the date of publishing · Check if any large study would radically change the recommendations 9. Rewrite and regrade recommendations if necessary · What are the cultural and consumer issues particular to New Zealand? · Are the practice points or I grades appropriate? 10. Implementation planning Redesign the implementation plan to meet local circumstances
64 Steps in guideline adaptation9. Rewrite and regrade recommendations if necessary - what are the cultural and consumer issues particular to local environment? ` - are the practice points or grades appropriate? 10. Implementation - for the local circumstances 9. Rewrite and regrade recommendations if necessary · What are the cultural and consumer issues particular to New Zealand? · Are the practice points or I grades appropriate? 10. Implementation planning Redesign the implementation plan to meet local circumstances
65 Methodological Challenges in Guideline DevelopmentUnderstanding patient preferences Dealing with uncertainty ? Grading levels of evidence/recommendations Adaptation of guidelines Updating of guidelines Incorporating quality indicators Electronic clinical decision support
66 Updating Guidelines Practitioners want up to date informationAHRQ: research suggested that “as a general rule guidelines should be reassessed for validity every 3 years” Shekelle, P. G., et al. (2001). "Validity of the Agency for Healthcare Research and Quality clinical practice guidelines: how quickly do guidelines become outdated?" JAMA 286(12): Shekelle, P., et al. (2001). "When should clinical guidelines be updated?" BMJ 323(7305):
67 Methodological Challenges in Guideline DevelopmentUnderstanding patient preferences Dealing with uncertainty Grading levels of evidence/recommendations Adaptation of guidelines Updating of guidelines Incorporating quality indicators Electronic clinical decision support
68 Quality indicators Quality indicators are used for audit and evaluation Can they be used between countries? US and UK indicators covering 18 conditions – 56% agreement Marshall, M. N., et al. (2003). "Can health care quality indicators be transferred between countries?" Qual Saf Health Care 12(1): 8-12.
69 Methodological Challenges in Guideline DevelopmentUnderstanding patient preferences Dealing with uncertainty ? Grading levels of evidence/recommendations Adaptation of guidelines Updating of guidelines Incorporating quality indicators Electronic clinical decision support
70 Electronic decision supportRequires clear actionable statements from guideline developers How to integrate into the electronic medical record? How to deal with complexity and comorbidities of patients?
71 AGREE II Introduction [email protected]
72 Recapitulation Need for quality in Health Care ServicesEvidence Based Medicine Evidence and Guidelines Guidelines Development Process
73 Cochrane Library http://www.cochranelibrary.com/The Cochrane Database of Systematic Reviews (CDSR) is the leading resource for systematic reviews in health care. The CDSR includes Cochrane Reviews (the systematic reviews) and protocols for Cochrane Reviews as well as editorials. The CDSR also has occasional supplements. The CDSR is updated regularly as Cochrane Reviews are published ‘when ready’ and form monthly issues; see publication schedule.
74 Cochrane Podcasts http://www. cochranelibraryCochrane evidence in under five minutes, deliver the latest Cochrane evidence in an easy to access audio format, allowing you to stay up to date on newly published reviews wherever you are.
75 MEDLINE/PUBMED http://www.ncbi.nlm.nih.gov/pubmedPubMed comprises over 25 million citations for biomedical literature from MEDLINE, life science journals, and online books. PubMed citations and abstracts include the fields of biomedicine and health, covering portions of the life sciences, behavioral sciences, chemical sciences, and bioengineering. PubMed also provides access to additional relevant web sites and links to the other NCBI molecular biology resources.
76 The National Institute for Health and Care Excellence (NICE) https://www.nice.org.uk/provides national guidance and advice to improve health and social care. Our guidance, advice, quality standards and information services for health, public health and social care. Also contains resources to help maximize use of evidence and guidance. Essential information for key groups including GPs, local government, public health professionals and members of the public.
77 G-I-N Guidelines International Network http://www.g-i-n.net/Our mission is to lead, strengthen and support collaboration in guideline development, adaptation and implementation. As a major player on the global healthcare quality stage, G-I-N facilitates networking, promotes excellence and helps our members create high quality clinical practice guidelines that foster safe and effective patient care.
78 Google Scholar https://scholar.google.com/
79 MediSys http://medisys.newsbrief.eu/medisys/homeedition/en/home.htmlMedISys is an internet monitoring and analysis system providing event based surveillance to rapidly identify potential threats to the public health using information from the internet. It displays only those articles with interest to Public Health, grouped by disease or disease type. It analyses the news and warns users with automatically generated alerts. The information processed by Medisys is derived from the Europe Media Monitor (EMM), developed by the JRC.
80 AGREE: Appraisal of Guidelines for Research and Evaluation http://wwwAGREE II Overview Tutorial This tool provides an Avatar-guided overview of the AGREE II tool. This tool takes approximately 10 minutes to complete. AGREE II Tutorial + Practice Exercise Expanding upon the Avatar-guided tutorial, the “Practice Exercise” tool provides trainees with the opportunity to appraise a test practice guideline with the AGREE II