1 PDSA Pragmatic Science Returning to the Deming Framework of Quality Improvement Presented to: National Quality Center NY State Dept. of Health Ryan White Community May, 2017 Ted Speroff, PhD 1

2 Outline of PresentationFramework: Deming’s CQI & PDSA Cycle Deming’s CQI = Applied Scientific Method Journal Club Article (2004) – QI Time Series Design RCT Gold Standard & QI Studies Physics of QI & Socio-Ecological Model Deming Model Updated & Examples Conclusion: Implication for QI 2

3 W. Edwards Deming (1900-1993) Out of the Crisis, 1982 System of Profound KnowledgeContinuous Quality Improvement Methods & Tactics Deming Cycle (PDSA) Knowledge of Variation (Statistical Process Control) Common Cause Special Cause Psychology of Change The Learning Organization Theory of knowledge Systems Thinking Test against data, data-driven Transformation is everyone’s job - context Management Leadership Drive out fear and eliminate management by objectives Loyalty & trust Breakdown barriers Training on the job 3

4 The PDSA is a Cycle of LearningAct Plan Objective(s) Questions and predictions (why) Plan to carry out the cycle (who, what, where, when) What changes are to be made? Next cycle? Study Do Complete the analysis of the data Compare data to predictions Summarize what was learned Carry out the plan Document problems and unexpected observations Begin analysis of the data 4 4

5 Deming’s PDSA Quality Improvement is Pragmatic SciencePDSA (Improvement Science) Plan: identify a goal or purpose, formulate a theory, define metrics, develop action plan Do: design and then implement an action plan Study: outcomes are monitored to test the validity of the plan for signs of progress and success or problems and areas for improvement Act: integrate the learning generated to adjust the goal, change methods, or reformulate the theory Spread: repeat over and over as a never-ending cycle of continual improvement Scientific Method (Experiments) Formulate Background, Rational, Aims and Hypotheses Protocol of Methods, Procedures, Implementation Plan, Analysis Plan Conduct Analysis, Results Conclusion, Discussion Publish, Disseminate Hence, PDSA project is systematic application of the Scientific Method 5

6 Continual Inquiry Repeated Use of the Cycle to Guide Effective ChangeOverall Aim is Profound Knowledge: Learning how the System works so that Changes can Result in Improvement and Better Operations Policies A P S D DATA D S P A A P S D A P S D Hunches Theories Ideas 6 6

7 Journal Club Article If one approaches PDSA as a Pragmatic Science, then Scientific methodology should be applied to bring rigor to Quality Improvement projects to learn with confidence what works Study Designs for PDSA Quality Improvement Research by Speroff & O’Connor, 2004 Reference: T Speroff, GT O’Connor. Q Manage Health Care 2004; 13(1): 7

8 The Dawn of Improvement and Implementation ScienceAt the time of the article: Typical QI Project was a Before-After Study Need to promote Quasi-Experimental Time Series Designs for more effective PDSA 8

9 Architecture of the Before-After Study Design Two Points in Time – Only 2 data points Extraneous Factors Confound Attribution Chp 3: Before-and-after design is Really Weak Threats to internal validity of before-and-after designs produce competing interpretations 3.5.1 History threat (secular trends, competing factors) 3.5.2 Instrumentation/reporting threat 3.5.3 Regression-to-the-mean threat 3.5.4 Testing threat Placebo and Hawthorne threats Maturation threat Dropout threat Change occurs, but, results are ambiguous. You cannot say with confidence that intervention (change idea) was effective. False interpretation leads to tampering. 9

10 Deming’s CQI Component Robust Study Designs for PDSA QI Journal Club ArticleQuasi-Experimental Designs study how changes/variation in process relate to outcomes Choose the study design and time dependent measures that rule out spurious factors Data Driven. Data collection methods, data quality, and meaningful, sound measurement. Time Series and SPC focus on the effect of the interventions. Multiple points in time, longitudinal data during both Pre and Post-Intervention. Apply Criteria for causal inference to evaluate the functional relationship between process intervention and outcome. Observe time-series charts for: Temporality of the Association - Immediacy (concurrence in timing of the discontinuity & timing of intervention) Strength of the Association - Magnitude (gap & dose effect) Stability-Constancy of Effect (trend, change in slope) Consistency across Replications (generalization, external validity, multiple baseline design) Standard of Evidence Problem: Quasi-experimental designs are viewed weak compared to experimental randomized controlled trial designs. 10

11 Health Sciences Paradigm for Standard of Evidence Internal Validity: RCT is Gold Standard11

12 Gold Standard Architecture of the RCT Design: Controlled Environment12

13 RCT Conceptual Framework: Example Typical Cluster QI RCT Study Measures, Moderating Variables Hierarchical Regression Equation – Path Analysis 13 Reference: Kelly M Simpson, Kristie Porter, Eleanor S McConnell, Cathleen Colόn-Emeric, Kathryn A Daily, Alyson Stalzer, Ruth A Anderson. Tool for evaluating research implementation challenges: A sense-making protocol for addressing implementation challenges in complex research settings. Implementation Science 2013; 8(2).

14 Critique of RCT Approach Challenges of Complex vs Controlled Setting InterdependenciesMultiple components: evidence-based practices, fidelity to protocol, dose Competing Demands: staff unable to attend to study activities, hinder protocol of implementation Interdependencies: Staff turnover, dropout, absenteeism, schedule changes, interactions among staff Human Relations: alienating, interrupting, unpredictable, adaptations, self-organizing interplay, mentorship, interaction networks, sense making, communications, trust, responsiveness. Diversity: literacy, language, skill and training Culture: hierarchical, punitive, vs enabling management practices QI Interventions are not static, not fixed, not controlled. QI is dynamic. Adjust implementation procedures 14

15 The Basic Flaw: Modeling Psychology of Change as a Linear Model Unidirectional: Input Output15 X

16 Unlike RCT, Implementation/Improvement Studies occur in Social-Ecological Environment16

17 Deming’s Psychology of Change Component In a Social-Ecological System, contextual mechanisms are linked to outcomes 17

18 Work System Example Social-Ecological Model: Within an Organization The Systems Engineering Initiative for Patient Safety (SEIPS) Model: a sociotechnical approach Patient Outcomes Quality of Care Patient Safety Processes Organizational Outcomes Job Performance Teamwork Reference: P Carayon, A Schoofs Hundt, B-T Karsh, A P Gurses, C J Alvarado, M Smith, P Flatley Brennan. Work system design for patient safety: the SEIPS mode. Qual Saf Health Care 2006; 15(Sullp 1): i50-i58. 18

19 Underlying Physics In Socio-Ecological Systems the Defining Feature is Dynamic Loops and Interplay Bi-Directional ( ) Mechanisms (A) Linear Models: cause-effect is unidirectional (i.e. RCT, context is covariate). (B&C) Dynamic Models: feedback loops in complex systems produce adaptive changes. 19

20 Paradigm Shift for Level of EvidenceImplementation Quality Improvement Knowledge Controlled Study Ecological Study System Dynamic Models RCT Factorial Time-Series Cohort Quasi-Experimental Case-Control Cross-Sectional Before-After 20

21 Implication: Deming Approach for QI Design and Program EvaluationPDSA Continuous Quality Improvement Pragmatic Science (Technical) Theory-driven Plan Evidence-Based Medicine + Underlying Mechanisms of Change Measurement: data used to evaluate sources of variation & impact over time Rigor PDSA Design (JC Article) Robust Methods for Implementation of strategic maneuvers to produce dose effect in outcomes Quasi-Experimental Times Series or Statistical Process Control for program evaluation Cycles of Learning – design probes for knowledge Coaches (tools, techniques, skills, training) Socio-Ecological System (Adaptive) Systems Thinking Implementation embedded in real world Goodness of Fit – How does PDSA integrate with real-world Physics: Closed-Loop Feedback Dynamic real-world mechanisms that define functional relationships & key contributing factors Interplay of Culture & Context Knowledge of Organizational Systems: Leadership, Attitudes, Management, Teamwork, Responsibilities, Motivation Social Sciences, Psychology of Change, & Dynamic Systems Engineering Adaptive changes over time 21

22 Implementation (Adaptive)QI Example Using real time process measures to reduce catheter related bloodstream infections in the ICU PDSA (Technical) BSI best practices Process measures BSI guidelines Fidelity Outcome measures Standardized CDC definition of BSI Time-series SPC g-chart feedback Implementation (Adaptive) Locate leverage points in microsystem, unmet needs Resources & Logistics: Interplay of supplies, pharmacy, infection control, managers, nurses, providers Checklist (airlines-crew resource management; new habits & routine care) Organizational Management Circumstances & resistance: burden of competing commitments, priorities, routines, assumptions & beliefs Motivation, enthusiasm, pride, satisfaction, respect Competence, training, turnover Connectivity, interdisciplinary cooperation, teamwork QI generates disequilibrium in existing systems to modify practice behaviors and routines. Underlying mechanisms are beginning to emerge. 22 Reference: RJ Wall, EW Ely, TA Elasy, RS Dittus, J Foss, KS Wilkerson, T Speroff. Qual. Saf Health Care 2005; 14:

23 Replication: Michigan Keystone BSI ICU Project“popular accounts of the program have often been simplistic and partial, and have perpetuated the myth that the program’s achievements can be traced to a ‘simple checklist’ rather than a complex social intervention”. Adaptive work (context) challenges and interventions far outweigh the technical work (PDSA methods and logistics). References: M Dixon-Woods, CL Bosk, EL Aveling, CA Goeschel, PJ Pronovost. Explaining Michigan: Developing an ex Post Theory of a Quality Improvement Program. The Milbank Quarterly 2011; 89(2): CA Goeschel, PJ Pronovost. Harnessing the potential of health care collaboratives: Lessons from the Keystone ICU Project. In Henriksen K, Battles JB, Keyes MA, Grady ML, editors. Advances in Patient Safety: New Directions and Alternative Approaches (Vol. 2: Culture and Redesign). Rockville (MD): Agency for Healthcare Research and Quality (US); 2008 Aug. Advances in Patient Safety. 23

24 Conclusion Quality Improvement is not easy.Quality Improvement requires a blend of commitments, both technical (PDSA Design) and adaptive (socio-ecological system). PDSA Design and System Dynamic Engineering are connected and must be aligned. 24

25 Conclusion: Design of QI Projects Technical AdaptiveConnect PDSA Pragmatic Science & Methods Socio-Ecological System Dynamic Engineering Align 25