1 National Institutes of HealthUse of E-Tools for Organizational Performance Assessments &Evidence-based Management: NIH GPRA & PART Deborah Duran, Ph.D. Systemic Assessments Branch Office of Portfolio Analysis & Strategic Initiatives
2 Overview of PresentationNIH Overview Government Performance Program Performance Monitoring System (PPMS) Information Data Collection Knowledge Management Reporting Portfolio Analysis for goal replacement
3 The Nation’s Leading Medical Research AgencyNIH Role World’s premier medical research center and steward of medical and behavioral research for the nation NIH Goal Acquire new knowledge to help prevent, detect, diagnose, and treat disease and disability
4 Immediate Office of the DirectorNIH Organizational Structure Office of the Director Staff Offices Office of Extramural Research Office of Intramural Research Office of Management/Chief Financial Officer Office of Science Policy Office of Communications and Public Liaison Office of Equal Opportunity and Diversity Management Office of Program Coordination Office of Legislative Policy and Analysis Office of Community Liaison Executive Office Office of the Ombudsman/Ctr. for Cooperative Resolution NIH Ethics Office Immediate Office of the Director Office of the Director Program Offices Office of Research on Women’s Health Office of AIDS Research Office of Behavioral and Social Sciences Research Office of Disease Prevention National Cancer Institute National Eye Institute National Heart Lung and Blood Institute National Human Genome Research Institute National Institute on Aging National Institute on Alcohol Abuse and Alcoholism National Institute of Allergy and Infectious Disease National Institute of Arthritis and Musculoskeletal and Skin Diseases National Institute of Biomedical Imaging and Bioengineering National Institute of Child Health and Human Development National Institute of Deafness and Other Communication Disorders National Institute of Dental and Craniofacial Research National Institute of Diabetes and Digestive and Kidney Diseases National Institute on Drug Abuse National Institute of Environmental Health Sciences National Institute of General Medical Sciences National Institute of Mental Health National Institute of Neurological Diseases and Stroke National Institute of Nursing Research National Library of Medicine John E. Fogarty Center for Advanced Study in the Health Sciences National Center for Complementary and Alternative Medicine National Center for Minority Health and Health Disparities National Center for Research Resources Clinical Center Center for Information Technology Center for Scientific Review
5 NIH Overview 27 Institutes and Centers (ICs) work individually and collectively Over ~$29 billion invested annually on medical research 80% of budget awarded through 50,000 competitive grants to support over 325,000 scientists and research personnel in the U.S. and abroad 10% of budget supports research led by ~6,000 intramural principal investigators
6 Research NIH HAS ONE PROGRAM: For NIH level performance reporting:GPRA PART
7 GPRA Performance Driver MapNIH mission: to uncover new knowledge that will lead to better health for everyone. GPRA Performance Driver Map DRIVERS FUNCTIONAL AREAS PERFORMANCE Scientific Research Outcomes Short/Long term Low/High risk Intramural/Extramural Generation of new scientific knowledge of new or improved prevention, diagnosis, and treatment of disease Knowledge Dissemination Translation Field input Data / Repositories Communication and Transfer of Results Shared knowledge for public benefit Capacity Building and Research Resources Training Technology Facilities Partnerships Highly trained basic and clinical scientists; leadership in technology and facilities Strategic Management of Human Capital Performance-based approach Workplace Management Adequate Workforce Availability of highly skilled and diverse workforce; enhanced work environment Program Oversight and Improvement Results-oriented Management Resource Management Stewardship Enhanced program management and compliance with relevant laws
8 Core Criteria for Planning and Reporting Performance GoalsRepresentative (Disease/Disorders/Management) Trans-NIH Approach Annual Budget & Performance Target-Based Approach Balanced Scientific Research Portfolio (Intramural/Extramural, Basic/Clinical, ICs) R&D Criteria (Quality, Relevance, Performance) 4 Ps Predictive, Preemptive, Personalized and Participatory Scientific Difficulty (Low-Medium-High) Time (Short-Term, Mid-Range, Long-Term)
9 Scientific Risk and Time MatrixHigh Medium Low Increasing Level of Difficulty 1-3 years years years Time
10 Annual Targets Goals span from 1 to 10 years; therefore, assess annual performance Prospective & Quantifiable Annual Scientific Targets All IC/Office annual activities or contributions are called Strategic Targets Reported Targets are called Performance Targets Monitor all Targets but only submit Performance Targets Representative Target Reporting SROs: 1 performance target per goal Non-SROs: 1-5 Performance targets per goal Adaptive measure – scientific justification
11 PART Scores
12 Information KnowledgeTransitions from Age of Information Technology / Data Management to Age of Knowledge / Collects & Connect Capture data Structure data Analyze information Design Application
13 Program Performance Monitoring System (PPMS)Centralizes performance monitoring & reporting Enhances coordination & communication among collaborators Reduces workforce burden Improves analysis of budget / performance information Two Components of PPMS: Systemic Assessments Branch (SAB) Website (VPS Interface) Useful links to relevant resources and performance-related information at NIH Provides single point of access to NIH’s first centralized budget-performance reporting system Calendar with relevant guidance Visual Performance Suite (VPS) Centralized online database for collecting, storing, analyzing and reporting trans-NIH performance and budget data Maintain information used to plan activities at NIH Central point of access
14 SAB Homepage – NIH View
15 SAB Homepage – GPRA Goal Performance Reporting
16 INFORMATION DATA COLLECTION
17 Budget Performance Integration (BPI) – Functional Area LevelInformation structured to match NIH business process Scorecard Configured to display budget and performance information User can drill down to the next level of granularity
18 Functional Level to Goal LevelGoal level displays all goals and funding associated with the functional area Displayed box illustrates navigation system designed to allow users to: Navigate to any part of the system Display summary of information for goal Upload documents or form discussion using blog tread
19 Goal Data Entry Goal data entry screen has been configured to capture data necessary to support reporting needs-tailored quantitative and qualitative data fields
20 Goal Level to Targets Level
21 Target Data Entry / Tailored FieldsTailored fields can be added to collect reportable and un-reportable information Data collected at target levels automatically aggregate to goal levels
22 Budget Reporting Budget is integrated with workflow of configuring Goals and Targets Comment box appears/required when threshold for % change is crossed alerts are sent to contacts when a goal is rejected or needs revision
23 Performance Page Annual performance is collected on the performance page An automated notification system alerts users when annual performance results are due or missed Alert contains hyperlink that links users directly to reporting page
24 PART Reporting Page
25 KNOWLEDGE MANAGEMENT
26 Analytic Tool Strategy Map with AttributesGoal linkage can be represented dramatically on a map Attributes are tailored for sorting Attributes provide snapshot of how the elements are performing
27 Decision-Makers’ Tools for Planning and ReportingVPS produce graphics of budget-performance data in system Graphical Analyses supports various types of charts & graphs and formats Summarize budget performance data Compare dimensions Filter information Drill into a dimension Trending Modeling Resource for organizational performance analysis and portfolio management
28 Summarized Data (Core Criteria: Trans-NIH Approach)X-Axis and Legend can be interchanged dynamically Chart labels can also be updated
29 Compare Dimensions (Core Criteria: Scientific Risk)Users can navigate from scorecard to graphics with a single mouse click Preconfigured graphic can be further analyzed using menus on left panel Control panel can be hidden to draw a full-screen chart
30 Filter Information Data can be filtered so only “top n” “bottom n” or “middle n” is displayed Graphics feature has the ability to query the database by specifying any number of properties of an element in a plan (For example, user can configure graphic to display all goals that has a specific word such as “cancer”).
31 Drill Into a Dimension (Core Criteria: Representative Performance Goals)Users can drill down to the lowest level of a dimension using a filter Chart can be display in multiple chart types, in 3D and with table
32 Charting Multiple MeasuresMultiple measures can be simultaneously charted on a stacked vertical bar chart
33 REPORTING
34 VPS Reports Analytic graphics can be saved as jpeg or gif files & imported into PowerPoint Customized reports for each module or reporting requirement 12 tailored budget performance reports are currently produced on VPS Reports can be generated with a button push printed, exported, and/or saved as MS Word documents MS Excel spreadsheets
35 Sample VPS Report – Performance Goal Narrative
36 VPS Summary Report – Budget/Performance
37 VPS GPRA Budget Report
38 VPS PART Report
39 PORTFOLIO ANALYSIS FOR GPRA GOAL REPLACEMENT
40 Portfolio Analysis Using Core CriteriaGoal End Date SRO goals completed Core Criteria for Portfolio Analysis of Performance Goals Risk Years Funding Science Low Medium High 1-3 4-6 7-10 Intramural Extra mural Both Basic Clin-ical 2004 1 2005 4 2 3 2006 2007 2008 2009 6 2010 2011 2012 2013 TOTAL 29 10 9 7 13 12 16
41 Portfolio Analysis Using Funding $Research, Condition, and Disease Categorization (RCDC) system can help identify High priority research areas appropriate for representation in a GPRA goal Potential goal lead
42 RCDC for GPRA $ reportingRCDC categorizes NIH projects into disease categories based on specific terminology (Finger print) QVR/RCDC can be used to retrieve RCDC categorized grants based on several criteria (Text search, by IC, FY etc)
43 GPRA goal 3.4a using RCDC/QVRBy 2013 identify and characterize at least 2 candidate genes influencing substance use, psychotic disorders using high risk family, twin, special populations Finger Print Concept Name Concept Rank Bipolar Disorder 95% Schizophrenia 85% Alcohol dependence 65% proband 65% Substance Use Disorder 65% endophenotype 60% Genome Scan 58% Family 58% Conduct Disorder 58% Major Depressive Disorder 58% Borderline pers Disorder 58% ADHD 58% Autistic Disorder 58% LD Mapping 58% Psychotic Disorders 50% Twin Multiple Birth 50% Alcoholism 50% Twin Studies 45% Stress Disorders 44% autism spectrum disorder 44% Pathological Gambling 41% Siblings 40% Personality Disorders 40% psychogenetics 40% Project list from intersection Mental Health Alcoholism Drug abuse Genetics Human Genome Once you have the Finger print, it can be used to obtain the type2 grants and their progress reports, for the next fiscal year The dollar amount associated with the projects can be obtained Using RCDC/QVR further analysis can be done (Program officers or PCC) ICs NIMH, NIDA, NIAAA Key word search
44 Project list GPRA3.4a 5 R01AA Alcoholism Susceptibility Genes in High Density Families 5 P50AA MARC: Genetic Epidemiology of Alcoholism & Comorbidity 5 R01MH Genetic Linkage Studies in Bipolar Disorder Families 5 R01DA Twin Study of ADHD, CD and Substance Abuse 5 R01MH Genetic Linkage and Association in Bipolar Disorder 3 R01MH S1 Genetics of Early-Onset Major Depression 1 R01MH Using Genetics to Dissect Schizhoprenia, Bipolar Disorder, and Depression 5 R01MH Genetics of Early Onset Depression 5 R01AA Molecular Epidemiology of Alcoholism 2- Big Sibships 5 R01MH Genetics of Bipolar Disorder in Latino Populations 5 R01MH Collaborative Genetic Study of Bipolar Disorder
45 Analytic Results It is not a question how much a person knows, but what use s/he can make of what s/he knows. Josiah Gilbert Holland 1866 Data Management Data Analysis Information Knowledge Intuition Systematic
46 Systemic Assessments Branch Office of Portfolio Analysis & Strategic Initiatives (OPASI) National Institutes of Health (OD) 9000 Rockville Pike Bldg 2, Rm 1E10 Bethesda, MD ph: fax: