1 My application has been reviewed…so now what?Meredith Temple-O’Connor, Ph.D. Senior Scientific Advisor to the NIH Deputy Director for Extramural research Office of Extramural Research Charisee Lamar, Ph.D., M.P.H., R.R.T. Program Officer, Reproductive Neuroendocrinology and Fertility Preservation Programs Division of Extramural Research Eunice Kennedy Shriver National Institute of Child Health and Human Development

2 Learning Objectives After this session, attendees should be able to:Explain what happens after peer review Identify who to speak with about their priority score and/or summary statement Assess the appropriate actions if their application is not selected for award Determine the actions necessary if the application is selected for award including resolving concerns from peer review and meeting just-in-time requirements

3 Specific Discussion PointsQuick primer on timeline and roles Understanding your score Interpreting your summary statement What should I do if I’m not selected for funding? What happens if I am likely to be selected for funding? Resolving concerns Just in time requirements and procedures Touch on Notice of Award Alternate text: The graphics at the top of this slide is a stick figure holding a magnifying glass. 3 3 3 3

4 NIH Extramural Administrative TeamProgram Officer Scientific Review Officer Grants Management Specialist Alternate text: This is a Venn diagram with three overlapping circles that describes the relationship between the Scientific Review officer, the Program Officer, and the Grants Management Specialist who comprise the NIH Extramural Administrative Team. Team is italicized to highlight the importance of that concept as it applies to how these individuals work together in the management and oversight of extramural grant applications and awards.

5 When to contact: Roles and Responsibilities:Program/Medical Officer (PO/ PD, MO) Roles and Responsibilities: Manages portfolio of grants Conduct site visits Participate in IC funding priorities discussion Attend scientific meetings; serve on professional society and IC committees Plan NIH-sponsored scientific meetings Communicate scientific advances to the Branch, Center, and Institute Directors When to contact: Prior to application submission and after receiving a summary statement through award close out

6 Roles and Responsibilities:Scientific Review Officer (SRO) SCIENTIFIC REVIEW OFFICER (SRO) Roles and Responsibilities: Recruits reviewers for study section meetings Assigns applications to reviewers; explains/interprets review criteria Runs study section meeting Calculates impact score and percentile Drafts narrative summaries, edits reviewer critiques (minimal edits), and releases summary statement When to contact: After application submission through release of summary statement

7 Roles and Responsibilities:Grants Management Specialist (GMS, GMO) Roles and Responsibilities: Acts as an agent of the GMO Assures compliance with Federal laws and NIH policies and procedures Analyzes grant applications prior to award Prepares award for GMO release Reviews and responds to grantee prior approval and re-budgeting requests Assures documentation of official grant files When to contact: After receiving a fundable score through award close out

8 Review System for GrantsScientific Review Group (SRG) Independent outside review managed by NIH Evaluate scientific merit, significance, other criteria Provides priority score Recommend length and level of funding 1st level Output: Impact Score and Summary Statement Advisory Council Assess quality of SRG process Offers recommendation to Institute/Center Director and staff Evaluates program priorities and relevance Advises on policy 3 - 7 months 2nd level Output: Funding Recommendations Alternate text: Box 1: Title is Scientific Review Group (SRG) This box describes the first level of review at the NIH. It includes an independent, outside review group managed by NIH that evaluate the scientific merit, significance and other review criteria. This group provides a priority score and recommends the length of award and level funding. The output is an impact score and summary statement. Box 2: Advisory Council Approximately 3-7 months later the second level of review is conducted at the NIH Institute or Center Advisory Council. This group assesses the quality of the SRG process, offers funding recommendations to the Institute/Center Director and staff. It evaluates program priorities and relevance and advises the Institute/Center Director and staff on policy. The output includes funding recommendations. Box 3: Institute/Center Director Approximately 1-3 months after Advisory Council meeting, the Institute/Center Director makes the final funding decisions based on Council input, assessment of programmatic priorities, etc. The applications must also pass administrative review to ensure compliance with policies, procedures, and regulations. The output is either an award to the grantee or a resubmission by the grantee. Institute/Center Director Makes final decision based on Council input, programmatic priorities Must also pass administrative review months Output: Awards or Resubmission 8

9 Understanding Your ScorePeer review scoring system What does the percentile mean? Why do I have a percentile but my colleague doesn’t? What to do when you get your score? Wait until you have a summary statement to go with it before contacting your Program Officer Alternate text: The graphics at the top right of this slide are three individuals sitting at a table holding up score cards.

10 Understanding Your ScoreAssigned reviewers: scores for each criterion and an overall impact/priority score All reviewers score privately from 1 (best) to 9 (worst). Overall scores are averaged, rounded to one decimal place, and multiplied by 10 e.g., a 1.34 average is a score of 13. Alternate text: The graphics at the bottom of this slide are three individuals sitting at a table holding up score cards.

11 Understanding your scoreAfter the meeting, reviewers may edit their criterion scores and critiques, but not the overall impact/priority scores. Note: Reviewers do not always edit to reflect the discussion. Depends on the reviewer and the panel. Your priority score in your Commons account within 3 business days after the peer review Your summary statement typically available within 30 days. Alternate text: The graphics at the bottom of this slide are three individuals sitting at a table holding up score cards.

12 Understanding your scoreScore with percentile Score with no percentile Unscored Scientific merit in the lower 50% of the applications or beyond the group likely to be competitive for funding Not Recommended for Further Consideration Indication of serious concerns. Usually not eligible for funding Alternate text: The graphics at the bottom of this slide are three individuals sitting at a table holding up score cards.

13 Understanding Paylines and PercentilesPercentiles indicate a relative rank of your application compared to others reviewed by your study section at the last three meetings. Percentiles are used counter “score creep” and variation in use of review criteria by different study sections. Percentiles range from 1 to 99 in whole numbers. Smaller number is better (e.g. 10 is better than 25). Alternate text: The graphics at the bottom of this slide are three individuals sitting at a table holding up score cards.

14 I have a percentile but my colleague doesn’t……Percentiles are only meaningful when there is a good comparison group such as all R01 applications reviewed by the CSR “XYZ” Study Section for the last 3 meetings Not meaningful when… Special emphasis panel Small number of one type Major change in process impacts some but not all applications Alternate text: The graphics at the bottom of this slide are three individuals sitting at a table holding up score cards.

15 Interpreting Your Summary StatementWhen will I get my summary statement? Usually about 4-8 weeks after the review panel meeting Tips about contacting NIH staff Review and “digest” it Review the “Next Steps” guidance provided on your summary statement Look under SRG Action: Impact Score on the SS Who do I talk with about my summary statement? Program Officer, not the Scientific Review Officer Colleagues and/or mentors Alternate text: The graphics at the top of this slide is a stick figure with a question mark over the head.

16 Interpreting Your Summary StatementHow do I find my program officer? eRA Commons Account Upper Left Corner of the Summary Statement What questions do I ask? Likelihood of funding Know that they probably won’t be able to tell you for certain but you can get a feel for it Thoughts about resubmitting Timing for funding consideration and next steps Alternate text: The graphics at the top of this slide is a stick figure with a question mark over the head.

17 Likelihood of Funding… it dependsOctober to January No budget yet (usually) Operate under a continuing resolution previous FY $ level Fund a limited number of applications. Other awards delayed Winter Until Spring Have a budget Congress passes appropriations bill, president signs into law, setting NIH's budget levels DHHS needs several weeks to analyze budget Alternate text: The graphics at the top of this slide are dollar bills with a $ and smiley face and feet.

18 Likelihood of Funding… it dependsSpring to Early Summer Budget arrives at Institutes Number crunching Each institute sets their financial management plan Fund grants within payline Fund a few grants through special funding programs July to September Close of fiscal year Number crunching Revisit payline if needed Pay deferred applications Special pay programs Alternate text: The graphics at the top of this slide are dollar bills with a $ and smiley face and feet.

19 Likelihood of Funding… it dependsAt the start of the fiscal year, you will more likely have to wait for an award because funding is limited. If your application missed the payline, it may get funded later in the fiscal year. Institutes may have funding pools to use for high priority or special need areas (e.g. selective pay, Bridge awards). Stay in touch with your Program Officer Alternate text: The graphics at the top of this slide are dollar bills with a $ and smiley face and feet.

20 Sample Summary Statement*After the meeting, all reviewed applications receive an overall impact score and summary statement prepared by the SRO. NIH releases scores to you and your program officer in the Commons within three business days and uploads your summary statement within 30 days. Your summary statement has a lot of information: bulleted critiques from your assigned reviewers brief summary of the discussion overall impact score and percentile for R01s criterion scores from your assigned reviewers recommended budget human and animal subjects codes any administrative comments If necessary, you'll use this information to revise a fixable application as we describe in Part 6. If Not Funded. But keep in mind that although your summary statement gives you critical feedback, it is not an exhaustive critique or a teaching tool containing every point reviewers found to be problematic. We tell you more in Assess Peer Review Results in What to Do if You Get Bad News in Part 6. *Summary statement shown with permission for training purposes

21 Sample Summary Statement*Alternate text: Image of the top part of a summary statement including headings on application information and the resume and summary of discussion section of the summary statement. Summary text: But keep in mind that although your summary statement gives you critical feedback, it is not an exhaustive critique or a teaching tool containing every point reviewers found to be problematic. *Summary statement shown with permission for training purposes

22 Sample Summary Statement*CRITIQUE 1: Significance: 2 Investigator(s): 1 Innovation: 1 Approach: 1 Environment: 1 Overall Impact: The research in this application should lead to a paradigm shift in our understanding of the development of the neonatal immune system, particularly the shift from a tolerogenic phenotyp utero to a highly responsive phenotype after birth. The hypothesis is clearly presented, and the experiments are expected to unequivocally prove or disprove the hypothesis. Either way, the r be of great interest. If the hypothesis proves to be true, it may become possible to identify those newborns at the highest risk for serious or life-threatening infections, and to tailor immunization strategies to an individual’s immune maturity status. *Summary statement shown with permission for training purposes

23 Sample Summary Statement*Significance: Strengths The limited ability of the neonatal immune system to effectively resist infections is a maj of morbidity and mortality. Understanding the reasons for this would allow us to identify at the highest risk for infections, might allow us to tailor organization approaches appro and might ultimately lead to immune modulatory approaches. The clear demonstration of layering in the human developing immune system would gr the prevailing paradigms of immune system development. Weaknesses Investigator(s): Strengths The PI has a long and highly accomplished record in this field, and is exceptionally wel for these studies. Innovation: Strengths The application of the concept of layering to the developing human immune system is innovative. The hypothesis that temporal differences in the layering process might lead to variation balance of Th1 vs Th2 type responses is highly innovative. The development of a transcriptional signature for fetal versus adult lineage T cells is hi innovative. Approach: Strengths The extensive preliminary data, all of which is highly relevant to this proposal, is a clear of the application. The extremely clear presentation of the experimental approach is another strength. The inclusion of multiple collaborators at different sites as sources for cord blood is good. The well-thought-out and clear data analysis plan is a strength. The overall synergy between the aims and between the sub aims is remarkable. The expected results and their interpretation are clearly presented. Environment: Strengths The environment at UCSF and the associated clinical sites is essentially perfect for this study. *Summary statement shown with permission for training purposes

24 Sample Summary Statement*Protections for Human Subjects: Acceptable Risks and/or Adequate Protections No concerns Data and Safety Monitoring Plan (Applicable for Clinical Trials Only): Inclusion of Women, Minorities and Children: G1A - Both Genders, Acceptable M1A - Minority and Non-minority, Acceptable C1A – Children Included, Acceptable no concerns Vertebrate Animals: Not Applicable (No Vertebrate Animals) Biohazards: Acceptable no concerns Roster follows *Summary statement shown with permission for training purposes

25 What if I’m not selected for funding?Understand your summary statement Discuss options with your Program Officer Discuss options with colleagues and/or mentors Evaluate your options Prepare to re-apply, if applicable Alternate text: The graphics at the top of this slide is a stick figure with a question mark over the head.

26 What if I’m not selected for funding?Assess peer review results and read summary statement carefully Can you readily fix reviewers concerns? Can you clarify things that reviewers misunderstood? Did reviewers have conceptual problems that you can address? Alternate text: The graphics at the top of this slide is a stick figure with a question mark over the head.

27 What if I’m not selected for funding?Talk to your Program Official Are there options for special funding?? Ask the PO about Reviewer support for your idea Additional insight from the review Any issues with the presentation Alternate text: The graphics at the top of this slide is a stick figure with a question mark over the head.

28 What if I’m not selected for funding?Hard to fix Problems – When to go in a new direction Low-impact research topic. Philosophical issues, e.g., the reviewers do not think the work is highly significant. Hypothesis is not sound or not supported by the data. Work has already been done. Methods proposed were not suitable for testing the hypothesis. Alternate text: The graphics at the top of this slide is a stick figure with a question mark over the head.

29 What to do if I am being considered for funding?Resolve concerns E.g., human subject protections, vertebrate animal protections, inclusion of women, minorities, and children Complete Just-In-Time Requirements Restricted Awards Alternate text: The graphics at the top of this slide is a stick figure with a question mark over the head.

30 Resolving Concerns A concern is a Bar to Award:Human and Animal Subjects Protection Concern code = 44 Inclusion of women, minorities, and children concerns = U (unacceptable) code How to resolve concerns Read your summary statement Determine the issue(s) e.g., source of materials, identifiers, risks, protections, informed consent, missing inclusion information, poorly justified sample Contact your Program Officer Submit requested materials via the just-in-time module Resolution may take a few weeks Restricted award may be issued Typically at the end of the fiscal year Activity with concern may NOT commence without resolution Alternate text: The graphics at the bottom of this slide is a stick figure with a clipboard with a checklist.

31 Other potential bars to fundingLack of assurance code = 20 Research Misconduct Institutions are required to establish and maintain an assurance that certifies they have a process for responding to allegations of research misconduct. This does not appear in the summary statement Generally due to institution not updating ORI their research misconduct assurance or not having a misconduct assurance in place (usually new institutions, first time grantees) Alternate text: The graphics at the bottom of this slide is a stick figure with a clipboard with a checklist.

32 Just in Time RequirementsWhat is just in time information? Information that NIH requests after the initial peer review Information NIH needs to make an award but doesn’t require in your application Application within the range of possible funding; JIT request doesn’t mean that your application will be funded. See Section of the NIH Grants Policy Statement for more information: st_in_time_procedures Alternate text: The graphics at the top of this slide is a stick figure sitting on an hourglass.

33 Just in Time RequirementsHow will I know to submit just in time? Standard from NIH. The NIH issues automatic just-in-time s for all applications that get an overall impact/priority score of 40 or less. These requests should arrive to you and your business office within 15 days after you get your percentile rank. Personalized from individual institute or center. Grants Management staff issues personal just- in-time requests when funding is likely. Alternate text: The graphics at the top of this slide is a stick figure sitting on an hourglass.

34 Just in Time RequirementsWhat is just in time information? Current Other Support for all key personnel (may need update after 120 days) Human subjects: IRB approval date or Institutional Assurance Documentation of required education in Protection of Human Subjects Research Participants for key personnel Animal Research: IACUC approval date Alternate text: The graphics at the top of this slide is a stick figure sitting on an hourglass.

35 Just in Time RequirementsWhat is just in time information? Confirm administrative information such as Entity Identification Number (EIN) Confirm F&A agreement date and rate Responses to Summary Statement Concerns Patient Care Costs Alternate text: The graphics at the top of this slide is a stick figure sitting on an hourglass.

36 Just in Time RequirementsWhere do I send the information? Electronic Submission of Just-in-Time Information NIH Guide Notice NOT-OD Your business official will submit through eRA Commons. Be prepared Know what we will ask for and when to send it Contact person: Grants Management Alternate text: The graphics at the top of this slide is a stick figure sitting on an hourglass.

37 After the Award… Now What?Understand what’s in your Notice of Award Post-review reporting requirements Research Progress Performance Report (RPPR) Alternate text: The graphics at the top of this slide are dollar bills with a $ and smiley face and feet. 37

38 Notice of Award (NoA) Legally binding documentAward data and fiscal information Grants payment information Terms and conditions of award Terms and Conditions in FOA Terms and Conditions in your NoA Used to clarify information, provide programmatic structure, or restrict how you can use funds. Where to get information? eRA Commons; use the Status module or Issued Notice of Award query. Your institution accepts the terms and conditions of award when you spend the money Alternate text: The graphics at the top of this slide are two stick figures shaking hands. 38

39 Differences Between Request and AwardPeer review may have recommended changes to budget, performance period, or your research plan Overlap may cause funding levels to be lower Programmatic reduction Some ICs reduce award budgets when the annual appropriation doesn’t allow funding at council approved levels. Alternate text: The graphics at the top of this slide are two stick figures shaking hands.

40 Reporting After Award Annual Research Progress Performance Report (RPPR) Annual federal financial reporting Subaward reports Invention reporting Yearly audits (as applicable) Closeout reporting Alternate text: The graphics at the top of this slide are two stick figures communicating through cans connected with string. 40

41 Additional Reporting related to the researchExamples: Clinical Research and/or Trials Protocols registered in Clinical trials.gov Inclusion enrollment reporting at least annually Institute/Center-specific reporting requirements Human subjects IRB approval every year Animal subjects IACUC approval every 3 years Data Sharing Model Organisms Public Access Alternate text: The graphics at the top of this slide are two stick figures communicating through cans connected with string.

42 Resources for InvestigatorsNIH Grants Process Overview Homepage: Award Management page: NIH Grants Process: The Big Picture (YouTube) https://www.youtube.com/watch?v=rNwsg_PR90w Policy pages dedicated to specific topics Examples: Human subject protections, inclusion of women, minorities, and children, animal protections, peer review, public access, etc. Institute/Center-specific resources Grants process Funding plans Reporting and award requirements Alternate text: The image at the top of the slide is a stick figure with a light bulb over their head and their hand raised by the light bulb.