1 Reviewing CONFERENCE abstractsPatricia O’Sullivan Research and Development in Medical Education

2 Acknowledgments Paul Hemmer, MD, MPH G. Dodd Denton, MD, MPHUniformed Services University of the Health Sciences G. Dodd Denton, MD, MPH Karen Szauter, MD University of Texas Medical Branch School of Medicine Sebastian Uijtdehaage, PhD UCLA (now at Uniformed Services University of the Health Sciences) Brian Mavis, PhD Michigan State University

3 Creative Commons LicenseAttribution-NonCommercial-Share Alike 3.0 Unported You are free: to Share — to copy, distribute and transmit the work to Remix — to adapt the work Under the following conditions: Attribution. You must give the original authors credit (but not in any way that suggests that they endorse you or your use of the work). Noncommercial. You may not use this work for commercial purposes. Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one. See for full license.

4 Workshop Objectives Define and describe major structural components of a conference abstract Recognize what makes each component effective Describe the elements required in a “Methods” and “Results”/ “Evaluation Plan” section Avoid common pitfalls in a “Discussion” section Describe the peer-review process for a conference abstract Write reviews for abstracts being considered for a conference

5 Abstracts are different than manuscriptsOften describe preliminary or pilot studies Word restrictions (300 max) limit the scope Exclude supporting documentation such as references, tables, etc. Remember that abstracts are a unique kind of document… it is common to think of them within the context of a manuscript. In the context of this workshop, the abstract stands alone without a supporting manuscript. As a result: Abstracts are brief (usually words) and more limited in what can be communicated Abstracts often report on preliminary or pilot studies, a new innovation, or present only part of a larger study Abstracts stand alone without references, extensive literature review, tables or figures.

6 Anatomy of abstract Title Introduction Methods Results/Evaluation PlanPurpose/Background Methods Results/Evaluation Plan Discussion/Dissemination Reflective Critique

7 THE INTRODUCTION

8 The Introduction provides background or rationaleReference to what is already known Reference to general knowledge or accepted standards of practice (“conceptual framework”) Addresses the “So what?” question Peer-feedback influences a learner’s knowledge and skill acquisition. As residents progress in their education, it is possible that their peers are valuable sources of feedback. The introduction provides the basis for the project. The opening sentence may describe what is known, or address issues that are currently "hot topics".

9 A clear Problem Statement/Purpose must be includedReviewer should clearly understand: What question was being investigated, and Why this question is important Abstract is unacceptable without a clear purpose/research question Our purpose is to describe the feedback surgical peers give each other about skills and how it varies with the use of a guide. The reviewer should want to know more because it is interesting, not because of a lack of clarity of what the study is about or the questions underlying the purpose of the study. If the reviewer is confused or not interested at the end of Introduction section, the remainder of the abstract is going to have to be very strong to pass muster as acceptable. The reviewer should understand the reason for investigating the research question stated at the end of the introduction/background. The reviewer should also have a sense of the question to be answered as a result of this study.

10 Problem Statement, Purpose, or HypothesisProblem statement/Research Question “We investigate the relationship between USMLE Step 1 scores and self-assessed study skills” Statement of Purpose “Our purpose is to develop a curriculum for students to improve skills in communicating bad news” Hypothesis “We hypothesized that our students would demonstrate improvement in exam scores after participating in on-line remediation” The Introduction should always make a statement of what the study is about. This might take the form of a problem statement, purpose statement, or hypothesis. The form of the problem statement reflects the nature of the study and should set the stage of the Methods to be used. The remainder of the abstract centers around the problem statement or hypothesis If this is not clear, the focus of the study will be difficult to follow given the brevity of the abstract, and the appropriateness of the methods, results and discussion cannot easily be assessed 10

11 References to the literature are not requiredWord limits preclude full reference citations Authors might include abbreviated citation, e.g.: Smith, 2008. Allows author and reviewer to differentiate between “opinion” and building on another’s work Word limitations often preclude the inclusion of complete references at the end of the abstract. Authors might consider an abbreviated citation (e.g.: Smith, 2008) in the body of the text if they feel it necessary, such as a study that is extending or replicating a previous study.

12 Example 1 INTRODUCTION Increasing the enrollment of students from underrepresented groups in medical schools continues to be a challenge. Recent data (Smith, Jackson & Jones, 2011 ) showed that only 7% percnt of physicians are minority whereas this proportion is 25% in the US population overall. Pipeline programs are designed to improve diversity in medical schools. This study focused on the academic performance of underrepresented (UR) students on courses that are required for applying to medical school (“gateway courses”). The introduction does not flow well. There is a disconnect between the context (difficulty of recruiting minority students through pipeline programs) and the purpose of the study. Moreover, the focus of the study remains vague. There is no clear research question or purpose stated. Citations are not required in abstracts and may not be the most effective use of the limited number of words allowed. Typos are annoying and suggest that the authors gave minimal attention to the abstract. It is doubtful that all authors signed off on this abstract.

13 Example 2 INTRODUCTION Increasing the enrollment of students from underrepresented population in medical schools continues to be a challenge. Minority students’ achievement in college courses that our required for application in medical school (“gateway courses”) is key to successful admission in medical schools. We sought to determine (a) whether minority students receive lower grades than non-minority students in pre-health gateway courses; (b) the extent to which lower grade performance might be explained by the differences in precollege academic achievement; and (c) whether minority students are less likely than non-minority students to persist in completing at least four gateway courses. This introductions flows better. It briefly describes the context and why this is important. It articulates the purpose of the study clearly. A minor point is that the authors could have mentioned a conceptual framework.

14 METHODS

15 General Considerations for MethodsOperationalizes the research question or the innovation Describes a quantitative, qualitative design, or mixed design OR Describes the framework for development of innovation Explicit statement about IRB approval or review is optimal, but not required Clarity: The abstract is a short story and although brief, it should all make sense, with each section flowing logically into the next. The research question drives the study design, and as previously stated, should be clear as to what the study is about. If you cannot follow the Methods section, it is not well written Operationalizes the Research Question Key variables and how they are measured, assessed, or explored will be discussed. The approach to the data collection and analysis should be apparent. Ethics review While very important, and ultimately necessary in most cases for presentation or publication, in writing an abstract it is often omitted because of the word limitation. At this point it is not a requirement. Not all institutions view medical education research as requiring ethics (also called IRB) review. This is an evolving standard, and in the future, we expect that almost every medical education research project will be required to show evidence of ethics review.

16 Methods describe: “Who, where, what, when, how?”Who is sampled? Patients, students, residents, fellows, faculty? What is the sample size? What is the setting? What is the source of data? Survey, papers, examinations, etc. What is the study design? When is this study done? How is the data analyzed? The Methods should describe: Who was sampled and was it appropriate? Is the sample representative or appropriate for the problem being studied? Does the sample size seem very small or very large? How was data collected? Is there enough information about instrumentation? What is the context of the study… did it take place in a hospital, community clinic, classroom, small group, etc? Is the research design clear? Optionally, when was the study done? How were data analyzed?

17 What reviewers look for in Qualitative StudiesWas sampling appropriate to the question? Was data collection method appropriate? Interviews, focus groups, survey responses, written comments Was data analysis done with rigor? Who analyzed the data? How were coding difference resolved? When reviewing qualitative research, here are some of the key questions to consider: A. Was sampling appropriate to the question? --Who was interviewed? What were the sources of information? B. Was data collection appropriate, explained? --Interviews, focus groups, survey responses, written comments, reflective essays, medical records, etc… --Triangulation may be used—different data sources or methods to address question

18 Innovation or Curriculum DevelopmentGuiding framework for development Methods used for steps taken Ensure the quality of the innovation or curriculum developed Data planned or collected

19 Example 1 METHOD Using regression analysis, we analyzed academic performance of 5,000 college students in three California colleges who matriculated in the 2009–2010 or 2010–2011 academic year. We compared grade performance of students across ethnic groups in gateway courses and the degree in which they completed at least four gateway courses, using regression methods to control for students’ college admission test scores. Statistical significance was set at Source of data is unclear. Statistical significance set at 0.05 is common practice and does not need to be described in an abstract. The authors describe who is sampled but ethnic groups unclear

20 Example 2 METHOD We obtained retrospective, administrative data from 3 California colleges on 5,000 college students who matriculated in the 2009–2010 or 2010–2011 academic years and enrolled in at least one college gateway course. We compared students across ethnic groups in physics, organic chemistry, biochemistry and calculus* grade performance and persistence in completing at least four gateway courses, using regression methods to control for students’ college admission test scores. * Made up “gateway courses” This methods section contains most required elements. It alludes to a retrospective analysis for archived data although the study design is not clear. The authors describe 1) who is sampled; 2) sample size; 3) setting; 4) source of data; 5) when the data was collected; 6) type of analysis employed. They do not mention IRB approval; but mentioning this is not required for an abstract.

21 Curricular Example 1 METHODWe held weekly seminar taught by faculty from medicine and law schools for ten weeks. The curriculum focuses on areas where medicine and law intersect, including foster care, prisoner health, caring for refugees, and homeless health policy. Participants are exposed to academic and community leaders who express interest in developing mentoring opportunities with students. Eleven law and thirteen medical students enrolled in the seminar during the first year. Seminar sessions are conducted in lecture-based format for one hour followed by half hour small-group discussions. Students simultaneously interact with each other in the clinical setting when providing direct services with faculty supervision, and are encouraged to discuss these experiences. Don’t know what guided the content, methods Mixing in a bit of purpose and results

22 Curricular Example 2 METHODWe developed the joint medical and law seminar following a curriculum development framework where we analyzed needs, set objectives, chose teaching strategies, implemented and evaluated the seminar. Needs assessment indicated selecting topics where medicine and law intersected such as foster care, prisoner health, caring for refugees, and homeless health policy. Since we were emphasizing the intersection, we chose a format that allowed for sharing as well as didactics. Didactics were reviewed by the development team prior to sessions. Success was assessed by numbers enrolling from both schools, attendance, and learner and faculty satisfaction.

23 RESULTS

24 Results must present key findingsActual data (numbers) must be presented Congruent with Methods section Present descriptive data Inferential analyses, with p values, should be present as appropriate. (but p values alone are uninformative) Key features: Actual data 2. Congruent with Methods. For instance, no analyses are left out from the Methods, and there are no “surprise” analyses that were not mentioned. 2. Key findings should be presented, understanding that space will limit inclusion of all results. 3. In a quantitative study, actual data (numbers) should be presented rather than a summary of findings. Descriptive information is usually present, such as sample size, response rate, total number of responses analyzed. This is often reported as the Number and percentage: N (%) Inferential analyses from the Methods should be reported to include: the actual data, the p value, and a measure of effect (e.g., effect size, confidence intervals, percent of variance explained). Recognize that due to word limitations in an abstract this section: might not include all information about statistical tests might not include all of the analyses

25 Results: Essential componentsResponse rate to a survey Participation rate in cohort studies Numbers in control and intervention groups Nice to have: Numbers and/or reasons for exclusion Demographic information Descriptive information may be limited in abstracts but should be sufficient to characterize the dataset and to substantiate the use of appropriate inferential statistical tests (e.g., presentation of means/standard deviations). Common descriptive results include: Response rate to a survey Participation rate in cohort studies Numbers in control and intervention groups Numbers and/or reasons for exclusion Demographic information The numbers should be expressed in commonly used terms (eg., means, medians, mode, range, standard deviation).

26 Data and p values must be reportedWRONG: “Group A scored significantly higher than group B (p < .05)” RIGHT: “Group A scored higher than group B (means: 10 ± 2 versus 4 ± 1, p<.05)”

27 What influences statistical significance?Sample size As sample size increases, smaller differences needed for significance Is the finding significant because of a large sample? If so, is it MEANINGFUL? Multiple comparisons As number of statistical tests increases, likelihood of at least one statistically significant finding by chance alone increases the p value can and should be adjusted for multiple comparisons How would you see this in an abstract? Lots of p values being reported! Reviewers should understand that many factors can affect statistical significance, including sample size and multiple comparisons.

28 “Negative” studies Negative findings can be important for learning what works (or doesn’t work) in medical education However, need to consider if negative or non-significant findings due to lack of statistical power Most common reason is insufficient sample size With negative findings, author should include a comment about power in the abstract If absent, this is a feedback point in the written comments Negative or non-significant findings in a study does not mean it was a poor study. Such findings can be important for learning what works or what doesn’t work in medical education. As a reviewer, you must judge whether the negative findings are important to share with the community. However, if a study does not demonstrate a difference, it may be because there is a lack of statistical power. The most common reason for a lack of power is too small of a sample size, such that meaningful differences cannot be uncovered by a statistical test. If negative findings are present in an abstract and statistical power is not mentioned, the reviewer should inquire about power in their written comments to the author.

29 Results: Qualitative StudiesDescription of themes and sub-themes Word limit generally precludes example quotes Differences among participants, groups may be discussed Quantification of the results often not included or relevant In medical education studies that used qualitative methods, the Results section is often a description of the findings of the analysis. This may be limited to presenting the Themes and sub-themes that emerged from the analysis. In contrast to a manuscript of a qualitative study, direct quotes that illustrate the themes often cannot be presented due to word limitations. As such, in an abstract, summarizing the themes identified is often sufficient to satisfy expectations and requirements. Differences among participants or groups studied may be addressed, if this is relevant It is important to understand that Quantification of the results (e.g., X% of the respondents noted Theme 1) is often not included or necessarily relevant.

30 Results: Common PitfallsNo data (actual numbers) are reported Results “will be presented at the meeting” is unacceptable [*may depend on conference but helps if have preliminary or provide plan] No or low response rate (with surveys) Not enough detail about measures Statistical analyses inappropriate for type of data Authors interpret data instead of just listing results: “Analyses support our hypothesis that our intervention was effective and should be broadly implemented (60 vs. 35, p<.05)”

31 Example 1 RESULTS Gateway courses included Biology, Organic Chemistry, Calculus and Physics . Calculating the GPA for these courses, we found that on average , the GPA was lower for UR students compared to non-UR students (p<.05). This gap could not be explained by differences in prior academic performance. However, no statistically significant differences were found in the rate in which UR students completed at least four gateway courses compared to white students. However , after accounting for the lower grades of UR students in their initial classes, UR students were more likely than non-URM students to complete four or more gateway courses (p<.01). “Gateway courses included Biology, Organic Chemistry, Calculus and Physics” This is a not a research finding but part of the methodology. This could be omitted given this information is widely known. Authors do not provide data, but just p values. The sample size of each group is required. Inconsistent terminology. Are “white students” the same as “non-URM” students? STYLE: The frequent use of however, makes it difficult to determine what the main finding is.

32 Example 2 RESULTS On average, the GPA on gateway courses was 0.63 lower for students who are UR (N=1,800) compared to students who are not UR (N=3,200, R2=0.3, p<.05). This gap persisted after adjusting for measures of prior academic performance. However, logistic regression showed that students who are UR were nearly as likely students who are not UR to persist in completing at least four gateway courses. After accounting for the lower grades of students who are UR in their initial classes, UR students were more likely than non-UR students to complete four or more gateway courses (adjusted odds ratio = 1.25, p<.01). The results section includes data and even effect sizes (R2 and odds ratio). It corresponds well with the research questions listed in the introduction. The authors appropriate avoid interpretation of the data which is reserved for the Discussion section

33 Evaluation Plan option

34 When evaluation is planned but not completedMost appropriate for curricular innovations where work has been done to develop the curriculum that has not yet been implemented Indicate type of data to be collected From whom How analyzed Timeline In a way, an extended methods section

35 Discussion

36 Discussion describes what the results meanInterpretation of results General reaction to findings Statements of limitations Thoughts on study’s impact Proposals for next steps Things to include when writing the Discussion: What do the results mean, especially in relation to initial problem statement or hypothesis? --were the results as expected? What are possible limitations of this study? What are possible next steps for investigation? Were any lessons learned?

37 Discussion pulls all information togetherDiscussion must relate to problem statement/ hypothesis Interpretation of study’s findings What do the results mean? Do they answer the original hypotheses or problem statement? The Results section is a statement of actual findings, the Discussion is the author’s interpretation of the meaning of the findings. The discussion is the section of the abstract that allows the authors to provide their interpretation of the study, its meaning and/ or practical significance. Statements made in the discussion should be supported by the Methods/Results and should add a “closure” to the project. Simply reiterating the Results in the Discussion without interpretation is inadequate. The authors should provide some insight into the meaning of what they found, or comment on unexpected findings. The Discussion should refer back to the original problem statement or hypothesis

38 Discussion: Common pitfallsRehash of the results without interpretation: “In this study we found that the intervention group performed significantly better than controls” Assertions are made that are not supported by the data Obvious limitations are ignored As a reviewer, it is important to question the Discussion/Conclusions: Is what the authors have stated balanced? Based on what they have presented in the Methods and Results, was the study both appropriately done and are the results adequately presented such that the conclusions made are reasonable? Does the Discussion appear to interpret information outside of what was presented?

39 Example 1 DISCUSSION Scores in Gateway courses were statistically lower for UR students compared to white students. However, they completed four or more of these courses at the same rate.  Our study clearly shows that academic support for UR students in college targeting gateway courses is crucial for increasing the diversity in medical schools. This is rehash of the results section. The DISCUSSION should focus on the meaning and implication of the results. It is debatable that this conclusion is supported by the findings.

40 Example 2 DISCUSSION Despite academic challenges the minority students experience in college, many persist in their pre-health course requirements. Interventions at the college level to improve minority student performance in gateway courses are particularly important for increasing the diversity of medical schools This discussion is fairly well written. It ties together the problem statement described in the Introduction. It also describes the implication of the study which could be followed up with another study. Word limit precluded authors to describe the limitations of the study.

41 Review criteria

42 UCSF Education ShowcasePurpose: clearly describe the project's goals and objectives. If you have a hypothesis, please state it here. Background: Describe how project was informed by knowledge of previous scholarly work or knowledge of educational principles. Methods: Describe the approaches used for this project. Who was studied or who will evaluate this project? What data were collected and how? Results/Evaluation Plan: Describe the findings of the study/evaluation including response rate, results, and statistical testing where appropriate. OR Describe the outcomes that will tell you whether your curriculum is effective. Discussion/Dissemination: State concisely the data-driven conclusion of your study/evaluation and how the results of the project are being (or will be) made available to colleagues. Describe your plans for making the results of your project available to colleagues (talks, posters, publications, websites, etc.). Reflective Critique: Describe the steps taken to solicit feedback regarding your project and how it has been (or will be) modified based on the feedback.

43 Reviewers make a final recommendationDefinitely Include Highly rated in most/all categories; highly relevant Probably Include Few concerning areas; can be improved before presenting. Relevant Probably Exclude Several areas of concern, will require significant revision to remedy Definitely Exclude Many areas of concern; fatal flaws present; needs substantial revision Multiple reviewers will be providing input on this decision so it is not entirely up to one reviewer 43

44 Education Showcase DecisionsDefinitions: Plenary Presentation: 10-minute presentations featured on the first day of the Showcase Mini-oral Presentation: 5-minute presentations take the place of the poster sessions at previous Showcases. Recommend for Plenary Presentation Recommend for Mini-oral Presentation Acceptable for Mini-oral Presentation Not Acceptable

45 Reviewers etiquette Here are some practical suggestions for reviewing abstracts gathered from experienced reviewers…

46 Conforming to Submission RequirementsAs a reviewer, it is important to be familiar with the submission requirements Some submission systems reject an abstract that exceeds the word count If the abstract seems long, check the word count and comment on it in the review if excessive The reviewer has an obligation to make a comment to the meeting planner if an abstract really goes over the word limit. Program committees may differ on how rigid they are with respect to “the rules”, but at least the issue should be brought to the committee’s attention.

47 General Rules Always write comments Your chance to convey:Engage authors in feedback Your chance to convey: Concerns/suggestions for improvement Compliments Reasons for ratings of abstract elements Consider writing comments FIRST, then use the rating form Your written comments to the authors and the meeting planners are essential and you should ALWAYS make written comments. These comments give you the chance to convey: Specific Concerns you have about the abstract, and how the authors might address those concerns. You can and should also provide compliments on studies that were well done, well executed, clear, and/or provide important findings. Written comments also allow you to provide the reasons for your ratings of each section of the abstract By writing your comments first, it will help you organize your thoughts, and outline your concerns and/or compliments; in this way, the rating should become more obvious Your final rating of the elements as well as the abstract overall should be a weighing of the strengths and weaknesses. Remember that there will be multiple reviewers so your decision along will not decide the fate of the abstract, but can help the conference planners make an informed decision.

48 Written Comments Be respectful Be specific Be constructive Be focused--if you wouldn’t say it to the author, then don’t write it to the author --being a reviewer protected by confidentiality does NOT give you license to be rude, disrespectful or disparaging Be specific: --make note of specific strengths of the abstract --identify what you think are specific weaknesses of the abstract --provide suggestions for how the authors might address your concerns or improve their work Be constructive: --give the authors suggestions on points they might want to clarify or add when they are making their conference presentation --if there is a key reference that you think might be helpful to illustrate your point or give guidance to the author, include it in your comments Be focused --Comment on the study that was done, not about what they should have done --If you believe the study has a fatal flaw, then you should comment on that and how the authors could address that in the future. --If you believe that additional analysis is necessary to improve the project, then suggest that --If additional study is needed, suggest to the authors what they might do as a next step

49 Example of Comments Poor Improved “The worst abstract I’ve ever read!”“Data presentation was pitiful.” “Wonderful study!” “The authors need to attend to several areas before resubmitting; these include:” “The authors need to present the actual data and statistical analyses.” “This was an excellent study for the following reasons:” Be specific in the comments section – statements like “clarify” or “revise” are not helpful unless they specify what needs clarification or revision, i.e., clarify methods section – specify the statistical test used. General comments like “well-written” or “I enjoyed reading this abstract” might soften the reviewers comments, but are not often useful to the authors. 49

50 Are you Available to Review?Most reviews must happen within a discrete (and often short) time period Look at your calendar before you agree to review If the timing is bad, politely DECLINE! Before you agree to participate as a reviewer, be very conscious about the period for abstract review. It is simply professional courtesy to know whether or not you are available to review. Requesting not to review in a given year will not prevent you from reviewing in the future, but failing to follow-through on a commitment can result in being dropped from a reviewer list. The conference planners typically have limited time to put the program together and need the reviews turned in by the deadline.

51 Comfortable with the Content?Are you comfortable addressing the specific topics of each submission? If you are not an appropriate reviewer, return it immediately so other reviewers can be found Scan abstracts as soon as you receive them to verify that you are comfortable addressing the specific topics covered in each submission If for any reason you feel that you are not an appropriate reviewer for a submission, return it immediately to the conference planning team so that it can routed elsewhere

52 Writing Review exerciseRead abstract Solicit comments Have them look at annotation Then look at write up and see what they think.

53 Reviewing 2017 Education ShowcaseRetrieve your three abstracts Read, write review, rate (do it in word first) Check in with your coach Submit your review and ratings through Qualtrics link