1 Diversity Considerations for Graduate AdmissionsCasey W. Miller Rochester Institute of Technology Director, Materials Science and Engineering School of Chemistry and Materials Science APS Committee on Minorities ( ) 2015 Inclusive Grant Initiative Division of Diversity and Inclusion

2 My Research: CM ExperimentEpitaxial Magnetite Thin Films Magnons and Spin Torque Magnetocaloric Effect in Heterostructures AFOSR-YIP (2008) EPMD (ECCS) (2008) CAREER (DMR) (2010) EPMD (ECCS) (2012)

3 APS News The Back Page Admissions Criteria and Diversity in Graduate School (Feb 2013)

4

5 Outline State of Physics Admissions Criteria and DiversityUsing Minimum Acceptable GRE Scores Negatively Impacts Diversity and does not select students with the most research potential

6 Physics is among the least diverse of the sciences.Percentage of Degrees Awarded to Underrpresented Minorities Credit: APS/Source: IPEDS Completion Survey

7 Physics is among the least diverse of the sciences.US pop. age 25-29 US pop. age 21-24 average of Credit: APS/Source: IPEDS Completion Survey

8 Physics is among the least diverse of the sciences.Fraction of Bachelor’s Degrees Earned by Women Credit: APS/Source: IPEDS Completion Survey

9 National Academies ReportTop Priority Actions Increase undergraduate retention and completion via strong academic, social, and financial support. Teacher prep, college prep programs, and Transition to Graduate Study. The EUMP report discusses several important factors for the STEM community to consider. These include student preparation (starting from elementary education), postsecondary access and motivation (xyx); affordability (xyz); and academic and social support (xyz). For each of these topics, there were several recommendations and action items set forth. Preparation: Early childhood education (womb through 3rd grade) to develop reading readiness and math skills and to introduce the concepts of creativity and discovery. Improving K-12 STEM education especially for URMs, including teacher preparation and retention. Postsecondary Success: improve postsecondary education and technical training, and increase URM student awarenes of and motivatino for STEM education and careers through improved information, counceling, and outreach; provide adequate support to URM students in ug and grad STEM education; Academic and Social Support: increase inclusion of and college completion and success in STEM education for URMs (they aspire to major in STEM fields at the same rate as majority students, but have significantly higher attrition rates during the early undergraduate years); They outline best practices that have been demonstrated by several programs, and note, importantly, that these practices help all students, but have been shown to be critical to URMs: summer programs targeting high schoolers and undergraduates; research experiences for undergraduates; professional development activites, especially at the graduate level; academic support and social integration within programs, especially by peer mentors; mentoring by engaged faculty, not necessarily research mentoring, but in a broad sense so as to allow the student access to sage advice for general and critical decisions. The report’s “Top Priority Actions” focus on undergraduate retention and completion; A cadre of qualified underrepresented minorities already exists who attend college, declare an interest in majoring in the natural sciences or engineering, and either do not complete a degree or switch out of STEM before graduating. An intense effort to reduce this attrition and bolster baccalaureate completion represents the most straightforward way to retain these students. • An increase in the completion of undergraduate STEM degrees by this population may also have impacts up and down the pathway. The visibility of increased undergraduate success may stimulate interest in STEM on the part of younger cohorts. And the increase in overall numbers will increase the pool of underrepresented minorities who may consider graduate education and careers in STEM. teacher preparation; college prep programs; and the transition to graduate studies. At the other end of the undergraduate years, the transition of underrepresented minorities to graduate work at top research universities, where they can contribute to research and leadership in our nation’s science and engineering enterprise, is also critical. Equally important to the undergraduate support recommended above, we believe, is that underrepresented minorities should constitute a similar proportion of new graduate students who are supported through portable fellowships, research assistantships, or institutional grants, in order to increase their overall representation and to move greater numbers into top graduate programs. 2011

10 Admissions is typically a two-tier process for all fieldsAdmissions Snapshot Admissions is typically a two-tier process for all fields

11 Undergraduate GPA GRE scores Undergraduate InstitutionTier 1: Filters Undergraduate GPA GRE scores Undergraduate Institution

12 Tier 2: Qualitative, HolisticResearch and leadership potential Creativity, Innovation Passion for research Diversity

13 It doesn’t really work much better than this

14 Many reasons for this, Admissions is only one

15 Issues with the norm GradesGrade Inflation undermines URM participation

16 Most URMs attend State CollegesURM Engineering #BA/BS Rank URM Physical Sciences #BA/BS University of Florida (240/yr) 1 Florida International University (85/yr) Florida International University 2 Xavier University of Louisiana Texas A & M University-College Station 3 The University of Texas at Austin University of Central Florida 4 University of California-Santa Barbara Georgia Institute of Technology-Main Campus 5 California State Polytechnic University-Pomona 6 The University of Texas at El Paso 7 University of California-Los Angeles 8 University of Florida North Carolina A & T State University 9 Spelman College The University of Texas-Pan American 10 University of California-Irvine Cal Polytechnic State University-San Luis Obispo 11 University of North Carolina at Chapel Hill The University of Texas at San Antonio 12 University of California-Santa Cruz Arizona State University-Tempe 13 University of Arizona University of California-San Diego 14 University of New Mexico-Main Campus University of Houston 15 Florida State University San Diego State University 16 Georgia State University Morgan State University 17 Jackson State University Prairie View A & M University 18 Alabama A & M University 19 Columbia University in the City of New York North Carolina State University at Raleigh 20 University of Memphis Southern University and A & M College 21 CUNY City College Howard University 22 CUNY Graduate School and University Center Tuskegee University 23 Savannah State University University of Maryland-College Park 24 University of South Florida-Main Campus 25 Georgia Southern University Virginia Tech (38/yr) 26 Tennessee State University (15/yr)

17 Issues with the norm Test Scores“Guide to Use of Scores” not followed (or even known of) Significant race/gender group differences Correlations with Success are questionable

18 CONSIDER INSTRUMENT RESOLUTION From ETS Guide to Use of Scores: It is an inexact measure; only score differences that exceed the standard error of measurement [are meaningful]” Translated to physics-ese: CONSIDER INSTRUMENT RESOLUTION S.E.M. ~60 points (on old GRE scale, ). 740 = 800 = perfect! Faculty on admissions committees are not well informed about how to properly use the GRE. While ETS publishes a “guide to the use of scores,” this document is unknown to most faculty. In it, ETS sets out many important limitations of the test, and makes recommendations about how to use the test scores. For example, they say that score differences are unimportant unless they exceed the standard error of the measurement. In physics, we would say to consider the instrument resolution. Many faculty believe there is a difference between a 780 and an 800 on the GRE general test. In fact, the standard error is 60 points, meaning that a 740 is equivalent to a perfect 800. We have a snapshot of what departments are actually doing with the exam, since the average GRE-Q was of matriculants was included in the NRC department rankings study. [Summarize these data with a histogram, include a median]. The importance given to the GRE is thus apparent. Half of those admitted to graduate school have scores above 760; only x% of programs have half of their students with statistically perfect scores (i.e., a dept with 740 in NRC, half a perfect, half are less than perfect).

19 US Physics PhD ProgramsAbout 180 graduate programs are listed in the AIP Graduate Programs book ~96% require the General GRE one quarter have a stated minimum score for admission, with the median being 700 (~64th%) ~48% require the Physics GRE one quarter have a stated minimum score for admission, with the median being 600 (32nd%).

20 From ETS Guide to Use of ScoresGuidelines A cutoff score based only on GRE scores should never be used as a sole criterion for denial of admission. Any department considering the use of a cutoff score should compile a rationale justifying the appropriateness of such a score for each measure: evidence that the proposed cutoff score for the measure usefully distinguishes between individuals who are likely to succeed in graduate school and those who are not, and the impact of the proposed cutoff score on the institution’s goals related to diversity.

21 GRE Quantitative Scores (2006-2007) Physical Sciences, US CitizensFrom ETS document "Factors that can influence performance on the GRE general test " 166 Median (NRC; Physics): 760 75% 25% 50% 155 148 GRE Quantitative Scores ( ) Physical Sciences, US Citizens 144 140 Other Hispanic N = 393 Mexican Am. N = 398 African Am. N = 1055 Puerto Rican N = 260 N = 12492 Men Women N = 7104 Asian Am. N =1474 N = 14957 White Native Am. N = 90

22 These performance disparities are:Not bias, according to psychometricians Nearly independent of intended graduate major Qualitatively unchanged when controlling for undergraduate GPA Qualitatively the same for GRE Subject test SAT Math 8th grade math achievement tests 4th grade math achievement tests A feature of standardized testing.

23 GRE Quantitative Score (2006-2007)SOURCE: ETS, "Factors that can influence performance on the GRE general test ”

24 GRE Quantitative Score (2006-2007)SOURCE: ETS, "Factors that can influence performance on the GRE general test ”

25 GRE Quantitative Score (2006-2007)SOURCE: ETS, "Factors that can influence performance on the GRE general test ”

26 GRE Quantitative Score (2006-2007)SOURCE: ETS, "Factors that can influence performance on the GRE general test ”

27 GRE Quantitative Score (2006-2007)SOURCE: ETS, "Factors that can influence performance on the GRE general test ”

28 GRE Quantitative Score (2006-2007)SOURCE: ETS, "Factors that can influence performance on the GRE general test ”

29 GRE Quantitative Score (2006-2007)SOURCE: ETS, "Factors that can influence performance on the GRE general test ”

30 GRE Quantitative Score (2006-2007)SOURCE: ETS, "Factors that can influence performance on the GRE general test ”

31 Mean SAT Math Score SOURCE: Total Group Profile Report, College Board, 2009 College-Bound Seniors.

32 Average Mathematics Scale Scores, 2007Find poverty rate vs race Asian White Hispanic Am. Ind. Afr. Am. SOURCE: NCES. The Nation’s Report Card: Mathematics 2009 (NCES ), National Center for Education Statistics, U.S. Department of Education.

33 Percent NOT living in povertyFind poverty rate vs race SOURCE: U.S. Bureau of the Census, Income, Poverty, and Health Insurance Coverage in the United States: 2010, Report P60, n. 238, Table B-2, pp ; Asian White Hispanic Am. Ind. Afr. Am.

34 Despite these R/E/G/SES issuesCut-off scores are used all the time by admissions committee members. Sorting spreadsheets is easy. Faculty are busy, often reluctantly serving Faculty are not trained in selection “Low scores must tell you something.” But correlation R~ is only with first year GPA; by second year, R<0.1a aNAS: In the Nation’s Compelling Interest: Ensuring Diversity in the Health-Care Workforce (2004)

35 Impact of Cut-off Scores?Number of Scores GRE-Q Score Only physical sciences & US citz.

36 Impact of Cut-offs: %Δ RepresentationAsian Am. White URMs Women Men 700 9.3% 82% 5.2% 27% 73% 250% Examinees 7.6% 77% 11% 36% 64% (155)

37 GRE Physics Subject TestWorking with APS And ETS

38 GRE Physics Subject TestFraction Above the Cut-off Cut-off on GRE Physics SOURCE: ETS (via APS)

39 GRE Physics Subject Test7172 467 155 821 Fraction Above the Cut-off Cut-off on GRE Physics SOURCE: ETS (via APS)

40 Gender Gap Persists ∀ R/E; ∀ Nationalities

41 From ETS Guide to Use of ScoresGuidelines A cutoff score based only on GRE scores should never be used as a sole criterion for denial of admission. Any department considering the use of a cutoff score should compile a rationale justifying the appropriateness of such a score for each measure: evidence that the proposed cutoff score for the measure usefully distinguishes between individuals who are likely to succeed in graduate school and those who are not, and the impact of the proposed cutoff score on the institution’s goals related to diversity.

42 Correlations with success are questionable

43 Working with APS to study this across Physics programs B+ Similar conclusions: Michigan; Berkeley Working with APS to study this across Physics programs (encourage your department to send us your data!) B B- C+ C

44 The usual weight given to GRE scores in admissions exceeds its predictive capabilities and has negative societal impact.

45 Diversity needn’t come at the expense of QualityLowering the bar perpetuates stereotypes exactly by definition What bar? Potential for what? RESEARCH!

46 Non-Cognitive ConstructsConscientiousness Trustworthiness Achievement Orientation Initiative Adaptability Optimism Measurable! Results from decades of Industrial-Organizational Psychology research non-cogs predict academic/job performance non-cogs show little if any group differences non-cogs are orthogonal variables to cognitive constructs (GPA, SAT/GRE) Needs to be as easy to use as GPA/GRE for 1st filter

47 Major Take-Aways Get a social scientist to help! Thanks!!!Traditional admissions process No better than a coin toss Strongly filters under-represented groups Non-cognitive assessments can help increase diversity by reducing barriers select students with enough grit to graduate Systematic evaluation protocols can limit implicit bias, increase diversity increase efficiency of screening applicants Get a social scientist to help! Thanks!!!

48 Self-Management competencies correlate with clinical grade.Victoroff and Boyatzis, J. Dent. Ed 77, 416 (2013): Correlating clinical performance with admissions criteria and non-cognitive competencies Self-Management competencies correlate with clinical grade. “Cognitive ability and knowledge are threshold aspects of professional work, necessary but not sufficient for outstanding professional performance.” Didactic Clinical Cognitive Yes No Non-Cognitive Maybe Achievement Orientation Trustworthiness Adaptability Conscientiousness Initiative Optimism Emotional Self-Control

49 Non Cogs: Implements Coarse Grained Rubrics can be implemented now!guides review of statements/letters/interviews can expedite the review process reduce implicit bias (expectations based on race/gender/name/culture); combats reviewer fatigue inter-rater reliability develop in conjunction with a social scientist F-V Program has a toolkit to help get started (workshop tomorrow at 9:30!)

50 utility in grad studies: not so novel[9] P. R. Sackett, N. Schmitt, J. E. Ellingson, and M. B. Kabin 2001, American Psychologist, 56, 302 [10] P. Kyllonen, A. M. Walters, and J. C. Kaufman 2005, Educational Assessment, 10, 153, [11] R. E. Ployhart and B. C. Holtz 2008, Personnel Psychology, 61, 153, [12] F. L. Oswald and L. M. Hough 2011, in APA Handbook of Industrial and Organizational Psychology, Vol 2: Selecting and Developing Members for the Organization (Washington, DC: American Psychological Association), p. 153, [13] A. Feingold 1994, Psychological Bulletin, 116, 429 [14] H. J. Foldes, E. E. Duehr, and D. S. Ones 2008, Personnel Psychology, 61, 579 [15] F. L. Schmidt and J. E. Hunter 1998, Psychological Bulletin, 124, 262 [16] T. Buyse and F. Lievens 2011, J. Dental Education, 75, 743, [17] F. Lievens and P. R. Sackett 2012, J. Applied Psychology, 97, 460 [18] R. J. Sternberg and W. M. Williams 1997, American Psychologist, 52, 630 [19] F. Lievens, D. S. Ones, and S. Dilchert 2009, Journal of Applied Psychology, 94, 1514 [20] M. M. Shultz and S. Zedeck 2012, Educational Psychologist, 47, 51 [21] K. Z. Victoro and R. E. Boyatzis 2013, J. Dental Ed., 77, 416, [22] R. Emmerling and R. E. Boyatzis 2012, Cross Cultural Management: An International Journal, 19, 4, [23] R. E. Boyatzis, E. C. Stubbs, and S. N. Taylor 2002, Academy of Management Learning & Education, 1, 150, [24] W. E. Sedlacek 2004, The Advisor: The Journal of the National Association of Advisors for the Health Professions, 2, 32, [25] W. E. Sedlacek 2004, Beyond the Big Test: Noncognitive Assessment in Higher Education (San Francisco: Jossey-Bass). [26] M. B. Wilson, W. E. Sedlacek, and B. L. Lowery 2014, Journal of Dental Education, 78, 567, [27] K. G. Stassun, S. Sturm, K. Holley-Bockelmann, A. Burger, D. J. Ernst, and D. Webb 2011, American Journal of Physics, 79, 374, [29] J. J. McHenry, L. M. Hough, J. L. Toquam, M. A. Hanson, and S. Ashworth 1990, Personnel Psychology, 43, 335 [30] A. E. Poropat 2009, Psychological Bulletin, 135, 322 [34] M. Walpole, N. Burton, K. Kanyi, and A. Jackenthal 2002, Selecting Successful Graduate Students: In-depth Interviews with GRE Users, GRE Board Research Rep. No R; ETS RR (Princeton, NJ: ETS) [35] M. K. Enright and D. Gitomer 1989, Toward a Description of Successful Graduate Students, GRE Board Research Report No R [36] M. R. Barrick and M. K. Mount 1991, Personnel Psychology, 44, 1, [37] K. S. Corker, F. L. Oswald, and M. B. Donnellan 2012, Journal of Personality, 80, 995, [38] P. Kyllonen 2013, (Princeton, NJ: Educational Testing Service), retrieved 16 October 2013,

51

52 The validity of the GRE tests, is limited to first year grades.Subject Test may tell you a little about preparation, convolved with R/E/G NOT about potential to become a PhD-level research scientist (the aim!)

53 Impact of Cut-offs: SelectivityAsian Am. White URMs Women Men 700 61.6% 52.8% 23.5% 37.5% 58.4%