After GRExit: Reducing Bias in Geoscience Graduate Admissions

Graduate admissions committees serve as gatekeepers, deciding who has access to higher education—and to a large extent, careers—in fields spanning the humanities and science, technology, engineering, and mathematics (STEM). These committees have long relied on Graduate Record Exam (GRE) results as a supposedly objective measure by which to compare and rank applicants. Recently, though, many graduate programs, including geoscience departments, have staged a “GRExit”; that is, they dropped the GRE as an application requirement because of its recognized biases and because of pandemic restrictions on access to the test. Amid this shift, programs are seeking to implement more holistic evaluations; however, there are not many examples within the geosciences to guide their practice.

The Geosciences graduate program at the University of Massachusetts Amherst (UMass-Amherst) piloted dropping its GRE requirement for graduate applicants in early 2018. In the first few admissions cycles after that decision, the admitted graduate students proved well-qualified and capable, but the admissions committee was relying on unstructured assessments that are themselves vulnerable to bias. For example, when two candidates appeared to be similarly strong, we might have distinguished between them on the basis of the reputations of their undergraduate schools (halo bias), their perceived love of the outdoors (affinity bias), slight differences in grade point average (GPA; ambiguity aversion), or the effectiveness of recommendation letter writers in championing their mentees (framing bias). Clearly, this approach was still far from ideal, and we found ourselves wondering how best to move on from dropping the GRE.

Evaluate the Person, Not Just Their Numbers

Holistic applicant reviews can reduce biases in graduate admissions by placing greater emphasis on skills and personal attributes than on quantitative metrics that may be influenced by disparities in applicants’ work conditions.

Holistic applicant reviews can reduce biases in graduate admissions by placing greater emphasis on skills and personal attributes—inherent qualities of candidates themselves—than on quantitative metrics, such as GPA, which may be influenced by disparities in applicants’ work conditions [Kent and McCarthy, 2016; Wilson et al., 2019]. However, many of the holistic admissions best practices applied in STEM were developed within other science disciplines that differ in critical ways from geosciences [e.g., Wilson et al., 2019].

First, the specific aspects of someone’s educational background that tend to make them a strong applicant (e.g., whether they’ve taken courses in mathematics) vary across the broad range of geoscience subdisciplines. Without consensus on what constitutes a strong applicant, geoscience admissions committees need a structured evaluation process to avoid falling back on criteria that perpetuate inequities, such as advantaging students from elite undergraduate programs or those with prior research experience [Posselt, 2016].

Second, compared with programs in other science fields, many graduate geoscience programs support a higher proportion of new students with research assistantships (RAs) as opposed to, say, teaching assistantships or fellowships. This reliance on RAs introduces additional sources of possible bias by putting admissions decisions largely into the hands of faculty principal investigators (PIs), who select students for specific projects, rather than with a centralized admissions committee [Kent and McCarthy, 2016].

The UMass-Amherst Geosciences graduate program recently adopted a holistic review process that excludes the GRE, gathers responses to customized supplemental prompts, and relies on multiple rubric assessments to judge the likelihood that applicants will be successful in our program through more equitable evaluations (Figure 1). The limited capacity of faculty to take on new advisees and the fact that our program does not admit a student without the approval of at least one potential faculty adviser mean, however, that some excellent graduate applicants are not admitted. Our primary goal with our new process, which we detail below and which may serve as a useful model for other programs, is thus not to achieve full equity, but to reduce biases in admissions decisions as much as possible.

Moving Toward Holistic Admissions

Prior to 2018, the graduate admissions process in the UMass-Amherst Geosciences program was similar to that of many other programs. Our admissions committee used a combination of GRE scores, GPA, a personal statement, recommendation letters and accompanying rankings (called “referee letters” and “referee rankings” in Figure 2), and input from potential advisers to evaluate each applicant. Advisers had access to all applicant materials and sometimes interviewed applicants as well.

This committee, including individuals representing different research areas in our program, engaged in a discussion process that as described by Posselt [2016], is common in nonconsensus disciplines where there isn’t consistent agreement on the required training for strong candidates. In our effort to reach agreeable outcomes, our admissions committee was vulnerable to making decisions for the sake of conformity and to implicit biases because we didn’t have a structure for making decisions. Additionally, our process relied heavily on candidate rankings by potential advisers, but individual approaches to these rankings varied widely across the program.

The admissions committee took a first step toward holistic admissions in winter 2020 by developing a list of traits that correlate with success in our graduate program.

Because dropping the GRE did not address the biases in our process, the admissions committee took a first step toward holistic admissions in winter 2020 by developing a list of traits that correlate with success in our graduate program. Focusing on applicants’ traits rather than experiences can reduce admissions bias [e.g., Megginson, 2009] and can create a consistent evaluation approach for nonconsensus disciplines like the geosciences [e.g., Posselt, 2016].

We initially compiled a list of traits that graduate schools and employers typically ask referees to address in their assessments of candidates. We then narrowed this list to six traits that most closely correlate with student success across all research groups in our program: perseverance, independence, curiosity, ability to work in teams, maturity, and communication skills.

For our admissions process to be effective in helping the committee to select the applicants most likely to thrive in our graduate program, application materials must reveal information about each of the six traits for success. However, when the admissions committee mapped the effectiveness of each application component in informing each trait (Figure 2), they found that these tools were not reliable indicators of potential success in our program. This finding confirmed what some of our graduate students had noted in spring 2019 (Figure 1).

The personal statement offered an opportunity for applicants to demonstrate this potential, but in practice, the degree to which the statements actually demonstrated potential for success varied widely. This observation suggested that applicants had varying access to mentoring about which strengths they should highlight in their personal statements. After discussing this mapping and the apparent ineffectiveness of the existing admissions process, the faculty agreed that we needed a new approach. Having this broad agreement was critical for instituting change in our program because the new approach would require greater effort from each faculty adviser.

A Change of Approach

One effective strategy for gathering information on traits for success in graduate school is to require supplementary application materials that specifically demonstrate the assessed traits. In our case, a four-person committee developed a supplemental form consisting of several open-ended prompts that map to desired traits. The committee included one current graduate student, whose perspective on how the six designated traits related to their own success in graduate school was critical for crafting supplemental prompts that would be both relatable to applicants and revealing to the admissions committee.

We focused on process-based questions, rather than outcome-based questions that would advantage applicants with particular achievements.

We focused on process-based questions, rather than outcome-based questions that would advantage applicants with particular achievements. For example, rather than asking about research experience, which is not available to all undergraduates, we ask candidates to describe their independence, curiosity, and problem-solving skills. Candidates can draw from any previous experience in answering this prompt, including a research project. We also took care to avoid leading questions. For example, our committee recognized that working successfully in teams does not necessarily mean being a leader. Asking applicants to describe their past success in any of a range of team roles is more inclusive of different personalities and encourages applicants to speak to their strengths.

Each prompt explains how the traits we aim to evaluate contribute to success in graduate school (Table 1). These explanations add transparency to how applications will be considered and reduces disparities among students with variable access to effective mentors. Applicants can see the prompts in advance of applying on our graduate program’s web page, and we suggest that they share the prompts with the people writing their letters of recommendation.

Table 1. UMass-Amherst Geosciences Supplemental Prompts for Graduate Admissions

Prompt	Explanation
1. What motivates you to apply to our program?	Graduate school is a multiyear commitment to learning and to a research project. The decision to attend graduate school is a major step in your career. Please tell us what motivates you to go to graduate school and why you think the Geosciences Department at UMass-Amherst is where you would like to go?
2. Tell us about your perseverance and maturity	Completing graduate-level coursework, independent study, and research definitely requires perseverance in handling deadlines, challenges, and even failures along the way. Briefly describe a challenging time (one where you did not fully succeed in achieving a goal or completing a task) and describe how you responded to that and what you learned from that experience. Then, briefly describe an experience where you struggled through a challenging time but eventually succeeded. What helped you get through that struggle and the stress associated with it?
3. Tell us about how you work in teams	As a graduate student, your research will often involve working as a team with your adviser and other collaborators. Think about a past experience being a part of a team (academic or extracurricular), and please answer the following questions: ● What role(s) did you play on the team (leader, synthesizer, heavy lifter, etc.) that contributed to its success? ● What did you learn about successful teamwork and/or team culture through the experience?
4. Tell us about your independence, curiosity, and problem-solving skills	Conducting research as a graduate student requires curiosity in developing a question or hypothesis and independence along with creative problem-solving in conducting self-directed research. Please describe a time when you took initiative on something, either academic or outside of academics, that highlights some of these qualities.

Following holistic admissions best practices [Michel et al., 2019], we use standardized rubrics that employ evidence-based scoring (on a scale of 1–3) to evaluate applicants and track how specific components of the application reveal each trait (Table 2). Each score must be substantiated with evidence from some part of the application, and we provide examples to guide this process. The faculty in our program requested that in addition to scoring for the six designated traits, the rubric include a score for motivation and alignment to the research program, which is typically assessed through personal interviews.

Table 2. UMass-Amherst Geosciences Admissions Rubric^a

Trait	Score	Comments and Evidence for Assessment
Perseverance	1–3	Applicant worked 20 hours per week while carrying full course load; double major with mathematics; applicant describes learning they were being paid less compared to similarly qualified employees, bringing up the issue with their manager, and ultimately receiving a promotion and salary raise—the applicant now is involved with hiring discussions in same company
Independence	1–3	Learned Matlab on their own; developed the project ideas; first person in family to go to college and move out of state
Curiosity	1–3	Wide range of previous research; reference letter says “likes the research journey”; applicant describes working on a project for a client, proactively identifying and investigating an environmental impact to regional soil contamination, and, ultimately, having the issue and their proposed solution accepted at the state level
Ability to work in teams	1–3	Reference letters says “positive influence in any group”; applicant organized a community theater production that more than 600 people attended; applicant created campus club to mentor low-income and underrepresented students
Maturity	1–3	Reference letter says “positive outlook in stressful environments”; applicant describes struggle with anxiety and depression, their journey to seek and accept help, and their ultimate success
Communication skills (written and oral)	1–3	Reference letter says “first draft read like a published paper”; reference letter describes inspirational speech applicant gave as part of a campus environmental advocacy event; applicant gave a talk at AGU as an undergraduate
Motivation for and alignment to research program	1–3	Has skills that map well to research groups future directions; expressed strong interest in ongoing research; applicant has background in physics and strong coding skills that will be helpful for the physical oceanography modeling project they wish to pursue.
Composite score	7–21
Additional considerations not included above?		Applicant has worked in environmental consulting industry for past 8 years; has well-thought reasons for pursuing a Ph.D.

^aAssessment scores are as follows: high = 3, good/average = 2, weak = 1.

Most holistic admissions processes depend on a central admissions committee trained in bias vulnerabilities to make equitable admissions decisions [Michel et al., 2019]. Because half of the graduate students in our program are supported by PI-funded RAs for particular research projects, we need a structure for including faculty advisers in the admissions process. Consequently, both our admissions committee and potential advisers complete rubrics for each student, and composite scores from each assessor are considered equally.

Advisers have been encouraged to watch a 15-minute video that provides training on how to recognize bias in admissions, assess applicant traits, and use the rubrics. Advisers are also provided with effective interview questions, which were compiled by the committee that developed the supplemental prompts. In nearly all cases, the variance among composite rubric scores from different assessors was within 1 point, which speaks to the strength of this approach for guiding admissions decisions. In the very rare cases when scores differ more substantially, the evidence-based structure of the rubric provides valuable insights that help the admissions committee chair understand and resolve the conflicting scores.

A Clearer View of the Candidates

Several faculty members have remarked that the process helped them recognize strengths of candidates who might not have been ranked as highly in our former process because of unseen biases.

Our new admissions process means that an adviser can no longer appeal to how well they think an applicant would fit in their specific research group as the sole argument for admission, but rather must demonstrate the applicant’s broader potential for success. Employing this process has required widespread buy-in from our faculty to the notion that it serves the best interests of their research groups.

Having used the rubrics for multiple admissions cycles now, several faculty members have remarked that the process helped them recognize strengths of candidates who might not have been ranked as highly in our former process because of unseen biases. For example, the supplemental prompts have revealed strengths, such as the ability to manage two jobs simultaneously, that likely would not have emerged through a traditional personal statement focused on a student’s research interests.

Another advantage of the new process is that all applications are read carefully, not just those of applicants who contact faculty directly. Faculty have reported that although the holistic process takes more effort, it has improved their ability to assess candidates compared with our previous approach. And they have strongly supported continuing this new practice. Going forward, committees of faculty and graduate students will periodically review and update our admissions process with attention to how the demographics and retention of our graduate students evolve.

We encourage geoscience graduate programs who haven’t done so already to consider employing a structured admissions process that uses supplemental prompts and rubrics, such as those shared here. These tools can and should be adjusted to best suit the goals and constraints of a given program.

In our program, we have seen that a holistic admissions process can open doors for qualified applicants who might otherwise be overlooked. If enough programs pursue a similar path, we can substantially reduce bias in admissions decisions and foster greater diversity in the next generation of geoscientists.

Acknowledgments

All UMass Geosciences program faculty contributed to the transition to holistic admissions. The authors thank Forrest Bowlick, Haiying Gao, Ray Bradley, Stephen Burns, Mike Williams, Isaac Larsen, and Steve Turner for serving on the various committees that instituted these changes.

References

Kent, J. D., and M. T. McCarthy (2016), Holistic review in graduate admissions, Counc. of Grad. Sch., Washington, D.C.

Megginson, L. (2009), Noncognitive constructs in graduate admissions: An integrative review of available instruments, Nurse Educ., 34(6), 254–261, https://doi.org/10.1097/NNE.0b013e3181bc7465.

Michel, R. S., et al. (2019), Graduate admissions practices: A targeted review of the literature, ETS Res. Rep. Ser., 2019(1), 1–18, https://doi.org/10.1002/ets2.12271.

Posselt, J. R. (2016), Inside Graduate Admissions, Harvard Univ. Press, Cambridge, Mass., https://doi.org/10.4159/9780674915640.

Wilson, M. A., et al. (2019), A model for holistic review in graduate admissions that decouples the GRE from race, ethnicity, and gender, CBE Life Sci. Educ., 18(1), https://doi.org/10.1187/cbe.18-06-0103.

Author Information

Michele L. Cooke (cooke@umass.edu), Hannah Baranes, Isla S. Castañeda, Jonathan D. Woodruff, and David F. Boutt, University of Massachusetts Amherst

Citation: Cooke, M. L., H. Baranes, I. S. Castañeda, J. D. Woodruff, and D. F. Boutt (2022), After GRExit: Reducing bias in geoscience graduate admissions, Eos, 103, https://doi.org/10.1029/2022EO220285. Published on 17 June 2022.

This article does not represent the opinion of AGU, Eos, or any of its affiliates. It is solely the opinion of the author.

Text © 2022. The authors. CC BY-NC-ND 3.0
Except where otherwise noted, images are subject to copyright. Any reuse without express permission from the copyright owner is prohibited.