High school seniors have been busy finalizing their college admission applications amidst an intense national conversation about what use of artificial intelligence (AI) is advisable, permissible, responsible, and detectable. Once they press submit on their application, they will turn their attention to wondering what is happening inside the “black box” of the admission office.
This year, that mystery is more nuanced than ever—some are wondering if humans or machines are deciding their fate. Ultimately, it is not a question of if artificial intelligence will be used by admission offices. The truth is that enrollment managers were employing AI in their work long before the ChatGPT boom last fall. The question is rather one of how much and in what ways this technology will be used.
What’s the Buzz?
While there is no shortage of articles about students using AI to write college essays, some recent pieces have focused on how colleges will use tools like ChatGPT. In The Chronicle of Higher Education, Taylor Swaak explored the use of AI in admission and whether it is the answer to “administrative drudgery,” but also in recruiting, predictive modeling, and application scanning and review. And while she noted that the sources she spoke with are proceeding with caution, they are proceeding.
Liam Knox at Inside Higher Ed reported on a survey by Intelligent, an online education magazine targeting college applicants. Among Intelligent’s key findings were that “Half of educational admissions departments currently use AI and 82% will by 2024” and the “Majority of schools using AI will allow it to have the final say on applicants.” The report suggests that admission offices are using AI in every way, from detection software to verify the authenticity of essays, to recommendation and transcript scanning programs.
Before you throw your hands in the air, keep in mind that the survey only had 399 respondents and 20% were not college admission officers. The findings were met with skepticism from admission leaders. Andy Borst, vice president for enrollment management at the University of Georgia reassures applicants and counselors, writing on X/Twitter, “There is a zero percent chance admissions offices are using AI to read files in the way this internet survey and article imply. Yes, Harvard is still reading students' essays. No, they are NOT using AI technology to skim and score your letters of recommendation. Yes, many colleges are using a formula based on historical data (usually GPA and test score) to ‘finalize admissions decisions’.” He added, “I will acknowledge that AI has the POTENTIAL to aid in the organization and standardization of admissions information. No one is there yet…students, parents, and those in higher ed should not be concerned about AI replacing caring humans in the admissions process. College admissions is flawed, but it also has not been turned over to AI.”
MORE FROM FORBES ADVISORTell Me What’s A-Happening
Talk to any college admission leader and you will undoubtedly hear the refrain, “It’s a human process.” They are attempting to dismiss the perception that it is all numbers-driven and impersonal. It also accentuates the reality that it is an imperfect system, with humans guiding it, making human decisions in as comprehensive a way as possible. In holistic admission, reviewers look beyond test scores and grades, incorporating a bevy of other factors—involvement, leadership, background, context, essays, interviews, recommendations, and more. Ideally, an institution’s mission dictates the aspects of an application that will influence a decision. It is a human process because educators decide what will “count” and how much. Whether using a rubric or another paradigm for review, they have also traditionally been responsible for assessing each part of the application personally and acutely. But is this sustainable?
While overall college enrollment is declining, at some schools application numbers continue to swell. Between 2000 and 2020 the number of students applying to 7 or more colleges increased by 184% from 13% to 37%. Some large universities, like New York University and The University of California, saw applications for fall 2023 admission surge well over 100,000.
Meanwhile, enrollment leaders are maxed out and stressed out. Due to the “Great Resignation,” admission offices, like many student affairs departments, are hopelessly understaffed. Admission leaders report record vacancies in their staff, some operating (barely) with limited capacity. It begs for an innovative solution. Colleges and universities are increasingly turning to part-time readers who review applications because full-time staff simply cannot handle the volume. Some universities are enlisting over 200 seasonal readers, having to train, manage, and pay this army of reinforcements. Is this the right approach, or is there a better, more fiscally responsible way? Perhaps the artificial admission officer is the answer to enrollment managers’ prayers.
You Cannot Predict the Weather
If you had your choice, who would evaluate your college application? Would you prefer an impartial reader with years of extensive training by a team of seasoned professionals and a historical perspective on which students are most successful? Or would you rather have a new application reader, trained for a few weeks, who may not have had their first cup of coffee, on a rainy day, after an argument with a family member? If that first impartial reader isn’t human, does it change your answer?
What if I told you about research from The University of Pennsylvania’s Uri Simonsohn showing the influence of weather on decision-making? In his Journal of Behavioral Decision Making paper, “Clouds Make Nerds Look Good: Field Evidence of the Impact of Incidental Factors on Decision Making”, Simonsohn shared his analysis of admission decisions, finding that “applicants' academic attributes are weighted more heavily on cloudier days and non-academic attributes on sunnier days.” The reality is that humans are emotional beings and despite significant anti-bias and other training for admission readers, they are not immune to human nature, or evidently the weather.
RoBERTa: An Admission Superstar or Gone Too Far?
A new study in the journal Science Advances suggests the future of AI admission officers might not be too far off. Researchers from The University of Colorado-Boulder and The University of Pennsylvania used artificial intelligence to assess personal qualities in college admission and concluded that “an AI approach to measuring personal qualities warrants both optimism and caution.”
The study's authors emphasize that “the holistic assessment of personal qualities in college admissions is opaque and resource intensive,” calling out the “secretive” nature of admission decisions in holistic review. Meanwhile, they cite longitudinal research supporting the use of non-cognitive skills, or character, in predicting college success and life outcomes, as well as the role of holistic admission as potentially advancing equity in admission. They point out that “with stunning efficiency, AI systems identify patterns in data and, with stunning fidelity, apply learned models to new cases.” The authors write, “For example, a computer algorithm could be trained to generate personal quality scores from student writing instantaneously, reliably, and at near-zero marginal cost.”
This is what they endeavored to do in their study. The researchers used “human-centered,” supervised machine learning to measure personal qualities in students' descriptions of their extracurriculars. They trained research assistants to code for personal qualities and recruited college admissions officers with professional expertise to do the same. Researchers trained the model by “identifying the presence or absence of seven different personal qualities commonly valued by universities and shown in prior research to predict college success.” These included attributes like prosocial purpose, intrinsic motivation, teamwork, and perseverance.
Research assistant and admissions officer ratings were then used to fine-tune RoBERTa: a Robustly Optimized BERT (Bidirectional Encoder Representations from Transformers) Pretraining Approach. For those whose computer science knowledge reflects my own, essentially this is a Large Language Model (LLM), or neural network, that has been trained on an immense amount of data. The researchers employed RoBERTa to review over 300,000 short essays. Here are some of their findings:
Benjamin Lira, one of the researchers from The University of Pennsylvania writes, "We found that the fine-tuned models could faithfully reproduce how human admission officers and research assistants rated applicants’ essays and predict whether an applicant would graduate college six years later as effectively as the human ratings.” He says, “Additionally, there was no evidence of bias in the ratings. This was evident in three different ways. First, the models matched the humans equally well across demographic groups. Second, the models gave similar scores to applicants from different backgrounds. Aside from female applicants being rated as more prosocial (consistent with decades of prior research), there were no advantages or disadvantages associated with any demographic characteristic. Finally, obtaining high personal quality scores from the models was equally predictive of graduation, regardless of the applicant’s background."
The authors conclude: “We recommend AI be used to augment, not replace, human judgment. No algorithm can decide what the goals of a university’s admissions process should be or what personal qualities matter most for that community.”
RoBERTa on Trial
Some worry that the use of AI in admission is a threat that has the potential to spiral out of control. Others reassure us that we need not be preoccupied by the future of these tools, but rather remain hopeful about their potential to simplify admission. Does relying on AI betray the humanity of admission or will it amplify the ability to stay true to institutional mission and priorities?
Matthew DeGreeff, dean of college counseling and student enrichment at Middlesex School says, “I don't know how the humanity and essence of an applicant, the nuances and character of a high school, and the complexities and context of a life lived in endlessly different communities can be measured by an algorithm.” He adds, “I can't imagine selecting a class of students getting more soulless and more separated from welcome human beings to a community of learners, educators, artists, athletes, researchers, and leaders.”
Bob Massa, vice president emeritus for enrollment and college relations at Dickinson College says “AI can indeed identify personal qualities and student character attributes through college essays and recommendations, and if trained properly, can even discern how strong a personal quality is (e.g. 4 years of volunteer work vs. one year). But the use of AI should be to supplement rather than to replace human evaluation.” He adds, “It could make our work more accurate, picking up some items that we might miss, but the opposite is also true, which is why an admission officer must always validate the work of an admission bot.”
Tom Bear, vice president of enrollment management at Rose-Hulman Institute of Technology agrees, saying, “AI could give a first read that would then allow the second and/or third reader greater efficiency. The readers would be alerted to possible character traits presented through the student’s application. The readers would then be responsible for discerning if those traits associate back to their respective college’s mission, vision, history, culture, etc.” He says, “I see the potential, but I am hesitant to lean entirely on AI for such decisions.”
Angel Pérez, CEO of the National Association for College Admission Counseling (NACAC) reflects that “the research effectively captures the essence of what the admission profession is currently trying to gauge: the potential for administrative and qualitative improvements to the application review process versus the as-yet-unknown and unanticipated effects on equity in the college admission process.” He says, “Our job as an association is to collect evidence from the field, both research and practice, to share information about how institutions are benefiting from this technology while also making progress on our shared commitment to equity.” Pérez adds, “This study provides us with some hope that, in a future in which AI will become more integrated with the college admission process, we can utilize technology to glean more about each student’s strengths than our current and past methods have allowed.” He emphasizes that the “study also confirms what we have heard from our member institutions, which is that AI is often only as good as the inputs that direct it. Ensuring that admission offices adopt a ‘do no harm’ approach to incorporating AI, particularly with respect to equity, will be a top priority moving forward.”
Everything's Alright
Kevin Roose is a technology columnist for The New York Times, host of the Hard Fork Podcast, and a New York Times bestselling author of three books, including his latest, Futureproof: 9 Rules for Humans in the Age of Automation. He says that we need not compete with machines but make ourselves more human. As the demands on enrollment managers increase and they face the pressure of scale, leaders are looking for ways to be more human. AI is not going to replace legendary educational leaders like Ted Spencer, former associate vice provost and executive director of admission at The University of Michigan. But if trained properly and developed ethically, AI has the potential to support his successors in doing their jobs effectively, sustainably, and equitably. Imagine if Spencer and other thoughtful and seasoned professionals collaborated to train these models based on intentional and equitable rubrics. Not only would we capture decades of experience but we would also exponentially increase capacity.
An AI admission officer is not a messiah who will save the application review process from all its challenges and nuances. But before we are quick to judge models like RoBERTa, we need to consider them in the context and limitations of the current review process, staffing, and other demands. If RoBERTa and other artificially intelligent admission officers can free their human colleagues to have more meaningful, and less transactional, relationships with applicants, then let’s start training them!