New Research Finds AI Hiring Systems More Prone to Bias Than Humans
New research presented at ICML suggests large language models can develop their own discriminatory stereotypes during simulated hiring tasks, often outperforming humans in their tendency to segregate candidates by group.
As companies increasingly automate recruitment, new research indicates that large language models (LLMs) may be developing their own discriminatory patterns independent of the data they were trained on. A study presented this July at the International Conference on Machine Learning (ICML) in Seoul found that models like ChatGPT, Claude, and Gemini can form stereotypes during simulated hiring tasks. According to MIT Tech Review, these systems were more likely to segregate candidates by demographic group than human participants in a similar psychology experiment, suggesting that the drive to optimize for successful outcomes can lead to unintended, biased decision-making.
Researchers from Princeton University and the University of Chicago tested the models using a simulated hiring game. The AI agents were tasked with filling 20 different job roles—ranging from doctors to janitors—using candidates from four fictional ethnic groups: Tufa, Aima, Reku, and Weki. While the models were instructed to maximize successful hires, they quickly began segregating candidates based on early, limited observations. For instance, if a model observed one failure from a specific group in a high-status role, it would begin steering that group toward lower-status roles, regardless of actual candidate capability.
The tendency to stereotype appeared to scale with model sophistication. On a segregation scale where 2.0 represents complete confinement of groups to specific job niches, human participants scored 0.84. In contrast, the models scored roughly 65% higher, with OpenAI’s o3 model reaching 1.83. Ryan Liu, a coauthor of the study and a PhD student at Princeton, noted that LLMs are optimized to generalize from limited data, an instinct that aids in solving logic puzzles but proves detrimental when applied to social decision-making. This 'exploration-exploitation' trade-off causes models to settle on biased hunches prematurely.
The study also examined potential mitigations for this behavior. Simply instructing a model to be fair proved largely ineffective, as the directive was often submerged by the system's primary goal of optimizing for successful hires. However, the researchers found that providing an explicit bonus for diverse hiring significantly reduced biased outcomes. Additionally, the models showed less tendency to segregate when provided with relevant personal data, such as education or age, rather than irrelevant traits like hair color. These findings suggest that the internal goals and the quality of input data are critical factors in how these systems form biases.
While the experiment provides a controlled environment, the real-world implications remain uncertain. Unlike the simulation, where models received immediate feedback on hiring success, real-world recruitment often lacks instant performance metrics. As noted by Angelina Wang, a computer scientist at Cornell University, the increasing use of personalization and memory features in chatbots may exacerbate these issues, as models risk over-indexing on past behaviors. The research highlights that without specific engineering to incorporate social values, AI hiring systems may inadvertently create novel, persistent biases that are difficult to detect or correct.
Sources
- 01 AI is more likely than humans to form biases when hiring — MIT Tech Review