Hiring Bias: What 25 Years of Fake Applications Show, and What Actually Reduces It

Hiring bias is the unequal treatment of equally qualified candidates because of who they are rather than what they can do, and it is one of the best-measured facts in social science. When researchers send matched fake applications to real job ads, white US applicants have received on average 36% more callbacks than identical African American applicants since 1989, with no improvement over 25 years. This post covers how hiring bias is measured, who it hits hardest, why the popular fixes often fail, and which changes to the decision itself have evidence behind them.

How researchers measure hiring bias

Hiring bias is measured with field experiments, not surveys. The standard design is the correspondence study, also called an audit study: researchers write pairs of CVs that are equal in every way that matters for the job, change one signal such as the name, age or a disability, send them to real vacancies and count who gets called back.

The design is blunt, which is its strength. Nobody is asked about their attitudes. Recruiters do not know they are in a study. The only thing that differs between the two applications is the trait being tested, so a gap in callbacks is the bias, in its plainest form.

The best-known example is Bertrand and Mullainathan's 2004 study in the American Economic Review. They answered help-wanted ads in Boston and Chicago with fictitious CVs carrying either distinctively white or distinctively African American names. White names received 50 percent more callbacks. The second finding is the one that should bother anyone who designs a hiring process: a stronger CV raised callbacks by 30 percent for white names and by far less for Black names. Being better on paper paid off unequally.

One study can be a fluke. The case rests on the meta-analyses, which pool dozens or hundreds of them.

Hiring bias by race has not moved in 25 years

The most cited meta-analysis says the racial callback gap in the United States is as large as it was a generation ago. Quillian, Pager, Hexel and Midtbøen (PNAS, 2017) collected every available field experiment on hiring discrimination against African Americans or Latinos: 28 studies, 55,842 applications, 26,326 positions.

Since 1989, white applicants received on average 36% more callbacks than African Americans and 24% more than Latinos. The authors found no change in discrimination against African Americans over the period, and only modest evidence of a decline for Latinos. The Northwestern release adds that the Latino trend rests on few studies, so it is not highly certain.

The picture is not uniquely American. A follow-up by Quillian and colleagues (Sociological Science, 2019) pooled 97 field experiments with more than 200,000 applications in nine countries in Europe and North America. They found significant discrimination against nonwhite natives in every country. In high-discrimination countries, white natives received nearly twice the callbacks of nonwhites; in low-discrimination countries, about 25 percent more. France had the highest rates, followed by Sweden.

Who is penalised beyond race

Race gets most of the research attention, but it is not where the largest callback penalties sit. The widest synthesis so far, by Lippens, Vermeiren and Baert (IZA Discussion Paper, 2021), pooled 306 correspondence experiments from 169 articles published between 2005 and 2020. Together they cover over 900,000 fictitious applications.

The pooled results, from the working paper version:

• Disability: about 44% less likely to get a positive response.
• Age: older candidates about 40% less likely.
• Physical appearance: less attractive candidates about 37% less likely.
• Race and ethnicity: a pooled ratio of 0.66, with Arab, Maghrebi and Middle Eastern applicants facing about 46% fewer positive responses, East and South-East Asian about 43%, and Black applicants about 36%.
• Gender: no overall penalty for women; after outliers were removed, a small advantage that faded in the more recent studies.

Two caveats matter. The variation between studies is very large, so a single average hides firms and countries that discriminate far more and some that do not at all. And on trend, the authors found a significant decrease only for ethnic discrimination in Europe, with no structural evidence of change for the other grounds.

The practical point for anyone who hires: if your fairness work covers race and gender only, it misses the candidates the experiments say are treated worst.

Discrimination concentrates in particular firms

Bias in hiring is not spread evenly; a minority of employers drive much of it. Kline, Rose and Walters (Quarterly Journal of Economics, 2022) sent more than 83,000 fictitious applications to jobs posted by 108 of the largest US employers.

On average, distinctively Black names reduced the probability of employer contact by 2.1 percentage points. The average hides the shape. Contact gaps were highly concentrated in particular companies, and the firms in the top fifth for racial discrimination were responsible for nearly half of the contacts Black applicants lost.

The same study, in its NBER version, reports what the lower-discrimination firms had in common. Company-specific racial gaps were smaller at more profitable firms, at federal contractors and at firms with more centralised recruiting. For gender there was no significant average gap, but some firms favoured men and others favoured women.

That last finding is the most useful one in this post. It suggests hiring bias lives in process: who screens, with what rules, and how much is left to one person's judgement on one CV.

Why the popular fixes for hiring bias disappoint

The two most common responses, bias training and anonymous CVs, have weak or even negative evidence. Both target the person rather than the decision.

Training. A network meta-analysis by Forscher and colleagues (Journal of Personality and Social Psychology, 2019) pooled 492 studies with 87,418 participants on procedures meant to change implicit bias. Implicit measures could be changed, but the effects were often weak (|d| < .30). The procedures produced trivial changes in behaviour, and changes in implicit measures did not mediate changes in behaviour. Most of the studies were brief lab interventions, so this is not the last word on long programmes. Field data are not more encouraging. In their HBR analysis, Frank Dobbin and Alexandra Kalev report that five years after firms introduced mandatory diversity training for managers, there was no improvement in management representation for white women, Black men or Hispanics, and the share of Black women in management fell by 9% on average. Voluntary training did better.

Anonymous CVs. Blind screening sounds like it should remove the signal the field experiments manipulate. In France, it did the opposite. In a randomised programme with the public employment service, studied by Behaghel, Crépon and Le Barbanchon, 1,005 firms volunteered and their vacancies were randomly assigned to anonymous or name-bearing CVs. According to J-PAL's summary, anonymisation widened the interview gap between majority and minority candidates from 2.4 to 13 percentage points. The explanation is uncomfortable: the volunteering firms were already more favourable to minority candidates, and once names and addresses were hidden, they could no longer read a gap in a CV in light of where the candidate came from. The government abandoned the policy in 2015.

The lesson is not that anonymity never helps. It is that removing information is a blunt tool, and its effect depends on who was making the decision before.

What reduces hiring bias: change the comparison

The interventions with the cleanest evidence change how a decision is made, not what people believe. The best experimental example is joint evaluation.

Bohnet, van Geen and Bazerman (Management Science, 2016) ran a hiring experiment at the Harvard Decision Science Laboratory. Evaluators chose candidates for a maths or a verbal task, knowing each candidate's past performance and sex. Some evaluators saw one candidate at a time; others saw two side by side.

Seen one at a time, candidates from the gender favoured by the stereotype were chosen with a probability of 0.65, against 0.49 for the others, and performance barely mattered: 62% chose a higher performer, 52% a lower one. Seen side by side, the gender gap disappeared, and evaluators became 46 percentage points more likely to pick the higher performer than the lower one. Gender was not correlated with performance in the task, so every stereotype-driven choice was simply a worse hire.

The mechanism is ordinary. A single candidate is judged against a vague mental picture of "a good engineer", and the picture carries the stereotype. Two candidates are judged against each other, on the evidence in front of you.

How to build a hiring process with less bias

Taken together, the studies point to one design principle: decide on evidence that is collected the same way for everyone and compared directly. In practice that means four things.

Fix the criteria before you see anyone. Write down what the job needs and how you will score it. This is the logic behind structured interview questions: every candidate gets the same questions and the same rating scale, so the score reflects the answer, not the rapport. The free interview scorecard template sets up the criteria, weights and rating scale in a few clicks.

Ask for evidence of the work. A CV is a list of signals, and signals are where the callback gaps appear. A work sample test asks candidates to do a slice of the job, which gives evaluators something to compare that is not a name, an address or a birth year.

Compare candidates side by side. Score answers question by question across the pool, not candidate by candidate. Promote from a shortlist, not from one name someone put forward.

Measure your own funnel. Kline and colleagues found bias concentrated in particular firms and smaller where recruiting was centralised. You cannot know which kind of firm you are without data: track pass-through rates at each stage by age band, disability disclosure and other groups you can lawfully record.

Automated screening needs the same audit, not an exemption. In a 2024 study by Wilson and Caliskan at the University of Washington, text-embedding models ranked over 500 real CVs against 500 job descriptions. They favoured white-associated names in 85.1% of cases and female-associated names in only 11.1%, and Black men were disadvantaged in up to 100% of cases. A model trained on past text can reproduce the gap the field experiments measure, at scale and faster.

This is also why the way you assess thinking matters. Tools that turn a candidate's reasoning into comparable metrics can support a side-by-side decision, as long as a person makes the call. NOOVA HR runs this as a 20-minute voice dialogue that produces Thinking metrics for each candidate, and a person always decides; there are no automated verdicts. Whatever you use, audit it like any other stage of the funnel. And if you want the cognitive side of why a single impression feels so convincing, our piece on metacognition covers how loosely confidence tracks accuracy.

A checklist for reducing hiring bias this quarter

1. Write the scoring criteria for your next open role before you read a single application.
2. Pull pass-through rates by stage and by age band from your last 12 months of hiring.
3. Replace one free-form interview round with the same questions and rating scale for every candidate.
4. Score interview answers question by question across all candidates instead of one candidate at a time.
5. Add a short work sample that every shortlisted candidate completes under the same conditions.
6. Ask any vendor of automated screening for an adverse impact audit by group before renewing.
7. Drop mandatory bias training as your main fix and spend the hours on process changes.
8. Re-run the funnel numbers after two hiring cycles and compare.

Sources

• Are Emily and Greg More Employable Than Lakisha and Jamal?, Bertrand and Mullainathan, American Economic Review, 2004
• Meta-analysis of field experiments shows no change in racial discrimination in hiring over time, Quillian, Pager, Hexel and Midtbøen, PNAS, 2017 (Russell Sage Foundation)
• Northwestern University news release on Quillian et al., EurekAlert, 2017
• Do Some Countries Discriminate More than Others? Evidence from 97 Field Experiments, Quillian et al., Sociological Science, 2019
• The State of Hiring Discrimination: A Meta-Analysis of (Almost) All Recent Correspondence Experiments, Lippens, Vermeiren and Baert, IZA DP 14966
• Systemic Discrimination Among Large U.S. Employers, Kline, Rose and Walters, Quarterly Journal of Economics, 2022
• Systemic Discrimination Among Large U.S. Employers, NBER Working Paper 29053
• A Meta-Analysis of Procedures to Change Implicit Measures, Forscher et al., Journal of Personality and Social Psychology, 2019
• Why Diversity Programs Fail, Dobbin and Kalev, Harvard Business Review, 2016
• Unintended Effects of Anonymous Resumes, Behaghel, Crépon and Le Barbanchon, IZA DP 8517
• Discrimination in Hiring and Anonymous CVs in France, J-PAL evaluation summary
• When Performance Trumps Gender Bias: Joint vs. Separate Evaluation, Bohnet, van Geen and Bazerman, Management Science, 2016
• Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval, Wilson and Caliskan, 2024