How to Audit AI Resume Screening Results: Monthly Steps, Sample Table and Human Review

To audit AI resume screening results, take a monthly sample of scored candidates from every score band, have a person review each CV blind to the score, record where the human and the tool disagree, find the cause of each disagreement, and fix the job requirements or process. Keep notes so you can show what you checked.

Why an audit is part of using screening at all

Buying an AI screening tool is not the end of the decision. The tool scores CVs against criteria, and criteria can be wrong, outdated or applied to people in ways you did not intend. An audit is a short, regular check that the scores still match what a careful person would conclude. Without it, small errors repeat across hundreds of candidates and nobody notices until a good candidate complains or a hire goes badly. The audit also gives you a record. If someone later asks how you screen, you can show what you checked and what you changed. Rules about automated tools in hiring differ by country and sometimes by city, and some require notices or bias checks, so check local law and ask a qualified adviser. This article is a practical method for small and mid-size teams, not legal advice. For background on how screening works, see /ai-resume-screening.

What you are actually checking

An audit asks four questions. Is the score accurate: does a high score mean a strong fit and a low score a weak one, when read by a person? Is it consistent: do two similar CVs get similar scores? Is it fair: do the results differ in ways that cannot be explained by the job requirements? And is it explainable: can you point to the lines in the CV and the requirements that produced the score? You are not trying to prove the tool perfect. You are looking for patterns of error and for requirements that do more harm than good, such as a must-have skill that nobody in the role actually needs, or a keyword that a good candidate would describe differently. The most common cause of bad results is not the software but the requirement list, so many fixes happen in the job setup, not in the tool.

Step 1: choose your sample

Audit a sample, not everything. Pull the candidates scored in the last month for one or two jobs, then sample from every band, not just the top. A good example for a job with 200 applicants is 10 from the top band, 10 from the middle, 10 from the bottom and 5 of those rejected early for any reason, which is 35 CVs. The bottom matters most, because that is where good candidates hide. Pick them randomly, for instance by taking every fifth candidate in each band. Record the job, the date range and how you chose the sample. If your volume is small, review every candidate in a short month. If it is large, rotate which jobs you audit so every active job is checked every quarter. Write the sample size in advance so you cannot quietly stop when the results look good.

Step 2: review blind to the score

Have a person read each sampled CV against the written job requirements without seeing the tool's score. Use a simple scale such as strong, possible, weak. Ideally the reviewer is not the person who set the requirements, because they may read the CV through their own expectations. Hide names if you can, since names can sway a reader, and judge only the evidence against the requirements. Record the rating and a one-line reason. Then reveal the tool's score and compare. Keeping the review blind matters. If you read the score first, you will tend to agree with it, and the audit will tell you nothing.

Step 3: record and classify disagreements

Put every sampled CV in a table. The table below shows an example with invented rows for a customer support role.

Count the agreements and disagreements. In this example, 3 of 6 agree. A real sample is larger, so look at rates, not single cases. Then group the disagreements by cause. Common causes are a keyword that appears without real experience, a skill described in unfamiliar words, an over-narrow must-have, a CV that is badly formatted or unreadable, and a requirement that does not reflect the actual job.

Step 4: look for patterns that could signal unfairness

Beyond accuracy, look at who is being scored low. You usually will not hold protected characteristics, and in many places you should not collect them casually, so use the information you legitimately have. Look at patterns such as career breaks, non-linear paths, unusual job titles, overseas qualifications, or CV formats that differ from the norm. If disagreements cluster on one kind of CV, the requirements or the reading of them may disadvantage that group. Rules about monitoring for bias, and what data you may collect to do it, differ by country, so check local law before you start collecting anything new. If you see a pattern, do not assume the tool is at fault yet. First check whether a requirement is a proxy for something that does not matter to the job, and consider removing it.

Step 5: fix the cause, then re-test

Match each fix to its cause.

After a change, re-score the same sample and compare the disagreement rate. If it did not improve, the cause was something else. Keep a change log with the date, what you changed and why. This is the record that proves the audit led to action.

Human review rules to write down

A tool should support a decision, not make it. Write your own rules and share them with everyone who screens. A person reads the CV before any candidate is rejected. Nobody is rejected on a score alone. Low scorers are spot-checked every month. Any candidate who asks for a review gets one from a person. Scores are never the only reason recorded for a decision, and the reason must refer to the job requirements. Recruiters can override a score and must note why. Keep these rules short enough to fit on one page. They protect candidates and they protect you, and they are easier to follow when they are simple and written.

Worked example: one month, one job

An example with invented numbers. A small team hires a sales coordinator and receives 180 applications. At month end the reviewer samples 30 CVs: 10 top, 10 middle and 10 bottom. Blind human ratings agree with the tool on 21 of 30 (70%). Of the 9 disagreements, 5 are strong CVs scored in the middle or bottom because the must-have list asked for a specific CRM, while candidates described similar tools. The team changes the CRM from must-have to nice-to-have, adds general wording about customer records, and re-scores the sample. Agreement rises to 26 of 30 (87%). They log the change and the numbers. The result was not a better tool but a better requirement list, which is the most common outcome of an audit.

What to record so the audit can be repeated

An audit is only useful if someone else could repeat it. Keep a one-page record for each review: the job, the date range, the number of candidates in the pool, how you chose the sample, who reviewed, what the requirements were at the time, the agreement rate, the causes you found and the changes you made. Store it with your hiring records, not in a personal note. The record should also state what you did not check, such as outcomes after hire, because honest limits are part of a credible audit. Keep it as short as you can. A record that takes five minutes to write will be written every month, while a long template tends to be skipped after the second month. If you hire in a place with specific rules on automated tools, ask an adviser which records you must keep and for how long, since this differs by country.

Metrics worth tracking month to month

Use few measures and keep them stable so you can see trends. The table shows examples with invented values.

The most telling number is missed strong CVs, because it shows who the tool may be hiding from you. A rising agreement rate after a requirements change is a good sign, while a flat or falling one means the cause was not found. Do not chase a perfect score; use the trend to decide where to look next.

Candidate complaints and review requests

Your audit should also cover the cases people raise themselves. Decide how a candidate can ask for a person to look again at their application, and who answers. Log each request, what was found and what you did. If several requests share a theme, such as a CV format that fails to read, treat it as an audit finding even if your sample missed it. Tell candidates in plain words that a person reviews applications and that scores are decision support. Being open about this builds trust and often prevents complaints. Check local law on what you must tell applicants about automated processing, as duties differ by country and sometimes by city.

A word-for-word monthly checklist

Print this and tick it off each month. Pick the job or jobs to audit and write the sample size. Pull candidates from the top, middle and bottom bands, and a few early rejections. Remove names where possible and hide the score. Have a reviewer rate each CV as strong, possible or weak against the written requirements, with a one-line reason. Reveal the scores and fill the table. Count agreements and calculate the agreement rate. Group each disagreement by cause. Decide on fixes, make them and re-score the same sample. Write the log entry with date, numbers and changes. Check the review requests received this month. Confirm that nobody was rejected on a score alone. Note anything you could not check. File the record where your hiring records are kept. The whole routine should take a couple of hours for a small team, which is why it can be repeated every month without becoming a burden.

Mistakes that make an audit worthless

The first mistake is auditing only the top of the list, which tells you whether the tool finds good people but not whether it hides them. The second is reading the score before the CV, which biases the reviewer. The third is letting the person who wrote the requirements judge alone, because they will read CVs through those requirements. The fourth is auditing once and never again, when requirements, applicant mix and the tool itself change over time. The fifth is fixing the tool settings without recording why, so nobody knows later which change helped. The sixth is treating agreement as proof of quality: a human and a tool can agree and both be wrong if the requirements are poor. Keep the human reviewer thinking about the job, not about matching the tool.

Where Rezumz fits

Rezumz is an AI hiring workspace for small and mid-size hiring teams. Its scoring compares a CV with the job's must-have and nice-to-have skills, with 80% of the score from the share of must-haves found and 20% from nice-to-haves. It shows the matched and missing requirements and quotes the CV lines that support them, so your reviewer can see what produced a score. It never rejects, emails or moves a candidate on its own, and it says a CV that does not mention a skill scores as if the candidate lacks it, so always read the CV before rejecting. If scores look wrong, the first thing to check is the skills list, which fits the audit method above. Rankings of interviewed candidates use anonymised written evidence. Account owners can see an audit log of who did what. Rezumz does not run a formal bias audit for you or certify compliance, so your sampling and records remain your job. It also does not suit large enterprises needing deep HRIS or payroll links. The free plan has 2 free candidate analyses; paid plans are $29 a month, $199 a year or $499 lifetime (limited), and Team is $79 a month or $699 a year for up to 10 teammates. Try /demo.

When an audit is not enough

A monthly sample catches drift and bad requirements, but it is not a legal compliance program. If you hire in a place with specific rules for automated employment tools, you may need notices, impact assessments or independent audits, and those need proper advice. If you receive very few applications, an audit of a handful of CVs tells you little and you may do better reading every application. And if people on your team do not read CVs before rejecting, fix that first. The audit only works when there is a real human review to compare against.