How to Evaluate AI Resume Screening Tools: Questions to Ask Vendors

To evaluate an AI resume screening tool, ask the vendor how scores are produced and shown, test it yourself on ten real CVs you have already judged, check what it does about bias and audit records, confirm how candidate data is stored and deleted, and score every vendor on the same sheet.

Why evaluation matters more than the demo

Every AI screening demo looks good. The vendor picks the CVs, the job and the story, and the ranking always seems sensible. The risk starts after you buy, when the tool meets your messy CVs, your unusual roles and your legal duties. A screening tool touches decisions about people, so a poor choice can cost you good candidates, create unfair outcomes you cannot explain, or leave you with candidate data you cannot account for. The good news is that you do not need a data science team to evaluate one. You need a short list of questions, a small test you run yourself, and a scoring sheet that stops the loudest demo winning. This guide gives you all three. It is written for small and mid-size hiring teams who will make the choice themselves, so it avoids jargon and focuses on what you can check in an afternoon. For background on what the category does, see our overview at /ai-resume-screening, and for the wider tool landscape see /recruitment-software. Rules on automated decisions and candidate data differ by country, so treat the legal points here as prompts to check local law, not as advice.

Start with what you need the tool to do

Before you talk to any vendor, write down the job you are hiring the tool to do. Most teams want one or more of these: reduce the time spent reading CVs, rank a large batch so the best are reviewed first, apply the same criteria to everyone, keep a record of why someone was shortlisted, or surface people you would have missed. Each need points to different features. If your problem is volume, ranking speed matters. If your problem is consistency, you need visible criteria. If your problem is defending decisions, you need an audit trail. Also write down your numbers as examples to plan with: roles per year, CVs per role, number of people who will use it, and whether hiring managers need access. Finally, list your non-negotiables, such as data staying in a region, no automatic rejection, or export of all records. Vendors that cannot meet a non-negotiable drop out early, which saves you from wasting demo time. Keep this one-page brief and give the same brief to every vendor so their answers can be compared fairly.

Vendor questions on how the scoring works

These questions test whether the tool can explain itself. Ask each one and write the answer down. One: what exactly does the score represent, and what inputs does it use? Two: can I see, for each candidate, the lines of the CV that supported the score? Three: can I define the must-have criteria myself, or does the tool decide what matters? Four: does it handle CVs in different layouts, languages and file types, including scanned documents? Five: what happens when a CV cannot be read, and does the tool tell me? Six: does the same CV get the same score twice? Seven: how does the tool treat career gaps, career changes and non-traditional paths? Eight: does the score change if I edit the job description, and can I see why? Good answers are specific and show the feature on screen. Weak answers use phrases such as proprietary algorithm or our model just knows. If a vendor cannot show the evidence behind a ranking, you cannot check it, and you cannot explain it to a candidate, a manager or a regulator.

Vendor questions on bias, fairness and audit records

Bias cannot be fully removed from screening, whether by people or software, so the useful question is how the vendor handles it. Ask: does the tool use or infer protected characteristics such as age, gender, ethnicity, disability or marital status? Does it use names, photos, addresses or school names when scoring? Can I hide those fields during review? Has the vendor run any fairness testing, and will they share the method and results in writing? Is there a human step before any candidate is rejected, or can the tool reject automatically? Can I export a record of who was scored, with what criteria, and what decision a person then made? Be careful with confident claims such as bias-free or fully compliant. No honest vendor can promise that for every country and role. Some places have rules on automated decision tools, bias audits or candidate notices, and these differ by country and sometimes by city or state. Ask the vendor which rules they support and check local law or take advice for yours. The strongest design is one where the tool shows evidence and people decide.

Vendor questions on transparency and data handling

Ask where candidate data is stored, who can access it, and which sub-processors or outside AI services receive CV text. Ask whether CVs are used to train models, and whether you can opt out. Ask how long data is kept, how deletion works for one candidate and for a whole job, and whether deleted data is also removed from backups on a stated schedule. Ask how a candidate's access or deletion request would be handled, and how fast. Ask about encryption, role-based access, single sign-on, and whether every view or change is logged. Ask for the privacy policy and data processing terms before you sign, and read them. Ask what happens to your data if you cancel: can you export everything, and how soon is it deleted? Check the vendor's current documents rather than relying on a sales call. Obligations on candidate data differ by country, and you may need a lawful basis, a retention limit and a notice to candidates. If the vendor cannot answer in writing, treat that as a finding.

Run your own test with ten CVs

The test is simple and the most reliable thing you can do. Pick one real, closed role. Choose ten CVs you already know well: two clear strong fits, two clear misses, and six in the middle, including one with a career gap, one with a non-standard layout, one scanned or image-based, and one from someone whose background is unusual but relevant. Before you run anything, write your own rank order from one to ten and a one-line reason for each. Then load the same ten CVs and the same job into each tool, using its default settings, and record its ranking and its stated reasons. Compare. Look for strong fits placed low, clear misses placed high, and any CV the tool failed to read. Read the evidence it shows: is it quoting real lines from the CV, or vague statements? Run the same CVs a second time to check consistency. Then change one must-have and see whether the ranking moves in a sensible way. Ten CVs will not prove accuracy, but they will expose obvious problems fast, and they let you compare tools on your own material instead of the vendor's. See also /resume-parser for how reading and parsing affect results.

A scoring sheet you can copy

Score each vendor from zero to three on each line, where zero is no or cannot show, one is partly, two is yes, and three is yes and demonstrated on your CVs. Lines and weights are example weights, so adjust them to your needs.

Multiply each score by its weight and add up. The weights add to 26 and the top score is 3, so the maximum with these example weights is 78. Treat any zero on a weight-3 line as a veto to discuss, whatever the total. Fill the sheet in together with a colleague, each scoring alone first, then compare. Disagreements are useful because they show which answers were unclear. Keep the completed sheets, since they record why you chose as you did.

Worked example with example numbers

These numbers are examples only. A 40-person company hires about eight people a year and sees around 120 CVs per role. Three tools are shortlisted and each is tested on the same ten CVs. Tool A ranks seven of the ten in the same order as the team and shows quoted evidence for each, but it misses the scanned CV. Tool B ranks six in the same order, shows no evidence, and gives different scores when the CVs are run twice. Tool C ranks six in the same order, shows evidence, reads every file, and lets the team edit the must-haves. On the scoring sheet the totals come out as in the table.

Tool B is out because it fails the evidence line and gives different scores when the CVs are run twice. Tool A and Tool C are close, so the team asks Tool A how it handles scanned files and asks both for written data terms. Tool A's answer is a roadmap promise, while Tool C gives a clear deletion process. The team picks Tool C and starts on a monthly plan rather than a long contract, so it can review after two roles. The point is the method: one test, one sheet, written answers, and a small commitment first.

Common mistakes when buying

Mistake one: judging by the vendor's demo CVs. Always test your own. Mistake two: treating a high score as a decision. A score is a starting point for a person to check. Mistake three: letting the tool reject automatically without a human step. Mistake four: ignoring unreadable CVs. A CV the tool cannot read is silently a rejection if nobody notices. Mistake five: asking only about features and never about data. Mistake six: signing a long contract before running a real role. Mistake seven: believing claims of bias-free, 100 percent accurate or fully compliant. Mistake eight: not involving the hiring managers who must trust the shortlist. Mistake nine: forgetting the cost of change, such as moving data, retraining people and rewriting processes. Mistake ten: not setting a review date. Put a date in the diary to check, after a few real roles, whether the shortlists matched what the team would have chosen and whether any good candidate was missed. For rival comparisons, see /compare, and always check each competitor's current pricing and features on their own site.

Where Rezumz fits

Rezumz is a hiring workspace for small and mid-size hiring teams. It scores each CV against the must-haves you set for a job and shows the lines of the CV it relied on, so a person can check the reasoning. It also covers the apply page, shortlisting, interview scorecards, offers with approval and signing, and handing a finished hire onward. People make every decision, and Rezumz does not reject candidates for you. It suits small and mid-size teams, and the Team plan is aimed at teams of four or more recruiters. It does not suit large enterprises that need deep HRIS or payroll connections, and it does not do one-click multiposting to job boards. Prices: a free plan with 2 free candidate analyses, $29 a month, $199 a year, or a limited $499 lifetime option. Team is $79 a month or $699 a year for up to 10 teammates. See /pricing for current details, and use the ten-CV test in this article on the live demo at /demo to judge it against your own standards.

Who this is not for

If you hire fewer than a handful of people a year and read every CV yourself, you may not need a screening tool at all, and a simple spreadsheet or a free tracker, see /free-applicant-tracking-system, may be enough. If your main problem is not enough applicants, a screening tool will not help, because it ranks the people you already have. If your process is dominated by technical assessments, video interviews or multiposting, you may need different software. And if a vendor will not let you test on your own CVs, will not show evidence, or will not give data terms in writing, you should not buy from them whatever the price. Automated screening also does not remove your legal duties as the employer. You remain responsible for fair decisions and for how you treat candidate data, and the rules differ by country, so check local law before you rely on any tool for decisions about people.