How to Calibrate Interviewers Before a Hiring Round (45-Minute Plan)

To calibrate interviewers, hold a 45-minute session before the first interview. Agree what the role needs, read the interview evaluation form and its rating anchors together, score one sample answer alone, compare the scores, discuss every gap of two points or more, and write down any rule you change.

What calibration means and why it is worth 45 minutes

Calibration means getting the people who will interview to use the same ruler. Two interviewers can hear the same answer and give it a 2 and a 5, not because one is careless but because they carry different standards, different pasts and different ideas of what good looks like. One has just come from a team of experts and finds everyone average. Another has been burned by a weak hire and now marks hard. Without a shared ruler, a candidate's result depends partly on who they happened to meet, and your data cannot be compared across interviewers. A short session before the round fixes much of this. It does not make people identical, and it should not. It narrows the gap, makes the remaining differences visible and gives everyone a common language. Forty-five minutes is enough for a small hiring round of one role and three to five interviewers. It costs far less than the time lost in debriefs where people argue about what a 3 means. This guide gives you the full plan, with timings, a practice exercise and a worked example. For the forms you will calibrate, see /interview-feedback-form.

Before the session: what to prepare

Prepare five things, and send them out a day ahead so the session can start on time. One: the job description and a one-paragraph summary of what success looks like at six months, which you can build with our /job-description-template. Two: the interview scorecard or evaluation form, with five to seven competencies and a 1 to 5 scale with anchors. Three: the interview plan, saying who covers which competency. Four: a sample answer for practice. You can write one yourself, take a real anonymised answer from a past hire with permission or record a short mock. Make it deliberately middling, with real strengths and real gaps, since clear cases do not teach much. Five: a blank grid for scores. Ask each interviewer to read the form beforehand and bring questions. Also decide who leads the session. A recruiter or hiring manager works well, and the leader should say little during scoring and more during discussion. Tell the group the aim up front: not to find the right score, but to understand why people differ.

The 45-minute agenda

Here is a plan you can follow as written. Timings are a guide and you can trim for smaller groups.

Keep the clock visible. The compare and discuss step is the heart of the session, so protect it. If the group is still debating at minute 32, move on and note the open question, since you will learn more from live interviews and a short follow-up than from a longer debate about an imaginary answer.

The practice scoring exercise

The exercise is simple. Everyone reads the same sample answer and scores it on the same competency, alone and in silence, with one line of evidence. Do not allow talking, and do not reveal your own score first. Here is a sample you can use for a customer success role, rated on the competency Handling a difficult customer.

Sample answer: A customer was angry that a feature was missing. I apologised, told them it was on the roadmap and that I would check back. I did follow up two weeks later, but the feature was not shipped yet, so I just told them again. They stayed with us. I think it was because I was friendly.

Is that a 2, a 3 or a 4? Reasonable people will differ. Some will credit the follow-up, some will mark down for no real solution and no honest assessment of why the customer stayed. That difference is the lesson. After scoring, collect the numbers before anyone speaks. Then use the grid in the next section. Repeat with a second sample if time allows, preferably one where the answer is clearly strong or clearly weak, to test whether the group agrees at the ends of the scale.

Compare scores and discuss the gaps

Write the scores in a grid. Here is an example with invented numbers for four interviewers and two samples.

Focus on the spread. For each gap of two or more, ask the highest and lowest scorers to read their evidence lines aloud, and ask only one question: what did you hear that made you choose that number? Often you will find one of three causes. The first is different standards, such as one person weighting the follow-up and another weighting the result. The second is different evidence, where one person noticed a detail the other missed. The third is an unclear anchor, where the form itself allows both readings. Fix the cause. Edit the anchor, agree which evidence counts, or note that a particular type of answer should score at a set level. Do not ask anyone to simply change their number to match the group. The aim is understanding, and forced agreement hides real differences that will reappear in live interviews.

Agree rules before the interviews begin

Use the last part of the session to write down the rules. Keep them short, and put them on the form or in a shared note. Typical rules: score only what you saw in the interview, with one line of evidence per score; submit your form before the debrief and do not discuss candidates until you have; a 3 meets the bar and is a good score; no half points; a recommendation uses four levels with no middle option; no hire if a must-have competency scores 1. Add a rule on notes: record facts and quotes, not guesses about age, family plans, health, religion or origin, since rules on what may be asked and recorded differ by country and you should check local law. Add a rule on bias controls such as a standard order of speaking in the debrief, with the most junior first. Finally, agree what happens if two interviewers disagree sharply on a live candidate: ask each to read the evidence, and if it remains unclear, add a short follow-up with a third person rather than averaging. Rules that everyone helped to write are followed far more often than rules that arrive by email.

A worked example of a first session

Here is an invented example of how one small team ran it. A 15-person software company hired two support engineers and gathered the recruiter, the support lead and two engineers. In the practice exercise, the answer described debugging a customer's integration problem. The engineers scored the technical competency 4 and 5, because the candidate named the right tool. The support lead scored 2, because the candidate never asked what the customer was trying to achieve. The recruiter scored 3. In the discussion, the group saw two causes. Engineers rewarded the correct tool, while the support lead rewarded understanding the customer. Neither was wrong, but the anchors did not say which mattered more. They edited the anchor for a 4 to read: identifies the cause and confirms the customer's goal before proposing a fix. They also agreed that technical accuracy and customer focus would be scored on separate lines, not blended. In the live round that followed, the spread on the technical line dropped, and the support lead said the debrief was shorter because they were arguing about evidence, not about meaning. The numbers here are illustrative, but the pattern is common.

Common mistakes in calibration

Avoid these pitfalls, which can make the session worse than none.

A specific caution: do not use calibration to push everyone toward the most senior person's taste. The ruler should come from the role's needs, written in the anchors, and not from whoever speaks last.

A checklist to score your calibration session

After the session, rate it on each line from zero to two. Twelve or above means the group is ready to start.

After the first few live interviews, run a ten-minute check. Look at each interviewer's average scores. If one person gives mostly 5s or mostly 2s, have a quiet conversation about the evidence behind them, not about their character. Some people are simply stricter, and knowing this lets you adjust how you read their forms.

When calibration is not worth it

A full session is not for every situation. If you are hiring one person and only one interviewer will meet them, there is nobody to align with, so spend the time on the form. If the interviewers have worked together on many rounds with the same form and the scores have stayed close, a five-minute refresher is enough. If you are in an emergency hire where speed is the main constraint, a short written note of the rules may be all you can manage. And calibration is no substitute for a good interview plan. If questions are random, a session will not rescue the data. Also be careful with sample answers drawn from real candidates. Remove names and details, get permission where needed and follow your local data rules, which differ by country. When in doubt, write an invented sample, which is safer and easier to control.

Where Rezumz fits

Rezumz is an AI hiring workspace for small and mid-size hiring teams. It does not run a calibration session for you, but it supports the parts that make one stick. On each job you set the interview scorecard once, with up to 10 items rated 1 to 5 and an optional note on what a strong answer shows, which is a good place to store the anchors you agree. Every new interview link starts from it, so everyone uses the same form. Each interviewer has a private link with a one-time email code, so scores are submitted independently, and submitted feedback is locked, with changes saved as versions and a reason. Rezumz flags feedback that looks copied between interviewers, which can show whether calibration is working or people are drifting together. People make every hiring decision. It suits small and mid-size teams, and the Team plan is for 4 or more recruiters. It does not suit large enterprises needing deep HRIS or payroll connections, and it does not do one-click multiposting to job boards. Prices: a free plan with 2 free candidate analyses, $29 a month, $199 a year, or a limited $499 lifetime option. Team is $79 a month or $699 a year for up to 10 teammates. See /pricing and the live demo at /demo.