Inside the fit judge: an LLM-as-judge that corrected my own labels
An LLM-as-judge scores how well a resume fits a role, gated by a regression suite over a human-labeled golden set. It disagreed with my labels twice and was right both times, caught overqualification that keyword matching cannot, and was non-deterministic at temperature zero until self-consistency voting fixed it. A field guide to building an eval you can actually trust.
By Russ. Built and validated 2026.07.
What this is
A small system that scores how well a resume fits a specific job description, on a 0 to 100 fit score and a strong / fair / weak band. It exists to answer one question before an application goes out: is this the right resume for this role, and where is it weak against what they actually asked for.
The interesting part is not the scoring. It is the machinery that keeps the scoring honest: a rubric, a human-labeled golden set, two graders held to different bars, a regression suite that gates changes, and self-consistency voting to stabilize a judge that turned out not to be deterministic. This post is about that machinery, because it is the reusable part. The resume domain is a vehicle.
Why keyword matching is not enough
The cheap version of this is an applicant-tracking-style keyword scorer: count how many of the job description's required terms appear in the resume, report a percentage, done. It is fast, free, and deterministic, and it is genuinely useful as a first pass. I kept it. But it is blind to the thing a recruiter actually does, which is judge fit.
The clearest example: overqualification. A strong senior resume dropped onto a junior data-entry role scores high on a keyword scorer, because it has all the skills and more. To a human it is a weak fit, because the person is aimed at the wrong level and will not be hired for it. Overqualification reads as strength on every surface signal, which is exactly why a keyword count cannot see it. You need something that reasons about the shape of the match, not just the overlap.
The rubric
Five weighted dimensions, each scored 1 to 5 against written anchors: skills coverage (0.30), title and seniority alignment (0.20), quantified impact (0.20), domain relevance (0.15), and evidence of shipping and ownership (0.15). The weighted sum maps to a 0 to 100 score and a band.
The anchors matter more than the weights. Each score from 1 to 5 has an explicit written definition, so the grader is choosing between described states, not guessing on a bare number line. A 1-to-5 anchored scale beats a 1-to-10 open scale because "is this a 6 or a 7" is exactly where inter-rater agreement goes to die. Collapse that ambiguous middle and both a human and a model score more consistently. The rubric is the shared contract: the human labels against it, the graders score against it, and the regression suite is written against it, so all three speak the same five dimensions.
Two graders, held to different bars
There are two graders behind one interface. A heuristic grader estimates each rubric dimension from signals it can compute deterministically: keyword coverage, title overlap, number density, shipping language. It needs no API key and runs instantly. And an LLM-as-judge that reads the rubric, the job description, and the resume, and returns per-dimension scores with reasoning.
The gates are grader-aware on purpose. The heuristic is a cheap proxy that cannot model overqualification, so it is held to "never wildly disagrees" (within one band of the human label) plus perfect stability. The LLM judge is held to exact band agreement, because judging the nuance is the entire reason it costs money. Holding a keyword proxy to the same bar as a reasoning judge would just be gaming the metric. Use the cheapest grader that answers the question, and hold each to the bar it can actually clear.
The regression suite, and three ways it embarrassed me
The golden set is ten job-description-to-resume pairs I labeled by hand as strong, fair, or weak, including deliberate mismatches so the suite has to prove the judge scores bad fits low, not just that it rewards good ones. The regression runner scores every case, checks band agreement against my labels, checks consistency by running each case more than once, and fails the build if a gate is missed. Standard eval plumbing. Here is what it caught.
One: the judge corrected my labels
On the first validated run the judge disagreed with me on two cases, and both times it was right. I had labeled a role that hard-requires vector databases and distributed systems at scale as a "fair" fit for a candidate who lacks both. The judge called it weak. It is weak. A role gated on requirements you do not meet is not a fair fit with a gap, it is the wrong tool. I changed my label. Ground truth is not automatically whatever the human wrote down. The eval improved my judgment, not the other way around. This is the uncomfortable, useful part of eval work that people skip: sometimes the disagreement means your labels were lazy.
Two: it caught the thing keyword matching cannot
The overqualified-candidate case, an AI product resume dropped on an entry-level data-entry role, scored high on the keyword heuristic and correctly weak from the LLM judge. That single case is the entire argument for paying for the judge layer at all. If the judge only reproduced the keyword score, it would be an expensive way to count words. It earns its cost precisely on the cases the cheap grader gets wrong.
Three: temperature zero was not deterministic
I assumed temperature zero meant the same input returns the same output, ran each case once, and trusted the number. Then a case scored 66 on one run and 53 on the next, on identical input. That is a two-band swing on a supposedly deterministic call. Small instruct models are not fully deterministic even at temperature zero, and a single sample of a judge is a coin flip anywhere near a band boundary. An eval you trust blindly is worse than no eval, because it launders a guess into a number.
Self-consistency voting
The fix for a non-deterministic judge is not to pretend it is deterministic. It is to sample it an odd number of times and take the median of each dimension. Three samples, median per dimension, and the boundary cases stopped flipping. Consistency went to 100 percent and stayed there across runs. Odd sample counts avoid median ties. This is the standard production move for LLM-as-judge, and I only reached for it because the regression suite made the instability impossible to ignore.
The line I would not cross
This judge feeds a tailoring step that adjusts a resume per application. The hard rule is that tailoring never invents. It does two things only: it sets the headline to the exact role title when the underlying experience genuinely matches, and it surfaces job-description keywords into the skills section, but only the ones that already map to real experience. The system splits missing keywords into "true, surface these" and "cannot claim, flag for a human," and it will not auto-add anything in the second bucket. The value of an eval is that it tells you where you are weak so you can fix it honestly, not that it papers the weakness over. A judge that helped you lie would fail the moment a human read what the machine wrote.
Model choice: do not use a reasoning model as a judge
My first instinct was to reach for a large reasoning model as the judge. It was the wrong instinct, and worth saying why. Reasoning models forced temperature to one, which is poison for a reproducible eval. They were slow, because they think before answering. And they burned the token budget on hidden chain-of-thought and returned empty output when the budget ran out. A judge does not need deep deliberation, it needs a stable score against a rubric. A small fast instruct model at temperature zero, four seconds a call, reliable JSON, is the right judge. Save the reasoning model for the task, not the grading of it.
What I would change
Three things. The golden set is ten cases, which is enough to catch gross miscalibration but thin for the boundaries; the honest next step is to grow it from real disagreements as they surface, the way any golden set should grow. The judge is measurably more conservative than I am on genuinely adjacent roles, a one-band offset I would rather correct with a calibration pass than by moving thresholds until it agrees with me. And the tailoring's title extraction is naive on messy job posts with no clean title line, so it falls back to the untailored resume rather than guessing, which is the safe failure but not the useful one.
The reusable takeaways are smaller and blunter than the system. Anchor your rubric so the middle of the scale is not a coin flip. Hold cheap and expensive graders to different bars. Let the eval argue with your labels and be willing to lose. Do not trust a single sample of a judge. And pick the model for the job it is doing, which for judging is small, fast, and boring, not large and clever.