Introduction

Consistency matters tremendously in the AI evals world. You don't want your eval results to jump around like they're dancing to House of Pain. You want to rely on the results you see. You want to be confident that when you change something in your implementation, the scores improve (hopefully) because of your change, and not because the DJ dropped a new beat. In the same way, if you use an eval as a unit test, you want it to fail only when your changes actually broke something. There are already plenty of real problems in our lives, no need to add fake ones. That's why we're here. This post is about how we make eval results more consistent. It's rooted in eval examples, but the principles apply to any flow that uses LLMs to score something.

How bad can it be?

We ran the same CV-fit metric ten times. Same metric, same requirements, same CV, same judge. The score wandered from 0 to 75. Five runs cleared our CI gate (the pass threshold, 55); five did not. A dashboard that only plots pass/fail would flip depending on the day. This is how bad it can be.

So what can be done about it? The internet has some advice.

The first two tips might have helped last year, but those params are gone now. New reasoning models ignore them, even if you pass them via the API. Probably because it's not easy to keep the reasoning process in a specific thinking lane. Running the same LLM call multiple times and taking the median can help, but is that median really representative of the expected value? Also, before multiplying your costs 4-5x without any real guarantee, maybe it's worth trying something else first.

What we try to do instead:

I know, it sounds vague and corporate. Let's try to make it more concrete.

The mixing desk

There are three dials that influence your final results: the algorithm, the model, and the data. Dial 1 is the most important one, and where you should spend most of your effort: how you build the flow around the LLM, what you do with the input before it goes to the LLM, how you write your prompts, and what you do with the output when it comes back. Control the hell out of it and refine it as much as you can. Dial 2 is the model. Most flows use an off-the-shelf model, so you can't really control it. But you can choose the one that fits your task best. Dial 3 is the data: the prompts, outputs, and specs our clients send us. We can't change it, but we can negotiate what they end up sending.

The rest of this post turns the dials one at a time, on real reruns. The question is not "is the model noisy?" but "which knob did we leave in the model's hands?"

The example

The test case is a staff backend role in payments, Berlin, EU hours. The job description is long on purpose. Sixteen requirements, six sections, one hard gate. The full pack is in The job description and CV.

The candidate must be able to work in the EU on their start date without sponsorship. We cannot sponsor. A candidate who fails the gate is not advanced, however strong the rest is. Everything below the gate is weighed, not vetoed.

The CV is strong on paper. Eight years of professional Python. FastAPI card-payments API, on-call, Kafka, idempotency keys, two juniors mentored, a finished CS degree at TU Berlin, four weeks' notice, will work EU hours after relocating. Then, in additional information:

Work authorization: Irish citizenship by descent; Foreign Births Register application submitted 2026-05, decision pending, expected before the start date; if it slips, I would need a work permit in the meantime.

That is the planted fail. Citizenship is pending. A permit may still be required. The visa gate says no.

A naive judge that emits one 0-100 score for the whole pack will average the visa problem with the Kafka and the mentoring. That is how you get a 72 that still "passes" — and, on the next rerun, a 0. The score is not wrong about the Python. It is wrong about what the number is for.

Below is the prompt that generated the jumpy scores between 0 and 75. This is how a lot of flows are built today: one prompt that asks the model to score the whole document and output a single number.

Dial 1 — Make the model decide less

We'll change this naive prompt step by step and observe what helps. The move is not "write a better prompt." It is "take decisions away from the model" until the only thing left is a verdict a human reviewer could also give. I'll use GPT-5.6 Luna with reasoning set to none for all steps.

Hack A — Steps, not one prompt

Split the job description into sixteen requirement chunks. Ask for a 0-100 score per chunk, in one call, then average. Ask for chunk ids in the output rather than the original requirement text. Let's not waste tokens on that.

σ drops from 23.4 to 2.6. The mean climbs into the low 80s because the easy chunks (education, seniority, most of experience) are now 100s and they are no longer averaged together with the visa problem. Every rerun now clears the CI gate. The remaining jitter is the model still inventing a number, just sixteen smaller ones.

Consistent, yes. Correct, no. This candidate shouldn't pass, and now every rerun passes with confidence.

Hack B — Make its life easier

In Hack A, the model numbered the requirements itself. Don't let it invent chunk ids. Python owns the keys. The input is { 0: "...", 1: "..." }. The output still has chunk_id, score, reason.

σ drops slightly, to 2.4. It doesn't look like much, but the effect goes beyond the score. LLMs are not good with numbers, even when it's just counting items in a list. We've had plenty of cases where the LLM returned a different number of items than the list had, or hallucinated indexes (e.g. 100 in a list of 16 items). Take the counting off its plate and the LLM only has to focus on assigning a score. You get more accurate results, better consistency, no hallucinated ids, and faster processing. Such a small change, such a big impact.

Hack C — Never ask a model for a number

Ask for a category. Map the category to a score in Python.

We used five labels, each with a short description so the model knows where the line is. The model returns chunk_id, category, reason. It never emits 0-100. Python maps the labels to scores: unfulfilled → 0, mostly_unfulfilled → 25, partially_fulfilled → 50, mostly_fulfilled → 75, fully_fulfilled → 100.

They are called Large Language Models for a reason. Models do not perceive numbers the way we do. They can easily jump between 70 and 80 on the same evidence. Their training data is full of word-to-word associations, not number-to-word ones. So let them talk words.

This heatmap shows where the earlier σ was coming from. Each row is one requirement; the number next to a section is how many requirements it has. Education (3), seniority (1), and experience (3) are 100s across the board. In Hacks A and B, one tech-skills chunk sits steadily in the middle, at 50. The red is constraints — work authorization, sponsorship, start date, EU hours. Hack A keeps most of the red steady, but lets the bottom constraint row flicker. Hack B lets the red band flicker run to run. Hack C locks the constraints, and the only row still wobbling is that tech-skills chunk — that is the 0.9. The model was not confused about the CS degree. It was improvising a number on the visa sentence.

Hack D — Weight the section, not the chunk

Up until now we used a simple average, but not all requirements are born equal. Constraints matter more than education. A CS degree doesn't help much if you don't have a work visa.

Since job postings don't have a fixed structure, we use an LLM to assign each chunk to a section. Python then takes the per-chunk scores from the previous step and calculates a weighted average based on the section each chunk belongs to.

The mean drops from 86.0 to 82.1 because constraints went from a quarter of the vote (4 of 16 chunks) to a third (weight 5 of 15). Variance stays in the same tight band. Is it enough though? 82 is still a comfortable pass for someone the job description says not to advance.

Hack E — A missed constraint scores zero

There is an infinite number of rules you can apply to get the best possible final result, for example:

at least 50% of the requirements need to be mostly_fulfilled or better

at least 20% of requirements from each section need to be fully_fulfilled

I chose a more rigid one for this example.

The point here is to pick a rule and implement it in Python. Don't let the LLM decide. You are the expert: you set the rules, and Python applies them the same way every time.

So, did it work?

Yes. σ went from 23.4 to 0.0, and the score finally says what the job description says. The model judges. The calculator calculates. We already invented calculators.

What about the cost? Of course a multi-step approach costs more than a single prompt. The upside is that once you split the task, you can often use a smaller model with no reasoning, which you wouldn't get away with on a single prompt. And if you try to fix the single prompt by switching to a better model, like GPT-5.6 Terra with low reasoning, the cost explodes and the consistency still isn't there (σ = 15.4). So the gain is real.

So are we done? With the CV-fit example, yes. But even an amazing algorithm doesn't make the model irrelevant.

Dial 2 — The model is a bias, not noise

We took a fake occupational-health briefing with planted gender, age, nationality, and disability stereotypes — some in the author's own voice, some as hostile quotes the text claims to reject — and scored bias 640 times (100 = clean, lower = harsher). Same document, same metric. Nine models, multiple reasoning levels, twenty repeats each.

Almost every run flagged the four planted categories. The 0-100 score still ranged from about 28 to 72. Higher reasoning made GPT-5.1, Opus 4.6, and Sonnet 4.6 harsher, and made Sonnet 5 more lenient. Luna at none was a different judge from Luna with reasoning on. Opus 5 barely moved.

So you're picking two things: how noisy a judge is (Figure 10), and what it believes (Figure 9).

That spread is not a seed. It is an opinion. Models do not only have variance. They have views — on whether quoted harm counts, on how much of a document is enough, on how sure they are that "certain national backgrounds" is nationality bias. You mostly choose this dial: family, reasoning effort, then pin the version.

We wrote the full analysis, including how judges split on quoted harm and on nationality, in The Judge Is Part of the Metric. The short version for this post: pick the judge like you pick a reviewer, for their biases, not their mood. Write the choice down. A re-baseline after a model upgrade is a release, not a bug.

Dial 3 — "Be concise" is a random number generator

We don't control this one. We negotiate it, with the client.

One prompt, four jobs

The first example is a flow that uses an LLM to analyze technician inspection notes. It has a huge prompt that asks the model to summarize, edit, diagnose, and extract. The client uses the latest model with reasoning enabled, but the results are still not good. And when we evaluate the output, the scores are quite jumpy too. Well, it's kind of obvious when you look at what the LLM is supposed to do in a single take: four tasks, some of them conflicting with each other. The model has to drop details while summarizing, keep the exact values (like temperature) while extracting, and come up with new information while diagnosing. Sounds quite complex and error-prone to me.

So what's the negotiation about? Convincing them that there is a better world in which this is split into three separate tasks (summarizing and editing fold into one), each with its own prompt. Only the Diagnostician needs a higher-reasoning model; the rest can run on a smaller one. The tasks can run in parallel. The prompts are shorter and simpler. You save time and money, and you get better results. And you can evaluate each task with the metric that fits it without getting loads of false positives.

Format alignment

The next experiment did not change the output and did not change the model. It changed the spec.

The task is a private equity (PE) workforce briefing. The judge is asked whether the output "follows the format." By default, our format alignment metric extracts the format instructions from the prompt, then asks the LLM to score the output against them. So if the prompt says "be opinionated" or "be direct", how is the model supposed to assess "opinionism"? I know it's not a word. But the thing is, the model will change its mind quite often. We started the negotiation by asking the client how they would assess whether the output is "concise". Is it 3 sentences, or 300 words? After some back and forth, we understood that they don't actually care that much about length, and these broader instructions were more of a guideline than a strict requirement. What they do care about is the markdown formatting and the headers. Those have to be exactly right, so the rendering never breaks. Making the rules more concrete helps, but what they actually needed was the headers. So this is what we checked: six headers, in this spelling, in this order. We specified them in the 'instructions' parameter of the metric.

With a CI gate of 80 and scores from 45.8 to 85.3, the same output passed or failed depending on the run.

"Be opinionated" is not a requirement. It is an invitation for the judge to have feelings. Once the metric checked for six headers, the feelings stopped showing up in the score.

What we don't stabilize

Not every movement is a dial you should chase. Boring is not the same as frozen.

The explanation. We return a small reason as part of our eval response, and we're fine with that being phrased differently. Prose is allowed to vary. Two reviewers describe the same 90 differently and both are right.

The float. Chasing bit-exact scores is the wrong goal. The contract is same input, same band, not identical decimals. If your LLM always returns 50.2, maybe the task is so simple that you don't even need an LLM. If you do need one, expect some variance.

The mean after a model upgrade. It will move. A re-baseline is a release, not a bug. Pin the judge; version the CI gate.

The judge's opinion. Dial 2 is a choice, not a defect. Choose it on purpose and write the choice down.

Conclusion

The wobble was not a sampler setting. Same CV, ten reruns, a seventy-five-point swing that passed the CI gate on a good day and scored 0 on a bad one — until we turned Dial 1: stopped asking the model for a number, weighted the sections the job description had already ranked, and put the visa rule in Python. A stronger model on the same one-shot prompt still had σ = 15.4. Categories tightened the score to a three-point band. The visa rule then told the truth.

The bias runs are the same lesson from the other side. Pin the protocol and the judge still has opinions. Means from 28 to 72, with more reasoning making some judges harsher and others more lenient. That is Dial 2. You choose it, then you pin it.

Dial 3 is the contract you thought you had. One prompt that summarizes, edits, diagnoses, and extracts is four specialists in a trench coat. "Follow the format" is 400 words of manners. Six headers, in this spelling, in this order, is a metric.

A metric that cannot commit is not a metric. It is a vibe.