Benchmarking AI-assisted application review criteria

A proposal for taking the guesswork out of AI-assisted screening

Ashby's AI-assisted review allows recruiters to sift through applications faster to find the right people for the role. You write criteria prompts describing the best-fit for the job, and the AI agent applies those prompts to shortlist candidates from your active pipeline. But the shortlist is only as good as the prompt behind it, and the inherent fuzziness of a prompt leaves it open to the agent's interpretation of what you meant.

An ineffective prompt can drop a strong candidate and let a weak one through in equal measure. Whether a prompt shortlists the right people only becomes apparent after you've evaluated your active candidates. Which begs the question, how do you judge the effectiveness of a prompt before it ever reaches your active pipeline? The answer to that sits at the core of this prototype.

The prototype adds a benchmark to the criteria form, where you test your prompts against candidates in your talent pool who applied for a related role in the past. A mismatch between the benchmark result and the candidate's known outcome exposes where the prompt fell short, which surfaces a rewrite suggestion.

But why does any of this matter?

Today an ineffective criterion can cost you time, money (a credit per evaluation), and strong hire while leaving no trace. Benchmarking your criteria against known outcomes not only surfaces mismatches early, but does so for the cost of a handful of evaluations rather than your entire pipeline.

While this prototype is an incremental step, the idea behind it goes much further. Imagine a fully autonomous agent that, at the flip of a switch drafts the first set of criteria from the job description, then refines them through the same benchmark until the results sit as close to your known outcomes as it can get them.