The screening pilot that saved nothing
Twelve weeks, 9,000 applications, and a time saving that vanished once review time was counted properly.
The pilot was set up to answer one question: does automated CV screening reduce the time a recruiter spends getting from application to shortlist? Twelve weeks, four job families, 9,000 applications, one vendor, and a control group that ran the old way.
The headline result was a 71% reduction in screening time. That number was true, and it was also the wrong number.
What the 71% measured
It measured the interval between an application arriving and a recruiter marking it reviewed. The tool did that in seconds. A recruiter did it in roughly ninety.
What it did not measure was everything the recruiter then did with the output: opening the borderline cases the model had scored between thresholds, checking the rejections in the job families where the model had least training data, and answering hiring managers who wanted to know why a particular profile had not come through.
Time tracking, 12-week pilot FY26 Q1 · four job families, three regions · n=9,000 applications
Add the pilot’s three bars together and you get 139 minutes against a control of 152. A saving of nine per cent, not seventy-one.
Every automation moves work rather than removing it. The question is only whether it moves to someone cheaper, and whether you counted where it landed.
Where it did pay
The nine per cent is an average across four job families, and averages hide the useful part.
In high-volume roles with well-defined requirements (customer support, warehouse, entry-level manufacturing) the saving was real and large. The model had thousands of examples per role and the dispute rate was near zero. In specialist engineering, where each requisition looked different from the last, the review and dispute time exceeded what was saved. The tool cost time.
Net time change by job family
| Job family | Applications | Control (min) | Pilot (min) | Net |
|---|---|---|---|---|
| Customer support | 4,100 | 148 | 71 | −52% |
| Manufacturing | 2,600 | 139 | 84 | −40% |
| Finance | 1,400 | 161 | 152 | −6% |
| Engineering | 900 | 174 | 219 | +26% |
| All | 9,000 | 152 | 139 | −9% |
Time tracking, 12-week pilot FY26 Q1 · minutes per 100 applications · n=9,000
What we did with it
We kept the tool and narrowed it to two job families. The licence cost went down because the volume tier changed, and the recruiters who had been fighting it in engineering got their afternoons back.
The uncomfortable part of the review was not the vendor. It was that the original business case had been written against the screening interval, and nobody, including me, had asked at the design stage what happened to the work downstream. A pilot that measures one step of a process will always find that step improved.
If you are about to run one: define the measurement boundary before the vendor does, and make it end at the point where a hiring manager sees a shortlist, not at the point where the model returns a score.
Open Req Letter · Letter 013 · 26 February 2026 · Piotr Knapik
https://openreqletter.com/letters/the-screening-pilot-that-saved-nothing