BlogSign inApply for the pilot

Test an AI on your past work before it touches a client

29 October 2026 · 2 min read · Keyrole team

Every AI product looks good in a demo, because the demo uses an example chosen to work. The question a firm needs answered is different: will it do our work, for our clients, the way we do it? There is a simple way to find out before any client is involved: give it jobs you have already done and compare its results with your team's.

Why past work is the right test

  • You already know the right answer. Your team's delivered work is the benchmark, including the judgement calls they made.
  • It uses your real inputs. The messy spreadsheet exports, scanned statements and late corrections that demos leave out.
  • No client is exposed. Mistakes found here cost nothing.

How to run a probation run

  1. Pick a few past jobs for one service, including at least one awkward case: a missing paper, an unusual transaction, a month that needed a correction.
  2. Give the AI the inputs your team had at the time, not the final answer.
  3. Compare its output with what your team delivered: the figures, the structure, the notes and the questions it raised.
  4. Look at the differences, not just the score. A difference may be its mistake, or it may reveal a step your method never wrote down. Both are worth knowing.
  5. Decide whether it goes live. If it does not match, teach it what was missing and run the test again.

Make the test compulsory where figures are involved

For work that produces figures, such as books, reconciliations, returns and payroll, a probation run should not be optional. In Keyrole, a skill that produces figures cannot go live until it has been tried on your past work, and if the skill changes afterwards, it has to be tested again. Probation runs never count towards its Key score; that score is built only from your reviewers' decisions on live work.

What a passed probation does not mean

Passing on a few past months shows the method works on your typical cases. It does not prove it will handle every unusual case, which is why review continues after it goes live, and why review should loosen only as a record builds up. Treat it the way you would a new hire's first months: promising, measured, and still checked.

Keyrole runs a probation on your own work before any AI employee starts. Apply for the pilot to try it on one role.