WORKFLOW BUILDER · GUIDE · EXPERTS AND THE AI JUDGE
Expert evaluation and the AI judge
This guide is for two groups of people: subject matter experts who grade a workflow's output, and the workflow owner who runs the panel and, if they choose, the AI judge. The general builder guide is here.
1. Who does what
- Workspace administrator. Gives the expert role to an active, verified member of the workspace, and can withdraw it. Find it in the builder under Experts, beside the security log.
- Workflow owner. The person who saved the workflow. Invites experts to that workflow's panel, shares runs for grading, approves or declines criteria, picks the cases for the blind baseline, chooses the release checks and runs the AI judge. The owner is accountable and can never sit on their own panel.
- Expert. Grades outputs, drafts criteria, checks critical test cases, answers the blind baseline and proposes test cases. An expert cannot approve anything, including their own drafts.
Everyone's entries are independent. You see other experts' grades, answers or verdicts on an item only after you have recorded your own, and nothing can be changed once it is recorded.
2. Invite experts to a panel (owner)
- Ask a workspace administrator to give the expert role to each person. They must already be workspace members.
- Open your saved workflow in the builder and expand Experts and evaluation. Under Panel, type the expert's verified email and choose Invite to panel.
- Under Runs shared for grading, choose Share for grading on finished runs. Share a mix of good and poor outputs: agreement cannot be measured if every output gets the same verdict.
Experts find the workflow under Workflows you evaluate as an expert when they sign in. If an administrator withdraws someone's expert role, the panel shows "expert role withdrawn" and they can no longer enter grades.
3. Grade before criteria exist (experts)
Before any criteria are approved, each shared run asks one question: is this output acceptable? Choose Pass, Fail or Abstain and add a short note on what made it good or bad. Abstain when you cannot tell from the documents. These notes are the evidence for writing criteria, so be specific. Do not type personal data into notes; the Studio refuses notes that look like personal or patient identifiers.
4. Draft criteria and owner approval
Experts draft a criteria version under Draft acceptance criteria: up to 12 short criteria, each one a statement a grader can mark pass or fail. You can give a criterion a higher agreement floor than the default of 0.6, and you can propose an acceptance bar: the share of the evaluation set the system must pass. A revision of an earlier draft keeps its authors as co-authors.
The owner approves or declines each draft with a written reason of at least 10 characters. Nobody approves a draft they helped write. An approved version is fixed: to change it, the experts draft a new version, and approving that one supersedes the old one. Under This release, the owner pins the approved version the next saved workflow version is evaluated against. Every agreement figure is reported against one criteria version at a time.
5. The blind baseline
The blind baseline measures how well people do on the same cases, so the acceptance bar is set against human performance rather than a guess.
- Owner: under Blind baseline sample, tick the test cases to include, then save a new version.
- Experts: under Blind baseline: answer these cases yourself, read the documents and write your answer. The system's output for that case stays hidden from you until you have answered the current version of the case. Your answer is checked with the same expectations as the system's output.
A baseline counts only once at least 10 cases in the sample have been answered. Until then no expert accuracy is used, and any acceptance bar needs the owner's written reason. Once the baseline counts, a bar above the experts' accuracy still needs that written sign-off. A workflow holds up to 10 test cases, so today a usable baseline means every test case is in the sample and each has at least one answer.
If the owner edits a case's documents or expectations, it becomes a new version of the case: earlier answers no longer count and its run is hidden again until you answer the new version. The dispute view then shows the system's result beside each expert's own answer. An even split between experts is shown as split; it is never resolved by a tie-break.
6. Critical cases and two-expert agreement
A critical test case is one where a failure is never offset by a good average (for example disclosure or data integrity). For these cases, two experts must agree independently that the expected result is right before the version can run on other documents.
- This check is on by default whenever the workflow's panel has two or more active experts. The owner can turn it off explicitly under This release; the choice is saved with the next version.
- Experts answer Critical test cases: is the expected result right? with Agree or Disagree, and say what they expect instead when they disagree.
- Any disagreement marks the case ambiguous. No single person, the owner included, can settle it: the owner rewrites the case, which needs two fresh agreeing reviews.
7. Run the AI judge (owner)
The AI judge is optional. It grades runs against an approved criteria version so you can see whether it agrees with your experts. It does not replace them.
- Approve a criteria version and have experts grade shared runs against it.
- Under Grades and the AI judge, pick the criteria version and choose Run the AI judge on shared and test-case runs. One click grades up to 40 finished runs, one at a time, and replaces any earlier judge grade on the same run and criteria version.
Key and cost. The judge uses the same key path as workflow steps: this workspace's AI key if an administrator added one, otherwise your own key from your account page. Your provider bills that key's account; RJL does not pay for it and the key is never stored with the grades or shown to anyone. Each graded run is one request to the provider, typically a few cents at list prices for short example documents; longer documents and premium models cost more, so a full click can reach a dollar or more. Calls count against the key's monthly spending cap, and documents with personal or patient identifiers are refused before anything is sent.
A missing or unrecognized verdict from the judge counts as an abstention, never a pass.
8. Read the agreement table
The table has one row per criterion of the chosen criteria version. Agreement is chance-adjusted: two graders who pass everything agree perfectly by accident, so raw percentages are not used.
- Experts (kappa, mean ± sd). Cohen's kappa for each pair of experts on the runs both graded, then the mean and spread across pairs. 1 is perfect agreement, 0 is no better than chance.
- Experts (alpha). Krippendorff's alpha across all experts together. It allows missing grades.
- Judge (kappa). The judge's mean kappa with each expert on the same runs.
- Floor. The minimum agreement for the criterion: 0.6 by default, or the higher floor set on the criterion.
- Status. May gate, Needs expert grades, Needs judge grades, Below expert band, Below floor, Rewrite criterion or Not measurable. Hover or read the line under the table for the reason.
Abstentions are left out of each pair and counted. A criterion where every grade falls in one category is not measurable; it is never treated as perfect agreement. If experts agree below the floor, the status is Rewrite criterion: reword it so people can apply it consistently. The floor is never lowered to match.
Before the judge may gate a criterion, all of these must hold:
- at least 3 experts, each sharing at least 10 graded runs with another expert (abstentions excluded);
- the judge shares at least 10 graded runs with at least one of those experts;
- the experts' mean kappa is at or above the floor;
- the judge's mean kappa is at or above the floor (0.6 by default);
- the judge's mean kappa is no more than one standard deviation below the experts' mean.
9. What the judge may and may not gate
Under This release, the owner can tick Let the AI judge gate release. The release gate applies when the workflow requires passing test cases before it runs on other documents.
- May gate: ordinary (non-critical) test cases, and only on criteria marked May gate for the criteria version pinned to the release. A judge abstention on such a criterion counts as not passed.
- May not gate: critical test cases. Critical categories always need people and the fixed rule-based checks: the critical-failure veto and two-expert agreement. The judge's grades never clear or block a critical case.
- If the judge is not calibrated on any criterion, it does not gate at all; people and the test-case checks still do.
10. Limits
Use example or approved business documents only. The judge and the experts grade written output; they do not make regulated decisions and their grades are not legal or professional sign-off. Expert oversight applies the reviewer rule to experts: an expert with 10 or more grades who failed under 2% of them is flagged as a prompt to look at a sample, not as a finding. The thresholds on this page (0.6 floor, 10 shared runs, 3 experts, 10 answered baseline cases) are the owner's current settings and may change.