Skip to content

fix(evaluator): answer each rubric judge within its model's max_tokens - #183

Merged
3mei merged 1 commit into
mainfrom
yev/fix/judge-token-budget
Nov 22, 2026
Merged

3mei merged 1 commit into
mainfrom
yev/fix/judge-token-budget

Conversation

@3mei

@3mei 3mei commented Nov 22, 2026

Copy link
Copy Markdown
Contributor

Related issue

None.

What changed and why

The rubric judge asked every model for at most max_judge_tokens (4,096, or 8,192 in locales whose script tokenizes densely), whatever max_tokens the judge's row in the models file set. A judge that reasons before answering can spend that budget before it finishes its JSON, so its answer doesn't parse and the row is stored without a judge verdict, with no setting that gives it more room. One commit.

  • A judge's model sets its budget. Each judge now answers within its own model's max_tokens, scaled by NON_ASCII_TOKEN_SCALE for those locales, read with scaled_max_tokens as the simulator's calls already are (TrajectoryEvaluatorGenerator._judge_max_tokens). Two judges in an ensemble each get their own model's budget. max_judge_tokens and max_judge_tokens_non_ascii keep their defaults and apply to a judge whose model sets no max_tokens, so a models file that sets none sees the same budget as before. The warning for an answer that doesn't parse now names the budget and where to raise it. architecture/evaluator.md says how the budget is set, and that the next usersim eval judges a row without a verdict again.

No prompt or eval version changes: the prompt and the envelope are unchanged.

How it was verified

New tests fail on the code before the change and pass after it: a judge whose model sets max_tokens answers within it in English and within twice it in Japanese, each judge of an ensemble within its own (test_evaluator_generator.py). Two more cases pin the fallback for a model that sets none, 4,096 and 8,192, and pass before and after.

On the commit's tree: make check-all, the pre-commit hooks on the commit's files and pytest --collect-only.

At the head:

$ make check-all
Lint clean. Run 'make lint-fix' to auto-fix issues.
471 files already formatted
Formatting clean. Run 'make format' to auto-fix issues.
$ make test
5281 passed in 56.22s
$ make smoke
  usersim simulate --locale en_US --num-rows 5 \
    --out output/trajectories
$ make check-license-headers
🎉 All files have correct license headers!
$ make check-doc-links
✓ every relative link, and every link into this repository's main branch, resolves
$ make check-notebooks-clean
OK 4 notebook(s) carry no outputs

Live, against a custom provider config: the end-to-end health check as configured, where the evaluator's row sets no max_tokens, and again with max_tokens = 12000 on the evaluator's row, which the ja_JP row sends as 24,000 through the existing max_completion_tokens retry for a model that rejects max_tokens:

$ make health-check PROVIDER_CONFIG=<custom config>
[INFO]   |-- requests: success=1, failed=0, total=1, rpm=3
✓ simulate, eval and report all completed
$ make health-check PROVIDER_CONFIG=<custom config, evaluator max_tokens = 12000>
[INFO]   |-- requests: success=1, failed=0, total=1, rpm=4
✓ simulate, eval and report all completed

Results impact

Runs whose models file sets no max_tokens on a judge's row are unchanged, and so are runs through the model catalog, whose entries set 4,096 or none. A run whose judge row sets max_tokens now gets that budget: more room where it is larger, so rows that reasoning judges left without a verdict get rubric scores, and less where it is smaller than 4,096.

The rubric judge asked every model for at most max_judge_tokens, whatever max_tokens the judge's row set, so a judge that reasons before answering could run out of room before its JSON and leave the row without a verdict. Each judge now answers within its own model's max_tokens, scaled for locales whose script tokenizes densely, as the simulator's calls are; max_judge_tokens and max_judge_tokens_non_ascii apply to a judge whose model sets none.

Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
@3mei
3mei requested a review from a team as a code owner November 22, 2026 19:21
@3mei
3mei merged commit 900c4b1 into main Nov 22, 2026
12 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant