Repository navigation
fix(evaluator): answer each rubric judge within its model's max_tokens - #183
Merged
Merged
Conversation
The rubric judge asked every model for at most max_judge_tokens, whatever max_tokens the judge's row set, so a judge that reasons before answering could run out of room before its JSON and leave the row without a verdict. Each judge now answers within its own model's max_tokens, scaled for locales whose script tokenizes densely, as the simulator's calls are; max_judge_tokens and max_judge_tokens_non_ascii apply to a judge whose model sets none. Signed-off-by: Yev Meyer <ymeyer@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Related issue
None.
What changed and why
The rubric judge asked every model for at most
max_judge_tokens(4,096, or 8,192 in locales whose script tokenizes densely), whatevermax_tokensthe judge's row in the models file set. A judge that reasons before answering can spend that budget before it finishes its JSON, so its answer doesn't parse and the row is stored without a judge verdict, with no setting that gives it more room. One commit.max_tokens, scaled byNON_ASCII_TOKEN_SCALEfor those locales, read withscaled_max_tokensas the simulator's calls already are (TrajectoryEvaluatorGenerator._judge_max_tokens). Two judges in an ensemble each get their own model's budget.max_judge_tokensandmax_judge_tokens_non_asciikeep their defaults and apply to a judge whose model sets nomax_tokens, so a models file that sets none sees the same budget as before. The warning for an answer that doesn't parse now names the budget and where to raise it.architecture/evaluator.mdsays how the budget is set, and that the nextusersim evaljudges a row without a verdict again.No prompt or eval version changes: the prompt and the envelope are unchanged.
How it was verified
New tests fail on the code before the change and pass after it: a judge whose model sets
max_tokensanswers within it in English and within twice it in Japanese, each judge of an ensemble within its own (test_evaluator_generator.py). Two more cases pin the fallback for a model that sets none, 4,096 and 8,192, and pass before and after.On the commit's tree:
make check-all, the pre-commit hooks on the commit's files andpytest --collect-only.At the head:
Live, against a custom provider config: the end-to-end health check as configured, where the evaluator's row sets no
max_tokens, and again withmax_tokens = 12000on the evaluator's row, which theja_JProw sends as 24,000 through the existingmax_completion_tokensretry for a model that rejectsmax_tokens:Results impact
Runs whose models file sets no
max_tokenson a judge's row are unchanged, and so are runs through the model catalog, whose entries set 4,096 or none. A run whose judge row setsmax_tokensnow gets that budget: more room where it is larger, so rows that reasoning judges left without a verdict get rubric scores, and less where it is smaller than 4,096.