Skip to content

fix lmeval start speed#1343

Merged
hiworldwzj merged 1 commit into
mainfrom
wzj_fix
Jun 11, 2026
Merged

fix lmeval start speed#1343
hiworldwzj merged 1 commit into
mainfrom
wzj_fix

Conversation

@hiworldwzj

Copy link
Copy Markdown
Collaborator

No description provided.

@hiworldwzj hiworldwzj merged commit 316b398 into main Jun 11, 2026
1 check passed
@hiworldwzj hiworldwzj deleted the wzj_fix branch June 11, 2026 00:29

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a new common guidance document (skills/test_model/SKILL.md) to accelerate lm_eval startup times by defaulting to offline caching configurations. The reviewer suggested updating the --model_args parameter format in the example command to use an escaped JSON string instead of a comma-separated string to maintain consistency with other sub-skills in the directory and prevent parsing issues.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Comment on lines +36 to +40
lm_eval --model local-completions \
--model_args "model=${MODEL_NAME},base_url=${BASE_URL},num_concurrent=64,max_retries=3,tokenized_requests=False,tokenizer=${MODEL_DIR}" \
--tasks gsm8k \
--batch_size 64 \
--confirm_run_unsafe_code

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The --model_args format in this example uses a comma-separated string, whereas other sub-skills in this directory (such as qwen3.5-0.8b-gsm8k-scenarios/SKILL.md and qwen2.5-14b-fp8kv-gsm8k/SKILL.md) consistently use an escaped JSON string format for --model_args when using the local-completions model. To ensure consistency and avoid potential parsing errors, please update the example to use the JSON format.

Suggested change
lm_eval --model local-completions \
--model_args "model=${MODEL_NAME},base_url=${BASE_URL},num_concurrent=64,max_retries=3,tokenized_requests=False,tokenizer=${MODEL_DIR}" \
--tasks gsm8k \
--batch_size 64 \
--confirm_run_unsafe_code
lm_eval --model local-completions \
--model_args "{\"model\":\"${MODEL_NAME}\",\"base_url\":\"${BASE_URL}\",\"num_concurrent\":64,\"max_retries\":3,\"tokenized_requests\":false,\"tokenizer\":\"${MODEL_DIR}\"}" \
--tasks gsm8k \
--batch_size 64 \
--confirm_run_unsafe_code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant