fix lmeval start speed#1343
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces a new common guidance document (skills/test_model/SKILL.md) to accelerate lm_eval startup times by defaulting to offline caching configurations. The reviewer suggested updating the --model_args parameter format in the example command to use an escaped JSON string instead of a comma-separated string to maintain consistency with other sub-skills in the directory and prevent parsing issues.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| lm_eval --model local-completions \ | ||
| --model_args "model=${MODEL_NAME},base_url=${BASE_URL},num_concurrent=64,max_retries=3,tokenized_requests=False,tokenizer=${MODEL_DIR}" \ | ||
| --tasks gsm8k \ | ||
| --batch_size 64 \ | ||
| --confirm_run_unsafe_code |
There was a problem hiding this comment.
The --model_args format in this example uses a comma-separated string, whereas other sub-skills in this directory (such as qwen3.5-0.8b-gsm8k-scenarios/SKILL.md and qwen2.5-14b-fp8kv-gsm8k/SKILL.md) consistently use an escaped JSON string format for --model_args when using the local-completions model. To ensure consistency and avoid potential parsing errors, please update the example to use the JSON format.
| lm_eval --model local-completions \ | |
| --model_args "model=${MODEL_NAME},base_url=${BASE_URL},num_concurrent=64,max_retries=3,tokenized_requests=False,tokenizer=${MODEL_DIR}" \ | |
| --tasks gsm8k \ | |
| --batch_size 64 \ | |
| --confirm_run_unsafe_code | |
| lm_eval --model local-completions \ | |
| --model_args "{\"model\":\"${MODEL_NAME}\",\"base_url\":\"${BASE_URL}\",\"num_concurrent\":64,\"max_retries\":3,\"tokenized_requests\":false,\"tokenizer\":\"${MODEL_DIR}\"}" \ | |
| --tasks gsm8k \ | |
| --batch_size 64 \ | |
| --confirm_run_unsafe_code |
No description provided.