瀏覽代碼

updated mmlu meta-eval for prompt migration

Justin Lee 2 月之前
父節點
當前提交
479b1fbbd7

+ 2 - 1
end-to-end-use-cases/benchmarks/llm_eval_harness/meta_eval/work_dir/meta_instruct.yaml

@@ -1,3 +1,4 @@
 group: meta_instruct
 task:
-- meta_mmlu_pro_instruct
+- meta_mmlu
+# - meta_mmlu_pro_instruct

+ 3 - 3
end-to-end-use-cases/benchmarks/llm_eval_harness/meta_eval/work_dir/mmlu/mmlu.yaml

@@ -1,6 +1,6 @@
 task: meta_mmlu
-dataset_path: meta-llama/Llama-3.1-70B-evals
-dataset_name: Llama-3.1-70B-evals__mmlu__details
+dataset_path: meta-llama/Llama-3.3-70B-Instruct-evals
+dataset_name: Llama-3.3-70B-Instruct-evals__mmlu__0_shot__cot__details
 test_split: latest
 output_type: multiple_choice
 process_docs: !function utils.process_docs
@@ -11,4 +11,4 @@ doc_to_choice: ["A", "B", "C", "D"]
 # So no need to generate
 num_fewshot: 0
 metadata:
-  version: 1.0
+  version: 1.0

文件差異過大導致無法顯示
+ 27 - 5
end-to-end-use-cases/benchmarks/llm_eval_harness/meta_eval/work_dir/mmlu/utils.py