浏览代码

fix issue with jsonl encoding

Elena Paraschiv 8 月之前
父节点
当前提交
97fa791e3a
共有 1 个文件被更改,包括 1 次插入1 次删除
  1. 1 1
      llmeval/commons/management/commands/import_medqa.py

+ 1 - 1
llmeval/commons/management/commands/import_medqa.py

@@ -29,7 +29,7 @@ class Command(BaseCommand):
         # check if dataset exists
         dataset, _ = Dataset.objects.get_or_create(name=dataset_name)
         
-        with open(file_path, 'r') as f:
+        with open(file_path, 'r', encoding='utf-8') as f:
             lines = f.readlines()
             for line in tqdm(lines, desc="Importing MedQA data"):
                 data = json.loads(line)