\n![\"Image\"](\"https://github.com/InternLM/xtuner/assets/36994684/a157638c-3500-44ed-bfab-d8d8249f91bb\")
\n
\n\n| Model | MMBench Test (EN) | MMBench Test (CN) | CCBench Dev | MMMU Val | SEED-IMG | AI2D Test | ScienceQA Test | HallusionBench aAcc | POPE | GQA | TextVQA | MME | MMStar | Configs |\n| :-------------------- | :---------------: | :---------------: | :---------: | :-------: | :------: | :-------: | :------------: | :-----------------: | :--: | :--: | :-----: | :------: | :----: | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |\n| LLaVA-v1.5-7B | 66.5 | 59.0 | 27.5 | 35.3 | 60.5 | 54.8 | 70.4 | 44.9 | 85.9 | 62.0 | 58.2 | 1511/348 | 30.3 | - |\n| LLaVA-Llama-3-8B | 68.9 | 61.6 | 30.4 | 36.8 | 69.8 | 60.9 | 73.3 | 47.3 | 87.2 | 63.5 | 58.0 | 1506/295 | 38.2 | [Pretrain](./pretrain/llava_llama3_8b_instruct_clip_vit_large_p14_336_e1_gpu8_pretrain.py) / [Fine-tune](./finetune/llava_llama3_8b_instruct_full_clip_vit_large_p14_336_lora_e1_gpu8_finetune.py) |\n| LLaVA-Llama-3-8B-v1.1 | 72.3 | 66.4 | 31.6 | 36.8 | 70.1 | 70.0 | 72.9 | 47.7 | 86.4 | 62.6 | 59.0 | 1469/349 | 45.1 | [Pretrain](./pretrain/llava_llama3_8b_instruct_clip_vit_large_p14_336_e1_gpu8_sharegpt4v_pretrain.py) / [Fine-tune](./finetune/llava_llama3_8b_instruct_full_clip_vit_large_p14_336_lora_e1_gpu8_internvl_finetune.py) |\n\n## Resources\n\n- LLaVA-Llama-3-8B-v1.1\n\n - Official LLaVA format model (`xtuner/llava-llama-3-8b-v1_1-hf`): 🤗 [HuggingFace](https://huggingface.co/xtuner/llava-llama-3-8b-v1_1-hf) / 🤖 [ModelScope](https://modelscope.cn/models/xtuner/llava-llama-3-8b-v1_1-hf)\n - HuggingFace LLaVA format model (`xtuner/llava-llama-3-8b-v1_1-transformers`): 🤗 [HuggingFace](https://huggingface.co/xtuner/llava-llama-3-8b-v1_1-transformers) / 🤖 [ModelScope](https://modelscope.cn/models/xtuner/llava-llama-3-8b-v1_1-transformers)\n - XTuner LLaVA format model (`xtuner/llava-llama-3-8b-v1_1`): 🤗 [HuggingFace](https://huggingface.co/xtuner/llava-llama-3-8b-v1_1) / 🤖 [ModelScope](https://modelscope.cn/models/xtuner/llava-llama-3-8b-v1_1)\n - GGUF model (`xtuner/llava-llama-3-8b-v1_1-gguf`): 🤗 [HuggingFace](https://huggingface.co/xtuner/llava-llama-3-8b-v1_1-gguf) / 🤖 [ModelScope](https://modelscope.cn/models/xtuner/llava-llama-3-8b-v1_1-gguf)\n - Pretrained projector weights: 🤗 [HuggingFace](https://huggingface.co/xtuner/llava-llama-3-8b-v1_1-pretrain) / 🤖 [ModelScope](https://modelscope.cn/models/xtuner/llava-llama-3-8b-v1_1-pretrain)\n\n- LLaVA-Llama-3-8B\n\n - Official LLaVA format model (`xtuner/llava-llama-3-8b-hf`): 🤗 [HuggingFace](https://huggingface.co/xtuner/llava-llama-3-8b-hf) / 🤖 [ModelScope](https://modelscope.cn/models/xtuner/llava-llama-3-8b-hf)\n - HuggingFace LLaVA format model (`xtuner/llava-llama-3-8b-transformers`): 🤗 [HuggingFace](https://huggingface.co/xtuner/llava-llama-3-8b-transformers) / 🤖 [ModelScope](https://modelscope.cn/models/xtuner/llava-llama-3-8b-transformers)\n - XTuner LLaVA format model (`xtuner/llava-llama-3-8b`): 🤗 [HuggingFace](https://huggingface.co/xtuner/llava-llama-3-8b) / 🤖 [ModelScope](https://modelscope.cn/models/xtuner/llava-llama-3-8b)\n - Pretrained projector weights: 🤗 [HuggingFace](https://huggingface.co/xtuner/llava-llama-3-8b-pretrain) / 🤖 [ModelScope](https://modelscope.cn/models/xtuner/llava-llama-3-8b-pretrain)\n\n## Data Preparation\n\n### LLaVA dataset\n\n#### File structure\n\n```\n./data/llava_data\n├── LLaVA-Pretrain\n│ ├── blip_laion_cc_sbu_558k.json\n│ ├── blip_laion_cc_sbu_558k_meta.json\n│ └── images\n├── LLaVA-Instruct-150K\n│ └── llava_v1_5_mix665k.json\n└── llava_images\n ├── coco\n │ └── train2017\n ├── gqa\n │ └── images\n ├── ocr_vqa\n │ └── images\n ├── textvqa\n │ └── train_images\n └── vg\n ├── VG_100K\n └── VG_100K_2\n```\n\n#### Pretrain\n\nLLaVA-Pretrain\n\n```shell\n# Make sure you have git-lfs installed (https://git-lfs.com)\ngit lfs install\ngit clone https://huggingface.co/datasets/liuhaotian/LLaVA-Pretrain --depth=1\n```\n\n#### Finetune\n\n1. Text data\n\n 1. LLaVA-Instruct-150K\n\n ```shell\n # Make sure you have git-lfs installed (https://git-lfs.com)\n git lfs install\n git clone https://huggingface.co/datasets/liuhaotian/LLaVA-Instruct-150K --depth=1\n ```\n\n2. Image data\n\n 1. COCO (coco): [download url](http://images.cocodataset.org/zips/train2017.zip)\n\n 2. GQA (gqa): [download url](https://downloads.cs.stanford.edu/nlp/data/gqa/images.zip)\n\n 3. OCR-VQA (ocr_vqa): [download script](https://drive.google.com/drive/folders/1_GYPY5UkUy7HIcR0zq3ZCFgeZN7BAfm_?usp=sharing)\n\n 1. ⚠️ Modify the name of OCR-VQA's images to keep the extension as `.jpg`!\n\n ```shell\n #!/bin/bash\n ocr_vqa_path=\"\n![\"Image\"](\"https://github.com/InternLM/xtuner/assets/36994684/78524f65-260d-4ae3-a687-03fc5a19dcbb\")
\n
\n\n| Model | MMBench Test (EN) | MMMU Val | SEED-IMG | AI2D Test | ScienceQA Test | HallusionBench aAcc | POPE | GQA | TextVQA | MME | MMStar | Configs |\n| :-------------------- | :---------------: | :-------: | :------: | :-------: | :------------: | :-----------------: | :--: | :--: | :-----: | :------: | :----: | :----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------: |\n| LLaVA-v1.5-7B | 66.5 | 35.3 | 60.5 | 54.8 | 70.4 | 44.9 | 85.9 | 62.0 | 58.2 | 1511/348 | 30.3 | - |\n| LLaVA-Llama-3-8B | 68.9 | 36.8 | 69.8 | 60.9 | 73.3 | 47.3 | 87.2 | 63.5 | 58.0 | 1506/295 | 38.2 | [Pretrain](https://github.com/InternLM/xtuner/blob/main/xtuner/configs/llava/llama3_8b_instruct_clip_vit_large_p14_336/pretrain/llava_llama3_8b_instruct_clip_vit_large_p14_336_e1_gpu8_pretrain.py) / [Fine-tune](https://github.com/InternLM/xtuner/blob/main/xtuner/configs/llava/llama3_8b_instruct_clip_vit_large_p14_336/finetune/llava_llama3_8b_instruct_full_clip_vit_large_p14_336_lora_e1_gpu8_finetune.py) |\n| LLaVA-Llama-3-8B-v1.1 | 72.3 | 37.1 | 70.1 | 70.0 | 72.9 | 47.7 | 86.4 | 62.6 | 59.0 | 1469/349 | 45.1 | [Pretrain](https://github.com/InternLM/xtuner/blob/main/xtuner/configs/llava/llama3_8b_instruct_clip_vit_large_p14_336/pretrain/llava_llama3_8b_instruct_clip_vit_large_p14_336_e1_gpu8_sharegpt4v_pretrain.py) / [Fine-tune](https://github.com/InternLM/xtuner/blob/main/xtuner/configs/llava/llama3_8b_instruct_clip_vit_large_p14_336/finetune/llava_llama3_8b_instruct_full_clip_vit_large_p14_336_lora_e1_gpu8_internvl_finetune.py) |\n| **LLaVA-Phi-3-mini** | 69.2 | 41.4 | 70.0 | 69.3 | 73.7 | 49.8 | 87.3 | 61.5 | 57.8 | 1477/313 | 43.7 | [Pretrain](./pretrain/llava_phi3_mini_4k_instruct_clip_vit_large_p14_336_e1_gpu8_sharegpt4v_pretrain.py) / [Fine-tune](./finetune/llava_phi3_mini_4k_instruct_full_clip_vit_large_p14_336_full_e2_gpu8_internvl_finetune.py) |\n\n## Resources\n\n- Official LLaVA format model (`xtuner/llava-phi-3-mini`): 🤗 [HuggingFace](https://huggingface.co/xtuner/llava-phi-3-mini) / 🤖 [ModelScope](https://modelscope.cn/models/xtuner/llava-phi-3-mini)\n- HuggingFace LLaVA format model (`xtuner/llava-phi-3-mini-hf`): 🤗 [HuggingFace](https://huggingface.co/xtuner/llava-phi-3-mini-hf) / 🤖 [ModelScope](https://modelscope.cn/models/xtuner/llava-phi-3-mini-hf)\n- XTuner LLaVA format model (`xtuner/llava-phi-3-mini-xtuner`): 🤗 [HuggingFace](https://huggingface.co/xtuner/llava-phi-3-mini-xtuner) / 🤖 [ModelScope](https://modelscope.cn/models/xtuner/llava-phi-3-mini-xtuner)\n- GGUF model (`xtuner/llava-phi-3-mini-gguf`): 🤗 [HuggingFace](https://huggingface.co/xtuner/llava-phi-3-mini-gguf) / 🤖 [ModelScope](https://modelscope.cn/models/xtuner/llava-phi-3-mini-gguf)\n- Pretrained projector weights: 🤗 [HuggingFace](https://huggingface.co/xtuner/llava-phi-3-mini-pretrain) / 🤖 [ModelScope](https://modelscope.cn/models/xtuner/llava-phi-3-mini-pretrain)\n\n## Data Preparation\n\nPlease refer to [here](https://github.com/InternLM/xtuner/tree/main/xtuner/configs/llava/llama3_8b_instruct_clip_vit_large_p14_336#data-preparation).\n\n## Training\n\n### LLaVA-Phi-3-mini\n\n1. Pretrain\n\n```bash\nNPROC_PER_NODE=8 xtuner train llava_phi3_mini_4k_instruct_clip_vit_large_p14_336_e1_gpu8_sharegpt4v_pretrain --deepspeed deepspeed_zero2 --seed 1024\n```\n\n2. Fine-tune\n\n```bash\nNPROC_PER_NODE=8 xtuner train llava_phi3_mini_4k_instruct_full_clip_vit_large_p14_336_full_e2_gpu8_internvl_finetune --deepspeed deepspeed_zero2 --seed 1024\n```\n\n## Model Conversion\n\n### Step 0. Convert `.pth` file to LLaVA model in xtuner format ([LLaVA-Phi-3-mini-xtuner](https://huggingface.co/xtuner/llava-phi-3-mini-xtuner))\n\nAfter training, we will obtain a set of weights (*i.e.*, `iter_xxx.pth`), which are not in the universal HuggingFace format. We first need to convert them to the LLaVA model in xtuner format.\n\n```bash\nxtuner convert pth_to_hf $FINETUNE_CFG $PTH_PATH $SAVE_PATH\n# e.g., xtuner convert pth_to_hf llava_phi3_mini_4k_instruct_full_clip_vit_large_p14_336_full_e2_gpu8_internvl_finetune ./iter_39620.pth ./iter_39620_xtuner\n```\n\n```\n./iter_39620_xtuner\n├── added_tokens.json\n├── config.json\n├── model-00001-of-00004.safetensors\n├── model-00002-of-00004.safetensors\n├── model-00003-of-00004.safetensors\n├── model-00004-of-00004.safetensors\n├── model.safetensors.index.json\n├── projector\n│ ├── config.json\n│ ├── configuration_projector.py\n│ ├── modeling_projector.py\n│ └── model.safetensors\n├── special_tokens_map.json\n├── tokenizer_config.json\n├── tokenizer.json\n├── tokenizer.model\n└── visual_encoder\n ├── config.json\n ├── model.safetensors\n └── preprocessor_config.json\n```\n\nAt this time, the LLaVA model of xtuner-format can engage in conversation using xtuner chat, by\n\n```bash\nxtuner chat ./iter_39620_xtuner \\\n --llava ./iter_39620_xtuner \\\n --prompt-template phi3_chat \\\n --image $IMAGE_PATH\n```\n\nand in MMBench evaluation, by\n\n```bash\nxtuner mmbench ./iter_39620_xtuner \\\n --llava ./iter_39620_xtuner \\\n --prompt-template phi3_chat \\\n --data-path $DATA_PATH \\\n --work-dir $RESULT_PATH\n```\n\nHere, `$DATA_PATH` refers to one of the mmbench datasets. You can download the expected data by\n\n```bash\nwget https://opencompass.openxlab.space/utils/VLMEval/MMBench_DEV_EN.tsv\nwget https://opencompass.openxlab.space/utils/VLMEval/MMBench_TEST_EN.tsv\nwget https://opencompass.openxlab.space/utils/VLMEval/MMBench_DEV_CN.tsv\nwget https://opencompass.openxlab.space/utils/VLMEval/MMBench_TEST_CN.tsv\nwget https://opencompass.openxlab.space/utils/VLMEval/CCBench.tsv\n```\n\n### Step 1. Convert LLaVA in xtuner format to official LLaVA format or HuggingFace LLaVA format\n\n- The official LLaVA format is structured similarly to the architecture of the [liuhaotian/llava-v1.5-7b](https://huggingface.co/liuhaotian/llava-v1.5-7b) model.\n- The HuggingFace LLaVA format is structured similarly to the architecture of the [llava-hf/llava-1.5-7b-hf](https://huggingface.co/llava-hf/llava-1.5-7b-hf) model.\n\nSince the official LLaVA format and the HuggingFace LLaVA format only support Llama architecture as the LLM, we need to first convert the phi-3 model to an equivalent Llama LLM.\n\n```bash\npython ./convert_phi_to_llama.py --phi_path ./iter_39620_xtuner --save_path ./iter_39620_xtuner_llama_llm\n```\n\nHere, `--phi_path` should specify the path to phi-3, which is the path obtained from Step.0 for the xtuner-format LLaVA model. `--save_path` should specify the save path for the converted Llama LLM.\n\n#### To official LLaVA format ([LLaVA-Phi-3-mini](https://huggingface.co/xtuner/llava-phi-3-mini))\n\nWe can utilize the following command to obtain the LLaVA model in the official LLaVA format.\n\n```bash\npython ./convert_xtuner_weights_to_llava.py --text_model_id ./iter_39620_xtuner_llama_llm --vision_model_id ./iter_39620_xtuner/visual_encoder --projector_weight ./iter_39620_xtuner/projector/model.safetensors --save_path ./iter_39620_llava\n```\n\nHere, the converted LLaVA model in official LLaVA format is saved to `./iter_39620_llava`.\n\n```\n./iter_39620_llava\n├── added_tokens.json\n├── config.json\n├── generation_config.json\n├── model-00001-of-00005.safetensors\n├── model-00002-of-00005.safetensors\n├── model-00003-of-00005.safetensors\n├── model-00004-of-00005.safetensors\n├── model-00005-of-00005.safetensors\n├── model.safetensors.index.json\n├── preprocessor_config.json\n├── special_tokens_map.json\n├── tokenizer_config.json\n├── tokenizer.json\n└── tokenizer.model\n```\n\n#### To HuggingFace LLaVA format ([LLaVA-Phi-3-mini-hf](https://huggingface.co/xtuner/llava-phi-3-mini-hf))\n\nWe can utilize the following command to obtain the LLaVA model in the HuggingFace LLaVA format.\n\n```bash\npython ./convert_xtuner_weights_to_hf.py --text_model_id ./iter_39620_xtuner_llama_llm --vision_model_id ./iter_39620_xtuner/visual_encoder --projector_weight ./iter_39620_xtuner/projector/model.safetensors --save_path ./iter_39620_hf\n```\n\nHere, the converted LLaVA model in HuggingFace LLaVA format is saved to `./iter_39620_hf`.\n\n```\n./iter_39620_hf\n├── added_tokens.json\n├── config.json\n├── generation_config.json\n├── model-00001-of-00002.safetensors\n├── model-00002-of-00002.safetensors\n├── model.safetensors.index.json\n├── preprocessor_config.json\n├── special_tokens_map.json\n├── tokenizer_config.json\n├── tokenizer.json\n└── tokenizer.model\n```\n\n## Chat\n\n- XTuner LLaVA format [docs](https://huggingface.co/xtuner/llava-phi-3-mini-xtuner#quickstart)\n- Official LLaVA format [docs](https://huggingface.co/xtuner/llava-phi-3-mini#quickstart)\n- HuggingFace LLaVA format [docs](https://huggingface.co/xtuner/llava-phi-3-mini-hf#quickstart)\n- GGUF format [docs](https://huggingface.co/xtuner/llava-phi-3-mini-gguf#quickstart)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/llava/phi3_mini_4k_instruct_clip_vit_large_p14_336/convert_phi_to_llama.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nimport argparse\nimport json\nimport os\n\nfrom mmengine.utils import mkdir_or_exist\nfrom safetensors import safe_open\nfrom safetensors.torch import save_file\nfrom tqdm import tqdm\nfrom transformers import AutoTokenizer\n\n\ndef convert_phi_to_llama(phi_path, save_path):\n files = [f for f in os.listdir(phi_path) if f.endswith('safetensors')]\n mkdir_or_exist(save_path)\n\n index_json = os.path.join(phi_path, 'model.safetensors.index.json')\n config_json = os.path.join(phi_path, 'config.json')\n\n with open(index_json) as f:\n index = json.load(f)\n\n with open(config_json) as f:\n config = json.load(f)\n\n config.pop('_name_or_path')\n if 'auto_map' in config:\n config.pop('auto_map')\n config.pop('embd_pdrop')\n config.pop('resid_pdrop')\n config['architectures'] = ['LlamaForCausalLM']\n config['model_type'] = 'llama'\n\n for file in tqdm(files, desc='Convert'):\n tensors = {}\n new_path = os.path.join(save_path, file)\n old_path = os.path.join(phi_path, file)\n with safe_open(old_path, framework='pt', device='cpu') as f:\n for key in f.keys():\n\n if 'qkv_proj' in key:\n qkv = f.get_tensor(key)\n\n q, k, v = qkv.chunk(3, dim=0)\n q_name = key.replace('qkv_proj', 'q_proj')\n k_name = key.replace('qkv_proj', 'k_proj')\n v_name = key.replace('qkv_proj', 'v_proj')\n\n tensors[q_name] = q\n tensors[k_name] = k\n tensors[v_name] = v\n\n index['weight_map'].pop(key)\n\n filename = os.path.basename(new_path)\n index['weight_map'][q_name] = filename\n index['weight_map'][k_name] = filename\n index['weight_map'][v_name] = filename\n\n elif 'gate_up_proj' in key:\n gate_up_proj = f.get_tensor(key)\n gate_proj, up_proj = gate_up_proj.chunk(2, dim=0)\n\n gate_name = key.replace('gate_up_proj', 'gate_proj')\n up_name = key.replace('gate_up_proj', 'up_proj')\n tensors[gate_name] = gate_proj\n tensors[up_name] = up_proj\n\n index['weight_map'].pop(key)\n filename = os.path.basename(new_path)\n index['weight_map'][gate_name] = filename\n index['weight_map'][up_name] = filename\n else:\n tensors[key] = f.get_tensor(key)\n metadata = f.metadata()\n save_file(tensors, new_path, metadata=metadata)\n\n new_config_json = os.path.join(save_path, 'config.json')\n with open(new_config_json, 'w') as f:\n json.dump(config, f, indent=2)\n\n new_index_json = os.path.join(save_path, 'model.safetensors.index.json')\n with open(new_index_json, 'w') as f:\n json.dump(index, f, indent=2)\n\n tokenizer = AutoTokenizer.from_pretrained(phi_path, trust_remote_code=True)\n tokenizer.save_pretrained(save_path)\n print(f'Saved to {save_path}')\n\n\ndef main():\n parser = argparse.ArgumentParser()\n parser.add_argument('--phi_path')\n parser.add_argument('--save_path')\n args = parser.parse_args()\n convert_phi_to_llama(args.phi_path, args.save_path)\n\n\nif __name__ == '__main__':\n main()\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/llava/phi3_mini_4k_instruct_clip_vit_large_p14_336/convert_xtuner_weights_to_hf.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\n# Modified from https://github.com/huggingface/transformers/blob/v4.40.1/src/transformers/models/llava/convert_llava_weights_to_hf.py # noqa: E501\nimport argparse\n\nimport torch\nfrom safetensors import safe_open\nfrom transformers import (AddedToken, AutoConfig, AutoModel,\n AutoModelForCausalLM, CLIPImageProcessor,\n LlamaTokenizerFast, LlavaConfig,\n LlavaForConditionalGeneration, LlavaProcessor)\n\nKEYS_TO_MODIFY_MAPPING_LLM = {\n 'model': 'language_model.model',\n 'lm_head': 'language_model.lm_head',\n}\nKEYS_TO_MODIFY_MAPPING_VIT = {\n 'vision_model': 'vision_tower.vision_model',\n}\nKEYS_TO_MODIFY_MAPPING_PROJECTOR = {\n 'model.0': 'multi_modal_projector.linear_1',\n 'model.2': 'multi_modal_projector.linear_2',\n}\n\n\ndef convert_state_dict_to_hf(state_dict, mapping):\n new_state_dict = {}\n for key, value in state_dict.items():\n if key.endswith('.inv_freq'):\n continue\n for key_to_modify, new_key in mapping.items():\n if key_to_modify in key:\n key = key.replace(key_to_modify, new_key)\n\n new_state_dict[key] = value\n return new_state_dict\n\n\ndef convert_to_hf(text_model_id, vision_model_id, projector_weight, save_path):\n torch.set_default_dtype(torch.float16)\n text_config = AutoConfig.from_pretrained(\n text_model_id, trust_remote_code=True)\n vision_config = AutoConfig.from_pretrained(vision_model_id)\n\n tokenizer = LlamaTokenizerFast.from_pretrained(text_model_id)\n tokenizer.add_tokens(\n AddedToken('\n ![](\"https://github.com/InternLM/xtuner/assets/41630003/48e4b6e3-1bcd-4349-90f0-dbbbc0f1cee7\")
\n
\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/qwen/qwen1_5/qwen1_5_110b_chat/qwen1_5_110b_chat_full_alpaca_e3.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom torch.optim import AdamW\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nfrom xtuner.dataset import process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import alpaca_map_fn, template_map_fn_factory\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n ThroughputHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'Qwen/Qwen1.5-110B-Chat'\nuse_varlen_attn = False\n\n# Data\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.qwen_chat\nmax_length = 2048\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\n# total batch = 32gpus * batch_size_per_device 1 * acc 1 = 32\naccumulative_counts = 1\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 4\nmax_epochs = 3\noptim_type = AdamW\nlr = 1e-5\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 500\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=alpaca_en,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template),\n dict(type=ThroughputHook)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/qwen/qwen1_5/qwen1_5_110b_chat/qwen1_5_110b_chat_qlora_alpaca_e3.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nimport torch\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom peft import LoraConfig\nfrom torch.optim import AdamW\nfrom transformers import (AutoModelForCausalLM, AutoTokenizer,\n BitsAndBytesConfig)\n\nfrom xtuner.dataset import process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import alpaca_map_fn, template_map_fn_factory\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n ThroughputHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'Qwen/Qwen1.5-110B-Chat'\nuse_varlen_attn = False\n\n# Data\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.qwen_chat\nmax_length = 2048\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 1 # total bs = 1 bs_per_device * 8 gpus * 1 acc = 8\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 3\noptim_type = AdamW\nlr = 1e-4 # 110B model use smaller lr\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 500\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n torch_dtype=torch.float16,\n quantization_config=dict(\n type=BitsAndBytesConfig,\n load_in_4bit=True,\n load_in_8bit=False,\n llm_int8_threshold=6.0,\n llm_int8_has_fp16_weight=False,\n bnb_4bit_compute_dtype=torch.float16,\n bnb_4bit_use_double_quant=True,\n bnb_4bit_quant_type='nf4',\n bnb_4bit_quant_storage=torch.float16)),\n lora=dict(\n type=LoraConfig,\n r=64,\n lora_alpha=16,\n lora_dropout=0.1,\n bias='none',\n task_type='CAUSAL_LM'))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=alpaca_en,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(type=ThroughputHook),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/qwen/qwen1_5/qwen1_5_110b_chat/qwen1_5_110b_chat_qlora_alpaca_e3_16k_2gpus.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nimport torch\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom peft import LoraConfig\nfrom torch.optim import AdamW\nfrom transformers import (AutoModelForCausalLM, AutoTokenizer,\n BitsAndBytesConfig)\n\nfrom xtuner.dataset import process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import alpaca_map_fn, template_map_fn_factory\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n ThroughputHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'Qwen/Qwen1.5-110B-Chat'\nuse_varlen_attn = False\n\n# Data\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.qwen_chat\nmax_length = 16384\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 2\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 1 # total bs = 1 bs_per_device * 2 gpus * 1 acc = 2\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 3\noptim_type = AdamW\nlr = 1e-4 # 110B model use smaller lr\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 50\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n torch_dtype=torch.float16,\n quantization_config=dict(\n type=BitsAndBytesConfig,\n load_in_4bit=True,\n load_in_8bit=False,\n llm_int8_threshold=6.0,\n llm_int8_has_fp16_weight=False,\n bnb_4bit_compute_dtype=torch.float16,\n bnb_4bit_use_double_quant=True,\n bnb_4bit_quant_type='nf4',\n bnb_4bit_quant_storage=torch.float16)),\n lora=dict(\n type=LoraConfig,\n r=64,\n lora_alpha=16,\n lora_dropout=0.1,\n bias='none',\n task_type='CAUSAL_LM'))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=alpaca_en,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(type=ThroughputHook),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=1),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False, window_size=1)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/qwen/qwen1_5/qwen1_5_14b/qwen1_5_14b_full_alpaca_e3.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom torch.optim import AdamW\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nfrom xtuner.dataset import process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import alpaca_map_fn, template_map_fn_factory\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'Qwen/Qwen1.5-14B'\nuse_varlen_attn = False\n\n# Data\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.default\nmax_length = 2048\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 3\noptim_type = AdamW\nlr = 2e-5\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 500\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=alpaca_en,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/qwen/qwen1_5/qwen1_5_14b/qwen1_5_14b_qlora_alpaca_e3.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nimport torch\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom peft import LoraConfig\nfrom torch.optim import AdamW\nfrom transformers import (AutoModelForCausalLM, AutoTokenizer,\n BitsAndBytesConfig)\n\nfrom xtuner.dataset import process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import alpaca_map_fn, template_map_fn_factory\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'Qwen/Qwen1.5-14B'\nuse_varlen_attn = False\n\n# Data\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.default\nmax_length = 2048\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 3\noptim_type = AdamW\nlr = 2e-4\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 500\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n torch_dtype=torch.float16,\n quantization_config=dict(\n type=BitsAndBytesConfig,\n load_in_4bit=True,\n load_in_8bit=False,\n llm_int8_threshold=6.0,\n llm_int8_has_fp16_weight=False,\n bnb_4bit_compute_dtype=torch.float16,\n bnb_4bit_use_double_quant=True,\n bnb_4bit_quant_type='nf4')),\n lora=dict(\n type=LoraConfig,\n r=64,\n lora_alpha=16,\n lora_dropout=0.1,\n bias='none',\n task_type='CAUSAL_LM'))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=alpaca_en,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/qwen/qwen1_5/qwen1_5_14b_chat/qwen1_5_14b_chat_full_alpaca_e3.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom torch.optim import AdamW\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nfrom xtuner.dataset import process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import alpaca_map_fn, template_map_fn_factory\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'Qwen/Qwen1.5-14B-Chat'\nuse_varlen_attn = False\n\n# Data\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.qwen_chat\nmax_length = 2048\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 3\noptim_type = AdamW\nlr = 2e-5\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 500\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=alpaca_en,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/qwen/qwen1_5/qwen1_5_14b_chat/qwen1_5_14b_chat_qlora_alpaca_e3.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nimport torch\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom peft import LoraConfig\nfrom torch.optim import AdamW\nfrom transformers import (AutoModelForCausalLM, AutoTokenizer,\n BitsAndBytesConfig)\n\nfrom xtuner.dataset import process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import alpaca_map_fn, template_map_fn_factory\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'Qwen/Qwen1.5-14B-Chat'\nuse_varlen_attn = False\n\n# Data\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.qwen_chat\nmax_length = 2048\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 3\noptim_type = AdamW\nlr = 2e-4\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 500\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n torch_dtype=torch.float16,\n quantization_config=dict(\n type=BitsAndBytesConfig,\n load_in_4bit=True,\n load_in_8bit=False,\n llm_int8_threshold=6.0,\n llm_int8_has_fp16_weight=False,\n bnb_4bit_compute_dtype=torch.float16,\n bnb_4bit_use_double_quant=True,\n bnb_4bit_quant_type='nf4')),\n lora=dict(\n type=LoraConfig,\n r=64,\n lora_alpha=16,\n lora_dropout=0.1,\n bias='none',\n task_type='CAUSAL_LM'))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=alpaca_en,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/qwen/qwen1_5/qwen1_5_1_8b/qwen1_5_1_8b_full_alpaca_e3.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom torch.optim import AdamW\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nfrom xtuner.dataset import process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import alpaca_map_fn, template_map_fn_factory\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'Qwen/Qwen1.5-1.8B'\nuse_varlen_attn = False\n\n# Data\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.default\nmax_length = 2048\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 3\noptim_type = AdamW\nlr = 2e-5\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 500\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=alpaca_en,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/qwen/qwen1_5/qwen1_5_1_8b/qwen1_5_1_8b_qlora_alpaca_e3.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nimport torch\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom peft import LoraConfig\nfrom torch.optim import AdamW\nfrom transformers import (AutoModelForCausalLM, AutoTokenizer,\n BitsAndBytesConfig)\n\nfrom xtuner.dataset import process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import alpaca_map_fn, template_map_fn_factory\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'Qwen/Qwen1.5-1.8B'\nuse_varlen_attn = False\n\n# Data\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.default\nmax_length = 2048\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 3\noptim_type = AdamW\nlr = 2e-4\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 500\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n torch_dtype=torch.float16,\n quantization_config=dict(\n type=BitsAndBytesConfig,\n load_in_4bit=True,\n load_in_8bit=False,\n llm_int8_threshold=6.0,\n llm_int8_has_fp16_weight=False,\n bnb_4bit_compute_dtype=torch.float16,\n bnb_4bit_use_double_quant=True,\n bnb_4bit_quant_type='nf4')),\n lora=dict(\n type=LoraConfig,\n r=64,\n lora_alpha=16,\n lora_dropout=0.1,\n bias='none',\n task_type='CAUSAL_LM'))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=alpaca_en,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/qwen/qwen1_5/qwen1_5_1_8b_chat/qwen1_5_1_8b_chat_full_alpaca_e3.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom torch.optim import AdamW\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nfrom xtuner.dataset import process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import alpaca_map_fn, template_map_fn_factory\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'Qwen/Qwen1.5-1.8B-Chat'\nuse_varlen_attn = False\n\n# Data\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.qwen_chat\nmax_length = 2048\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 3\noptim_type = AdamW\nlr = 2e-5\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 500\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=alpaca_en,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/qwen/qwen1_5/qwen1_5_1_8b_chat/qwen1_5_1_8b_chat_qlora_alpaca_e3.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nimport torch\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom peft import LoraConfig\nfrom torch.optim import AdamW\nfrom transformers import (AutoModelForCausalLM, AutoTokenizer,\n BitsAndBytesConfig)\n\nfrom xtuner.dataset import process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import alpaca_map_fn, template_map_fn_factory\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'Qwen/Qwen1.5-1.8B-Chat'\nuse_varlen_attn = False\n\n# Data\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.qwen_chat\nmax_length = 2048\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 3\noptim_type = AdamW\nlr = 2e-4\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 500\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n torch_dtype=torch.float16,\n quantization_config=dict(\n type=BitsAndBytesConfig,\n load_in_4bit=True,\n load_in_8bit=False,\n llm_int8_threshold=6.0,\n llm_int8_has_fp16_weight=False,\n bnb_4bit_compute_dtype=torch.float16,\n bnb_4bit_use_double_quant=True,\n bnb_4bit_quant_type='nf4')),\n lora=dict(\n type=LoraConfig,\n r=64,\n lora_alpha=16,\n lora_dropout=0.1,\n bias='none',\n task_type='CAUSAL_LM'))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=alpaca_en,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/qwen/qwen1_5/qwen1_5_4b/qwen1_5_4b_full_alpaca_e3.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom torch.optim import AdamW\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nfrom xtuner.dataset import process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import alpaca_map_fn, template_map_fn_factory\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'Qwen/Qwen1.5-4B'\nuse_varlen_attn = False\n\n# Data\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.default\nmax_length = 2048\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 3\noptim_type = AdamW\nlr = 2e-5\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 500\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=alpaca_en,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/qwen/qwen1_5/qwen1_5_4b/qwen1_5_4b_qlora_alpaca_e3.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nimport torch\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom peft import LoraConfig\nfrom torch.optim import AdamW\nfrom transformers import (AutoModelForCausalLM, AutoTokenizer,\n BitsAndBytesConfig)\n\nfrom xtuner.dataset import process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import alpaca_map_fn, template_map_fn_factory\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'Qwen/Qwen1.5-4B'\nuse_varlen_attn = False\n\n# Data\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.default\nmax_length = 2048\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 3\noptim_type = AdamW\nlr = 2e-4\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 500\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n torch_dtype=torch.float16,\n quantization_config=dict(\n type=BitsAndBytesConfig,\n load_in_4bit=True,\n load_in_8bit=False,\n llm_int8_threshold=6.0,\n llm_int8_has_fp16_weight=False,\n bnb_4bit_compute_dtype=torch.float16,\n bnb_4bit_use_double_quant=True,\n bnb_4bit_quant_type='nf4')),\n lora=dict(\n type=LoraConfig,\n r=64,\n lora_alpha=16,\n lora_dropout=0.1,\n bias='none',\n task_type='CAUSAL_LM'))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=alpaca_en,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/qwen/qwen1_5/qwen1_5_4b_chat/qwen1_5_4b_chat_full_alpaca_e3.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom torch.optim import AdamW\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nfrom xtuner.dataset import process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import alpaca_map_fn, template_map_fn_factory\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'Qwen/Qwen1.5-4B-Chat'\nuse_varlen_attn = False\n\n# Data\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.qwen_chat\nmax_length = 2048\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 3\noptim_type = AdamW\nlr = 2e-5\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 500\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=alpaca_en,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/qwen/qwen1_5/qwen1_5_4b_chat/qwen1_5_4b_chat_qlora_alpaca_e3.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nimport torch\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom peft import LoraConfig\nfrom torch.optim import AdamW\nfrom transformers import (AutoModelForCausalLM, AutoTokenizer,\n BitsAndBytesConfig)\n\nfrom xtuner.dataset import process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import alpaca_map_fn, template_map_fn_factory\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'Qwen/Qwen1.5-4B-Chat'\nuse_varlen_attn = False\n\n# Data\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.qwen_chat\nmax_length = 2048\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 3\noptim_type = AdamW\nlr = 2e-4\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 500\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n torch_dtype=torch.float16,\n quantization_config=dict(\n type=BitsAndBytesConfig,\n load_in_4bit=True,\n load_in_8bit=False,\n llm_int8_threshold=6.0,\n llm_int8_has_fp16_weight=False,\n bnb_4bit_compute_dtype=torch.float16,\n bnb_4bit_use_double_quant=True,\n bnb_4bit_quant_type='nf4')),\n lora=dict(\n type=LoraConfig,\n r=64,\n lora_alpha=16,\n lora_dropout=0.1,\n bias='none',\n task_type='CAUSAL_LM'))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=alpaca_en,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/qwen/qwen1_5/qwen1_5_72b/qwen1_5_72b_full_alpaca_e3.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom torch.optim import AdamW\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nfrom xtuner.dataset import process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import alpaca_map_fn, template_map_fn_factory\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'Qwen/Qwen1.5-72B'\nuse_varlen_attn = False\n\n# Data\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.default\nmax_length = 2048\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 3\noptim_type = AdamW\nlr = 2e-5\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 500\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=alpaca_en,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/qwen/qwen1_5/qwen1_5_72b/qwen1_5_72b_qlora_alpaca_e3.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nimport torch\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom peft import LoraConfig\nfrom torch.optim import AdamW\nfrom transformers import (AutoModelForCausalLM, AutoTokenizer,\n BitsAndBytesConfig)\n\nfrom xtuner.dataset import process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import alpaca_map_fn, template_map_fn_factory\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'Qwen/Qwen1.5-72B'\nuse_varlen_attn = False\n\n# Data\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.default\nmax_length = 2048\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 3\noptim_type = AdamW\nlr = 2e-4\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 500\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n torch_dtype=torch.float16,\n quantization_config=dict(\n type=BitsAndBytesConfig,\n load_in_4bit=True,\n load_in_8bit=False,\n llm_int8_threshold=6.0,\n llm_int8_has_fp16_weight=False,\n bnb_4bit_compute_dtype=torch.float16,\n bnb_4bit_use_double_quant=True,\n bnb_4bit_quant_type='nf4')),\n lora=dict(\n type=LoraConfig,\n r=64,\n lora_alpha=16,\n lora_dropout=0.1,\n bias='none',\n task_type='CAUSAL_LM'))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=alpaca_en,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/qwen/qwen1_5/qwen1_5_72b_chat/qwen1_5_72b_chat_full_alpaca_e3.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom torch.optim import AdamW\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nfrom xtuner.dataset import process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import alpaca_map_fn, template_map_fn_factory\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'Qwen/Qwen1.5-72B-Chat'\nuse_varlen_attn = False\n\n# Data\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.qwen_chat\nmax_length = 2048\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 3\noptim_type = AdamW\nlr = 2e-5\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 500\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=alpaca_en,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/qwen/qwen1_5/qwen1_5_72b_chat/qwen1_5_72b_chat_qlora_alpaca_e3.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nimport torch\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom peft import LoraConfig\nfrom torch.optim import AdamW\nfrom transformers import (AutoModelForCausalLM, AutoTokenizer,\n BitsAndBytesConfig)\n\nfrom xtuner.dataset import process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import alpaca_map_fn, template_map_fn_factory\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'Qwen/Qwen1.5-72B-Chat'\nuse_varlen_attn = False\n\n# Data\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.qwen_chat\nmax_length = 2048\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 3\noptim_type = AdamW\nlr = 2e-4\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 500\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n torch_dtype=torch.float16,\n quantization_config=dict(\n type=BitsAndBytesConfig,\n load_in_4bit=True,\n load_in_8bit=False,\n llm_int8_threshold=6.0,\n llm_int8_has_fp16_weight=False,\n bnb_4bit_compute_dtype=torch.float16,\n bnb_4bit_use_double_quant=True,\n bnb_4bit_quant_type='nf4')),\n lora=dict(\n type=LoraConfig,\n r=64,\n lora_alpha=16,\n lora_dropout=0.1,\n bias='none',\n task_type='CAUSAL_LM'))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=alpaca_en,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/qwen/qwen1_5/qwen1_5_7b/qwen1_5_7b_full_alpaca_e3.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom torch.optim import AdamW\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nfrom xtuner.dataset import process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import alpaca_map_fn, template_map_fn_factory\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'Qwen/Qwen1.5-7B'\nuse_varlen_attn = False\n\n# Data\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.default\nmax_length = 2048\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 3\noptim_type = AdamW\nlr = 2e-5\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 500\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=alpaca_en,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/qwen/qwen1_5/qwen1_5_7b/qwen1_5_7b_qlora_alpaca_e3.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nimport torch\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom peft import LoraConfig\nfrom torch.optim import AdamW\nfrom transformers import (AutoModelForCausalLM, AutoTokenizer,\n BitsAndBytesConfig)\n\nfrom xtuner.dataset import process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import alpaca_map_fn, template_map_fn_factory\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'Qwen/Qwen1.5-7B'\nuse_varlen_attn = False\n\n# Data\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.default\nmax_length = 2048\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 3\noptim_type = AdamW\nlr = 2e-4\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 500\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n torch_dtype=torch.float16,\n quantization_config=dict(\n type=BitsAndBytesConfig,\n load_in_4bit=True,\n load_in_8bit=False,\n llm_int8_threshold=6.0,\n llm_int8_has_fp16_weight=False,\n bnb_4bit_compute_dtype=torch.float16,\n bnb_4bit_use_double_quant=True,\n bnb_4bit_quant_type='nf4')),\n lora=dict(\n type=LoraConfig,\n r=64,\n lora_alpha=16,\n lora_dropout=0.1,\n bias='none',\n task_type='CAUSAL_LM'))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=alpaca_en,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/qwen/qwen1_5/qwen1_5_7b_chat/qwen1_5_7b_chat_full_alpaca_e3.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom torch.optim import AdamW\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nfrom xtuner.dataset import process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import alpaca_map_fn, template_map_fn_factory\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'Qwen/Qwen1.5-7B-Chat'\nuse_varlen_attn = False\n\n# Data\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.qwen_chat\nmax_length = 2048\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 3\noptim_type = AdamW\nlr = 2e-5\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 500\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=alpaca_en,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/qwen/qwen1_5/qwen1_5_7b_chat/qwen1_5_7b_chat_qlora_alpaca_e3.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nimport torch\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom peft import LoraConfig\nfrom torch.optim import AdamW\nfrom transformers import (AutoModelForCausalLM, AutoTokenizer,\n BitsAndBytesConfig)\n\nfrom xtuner.dataset import process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import alpaca_map_fn, template_map_fn_factory\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'Qwen/Qwen1.5-7B-Chat'\nuse_varlen_attn = False\n\n# Data\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.qwen_chat\nmax_length = 2048\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 3\noptim_type = AdamW\nlr = 2e-4\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 500\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n torch_dtype=torch.float16,\n quantization_config=dict(\n type=BitsAndBytesConfig,\n load_in_4bit=True,\n load_in_8bit=False,\n llm_int8_threshold=6.0,\n llm_int8_has_fp16_weight=False,\n bnb_4bit_compute_dtype=torch.float16,\n bnb_4bit_use_double_quant=True,\n bnb_4bit_quant_type='nf4')),\n lora=dict(\n type=LoraConfig,\n r=64,\n lora_alpha=16,\n lora_dropout=0.1,\n bias='none',\n task_type='CAUSAL_LM'))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=alpaca_en,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/qwen_moe/qwen1_5/qwen1_5_moe_a2_7_b_chat/qwen1_5_moe_a2_7_b_chat_full_alpaca_e3.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom torch.optim import AdamW\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nfrom xtuner.dataset import ConcatDataset, process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import (alpaca_map_fn, alpaca_zh_map_fn,\n template_map_fn_factory)\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n ThroughputHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'Qwen/Qwen1.5-MoE-A2.7B-Chat'\nuse_varlen_attn = False\n\n# Data\nalpaca_zh_path = 'silk-road/alpaca-data-gpt4-chinese'\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.qwen_chat\nmax_length = 32768\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 1\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 3\noptim_type = AdamW\nlr = 2e-5\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 50\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=False,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nalpaca_zh = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_zh_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_zh_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=False,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\ntrain_dataset = dict(type=ConcatDataset, datasets=[alpaca_en, alpaca_zh])\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=train_dataset,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template),\n dict(type=ThroughputHook),\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=1),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=-1,\n save_last=False,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False, window_size=1)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/reward_model/internlm/internlm2_chat_1_8b_reward_full_ultrafeedback.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom torch.optim import AdamW\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nfrom xtuner.dataset.collate_fns.preference_collate_fn import \\\n preference_collate_fn\nfrom xtuner.dataset.preference_dataset import (build_preference_dataset,\n orpo_dpo_mix_40k_map_fn)\nfrom xtuner.engine.hooks import VarlenAttnArgsToMessageHubHook\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model.reward import RewardModel\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'internlm/internlm2-chat-1_8b-sft'\nuse_varlen_attn = False\nreward_token_id = 92527 # use [UNUSED_TOKEN_130] as reward token\nloss_type = 'focal'\npenalty_type = 'log_barrier'\n\n# Data\nmax_length = 2048\n\n# Scheduler & Optimizer\nbatch_size = 4 # per_device\naccumulative_counts = 16\ndataloader_num_workers = 0\nmax_epochs = 1 # reward model should not be trained for more than 1 epoch to avoid overfitting # noqa: E501\noptim_type = AdamW\nlr = 2e-5\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 500\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=RewardModel,\n use_varlen_attn=use_varlen_attn,\n loss_type=loss_type,\n penalty_type=penalty_type,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\ntrain_dataset = dict(\n type=build_preference_dataset,\n dataset=dict(\n type=load_dataset,\n path='argilla/ultrafeedback-binarized-preferences-cleaned'),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=orpo_dpo_mix_40k_map_fn,\n is_dpo=False,\n is_reward=True,\n reward_token_id=reward_token_id,\n num_proc=32,\n use_varlen_attn=use_varlen_attn,\n shuffle_before_pack=True,\n)\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=train_dataset,\n sampler=dict(type=DefaultSampler, shuffle=True),\n collate_fn=dict(\n type=preference_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = []\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/reward_model/internlm/internlm2_chat_1_8b_reward_full_varlenattn_jsonl_dataset.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom torch.optim import AdamW\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nfrom xtuner.dataset.collate_fns.preference_collate_fn import \\\n preference_collate_fn\nfrom xtuner.dataset.preference_dataset import (build_preference_dataset,\n load_jsonl_dataset)\nfrom xtuner.engine.hooks import VarlenAttnArgsToMessageHubHook\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model.reward import RewardModel\nfrom xtuner.parallel.sequence import SequenceParallelSampler\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'internlm/internlm2-chat-1_8b-sft'\nuse_varlen_attn = True\nreward_token_id = 92527 # use [UNUSED_TOKEN_130] as reward token\nloss_type = 'focal'\npenalty_type = 'log_barrier'\n\n# Data\nmax_length = 2048\nmax_packed_length = max_length * 2\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 1 # reward model should not be trained for more than 1 epoch to avoid overfitting # noqa: E501\noptim_type = AdamW\nlr = 2e-5\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\n# TODO: eval\n# evaluation_freq = 500\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=RewardModel,\n use_varlen_attn=use_varlen_attn,\n loss_type=loss_type,\n penalty_type=penalty_type,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataset = dict(\n type=build_preference_dataset,\n dataset=dict(\n type=load_jsonl_dataset,\n data_files=[\n '/your/jsonl/path/here.jsonl',\n '/your/another/jsonl/path/here.jsonl'\n ]),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=None,\n is_dpo=False,\n is_reward=True,\n reward_token_id=reward_token_id,\n num_proc=32,\n use_varlen_attn=use_varlen_attn,\n max_packed_length=max_packed_length,\n shuffle_before_pack=True,\n)\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=train_dataset,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(\n type=preference_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = []\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/reward_model/internlm/internlm2_chat_1_8b_reward_full_varlenattn_ultrafeedback.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom torch.optim import AdamW\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nfrom xtuner.dataset.collate_fns.preference_collate_fn import \\\n preference_collate_fn\nfrom xtuner.dataset.preference_dataset import (build_preference_dataset,\n orpo_dpo_mix_40k_map_fn)\nfrom xtuner.engine.hooks import VarlenAttnArgsToMessageHubHook\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model.reward import RewardModel\nfrom xtuner.parallel.sequence import SequenceParallelSampler\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'internlm/internlm2-chat-1_8b-sft'\nuse_varlen_attn = True\nreward_token_id = 92527 # use [UNUSED_TOKEN_130] as reward token\nloss_type = 'focal'\npenalty_type = 'log_barrier'\n\n# Data\nmax_length = 2048\nmax_packed_length = max_length * 2\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 1 # reward model should not be trained for more than 1 epoch to avoid overfitting # noqa: E501\noptim_type = AdamW\nlr = 2e-5\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\n# TODO: eval\n# evaluation_freq = 500\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=RewardModel,\n use_varlen_attn=use_varlen_attn,\n loss_type=loss_type,\n penalty_type=penalty_type,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataset = dict(\n type=build_preference_dataset,\n dataset=dict(\n type=load_dataset,\n path='argilla/ultrafeedback-binarized-preferences-cleaned'),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=orpo_dpo_mix_40k_map_fn,\n is_dpo=False,\n is_reward=True,\n reward_token_id=reward_token_id,\n num_proc=32,\n use_varlen_attn=use_varlen_attn,\n max_packed_length=max_packed_length,\n shuffle_before_pack=True,\n)\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=train_dataset,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(\n type=preference_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = []\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/reward_model/internlm/internlm2_chat_1_8b_reward_qlora_varlenattn_ultrafeedback.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nimport torch\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom peft import LoraConfig\nfrom torch.optim import AdamW\nfrom transformers import (AutoModelForCausalLM, AutoTokenizer,\n BitsAndBytesConfig)\n\nfrom xtuner.dataset.collate_fns.preference_collate_fn import \\\n preference_collate_fn\nfrom xtuner.dataset.preference_dataset import (build_preference_dataset,\n orpo_dpo_mix_40k_map_fn)\nfrom xtuner.engine.hooks import VarlenAttnArgsToMessageHubHook\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model.reward import RewardModel\nfrom xtuner.parallel.sequence import SequenceParallelSampler\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'internlm/internlm2-chat-1_8b-sft'\nuse_varlen_attn = True\nreward_token_id = 92527 # use [UNUSED_TOKEN_130] as reward token\nloss_type = 'focal'\npenalty_type = 'log_barrier'\n\n# Data\nmax_length = 2048\nmax_packed_length = max_length * 2\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 1 # reward model should not be trained for more than 1 epoch to avoid overfitting # noqa: E501\noptim_type = AdamW\nlr = 1e-4\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\n# TODO: eval\n# evaluation_freq = 500\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=RewardModel,\n use_varlen_attn=use_varlen_attn,\n loss_type=loss_type,\n penalty_type=penalty_type,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n torch_dtype=torch.float16,\n quantization_config=dict(\n type=BitsAndBytesConfig,\n load_in_4bit=True,\n load_in_8bit=False,\n llm_int8_threshold=6.0,\n llm_int8_has_fp16_weight=False,\n bnb_4bit_compute_dtype=torch.float16,\n bnb_4bit_use_double_quant=True,\n bnb_4bit_quant_type='nf4')),\n lora=dict(\n type=LoraConfig,\n r=64,\n lora_alpha=16,\n lora_dropout=0.1,\n bias='none',\n task_type='FEATURE_EXTRACTION')) # this setting is important\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataset = dict(\n type=build_preference_dataset,\n dataset=dict(\n type=load_dataset,\n path='argilla/ultrafeedback-binarized-preferences-cleaned'),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=orpo_dpo_mix_40k_map_fn,\n is_dpo=False,\n is_reward=True,\n reward_token_id=reward_token_id,\n num_proc=32,\n use_varlen_attn=use_varlen_attn,\n max_packed_length=max_packed_length,\n shuffle_before_pack=True,\n)\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=train_dataset,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(\n type=preference_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = []\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/reward_model/llama/llama3_8b_instruct_reward_full_varlenattn_ultrafeedback.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom torch.optim import AdamW\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\nfrom xtuner.dataset.collate_fns.preference_collate_fn import \\\n preference_collate_fn\nfrom xtuner.dataset.preference_dataset import (build_preference_dataset,\n orpo_dpo_mix_40k_map_fn)\nfrom xtuner.engine.hooks import VarlenAttnArgsToMessageHubHook\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model.reward import RewardModel\nfrom xtuner.parallel.sequence import SequenceParallelSampler\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'meta-llama/Meta-Llama-3-8B-Instruct'\nuse_varlen_attn = True\nreward_token_id = 128002 # use <|reserved_special_token_0|> as reward token\nloss_type = 'focal'\npenalty_type = 'log_barrier'\n\n# Data\nmax_length = 2048\nmax_packed_length = max_length * 2\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 1 # reward model should not be trained for more than 1 epoch to avoid overfitting # noqa: E501\noptim_type = AdamW\nlr = 2e-5\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\n# TODO: eval\n# evaluation_freq = 500\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=RewardModel,\n use_varlen_attn=use_varlen_attn,\n loss_type=loss_type,\n penalty_type=penalty_type,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\n\ntrain_dataset = dict(\n type=build_preference_dataset,\n dataset=dict(\n type=load_dataset,\n path='argilla/ultrafeedback-binarized-preferences-cleaned'),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=orpo_dpo_mix_40k_map_fn,\n is_dpo=False,\n is_reward=True,\n reward_token_id=reward_token_id,\n num_proc=32,\n use_varlen_attn=use_varlen_attn,\n max_packed_length=max_packed_length,\n shuffle_before_pack=True,\n)\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=train_dataset,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(\n type=preference_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = []\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/starcoder/starcoder_qlora_stack_exchange_example.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nimport torch\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom peft import LoraConfig\nfrom torch.optim import AdamW\nfrom transformers import (AutoModelForCausalLM, AutoTokenizer,\n BitsAndBytesConfig)\n\nfrom xtuner.dataset import process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import (stack_exchange_map_fn,\n template_map_fn_factory)\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.utils import PROMPT_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'bigcode/starcoder'\nuse_varlen_attn = False\n\n# Data\ndata_path = 'ArmelR/stack-exchange-instruction'\nprompt_template = PROMPT_TEMPLATE.default\nmax_length = 2048\n# randomly select 20000 samples from the original dataset\nmax_dataset_length = 20000\npack_to_max_length = True\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16 # 1bs * 16acc * 1gpu = 16 batchsize\ndataloader_num_workers = 0\nmax_epochs = 1\noptim_type = AdamW\nlr = 1e-4\nbetas = (0.9, 0.999)\nweight_decay = 0.05\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 200\nSYSTEM = ''\nevaluation_inputs = [\n 'from typing import List def has_close_elements(numbers: List[float], threshold: float) -> bool: \"\"\" Check if in given list of numbers, are any two numbers closer to each other than given threshold. >>> has_close_elements([1.0, 2.0, 3.0], 0.5) False >>> has_close_elements([1.0, 2.8, 3.0, 4.0, 5.0, 2.0], 0.3) True \"\"\"' # noqa: E501\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n torch_dtype=torch.float16,\n quantization_config=dict(\n type=BitsAndBytesConfig,\n load_in_4bit=True,\n load_in_8bit=False,\n llm_int8_threshold=6.0,\n llm_int8_has_fp16_weight=False,\n bnb_4bit_compute_dtype=torch.float16,\n bnb_4bit_use_double_quant=True,\n bnb_4bit_quant_type='nf4')),\n lora=dict(\n type=LoraConfig,\n r=16,\n lora_alpha=32,\n lora_dropout=0.05,\n bias='none',\n target_modules=['c_proj', 'c_attn', 'q_attn'],\n task_type='CAUSAL_LM'))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\ntrain_dataset = dict(\n type=process_hf_dataset,\n dataset=dict(\n type=load_dataset,\n path=data_path,\n data_dir='data/finetune',\n split='train'),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=stack_exchange_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n max_dataset_length=max_dataset_length,\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=train_dataset,\n sampler=dict(type=DefaultSampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/yi/yi_34b/yi_34b_qlora_alpaca_enzh_e3.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nimport torch\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom peft import LoraConfig\nfrom torch.optim import AdamW\nfrom transformers import (AutoModelForCausalLM, AutoTokenizer,\n BitsAndBytesConfig)\n\nfrom xtuner.dataset import ConcatDataset, process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import (alpaca_map_fn, alpaca_zh_map_fn,\n template_map_fn_factory)\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = '01-ai/Yi-34B'\nuse_varlen_attn = False\n\n# Data\nalpaca_zh_path = 'silk-road/alpaca-data-gpt4-chinese'\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.default\nmax_length = 2048\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 3\noptim_type = AdamW\nlr = 2e-4\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 500\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n torch_dtype=torch.float16,\n quantization_config=dict(\n type=BitsAndBytesConfig,\n load_in_4bit=True,\n load_in_8bit=False,\n llm_int8_threshold=6.0,\n llm_int8_has_fp16_weight=False,\n bnb_4bit_compute_dtype=torch.float16,\n bnb_4bit_use_double_quant=True,\n bnb_4bit_quant_type='nf4')),\n lora=dict(\n type=LoraConfig,\n r=64,\n lora_alpha=16,\n lora_dropout=0.1,\n bias='none',\n task_type='CAUSAL_LM'))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nalpaca_zh = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_zh_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_zh_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\ntrain_dataset = dict(type=ConcatDataset, datasets=[alpaca_en, alpaca_zh])\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=train_dataset,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/yi/yi_6b/yi_6b_qlora_alpaca_enzh_e3.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nimport torch\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom peft import LoraConfig\nfrom torch.optim import AdamW\nfrom transformers import (AutoModelForCausalLM, AutoTokenizer,\n BitsAndBytesConfig)\n\nfrom xtuner.dataset import ConcatDataset, process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import (alpaca_map_fn, alpaca_zh_map_fn,\n template_map_fn_factory)\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = '01-ai/Yi-6B'\nuse_varlen_attn = False\n\n# Data\nalpaca_zh_path = 'silk-road/alpaca-data-gpt4-chinese'\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.default\nmax_length = 2048\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 3\noptim_type = AdamW\nlr = 2e-4\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 500\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n torch_dtype=torch.float16,\n quantization_config=dict(\n type=BitsAndBytesConfig,\n load_in_4bit=True,\n load_in_8bit=False,\n llm_int8_threshold=6.0,\n llm_int8_has_fp16_weight=False,\n bnb_4bit_compute_dtype=torch.float16,\n bnb_4bit_use_double_quant=True,\n bnb_4bit_quant_type='nf4')),\n lora=dict(\n type=LoraConfig,\n r=64,\n lora_alpha=16,\n lora_dropout=0.1,\n bias='none',\n task_type='CAUSAL_LM'))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nalpaca_zh = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_zh_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_zh_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\ntrain_dataset = dict(type=ConcatDataset, datasets=[alpaca_en, alpaca_zh])\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=train_dataset,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/configs/zephyr/zephyr_7b_beta_qlora_alpaca_e3.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nimport torch\nfrom datasets import load_dataset\nfrom mmengine.dataset import DefaultSampler\nfrom mmengine.hooks import (CheckpointHook, DistSamplerSeedHook, IterTimerHook,\n LoggerHook, ParamSchedulerHook)\nfrom mmengine.optim import AmpOptimWrapper, CosineAnnealingLR, LinearLR\nfrom peft import LoraConfig\nfrom torch.optim import AdamW\nfrom transformers import (AutoModelForCausalLM, AutoTokenizer,\n BitsAndBytesConfig)\n\nfrom xtuner.dataset import process_hf_dataset\nfrom xtuner.dataset.collate_fns import default_collate_fn\nfrom xtuner.dataset.map_fns import alpaca_map_fn, template_map_fn_factory\nfrom xtuner.engine.hooks import (DatasetInfoHook, EvaluateChatHook,\n VarlenAttnArgsToMessageHubHook)\nfrom xtuner.engine.runner import TrainLoop\nfrom xtuner.model import SupervisedFinetune\nfrom xtuner.parallel.sequence import SequenceParallelSampler\nfrom xtuner.utils import PROMPT_TEMPLATE, SYSTEM_TEMPLATE\n\n#######################################################################\n# PART 1 Settings #\n#######################################################################\n# Model\npretrained_model_name_or_path = 'HuggingFaceH4/zephyr-7b-beta'\nuse_varlen_attn = False\n\n# Data\nalpaca_en_path = 'tatsu-lab/alpaca'\nprompt_template = PROMPT_TEMPLATE.zephyr\nmax_length = 2048\npack_to_max_length = True\n\n# parallel\nsequence_parallel_size = 1\n\n# Scheduler & Optimizer\nbatch_size = 1 # per_device\naccumulative_counts = 16\naccumulative_counts *= sequence_parallel_size\ndataloader_num_workers = 0\nmax_epochs = 3\noptim_type = AdamW\nlr = 2e-4\nbetas = (0.9, 0.999)\nweight_decay = 0\nmax_norm = 1 # grad clip\nwarmup_ratio = 0.03\n\n# Save\nsave_steps = 500\nsave_total_limit = 2 # Maximum checkpoints to keep (-1 means unlimited)\n\n# Evaluate the generation performance during the training\nevaluation_freq = 500\nSYSTEM = SYSTEM_TEMPLATE.alpaca\nevaluation_inputs = [\n '请给我介绍五个上海的景点', 'Please tell me five scenic spots in Shanghai'\n]\n\n#######################################################################\n# PART 2 Model & Tokenizer #\n#######################################################################\ntokenizer = dict(\n type=AutoTokenizer.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n padding_side='right')\n\nmodel = dict(\n type=SupervisedFinetune,\n use_varlen_attn=use_varlen_attn,\n llm=dict(\n type=AutoModelForCausalLM.from_pretrained,\n pretrained_model_name_or_path=pretrained_model_name_or_path,\n trust_remote_code=True,\n torch_dtype=torch.float16,\n quantization_config=dict(\n type=BitsAndBytesConfig,\n load_in_4bit=True,\n load_in_8bit=False,\n llm_int8_threshold=6.0,\n llm_int8_has_fp16_weight=False,\n bnb_4bit_compute_dtype=torch.float16,\n bnb_4bit_use_double_quant=True,\n bnb_4bit_quant_type='nf4')),\n lora=dict(\n type=LoraConfig,\n r=64,\n lora_alpha=16,\n lora_dropout=0.1,\n bias='none',\n task_type='CAUSAL_LM'))\n\n#######################################################################\n# PART 3 Dataset & Dataloader #\n#######################################################################\nalpaca_en = dict(\n type=process_hf_dataset,\n dataset=dict(type=load_dataset, path=alpaca_en_path),\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=alpaca_map_fn,\n template_map_fn=dict(\n type=template_map_fn_factory, template=prompt_template),\n remove_unused_columns=True,\n shuffle_before_pack=True,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn)\n\nsampler = SequenceParallelSampler \\\n if sequence_parallel_size > 1 else DefaultSampler\ntrain_dataloader = dict(\n batch_size=batch_size,\n num_workers=dataloader_num_workers,\n dataset=alpaca_en,\n sampler=dict(type=sampler, shuffle=True),\n collate_fn=dict(type=default_collate_fn, use_varlen_attn=use_varlen_attn))\n\n#######################################################################\n# PART 4 Scheduler & Optimizer #\n#######################################################################\n# optimizer\noptim_wrapper = dict(\n type=AmpOptimWrapper,\n optimizer=dict(\n type=optim_type, lr=lr, betas=betas, weight_decay=weight_decay),\n clip_grad=dict(max_norm=max_norm, error_if_nonfinite=False),\n accumulative_counts=accumulative_counts,\n loss_scale='dynamic',\n dtype='float16')\n\n# learning policy\n# More information: https://github.com/open-mmlab/mmengine/blob/main/docs/en/tutorials/param_scheduler.md # noqa: E501\nparam_scheduler = [\n dict(\n type=LinearLR,\n start_factor=1e-5,\n by_epoch=True,\n begin=0,\n end=warmup_ratio * max_epochs,\n convert_to_iter_based=True),\n dict(\n type=CosineAnnealingLR,\n eta_min=0.0,\n by_epoch=True,\n begin=warmup_ratio * max_epochs,\n end=max_epochs,\n convert_to_iter_based=True)\n]\n\n# train, val, test setting\ntrain_cfg = dict(type=TrainLoop, max_epochs=max_epochs)\n\n#######################################################################\n# PART 5 Runtime #\n#######################################################################\n# Log the dialogue periodically during the training process, optional\ncustom_hooks = [\n dict(type=DatasetInfoHook, tokenizer=tokenizer),\n dict(\n type=EvaluateChatHook,\n tokenizer=tokenizer,\n every_n_iters=evaluation_freq,\n evaluation_inputs=evaluation_inputs,\n system=SYSTEM,\n prompt_template=prompt_template)\n]\n\nif use_varlen_attn:\n custom_hooks += [dict(type=VarlenAttnArgsToMessageHubHook)]\n\n# configure default hooks\ndefault_hooks = dict(\n # record the time of every iteration.\n timer=dict(type=IterTimerHook),\n # print log every 10 iterations.\n logger=dict(type=LoggerHook, log_metric_by_epoch=False, interval=10),\n # enable the parameter scheduler.\n param_scheduler=dict(type=ParamSchedulerHook),\n # save checkpoint per `save_steps`.\n checkpoint=dict(\n type=CheckpointHook,\n by_epoch=False,\n interval=save_steps,\n max_keep_ckpts=save_total_limit),\n # set sampler seed in distributed evrionment.\n sampler_seed=dict(type=DistSamplerSeedHook),\n)\n\n# configure environment\nenv_cfg = dict(\n # whether to enable cudnn benchmark\n cudnn_benchmark=False,\n # set multi process parameters\n mp_cfg=dict(mp_start_method='fork', opencv_num_threads=0),\n # set distributed parameters\n dist_cfg=dict(backend='nccl'),\n)\n\n# set visualizer\nvisualizer = None\n\n# set log level\nlog_level = 'INFO'\n\n# load from which checkpoint\nload_from = None\n\n# whether to resume training from the loaded checkpoint\nresume = False\n\n# Defaults to use random seed and disable `deterministic`\nrandomness = dict(seed=None, deterministic=False)\n\n# set log processor\nlog_processor = dict(by_epoch=False)\n"
},
{
"path": "xtuner-eval_niah/xtuner/dataset/__init__.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nimport warnings\n\nfrom .concat_dataset import ConcatDataset\nfrom .huggingface import process_hf_dataset\nfrom .intern_repo import (build_packed_dataset,\n load_intern_repo_tokenized_dataset,\n load_intern_repo_untokenized_dataset)\nfrom .json_dataset import load_json_file\nfrom .llava import LLaVADataset\nfrom .modelscope import process_ms_dataset\nfrom .moss_sft import MOSSSFTDataset\nfrom .refcoco_json import (InvRefCOCOJsonDataset, RefCOCOJsonDataset,\n RefCOCOJsonEvalDataset)\nfrom .utils import decode_base64_to_image, expand2square, load_image\n\n# ignore FutureWarning in hf datasets\nwarnings.simplefilter(action='ignore', category=FutureWarning)\n\n__all__ = [\n 'process_hf_dataset', 'ConcatDataset', 'MOSSSFTDataset',\n 'process_ms_dataset', 'LLaVADataset', 'expand2square',\n 'decode_base64_to_image', 'load_image', 'process_ms_dataset',\n 'load_intern_repo_tokenized_dataset',\n 'load_intern_repo_untokenized_dataset', 'build_packed_dataset',\n 'RefCOCOJsonDataset', 'RefCOCOJsonEvalDataset', 'InvRefCOCOJsonDataset',\n 'load_json_file'\n]\n"
},
{
"path": "xtuner-eval_niah/xtuner/dataset/collate_fns/__init__.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nfrom .default_collate_fn import default_collate_fn\nfrom .mmlu_collate_fn import mmlu_collate_fn\n\n__all__ = ['default_collate_fn', 'mmlu_collate_fn']\n"
},
{
"path": "xtuner-eval_niah/xtuner/dataset/collate_fns/default_collate_fn.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nfrom typing import Dict, Sequence\n\nimport torch\nfrom torch.nn.utils.rnn import pad_sequence\n\nfrom xtuner.parallel.sequence import (get_sequence_parallel_world_size,\n pad_for_sequence_parallel)\nfrom xtuner.utils import DEFAULT_PAD_TOKEN_INDEX, IGNORE_INDEX\n\n\ndef default_collate_fn(instances: Sequence[Dict],\n pad_index: int = DEFAULT_PAD_TOKEN_INDEX,\n return_hf_format: bool = False,\n use_varlen_attn: bool = False):\n seq_parallel_world_size = get_sequence_parallel_world_size()\n\n input_ids, labels = [], []\n has_image = any(inst.get('pixel_values') is not None for inst in instances)\n if use_varlen_attn:\n position_ids, cumulative_len = [], []\n assert len(instances) == 1, (\n f'If utilizing varlen attention, the batch size should be'\n f' set to 1, but got {len(instances)}')\n assert not has_image, 'Currently, it is not configured to '\n 'accommodate the use of varlen Attention in multimodal training'\n\n if has_image:\n pixel_values = []\n\n for example in instances:\n input_ids.append(torch.LongTensor(example['input_ids']))\n labels.append(torch.LongTensor(example['labels']))\n if use_varlen_attn:\n cumulative_len.append(torch.IntTensor(example['cumulative_len']))\n position_ids.append(torch.LongTensor(example['position_ids']))\n\n if has_image:\n pixel_values.append(example['pixel_values'])\n\n ori_length = [len(ids) for ids in input_ids]\n if len(instances) > 1:\n input_ids = pad_sequence(\n input_ids, batch_first=True, padding_value=pad_index)\n labels = pad_sequence(\n labels, batch_first=True, padding_value=IGNORE_INDEX)\n else:\n input_ids = torch.stack(input_ids)\n labels = torch.stack(labels)\n\n if use_varlen_attn:\n assert input_ids.size(1) % seq_parallel_world_size == 0\n attention_mask = None\n position_ids = torch.stack(position_ids, dim=0)\n else:\n # Some tokenizers have the same eos token and pad token, so input_ids\n # cannot be masked directly based on the pad token id.\n attention_mask = torch.zeros_like(input_ids).bool()\n for i in ori_length:\n attention_mask[:i] = True\n\n bs, seq_len = input_ids.shape\n position_ids = torch.arange(seq_len).unsqueeze(0).long().repeat(bs, 1)\n\n if seq_parallel_world_size > 1:\n input_ids = pad_for_sequence_parallel(input_ids, pad_index)\n labels = pad_for_sequence_parallel(labels, IGNORE_INDEX)\n position_ids = pad_for_sequence_parallel(position_ids, 0)\n if attention_mask is not None:\n attention_mask = pad_for_sequence_parallel(attention_mask, 0)\n\n if use_varlen_attn:\n max_seqlen = (\n cumulative_len[0][1:] - # noqa: W504\n cumulative_len[0][:-1]).max().item()\n data_dict = {\n 'input_ids': input_ids,\n 'cumulative_len': cumulative_len,\n 'position_ids': position_ids,\n 'labels': labels,\n 'max_seqlen': max_seqlen\n }\n else:\n data_dict = {\n 'input_ids': input_ids,\n 'attention_mask': attention_mask,\n 'position_ids': position_ids,\n 'labels': labels\n }\n\n if has_image:\n pixel_values = torch.stack(pixel_values)\n data_dict['pixel_values'] = pixel_values\n\n if return_hf_format:\n return data_dict\n else:\n return {'data': data_dict, 'data_samples': None}\n"
},
{
"path": "xtuner-eval_niah/xtuner/dataset/collate_fns/mmlu_collate_fn.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nfrom typing import Dict, Sequence\n\nimport torch\nfrom torch.nn.utils.rnn import pad_sequence\n\nfrom xtuner.utils import DEFAULT_PAD_TOKEN_INDEX, IGNORE_INDEX\n\n\ndef mmlu_collate_fn(instances: Sequence[Dict],\n pad_index: int = DEFAULT_PAD_TOKEN_INDEX,\n return_hf_format: bool = False) -> Dict[str, torch.Tensor]:\n input_ids = []\n labels = []\n data_samples = {'labels': [], 'subjects': []}\n for example in instances:\n input_ids.append(torch.tensor(example['input_ids']))\n labels.append(torch.tensor(example['labels']))\n data_samples['labels'].append(example['output'])\n data_samples['subjects'].append(example['subject'])\n if len(instances) > 1:\n input_ids = pad_sequence(\n input_ids, batch_first=True, padding_value=pad_index)\n labels = pad_sequence(\n labels, batch_first=True, padding_value=IGNORE_INDEX)\n else:\n input_ids = torch.stack(input_ids)\n labels = torch.stack(labels)\n\n data_dict = {\n 'input_ids': input_ids,\n 'attention_mask': input_ids.ne(pad_index),\n 'labels': labels\n }\n\n if return_hf_format:\n return data_dict\n else:\n return {'data': data_dict, 'data_samples': data_samples}\n"
},
{
"path": "xtuner-eval_niah/xtuner/dataset/collate_fns/preference_collate_fn.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nfrom typing import Dict, Sequence\n\nimport torch\nfrom torch.nn.utils.rnn import pad_sequence\n\nfrom xtuner.parallel.sequence import (get_sequence_parallel_world_size,\n pad_cumulative_len_for_sequence_parallel,\n pad_for_sequence_parallel)\nfrom xtuner.utils import DEFAULT_PAD_TOKEN_INDEX, IGNORE_INDEX\n\n\ndef preference_collate_fn(instances: Sequence[Dict],\n pad_index: int = DEFAULT_PAD_TOKEN_INDEX,\n return_hf_format: bool = False,\n use_varlen_attn: bool = False):\n seq_parallel_world_size = get_sequence_parallel_world_size()\n ds_names = []\n if not use_varlen_attn:\n # split chosen and rejected into two instances\n splited_instances = []\n for d in instances:\n splited_instances.append({\n 'input_ids': d['chosen_ids'],\n 'labels': d['chosen_labels']\n })\n splited_instances.append({\n 'input_ids': d['rejected_ids'],\n 'labels': d['rejected_labels']\n })\n ds_names.append(d.get('ds_name', None))\n instances = splited_instances\n\n input_ids, labels = [], []\n if use_varlen_attn:\n position_ids, cumulative_len = [], []\n assert len(instances) == 1, (\n f'If utilizing varlen attention, the batch size should be'\n f' set to 1, but got {len(instances)}')\n\n for example in instances:\n input_ids.append(torch.LongTensor(example['input_ids']))\n labels.append(torch.LongTensor(example['labels']))\n if use_varlen_attn:\n cumulative_len.append(torch.IntTensor(example['cumulative_len']))\n position_ids.append(torch.LongTensor(example['position_ids']))\n num_samples = (len(example['cumulative_len']) - 1) // 2\n ds_names.extend(example.get('ds_names', [None] * num_samples))\n\n ori_length = [len(ids) for ids in input_ids]\n if len(instances) > 1:\n input_ids = pad_sequence(\n input_ids, batch_first=True, padding_value=pad_index)\n labels = pad_sequence(\n labels, batch_first=True, padding_value=IGNORE_INDEX)\n else:\n input_ids = torch.stack(input_ids)\n labels = torch.stack(labels)\n\n if use_varlen_attn:\n attention_mask = None\n position_ids = torch.stack(position_ids, dim=0)\n else:\n # Some tokenizers have the same eos token and pad token, so input_ids\n # cannot be masked directly based on the pad token id.\n attention_mask = torch.zeros_like(input_ids).bool()\n for i in ori_length:\n attention_mask[:i] = True\n\n bs, seq_len = input_ids.shape\n position_ids = torch.arange(seq_len).unsqueeze(0).long().repeat(bs, 1)\n\n if seq_parallel_world_size > 1:\n input_ids = pad_for_sequence_parallel(input_ids, pad_index)\n labels = pad_for_sequence_parallel(labels, IGNORE_INDEX)\n position_ids = pad_for_sequence_parallel(position_ids, 0)\n if attention_mask is not None:\n attention_mask = pad_for_sequence_parallel(attention_mask, 0)\n if use_varlen_attn:\n (cumulative_len, attention_mask\n ) = pad_cumulative_len_for_sequence_parallel(cumulative_len)\n\n if use_varlen_attn:\n max_seqlen = (\n cumulative_len[0][1:] - # noqa: W504\n cumulative_len[0][:-1]).max().item()\n data_dict = {\n 'input_ids': input_ids,\n 'attention_mask': attention_mask,\n 'cumulative_len': cumulative_len,\n 'position_ids': position_ids,\n 'labels': labels,\n 'max_seqlen': max_seqlen\n }\n else:\n data_dict = {\n 'input_ids': input_ids,\n 'attention_mask': attention_mask,\n 'position_ids': position_ids,\n 'labels': labels\n }\n\n if return_hf_format:\n return data_dict\n else:\n return {'data': data_dict, 'data_samples': {'ds_names': ds_names}}\n"
},
{
"path": "xtuner-eval_niah/xtuner/dataset/concat_dataset.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nfrom torch.utils.data import ConcatDataset as TorchConcatDataset\n\nfrom xtuner.registry import BUILDER\n\n\nclass ConcatDataset(TorchConcatDataset):\n\n def __init__(self, datasets):\n datasets_instance = []\n for cfg in datasets:\n datasets_instance.append(BUILDER.build(cfg))\n super().__init__(datasets=datasets_instance)\n\n def __repr__(self):\n main_str = 'Dataset as a concatenation of multiple datasets. \\n'\n main_str += ',\\n'.join(\n [f'{repr(dataset)}' for dataset in self.datasets])\n return main_str\n"
},
{
"path": "xtuner-eval_niah/xtuner/dataset/huggingface.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nimport logging\nimport os\nfrom datetime import timedelta\nfrom functools import partial\n\nimport numpy as np\nfrom datasets import DatasetDict, concatenate_datasets\nfrom mmengine import print_log\nfrom mmengine.config import Config, ConfigDict\nfrom mmengine.utils.misc import get_object_from_string\nfrom torch import distributed as dist\n\nfrom xtuner.registry import BUILDER, MAP_FUNC\nfrom .utils import Packer, encode_fn\n\n\ndef get_lengths(example):\n return {'length': len(example['input_ids'])}\n\n\ndef build_origin_dataset(dataset, split):\n if isinstance(dataset, DatasetDict):\n if split is None:\n dataset = concatenate_datasets(dataset.values())\n else:\n dataset = dataset[split]\n elif isinstance(dataset, dict) or isinstance(\n dataset, Config) or isinstance(dataset, ConfigDict):\n dataset = BUILDER.build(dataset)\n if isinstance(dataset, DatasetDict):\n if split is None:\n dataset = concatenate_datasets(dataset.values())\n else:\n dataset = dataset[split]\n return dataset\n\n\ndef map_dataset(dataset, dataset_map_fn, map_num_proc):\n if isinstance(dataset_map_fn, str):\n map_fn_obj = MAP_FUNC.get(dataset_map_fn) or get_object_from_string(\n dataset_map_fn)\n if map_fn_obj is not None:\n dataset_map_fn = map_fn_obj\n else:\n raise TypeError('dataset_map_fn must be a function or a '\n \"registered function's string in MAP_FUNC, \"\n f\"but got a string of '{dataset_map_fn}'\")\n\n dataset = dataset.map(dataset_map_fn, num_proc=map_num_proc)\n return dataset\n\n\ndef add_template_to_dataset(dataset, template_map_fn, map_num_proc):\n if isinstance(template_map_fn,\n dict) or isinstance(template_map_fn, Config) or isinstance(\n template_map_fn, ConfigDict):\n template_map_fn = BUILDER.build(template_map_fn)\n dataset = dataset.map(template_map_fn, num_proc=map_num_proc)\n # remove invalid data\n dataset = dataset.filter(\n lambda example: len(example['conversation']) > 0,\n num_proc=map_num_proc)\n return dataset\n\n\ndef tokenize_dataset(dataset, tokenizer, max_length, with_image_token,\n input_ids_with_output, remove_unused_columns,\n map_num_proc):\n assert (tokenizer is not None) and (max_length is not None), \\\n f'({tokenizer}, {max_length})'\n if isinstance(tokenizer, dict) or isinstance(\n tokenizer, Config) or isinstance(tokenizer, ConfigDict):\n tokenizer = BUILDER.build(tokenizer)\n dataset = dataset.map(\n partial(\n encode_fn,\n tokenizer=tokenizer,\n max_length=max_length,\n with_image_token=with_image_token,\n input_ids_with_output=input_ids_with_output),\n remove_columns=list(dataset.column_names)\n if remove_unused_columns else None,\n num_proc=map_num_proc)\n return dataset\n\n\ndef pack_dataset(dataset, max_length, use_varlen_attn, shuffle_before_pack,\n map_num_proc):\n if shuffle_before_pack:\n dataset = dataset.shuffle()\n dataset = dataset.flatten_indices(num_proc=map_num_proc)\n dataset = dataset.map(\n Packer(max_length, use_varlen_attn=use_varlen_attn),\n batched=True,\n num_proc=map_num_proc)\n return dataset\n\n\ndef process(dataset,\n do_dataset_tokenization=True,\n tokenizer=None,\n max_length=None,\n dataset_map_fn=None,\n template_map_fn=None,\n max_dataset_length=None,\n split='train',\n remove_unused_columns=False,\n rename_maps=[],\n shuffle_before_pack=True,\n pack_to_max_length=True,\n use_varlen_attn=False,\n input_ids_with_output=True,\n with_image_token=False,\n map_num_proc=32):\n \"\"\"Post-process the dataset loaded from the Hugging Face Hub, or a local\n dataset.\n\n Args:\n dataset: The dataset to be post-processed.\n do_dataset_tokenization: Whether the dataset need to be tokenized\n in this function. Default to True.\n tokenizer: The tokenizer processes some raw text as input and outputs\n an Encoding. If `do_dataset_tokenization` is True, this argument\n should not be None. Default to None.\n max_length: Max length of the sequence. If `do_dataset_tokenization`\n or `pack_to_max_length` is True, this argument should not be None.\n Default to None.\n dataset_map_fn: Map the original dataset format to the one defined\n by xTuner.\n template_map_fn: Add the prompt template to the dataset\n max_dataset_length: If the length of the dataset is too long, we can\n randomly extract `max_dataset_length` from it.\n split: Which split of the data to load.\n If `None`, will return a single concatenated dataset with all\n splits (typically `datasets.Split.TRAIN` and\n `datasets.Split.TEST`).\n If given, will return a single Dataset.\n remove_unused_columns: Whether to remove columns from the dataset\n that are not used during training.\n rename_maps: Rename the column name of the dataset.\n shuffle_before_pack: Whether to shuffle the dataset before\n packing them.\n pack_to_max_length: Whether to pack the dataset to the `max_length `.\n This usually improves gpu utilization and therefore reduces\n training time.\n use_varlen_attn: If use_varlen_attn is True, we calculate attention\n the actual length of the sequence rather than the actual length\n of the sequence\n input_ids_with_output: Whether to put the groundtruth output\n corresponding to the question into the dataset. Typically set\n it to True during training and False during testing.\n with_image_token: Whether to convert DEFAULT_IMAGE_TOKEN to\n IMAGE_TOKEN_INDEX. Typically set it to True during the training\n of VLM.\n map_num_proc: Max number of processes when mapping the dataset.\n \"\"\"\n if use_varlen_attn:\n assert pack_to_max_length, \\\n '`pack_to_max_length` in `process_hf_dataset` should be set to ' \\\n 'True if `use_varlen_attn` is True.'\n if pack_to_max_length:\n assert split == 'train' or split is None, \\\n ('`split` should be `train` or `None` if `pack_to_max_length` is '\n f'True, but got {split}.')\n\n dataset = build_origin_dataset(dataset, split)\n\n # sample `max_dataset_length` items from the original dataset to\n # save time consumed by map function\n if max_dataset_length is not None:\n max_dataset_length = min(max_dataset_length, len(dataset))\n indices = np.random.choice(\n len(dataset), max_dataset_length, replace=False)\n dataset = dataset.select(indices)\n\n # Extract the useful data for training from the original dataset.\n if dataset_map_fn is not None:\n dataset = map_dataset(dataset, dataset_map_fn, map_num_proc)\n\n # Add prompt template, such as <|System|>: xxx <|User|>: xxx <|Bot|>: xxx\n if template_map_fn is not None:\n dataset = add_template_to_dataset(dataset, template_map_fn,\n map_num_proc)\n\n for old, new in rename_maps:\n dataset = dataset.rename_column(old, new)\n\n # remove unused columns\n if pack_to_max_length and (not remove_unused_columns):\n print_log(\n 'We have to remove unused columns if '\n '`pack_to_max_length` is set to True.',\n logger='current',\n level=logging.WARNING)\n remove_unused_columns = True\n\n if do_dataset_tokenization:\n dataset = tokenize_dataset(dataset, tokenizer, max_length,\n with_image_token, input_ids_with_output,\n remove_unused_columns, map_num_proc)\n\n if input_ids_with_output:\n assert {'input_ids', 'labels'}.issubset(dataset.column_names)\n # remove data that does not have the valid labels.\n dataset = dataset.filter(\n lambda example: any(label >= 0 for label in example['labels']),\n num_proc=map_num_proc)\n\n # pack to max length\n if pack_to_max_length:\n dataset = pack_dataset(dataset, max_length, use_varlen_attn,\n shuffle_before_pack, map_num_proc)\n\n # add 'length'\n dataset = dataset.map(get_lengths, num_proc=map_num_proc)\n setattr(dataset, 'length', dataset['length'])\n\n return dataset\n\n\ndef process_hf_dataset(dataset,\n do_dataset_tokenization=True,\n tokenizer=None,\n max_length=None,\n dataset_map_fn=None,\n template_map_fn=None,\n max_dataset_length=None,\n split='train',\n remove_unused_columns=False,\n rename_maps=[],\n shuffle_before_pack=True,\n pack_to_max_length=True,\n use_varlen_attn=False,\n input_ids_with_output=True,\n with_image_token=False,\n map_num_proc=32):\n \"\"\"Post-process the dataset loaded from the Hugging Face Hub, or a local\n dataset.\n\n Args:\n dataset: The dataset to be post-processed.\n do_dataset_tokenization: Whether the dataset need to be tokenized\n in this function. Default to True.\n tokenizer: The tokenizer processes some raw text as input and outputs\n an Encoding. If `do_dataset_tokenization` is True, this argument\n should not be None. Default to None.\n max_length: Max length of the sequence. If `do_dataset_tokenization`\n or `pack_to_max_length` is True, this argument should not be None.\n Default to None.\n dataset_map_fn: Map the original dataset format to the one defined\n by xTuner.\n template_map_fn: Add the prompt template to the dataset\n max_dataset_length: If the length of the dataset is too long, we can\n randomly extract `max_dataset_length` from it.\n split: Which split of the data to load.\n If `None`, will return a single concatenated dataset with all\n splits (typically `datasets.Split.TRAIN` and\n `datasets.Split.TEST`).\n If given, will return a single Dataset.\n remove_unused_columns: Whether to remove columns from the dataset\n that are not used during training.\n rename_maps: Rename the column name of the dataset.\n shuffle_before_pack: Whether to shuffle the dataset before\n packing them.\n pack_to_max_length: Whether to pack the dataset to the `max_length `.\n This usually improves gpu utilization and therefore reduces\n training time.\n use_varlen_attn: If use_varlen_attn is True, we calculate attention\n the actual length of the sequence rather than the actual length\n of the sequence\n input_ids_with_output: Whether to put the groundtruth output\n corresponding to the question into the dataset. Typically set\n it to True during training and False during testing.\n with_image_token: Whether to convert DEFAULT_IMAGE_TOKEN to\n IMAGE_TOKEN_INDEX. Typically set it to True during the training\n of VLM.\n map_num_proc: Max number of processes when mapping the dataset.\n \"\"\"\n kwargs = dict(\n dataset=dataset,\n do_dataset_tokenization=do_dataset_tokenization,\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=dataset_map_fn,\n template_map_fn=template_map_fn,\n max_dataset_length=max_dataset_length,\n split=split,\n remove_unused_columns=remove_unused_columns,\n rename_maps=rename_maps,\n shuffle_before_pack=shuffle_before_pack,\n pack_to_max_length=pack_to_max_length,\n use_varlen_attn=use_varlen_attn,\n input_ids_with_output=input_ids_with_output,\n with_image_token=with_image_token,\n map_num_proc=map_num_proc)\n if not (dist.is_available() and dist.is_initialized()):\n return process(**kwargs)\n\n xtuner_dataset_timeout = timedelta(\n minutes=int(os.getenv('XTUNER_DATASET_TIMEOUT', default=60)))\n print_log(\n f'xtuner_dataset_timeout = {xtuner_dataset_timeout}', logger='current')\n # monitored barrier requires gloo process group to perform host-side sync.\n group_gloo = dist.new_group(backend='gloo', timeout=xtuner_dataset_timeout)\n\n if dist.get_rank() == 0:\n dataset = process(**kwargs)\n objects = [dataset]\n else:\n objects = [None]\n\n dist.monitored_barrier(group=group_gloo, timeout=xtuner_dataset_timeout)\n dist.broadcast_object_list(objects, src=0)\n return objects[0]\n"
},
{
"path": "xtuner-eval_niah/xtuner/dataset/intern_repo.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nimport itertools as it\nimport json\nimport mmap\nimport operator\nimport os\nimport threading\nfrom pathlib import Path\n\nimport numpy as np\nimport torch\nfrom datasets import Dataset, load_dataset, load_from_disk\nfrom mmengine import print_log\nfrom torch import distributed as dist\nfrom torch.utils.data import ConcatDataset\n\nfrom xtuner.dataset.map_fns import openai_map_fn\nfrom xtuner.registry import BUILDER\nfrom .huggingface import process\n\n\nclass JsonlDataset(torch.utils.data.Dataset):\n \"\"\"\n\n JSONL format is expected to roughly follow that of The Pile.\n One-line-per-document of the form:\n ```\n {\n \"input_ids\": List[int],\n \"labels\": List[int]\n }\n ```\n\n \"\"\"\n\n def __init__(self, path: str, min_length=50):\n self.path = path\n self.threadlocal = threading.local()\n resolved_path = Path(path).resolve()\n self.resolved_path = resolved_path\n self.meta = Path(f'{resolved_path}.meta')\n\n # only build the cache in on the primary worker to prevent\n # overloading nfs\n assert os.path.exists(\n self.meta\n ), f'The cache file:{self.meta} is not found for file:{self.path}'\n try:\n with open(self.meta, 'rb') as f:\n meta = np.load(f)\n except Exception as e:\n print(f'Cannot load file {self.meta}...')\n raise e\n self.offsets = meta[:, 0]\n self.length = meta[:, -1]\n\n if min_length > 0:\n mask = self.length >= min_length\n self.offsets = self.offsets[mask]\n self.length = self.length[mask]\n\n def __getitem__(self, idx):\n f = self._get_mmap()\n position = self.offsets[idx]\n f.seek(position)\n item = f.readline().decode('utf-8')\n try:\n item = json.loads(item)\n item['input_ids'] = item['tokens']\n del item['tokens']\n labels = [x if x > 0 else -100 for x in item['input_ids']]\n item['input_ids'] = [abs(x) for x in item['input_ids']]\n item['labels'] = labels\n item['length'] = len(item['input_ids']) # add a length info\n except Exception as err:\n raise json.decoder.JSONDecodeError(\n doc=self.path,\n pos=position,\n msg=(f'Error while loading JSONL line in file {self.path} '\n f'at byte {position}. Contents of line:\\n{item}\\n{err}'),\n )\n return item\n\n def get_dataset_name(self):\n return str(self.resolved_path)\n\n def _get_mmap(self):\n if not hasattr(self.threadlocal, 'handles'):\n with open(self.path, 'rb') as f:\n mm = mmap.mmap(f.fileno(), 0, access=mmap.ACCESS_READ)\n self.threadlocal.handles = [f, mm]\n if self.path.endswith('.gz') or self.path.endswith(\n '.bz') or self.path.endswith('.bz2'):\n raise NotImplementedError(\n 'Compressed files are not supported because .seek() '\n 'would require rereading the entire file, making '\n 'performance too slow.')\n return self.threadlocal.handles[-1]\n\n def __setstate__(self, state):\n self.__dict__ = state\n self.threadlocal = threading.local()\n\n def __getstate__(self):\n d = {}\n for i, v in self.__dict__.items():\n if i != 'threadlocal':\n d[i] = v\n return d\n\n def __del__(self):\n if hasattr(self.threadlocal, 'handles'):\n # cleanup files we opened on initialization\n while self.threadlocal.handles:\n self.threadlocal.handles.pop().close()\n\n @staticmethod\n def exists(path):\n return os.path.exists(path)\n\n def __len__(self):\n # Virtual length of the dataset depends on the epoch number\n # if the number of documents is not perfectly divisible by the\n # data_subshard_count\n return len(self.offsets)\n\n\nclass PackedDataset(torch.utils.data.Dataset):\n \"\"\"The class PackedDataset takes in a dataset and aggregates samples of\n different lengths together based on the packed_length.\n\n Args:\n dataset: The original dataset to pack.\n packed_length: The length of each packed sample. Default is 8192.\n \"\"\"\n\n def __init__(self, dataset, packed_length: int = 8192, seed: int = 1024):\n self.dataset = dataset\n self.packed_length = packed_length\n if isinstance(dataset, JsonlDataset):\n self.length = dataset.length\n elif isinstance(dataset, Dataset):\n if hasattr(dataset, 'length'):\n length = dataset.length\n else:\n length = [len(i['input_ids']) for i in dataset]\n self.length = length\n else:\n raise NotImplementedError\n self.seed = seed\n\n rng = np.random.RandomState(self.seed)\n shuffled_indices = np.arange(len(self.length))\n rng.shuffle(shuffled_indices)\n self.shuffled_indices = shuffled_indices.tolist()\n self.shuffled_samples_len = list(\n map(self.length.__getitem__, shuffled_indices))\n self.shuffled_accumulated_samples_len = list(\n it.accumulate(self.shuffled_samples_len, operator.add))\n self.num_tokens = sum(self.length)\n\n def __len__(self):\n return self.num_tokens // self.packed_length\n\n def search_sample_index(self, pack_idx: int = 0):\n assert pack_idx >= 0\n length_train = (pack_idx + 1) * self.packed_length\n sample_index = np.searchsorted(\n self.shuffled_accumulated_samples_len, length_train, side='left')\n return sample_index\n\n def mapping(self, pack_idx: int = 0):\n begin_sample_idx, begin_token_id = 0, 0\n if pack_idx > 0:\n begin_sample_idx = self.search_sample_index(pack_idx - 1)\n # The position where the previous packed data ends\n begin_token_id = self.shuffled_samples_len[begin_sample_idx] - (\n self.shuffled_accumulated_samples_len[begin_sample_idx]\n - # noqa: W504,W503\n (pack_idx) * self.packed_length)\n if begin_token_id == self.shuffled_samples_len[begin_sample_idx]:\n begin_sample_idx += 1\n begin_token_id = 0\n\n end_sample_idx = self.search_sample_index(pack_idx)\n end_token_id = self.shuffled_samples_len[end_sample_idx] - (\n self.shuffled_accumulated_samples_len[end_sample_idx]\n - # noqa: W504,W503\n (pack_idx + 1) * self.packed_length)\n return begin_sample_idx, begin_token_id, end_sample_idx, end_token_id\n\n def build_pack(self, begin_sample_idx: int, begin_token_id: int,\n end_sample_idx: int, end_token_id: int):\n pack, cumulative_len, position_ids, labels = [], [0], [], []\n\n while begin_sample_idx < end_sample_idx:\n sample_idx = self.shuffled_indices[begin_sample_idx]\n sample = self.dataset[sample_idx]\n chunk = sample['input_ids'][begin_token_id:]\n pack.extend(chunk)\n _labels = sample['labels'][begin_token_id:]\n assert len(_labels) == len(chunk), (_labels, chunk)\n labels.extend(_labels)\n cumulative_len.append(cumulative_len[-1] + len(chunk))\n position_ids.extend(list(range(len(chunk))))\n begin_sample_idx = begin_sample_idx + 1\n begin_token_id = 0\n\n sample_idx = self.shuffled_indices[end_sample_idx]\n sample = self.dataset[sample_idx]\n chunk = sample['input_ids'][begin_token_id:\n end_token_id] # fragment of a sample\n _labels = sample['labels'][begin_token_id:end_token_id]\n pack.extend(chunk)\n assert len(_labels) == len(chunk), (_labels, chunk)\n labels.extend(_labels)\n cumulative_len.append(cumulative_len[-1] + len(chunk))\n position_ids.extend(list(range(len(chunk))))\n\n out = {\n 'input_ids': pack,\n 'cumulative_len': cumulative_len,\n 'position_ids': position_ids,\n 'labels': labels\n }\n return out\n\n def __getitem__(self, item: int):\n pos_before, token_id_before, pos_after, token_id_after = self.mapping(\n item)\n return self.build_pack(pos_before, token_id_before, pos_after,\n token_id_after)\n\n\ndef load_intern_repo_tokenized_dataset(folder,\n min_length=0,\n data_order_path=None,\n file_type='.bin'):\n assert os.path.exists(folder), f'{folder} does not exist.'\n datasets = []\n\n if data_order_path is not None:\n data_order = load_dataset(\n 'text', data_files=data_order_path, split='train')['text']\n for i, fp in enumerate(data_order):\n data_order[i] = os.path.join(folder, fp)\n else:\n triples = list(os.walk(folder, followlinks=True))\n data_order = []\n for root, dirs, files in triples:\n dirs.sort()\n for fn in sorted(files):\n if fn.endswith(file_type):\n fp = os.path.join(root, fn)\n data_order.append(fp)\n\n for fp in data_order:\n print_log(f'Reading {fp}...', logger='current')\n ds = JsonlDataset(fp, min_length=min_length)\n\n if len(ds) == 0:\n continue\n datasets.append(ds)\n\n return datasets\n\n\ndef load_intern_repo_untokenized_dataset(processed_dataset_dict_path=None,\n folder=None,\n tokenizer=None,\n max_length=None,\n template_map_fn=None,\n data_order_path=None,\n file_type='.json'):\n\n assert processed_dataset_dict_path or (folder and tokenizer and max_length)\n\n if processed_dataset_dict_path is not None:\n ds = load_from_disk(processed_dataset_dict_path)\n datasets = []\n for key, data in ds.items():\n datasets.append((key, data))\n datasets = sorted(datasets, key=lambda x: int(x[0]))\n datasets = [x[1] for x in datasets]\n return datasets\n\n assert os.path.exists(folder), f'{folder} does not exist.'\n datasets = []\n\n if data_order_path is not None:\n data_order = load_dataset(\n 'text', data_files=data_order_path, split='train')['text']\n for i, fp in enumerate(data_order):\n data_order[i] = os.path.join(folder, fp)\n else:\n triples = list(os.walk(folder, followlinks=True))\n data_order = []\n for root, dirs, files in triples:\n dirs.sort()\n for fn in sorted(files):\n if fn.endswith(file_type):\n fp = os.path.join(root, fn)\n data_order.append(fp)\n\n for fp in data_order:\n print_log(f'Reading {fp}...', logger='current')\n dataset = []\n with open(fp) as file:\n lines = file.readlines()\n for line in lines:\n line = json.loads(line)\n dataset.append({'messages': line})\n dataset = Dataset.from_list(dataset)\n dataset = process(\n dataset,\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=openai_map_fn,\n template_map_fn=template_map_fn,\n remove_unused_columns=True,\n pack_to_max_length=False,\n map_num_proc=32)\n\n if len(dataset) == 0:\n continue\n\n datasets.append(dataset)\n\n return datasets\n\n\ndef build_packed_dataset_rank0(dataset_cfg, packed_length=8192, seed=1024):\n if isinstance(dataset_cfg, dict):\n datasets = BUILDER.build(dataset_cfg)\n else:\n datasets = dataset_cfg\n\n if not isinstance(datasets, list):\n datasets = [datasets]\n\n packed_datasets = []\n\n for dataset in datasets:\n ds = PackedDataset(dataset, packed_length, seed=seed)\n packed_datasets.append(ds)\n\n dataset = ConcatDataset(datasets=packed_datasets)\n\n return dataset\n\n\ndef build_packed_dataset(*args, **kwargs):\n if not (dist.is_available() and dist.is_initialized()):\n return build_packed_dataset_rank0(*args, **kwargs)\n\n if dist.get_rank() == 0:\n dataset = build_packed_dataset_rank0(*args, **kwargs)\n objects = [dataset]\n else:\n objects = [None]\n dist.broadcast_object_list(objects, src=0)\n return objects[0]\n"
},
{
"path": "xtuner-eval_niah/xtuner/dataset/json_dataset.py",
"content": "import json\nimport os\n\nfrom datasets import Dataset, concatenate_datasets\n\n\ndef load_json_file(data_files=None, data_dir=None, suffix=None):\n assert (data_files is not None) != (data_dir is not None)\n if data_dir is not None:\n data_files = os.listdir(data_dir)\n data_files = [os.path.join(data_dir, fn) for fn in data_files]\n if suffix is not None:\n data_files = [fp for fp in data_files if fp.endswith(suffix)]\n elif isinstance(data_files, str):\n data_files = [data_files]\n\n dataset_list = []\n for fp in data_files:\n with open(fp, encoding='utf-8') as file:\n data = json.load(file)\n ds = Dataset.from_list(data)\n dataset_list.append(ds)\n dataset = concatenate_datasets(dataset_list)\n return dataset\n"
},
{
"path": "xtuner-eval_niah/xtuner/dataset/llava.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nimport json\nimport logging\nimport os\n\nimport torch\nfrom datasets import Dataset as HFDataset\nfrom datasets import DatasetDict, load_from_disk\nfrom mmengine import print_log\nfrom mmengine.config import Config, ConfigDict\nfrom PIL import Image\nfrom torch.utils.data import Dataset\n\nfrom xtuner.registry import BUILDER\nfrom .huggingface import process_hf_dataset\nfrom .utils import expand2square\n\n\ndef load_jsonl(json_file):\n with open(json_file) as f:\n lines = f.readlines()\n data = []\n for line in lines:\n data.append(json.loads(line))\n return data\n\n\nclass LLaVADataset(Dataset):\n\n def __init__(self,\n image_folder,\n image_processor,\n data_path=None,\n tokenizer=None,\n offline_processed_text_folder=None,\n max_dataset_length=None,\n dataset_map_fn=None,\n template_map_fn=None,\n max_length=2048,\n pad_image_to_square=False):\n super().__init__()\n\n assert offline_processed_text_folder or (data_path and tokenizer)\n if offline_processed_text_folder and data_path:\n print_log(\n 'Both `offline_processed_text_folder` and '\n '`data_path` are set, and we load dataset from'\n '`offline_processed_text_folder` '\n f'({offline_processed_text_folder})',\n logger='current',\n level=logging.WARNING)\n\n if offline_processed_text_folder is not None:\n self.text_data = load_from_disk(offline_processed_text_folder)\n else:\n if data_path.endswith('.json'):\n json_data = json.load(open(data_path))\n elif data_path.endswith('.jsonl'):\n json_data = load_jsonl(data_path)\n else:\n raise NotImplementedError\n\n for idx in range(len(json_data)):\n if isinstance(json_data[idx]['id'], int):\n json_data[idx]['id'] = str(json_data[idx]['id'])\n json_data = DatasetDict({'train': HFDataset.from_list(json_data)})\n self.text_data = process_hf_dataset(\n dataset=json_data,\n tokenizer=tokenizer,\n max_length=max_length,\n dataset_map_fn=dataset_map_fn,\n template_map_fn=template_map_fn,\n split='train',\n max_dataset_length=max_dataset_length,\n remove_unused_columns=False,\n pack_to_max_length=False,\n with_image_token=True)\n\n self.image_folder = image_folder\n if isinstance(image_processor, dict) or isinstance(\n image_processor, Config) or isinstance(image_processor,\n ConfigDict):\n self.image_processor = BUILDER.build(image_processor)\n else:\n self.image_processor = image_processor\n self.pad_image_to_square = pad_image_to_square\n\n @property\n def modality_length(self):\n length_list = []\n for data_dict in self.text_data:\n cur_len = len(data_dict['input_ids'])\n if data_dict.get('image', None) is None:\n cur_len = -cur_len\n length_list.append(cur_len)\n return length_list\n\n def __len__(self):\n return len(self.text_data)\n\n def __getitem__(self, index):\n data_dict = self.text_data[index]\n if data_dict.get('image', None) is not None:\n image_file = data_dict['image']\n image = Image.open(os.path.join(self.image_folder,\n image_file)).convert('RGB')\n if self.pad_image_to_square:\n image = expand2square(\n image,\n tuple(\n int(x * 255) for x in self.image_processor.image_mean))\n image = self.image_processor.preprocess(\n image, return_tensors='pt')['pixel_values'][0]\n data_dict['pixel_values'] = image\n else:\n if hasattr(self.image_processor, 'crop_size'):\n crop_size = self.image_processor.crop_size\n else:\n crop_size = self.image_processor.size\n data_dict['pixel_values'] = torch.zeros(3, crop_size['height'],\n crop_size['width'])\n return data_dict\n"
},
{
"path": "xtuner-eval_niah/xtuner/dataset/map_fns/__init__.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nfrom .dataset_map_fns import * # noqa: F401, F403\nfrom .template_map_fn import template_map_fn # noqa: F401\nfrom .template_map_fn import template_map_fn_factory # noqa: F401\n"
},
{
"path": "xtuner-eval_niah/xtuner/dataset/map_fns/dataset_map_fns/__init__.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\nfrom .alpaca_map_fn import alpaca_map_fn\nfrom .alpaca_zh_map_fn import alpaca_zh_map_fn\nfrom .arxiv_map_fn import arxiv_map_fn\nfrom .code_alpaca_map_fn import code_alpaca_map_fn\nfrom .colors_map_fn import colors_map_fn\nfrom .crime_kg_assitant_map_fn import crime_kg_assitant_map_fn\nfrom .default_map_fn import default_map_fn\nfrom .law_reference_map_fn import law_reference_map_fn\nfrom .llava_map_fn import llava_image_only_map_fn, llava_map_fn\nfrom .medical_map_fn import medical_map_fn\nfrom .msagent_map_fn import msagent_react_map_fn\nfrom .oasst1_map_fn import oasst1_map_fn\nfrom .openai_map_fn import openai_map_fn\nfrom .openorca_map_fn import openorca_map_fn\nfrom .pretrain_map_fn import pretrain_map_fn\nfrom .sql_map_fn import sql_map_fn\nfrom .stack_exchange_map_fn import stack_exchange_map_fn\nfrom .tiny_codes_map_fn import tiny_codes_map_fn\nfrom .wizardlm_map_fn import wizardlm_map_fn\n\nDATASET_FORMAT_MAPPING = dict(\n alpaca=alpaca_map_fn,\n alpaca_zh=alpaca_zh_map_fn,\n arxiv=arxiv_map_fn,\n code_alpaca=code_alpaca_map_fn,\n colors=colors_map_fn,\n crime_kg_assitan=crime_kg_assitant_map_fn,\n default=default_map_fn,\n law_reference=law_reference_map_fn,\n llava_image_only=llava_image_only_map_fn,\n llava=llava_map_fn,\n medical=medical_map_fn,\n msagent_react=msagent_react_map_fn,\n oasst1=oasst1_map_fn,\n openai=openai_map_fn,\n openorca=openorca_map_fn,\n pretrain=pretrain_map_fn,\n sql=sql_map_fn,\n stack_exchange=stack_exchange_map_fn,\n tiny_codes=tiny_codes_map_fn,\n wizardlm=wizardlm_map_fn,\n)\n\n__all__ = [\n 'alpaca_map_fn', 'alpaca_zh_map_fn', 'oasst1_map_fn', 'arxiv_map_fn',\n 'medical_map_fn', 'openorca_map_fn', 'code_alpaca_map_fn',\n 'tiny_codes_map_fn', 'colors_map_fn', 'law_reference_map_fn',\n 'crime_kg_assitant_map_fn', 'sql_map_fn', 'openai_map_fn',\n 'wizardlm_map_fn', 'stack_exchange_map_fn', 'msagent_react_map_fn',\n 'pretrain_map_fn', 'default_map_fn', 'llava_image_only_map_fn',\n 'llava_map_fn', 'DATASET_FORMAT_MAPPING'\n]\n"
},
{
"path": "xtuner-eval_niah/xtuner/dataset/map_fns/dataset_map_fns/alpaca_map_fn.py",
"content": "# Copyright (c) OpenMMLab. All rights reserved.\n\n\ndef alpaca_map_fn(example):\n if example.get('output') == '