[
{
"path": "LICENSE.md",
"content": " MIT License\n\n Copyright (c) Microsoft Corporation.\n\n Permission is hereby granted, free of charge, to any person obtaining a copy\n of this software and associated documentation files (the \"Software\"), to deal\n in the Software without restriction, including without limitation the rights\n to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n copies of the Software, and to permit persons to whom the Software is\n furnished to do so, subject to the following conditions:\n\n The above copyright notice and this permission notice shall be included in all\n copies or substantial portions of the Software.\n\n THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n SOFTWARE\n"
},
{
"path": "README.md",
"content": "# LoRA: Low-Rank Adaptation of Large Language Models\n\nThis repo contains the source code of the Python package `loralib` and several examples of how to integrate it with PyTorch models, such as those in Hugging Face.\nWe only support PyTorch for now.\nSee our paper for a detailed description of LoRA.\n\n**LoRA: Low-Rank Adaptation of Large Language Models**
\n*Edward J. Hu\\*, Yelong Shen\\*, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen*
\nPaper: https://arxiv.org/abs/2106.09685
\nVideo explainer: https://www.youtube.com/watch?v=DhRoTONcyZE
\n\n*Update 2/2023: LoRA is now supported by the [State-of-the-art Parameter-Efficient Fine-Tuning (PEFT)](https://github.com/huggingface/peft) library by Hugging Face.*\n\nLoRA reduces the number of trainable parameters by learning pairs of rank-decompostion matrices while freezing the original weights.\nThis vastly reduces the storage requirement for large language models adapted to specific tasks and enables efficient task-switching during deployment all without introducing inference latency.\nLoRA also outperforms several other adaptation methods including adapter, prefix-tuning, and fine-tuning.\n\nWe obtain result comparable or superior to full finetuning on the GLUE benchmark using [RoBERTa (Liu et al., 2019)](https://arxiv.org/abs/1907.11692) base and large and [DeBERTa (He et al., 2020)](https://arxiv.org/abs/2006.03654) XXL 1.5B, while only training and storing a fraction of the parameters. Click the numbers below to download the RoBERTa and DeBERTa LoRA checkpoints.\n\n| | | RoBERTa base
Fine-tune | RoBERTa base
LoRA | DeBERTa XXL
Fine-tune | DeBERTa XXL
LoRA |\n|---|-------------------------|----------------|--------------------------|-----------------|-----------------|\n| | # of Trainable Params. | 125M | 0.8M | 1.5B | 4.7M |\n| | MNLI (m-Acc/mm-Acc) | 87.6 | [87.5±.3/86.9±.3](https://github.com/microsoft/LoRA/releases/download/RoBERTa-base/roberta_base_lora_mnli.bin) |91.7/91.9| [91.9±.1/91.9±.2](https://github.com/microsoft/LoRA/releases/download/DeBERTa/deberta_v2_xxlarge_lora_mnli.bin) |\n| | SST2 (Acc) | 94.8 | [95.1±.2](https://github.com/microsoft/LoRA/releases/download/RoBERTa-base/roberta_base_lora_sst2.bin) | 97.2 | [96.9±.2](https://github.com/microsoft/LoRA/releases/download/DeBERTa/deberta_v2_xxlarge_lora_sst2.bin) |\n| | MRPC (Acc) | 90.2 | [89.7±.7](https://github.com/microsoft/LoRA/releases/download/RoBERTa-base/roberta_base_lora_mrpc.bin) | 92.0 | [92.6±.6](https://github.com/microsoft/LoRA/releases/download/DeBERTa/deberta_v2_xxlarge_lora_mrpc.bin) |\n| | CoLA (Matthew's Corr) | 63.6 | [63.4±1.2](https://github.com/microsoft/LoRA/releases/download/RoBERTa-base/roberta_base_lora_cola.bin) | 72.0 | [72.4±1.1](https://github.com/microsoft/LoRA/releases/download/DeBERTa/deberta_v2_xxlarge_lora_cola.bin) |\n| | QNLI (Acc) | 92.8 | [93.3±.3](https://github.com/microsoft/LoRA/releases/download/RoBERTa-base/roberta_base_lora_qnli.bin) | 96.0 | [96.0±.1](https://github.com/microsoft/LoRA/releases/download/DeBERTa/deberta_v2_xxlarge_lora_qnli.bin) |\n| | QQP (Acc) | 91.9 | [90.8±.1](https://github.com/microsoft/LoRA/releases/download/RoBERTa-base/roberta_base_lora_qqp.bin) | 92.7 | [92.9±.1](https://github.com/microsoft/LoRA/releases/download/DeBERTa/deberta_v2_xxlarge_lora_qqp.bin) |\n| | RTE (Acc) | 78.7 | [86.6±.7](https://github.com/microsoft/LoRA/releases/download/RoBERTa-base/roberta_base_lora_rte.bin) | 93.9 | [94.9±.4](https://github.com/microsoft/LoRA/releases/download/DeBERTa/deberta_v2_xxlarge_lora_rte.bin) |\n| | STSB (Pearson/Spearman Corr) | 91.2 | [91.5±.2/91.3±.2](https://github.com/microsoft/LoRA/releases/download/RoBERTa-base/roberta_base_lora_stsb.bin) |92.9/92.6| [93.0±.2/92.9±.3](https://github.com/microsoft/LoRA/releases/download/DeBERTa/deberta_v2_xxlarge_lora_stsb.bin) |\n| | Average | 86.40 | 87.24 | 91.06 | 91.32 |\n\nNote: You still need the original pre-trained checkpoint from [Hugging Face](https://huggingface.co/) to use the LoRA checkpoints.\n\nFine-tuning numbers are taken from [Liu et al. (2019)](https://arxiv.org/abs/1907.11692) and [He et al. (2020)](https://arxiv.org/abs/2006.03654). We include confidence intervals on results from our experiments. Please follow the instructions in `examples/NLU/` to reproduce our results.\n\nOn GPT-2, LoRA compares favorably to both full finetuning and other efficient tuning methods, such as [adapter (Houlsby et al., 2019)](https://arxiv.org/abs/1902.00751) and [prefix tuning (Li and Liang, 2021)](https://arxiv.org/abs/2101.00190). We evaluated on E2E NLG Challenge, DART, and WebNLG:\n\n| | Method | # of Trainable Params | E2E (BLEU) | DART (BLEU) | WebNLG (BLEU-U/S/A) |\n|---|---------------------|-----------------------|--------------|--------------|--------------------------------|\n| | GPT-2 M (Fine-Tune) | 354.92M | 68.2 | 46.0 | 30.4/63.2/47.6 |\n| | GPT-2 M (Adapter) | 0.37M | 66.3 | 42.4 | 45.1/54.5/50.2 |\n| | GPT-2 M (Prefix) | 0.35M | 69.7 | 45.7 | 44.1/63.1/54.4 |\n| | GPT-2 M (LoRA) | 0.35M |70.4±.1|47.1±.2| 46.7±.4/62.1±.2/55.3±.2 |\n| | GPT-2 L (Fine-Tune) | 774.03M | 68.5 | 46.5 | 41.7/64.6/54.2 |\n| | GPT-2 L (Adapter) | 0.88M | 69.1±.1 | 45.7±.1 | 49.8±.0/61.1±.0/56.0±.0 |\n| | GPT-2 L (Prefix) | 0.77M | 70.3 | 46.5 | 47.0/64.2/56.4 |\n| | GPT-2 L (LoRA) | 0.77M |70.4±.1|47.5±.1| 48.4±.3/64.0±.3/57.0±.1 |\n\nNon-LoRA baselines, except for adapter on GPT-2 large, are taken from [Li and Liang (2021)](https://arxiv.org/abs/2101.00190). We include confidence intervals on results from our experiments.\n\nDownload the GPT-2 LoRA checkpoints:\n * [GPT-2 Medium E2E](https://github.com/microsoft/LoRA/releases/download/GPT-2/gpt2_md_lora_e2e.pt) (1.5 MB)\n * [GPT-2 Medium DART](https://github.com/microsoft/LoRA/releases/download/GPT-2/gpt2_md_lora_dart.pt) (1.5 MB)\n * [GPT-2 Medium WebNLG](https://github.com/microsoft/LoRA/releases/download/GPT-2/gpt2_md_lora_webnlg.pt) (1.5 MB)\n * [GPT-2 Large E2E](https://github.com/microsoft/LoRA/releases/download/GPT-2/gpt2_lg_lora_e2e.pt) (2.3 MB)\n * [GPT-2 Large DART](https://github.com/microsoft/LoRA/releases/download/GPT-2/gpt2_lg_lora_dart.pt) (2.3 MB)\n * [GPT-2 Large WebNLG](https://github.com/microsoft/LoRA/releases/download/GPT-2/gpt2_lg_lora_webnlg.pt) (2.3 MB)\n\nPlease follow the instructions in `examples/NLG/` to reproduce our result.\n## Repository Overview\n\n(The initial release of this repo has been archived in the branch \"snapshot-9-15-2021\")\n\nThere are several directories in this repo:\n* [loralib/](loralib) contains the source code for the package `loralib`, which needs to be installed to run the examples we provide;\n* [examples/NLG/](examples/NLG) contains an example implementation of LoRA in GPT-2 using our package, which can be used to reproduce the result in our paper;\n* [examples/NLU/](examples/NLU) contains an example implementation of LoRA in RoBERTa and DeBERTa using our package, which produces competitive results on the GLUE benchmark;\n* See how we use `loralib` in [GPT-2](examples/NLG/src/model.py), [RoBERTa](examples/NLU/src/transformers/models/roberta/modeling_roberta.py), and [DeBERTa v2](examples/NLU/src/transformers/models/deberta_v2/modeling_deberta_v2.py)\n\n## Quickstart\n\n 1. Installing `loralib` is simply\n ```bash\n pip install loralib\n # Alternatively\n # pip install git+https://github.com/microsoft/LoRA\n ```\n\n 2. You can choose to adapt some layers by replacing them with counterparts implemented in `loralib`. We only support `nn.Linear`, `nn.Embedding`, and `nn.Conv2d` for now. We also support a `MergedLinear` for cases where a single `nn.Linear` represents more than one layers, such as in some implementations of the attention `qkv` projection (see Additional Notes for more).\n ```python\n # ===== Before =====\n # layer = nn.Linear(in_features, out_features)\n\n # ===== After ======\n import loralib as lora\n # Add a pair of low-rank adaptation matrices with rank r=16\n layer = lora.Linear(in_features, out_features, r=16)\n ```\n\n 3. Before the training loop begins, mark only LoRA parameters as trainable.\n ```python\n import loralib as lora\n model = BigModel()\n # This sets requires_grad to False for all parameters without the string \"lora_\" in their names\n lora.mark_only_lora_as_trainable(model)\n # Training loop\n for batch in dataloader:\n ...\n ```\n 4. When saving a checkpoint, generate a `state_dict` that only contains LoRA parameters.\n ```python\n # ===== Before =====\n # torch.save(model.state_dict(), checkpoint_path)\n # ===== After =====\n torch.save(lora.lora_state_dict(model), checkpoint_path)\n ```\n 5. When loading a checkpoint using `load_state_dict`, be sure to set `strict=False`.\n ```python\n # Load the pretrained checkpoint first\n model.load_state_dict(torch.load('ckpt_pretrained.pt'), strict=False)\n # Then load the LoRA checkpoint\n model.load_state_dict(torch.load('ckpt_lora.pt'), strict=False)\n ```\n\n#### Now training can proceed as usual.\n\n## Additional Notes\n\n1. While we focus on a simple yet effect setup, namely adapting only the `q` and `v` projection in a Transformer, in our examples, LoRA can be apply to any subsets of pre-trained weights. We encourage you to explore different configurations, such as adapting the embedding layer by replacing `nn.Embedding` with `lora.Embedding` and/or adapting the MLP layers. It's very likely that the optimal configuration varies for different model architectures and tasks.\n\n2. Some Transformer implementation uses a single `nn.Linear` for the projection matrices for query, key, and value. If one wishes to constrain the rank of the updates to the individual matrices, one has to either break it up into three separate matrices or use `lora.MergedLinear`. Make sure to modify the checkpoint accordingly if you choose to break up the layer.\n```python\n# ===== Before =====\n# qkv_proj = nn.Linear(d_model, 3*d_model)\n# ===== After =====\n# Break it up (remember to modify the pretrained checkpoint accordingly)\nq_proj = lora.Linear(d_model, d_model, r=8)\nk_proj = nn.Linear(d_model, d_model)\nv_proj = lora.Linear(d_model, d_model, r=8)\n# Alternatively, use lora.MergedLinear (recommended)\nqkv_proj = lora.MergedLinear(d_model, 3*d_model, r=8, enable_lora=[True, False, True])\n```\n3. Training bias vectors in tandem with LoRA might be a cost-efficient way to squeeze out extra task performance (if you tune the learning rate carefully). While we did not study its effect thoroughly in our paper, we make it easy to try in `lora`. You can mark some biases as trainable by passing \"all\" or \"lora_only\" to `bias=` when calling `mark_only_lora_as_trainable`. Remember to pass the corresponding `bias=` argument to `lora_state_dict` when saving a checkpoint.\n```python\n# ===== Before =====\n# lora.mark_only_lora_as_trainable(model) # Not training any bias vectors\n# ===== After =====\n# Training all bias vectors associated with modules we apply LoRA to \nlora.mark_only_lora_as_trainable(model, bias='lora_only')\n# Alternatively, we can train *all* bias vectors in the model, including LayerNorm biases\nlora.mark_only_lora_as_trainable(model, bias='all')\n# When saving a checkpoint, use the same bias= ('all' or 'lora_only')\ntorch.save(lora.lora_state_dict(model, bias='all'), checkpoint_path)\n```\n4. Calling `model.eval()` will trigger the merging of LoRA parameters with the corresponding pretrained ones, which eliminates additional latency for subsequent forward passes. Calling `model.train()` again will undo the merge. This can be disabled by passing `merge_weights=False` to LoRA layers.\n\n## Contact\nPlease contact us or post an issue if you have any questions.\n\nFor questions related to the package `loralib`:\n* Edward Hu (edward@edwardjhu.com)\n* Phillip Wallis (phwallis@microsoft.com)\n* Weizhu Chen (wzchen@microsoft.com)\n\nThe GPT-2 example:\n* Phillip Wallis (phwallis@microsoft.com)\n* Yelong Shen (yeshe@microsoft.com)\n\nThe RoBERTa/DeBERTa example:\n* Lu Wang (luw@microsoft.com)\n\n## Acknowledgements\nWe thank in alphabetical order Jianfeng Gao, Jade Huang, Jiayuan Huang, Lisa Xiang Li, Xiaodong Liu, Yabin Liu, Benjamin Van Durme, Luis Vargas, Haoran Wei, Peter Welinder, and Greg Yang for providing valuable feedback.\n\n## Citation\n```BibTeX\n@inproceedings{\nhu2022lora,\ntitle={Lo{RA}: Low-Rank Adaptation of Large Language Models},\nauthor={Edward J Hu and Yelong Shen and Phillip Wallis and Zeyuan Allen-Zhu and Yuanzhi Li and Shean Wang and Lu Wang and Weizhu Chen},\nbooktitle={International Conference on Learning Representations},\nyear={2022},\nurl={https://openreview.net/forum?id=nZeVKeeFYf9}\n}\n```\n\n## Contributing\n\nThis project welcomes contributions and suggestions. Most contributions require you to agree to a\nContributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us\nthe rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.\n\nWhen you submit a pull request, a CLA bot will automatically determine whether you need to provide\na CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions\nprovided by the bot. You will only need to do this once across all repos using our CLA.\n\nThis project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).\nFor more information see the [Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/) or\ncontact [opencode@microsoft.com](mailto:opencode@microsoft.com) with any additional questions or comments.\n"
},
{
"path": "examples/NLG/CODE_OF_CONDUCT.md",
"content": "# Microsoft Open Source Code of Conduct\n\nThis project has adopted the [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/).\n\nResources:\n\n- [Microsoft Open Source Code of Conduct](https://opensource.microsoft.com/codeofconduct/)\n- [Microsoft Code of Conduct FAQ](https://opensource.microsoft.com/codeofconduct/faq/)\n- Contact [opencode@microsoft.com](mailto:opencode@microsoft.com) with questions or concerns\n"
},
{
"path": "examples/NLG/LICENSE",
"content": " MIT License\n\n Copyright (c) Microsoft Corporation.\n\n Permission is hereby granted, free of charge, to any person obtaining a copy\n of this software and associated documentation files (the \"Software\"), to deal\n in the Software without restriction, including without limitation the rights\n to use, copy, modify, merge, publish, distribute, sublicense, and/or sell\n copies of the Software, and to permit persons to whom the Software is\n furnished to do so, subject to the following conditions:\n\n The above copyright notice and this permission notice shall be included in all\n copies or substantial portions of the Software.\n\n THE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\n IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\n FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\n AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\n LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\n OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\n SOFTWARE\n"
},
{
"path": "examples/NLG/README.md",
"content": "# Adapting GPT-2 using LoRA\n\nThis folder contains the implementation of LoRA in GPT-2 using the Python package `lora` and steps to replicate the results in our recent paper\n\n**LoRA: Low-Rank Adaptation of Large Language Models**
\n*Edward J. Hu\\*, Yelong Shen\\*, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen*
\nPaper: https://arxiv.org/abs/2106.09685
\n\n\n![](\"figures/LoRA_GPT2.PNG\")
\n
\n\nThis repo reproduces our experiments on GPT-2.\n\n## Repository Overview\n\nOur implementation is based on the fine-tuning code for GPT-2 in [Hugging Face](https://huggingface.co/).\nThere are several directories in this repo:\n* [src/](src) contains the source code used for data processing, training, and decoding.\n* [eval/](eval) contains the code for task-specific evaluation scripts.\n* [data/](data) contains the raw data we used in our experiments.\n* [vocab/](vocab) contains the GPT-2 vocabulary files.\n\n## Getting Started\n\n 1. You can start with the following docker image: `nvcr.io/nvidia/pytorch:20.03-py3` on a GPU-capable machine, but any generic PyTorch image should work.\n ```\n docker run -it nvcr.io/nvidia/pytorch:20.03-py3\n ```\n\n 2. Clone the repo and install dependencies in a virtual environment (remove sudo if running in docker container):\n ```\n sudo apt-get update\n sudo apt-get -y install git jq virtualenv\n git clone https://github.com/microsoft/LoRA.git; cd LoRA\n virtualenv -p `which python3` ./venv\n . ./venv/bin/activate\n pip install -r requirement.txt\n bash download_pretrained_checkpoints.sh\n bash create_datasets.sh\n cd ./eval\n bash download_evalscript.sh\n cd ..\n ```\n\n#### Now we are ready to replicate the results in our paper.\n\n## Replicating Our Result on E2E\n\n1. Train GPT-2 Medium with LoRA (see our paper for hyperparameters for GPT-2 Medium)\n```\npython -m torch.distributed.launch --nproc_per_node=1 src/gpt2_ft.py \\\n --train_data ./data/e2e/train.jsonl \\\n --valid_data ./data/e2e/valid.jsonl \\\n --train_batch_size 8 \\\n --grad_acc 1 \\\n --valid_batch_size 4 \\\n --seq_len 512 \\\n --model_card gpt2.md \\\n --init_checkpoint ./pretrained_checkpoints/gpt2-medium-pytorch_model.bin \\\n --platform local \\\n --clip 0.0 \\\n --lr 0.0002 \\\n --weight_decay 0.01 \\\n --correct_bias \\\n --adam_beta2 0.999 \\\n --scheduler linear \\\n --warmup_step 500 \\\n --max_epoch 5 \\\n --save_interval 1000 \\\n --lora_dim 4 \\\n --lora_alpha 32 \\\n --lora_dropout 0.1 \\\n --label_smooth 0.1 \\\n --work_dir ./trained_models/GPT2_M/e2e \\\n --random_seed 110\n```\n\n2. Generate outputs from the trained model using beam search:\n```\npython -m torch.distributed.launch --nproc_per_node=1 src/gpt2_beam.py \\\n --data ./data/e2e/test.jsonl \\\n --batch_size 1 \\\n --seq_len 512 \\\n --eval_len 64 \\\n --model_card gpt2.md \\\n --init_checkpoint ./trained_models/GPT2_M/e2e/model.26289.pt \\\n --platform local \\\n --lora_dim 4 \\\n --lora_alpha 32 \\\n --beam 10 \\\n --length_penalty 0.8 \\\n --no_repeat_ngram_size 4 \\\n --repetition_penalty 1.0 \\\n --eos_token_id 628 \\\n --work_dir ./trained_models/GPT2_M/e2e \\\n --output_file predict.26289.b10p08r4.jsonl\n```\n\n3. Decode outputs from step (2)\n```\npython src/gpt2_decode.py \\\n --vocab ./vocab \\\n --sample_file ./trained_models/GPT2_M/e2e/predict.26289.b10p08r4.jsonl \\\n --input_file ./data/e2e/test_formatted.jsonl \\\n --output_ref_file e2e_ref.txt \\\n --output_pred_file e2e_pred.txt\n```\n\n4. Run evaluation on E2E test set\n\n```\npython eval/e2e/measure_scores.py e2e_ref.txt e2e_pred.txt -p\n```\n\n## Replicating Our Result on WebNLG\n\n1. Follow steps 1 and 2 from E2E pipeline by replacing references to E2E with webnlg (see our paper for hyperparameters)\n\n2. Decode outputs from beam search (step 2 above)\n```\npython src/gpt2_decode.py \\\n --vocab ./vocab \\\n --sample_file ./trained_models/GPT2_M/webnlg/predict.20000.b10p08.jsonl \\\n --input_file ./data/webnlg_challenge_2017/test_formatted.jsonl \\\n --ref_type webnlg \\\n --ref_num 6 \\\n --output_ref_file eval/GenerationEval/data/references_webnlg \\\n --output_pred_file eval/GenerationEval/data/hypothesis_webnlg \\\n --tokenize --lower\n```\n\n3. Run evaluation on WebNLG test set\n```\ncd ./eval/GenerationEval/\npython eval.py \\\n -R data/references_webnlg/reference \\\n -H data/hypothesis_webnlg \\\n -nr 6 \\\n -m bleu,meteor,ter \ncd ../..\n```\n\n## Replicating Our Result on DART\n\n1. Follow steps 1 and 2 from E2E pipeline by replacing references to E2E with dart (see our paper for hyperparameters)\n\n2. Decode outputs from beam search (step 2 above)\n```\npython src/gpt2_decode.py \\\n --vocab ./vocab \\\n --sample_file ./trained_models/GPT2_M/dart/predict.20000.b10p08.jsonl \\\n --input_file ./data/dart/test_formatted.jsonl \\\n --ref_type dart \\\n --ref_num 6 \\\n --output_ref_file eval/GenerationEval/data/references_dart \\\n --output_pred_file eval/GenerationEval/data/hypothesis_dart \\\n --tokenize --lower\n```\n\n3. Run evaluation on Dart test set\n```\ncd ./eval/GenerationEval/\npython eval.py \\\n -R data/references_dart/reference \\\n -H data/hypothesis_dart \\\n -nr 6 \\\n -m bleu,meteor,ter \ncd ../..\n```\n\n## Citation\n```\n@misc{hu2021lora,\n title={LoRA: Low-Rank Adaptation of Large Language Models},\n author={Hu, Edward and Shen, Yelong and Wallis, Phil and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Lu and Chen, Weizhu},\n year={2021},\n eprint={2106.09685},\n archivePrefix={arXiv},\n primaryClass={cs.CL}\n}\n```"
},
{
"path": "examples/NLG/SECURITY.md",
"content": "\n\n## Security\n\nMicrosoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/).\n\nIf you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://docs.microsoft.com/en-us/previous-versions/tn-archive/cc751383(v=technet.10)), please report it to us as described below.\n\n## Reporting Security Issues\n\n**Please do not report security vulnerabilities through public GitHub issues.**\n\nInstead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://msrc.microsoft.com/create-report).\n\nIf you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com). If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://www.microsoft.com/en-us/msrc/pgp-key-msrc).\n\nYou should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://www.microsoft.com/msrc). \n\nPlease include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue:\n\n * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.)\n * Full paths of source file(s) related to the manifestation of the issue\n * The location of the affected source code (tag/branch/commit or direct URL)\n * Any special configuration required to reproduce the issue\n * Step-by-step instructions to reproduce the issue\n * Proof-of-concept or exploit code (if possible)\n * Impact of the issue, including how an attacker might exploit the issue\n\nThis information will help us triage your report more quickly.\n\nIf you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://microsoft.com/msrc/bounty) page for more details about our active programs.\n\n## Preferred Languages\n\nWe prefer all communications to be in English.\n\n## Policy\n\nMicrosoft follows the principle of [Coordinated Vulnerability Disclosure](https://www.microsoft.com/en-us/msrc/cvd).\n\n