[ { "path": ".gitignore", "content": "*/__pycache__/*\n*cache" }, { "path": "README.md", "content": "# LLaMA3-Quantization\n\nLLaMA3-Quantization is the official implementation of our paper How Good Are Low-bit Quantized LLAMA3 Models?\nAn Empirical Study [[PDF](https://arxiv.org/abs/2404.14047)]. Created by researchers from The University of Hong Kong, Beihang University and ETH Zürich.\n\n## Introduction\nMeta's LLaMa family has become one of the most powerful open-source Large Language Model (LLM) series. Notably, LLaMa3 models have recently been released and achieve impressive performance across various with super-large scale pre-training on over 15T tokens of data. Given the wide application of low-bit quantization for LLMs in resource-limited scenarios, we explore LLaMa3's capabilities when quantized to low bit-width. This exploration holds the potential to unveil new insights and challenges for low-bit quantization of LLaMa3 and other forthcoming LLMs, especially in addressing performance degradation problems that suffer in LLM compression. Specifically, we evaluate the 10 existing post-training quantization and LoRA-finetuning methods of LLaMa3 on 1-8 bits and diverse datasets to comprehensively reveal LLaMa3's low-bit quantization performance. Our experiment results indicate that LLaMa3 still suffers non-negligent degradation in these scenarios, especially in ultra-low bit-width. This highlights the significant performance gap under low bit-width that needs to be bridged in future developments. We expect that this empirical study will prove valuable in advancing future models, pushing the LLMs to lower bit-width with higher accuracy for being practical. Our project is released on [https://github.com/Macaronlin/LLaMA3-Quantization](https://github.com/Macaronlin/LLaMA3-Quantization) and quantized LLaMa3 models are released in [https://huggingface.co/Efficient-ML](https://huggingface.co/Efficient-ML).\n\n![img](images/overview.png)\n\n## Usage\n\nWe provide full script to evaluate various quantization methods in `./scripts/`. We use LLaMa-3-8B in IR-QLoRA method as an example here:\n\n```shell\npython main.py \\ \n --model meta-llama/Meta-Llama-3-8B \\ \n --peft LLMQ/LLaMA-3-8B-IR-QLoRA \\ \n --tau_range 0.1 --tau_n 100--blocksize 256 \\ \n --epochs 0 \\ \n --output_dir ./log/llama-3-8b-irqlora \\ \n --wbits 4 \\ \n --tasks piqa,arc_easy,arc_challenge,hellaswag,winogrande\n```\n\n## Results\n\n### Track1: Post-Training Quantization\n\n- Evaluation results of post-training quantization on LLAMA3-8B model.\n ![img](images/result_ptq_1.png)\n\n- Evaluation results of post-training quantization on LLAMA3-70B model.\n ![img](images/result_ptq_2.png)\n\n### Track2: LoRA-FineTuning Quantization\n- LoRA-FT on LLAMA3-8B with Alpaca dataset.\n ![img](images/result_lora_ft_1.png)\n\n## Related Project\n\n[QUIP](https://github.com/Cornell-RelaxML/QuIP)\n\n[GPTQ: Accurate Post-training Compression for Generative Pretrained Transformers](https://github.com/IST-DASLab/gptq)\n\n[AutoGPTQ](https://github.com/PanQiWei/AutoGPTQ)\n\n[AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://github.com/mit-han-lab/llm-awq)\n\n[RPTQ: Reorder-Based Post-Training Quantization for Large Language Models](https://github.com/hahnyuan/RPTQ4LLM)\n\n[OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models](https://github.com/OpenGVLab/OmniQuant)\n\n[PB-LLM: Partially Binarized Large Language Models](https://github.com/hahnyuan/PB-LLM)\n\n[BiLLM: Pushing the Limit of Post-Training Quantization for LLMs](https://github.com/Aaronhuang-778/BiLLM)\n\n[SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models](https://github.com/mit-han-lab/smoothquant)\n\n[QLoRA: Efficient Finetuning of Quantized LLMs](https://github.com/artidoro/qlora)\n\n[IR-QLoRA: Accurate LoRA-Finetuning Quantization of LLMs via Information Retention](https://github.com/htqin/IR-QLoRA)\n\n\n\n" }, { "path": "categories.py", "content": "subcategories = {\n \"abstract_algebra\": [\"math\"],\n \"anatomy\": [\"health\"],\n \"astronomy\": [\"physics\"],\n \"business_ethics\": [\"business\"],\n \"clinical_knowledge\": [\"health\"],\n \"college_biology\": [\"biology\"],\n \"college_chemistry\": [\"chemistry\"],\n \"college_computer_science\": [\"computer science\"],\n \"college_mathematics\": [\"math\"],\n \"college_medicine\": [\"health\"],\n \"college_physics\": [\"physics\"],\n \"computer_security\": [\"computer science\"],\n \"conceptual_physics\": [\"physics\"],\n \"econometrics\": [\"economics\"],\n \"electrical_engineering\": [\"engineering\"],\n \"elementary_mathematics\": [\"math\"],\n \"formal_logic\": [\"philosophy\"],\n \"global_facts\": [\"other\"],\n \"high_school_biology\": [\"biology\"],\n \"high_school_chemistry\": [\"chemistry\"],\n \"high_school_computer_science\": [\"computer science\"],\n \"high_school_european_history\": [\"history\"],\n \"high_school_geography\": [\"geography\"],\n \"high_school_government_and_politics\": [\"politics\"],\n \"high_school_macroeconomics\": [\"economics\"],\n \"high_school_mathematics\": [\"math\"],\n \"high_school_microeconomics\": [\"economics\"],\n \"high_school_physics\": [\"physics\"],\n \"high_school_psychology\": [\"psychology\"],\n \"high_school_statistics\": [\"math\"],\n \"high_school_us_history\": [\"history\"],\n \"high_school_world_history\": [\"history\"],\n \"human_aging\": [\"health\"],\n \"human_sexuality\": [\"culture\"],\n \"international_law\": [\"law\"],\n \"jurisprudence\": [\"law\"],\n \"logical_fallacies\": [\"philosophy\"],\n \"machine_learning\": [\"computer science\"],\n \"management\": [\"business\"],\n \"marketing\": [\"business\"],\n \"medical_genetics\": [\"health\"],\n \"miscellaneous\": [\"other\"],\n \"moral_disputes\": [\"philosophy\"],\n \"moral_scenarios\": [\"philosophy\"],\n \"nutrition\": [\"health\"],\n \"philosophy\": [\"philosophy\"],\n \"prehistory\": [\"history\"],\n \"professional_accounting\": [\"other\"],\n \"professional_law\": [\"law\"],\n \"professional_medicine\": [\"health\"],\n \"professional_psychology\": [\"psychology\"],\n \"public_relations\": [\"politics\"],\n \"security_studies\": [\"politics\"],\n \"sociology\": [\"culture\"],\n \"us_foreign_policy\": [\"politics\"],\n \"virology\": [\"health\"],\n \"world_religions\": [\"philosophy\"],\n}\n\ncategories = {\n \"STEM\": [\"physics\", \"chemistry\", \"biology\", \"computer science\", \"math\", \"engineering\"],\n \"humanities\": [\"history\", \"philosophy\", \"law\"],\n \"social sciences\": [\"politics\", \"culture\", \"economics\", \"geography\", \"psychology\"],\n \"other (business, health, misc.)\": [\"other\", \"business\", \"health\"],\n}" }, { "path": "datautils.py", "content": "import pdb\nfrom transformers import AutoTokenizer\nfrom datasets import load_dataset\nimport numpy as np\nimport torch\nimport random\n\n\ndef set_seed(seed):\n np.random.seed(seed)\n torch.random.manual_seed(seed)\n\n\n\n\ndef get_pile(nsamples, seed, seqlen, model):\n print(\"get_pile\")\n traindata = load_dataset(\"json\", data_files='/cpfs01/user/chenmengzhao/prompt_quantization/val.jsonl.zst', split=\"train\")\n\n tokenizer = AutoTokenizer.from_pretrained(model, use_fast=False)\n trainenc = tokenizer(\"\\n\\n\".join(traindata['text'][:1000]), return_tensors='pt')\n\n random.seed(seed)\n trainloader = []\n for _ in range(nsamples):\n i = random.randint(0, trainenc.input_ids.shape[1] - seqlen - 1)\n j = i + seqlen\n inp = trainenc.input_ids[:, i:j]\n tar = inp.clone()\n tar[:, :-1] = -100\n trainloader.append((inp, tar))\n return trainloader, None\n\n\ndef get_wikitext2(nsamples, seed, seqlen, model):\n print(\"get_wikitext2\")\n traindata = load_dataset('wikitext', 'wikitext-2-raw-v1', split='train')\n testdata = load_dataset('wikitext', 'wikitext-2-raw-v1', split='test')\n\n tokenizer = AutoTokenizer.from_pretrained(model, use_fast=False)\n trainenc = tokenizer(\"\\n\\n\".join(traindata['text']), return_tensors='pt')\n testenc = tokenizer(\"\\n\\n\".join(testdata['text']), return_tensors='pt')\n\n \n random.seed(seed)\n trainloader = []\n for _ in range(nsamples):\n i = random.randint(0, trainenc.input_ids.shape[1] - seqlen - 1)\n j = i + seqlen\n inp = trainenc.input_ids[:, i:j]\n tar = inp.clone()\n tar[:, :-1] = -100\n trainloader.append((inp, tar))\n return trainloader, testenc\n\ndef get_ptb(nsamples, seed, seqlen, model):\n print(\"get_ptb\")\n traindata = load_dataset('ptb_text_only', 'penn_treebank', split='train')\n valdata = load_dataset('ptb_text_only', 'penn_treebank', split='validation')\n\n\n tokenizer = AutoTokenizer.from_pretrained(model, use_fast=False)\n\n trainenc = tokenizer(\"\\n\\n\".join(traindata['sentence']), return_tensors='pt')\n testenc = tokenizer(\"\\n\\n\".join(valdata['sentence']), return_tensors='pt')\n\n random.seed(seed)\n trainloader = []\n for _ in range(nsamples):\n i = random.randint(0, trainenc.input_ids.shape[1] - seqlen - 1)\n j = i + seqlen\n inp = trainenc.input_ids[:, i:j]\n tar = inp.clone()\n tar[:, :-1] = -100\n trainloader.append((inp, tar))\n return trainloader, testenc\n\ndef get_c4(nsamples, seed, seqlen, model):\n print(\"get_c4\")\n traindata = load_dataset(\n 'allenai/c4', data_files={'train': 'en/c4-train.00000-of-01024.json.gz'}, split='train'\n )\n valdata = load_dataset(\n 'allenai/c4', data_files={'validation': 'en/c4-validation.00000-of-00008.json.gz'}, split='validation'\n )\n\n\n tokenizer = AutoTokenizer.from_pretrained(model, use_fast=False)\n\n random.seed(seed)\n trainloader = []\n for _ in range(nsamples):\n while True:\n i = random.randint(0, len(traindata) - 1)\n trainenc = tokenizer(traindata[i]['text'], return_tensors='pt')\n if trainenc.input_ids.shape[1] >= seqlen:\n break\n i = random.randint(0, trainenc.input_ids.shape[1] - seqlen - 1)\n j = i + seqlen\n inp = trainenc.input_ids[:, i:j]\n tar = inp.clone()\n tar[:, :-1] = -100\n trainloader.append((inp, tar))\n\n random.seed(0)\n valenc = []\n for _ in range(256):\n while True:\n i = random.randint(0, len(valdata) - 1)\n tmp = tokenizer(valdata[i]['text'], return_tensors='pt')\n if tmp.input_ids.shape[1] >= seqlen:\n break\n i = random.randint(0, tmp.input_ids.shape[1] - seqlen - 1)\n j = i + seqlen\n valenc.append(tmp.input_ids[:, i:j])\n valenc = torch.hstack(valenc)\n\n return trainloader, valenc \n\ndef get_ptb_new(nsamples, seed, seqlen, model):\n print(\"get_ptb_new\")\n traindata = load_dataset('ptb_text_only', 'penn_treebank', split='train')\n testdata = load_dataset('ptb_text_only', 'penn_treebank', split='test')\n\n\n tokenizer = AutoTokenizer.from_pretrained(model, use_fast=False)\n\n trainenc = tokenizer(\" \".join(traindata[\"sentence\"]), return_tensors=\"pt\")\n testenc = tokenizer(\" \".join(testdata [\"sentence\"]), return_tensors=\"pt\")\n\n random.seed(seed)\n trainloader = []\n for _ in range(nsamples):\n i = random.randint(0, trainenc.input_ids.shape[1] - seqlen - 1)\n j = i + seqlen\n inp = trainenc.input_ids[:, i:j]\n tar = inp.clone()\n tar[:, :-1] = -100\n trainloader.append((inp, tar))\n return trainloader, testenc\n\n\ndef get_c4_new(nsamples, seed, seqlen, model):\n print(\"get_c4_new\")\n traindata = load_dataset(\n 'allenai/c4', data_files={'train': 'en/c4-train.00000-of-01024.json.gz'}, split='train'\n )\n valdata = load_dataset(\n 'allenai/c4', data_files={'validation': 'en/c4-validation.00000-of-00008.json.gz'}, split='validation'\n )\n\n tokenizer = AutoTokenizer.from_pretrained(model, use_fast=False)\n \n random.seed(seed)\n trainloader = []\n for _ in range(nsamples):\n while True:\n i = random.randint(0, len(traindata) - 1)\n trainenc = tokenizer(traindata[i][\"text\"], return_tensors=\"pt\")\n if trainenc.input_ids.shape[1] >= seqlen:\n break\n i = random.randint(0, trainenc.input_ids.shape[1] - seqlen - 1)\n j = i + seqlen\n inp = trainenc.input_ids[:, i:j]\n tar = inp.clone()\n tar[:, :-1] = -100\n trainloader.append((inp, tar))\n\n valenc = tokenizer(\" \".join(valdata[:1100][\"text\"]), return_tensors=\"pt\")\n valenc = valenc.input_ids[:, : (256 * seqlen)]\n return trainloader, valenc\n\n\ndef get_loaders(\n name, nsamples=128, seed=0, seqlen=2048, model='',\n):\n if 'wikitext2' in name:\n return get_wikitext2(nsamples, seed, seqlen, model)\n if 'pile' in name:\n return get_pile(nsamples, seed, seqlen, model)\n if 'ptb' in name:\n if 'new' in name:\n return get_ptb_new(nsamples, seed, seqlen, model) \n return get_ptb(nsamples, seed, seqlen, model)\n if 'c4' in name:\n if 'new' in name:\n return get_c4_new(nsamples, seed, seqlen, model) \n return get_c4(nsamples, seed, seqlen, model)\n if 'mix' in name:\n wiki_train,wiki_val=get_wikitext2(nsamples//3, seed, seqlen, model)\n ptb_train,ptb_val=get_ptb(nsamples//3, seed, seqlen, model)\n c4_train,c4_val=get_c4(nsamples//3, seed, seqlen, model)\n train=wiki_train+ptb_train+c4_train\n val=None\n return train,val\n" }, { "path": "gptq.py", "content": "import math\r\nimport time\r\n\r\nimport torch\r\nimport torch.nn as nn\r\nimport transformers\r\nimport quant\r\nfrom texttable import Texttable\r\nfrom utils import torch_snr_error\r\n\r\ntorch.backends.cuda.matmul.allow_tf32 = False\r\ntorch.backends.cudnn.allow_tf32 = False\r\n\r\n\r\nclass Observer:\r\n\r\n def __init__(self, topk=32):\r\n self.loss_list = []\r\n self.topk = topk\r\n\r\n def submit(self, name: str, layerid: int, gptq, error: float):\r\n\r\n item = (name, layerid, {'gptq': gptq, 'error': error})\r\n\r\n if len(self.loss_list) < self.topk:\r\n self.loss_list.append(item)\r\n return\r\n\r\n min_error = error\r\n min_idx = -1\r\n for idx, data in enumerate(self.loss_list):\r\n if min_error > data[2]['error']:\r\n min_idx = idx\r\n min_error = data[2]['error']\r\n\r\n if min_idx >= 0:\r\n self.loss_list[min_idx] = item\r\n\r\n def print(self):\r\n self.loss_list = sorted(self.loss_list, key=lambda s: s[2]['error'], reverse=True)\r\n\r\n table = Texttable()\r\n\r\n table.header(['name', 'error'])\r\n table.set_cols_dtype(['t', 'f'])\r\n\r\n for item in self.loss_list:\r\n table.add_row([f\"{item[0]}.{item[1]}\", item[2]['error']])\r\n print(table.draw())\r\n print('\\n')\r\n\r\n def items(self):\r\n return self.loss_list\r\n\r\n\r\nclass GPTQ:\r\n\r\n def __init__(self, layer, observe=False):\r\n self.layer = layer\r\n self.dev = self.layer.weight.device\r\n W = layer.weight.data.clone()\r\n if isinstance(self.layer, nn.Conv2d):\r\n W = W.flatten(1)\r\n if isinstance(self.layer, transformers.Conv1D):\r\n W = W.t()\r\n self.rows = W.shape[0]\r\n self.columns = W.shape[1]\r\n self.H = torch.zeros((self.columns, self.columns), device=self.dev)\r\n self.nsamples = 0\r\n self.quantizer = quant.Quantizer()\r\n self.observe = observe\r\n\r\n def add_batch(self, inp, out):\r\n # Hessian H = 2 X XT + λ I\r\n if self.observe:\r\n self.inp1 = inp\r\n self.out1 = out\r\n else:\r\n self.inp1 = None\r\n self.out1 = None\r\n\r\n if len(inp.shape) == 2:\r\n inp = inp.unsqueeze(0)\r\n tmp = inp.shape[0]\r\n if isinstance(self.layer, nn.Linear) or isinstance(self.layer, transformers.Conv1D):\r\n if len(inp.shape) == 3:\r\n inp = inp.reshape((-1, inp.shape[-1]))\r\n inp = inp.t()\r\n if isinstance(self.layer, nn.Conv2d):\r\n unfold = nn.Unfold(self.layer.kernel_size, dilation=self.layer.dilation, padding=self.layer.padding, stride=self.layer.stride)\r\n inp = unfold(inp)\r\n inp = inp.permute([1, 0, 2])\r\n inp = inp.flatten(1)\r\n self.H *= self.nsamples / (self.nsamples + tmp)\r\n self.nsamples += tmp\r\n # inp = inp.float()\r\n inp = math.sqrt(2 / self.nsamples) * inp.float()\r\n # self.H += 2 / self.nsamples * inp.matmul(inp.t())\r\n self.H += inp.matmul(inp.t())\r\n\r\n def print_loss(self, name, q_weight, weight_error, timecost):\r\n table = Texttable()\r\n name += ' ' * (16 - len(name))\r\n\r\n table.header(['name', 'weight_error', 'fp_inp_SNR', 'q_inp_SNR', 'time'])\r\n\r\n # assign weight\r\n self.layer.weight.data = q_weight.reshape(self.layer.weight.shape).to(self.layer.weight.data.dtype)\r\n\r\n if self.inp1 is not None:\r\n # quantize input to int8\r\n quantizer = quant.Quantizer()\r\n quantizer.configure(8, perchannel=False, sym=True, mse=False)\r\n quantizer.find_params(self.inp1)\r\n q_in = quantizer.quantize(self.inp1).type(torch.float16)\r\n q_out = self.layer(q_in)\r\n\r\n # get kinds of SNR\r\n q_SNR = torch_snr_error(q_out, self.out1).item()\r\n fp_SNR = torch_snr_error(self.layer(self.inp1), self.out1).item()\r\n else:\r\n q_SNR = '-'\r\n fp_SNR = '-'\r\n\r\n table.add_row([name, weight_error, fp_SNR, q_SNR, timecost])\r\n print(table.draw().split('\\n')[-2])\r\n\r\n def fasterquant(self, blocksize=128, percdamp=.01, groupsize=-1, actorder=False, name=''):\r\n self.layer.to(self.dev)\r\n\r\n W = self.layer.weight.data.clone()\r\n if blocksize == -1:\r\n blocksize = W.shape[1]\r\n print(blocksize)\r\n if isinstance(self.layer, nn.Conv2d):\r\n W = W.flatten(1)\r\n if isinstance(self.layer, transformers.Conv1D):\r\n W = W.t()\r\n W = W.float()\r\n\r\n tick = time.time()\r\n\r\n if not self.quantizer.ready():\r\n self.quantizer.find_params(W, weight=True)\r\n\r\n H = self.H\r\n if not self.observe:\r\n del self.H\r\n dead = torch.diag(H) == 0\r\n H[dead, dead] = 1\r\n W[:, dead] = 0\r\n\r\n if actorder:\r\n perm = torch.argsort(torch.diag(H), descending=True)\r\n W = W[:, perm]\r\n H = H[perm][:, perm]\r\n\r\n Losses = torch.zeros_like(W)\r\n Q = torch.zeros_like(W)\r\n\r\n damp = percdamp * torch.mean(torch.diag(H))\r\n diag = torch.arange(self.columns, device=self.dev)\r\n H[diag, diag] += damp\r\n H = torch.linalg.cholesky(H)\r\n H = torch.cholesky_inverse(H)\r\n H = torch.linalg.cholesky(H, upper=True)\r\n Hinv = H\r\n\r\n g_idx = []\r\n scale = []\r\n zero = []\r\n now_idx = 1\r\n\r\n for i1 in range(0, self.columns, blocksize):\r\n i2 = min(i1 + blocksize, self.columns)\r\n count = i2 - i1\r\n\r\n W1 = W[:, i1:i2].clone()\r\n Q1 = torch.zeros_like(W1)\r\n Err1 = torch.zeros_like(W1)\r\n Losses1 = torch.zeros_like(W1)\r\n Hinv1 = Hinv[i1:i2, i1:i2]\r\n\r\n for i in range(count):\r\n w = W1[:, i]\r\n d = Hinv1[i, i]\r\n\r\n if groupsize != -1:\r\n if (i1 + i) % groupsize == 0:\r\n self.quantizer.find_params(W[:, (i1 + i):(i1 + i + groupsize)], weight=True)\r\n\r\n if ((i1 + i) // groupsize) - now_idx == -1:\r\n scale.append(self.quantizer.scale)\r\n zero.append(self.quantizer.zero)\r\n now_idx += 1\r\n\r\n q = self.quantizer.quantize(w.unsqueeze(1)).flatten()\r\n Q1[:, i] = q\r\n Losses1[:, i] = (w - q)**2 / d**2\r\n\r\n err1 = (w - q) / d\r\n W1[:, i:] -= err1.unsqueeze(1).matmul(Hinv1[i, i:].unsqueeze(0))\r\n Err1[:, i] = err1\r\n\r\n Q[:, i1:i2] = Q1\r\n Losses[:, i1:i2] = Losses1 / 2\r\n\r\n W[:, i2:] -= Err1.matmul(Hinv[i1:i2, i2:])\r\n\r\n torch.cuda.synchronize()\r\n error = torch.sum(Losses).item()\r\n\r\n groupsize = groupsize if groupsize != -1 else self.columns\r\n g_idx = [i // groupsize for i in range(self.columns)]\r\n g_idx = torch.tensor(g_idx, dtype=torch.int32, device=Q.device)\r\n if actorder:\r\n invperm = torch.argsort(perm)\r\n Q = Q[:, invperm]\r\n g_idx = g_idx[invperm]\r\n\r\n if isinstance(self.layer, transformers.Conv1D):\r\n Q = Q.t()\r\n\r\n self.print_loss(name=name, q_weight=Q, weight_error=error, timecost=(time.time() - tick))\r\n\r\n if scale == []:\r\n scale.append(self.quantizer.scale)\r\n zero.append(self.quantizer.zero)\r\n scale = torch.cat(scale, dim=1)\r\n zero = torch.cat(zero, dim=1)\r\n return scale, zero, g_idx, error\r\n\r\n def free(self):\r\n self.inp1 = None\r\n self.out1 = None\r\n self.H = None\r\n self.Losses = None\r\n self.Trace = None\r\n torch.cuda.empty_cache()\r\n" }, { "path": "irqlora.py", "content": "from tqdm import tqdm\nimport peft\nimport torch\nimport operator\nimport numpy as np\nimport bitsandbytes as bnb\nfrom peft.tuners.lora import LoraLayer\nfrom functools import reduce # Required in Python 3\nimport bitsandbytes.functional as bnb_F\nfrom torch import Tensor\nfrom scipy.stats import norm\nfrom bitsandbytes.functional import create_fp8_map, create_dynamic_map\n\ncache_folder_path = ''\nmodule_num = 0\nsigma = 1 / norm.ppf(torch.linspace(0.9677083, 0.5, 9)[:-1]).tolist()[0]\n\ndef replace_to_qlora_model(model, model_fp, blocksize2=256, tau_range=0.1, tau_n=100):\n model.model = _replace_with_ours_lora_4bit_linear(model.model, model_fp=model_fp, blocksize2=blocksize2, tau_range=tau_range, tau_n=tau_n)\n return model\n\ndef prod(iterable):\n return reduce(operator.mul, iterable, 1)\n \nnormal_map_fp8 = create_dynamic_map()\ndef quantize_tensor(X, L, idx=False):\n L = L.to(X.device)\n X_shape = X.shape\n X_expanded = X.reshape(-1, 1)\n L_reshaped = L.reshape(1, -1)\n abs_diff = torch.abs(X_expanded - L_reshaped)\n min_index = torch.argmin(abs_diff, dim=-1)\n min_index = torch.tensor(min_index, dtype=torch.uint8).to(L.device).reshape(X_shape)\n return min_index\n\ndef dequantize_tensor(X, L):\n L = L.to(X.device)\n return torch.index_select(L, dim=0, index=torch.as_tensor(X, dtype=torch.int32).reshape(-1)).reshape(X.shape)\n\n@torch.no_grad()\ndef nf4_quant(weight, weight_shape, tau, compress_statistics, quant_type, device):\n weight = weight.reshape(-1, 256, 64).to(device)\n tau = tau.reshape(-1, 256, 1).to(device)\n _weight = (weight - tau).reshape(weight_shape)\n nf4_weight = bnb.nn.Params4bit(_weight, requires_grad=False, compress_statistics=compress_statistics, quant_type=quant_type).cuda(0)\n tau2 = tau.abs().max(dim=1, keepdim=True)[0]\n tau1 = quantize_tensor(tau / tau2, normal_map_fp8)\n return nf4_weight, tau1.reshape(-1, 256), tau2.reshape(-1, 1)\n\n@torch.no_grad()\ndef evaluate_entropy(weight_int8, blocksize):\n device = weight_int8.device\n _weight_int8 = weight_int8.reshape(-1, 1)\n weight_nf4 = torch.cat((_weight_int8//16, _weight_int8%16), 1).reshape(1, -1, blocksize)\n weight_nf4_repeat = weight_nf4.repeat(16, 1, 1).to(device)\n values = torch.tensor(range(16)).reshape(16, 1, 1).to(device)\n freqs = (weight_nf4_repeat==values).sum(dim=-1, keepdim=True) / blocksize\n entropy = -freqs * torch.log2(freqs)\n entropy = torch.where(torch.isnan(entropy), 0, entropy)\n entropy = entropy.sum(dim=0)\n return entropy\n\n@torch.no_grad()\ndef search(fp_weight: Tensor, fp_weight_shape, compress_statistics, quant_type, device, tau_range=0.1, tau_n=51, blocksize=64, blocksize2=256):\n fp_weight = fp_weight.reshape(-1, blocksize2, blocksize).to(device)\n tau0 = fp_weight.median(2, keepdim=True)[0] # [-1, 256, 1]\n absmax = (fp_weight - tau0).abs().max(2, keepdim=True)[0]\n \n entropy_max, factor_best = None, None\n for factor in tqdm(np.linspace(-tau_range*sigma, tau_range*sigma, tau_n*2+1)):\n tau = factor * absmax + tau0\n nf4_weight, _, _ = nf4_quant(fp_weight, fp_weight_shape, tau, compress_statistics, quant_type, device)\n entropy = evaluate_entropy(nf4_weight, blocksize)\n \n if entropy_max is None:\n entropy_max = entropy\n factor_best = torch.full_like(entropy, factor)\n else:\n factor_best = torch.where(entropy > entropy_max, factor, factor_best)\n entropy_max = torch.max(entropy_max, entropy)\n \n tau = factor_best.reshape(-1, 256, 1) * absmax + tau0\n nf4_weight, tau1, tau2 = nf4_quant(fp_weight, fp_weight_shape, tau, compress_statistics, quant_type, device)\n return nf4_weight, tau1, tau2\n\nclass IRQLoraLinear4bit(bnb.nn.Linear4bit, LoraLayer):\n def __init__(\n self, old_model, model_fp=None, blocksize2=256, tau_range=0.1, tau_n=51\n ):\n for key, value in old_model.__dict__.items():\n setattr(self, key, value)\n\n fp_weight = model_fp.weight.data.contiguous().to('cpu')\n fp_weight_shape = fp_weight.shape\n \n compress_statistics, quant_type, device = self.base_layer.weight.compress_statistics, self.base_layer.weight.quant_type, self.base_layer.weight.device\n del self.base_layer.weight, model_fp\n torch.cuda.empty_cache()\n\n self.base_layer.weight, self.base_layer.tau_quant, self.base_layer.tau_absmax = search(\n fp_weight=fp_weight,\n fp_weight_shape=fp_weight_shape,\n compress_statistics=compress_statistics,\n quant_type=quant_type,\n device=device,\n tau_range=tau_range, tau_n=tau_n,\n blocksize2=blocksize2\n )\n self.base_layer.tau_quant = self.base_layer.tau_quant.to(device)\n self.base_layer.tau_absmax = self.base_layer.tau_absmax.to(device)\n \n del fp_weight\n torch.cuda.empty_cache()\n \n self.lora_default_A_scale = torch.nn.Parameter(torch.zeros([1], dtype=self.lora_A.default.weight.dtype).to(self.base_layer.weight.device), requires_grad=True)\n self.lora_default_B_scale = torch.nn.Parameter(torch.zeros([1], dtype=self.lora_A.default.weight.dtype).to(self.base_layer.weight.device), requires_grad=True)\n \n def forward(self, x: torch.Tensor):\n \n if self.base_layer.bias is not None and self.base_layer.bias.dtype != x.dtype:\n self.base_layer.bias.data = self.base_layer.bias.data.to(x.dtype)\n\n if getattr(self.base_layer.weight, 'quant_state', None) is None:\n print('FP4 quantization state not initialized. Please call .cuda() or .to(device) on the LinearFP4 layer first.')\n inp_dtype = x.dtype\n if self.base_layer.compute_dtype is not None:\n x = x.to(self.base_layer.compute_dtype)\n\n bias = None if self.base_layer.bias is None else self.base_layer.bias.to(self.base_layer.compute_dtype)\n \n with torch.no_grad():\n fp_B = bnb_F.dequantize_fp4(self.base_layer.weight, self.base_layer.weight.quant_state).to(x.dtype)\n tau = (dequantize_tensor(self.base_layer.tau_quant, normal_map_fp8).reshape(-1, 256, 1) * self.base_layer.tau_absmax.reshape(-1, 1, 1)).to(fp_B.device)\n blocksize = torch.prod(torch.tensor(fp_B.shape)) / torch.prod(torch.tensor(tau.shape))\n fp_B = (fp_B.reshape(-1, blocksize.int().item()) + tau.reshape(-1, 1)).reshape(fp_B.shape).to(x.dtype)\n out = torch.nn.functional.linear(x, fp_B, bias)\n\n out = out.to(inp_dtype)\n result = out\n \n if self.disable_adapters or self.active_adapter[0] not in self.lora_A.keys():\n return result\n elif self.r[self.active_adapter[0]] > 0:\n result = result.clone()\n if not torch.is_autocast_enabled():\n expected_dtype = result.dtype\n x = x.to(self.lora_A[self.active_adapter[0]].weight.dtype)\n x = self.lora_A[self.active_adapter[0]](self.lora_dropout[self.active_adapter[0]](x)) + self.lora_default_A_scale * x.reshape([_ for _ in x.shape[:-1]] + [self.lora_A[self.active_adapter[0]].out_features] + [-1]).mean(dim=-1)\n x = (self.lora_B[self.active_adapter[0]](x).reshape([_ for _ in x.shape] + [-1]) + self.lora_default_B_scale * x.unsqueeze(-1)).reshape([_ for _ in x.shape[:-1]] + [-1])\n output = x.to(expected_dtype) * self.scaling[self.active_adapter[0]]\n else:\n x = self.lora_A[self.active_adapter[0]](self.lora_dropout[self.active_adapter[0]](x)) + self.lora_default_A_scale * x.reshape([_ for _ in x.shape[:-1]] + [self.lora_A[self.active_adapter[0]].out_features] + [-1]).mean(dim=-1)\n x = (self.lora_B[self.active_adapter[0]](x).reshape([_ for _ in x.shape] + [-1]) + self.lora_default_B_scale * x.unsqueeze(-1)).reshape([_ for _ in x.shape[:-1]] + [-1])\n output = x * self.scaling[self.active_adapter[0]]\n result += output\n\n return result\n\ndef _replace_with_ours_lora_4bit_linear(\n model, current_key_name=None, model_fp=None, blocksize2=256, tau_range=0.5, tau_n=51\n):\n for name, module in model.named_children():\n if current_key_name is None:\n current_key_name = []\n current_key_name.append(name)\n\n if isinstance(module, peft.tuners.lora.Linear4bit):\n model._modules[name] = IRQLoraLinear4bit(model._modules[name], model_fp=model_fp._modules[name], blocksize2=blocksize2, tau_range=tau_range, tau_n=tau_n)\n \n if len(list(module.children())) > 0:\n if name in model_fp._modules:\n _ = _replace_with_ours_lora_4bit_linear(\n module,\n current_key_name, model_fp._modules[name], blocksize2, tau_range, tau_n\n )\n else:\n _ = _replace_with_ours_lora_4bit_linear(\n module,\n current_key_name, None, blocksize2, tau_range, tau_n\n )\n current_key_name.pop(-1)\n return model\n" }, { "path": "llama.py", "content": "import argparse\nimport time\nimport numpy as np\nimport torch\nimport torch.nn as nn\nimport quant\n\nfrom gptq import GPTQ, Observer\nfrom utils import find_layers, DEV, set_seed, get_wikitext2, get_ptb, get_c4, get_ptb_new, get_c4_new, get_loaders, export_quant_table, gen_conditions\nfrom texttable import Texttable\n\n\ndef get_llama(model):\n\n def skip(*args, **kwargs):\n pass\n\n torch.nn.init.kaiming_uniform_ = skip\n torch.nn.init.uniform_ = skip\n torch.nn.init.normal_ = skip\n from transformers import LlamaForCausalLM\n model = LlamaForCausalLM.from_pretrained(model, torch_dtype=torch.float16)\n model.seqlen = 2048\n return model\n\n\n@torch.no_grad()\ndef llama_sequential(model, dataloader, dev):\n print('Starting ...')\n\n use_cache = model.config.use_cache\n model.config.use_cache = False\n layers = model.model.layers\n\n model.model.embed_tokens = model.model.embed_tokens.to(dev)\n model.model.norm = model.model.norm.to(dev)\n layers[0] = layers[0].to(dev)\n\n dtype = next(iter(model.parameters())).dtype\n inps = torch.zeros((args.nsamples, model.seqlen, model.config.hidden_size), dtype=dtype, device=dev)\n cache = {'i': 0, 'attention_mask': None}\n\n class Catcher(nn.Module):\n\n def __init__(self, module):\n super().__init__()\n self.module = module\n\n def forward(self, inp, **kwargs):\n inps[cache['i']] = inp\n cache['i'] += 1\n cache['attention_mask'] = kwargs['attention_mask']\n cache['position_ids'] = kwargs['position_ids']\n raise ValueError\n\n layers[0] = Catcher(layers[0])\n for batch in dataloader:\n try:\n model(batch[0].to(dev))\n except ValueError:\n pass\n layers[0] = layers[0].module\n\n layers[0] = layers[0].cpu()\n model.model.embed_tokens = model.model.embed_tokens.cpu()\n model.model.norm = model.model.norm.cpu()\n torch.cuda.empty_cache()\n\n outs = torch.zeros_like(inps)\n attention_mask = cache['attention_mask']\n position_ids = cache['position_ids']\n\n print('Ready.')\n\n quantizers = {}\n observer = Observer()\n for i in range(len(layers)):\n\n print(f'Quantizing layer {i+1}/{len(layers)}..')\n print('+------------------+--------------+------------+-----------+-------+')\n print('| name | weight_error | fp_inp_SNR | q_inp_SNR | time |')\n print('+==================+==============+============+===========+=======+')\n\n layer = layers[i].to(dev)\n full = find_layers(layer)\n if args.true_sequential:\n sequential = [['self_attn.k_proj', 'self_attn.v_proj', 'self_attn.q_proj'], ['self_attn.o_proj'], ['mlp.up_proj', 'mlp.gate_proj'], ['mlp.down_proj']]\n else:\n sequential = [list(full.keys())]\n\n for names in sequential:\n subset = {n: full[n] for n in names}\n gptq = {}\n for name in subset:\n gptq[name] = GPTQ(subset[name], observe=args.observe)\n gptq[name].quantizer.configure(args.wbits, perchannel=True, sym=args.sym, mse=False)\n\n def add_batch(name):\n\n def tmp(_, inp, out):\n gptq[name].add_batch(inp[0].data, out.data)\n\n return tmp\n\n handles = []\n for name in subset:\n handles.append(subset[name].register_forward_hook(add_batch(name)))\n for j in range(args.nsamples):\n outs[j] = layer(inps[j].unsqueeze(0), attention_mask=attention_mask, position_ids=position_ids)[0]\n for h in handles:\n h.remove()\n\n for name in subset:\n scale, zero, g_idx, error = gptq[name].fasterquant(blocksize=args.blocksize, percdamp=args.percdamp, groupsize=args.groupsize, actorder=args.act_order, name=name)\n quantizers['model.layers.%d.%s' % (i, name)] = (gptq[name].quantizer.cpu(), scale.cpu(), zero.cpu(), g_idx.cpu(), args.wbits, args.groupsize)\n\n if args.observe:\n observer.submit(name=name, layerid=i, gptq=gptq[name], error=error)\n else:\n gptq[name].free()\n\n for j in range(args.nsamples):\n outs[j] = layer(inps[j].unsqueeze(0), attention_mask=attention_mask, position_ids=position_ids)[0]\n\n layers[i] = layer.cpu()\n del layer\n del gptq\n torch.cuda.empty_cache()\n\n inps, outs = outs, inps\n print('+------------------+--------------+------------+-----------+-------+')\n print('\\n')\n\n if args.observe:\n observer.print()\n conditions = gen_conditions(args.wbits, args.groupsize)\n for item in observer.items():\n name = item[0]\n layerid = item[1]\n gptq = item[2]['gptq']\n error = item[2]['error']\n target = error / 2\n\n table = Texttable()\n table.header(['wbits', 'groupsize', 'error'])\n table.set_cols_dtype(['i', 'i', 'f'])\n table.add_row([args.wbits, args.groupsize, error])\n\n print('Optimizing {} {} ..'.format(name, layerid))\n for wbits, groupsize in conditions:\n\n if error < target:\n # if error dropped 50%, skip\n break\n\n gptq.quantizer.configure(wbits, perchannel=True, sym=args.sym, mse=False)\n\n scale, zero, g_idx, error = gptq.fasterquant(percdamp=args.percdamp, groupsize=groupsize, actorder=args.act_order, name=name)\n\n table.add_row([wbits, groupsize, error])\n quantizers['model.layers.%d.%s' % (layerid, name)] = (gptq.quantizer.cpu(), scale.cpu(), zero.cpu(), g_idx.cpu(), wbits, groupsize)\n\n print(table.draw())\n print('\\n')\n gptq.layer.to('cpu')\n gptq.free()\n\n model.config.use_cache = use_cache\n\n return quantizers\n\n\n@torch.no_grad()\ndef llama_eval(model, testenc, dev):\n print('Evaluating ...')\n\n testenc = testenc.input_ids\n nsamples = testenc.numel() // model.seqlen\n\n use_cache = model.config.use_cache\n model.config.use_cache = False\n layers = model.model.layers\n\n model.model.embed_tokens = model.model.embed_tokens.to(dev)\n layers[0] = layers[0].to(dev)\n\n dtype = next(iter(model.parameters())).dtype\n inps = torch.zeros((nsamples, model.seqlen, model.config.hidden_size), dtype=dtype, device=dev)\n cache = {'i': 0, 'attention_mask': None}\n\n class Catcher(nn.Module):\n\n def __init__(self, module):\n super().__init__()\n self.module = module\n\n def forward(self, inp, **kwargs):\n inps[cache['i']] = inp\n cache['i'] += 1\n cache['attention_mask'] = kwargs['attention_mask']\n cache['position_ids'] = kwargs['position_ids']\n raise ValueError\n\n layers[0] = Catcher(layers[0])\n for i in range(nsamples):\n batch = testenc[:, (i * model.seqlen):((i + 1) * model.seqlen)].to(dev)\n try:\n model(batch)\n except ValueError:\n pass\n layers[0] = layers[0].module\n\n layers[0] = layers[0].cpu()\n model.model.embed_tokens = model.model.embed_tokens.cpu()\n torch.cuda.empty_cache()\n\n outs = torch.zeros_like(inps)\n attention_mask = cache['attention_mask']\n position_ids = cache['position_ids']\n\n for i in range(len(layers)):\n print(i)\n layer = layers[i].to(dev)\n\n if args.nearest:\n subset = find_layers(layer)\n for name in subset:\n quantizer = quant.Quantizer()\n quantizer.configure(args.wbits, perchannel=True, sym=args.sym, mse=False)\n W = subset[name].weight.data\n quantizer.find_params(W, weight=True)\n subset[name].weight.data = quantizer.quantize(W).to(next(iter(layer.parameters())).dtype)\n\n for j in range(nsamples):\n outs[j] = layer(inps[j].unsqueeze(0), attention_mask=attention_mask, position_ids=position_ids)[0]\n layers[i] = layer.cpu()\n del layer\n torch.cuda.empty_cache()\n inps, outs = outs, inps\n\n if model.model.norm is not None:\n model.model.norm = model.model.norm.to(dev)\n model.lm_head = model.lm_head.to(dev)\n\n testenc = testenc.to(dev)\n nlls = []\n for i in range(nsamples):\n hidden_states = inps[i].unsqueeze(0)\n if model.model.norm is not None:\n hidden_states = model.model.norm(hidden_states)\n lm_logits = model.lm_head(hidden_states)\n shift_logits = lm_logits[:, :-1, :].contiguous()\n shift_labels = testenc[:, (i * model.seqlen):((i + 1) * model.seqlen)][:, 1:]\n loss_fct = nn.CrossEntropyLoss()\n loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))\n neg_log_likelihood = loss.float() * model.seqlen\n nlls.append(neg_log_likelihood)\n ppl = torch.exp(torch.stack(nlls).sum() / (nsamples * model.seqlen))\n print(ppl.item())\n\n model.config.use_cache = use_cache\n\n\n# TODO: perform packing on GPU\ndef llama_pack(model, quantizers, wbits, groupsize):\n layers = find_layers(model)\n layers = {n: layers[n] for n in quantizers}\n quant.make_quant_linear(model, quantizers, wbits, groupsize)\n qlayers = find_layers(model, [quant.QuantLinear])\n print('Packing ...')\n for name in qlayers:\n print(name)\n quantizers[name], scale, zero, g_idx, _, _ = quantizers[name]\n qlayers[name].pack(layers[name], scale, zero, g_idx)\n print('Done.')\n return model\n\n\ndef load_quant(model, checkpoint, wbits, groupsize=-1, fused_mlp=True, eval=True, warmup_autotune=True):\n from transformers import LlamaConfig, LlamaForCausalLM, modeling_utils\n config = LlamaConfig.from_pretrained(model)\n\n def noop(*args, **kwargs):\n pass\n\n torch.nn.init.kaiming_uniform_ = noop\n torch.nn.init.uniform_ = noop\n torch.nn.init.normal_ = noop\n\n torch.set_default_dtype(torch.half)\n modeling_utils._init_weights = False\n torch.set_default_dtype(torch.half)\n model = LlamaForCausalLM(config)\n torch.set_default_dtype(torch.float)\n if eval:\n model = model.eval()\n layers = find_layers(model)\n for name in ['lm_head']:\n if name in layers:\n del layers[name]\n quant.make_quant_linear(model, layers, wbits, groupsize)\n\n del layers\n\n print('Loading model ...')\n if checkpoint.endswith('.safetensors'):\n from safetensors.torch import load_file as safe_load\n model.load_state_dict(safe_load(checkpoint))\n else:\n model.load_state_dict(torch.load(checkpoint))\n\n if eval:\n quant.make_quant_attn(model)\n quant.make_quant_norm(model)\n if fused_mlp:\n quant.make_fused_mlp(model)\n\n if warmup_autotune:\n quant.autotune_warmup_linear(model, transpose=not (eval))\n if eval and fused_mlp:\n quant.autotune_warmup_fused(model)\n model.seqlen = 2048\n print('Done.')\n\n return model\n\n\ndef llama_multigpu(model, gpus, gpu_dist):\n model.model.embed_tokens = model.model.embed_tokens.to(gpus[0])\n if hasattr(model.model, 'norm') and model.model.norm:\n model.model.norm = model.model.norm.to(gpus[0])\n import copy\n model.lm_head = copy.deepcopy(model.lm_head).to(gpus[0])\n\n cache = {'mask': None, 'position_ids': None}\n\n class MoveModule(nn.Module):\n\n def __init__(self, module, invalidate_cache):\n super().__init__()\n self.module = module\n self.dev = next(iter(self.module.parameters())).device\n self.invalidate_cache=invalidate_cache\n\n def forward(self, *inp, **kwargs):\n inp = list(inp)\n if inp[0].device != self.dev:\n inp[0] = inp[0].to(self.dev)\n\n if cache['mask'] is None or cache['mask'].device != self.dev or self.invalidate_cache:\n cache['mask'] = kwargs['attention_mask'].to(self.dev)\n kwargs['attention_mask'] = cache['mask']\n\n if cache['position_ids'] is None or cache['position_ids'].device != self.dev or self.invalidate_cache:\n cache['position_ids'] = kwargs['position_ids'].to(self.dev)\n kwargs['position_ids'] = cache['position_ids']\n \n tmp = self.module(*inp, **kwargs)\n return tmp\n\n layers = model.model.layers\n from math import ceil\n if not gpu_dist:\n pergpu = ceil(len(layers) / len(gpus))\n for i in range(len(layers)):\n layers[i] = MoveModule(layers[i].to(0 if i == 0 or i == len(layers) -1 else gpus[(i-1) // pergpu]), i==0)\n else:\n assert gpu_dist[0] >= 2, \"At least two layers must be on GPU 0.\"\n assigned_gpus = [0] * (gpu_dist[0]-1)\n for i in range(1, len(gpu_dist)):\n assigned_gpus = assigned_gpus + [i] * gpu_dist[i]\n\n remaining_assignments = len(layers)-len(assigned_gpus) - 1\n if remaining_assignments > 0:\n assigned_gpus = assigned_gpus + [-1] * remaining_assignments\n\n assigned_gpus = assigned_gpus + [0]\n\n for i in range(len(layers)):\n layers[i] = MoveModule(layers[i].to(gpus[assigned_gpus[i]]), i==0)\n\n model.gpus = gpus\n\n\ndef benchmark(model, input_ids, check=False):\n input_ids = input_ids.to(model.gpus[0] if hasattr(model, 'gpus') else DEV)\n torch.cuda.synchronize()\n\n cache = {'past': None}\n\n def clear_past(i):\n\n def tmp(layer, inp, out):\n if cache['past']:\n cache['past'][i] = None\n\n return tmp\n\n for i, layer in enumerate(model.model.layers):\n layer.register_forward_hook(clear_past(i))\n\n print('Benchmarking ...')\n\n if check:\n loss = nn.CrossEntropyLoss()\n tot = 0.\n\n def sync():\n if hasattr(model, 'gpus'):\n for gpu in model.gpus:\n torch.cuda.synchronize(gpu)\n else:\n torch.cuda.synchronize()\n\n max_memory = 0\n with torch.no_grad():\n attention_mask = torch.ones((1, input_ids.numel()), device=DEV)\n times = []\n for i in range(input_ids.numel()):\n tick = time.time()\n out = model(input_ids[:, i:i + 1], past_key_values=cache['past'], attention_mask=attention_mask[:, :(i + 1)].reshape((1, -1)))\n sync()\n times.append(time.time() - tick)\n print(i, times[-1])\n if hasattr(model, 'gpus'):\n mem_allocated = sum(torch.cuda.memory_allocated(gpu) for gpu in model.gpus) / 1024 / 1024\n else:\n mem_allocated = torch.cuda.memory_allocated() / 1024 / 1024\n max_memory = max(max_memory, mem_allocated)\n if check and i != input_ids.numel() - 1:\n tot += loss(out.logits[0].to(DEV), input_ids[:, (i + 1)].to(DEV)).float()\n cache['past'] = list(out.past_key_values)\n del out\n sync()\n print('Median:', np.median(times))\n if check:\n print('PPL:', torch.exp(tot / (input_ids.numel() - 1)).item())\n print('max memory(MiB):', max_memory)\n\n\nif __name__ == '__main__':\n\n parser = argparse.ArgumentParser()\n\n parser.add_argument('model', type=str, help='llama model to load')\n parser.add_argument('dataset', type=str, choices=['wikitext2', 'ptb', 'c4'], help='Where to extract calibration data from.')\n parser.add_argument('--seed', type=int, default=0, help='Seed for sampling the calibration data.')\n parser.add_argument('--nsamples', type=int, default=128, help='Number of calibration data samples.')\n parser.add_argument('--percdamp', type=float, default=.01, help='Percent of the average Hessian diagonal to use for dampening.')\n parser.add_argument('--nearest', action='store_true', help='Whether to run the RTN baseline.')\n parser.add_argument('--wbits', type=int, default=16, choices=[2, 3, 4, 8, 16], help='#bits to use for quantization; use 16 for evaluating base model.')\n parser.add_argument('--trits', action='store_true', help='Whether to use trits for quantization.')\n parser.add_argument('--blocksize', type=int, default=128, help='blocksize')\n parser.add_argument('--groupsize', type=int, default=-1, help='Groupsize to use for quantization; default uses full row.')\n parser.add_argument('--eval', action='store_true', help='evaluate quantized model.')\n parser.add_argument('--test-generation', action='store_true', help='test generation.')\n parser.add_argument('--save', type=str, default='', help='Save quantized checkpoint under this name.')\n parser.add_argument('--save_safetensors', type=str, default='', help='Save quantized `.safetensors` checkpoint under this name.')\n parser.add_argument('--load', type=str, default='', help='Load quantized model.')\n parser.add_argument('--benchmark', type=int, default=0, help='Number of tokens to use for benchmarking.')\n parser.add_argument('--check', action='store_true', help='Whether to compute perplexity during benchmarking for verification.')\n parser.add_argument('--sym', action='store_true', help='Whether to perform symmetric quantization.')\n parser.add_argument('--act-order', action='store_true', help='Whether to apply the activation order GPTQ heuristic')\n parser.add_argument('--true-sequential', action='store_true', help='Whether to run in true sequential model.')\n parser.add_argument('--new-eval', action='store_true', help='Whether to use the new PTB and C4 eval')\n parser.add_argument('--layers-dist', type=str, default='', help='Distribution of layers across GPUs. e.g. 2:1:1 for 2 layers on GPU 0, 1 layer on GPU 1, and 1 layer on GPU 2. Any remaining layers will be assigned to your last GPU.')\n parser.add_argument('--observe',\n action='store_true',\n help='Auto upgrade layer precision to higher precision, for example int2 to int4, groupsize 128 to 64. \\\n When this feature enabled, `--save` or `--save_safetensors` would be disable.')\n parser.add_argument('--quant-directory', type=str, default=None, help='Specify the directory for export quantization parameters to toml format. `None` means no export by default.')\n\n args = parser.parse_args()\n\n if args.layers_dist:\n gpu_dist = [int(x) for x in args.layers_dist.split(':')]\n else:\n gpu_dist = []\n\n if type(args.load) is not str:\n args.load = args.load.as_posix()\n\n if args.load:\n model = load_quant(args.model, args.load, args.wbits, args.groupsize)\n else:\n model = get_llama(args.model)\n model.eval()\n\n dataloader, testloader = get_loaders(args.dataset, nsamples=args.nsamples, seed=args.seed, model=args.model, seqlen=model.seqlen)\n\n if not args.load and args.wbits < 16 and not args.nearest:\n tick = time.time()\n quantizers = llama_sequential(model, dataloader, DEV)\n print(time.time() - tick)\n\n if args.benchmark:\n gpus = [torch.device('cuda:%d' % i) for i in range(torch.cuda.device_count())]\n if len(gpus) > 1:\n llama_multigpu(model, gpus, gpu_dist)\n else:\n model = model.to(DEV)\n if args.benchmark:\n input_ids = next(iter(dataloader))[0][:, :args.benchmark]\n benchmark(model, input_ids, check=args.check)\n\n if args.eval:\n datasets = ['wikitext2', 'ptb', 'c4']\n if args.new_eval:\n datasets = ['wikitext2', 'ptb-new', 'c4-new']\n for dataset in datasets:\n dataloader, testloader = get_loaders(dataset, seed=args.seed, model=args.model, seqlen=model.seqlen)\n print(dataset)\n llama_eval(model, testloader, DEV)\n from utils.datautils import zeroshot_evaluate\n zeroshot_evaluate(model, args, DEV)\n \n if args.test_generation:\n gpus = [torch.device('cuda:%d' % i) for i in range(torch.cuda.device_count())]\n if len(gpus) > 1:\n llama_multigpu(model, gpus, gpu_dist)\n else:\n model = model.to(DEV)\n\n from transformers import LlamaTokenizer, TextStreamer\n tokenizer = LlamaTokenizer.from_pretrained(args.model, use_fast=False)\n input_ids = tokenizer([\"The capital of New Mexico is\"], return_tensors=\"pt\").input_ids.to(gpus[0])\n streamer = TextStreamer(tokenizer)\n with torch.no_grad():\n generated_ids = model.generate(input_ids, streamer=streamer)\n \n\n\n if args.quant_directory is not None:\n export_quant_table(quantizers, args.quant_directory)\n\n if not args.observe and args.save:\n llama_pack(model, quantizers, args.wbits, args.groupsize)\n torch.save(model.state_dict(), args.save)\n\n if not args.observe and args.save_safetensors:\n llama_pack(model, quantizers, args.wbits, args.groupsize)\n from safetensors.torch import save_file as safe_save\n state_dict = model.state_dict()\n state_dict = {k: v.clone().contiguous() for k, v in state_dict.items()}\n safe_save(state_dict, args.save_safetensors)\n" }, { "path": "lm_eval/__init__.py", "content": "" }, { "path": "lm_eval/base.py", "content": "import abc\nfrom typing import Iterable\nimport numpy as np\nimport random\nimport re\nimport os\nimport json\nimport hashlib\nimport datasets\nfrom sqlitedict import SqliteDict\nfrom tqdm import tqdm\nimport torch\nimport torch.nn.functional as F\n\nfrom lm_eval.metrics import mean, weighted_perplexity, weighted_mean, bits_per_byte\nfrom lm_eval import utils\nfrom abc import abstractmethod\n\n\nclass LM(abc.ABC):\n def __init__(self):\n self.cache_hook = CacheHook(None)\n\n @abstractmethod\n def loglikelihood(self, requests):\n \"\"\"Compute log-likelihood of generating a continuation from a context.\n Downstream tasks should attempt to use loglikelihood instead of other\n LM calls whenever possible.\n\n :param requests: list\n A list of pairs (context, continuation)\n context: str\n Context string. Implementations of LM must be able to handle an\n empty context string.\n continuation: str\n The continuation over which log likelihood will be calculated. If\n there is a word boundary, the space should be in the continuation.\n For example, context=\"hello\" continuation=\" world\" is correct.\n :return: list\n A list of pairs (logprob, isgreedy)\n logprob: float\n The log probability of `continuation`\n isgreedy:\n Whether `continuation` would be generated by greedy sampling from `context`\n \"\"\"\n pass\n\n @abstractmethod\n def loglikelihood_rolling(self, requests):\n \"\"\"Compute full log-likelihood of a string, with no truncation, for perplexity computation\n - We will use the full max context length of the model.\n - For inputs that exceed the max context length, we divide the tokenized string into chunks of up to\n the max context length.\n - IMPORTANT: Each document's loglikelihood/perplexity is computed *separately*, unlike other implementations\n which may simply concatenate multiple documents together.\n - IMPORTANT: We maximize the amount of context for each prediction. Specifically, for inputs that we break into\n multiple chunks, the last input will still a full-sized context.\n Example:\n Input tokens: [ 0 1 2 3 4 5 6 7 8 9 ]\n Prefix: EOT\n Max context length: 4\n Resulting input/prediction pairs:\n\n INPUT: EOT 0 1 2\n PRED: 0 1 2 3\n\n INPUT: 3 4 5 6\n PRED: 4 5 6 7\n\n INPUT: 5 6 7 8\n PRED: 8 9\n\n Observe that:\n 1. Each token is predicted exactly once\n 2. For the last pair, we provide the full context, but only score the last two tokens\n\n :param requests: list\n A list of strings\n string: str\n String for which we are computing per-toke loglikelihood\n :return: list\n A list of pairs (logprob, isgreedy)\n logprob: float\n The log probability of `continuation`\n isgreedy:\n Whether `continuation` would be generated by greedy sampling from `context`\n \"\"\"\n pass\n\n # TODO: Add an optional max length\n @abstractmethod\n def greedy_until(self, requests):\n \"\"\"Generate greedily until a stopping sequence\n\n :param requests: list\n A list of pairs (context, until)\n context: str\n Context string\n until: [str]\n The string sequences to generate until. These string sequences\n may each span across multiple tokens, or may be part of one token.\n :return: list\n A list of strings continuation\n continuation: str\n The generated continuation.\n \"\"\"\n pass\n\n @classmethod\n def create_from_arg_string(cls, arg_string, additional_config=None):\n additional_config = {} if additional_config is None else additional_config\n args = utils.simple_parse_args_string(arg_string)\n args2 = {k: v for k, v in additional_config.items() if v is not None}\n return cls(**args, **args2)\n\n def set_cache_hook(self, cache_hook):\n self.cache_hook = cache_hook\n\n\nclass BaseLM(LM):\n @property\n @abstractmethod\n def eot_token_id(self):\n pass\n\n @property\n @abstractmethod\n def max_length(self):\n pass\n\n @property\n @abstractmethod\n def max_gen_toks(self):\n pass\n\n @property\n @abstractmethod\n def batch_size(self):\n pass\n\n @property\n @abstractmethod\n def device(self):\n pass\n\n @abstractmethod\n def tok_encode(self, string: str):\n pass\n\n @abstractmethod\n def tok_decode(self, tokens: Iterable[int]):\n pass\n\n @abstractmethod\n def _model_generate(self, context, max_length, eos_token_id):\n pass\n\n @abstractmethod\n def _model_call(self, inps):\n \"\"\"\n inps: a torch tensor of shape [batch, sequence]\n the size of sequence may vary from call to call\n\n returns: a torch tensor of shape [batch, sequence, vocab] with the\n logits returned from the model\n \"\"\"\n pass\n\n # subclass must implement properties vocab_size, eot_token_id, max_gen_toks, batch_size, device, max_length.\n # TODO: enforce this somehow\n\n def loglikelihood(self, requests):\n new_reqs = []\n for context, continuation in requests:\n if context == \"\":\n # end of text as context\n context_enc = [self.eot_token_id]\n else:\n context_enc = self.tok_encode(context)\n\n continuation_enc = self.tok_encode(continuation)\n\n new_reqs.append(((context, continuation), context_enc, continuation_enc))\n\n return self._loglikelihood_tokens(new_reqs)\n\n def loglikelihood_rolling(self, requests):\n # TODO: Implement caching once we've confirmed the perplexity implementation\n # TODO: automatic batch size detection for vectorization\n\n loglikelihoods = []\n for (string,) in tqdm(requests):\n rolling_token_windows = list(\n map(\n utils.make_disjoint_window,\n utils.get_rolling_token_windows(\n token_list=self.tok_encode(string),\n prefix_token=self.eot_token_id,\n max_seq_len=self.max_length,\n context_len=1,\n ),\n )\n )\n\n rolling_token_windows = [(None,) + x for x in rolling_token_windows]\n\n # TODO: extract out this call so it only gets called once and also somehow figure out partial caching for\n # that\n string_nll = self._loglikelihood_tokens(\n rolling_token_windows, disable_tqdm=True\n )\n\n # discard is_greedy\n string_nll = [x[0] for x in string_nll]\n\n string_nll = sum(string_nll)\n loglikelihoods.append(string_nll)\n\n return loglikelihoods\n\n def _loglikelihood_tokens(self, requests, disable_tqdm=False):\n # TODO: implement some kind of efficient-request-middleware that lumps together requests with the same context\n res = []\n\n def _collate(x):\n # the negative sign on len(toks) sorts descending - this has a few advantages:\n # - time estimates will always be over not underestimates, which is more useful for planning\n # - to know the size of a batch when going through the list, you know the first one is always the batch\n # padded context length. this is useful to simplify the batching logic and more importantly to make\n # automatic adaptive batches much much easier to implement\n # - any OOMs will happen right away rather than near the end\n\n toks = x[1] + x[2]\n return -len(toks), tuple(toks)\n\n # TODO: automatic (variable) batch size detection for vectorization\n re_ord = utils.Reorderer(requests, _collate)\n for chunk in utils.chunks(\n tqdm(re_ord.get_reordered(), disable=disable_tqdm), self.batch_size\n ):\n inps = []\n cont_toks_list = []\n inplens = []\n\n padding_length = None\n\n # because vectorizing is annoying, we first convert each (context, continuation) pair to padded\n # tensors, then we pack them together into a batch, call the model, and then pick it all apart\n # again because vectorizing is annoying\n\n for _, context_enc, continuation_enc in chunk:\n # sanity check\n assert len(context_enc) > 0\n assert len(continuation_enc) > 0\n assert len(continuation_enc) <= self.max_length\n\n # how this all works:\n # CTX CONT\n # inp 0 1 2 3|4 5 6 7 8 9 <- last token is deleted by inp[:, :-1]\n # gpt2 \\ \\\n # logits 1 2 3|4 5 6 7 8 9 <- the ctx half gets tossed out by the\n # cont_toks 4 5 6 7 8 9 [:, -len(continuation_enc):, :self.vocab_size] slice\n\n # when too long to fit in context, truncate from the left\n inp = torch.tensor(\n (context_enc + continuation_enc)[-(self.max_length + 1) :][:-1],\n dtype=torch.long,\n ).to(self.device)\n (inplen,) = inp.shape\n\n cont = continuation_enc\n\n # since in _collate we make sure length is descending, the longest is always the first one.\n padding_length = (\n padding_length if padding_length is not None else inplen\n )\n\n # pad length from seq to padding_length\n inp = torch.cat(\n [\n inp, # [seq]\n torch.zeros(padding_length - inplen, dtype=torch.long).to(\n inp.device\n ), # [padding_length - seq]\n ],\n dim=0,\n )\n\n inps.append(inp.unsqueeze(0)) # [1, padding_length]\n cont_toks_list.append(cont)\n inplens.append(inplen)\n\n batched_inps = torch.cat(inps, dim=0) # [batch, padding_length\n multi_logits = F.log_softmax(\n self._model_call(batched_inps), dim=-1\n ).cpu() # [batch, padding_length, vocab]\n\n for (cache_key, _, _), logits, inp, inplen, cont_toks in zip(\n chunk, multi_logits, inps, inplens, cont_toks_list\n ):\n\n # Slice to original seq length\n contlen = len(cont_toks)\n logits = logits[inplen - contlen : inplen].unsqueeze(\n 0\n ) # [1, seq, vocab]\n\n # Check if per-token argmax is exactly equal to continuation\n greedy_tokens = logits.argmax(dim=-1)\n cont_toks = torch.tensor(cont_toks, dtype=torch.long).unsqueeze(\n 0\n ) # [1, seq]\n max_equal = (greedy_tokens == cont_toks).all()\n\n # Obtain log-probs at the corresponding continuation token indices\n # last_token_slice = logits[:, -1, :].squeeze(0).tolist()\n logits = torch.gather(logits, 2, cont_toks.unsqueeze(-1)).squeeze(\n -1\n ) # [1, seq]\n\n # Answer: (log prob, is-exact-match)\n answer = (float(logits.sum()), bool(max_equal))\n\n # partial caching\n if cache_key is not None:\n self.cache_hook.add_partial(\"loglikelihood\", cache_key, answer)\n\n res.append(answer)\n\n return re_ord.get_original(res)\n\n def greedy_until(self, requests):\n # TODO: implement fully general `until` that handles until that are\n # multiple tokens or that span multiple tokens correctly\n\n # TODO: extract to TokenizedLM?\n res = []\n\n def _collate(x):\n toks = self.tok_encode(x[0])\n return len(toks), x[0]\n\n re_ord = utils.Reorderer(requests, _collate)\n\n for context, until in tqdm(re_ord.get_reordered()):\n if isinstance(until, str):\n until = [until]\n\n (primary_until,) = self.tok_encode(until[0])\n\n context_enc = torch.tensor(\n [self.tok_encode(context)[self.max_gen_toks - self.max_length :]]\n ).to(self.device)\n\n cont = self._model_generate(\n context_enc, context_enc.shape[1] + self.max_gen_toks, primary_until\n )\n\n s = self.tok_decode(cont[0].tolist()[context_enc.shape[1] :])\n\n for term in until:\n s = s.split(term)[0]\n\n # partial caching\n self.cache_hook.add_partial(\"greedy_until\", (context, until), s)\n\n res.append(s)\n\n return re_ord.get_original(res)\n\n\nclass Task(abc.ABC):\n \"\"\"A task represents an entire benchmark including its dataset, problems,\n answers, and evaluation methods. See BoolQ for a simple example implementation\n\n A `doc` can be any python object which represents one instance of evaluation.\n This is usually a dictionary e.g.\n {\"question\": ..., \"answer\": ...} or\n {\"question\": ..., question, answer)\n \"\"\"\n\n # The name of the `Task` benchmark as denoted in the HuggingFace datasets Hub\n # or a path to a custom `datasets` loading script.\n DATASET_PATH: str = None\n\n # The name of a subset within `DATASET_PATH`.\n DATASET_NAME: str = None\n\n def __init__(self, data_dir=None, cache_dir=None, download_mode=None):\n \"\"\"\n :param data_dir: str\n Stores the path to a local folder containing the `Task`'s data files.\n Use this to specify the path to manually downloaded data (usually when\n the dataset is not publicly accessible).\n :param cache_dir: str\n The directory to read/write the `Task` dataset. This follows the\n HuggingFace `datasets` API with the default cache directory located at:\n `~/.cache/huggingface/datasets`\n NOTE: You can change the cache location globally for a given process\n by setting the shell environment variable, `HF_DATASETS_CACHE`,\n to another directory:\n `export HF_DATASETS_CACHE=\"/path/to/another/directory\"`\n :param download_mode: datasets.DownloadMode\n How to treat pre-existing `Task` downloads and data.\n - `datasets.DownloadMode.REUSE_DATASET_IF_EXISTS`\n Reuse download and reuse dataset.\n - `datasets.DownloadMode.REUSE_CACHE_IF_EXISTS`\n Reuse download with fresh dataset.\n - `datasets.DownloadMode.FORCE_REDOWNLOAD`\n Fresh download and fresh dataset.\n \"\"\"\n self.download(data_dir, cache_dir, download_mode)\n self._training_docs = None\n self._fewshot_docs = None\n\n def download(self, data_dir=None, cache_dir=None, download_mode=None):\n \"\"\"Downloads and returns the task dataset.\n Override this method to download the dataset from a custom API.\n\n :param data_dir: str\n Stores the path to a local folder containing the `Task`'s data files.\n Use this to specify the path to manually downloaded data (usually when\n the dataset is not publicly accessible).\n :param cache_dir: str\n The directory to read/write the `Task` dataset. This follows the\n HuggingFace `datasets` API with the default cache directory located at:\n `~/.cache/huggingface/datasets`\n NOTE: You can change the cache location globally for a given process\n by setting the shell environment variable, `HF_DATASETS_CACHE`,\n to another directory:\n `export HF_DATASETS_CACHE=\"/path/to/another/directory\"`\n :param download_mode: datasets.DownloadMode\n How to treat pre-existing `Task` downloads and data.\n - `datasets.DownloadMode.REUSE_DATASET_IF_EXISTS`\n Reuse download and reuse dataset.\n - `datasets.DownloadMode.REUSE_CACHE_IF_EXISTS`\n Reuse download with fresh dataset.\n - `datasets.DownloadMode.FORCE_REDOWNLOAD`\n Fresh download and fresh dataset.\n \"\"\"\n self.dataset = datasets.load_dataset(\n path=self.DATASET_PATH,\n name=self.DATASET_NAME,\n data_dir=data_dir,\n cache_dir=cache_dir,\n download_mode=download_mode,\n )\n\n def should_decontaminate(self):\n \"\"\"Whether this task supports decontamination against model training set.\"\"\"\n return False\n\n @abstractmethod\n def has_training_docs(self):\n \"\"\"Whether the task has a training set\"\"\"\n pass\n\n @abstractmethod\n def has_validation_docs(self):\n \"\"\"Whether the task has a validation set\"\"\"\n pass\n\n @abstractmethod\n def has_test_docs(self):\n \"\"\"Whether the task has a test set\"\"\"\n pass\n\n def training_docs(self):\n \"\"\"\n :return: Iterable[obj]\n A iterable of any object, that doc_to_text can handle\n \"\"\"\n return []\n\n def validation_docs(self):\n \"\"\"\n :return: Iterable[obj]\n A iterable of any object, that doc_to_text can handle\n \"\"\"\n return []\n\n def test_docs(self):\n \"\"\"\n :return: Iterable[obj]\n A iterable of any object, that doc_to_text can handle\n \"\"\"\n return []\n\n def _process_doc(self, doc):\n \"\"\"\n Override this to process (detokenize, strip, replace, etc.) individual\n documents. This can be used in a map over documents of a data split.\n E.g. `map(self._process_doc, self.dataset[\"validation\"])`\n\n :return: dict\n The processed version of the specified `doc`.\n \"\"\"\n return doc\n\n def fewshot_examples(self, k, rnd):\n if self._training_docs is None:\n self._training_docs = list(self.training_docs())\n\n return rnd.sample(self._training_docs, k)\n\n def doc_to_decontamination_query(self, doc):\n print(\n \"Override doc_to_decontamination_query with document specific decontamination query.\"\n )\n assert False\n\n @abstractmethod\n def doc_to_text(self, doc):\n pass\n\n @abstractmethod\n def doc_to_target(self, doc):\n pass\n\n @abstractmethod\n def construct_requests(self, doc, ctx):\n \"\"\"Uses RequestFactory to construct Requests and returns an iterable of\n Requests which will be sent to the LM.\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param ctx: str\n The context string, generated by fewshot_context. This includes the natural\n language description, as well as the few shot examples, and the question\n part of the document for `doc`.\n \"\"\"\n pass\n\n @abstractmethod\n def process_results(self, doc, results):\n \"\"\"Take a single document and the LM results and evaluates, returning a\n dict where keys are the names of submetrics and values are the values of\n the metric for that one document\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param results:\n The results of the requests created in construct_requests.\n \"\"\"\n pass\n\n @abstractmethod\n def aggregation(self):\n \"\"\"\n :returns: {str: [metric_score] -> float}\n A dictionary where keys are the names of submetrics and values are\n functions that aggregate a list of metric scores\n \"\"\"\n pass\n\n @abstractmethod\n def higher_is_better(self):\n \"\"\"\n :returns: {str: bool}\n A dictionary where keys are the names of submetrics and values are\n whether a higher value of the submetric is better\n \"\"\"\n pass\n\n def fewshot_description(self):\n import warnings\n\n warnings.warn(\n \"`fewshot_description` will be removed in futures versions. Pass \"\n \"any custom descriptions to the `evaluate` function instead.\",\n DeprecationWarning,\n )\n return \"\"\n\n @utils.positional_deprecated\n def fewshot_context(\n self, doc, num_fewshot, provide_description=None, rnd=None, description=None\n ):\n \"\"\"Returns a fewshot context string that is made up of a prepended description\n (if provided), the `num_fewshot` number of examples, and an appended prompt example.\n\n :param doc: str\n The document as returned from training_docs, validation_docs, or test_docs.\n :param num_fewshot: int\n The number of fewshot examples to provide in the returned context string.\n :param provide_description: bool\n Not implemented, and this option is deprecated and will be removed in a future version in favor of a different description providing method\n :param rnd: random.Random\n The pseudo-random number generator used to randomly sample examples.\n WARNING: This is currently a required arg although it's optionalized with a default `None`.\n :param description: str\n The task's description that will be prepended to the fewshot examples.\n :returns: str\n The fewshot context.\n \"\"\"\n assert (\n rnd is not None\n ), \"A `random.Random` generator argument must be provided to `rnd`\"\n assert not provide_description, (\n \"The `provide_description` arg will be removed in future versions. To prepend \"\n \"a custom description to the context, supply the corresponding string via the \"\n \"`description` arg.\"\n )\n if provide_description is not None:\n # nudge people to not specify it at all\n print(\n \"WARNING: provide_description is deprecated and will be removed in a future version in favor of description_dict\"\n )\n\n description = description + \"\\n\\n\" if description else \"\"\n\n if num_fewshot == 0:\n labeled_examples = \"\"\n else:\n # for sets with no training docs, draw from other set *but ensure no overlap with current doc*\n if self.has_training_docs():\n fewshotex = self.fewshot_examples(k=num_fewshot, rnd=rnd)\n else:\n if self._fewshot_docs is None:\n self._fewshot_docs = list(\n self.validation_docs()\n if self.has_validation_docs()\n else self.test_docs()\n )\n\n fewshotex = rnd.sample(self._fewshot_docs, num_fewshot + 1)\n\n # get rid of the doc that's the one we're evaluating, if it's in the fewshot\n fewshotex = [x for x in fewshotex if x != doc][:num_fewshot]\n\n labeled_examples = (\n \"\\n\\n\".join(\n [\n self.doc_to_text(doc) + self.doc_to_target(doc)\n for doc in fewshotex\n ]\n )\n + \"\\n\\n\"\n )\n\n example = self.doc_to_text(doc)\n return description + labeled_examples + example\n\n\nclass MultipleChoiceTask(Task):\n def doc_to_target(self, doc):\n return \" \" + doc[\"choices\"][doc[\"gold\"]]\n\n def construct_requests(self, doc, ctx):\n lls = [\n rf.loglikelihood(ctx, \" {}\".format(choice))[0] for choice in doc[\"choices\"]\n ]\n\n return lls\n\n def process_results(self, doc, results):\n gold = doc[\"gold\"]\n\n acc = 1.0 if np.argmax(results) == gold else 0.0\n completion_len = np.array([float(len(i)) for i in doc[\"choices\"]])\n acc_norm = 1.0 if np.argmax(results / completion_len) == gold else 0.0\n\n return {\n \"acc\": acc,\n \"acc_norm\": acc_norm,\n }\n\n def higher_is_better(self):\n return {\n \"acc\": True,\n \"acc_norm\": True,\n }\n\n def aggregation(self):\n return {\n \"acc\": mean,\n \"acc_norm\": mean,\n }\n\n\nclass PerplexityTask(Task, abc.ABC):\n def should_decontaminate(self):\n \"\"\"Whether this task supports decontamination against model training set.\"\"\"\n return True\n\n def has_training_docs(self):\n return False\n\n def fewshot_examples(self, k, rnd):\n assert k == 0\n return []\n\n def fewshot_context(\n self, doc, num_fewshot, provide_description=None, rnd=None, description=None\n ):\n assert (\n num_fewshot == 0\n ), \"The number of fewshot examples must be 0 for perplexity tasks.\"\n assert (\n rnd is not None\n ), \"A `random.Random` generator argument must be provided to `rnd`.\"\n assert not provide_description, (\n \"The `provide_description` arg will be removed in future versions. To prepend \"\n \"a custom description to the context, supply the corresponding string via the \"\n \"`description` arg.\"\n )\n if provide_description is not None:\n # nudge people to not specify it at all\n print(\n \"WARNING: provide_description is deprecated and will be removed in a future version in favor of description_dict\"\n )\n\n return \"\"\n\n def higher_is_better(self):\n return {\n \"word_perplexity\": False,\n \"byte_perplexity\": False,\n \"bits_per_byte\": False,\n }\n\n def doc_to_decontamination_query(self, doc):\n return doc\n\n def doc_to_text(self, doc):\n return \"\"\n\n def doc_to_target(self, doc):\n return doc\n\n def construct_requests(self, doc, ctx):\n assert not ctx\n req = rf.loglikelihood_rolling(self.doc_to_target(doc))\n return req\n\n def process_results(self, doc, results):\n (loglikelihood,) = results\n words = self.count_words(doc)\n bytes_ = self.count_bytes(doc)\n return {\n \"word_perplexity\": (loglikelihood, words),\n \"byte_perplexity\": (loglikelihood, bytes_),\n \"bits_per_byte\": (loglikelihood, bytes_),\n }\n\n def aggregation(self):\n return {\n \"word_perplexity\": weighted_perplexity,\n \"byte_perplexity\": weighted_perplexity,\n \"bits_per_byte\": bits_per_byte,\n }\n\n @classmethod\n def count_bytes(cls, doc):\n return len(doc.encode(\"utf-8\"))\n\n @classmethod\n def count_words(cls, doc):\n \"\"\"Downstream tasks with custom word boundaries should override this!\"\"\"\n return len(re.split(r\"\\s+\", doc))\n\n\ndef hash_args(attr, args):\n dat = json.dumps([attr] + list(args))\n return hashlib.sha256(dat.encode(\"utf-8\")).hexdigest()\n\n\nclass CacheHook:\n def __init__(self, cachinglm):\n if cachinglm is None:\n self.dbdict = None\n return\n\n self.dbdict = cachinglm.dbdict\n\n def add_partial(self, attr, req, res):\n if self.dbdict is None:\n return\n hsh = hash_args(attr, req)\n self.dbdict[hsh] = res\n\n\nclass CachingLM:\n def __init__(self, lm, cache_db):\n \"\"\"LM wrapper that returns cached results if they exist, and uses the underlying LM if not.\n\n :param lm: LM\n Underlying LM\n :param cache_db: str\n Path to cache db\n \"\"\"\n self.lm = lm\n self.cache_db = cache_db\n if os.path.dirname(cache_db):\n os.makedirs(os.path.dirname(cache_db), exist_ok=True)\n self.dbdict = SqliteDict(cache_db, autocommit=True)\n\n # add hook to lm\n lm.set_cache_hook(self.get_cache_hook())\n\n def __getattr__(self, attr):\n def fn(requests):\n res = []\n remaining_reqs = []\n\n # figure out which ones are cached and which ones are new\n for req in requests:\n hsh = hash_args(attr, req)\n if hsh in self.dbdict:\n ob = self.dbdict[hsh]\n\n assert ob is not None\n\n res.append(ob)\n else:\n res.append(None)\n remaining_reqs.append(req)\n\n # actually run the LM on the requests that do not have cached results\n rem_res = getattr(self.lm, attr)(remaining_reqs)\n\n # stick the new ones back into the list and also cache any of the new ones\n resptr = 0\n for req, r in zip(remaining_reqs, rem_res):\n while res[resptr] is not None:\n resptr += 1\n\n res[resptr] = r\n\n # caching\n hsh = hash_args(attr, req)\n self.dbdict[hsh] = r\n self.dbdict.commit()\n\n return res\n\n return fn\n\n def get_cache_hook(self):\n return CacheHook(self)\n\n\nREQUEST_RETURN_LENGTHS = {\n \"loglikelihood\": 2,\n \"greedy_until\": None,\n \"loglikelihood_rolling\": None,\n}\n\n\nclass Request:\n def __init__(self, request_type, args, index=None):\n if request_type not in REQUEST_RETURN_LENGTHS.keys():\n raise NotImplementedError(\n \"The request type {} is not implemented!\".format(request_type)\n )\n\n self.request_type = request_type\n self.args = args\n self.index = index\n\n def __iter__(self):\n if REQUEST_RETURN_LENGTHS[self.request_type] is None:\n raise IndexError(\"This request type does not return multiple arguments!\")\n for i in range(REQUEST_RETURN_LENGTHS[self.request_type]):\n yield Request(self.request_type, self.args, i)\n\n def __getitem__(self, i):\n if REQUEST_RETURN_LENGTHS[self.request_type] is None:\n raise IndexError(\"This request type does not return multiple arguments!\")\n return Request(self.request_type, self.args, i)\n\n def __eq__(self, other):\n return (\n self.request_type == other.request_type\n and self.args == other.args\n and self.index == other.index\n )\n\n def __repr__(self):\n return f\"Req_{self.request_type}{self.args}[{self.index}]\\n\"\n\n\nclass RequestFactory:\n def __getattr__(self, attr):\n def fn(*args):\n return Request(attr, args)\n\n return fn\n\n\nrf = RequestFactory()\n" }, { "path": "lm_eval/datasets/README.md", "content": "# datasets\n\nThis directory contains custom HuggingFace [dataset loading scripts](https://huggingface.co/docs/datasets/dataset_script). They are provided to maintain backward compatibility with the ad-hoc data downloaders in earlier versions of the `lm-evaluation-harness` before HuggingFace [`datasets`](https://huggingface.co/docs/datasets/index) was adopted as the default downloading manager. For example, some instances in the HuggingFace `datasets` repository process features (e.g. whitespace stripping, lower-casing, etc.) in ways that the `lm-evaluation-harness` did not.\n\n__NOTE__: We are __not__ accepting any additional loading scripts into the main branch! If you'd like to use a custom dataset, fork the repo and follow HuggingFace's loading script guide found [here](https://huggingface.co/docs/datasets/dataset_script). You can then override your `Task`'s `DATASET_PATH` attribute to point to this script's local path.\n\n\n__WARNING__: A handful of loading scripts are included in this collection because they have not yet been pushed to the Huggingface Hub or a HuggingFace organization repo. We will remove such scripts once pushed.\n" }, { "path": "lm_eval/datasets/__init__.py", "content": "" }, { "path": "lm_eval/datasets/asdiv/__init__.py", "content": "" }, { "path": "lm_eval/datasets/asdiv/asdiv.py", "content": "# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"ASDIV dataset.\"\"\"\n\n\nimport os\nimport xml.etree.ElementTree as ET\n\nimport datasets\n\n\n_CITATION = \"\"\"\\\n@misc{miao2021diverse,\n title={A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers},\n author={Shen-Yun Miao and Chao-Chun Liang and Keh-Yih Su},\n year={2021},\n eprint={2106.15772},\n archivePrefix={arXiv},\n primaryClass={cs.AI}\n}\n\"\"\"\n\n_DESCRIPTION = \"\"\"\\\nASDiv (Academia Sinica Diverse MWP Dataset) is a diverse (in terms of both language\npatterns and problem types) English math word problem (MWP) corpus for evaluating\nthe capability of various MWP solvers. Existing MWP corpora for studying AI progress\nremain limited either in language usage patterns or in problem types. We thus present\na new English MWP corpus with 2,305 MWPs that cover more text patterns and most problem\ntypes taught in elementary school. Each MWP is annotated with its problem type and grade\nlevel (for indicating the level of difficulty).\n\"\"\"\n\n_HOMEPAGE = \"https://github.com/chaochun/nlu-asdiv-dataset\"\n\n# TODO: Add the licence for the dataset here if you can find it\n_LICENSE = \"\"\n\n_URLS = \"https://github.com/chaochun/nlu-asdiv-dataset/archive/55790e5270bb91ccfa5053194b25732534696b50.zip\"\n\n\nclass ASDiv(datasets.GeneratorBasedBuilder):\n \"\"\"ASDiv: A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers\"\"\"\n\n VERSION = datasets.Version(\"0.0.1\")\n\n BUILDER_CONFIGS = [\n datasets.BuilderConfig(\n name=\"asdiv\",\n version=VERSION,\n description=\"A diverse corpus for evaluating and developing english math word problem solvers\",\n )\n ]\n\n def _info(self):\n features = datasets.Features(\n {\n \"body\": datasets.Value(\"string\"),\n \"question\": datasets.Value(\"string\"),\n \"solution_type\": datasets.Value(\"string\"),\n \"answer\": datasets.Value(\"string\"),\n \"formula\": datasets.Value(\"string\"),\n }\n )\n return datasets.DatasetInfo(\n description=_DESCRIPTION,\n features=features,\n homepage=_HOMEPAGE,\n license=_LICENSE,\n citation=_CITATION,\n )\n\n def _split_generators(self, dl_manager):\n urls = _URLS\n data_dir = dl_manager.download_and_extract(urls)\n base_filepath = \"nlu-asdiv-dataset-55790e5270bb91ccfa5053194b25732534696b50\"\n return [\n datasets.SplitGenerator(\n name=datasets.Split.VALIDATION,\n # These kwargs will be passed to _generate_examples\n gen_kwargs={\n \"filepath\": os.path.join(\n data_dir, base_filepath, \"dataset\", \"ASDiv.xml\"\n ),\n \"split\": datasets.Split.VALIDATION,\n },\n ),\n ]\n\n # method parameters are unpacked from `gen_kwargs` as given in `_split_generators`\n def _generate_examples(self, filepath, split):\n tree = ET.parse(filepath)\n root = tree.getroot()\n for key, problem in enumerate(root.iter(\"Problem\")):\n yield key, {\n \"body\": problem.find(\"Body\").text,\n \"question\": problem.find(\"Question\").text,\n \"solution_type\": problem.find(\"Solution-Type\").text,\n \"answer\": problem.find(\"Answer\").text,\n \"formula\": problem.find(\"Formula\").text,\n }\n" }, { "path": "lm_eval/datasets/asdiv/dataset_infos.json", "content": "{\"asdiv\": {\"description\": \"ASDiv (Academia Sinica Diverse MWP Dataset) is a diverse (in terms of both language\\npatterns and problem types) English math word problem (MWP) corpus for evaluating\\nthe capability of various MWP solvers. Existing MWP corpora for studying AI progress\\nremain limited either in language usage patterns or in problem types. We thus present\\na new English MWP corpus with 2,305 MWPs that cover more text patterns and most problem\\ntypes taught in elementary school. Each MWP is annotated with its problem type and grade\\nlevel (for indicating the level of difficulty).\\n\", \"citation\": \"@misc{miao2021diverse,\\n title={A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers},\\n author={Shen-Yun Miao and Chao-Chun Liang and Keh-Yih Su},\\n year={2021},\\n eprint={2106.15772},\\n archivePrefix={arXiv},\\n primaryClass={cs.AI}\\n}\\n\", \"homepage\": \"https://github.com/chaochun/nlu-asdiv-dataset\", \"license\": \"\", \"features\": {\"body\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"question\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"solution_type\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"answer\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"formula\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"as_div\", \"config_name\": \"asdiv\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"validation\": {\"name\": \"validation\", \"num_bytes\": 501489, \"num_examples\": 2305, \"dataset_name\": \"as_div\"}}, \"download_checksums\": {\"https://github.com/chaochun/nlu-asdiv-dataset/archive/55790e5270bb91ccfa5053194b25732534696b50.zip\": {\"num_bytes\": 440966, \"checksum\": \"8f1fe4f6d5f170ec1e24ab78c244153c14c568b1bb2b1dad0324e71f37939a2d\"}}, \"download_size\": 440966, \"post_processing_size\": null, \"dataset_size\": 501489, \"size_in_bytes\": 942455}}\n" }, { "path": "lm_eval/datasets/coqa/__init__.py", "content": "" }, { "path": "lm_eval/datasets/coqa/coqa.py", "content": "# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"CoQA dataset.\n\nThis `CoQA` adds the \"additional_answers\" feature that's missing in the original\ndatasets version:\nhttps://github.com/huggingface/datasets/blob/master/datasets/coqa/coqa.py\n\"\"\"\n\n\nimport json\n\nimport datasets\n\n\n_CITATION = \"\"\"\\\n@misc{reddy2018coqa,\n title={CoQA: A Conversational Question Answering Challenge},\n author={Siva Reddy and Danqi Chen and Christopher D. Manning},\n year={2018},\n eprint={1808.07042},\n archivePrefix={arXiv},\n primaryClass={cs.CL}\n}\n\"\"\"\n\n_DESCRIPTION = \"\"\"\\\nCoQA is a large-scale dataset for building Conversational Question Answering\nsystems. The goal of the CoQA challenge is to measure the ability of machines to\nunderstand a text passage and answer a series of interconnected questions that\nappear in a conversation.\n\"\"\"\n\n_HOMEPAGE = \"https://stanfordnlp.github.io/coqa/\"\n\n# TODO: Add the licence for the dataset here if you can find it\n_LICENSE = \"\"\n\n_URLS = {\n \"train\": \"https://nlp.stanford.edu/data/coqa/coqa-train-v1.0.json\",\n \"validation\": \"https://nlp.stanford.edu/data/coqa/coqa-dev-v1.0.json\",\n}\n\n# `additional_answers` are not available in the train set so we fill them with\n# empty dicts of the same form.\n_EMPTY_ADDITIONAL_ANSWER = {\n \"0\": [\n {\n \"span_start\": -1,\n \"span_end\": -1,\n \"span_text\": \"\",\n \"input_text\": \"\",\n \"turn_id\": -1,\n }\n ],\n \"1\": [\n {\n \"span_start\": -1,\n \"span_end\": -1,\n \"span_text\": \"\",\n \"input_text\": \"\",\n \"turn_id\": -1,\n }\n ],\n \"2\": [\n {\n \"span_start\": -1,\n \"span_end\": -1,\n \"span_text\": \"\",\n \"input_text\": \"\",\n \"turn_id\": -1,\n }\n ],\n}\n\n\nclass Coqa(datasets.GeneratorBasedBuilder):\n \"\"\"CoQA is a large-scale dataset for building Conversational Question Answering systems.\"\"\"\n\n VERSION = datasets.Version(\"0.0.1\")\n\n BUILDER_CONFIGS = [\n datasets.BuilderConfig(\n name=\"coqa\", version=VERSION, description=\"The CoQA dataset.\"\n ),\n ]\n\n def _info(self):\n features = datasets.Features(\n {\n \"id\": datasets.Value(\"string\"),\n \"source\": datasets.Value(\"string\"),\n \"story\": datasets.Value(\"string\"),\n \"questions\": datasets.features.Sequence(\n {\n \"input_text\": datasets.Value(\"string\"),\n \"turn_id\": datasets.Value(\"int32\"),\n }\n ),\n \"answers\": datasets.features.Sequence(\n {\n \"span_start\": datasets.Value(\"int32\"),\n \"span_end\": datasets.Value(\"int32\"),\n \"span_text\": datasets.Value(\"string\"),\n \"input_text\": datasets.Value(\"string\"),\n \"turn_id\": datasets.Value(\"int32\"),\n }\n ),\n \"additional_answers\": {\n \"0\": datasets.features.Sequence(\n {\n \"span_start\": datasets.Value(\"int32\"),\n \"span_end\": datasets.Value(\"int32\"),\n \"span_text\": datasets.Value(\"string\"),\n \"input_text\": datasets.Value(\"string\"),\n \"turn_id\": datasets.Value(\"int32\"),\n }\n ),\n \"1\": datasets.features.Sequence(\n {\n \"span_start\": datasets.Value(\"int32\"),\n \"span_end\": datasets.Value(\"int32\"),\n \"span_text\": datasets.Value(\"string\"),\n \"input_text\": datasets.Value(\"string\"),\n \"turn_id\": datasets.Value(\"int32\"),\n }\n ),\n \"2\": datasets.features.Sequence(\n {\n \"span_start\": datasets.Value(\"int32\"),\n \"span_end\": datasets.Value(\"int32\"),\n \"span_text\": datasets.Value(\"string\"),\n \"input_text\": datasets.Value(\"string\"),\n \"turn_id\": datasets.Value(\"int32\"),\n }\n ),\n },\n }\n )\n return datasets.DatasetInfo(\n description=_DESCRIPTION,\n features=features,\n homepage=_HOMEPAGE,\n license=_LICENSE,\n citation=_CITATION,\n )\n\n def _split_generators(self, dl_manager):\n urls = {\"train\": _URLS[\"train\"], \"validation\": _URLS[\"validation\"]}\n data_dirs = dl_manager.download_and_extract(urls)\n return [\n datasets.SplitGenerator(\n name=datasets.Split.TRAIN,\n # These kwargs will be passed to _generate_examples\n gen_kwargs={\n \"filepath\": data_dirs[\"train\"],\n \"split\": datasets.Split.TRAIN,\n },\n ),\n datasets.SplitGenerator(\n name=datasets.Split.VALIDATION,\n # These kwargs will be passed to _generate_examples\n gen_kwargs={\n \"filepath\": data_dirs[\"validation\"],\n \"split\": datasets.Split.VALIDATION,\n },\n ),\n ]\n\n # method parameters are unpacked from `gen_kwargs` as given in `_split_generators`\n def _generate_examples(self, filepath, split):\n with open(filepath, encoding=\"utf-8\") as f:\n data = json.load(f)\n for row in data[\"data\"]:\n id = row[\"id\"]\n source = row[\"source\"]\n story = row[\"story\"]\n questions = [\n {\"input_text\": q[\"input_text\"], \"turn_id\": q[\"turn_id\"]}\n for q in row[\"questions\"]\n ]\n answers = [\n {\n \"span_start\": a[\"span_start\"],\n \"span_end\": a[\"span_end\"],\n \"span_text\": a[\"span_text\"],\n \"input_text\": a[\"input_text\"],\n \"turn_id\": a[\"turn_id\"],\n }\n for a in row[\"answers\"]\n ]\n if split == datasets.Split.TRAIN:\n additional_answers = _EMPTY_ADDITIONAL_ANSWER\n else:\n additional_answers = {\n \"0\": [\n {\n \"span_start\": a0[\"span_start\"],\n \"span_end\": a0[\"span_end\"],\n \"span_text\": a0[\"span_text\"],\n \"input_text\": a0[\"input_text\"],\n \"turn_id\": a0[\"turn_id\"],\n }\n for a0 in row[\"additional_answers\"][\"0\"]\n ],\n \"1\": [\n {\n \"span_start\": a1[\"span_start\"],\n \"span_end\": a1[\"span_end\"],\n \"span_text\": a1[\"span_text\"],\n \"input_text\": a1[\"input_text\"],\n \"turn_id\": a1[\"turn_id\"],\n }\n for a1 in row[\"additional_answers\"][\"1\"]\n ],\n \"2\": [\n {\n \"span_start\": a2[\"span_start\"],\n \"span_end\": a2[\"span_end\"],\n \"span_text\": a2[\"span_text\"],\n \"input_text\": a2[\"input_text\"],\n \"turn_id\": a2[\"turn_id\"],\n }\n for a2 in row[\"additional_answers\"][\"2\"]\n ],\n }\n yield row[\"id\"], {\n \"id\": id,\n \"story\": story,\n \"source\": source,\n \"questions\": questions,\n \"answers\": answers,\n \"additional_answers\": additional_answers,\n }\n" }, { "path": "lm_eval/datasets/coqa/dataset_infos.json", "content": "{\"coqa\": {\"description\": \"CoQA is a large-scale dataset for building Conversational Question Answering\\nsystems. The goal of the CoQA challenge is to measure the ability of machines to\\nunderstand a text passage and answer a series of interconnected questions that\\nappear in a conversation.\\n\", \"citation\": \"@misc{reddy2018coqa,\\n title={CoQA: A Conversational Question Answering Challenge},\\n author={Siva Reddy and Danqi Chen and Christopher D. Manning},\\n year={2018},\\n eprint={1808.07042},\\n archivePrefix={arXiv},\\n primaryClass={cs.CL}\\n}\\n\", \"homepage\": \"https://stanfordnlp.github.io/coqa/\", \"license\": \"\", \"features\": {\"id\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"source\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"story\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"questions\": {\"feature\": {\"input_text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"turn_id\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}}, \"length\": -1, \"id\": null, \"_type\": \"Sequence\"}, \"answers\": {\"feature\": {\"span_start\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}, \"span_end\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}, \"span_text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"input_text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"turn_id\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}}, \"length\": -1, \"id\": null, \"_type\": \"Sequence\"}, \"additional_answers\": {\"0\": {\"feature\": {\"span_start\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}, \"span_end\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}, \"span_text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"input_text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"turn_id\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}}, \"length\": -1, \"id\": null, \"_type\": \"Sequence\"}, \"1\": {\"feature\": {\"span_start\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}, \"span_end\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}, \"span_text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"input_text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"turn_id\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}}, \"length\": -1, \"id\": null, \"_type\": \"Sequence\"}, \"2\": {\"feature\": {\"span_start\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}, \"span_end\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}, \"span_text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"input_text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"turn_id\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}}, \"length\": -1, \"id\": null, \"_type\": \"Sequence\"}}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"coqa\", \"config_name\": \"coqa\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"train\": {\"name\": \"train\", \"num_bytes\": 26250528, \"num_examples\": 7199, \"dataset_name\": \"coqa\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 3765933, \"num_examples\": 500, \"dataset_name\": \"coqa\"}}, \"download_checksums\": {\"https://nlp.stanford.edu/data/coqa/coqa-train-v1.0.json\": {\"num_bytes\": 49001836, \"checksum\": \"b0fdb2bc1bd38dd3ca2ce5fa2ac3e02c6288ac914f241ac409a655ffb6619fa6\"}, \"https://nlp.stanford.edu/data/coqa/coqa-dev-v1.0.json\": {\"num_bytes\": 9090845, \"checksum\": \"dfa367a9733ce53222918d0231d9b3bedc2b8ee831a2845f62dfc70701f2540a\"}}, \"download_size\": 58092681, \"post_processing_size\": null, \"dataset_size\": 30016461, \"size_in_bytes\": 88109142}}\n" }, { "path": "lm_eval/datasets/drop/__init__.py", "content": "" }, { "path": "lm_eval/datasets/drop/dataset_infos.json", "content": "{\"drop\": {\"description\": \"DROP is a QA dataset which tests comprehensive understanding of paragraphs. In \\nthis crowdsourced, adversarially-created, 96k question-answering benchmark, a \\nsystem must resolve multiple references in a question, map them onto a paragraph,\\nand perform discrete operations over them (such as addition, counting, or sorting).\\n\", \"citation\": \"@misc{dua2019drop,\\n title={DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs}, \\n author={Dheeru Dua and Yizhong Wang and Pradeep Dasigi and Gabriel Stanovsky and Sameer Singh and Matt Gardner},\\n year={2019},\\n eprint={1903.00161},\\n archivePrefix={arXiv},\\n primaryClass={cs.CL}\\n}\\n\", \"homepage\": \"https://allenai.org/data/drop\", \"license\": \"\", \"features\": {\"section_id\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"passage\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"question\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"query_id\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"answer\": {\"number\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"date\": {\"day\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"month\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"year\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"spans\": {\"feature\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"length\": -1, \"id\": null, \"_type\": \"Sequence\"}, \"worker_id\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"hit_id\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"validated_answers\": {\"feature\": {\"number\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"date\": {\"day\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"month\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"year\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"spans\": {\"feature\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"length\": -1, \"id\": null, \"_type\": \"Sequence\"}, \"worker_id\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"hit_id\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"length\": -1, \"id\": null, \"_type\": \"Sequence\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"drop\", \"config_name\": \"drop\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"train\": {\"name\": \"train\", \"num_bytes\": 108858121, \"num_examples\": 77409, \"dataset_name\": \"drop\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 12560739, \"num_examples\": 9536, \"dataset_name\": \"drop\"}}, \"download_checksums\": {\"https://s3-us-west-2.amazonaws.com/allennlp/datasets/drop/drop_dataset.zip\": {\"num_bytes\": 8308692, \"checksum\": \"39d2278a29fd729de301b111a45f434c24834f40df8f4ff116d864589e3249d6\"}}, \"download_size\": 8308692, \"post_processing_size\": null, \"dataset_size\": 121418860, \"size_in_bytes\": 129727552}}\n" }, { "path": "lm_eval/datasets/drop/drop.py", "content": "# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n# Custom DROP dataset that, unlike HF, keeps all question-answer pairs\n# even if there are multiple types of answers for the same question.\n\"\"\"DROP dataset.\"\"\"\n\n\nimport json\nimport os\n\nimport datasets\n\n\n_CITATION = \"\"\"\\\n@misc{dua2019drop,\n title={DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs},\n author={Dheeru Dua and Yizhong Wang and Pradeep Dasigi and Gabriel Stanovsky and Sameer Singh and Matt Gardner},\n year={2019},\n eprint={1903.00161},\n archivePrefix={arXiv},\n primaryClass={cs.CL}\n}\n\"\"\"\n\n_DESCRIPTION = \"\"\"\\\nDROP is a QA dataset which tests comprehensive understanding of paragraphs. In\nthis crowdsourced, adversarially-created, 96k question-answering benchmark, a\nsystem must resolve multiple references in a question, map them onto a paragraph,\nand perform discrete operations over them (such as addition, counting, or sorting).\n\"\"\"\n\n_HOMEPAGE = \"https://allenai.org/data/drop\"\n\n# TODO: Add the licence for the dataset here if you can find it\n_LICENSE = \"\"\n\n_URLS = {\n \"drop\": \"https://s3-us-west-2.amazonaws.com/allennlp/datasets/drop/drop_dataset.zip\",\n}\n\n_EMPTY_VALIDATED_ANSWER = [\n {\n \"number\": \"\",\n \"date\": {\n \"day\": \"\",\n \"month\": \"\",\n \"year\": \"\",\n },\n \"spans\": [],\n \"worker_id\": \"\",\n \"hit_id\": \"\",\n }\n]\n\n\nclass Drop(datasets.GeneratorBasedBuilder):\n \"\"\"DROP is a QA dataset which tests comprehensive understanding of paragraphs.\"\"\"\n\n VERSION = datasets.Version(\"0.0.1\")\n\n BUILDER_CONFIGS = [\n datasets.BuilderConfig(\n name=\"drop\", version=VERSION, description=\"The DROP dataset.\"\n ),\n ]\n\n def _info(self):\n features = datasets.Features(\n {\n \"section_id\": datasets.Value(\"string\"),\n \"passage\": datasets.Value(\"string\"),\n \"question\": datasets.Value(\"string\"),\n \"query_id\": datasets.Value(\"string\"),\n \"answer\": {\n \"number\": datasets.Value(\"string\"),\n \"date\": {\n \"day\": datasets.Value(\"string\"),\n \"month\": datasets.Value(\"string\"),\n \"year\": datasets.Value(\"string\"),\n },\n \"spans\": datasets.features.Sequence(datasets.Value(\"string\")),\n \"worker_id\": datasets.Value(\"string\"),\n \"hit_id\": datasets.Value(\"string\"),\n },\n \"validated_answers\": datasets.features.Sequence(\n {\n \"number\": datasets.Value(\"string\"),\n \"date\": {\n \"day\": datasets.Value(\"string\"),\n \"month\": datasets.Value(\"string\"),\n \"year\": datasets.Value(\"string\"),\n },\n \"spans\": datasets.features.Sequence(datasets.Value(\"string\")),\n \"worker_id\": datasets.Value(\"string\"),\n \"hit_id\": datasets.Value(\"string\"),\n }\n ),\n }\n )\n return datasets.DatasetInfo(\n description=_DESCRIPTION,\n features=features,\n homepage=_HOMEPAGE,\n license=_LICENSE,\n citation=_CITATION,\n )\n\n def _split_generators(self, dl_manager):\n urls = _URLS[self.config.name]\n data_dir = dl_manager.download_and_extract(urls)\n return [\n datasets.SplitGenerator(\n name=datasets.Split.TRAIN,\n # These kwargs will be passed to _generate_examples\n gen_kwargs={\n \"filepath\": os.path.join(\n data_dir, \"drop_dataset\", \"drop_dataset_train.json\"\n ),\n \"split\": \"train\",\n },\n ),\n datasets.SplitGenerator(\n name=datasets.Split.VALIDATION,\n # These kwargs will be passed to _generate_examples\n gen_kwargs={\n \"filepath\": os.path.join(\n data_dir, \"drop_dataset\", \"drop_dataset_dev.json\"\n ),\n \"split\": \"validation\",\n },\n ),\n ]\n\n # method parameters are unpacked from `gen_kwargs` as given in `_split_generators`\n def _generate_examples(self, filepath, split):\n with open(filepath, encoding=\"utf-8\") as f:\n data = json.load(f)\n key = 0\n for section_id, example in data.items():\n # Each example (passage) has multiple sub-question-answer pairs.\n for qa in example[\"qa_pairs\"]:\n # Build answer.\n answer = qa[\"answer\"]\n answer = {\n \"number\": answer[\"number\"],\n \"date\": {\n \"day\": answer[\"date\"].get(\"day\", \"\"),\n \"month\": answer[\"date\"].get(\"month\", \"\"),\n \"year\": answer[\"date\"].get(\"year\", \"\"),\n },\n \"spans\": answer[\"spans\"],\n \"worker_id\": answer.get(\"worker_id\", \"\"),\n \"hit_id\": answer.get(\"hit_id\", \"\"),\n }\n validated_answers = []\n if \"validated_answers\" in qa:\n for validated_answer in qa[\"validated_answers\"]:\n va = {\n \"number\": validated_answer.get(\"number\", \"\"),\n \"date\": {\n \"day\": validated_answer[\"date\"].get(\"day\", \"\"),\n \"month\": validated_answer[\"date\"].get(\"month\", \"\"),\n \"year\": validated_answer[\"date\"].get(\"year\", \"\"),\n },\n \"spans\": validated_answer.get(\"spans\", \"\"),\n \"worker_id\": validated_answer.get(\"worker_id\", \"\"),\n \"hit_id\": validated_answer.get(\"hit_id\", \"\"),\n }\n validated_answers.append(va)\n else:\n validated_answers = _EMPTY_VALIDATED_ANSWER\n yield key, {\n \"section_id\": section_id,\n \"passage\": example[\"passage\"],\n \"question\": qa[\"question\"],\n \"query_id\": qa[\"query_id\"],\n \"answer\": answer,\n \"validated_answers\": validated_answers,\n }\n key += 1\n" }, { "path": "lm_eval/datasets/headqa/__init__.py", "content": "" }, { "path": "lm_eval/datasets/headqa/dataset_infos.json", "content": "{\"es\": {\"description\": \"HEAD-QA is a multi-choice HEAlthcare Dataset. The questions come from exams to access a specialized position in the\\nSpanish healthcare system, and are challenging even for highly specialized humans. They are designed by the Ministerio\\nde Sanidad, Consumo y Bienestar Social.\\nThe dataset contains questions about the following topics: medicine, nursing, psychology, chemistry, pharmacology and biology.\\n\", \"citation\": \"@inproceedings{vilares-gomez-rodriguez-2019-head,\\n title = \\\"{HEAD}-{QA}: A Healthcare Dataset for Complex Reasoning\\\",\\n author = \\\"Vilares, David and\\n G{'o}mez-Rodr{'i}guez, Carlos\\\",\\n booktitle = \\\"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics\\\",\\n month = jul,\\n year = \\\"2019\\\",\\n address = \\\"Florence, Italy\\\",\\n publisher = \\\"Association for Computational Linguistics\\\",\\n url = \\\"https://www.aclweb.org/anthology/P19-1092\\\",\\n doi = \\\"10.18653/v1/P19-1092\\\",\\n pages = \\\"960--966\\\",\\n abstract = \\\"We present HEAD-QA, a multi-choice question answering testbed to encourage research on complex reasoning. The questions come from exams to access a specialized position in the Spanish healthcare system, and are challenging even for highly specialized humans. We then consider monolingual (Spanish) and cross-lingual (to English) experiments with information retrieval and neural techniques. We show that: (i) HEAD-QA challenges current methods, and (ii) the results lag well behind human performance, demonstrating its usefulness as a benchmark for future work.\\\",\\n}\\n\", \"homepage\": \"https://aghie.github.io/head-qa/\", \"license\": \"MIT License\", \"features\": {\"name\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"year\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"category\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"qid\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}, \"qtext\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"ra\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}, \"answers\": [{\"aid\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}, \"atext\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}]}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"head_qa\", \"config_name\": \"es\", \"version\": {\"version_str\": \"1.1.0\", \"description\": null, \"major\": 1, \"minor\": 1, \"patch\": 0}, \"splits\": {\"train\": {\"name\": \"train\", \"num_bytes\": 1196021, \"num_examples\": 2657, \"dataset_name\": \"head_qa\"}, \"test\": {\"name\": \"test\", \"num_bytes\": 1169819, \"num_examples\": 2742, \"dataset_name\": \"head_qa\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 556924, \"num_examples\": 1366, \"dataset_name\": \"head_qa\"}}, \"download_checksums\": {\"https://drive.google.com/uc?export=download&confirm=t&id=1a_95N5zQQoUCq8IBNVZgziHbeM-QxG2t\": {\"num_bytes\": 79365502, \"checksum\": \"6ec29a3f55153d167f0bdf05395558919ba0b1df9c63e79ffceda2a09884ad8b\"}}, \"download_size\": 79365502, \"post_processing_size\": null, \"dataset_size\": 2922764, \"size_in_bytes\": 82288266}, \"en\": {\"description\": \"HEAD-QA is a multi-choice HEAlthcare Dataset. The questions come from exams to access a specialized position in the\\nSpanish healthcare system, and are challenging even for highly specialized humans. They are designed by the Ministerio\\nde Sanidad, Consumo y Bienestar Social.\\nThe dataset contains questions about the following topics: medicine, nursing, psychology, chemistry, pharmacology and biology.\\n\", \"citation\": \"@inproceedings{vilares-gomez-rodriguez-2019-head,\\n title = \\\"{HEAD}-{QA}: A Healthcare Dataset for Complex Reasoning\\\",\\n author = \\\"Vilares, David and\\n G{'o}mez-Rodr{'i}guez, Carlos\\\",\\n booktitle = \\\"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics\\\",\\n month = jul,\\n year = \\\"2019\\\",\\n address = \\\"Florence, Italy\\\",\\n publisher = \\\"Association for Computational Linguistics\\\",\\n url = \\\"https://www.aclweb.org/anthology/P19-1092\\\",\\n doi = \\\"10.18653/v1/P19-1092\\\",\\n pages = \\\"960--966\\\",\\n abstract = \\\"We present HEAD-QA, a multi-choice question answering testbed to encourage research on complex reasoning. The questions come from exams to access a specialized position in the Spanish healthcare system, and are challenging even for highly specialized humans. We then consider monolingual (Spanish) and cross-lingual (to English) experiments with information retrieval and neural techniques. We show that: (i) HEAD-QA challenges current methods, and (ii) the results lag well behind human performance, demonstrating its usefulness as a benchmark for future work.\\\",\\n}\\n\", \"homepage\": \"https://aghie.github.io/head-qa/\", \"license\": \"MIT License\", \"features\": {\"name\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"year\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"category\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"qid\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}, \"qtext\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"ra\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}, \"answers\": [{\"aid\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}, \"atext\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}]}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"head_qa\", \"config_name\": \"en\", \"version\": {\"version_str\": \"1.1.0\", \"description\": null, \"major\": 1, \"minor\": 1, \"patch\": 0}, \"splits\": {\"train\": {\"name\": \"train\", \"num_bytes\": 1123151, \"num_examples\": 2657, \"dataset_name\": \"head_qa\"}, \"test\": {\"name\": \"test\", \"num_bytes\": 1097349, \"num_examples\": 2742, \"dataset_name\": \"head_qa\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 523462, \"num_examples\": 1366, \"dataset_name\": \"head_qa\"}}, \"download_checksums\": {\"https://drive.google.com/uc?export=download&confirm=t&id=1a_95N5zQQoUCq8IBNVZgziHbeM-QxG2t\": {\"num_bytes\": 79365502, \"checksum\": \"6ec29a3f55153d167f0bdf05395558919ba0b1df9c63e79ffceda2a09884ad8b\"}}, \"download_size\": 79365502, \"post_processing_size\": null, \"dataset_size\": 2743962, \"size_in_bytes\": 82109464}}\n" }, { "path": "lm_eval/datasets/headqa/headqa.py", "content": "# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n# NOTE: This is an exact copy of\n# https://github.com/huggingface/datasets/blob/3804442bb7cfcb9d52044d92688115cfdc69c2da/datasets/head_qa/head_qa.py\n# with the exception of the `image` feature. This is to avoid adding `Pillow`\n# as a dependency.\n\"\"\"HEAD-QA: A Healthcare Dataset for Complex Reasoning.\"\"\"\n\n\nimport json\nimport os\n\nimport datasets\n\n\n_CITATION = \"\"\"\\\n@inproceedings{vilares-gomez-rodriguez-2019-head,\n title = \"{HEAD}-{QA}: A Healthcare Dataset for Complex Reasoning\",\n author = \"Vilares, David and\n G{\\'o}mez-Rodr{\\'i}guez, Carlos\",\n booktitle = \"Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics\",\n month = jul,\n year = \"2019\",\n address = \"Florence, Italy\",\n publisher = \"Association for Computational Linguistics\",\n url = \"https://www.aclweb.org/anthology/P19-1092\",\n doi = \"10.18653/v1/P19-1092\",\n pages = \"960--966\",\n abstract = \"We present HEAD-QA, a multi-choice question answering testbed to encourage research on complex reasoning. The questions come from exams to access a specialized position in the Spanish healthcare system, and are challenging even for highly specialized humans. We then consider monolingual (Spanish) and cross-lingual (to English) experiments with information retrieval and neural techniques. We show that: (i) HEAD-QA challenges current methods, and (ii) the results lag well behind human performance, demonstrating its usefulness as a benchmark for future work.\",\n}\n\"\"\"\n\n_DESCRIPTION = \"\"\"\\\nHEAD-QA is a multi-choice HEAlthcare Dataset. The questions come from exams to access a specialized position in the\nSpanish healthcare system, and are challenging even for highly specialized humans. They are designed by the Ministerio\nde Sanidad, Consumo y Bienestar Social.\nThe dataset contains questions about the following topics: medicine, nursing, psychology, chemistry, pharmacology and biology.\n\"\"\"\n\n_HOMEPAGE = \"https://aghie.github.io/head-qa/\"\n\n_LICENSE = \"MIT License\"\n\n_URL = \"https://drive.google.com/uc?export=download&confirm=t&id=1a_95N5zQQoUCq8IBNVZgziHbeM-QxG2t\"\n\n_DIRS = {\"es\": \"HEAD\", \"en\": \"HEAD_EN\"}\n\n\nclass HeadQA(datasets.GeneratorBasedBuilder):\n \"\"\"HEAD-QA: A Healthcare Dataset for Complex Reasoning\"\"\"\n\n VERSION = datasets.Version(\"1.1.0\")\n\n BUILDER_CONFIGS = [\n datasets.BuilderConfig(\n name=\"es\", version=VERSION, description=\"Spanish HEAD dataset\"\n ),\n datasets.BuilderConfig(\n name=\"en\", version=VERSION, description=\"English HEAD dataset\"\n ),\n ]\n\n DEFAULT_CONFIG_NAME = \"es\"\n\n def _info(self):\n return datasets.DatasetInfo(\n description=_DESCRIPTION,\n features=datasets.Features(\n {\n \"name\": datasets.Value(\"string\"),\n \"year\": datasets.Value(\"string\"),\n \"category\": datasets.Value(\"string\"),\n \"qid\": datasets.Value(\"int32\"),\n \"qtext\": datasets.Value(\"string\"),\n \"ra\": datasets.Value(\"int32\"),\n \"answers\": [\n {\n \"aid\": datasets.Value(\"int32\"),\n \"atext\": datasets.Value(\"string\"),\n }\n ],\n }\n ),\n supervised_keys=None,\n homepage=_HOMEPAGE,\n license=_LICENSE,\n citation=_CITATION,\n )\n\n def _split_generators(self, dl_manager):\n \"\"\"Returns SplitGenerators.\"\"\"\n data_dir = dl_manager.download_and_extract(_URL)\n\n dir = _DIRS[self.config.name]\n data_lang_dir = os.path.join(data_dir, dir)\n\n return [\n datasets.SplitGenerator(\n name=datasets.Split.TRAIN,\n gen_kwargs={\n \"data_dir\": data_dir,\n \"filepath\": os.path.join(data_lang_dir, f\"train_{dir}.json\"),\n },\n ),\n datasets.SplitGenerator(\n name=datasets.Split.TEST,\n gen_kwargs={\n \"data_dir\": data_dir,\n \"filepath\": os.path.join(data_lang_dir, f\"test_{dir}.json\"),\n },\n ),\n datasets.SplitGenerator(\n name=datasets.Split.VALIDATION,\n gen_kwargs={\n \"data_dir\": data_dir,\n \"filepath\": os.path.join(data_lang_dir, f\"dev_{dir}.json\"),\n },\n ),\n ]\n\n def _generate_examples(self, data_dir, filepath):\n \"\"\"Yields examples.\"\"\"\n with open(filepath, encoding=\"utf-8\") as f:\n head_qa = json.load(f)\n for exam_id, exam in enumerate(head_qa[\"exams\"]):\n content = head_qa[\"exams\"][exam]\n name = content[\"name\"].strip()\n year = content[\"year\"].strip()\n category = content[\"category\"].strip()\n for question in content[\"data\"]:\n qid = int(question[\"qid\"].strip())\n qtext = question[\"qtext\"].strip()\n ra = int(question[\"ra\"].strip())\n\n aids = [answer[\"aid\"] for answer in question[\"answers\"]]\n atexts = [answer[\"atext\"].strip() for answer in question[\"answers\"]]\n answers = [\n {\"aid\": aid, \"atext\": atext} for aid, atext in zip(aids, atexts)\n ]\n\n id_ = f\"{exam_id}_{qid}\"\n yield id_, {\n \"name\": name,\n \"year\": year,\n \"category\": category,\n \"qid\": qid,\n \"qtext\": qtext,\n \"ra\": ra,\n \"answers\": answers,\n }\n" }, { "path": "lm_eval/datasets/hendrycks_ethics/__init__.py", "content": "" }, { "path": "lm_eval/datasets/hendrycks_ethics/dataset_infos.json", "content": "{\"commonsense\": {\"description\": \"The ETHICS dataset is a benchmark that spans concepts in justice, well-being,\\nduties, virtues, and commonsense morality. Models predict widespread moral\\njudgments about diverse text scenarios. This requires connecting physical and\\nsocial world knowledge to value judgements, a capability that may enable us\\nto steer chatbot outputs or eventually regularize open-ended reinforcement\\nlearning agents.\\n\\nThe Commonsense subset contains examples focusing on moral standards and principles that most people intuitively accept.\", \"citation\": \"@article{hendrycks2021ethics\\n title={Aligning AI With Shared Human Values},\\n author={Dan Hendrycks and Collin Burns and Steven Basart and Andrew Critch and Jerry Li and Dawn Song and Jacob Steinhardt},\\n journal={Proceedings of the International Conference on Learning Representations (ICLR)},\\n year={2021}\\n}\\n\", \"homepage\": \"https://github.com/hendrycks/ethics\", \"license\": \"\", \"features\": {\"label\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}, \"input\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"is_short\": {\"dtype\": \"bool\", \"id\": null, \"_type\": \"Value\"}, \"edited\": {\"dtype\": \"bool\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"hendrycks_ethics\", \"config_name\": \"commonsense\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"train\": {\"name\": \"train\", \"num_bytes\": 14435215, \"num_examples\": 13910, \"dataset_name\": \"hendrycks_ethics\"}, \"test\": {\"name\": \"test\", \"num_bytes\": 3150094, \"num_examples\": 3885, \"dataset_name\": \"hendrycks_ethics\"}}, \"download_checksums\": {\"https://people.eecs.berkeley.edu/~hendrycks/ethics.tar\": {\"num_bytes\": 35585024, \"checksum\": \"40acbf1ac0da79a2aabef394d58889136b8d38b05be09482006de2453fb06333\"}}, \"download_size\": 35585024, \"post_processing_size\": null, \"dataset_size\": 17585309, \"size_in_bytes\": 53170333}, \"deontology\": {\"description\": \"The ETHICS dataset is a benchmark that spans concepts in justice, well-being,\\nduties, virtues, and commonsense morality. Models predict widespread moral\\njudgments about diverse text scenarios. This requires connecting physical and\\nsocial world knowledge to value judgements, a capability that may enable us\\nto steer chatbot outputs or eventually regularize open-ended reinforcement\\nlearning agents.\\n\\nThe Deontology subset contains examples focusing on whether an act is required, permitted, or forbidden according to a set of rules or constraints\", \"citation\": \"@article{hendrycks2021ethics\\n title={Aligning AI With Shared Human Values},\\n author={Dan Hendrycks and Collin Burns and Steven Basart and Andrew Critch and Jerry Li and Dawn Song and Jacob Steinhardt},\\n journal={Proceedings of the International Conference on Learning Representations (ICLR)},\\n year={2021}\\n}\\n\", \"homepage\": \"https://github.com/hendrycks/ethics\", \"license\": \"\", \"features\": {\"group_id\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}, \"label\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}, \"scenario\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"excuse\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"hendrycks_ethics\", \"config_name\": \"deontology\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"train\": {\"name\": \"train\", \"num_bytes\": 1931475, \"num_examples\": 18164, \"dataset_name\": \"hendrycks_ethics\"}, \"test\": {\"name\": \"test\", \"num_bytes\": 384602, \"num_examples\": 3596, \"dataset_name\": \"hendrycks_ethics\"}}, \"download_checksums\": {\"https://people.eecs.berkeley.edu/~hendrycks/ethics.tar\": {\"num_bytes\": 35585024, \"checksum\": \"40acbf1ac0da79a2aabef394d58889136b8d38b05be09482006de2453fb06333\"}}, \"download_size\": 35585024, \"post_processing_size\": null, \"dataset_size\": 2316077, \"size_in_bytes\": 37901101}, \"justice\": {\"description\": \"The ETHICS dataset is a benchmark that spans concepts in justice, well-being,\\nduties, virtues, and commonsense morality. Models predict widespread moral\\njudgments about diverse text scenarios. This requires connecting physical and\\nsocial world knowledge to value judgements, a capability that may enable us\\nto steer chatbot outputs or eventually regularize open-ended reinforcement\\nlearning agents.\\n\\nThe Justice subset contains examples focusing on how a character treats another person\", \"citation\": \"@article{hendrycks2021ethics\\n title={Aligning AI With Shared Human Values},\\n author={Dan Hendrycks and Collin Burns and Steven Basart and Andrew Critch and Jerry Li and Dawn Song and Jacob Steinhardt},\\n journal={Proceedings of the International Conference on Learning Representations (ICLR)},\\n year={2021}\\n}\\n\", \"homepage\": \"https://github.com/hendrycks/ethics\", \"license\": \"\", \"features\": {\"group_id\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}, \"label\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}, \"scenario\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"hendrycks_ethics\", \"config_name\": \"justice\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"train\": {\"name\": \"train\", \"num_bytes\": 2516501, \"num_examples\": 21791, \"dataset_name\": \"hendrycks_ethics\"}, \"test\": {\"name\": \"test\", \"num_bytes\": 309427, \"num_examples\": 2704, \"dataset_name\": \"hendrycks_ethics\"}}, \"download_checksums\": {\"https://people.eecs.berkeley.edu/~hendrycks/ethics.tar\": {\"num_bytes\": 35585024, \"checksum\": \"40acbf1ac0da79a2aabef394d58889136b8d38b05be09482006de2453fb06333\"}}, \"download_size\": 35585024, \"post_processing_size\": null, \"dataset_size\": 2825928, \"size_in_bytes\": 38410952}, \"utilitarianism\": {\"description\": \"The ETHICS dataset is a benchmark that spans concepts in justice, well-being,\\nduties, virtues, and commonsense morality. Models predict widespread moral\\njudgments about diverse text scenarios. This requires connecting physical and\\nsocial world knowledge to value judgements, a capability that may enable us\\nto steer chatbot outputs or eventually regularize open-ended reinforcement\\nlearning agents.\\n\\nThe Utilitarianism subset contains scenarios that should be ranked from most pleasant to least pleasant for the person in the scenario\", \"citation\": \"@article{hendrycks2021ethics\\n title={Aligning AI With Shared Human Values},\\n author={Dan Hendrycks and Collin Burns and Steven Basart and Andrew Critch and Jerry Li and Dawn Song and Jacob Steinhardt},\\n journal={Proceedings of the International Conference on Learning Representations (ICLR)},\\n year={2021}\\n}\\n\", \"homepage\": \"https://github.com/hendrycks/ethics\", \"license\": \"\", \"features\": {\"activity\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"baseline\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"rating\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"hendrycks_ethics\", \"config_name\": \"utilitarianism\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"train\": {\"name\": \"train\", \"num_bytes\": 2241770, \"num_examples\": 13738, \"dataset_name\": \"hendrycks_ethics\"}, \"test\": {\"name\": \"test\", \"num_bytes\": 749768, \"num_examples\": 4808, \"dataset_name\": \"hendrycks_ethics\"}}, \"download_checksums\": {\"https://people.eecs.berkeley.edu/~hendrycks/ethics.tar\": {\"num_bytes\": 35585024, \"checksum\": \"40acbf1ac0da79a2aabef394d58889136b8d38b05be09482006de2453fb06333\"}}, \"download_size\": 35585024, \"post_processing_size\": null, \"dataset_size\": 2991538, \"size_in_bytes\": 38576562}, \"virtue\": {\"description\": \"The ETHICS dataset is a benchmark that spans concepts in justice, well-being,\\nduties, virtues, and commonsense morality. Models predict widespread moral\\njudgments about diverse text scenarios. This requires connecting physical and\\nsocial world knowledge to value judgements, a capability that may enable us\\nto steer chatbot outputs or eventually regularize open-ended reinforcement\\nlearning agents.\\n\\nThe Virtue subset contains scenarios focusing on whether virtues or vices are being exemplified\", \"citation\": \"@article{hendrycks2021ethics\\n title={Aligning AI With Shared Human Values},\\n author={Dan Hendrycks and Collin Burns and Steven Basart and Andrew Critch and Jerry Li and Dawn Song and Jacob Steinhardt},\\n journal={Proceedings of the International Conference on Learning Representations (ICLR)},\\n year={2021}\\n}\\n\", \"homepage\": \"https://github.com/hendrycks/ethics\", \"license\": \"\", \"features\": {\"group_id\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}, \"label\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}, \"scenario\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"trait\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"hendrycks_ethics\", \"config_name\": \"virtue\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"train\": {\"name\": \"train\", \"num_bytes\": 2640328, \"num_examples\": 28245, \"dataset_name\": \"hendrycks_ethics\"}, \"test\": {\"name\": \"test\", \"num_bytes\": 473473, \"num_examples\": 4975, \"dataset_name\": \"hendrycks_ethics\"}}, \"download_checksums\": {\"https://people.eecs.berkeley.edu/~hendrycks/ethics.tar\": {\"num_bytes\": 35585024, \"checksum\": \"40acbf1ac0da79a2aabef394d58889136b8d38b05be09482006de2453fb06333\"}}, \"download_size\": 35585024, \"post_processing_size\": null, \"dataset_size\": 3113801, \"size_in_bytes\": 38698825}}\n" }, { "path": "lm_eval/datasets/hendrycks_ethics/hendrycks_ethics.py", "content": "# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"ETHICS dataset.\"\"\"\n# TODO: Add the `hard` dataset splits.\n\n\nimport csv\nimport os\n\nimport datasets\n\n\n_CITATION = \"\"\"\\\n@article{hendrycks2021ethics\n title={Aligning AI With Shared Human Values},\n author={Dan Hendrycks and Collin Burns and Steven Basart and Andrew Critch and Jerry Li and Dawn Song and Jacob Steinhardt},\n journal={Proceedings of the International Conference on Learning Representations (ICLR)},\n year={2021}\n}\n\"\"\"\n\n_DESCRIPTION = \"\"\"\\\nThe ETHICS dataset is a benchmark that spans concepts in justice, well-being,\nduties, virtues, and commonsense morality. Models predict widespread moral\njudgments about diverse text scenarios. This requires connecting physical and\nsocial world knowledge to value judgements, a capability that may enable us\nto steer chatbot outputs or eventually regularize open-ended reinforcement\nlearning agents.\n\"\"\"\n\n_HOMEPAGE = \"https://github.com/hendrycks/ethics\"\n\n# TODO: Add the licence for the dataset here if you can find it\n_LICENSE = \"\"\n\n_URLS = \"https://people.eecs.berkeley.edu/~hendrycks/ethics.tar\"\n\n\nclass EthicsConfig(datasets.BuilderConfig):\n \"\"\"BuilderConfig for Hendrycks ETHICS.\"\"\"\n\n def __init__(self, prefix, features, **kwargs):\n \"\"\"BuilderConfig for Hendrycks ETHICS.\n\n Args:\n prefix: *string*, prefix to add to the dataset name for path location.\n features: *list[string]*, list of the features that will appear in the\n feature dict.\n \"\"\"\n # Version history:\n super().__init__(version=datasets.Version(\"0.0.1\"), **kwargs)\n self.prefix = prefix\n self.features = features\n\n\nclass HendrycksEthics(datasets.GeneratorBasedBuilder):\n \"\"\"The ETHICS dataset is a benchmark that spans concepts in justice, well-being, duties, virtues, and commonsense morality.\"\"\"\n\n BUILDER_CONFIGS = [\n EthicsConfig(\n name=\"commonsense\",\n prefix=\"cm\",\n features=datasets.Features(\n {\n \"label\": datasets.Value(\"int32\"),\n \"input\": datasets.Value(\"string\"),\n \"is_short\": datasets.Value(\"bool\"),\n \"edited\": datasets.Value(\"bool\"),\n }\n ),\n description=\"The Commonsense subset contains examples focusing on moral standards and principles that most people intuitively accept.\",\n ),\n EthicsConfig(\n name=\"deontology\",\n prefix=\"deontology\",\n features=datasets.Features(\n {\n \"group_id\": datasets.Value(\"int32\"),\n \"label\": datasets.Value(\"int32\"),\n \"scenario\": datasets.Value(\"string\"),\n \"excuse\": datasets.Value(\"string\"),\n }\n ),\n description=\"The Deontology subset contains examples focusing on whether an act is required, permitted, or forbidden according to a set of rules or constraints\",\n ),\n EthicsConfig(\n name=\"justice\",\n prefix=\"justice\",\n features=datasets.Features(\n {\n \"group_id\": datasets.Value(\"int32\"),\n \"label\": datasets.Value(\"int32\"),\n \"scenario\": datasets.Value(\"string\"),\n }\n ),\n description=\"The Justice subset contains examples focusing on how a character treats another person\",\n ),\n EthicsConfig(\n name=\"utilitarianism\",\n prefix=\"util\",\n features=datasets.Features(\n {\n \"activity\": datasets.Value(\"string\"),\n \"baseline\": datasets.Value(\"string\"),\n \"rating\": datasets.Value(\"string\"), # Empty rating.\n }\n ),\n description=\"The Utilitarianism subset contains scenarios that should be ranked from most pleasant to least pleasant for the person in the scenario\",\n ),\n EthicsConfig(\n name=\"virtue\",\n prefix=\"virtue\",\n features=datasets.Features(\n {\n \"group_id\": datasets.Value(\"int32\"),\n \"label\": datasets.Value(\"int32\"),\n \"scenario\": datasets.Value(\"string\"),\n \"trait\": datasets.Value(\"string\"),\n }\n ),\n description=\"The Virtue subset contains scenarios focusing on whether virtues or vices are being exemplified\",\n ),\n ]\n\n def _info(self):\n return datasets.DatasetInfo(\n description=f\"{_DESCRIPTION}\\n{self.config.description}\",\n features=self.config.features,\n homepage=_HOMEPAGE,\n license=_LICENSE,\n citation=_CITATION,\n )\n\n def _split_generators(self, dl_manager):\n urls = _URLS\n data_dir = dl_manager.download_and_extract(urls)\n return [\n datasets.SplitGenerator(\n name=datasets.Split.TRAIN,\n # These kwargs will be passed to _generate_examples\n gen_kwargs={\n \"filepath\": os.path.join(\n data_dir,\n \"ethics\",\n self.config.name,\n f\"{self.config.prefix}_train.csv\",\n ),\n \"split\": \"train\",\n },\n ),\n datasets.SplitGenerator(\n name=datasets.Split.TEST,\n # These kwargs will be passed to _generate_examples\n gen_kwargs={\n \"filepath\": os.path.join(\n data_dir,\n \"ethics\",\n self.config.name,\n f\"{self.config.prefix}_test.csv\",\n ),\n \"split\": \"test\",\n },\n ),\n ]\n\n # method parameters are unpacked from `gen_kwargs` as given in `_split_generators`\n def _generate_examples(self, filepath, split):\n with open(filepath, newline=\"\") as f:\n if self.config.name == \"utilitarianism\":\n contents = csv.DictReader(f, fieldnames=[\"activity\", \"baseline\"])\n else:\n contents = csv.DictReader(f)\n # For subsets with grouped scenarios, tag them with an id.\n group_id = 0\n for key, row in enumerate(contents):\n if self.config.name == \"deontology\":\n # Scenarios come in groups of 4.\n if key % 4 == 0 and key != 0:\n group_id += 1\n yield key, {\n \"group_id\": group_id,\n \"label\": row[\"label\"],\n \"scenario\": row[\"scenario\"],\n \"excuse\": row[\"excuse\"],\n }\n elif self.config.name == \"justice\":\n # Scenarios come in groups of 4.\n if key % 4 == 0 and key != 0:\n group_id += 1\n yield key, {\n \"group_id\": group_id,\n \"label\": row[\"label\"],\n \"scenario\": row[\"scenario\"],\n }\n elif self.config.name == \"commonsense\":\n yield key, {\n \"label\": row[\"label\"],\n \"input\": row[\"input\"],\n \"is_short\": row[\"is_short\"],\n \"edited\": row[\"edited\"],\n }\n elif self.config.name == \"virtue\":\n # Scenarios come in groups of 5.\n if key % 5 == 0 and key != 0:\n group_id += 1\n scenario, trait = row[\"scenario\"].split(\" [SEP] \")\n yield key, {\n \"group_id\": group_id,\n \"label\": row[\"label\"],\n \"scenario\": scenario,\n \"trait\": trait,\n }\n elif self.config.name == \"utilitarianism\":\n yield key, {\n \"activity\": row[\"activity\"],\n \"baseline\": row[\"baseline\"],\n \"rating\": \"\",\n }\n" }, { "path": "lm_eval/datasets/hendrycks_math/__init__.py", "content": "" }, { "path": "lm_eval/datasets/hendrycks_math/dataset_infos.json", "content": "{\"algebra\": {\"description\": \"MATH is a dataset of 12,500 challenging competition mathematics problems. Each\\nproblem in Math has a full step-by-step solution which can be used to teach\\nmodels to generate answer derivations and explanations.\\n\", \"citation\": \"@article{hendrycksmath2021,\\n title={Measuring Mathematical Problem Solving With the Math Dataset},\\n author={Dan Hendrycks and Collin Burns and Saurav Kadavath and Akul Arora and Steven Basart and Eric Tang and Dawn Song and Jacob Steinhardt},\\n journal={NeurIPS},\\n year={2021}\\n}\\n\", \"homepage\": \"https://github.com/hendrycks/math\", \"license\": \"\", \"features\": {\"problem\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"level\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"type\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"solution\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"hendrycks_math\", \"config_name\": \"algebra\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"train\": {\"name\": \"train\", \"num_bytes\": 955021, \"num_examples\": 1744, \"dataset_name\": \"hendrycks_math\"}, \"test\": {\"name\": \"test\", \"num_bytes\": 648291, \"num_examples\": 1187, \"dataset_name\": \"hendrycks_math\"}}, \"download_checksums\": {\"https://people.eecs.berkeley.edu/~hendrycks/MATH.tar\": {\"num_bytes\": 20327936, \"checksum\": \"0fbe4fad0df66942db6c221cdcc95b298cc7f4595a2f0f518360cce84e90d9ac\"}}, \"download_size\": 20327936, \"post_processing_size\": null, \"dataset_size\": 1603312, \"size_in_bytes\": 21931248}, \"counting_and_probability\": {\"description\": \"MATH is a dataset of 12,500 challenging competition mathematics problems. Each\\nproblem in Math has a full step-by-step solution which can be used to teach\\nmodels to generate answer derivations and explanations.\\n\", \"citation\": \"@article{hendrycksmath2021,\\n title={Measuring Mathematical Problem Solving With the Math Dataset},\\n author={Dan Hendrycks and Collin Burns and Saurav Kadavath and Akul Arora and Steven Basart and Eric Tang and Dawn Song and Jacob Steinhardt},\\n journal={NeurIPS},\\n year={2021}\\n}\\n\", \"homepage\": \"https://github.com/hendrycks/math\", \"license\": \"\", \"features\": {\"problem\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"level\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"type\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"solution\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"hendrycks_math\", \"config_name\": \"counting_and_probability\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"train\": {\"name\": \"train\", \"num_bytes\": 667385, \"num_examples\": 771, \"dataset_name\": \"hendrycks_math\"}, \"test\": {\"name\": \"test\", \"num_bytes\": 353803, \"num_examples\": 474, \"dataset_name\": \"hendrycks_math\"}}, \"download_checksums\": {\"https://people.eecs.berkeley.edu/~hendrycks/MATH.tar\": {\"num_bytes\": 20327936, \"checksum\": \"0fbe4fad0df66942db6c221cdcc95b298cc7f4595a2f0f518360cce84e90d9ac\"}}, \"download_size\": 20327936, \"post_processing_size\": null, \"dataset_size\": 1021188, \"size_in_bytes\": 21349124}, \"geometry\": {\"description\": \"MATH is a dataset of 12,500 challenging competition mathematics problems. Each\\nproblem in Math has a full step-by-step solution which can be used to teach\\nmodels to generate answer derivations and explanations.\\n\", \"citation\": \"@article{hendrycksmath2021,\\n title={Measuring Mathematical Problem Solving With the Math Dataset},\\n author={Dan Hendrycks and Collin Burns and Saurav Kadavath and Akul Arora and Steven Basart and Eric Tang and Dawn Song and Jacob Steinhardt},\\n journal={NeurIPS},\\n year={2021}\\n}\\n\", \"homepage\": \"https://github.com/hendrycks/math\", \"license\": \"\", \"features\": {\"problem\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"level\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"type\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"solution\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"hendrycks_math\", \"config_name\": \"geometry\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"train\": {\"name\": \"train\", \"num_bytes\": 1077241, \"num_examples\": 870, \"dataset_name\": \"hendrycks_math\"}, \"test\": {\"name\": \"test\", \"num_bytes\": 523126, \"num_examples\": 479, \"dataset_name\": \"hendrycks_math\"}}, \"download_checksums\": {\"https://people.eecs.berkeley.edu/~hendrycks/MATH.tar\": {\"num_bytes\": 20327936, \"checksum\": \"0fbe4fad0df66942db6c221cdcc95b298cc7f4595a2f0f518360cce84e90d9ac\"}}, \"download_size\": 20327936, \"post_processing_size\": null, \"dataset_size\": 1600367, \"size_in_bytes\": 21928303}, \"intermediate_algebra\": {\"description\": \"MATH is a dataset of 12,500 challenging competition mathematics problems. Each\\nproblem in Math has a full step-by-step solution which can be used to teach\\nmodels to generate answer derivations and explanations.\\n\", \"citation\": \"@article{hendrycksmath2021,\\n title={Measuring Mathematical Problem Solving With the Math Dataset},\\n author={Dan Hendrycks and Collin Burns and Saurav Kadavath and Akul Arora and Steven Basart and Eric Tang and Dawn Song and Jacob Steinhardt},\\n journal={NeurIPS},\\n year={2021}\\n}\\n\", \"homepage\": \"https://github.com/hendrycks/math\", \"license\": \"\", \"features\": {\"problem\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"level\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"type\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"solution\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"hendrycks_math\", \"config_name\": \"intermediate_algebra\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"train\": {\"name\": \"train\", \"num_bytes\": 1157476, \"num_examples\": 1295, \"dataset_name\": \"hendrycks_math\"}, \"test\": {\"name\": \"test\", \"num_bytes\": 795070, \"num_examples\": 903, \"dataset_name\": \"hendrycks_math\"}}, \"download_checksums\": {\"https://people.eecs.berkeley.edu/~hendrycks/MATH.tar\": {\"num_bytes\": 20327936, \"checksum\": \"0fbe4fad0df66942db6c221cdcc95b298cc7f4595a2f0f518360cce84e90d9ac\"}}, \"download_size\": 20327936, \"post_processing_size\": null, \"dataset_size\": 1952546, \"size_in_bytes\": 22280482}, \"number_theory\": {\"description\": \"MATH is a dataset of 12,500 challenging competition mathematics problems. Each\\nproblem in Math has a full step-by-step solution which can be used to teach\\nmodels to generate answer derivations and explanations.\\n\", \"citation\": \"@article{hendrycksmath2021,\\n title={Measuring Mathematical Problem Solving With the Math Dataset},\\n author={Dan Hendrycks and Collin Burns and Saurav Kadavath and Akul Arora and Steven Basart and Eric Tang and Dawn Song and Jacob Steinhardt},\\n journal={NeurIPS},\\n year={2021}\\n}\\n\", \"homepage\": \"https://github.com/hendrycks/math\", \"license\": \"\", \"features\": {\"problem\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"level\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"type\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"solution\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"hendrycks_math\", \"config_name\": \"number_theory\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"train\": {\"name\": \"train\", \"num_bytes\": 595793, \"num_examples\": 869, \"dataset_name\": \"hendrycks_math\"}, \"test\": {\"name\": \"test\", \"num_bytes\": 349455, \"num_examples\": 540, \"dataset_name\": \"hendrycks_math\"}}, \"download_checksums\": {\"https://people.eecs.berkeley.edu/~hendrycks/MATH.tar\": {\"num_bytes\": 20327936, \"checksum\": \"0fbe4fad0df66942db6c221cdcc95b298cc7f4595a2f0f518360cce84e90d9ac\"}}, \"download_size\": 20327936, \"post_processing_size\": null, \"dataset_size\": 945248, \"size_in_bytes\": 21273184}, \"prealgebra\": {\"description\": \"MATH is a dataset of 12,500 challenging competition mathematics problems. Each\\nproblem in Math has a full step-by-step solution which can be used to teach\\nmodels to generate answer derivations and explanations.\\n\", \"citation\": \"@article{hendrycksmath2021,\\n title={Measuring Mathematical Problem Solving With the Math Dataset},\\n author={Dan Hendrycks and Collin Burns and Saurav Kadavath and Akul Arora and Steven Basart and Eric Tang and Dawn Song and Jacob Steinhardt},\\n journal={NeurIPS},\\n year={2021}\\n}\\n\", \"homepage\": \"https://github.com/hendrycks/math\", \"license\": \"\", \"features\": {\"problem\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"level\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"type\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"solution\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"hendrycks_math\", \"config_name\": \"prealgebra\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"train\": {\"name\": \"train\", \"num_bytes\": 715611, \"num_examples\": 1205, \"dataset_name\": \"hendrycks_math\"}, \"test\": {\"name\": \"test\", \"num_bytes\": 510195, \"num_examples\": 871, \"dataset_name\": \"hendrycks_math\"}}, \"download_checksums\": {\"https://people.eecs.berkeley.edu/~hendrycks/MATH.tar\": {\"num_bytes\": 20327936, \"checksum\": \"0fbe4fad0df66942db6c221cdcc95b298cc7f4595a2f0f518360cce84e90d9ac\"}}, \"download_size\": 20327936, \"post_processing_size\": null, \"dataset_size\": 1225806, \"size_in_bytes\": 21553742}, \"precalculus\": {\"description\": \"MATH is a dataset of 12,500 challenging competition mathematics problems. Each\\nproblem in Math has a full step-by-step solution which can be used to teach\\nmodels to generate answer derivations and explanations.\\n\", \"citation\": \"@article{hendrycksmath2021,\\n title={Measuring Mathematical Problem Solving With the Math Dataset},\\n author={Dan Hendrycks and Collin Burns and Saurav Kadavath and Akul Arora and Steven Basart and Eric Tang and Dawn Song and Jacob Steinhardt},\\n journal={NeurIPS},\\n year={2021}\\n}\\n\", \"homepage\": \"https://github.com/hendrycks/math\", \"license\": \"\", \"features\": {\"problem\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"level\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"type\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"solution\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"hendrycks_math\", \"config_name\": \"precalculus\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"train\": {\"name\": \"train\", \"num_bytes\": 816245, \"num_examples\": 746, \"dataset_name\": \"hendrycks_math\"}, \"test\": {\"name\": \"test\", \"num_bytes\": 552893, \"num_examples\": 546, \"dataset_name\": \"hendrycks_math\"}}, \"download_checksums\": {\"https://people.eecs.berkeley.edu/~hendrycks/MATH.tar\": {\"num_bytes\": 20327936, \"checksum\": \"0fbe4fad0df66942db6c221cdcc95b298cc7f4595a2f0f518360cce84e90d9ac\"}}, \"download_size\": 20327936, \"post_processing_size\": null, \"dataset_size\": 1369138, \"size_in_bytes\": 21697074}}\n" }, { "path": "lm_eval/datasets/hendrycks_math/hendrycks_math.py", "content": "# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"MATH dataset.\"\"\"\n\n\nimport json\nimport os\nimport pathlib\n\nimport datasets\n\n\n_CITATION = \"\"\"\\\n@article{hendrycksmath2021,\n title={Measuring Mathematical Problem Solving With the Math Dataset},\n author={Dan Hendrycks and Collin Burns and Saurav Kadavath and Akul Arora and Steven Basart and Eric Tang and Dawn Song and Jacob Steinhardt},\n journal={NeurIPS},\n year={2021}\n}\n\"\"\"\n\n_DESCRIPTION = \"\"\"\\\nMATH is a dataset of 12,500 challenging competition mathematics problems. Each\nproblem in Math has a full step-by-step solution which can be used to teach\nmodels to generate answer derivations and explanations.\n\"\"\"\n\n_HOMEPAGE = \"https://github.com/hendrycks/math\"\n\n# TODO: Add the licence for the dataset here if you can find it\n_LICENSE = \"\"\n\n_URLS = \"https://people.eecs.berkeley.edu/~hendrycks/MATH.tar\"\n\n_NAMES = [\n \"algebra\",\n \"counting_and_probability\",\n \"geometry\",\n \"intermediate_algebra\",\n \"number_theory\",\n \"prealgebra\",\n \"precalculus\",\n]\n\n\nclass HendrycksMath(datasets.GeneratorBasedBuilder):\n \"\"\"MATH is a dataset of 12,500 challenging competition mathematics problems.\"\"\"\n\n VERSION = datasets.Version(\"0.0.1\")\n\n BUILDER_CONFIGS = [\n datasets.BuilderConfig(name=name, version=version, description=name)\n for name, version in zip(_NAMES, [VERSION] * len(_NAMES))\n ]\n\n def _info(self):\n features = datasets.Features(\n {\n \"problem\": datasets.Value(\"string\"),\n \"level\": datasets.Value(\"string\"),\n \"type\": datasets.Value(\"string\"),\n \"solution\": datasets.Value(\"string\"),\n }\n )\n return datasets.DatasetInfo(\n description=_DESCRIPTION,\n features=features,\n homepage=_HOMEPAGE,\n license=_LICENSE,\n citation=_CITATION,\n )\n\n def _split_generators(self, dl_manager):\n urls = _URLS\n data_dir = dl_manager.download_and_extract(urls)\n return [\n datasets.SplitGenerator(\n name=datasets.Split.TRAIN,\n # These kwargs will be passed to _generate_examples\n gen_kwargs={\n \"basepath\": os.path.join(\n data_dir, \"MATH\", \"train\", self.config.name\n ),\n \"split\": \"train\",\n },\n ),\n datasets.SplitGenerator(\n name=datasets.Split.TEST,\n # These kwargs will be passed to _generate_examples\n gen_kwargs={\n \"basepath\": os.path.join(\n data_dir, \"MATH\", \"test\", self.config.name\n ),\n \"split\": \"test\",\n },\n ),\n ]\n\n # method parameters are unpacked from `gen_kwargs` as given in `_split_generators`\n def _generate_examples(self, basepath, split):\n key = 0\n for file in sorted(pathlib.Path(basepath).iterdir()):\n with open(file, \"r\", encoding=\"utf-8\") as f:\n data = json.load(f)\n yield key, {\n \"problem\": data[\"problem\"],\n \"level\": data[\"level\"],\n \"type\": data[\"type\"],\n \"solution\": data[\"solution\"],\n }\n key += 1\n" }, { "path": "lm_eval/datasets/logiqa/__init__.py", "content": "" }, { "path": "lm_eval/datasets/logiqa/dataset_infos.json", "content": "{\"logiqa\": {\"description\": \"LogiQA is a dataset for testing human logical reasoning. It consists of 8,678 QA\\ninstances, covering multiple types of deductive reasoning. Results show that state-\\nof-the-art neural models perform by far worse than human ceiling. The dataset can\\nalso serve as a benchmark for reinvestigating logical AI under the deep learning\\nNLP setting.\\n\", \"citation\": \"@misc{liu2020logiqa,\\n title={LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning}, \\n author={Jian Liu and Leyang Cui and Hanmeng Liu and Dandan Huang and Yile Wang and Yue Zhang},\\n year={2020},\\n eprint={2007.08124},\\n archivePrefix={arXiv},\\n primaryClass={cs.CL}\\n}\\n\", \"homepage\": \"https://github.com/lgw863/LogiQA-dataset\", \"license\": \"\", \"features\": {\"label\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"context\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"question\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"options\": {\"feature\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"length\": -1, \"id\": null, \"_type\": \"Sequence\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"logiqa\", \"config_name\": \"logiqa\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"train\": {\"name\": \"train\", \"num_bytes\": 6419852, \"num_examples\": 7376, \"dataset_name\": \"logiqa\"}, \"test\": {\"name\": \"test\", \"num_bytes\": 571705, \"num_examples\": 651, \"dataset_name\": \"logiqa\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 562437, \"num_examples\": 651, \"dataset_name\": \"logiqa\"}}, \"download_checksums\": {\"https://raw.githubusercontent.com/lgw863/LogiQA-dataset/master/Train.txt\": {\"num_bytes\": 6281272, \"checksum\": \"7d5bb1f58278e33b395744cd2ad8d7600faa0b3c4d615c659a44ec1181d759fa\"}, \"https://raw.githubusercontent.com/lgw863/LogiQA-dataset/master/Test.txt\": {\"num_bytes\": 559060, \"checksum\": \"359acb78c37802208f7fde9e2f6574b8526527c63d6a336f90a53f1932cb4701\"}, \"https://raw.githubusercontent.com/lgw863/LogiQA-dataset/master/Eval.txt\": {\"num_bytes\": 550021, \"checksum\": \"4c49e6753b7262c001506b9151135abf722247035ab075dad93acdea5789c01f\"}}, \"download_size\": 7390353, \"post_processing_size\": null, \"dataset_size\": 7553994, \"size_in_bytes\": 14944347}}\n" }, { "path": "lm_eval/datasets/logiqa/logiqa.py", "content": "# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"LogiQA dataset.\"\"\"\n\n\nimport datasets\n\n\n_CITATION = \"\"\"\\\n@misc{liu2020logiqa,\n title={LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning},\n author={Jian Liu and Leyang Cui and Hanmeng Liu and Dandan Huang and Yile Wang and Yue Zhang},\n year={2020},\n eprint={2007.08124},\n archivePrefix={arXiv},\n primaryClass={cs.CL}\n}\n\"\"\"\n\n_DESCRIPTION = \"\"\"\\\nLogiQA is a dataset for testing human logical reasoning. It consists of 8,678 QA\ninstances, covering multiple types of deductive reasoning. Results show that state-\nof-the-art neural models perform by far worse than human ceiling. The dataset can\nalso serve as a benchmark for reinvestigating logical AI under the deep learning\nNLP setting.\n\"\"\"\n\n_HOMEPAGE = \"https://github.com/lgw863/LogiQA-dataset\"\n\n# TODO: Add the licence for the dataset here if you can find it\n_LICENSE = \"\"\n\n_URLS = {\n \"train\": \"https://raw.githubusercontent.com/lgw863/LogiQA-dataset/master/Train.txt\",\n \"validation\": \"https://raw.githubusercontent.com/lgw863/LogiQA-dataset/master/Eval.txt\",\n \"test\": \"https://raw.githubusercontent.com/lgw863/LogiQA-dataset/master/Test.txt\",\n}\n\n\nclass Logiqa(datasets.GeneratorBasedBuilder):\n \"\"\"LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning\"\"\"\n\n VERSION = datasets.Version(\"0.0.1\")\n\n BUILDER_CONFIGS = [\n datasets.BuilderConfig(\n name=\"logiqa\", version=VERSION, description=\"The LogiQA dataset.\"\n ),\n ]\n\n def _info(self):\n features = datasets.Features(\n {\n \"label\": datasets.Value(\"string\"),\n \"context\": datasets.Value(\"string\"),\n \"question\": datasets.Value(\"string\"),\n \"options\": datasets.features.Sequence(datasets.Value(\"string\")),\n }\n )\n return datasets.DatasetInfo(\n description=_DESCRIPTION,\n features=features,\n homepage=_HOMEPAGE,\n license=_LICENSE,\n citation=_CITATION,\n )\n\n def _split_generators(self, dl_manager):\n urls = {\n \"train\": _URLS[\"train\"],\n \"test\": _URLS[\"test\"],\n \"validation\": _URLS[\"validation\"],\n }\n data_dir = dl_manager.download_and_extract(urls)\n return [\n datasets.SplitGenerator(\n name=datasets.Split.TRAIN,\n # These kwargs will be passed to _generate_examples\n gen_kwargs={\n \"filepath\": data_dir[\"train\"],\n \"split\": \"train\",\n },\n ),\n datasets.SplitGenerator(\n name=datasets.Split.TEST,\n # These kwargs will be passed to _generate_examples\n gen_kwargs={\"filepath\": data_dir[\"test\"], \"split\": \"test\"},\n ),\n datasets.SplitGenerator(\n name=datasets.Split.VALIDATION,\n # These kwargs will be passed to _generate_examples\n gen_kwargs={\n \"filepath\": data_dir[\"validation\"],\n \"split\": \"validation\",\n },\n ),\n ]\n\n # method parameters are unpacked from `gen_kwargs` as given in `_split_generators`\n def _generate_examples(self, filepath, split):\n def normalize(text):\n return text.replace(\".\", \". \").strip()\n\n with open(filepath, encoding=\"utf-8\") as f:\n data = f.read().strip().split(\"\\n\\n\")\n for key, row in enumerate(data):\n example = row.split(\"\\n\")\n yield key, {\n \"label\": example[0].strip(),\n \"context\": normalize(example[1]),\n \"question\": normalize(example[2]),\n \"options\": [normalize(option[2:]) for option in example[3:]],\n }\n" }, { "path": "lm_eval/datasets/mutual/__init__.py", "content": "" }, { "path": "lm_eval/datasets/mutual/dataset_infos.json", "content": "{\"mutual\": {\"description\": \"MuTual is a retrieval-based dataset for multi-turn dialogue reasoning, which is\\nmodified from Chinese high school English listening comprehension test data.\\n\\nThe MuTual dataset.\", \"citation\": \"@inproceedings{mutual,\\n title = \\\"MuTual: A Dataset for Multi-Turn Dialogue Reasoning\\\",\\n author = \\\"Cui, Leyang and Wu, Yu and Liu, Shujie and Zhang, Yue and Zhou, Ming\\\" ,\\n booktitle = \\\"Proceedings of the 58th Conference of the Association for Computational Linguistics\\\",\\n year = \\\"2020\\\",\\n publisher = \\\"Association for Computational Linguistics\\\",\\n}\\n\", \"homepage\": \"https://github.com/Nealcly/MuTual\", \"license\": \"\", \"features\": {\"answers\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"options\": {\"feature\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"length\": -1, \"id\": null, \"_type\": \"Sequence\"}, \"article\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"id\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"mutual\", \"config_name\": \"mutual\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"train\": {\"name\": \"train\", \"num_bytes\": 5141602, \"num_examples\": 7088, \"dataset_name\": \"mutual\"}, \"test\": {\"name\": \"test\", \"num_bytes\": 634396, \"num_examples\": 886, \"dataset_name\": \"mutual\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 624271, \"num_examples\": 886, \"dataset_name\": \"mutual\"}}, \"download_checksums\": {\"https://github.com/Nealcly/MuTual/archive/master.zip\": {\"num_bytes\": 10997878, \"checksum\": \"bb325cf6c672f0f02699993a37138b0fa0af6fcfc77ec81dfbe46add4d7b29f9\"}}, \"download_size\": 10997878, \"post_processing_size\": null, \"dataset_size\": 6400269, \"size_in_bytes\": 17398147}, \"mutual_plus\": {\"description\": \"MuTual is a retrieval-based dataset for multi-turn dialogue reasoning, which is\\nmodified from Chinese high school English listening comprehension test data.\\n\\nMuTualPlus is a more difficult MuTual that replaces positive responses with a safe responses.\", \"citation\": \"@inproceedings{mutual,\\n title = \\\"MuTual: A Dataset for Multi-Turn Dialogue Reasoning\\\",\\n author = \\\"Cui, Leyang and Wu, Yu and Liu, Shujie and Zhang, Yue and Zhou, Ming\\\" ,\\n booktitle = \\\"Proceedings of the 58th Conference of the Association for Computational Linguistics\\\",\\n year = \\\"2020\\\",\\n publisher = \\\"Association for Computational Linguistics\\\",\\n}\\n\", \"homepage\": \"https://github.com/Nealcly/MuTual\", \"license\": \"\", \"features\": {\"answers\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"options\": {\"feature\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"length\": -1, \"id\": null, \"_type\": \"Sequence\"}, \"article\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"id\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"mutual\", \"config_name\": \"mutual_plus\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"train\": {\"name\": \"train\", \"num_bytes\": 4921179, \"num_examples\": 7088, \"dataset_name\": \"mutual\"}, \"test\": {\"name\": \"test\", \"num_bytes\": 606620, \"num_examples\": 886, \"dataset_name\": \"mutual\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 597340, \"num_examples\": 886, \"dataset_name\": \"mutual\"}}, \"download_checksums\": {\"https://github.com/Nealcly/MuTual/archive/master.zip\": {\"num_bytes\": 10997878, \"checksum\": \"bb325cf6c672f0f02699993a37138b0fa0af6fcfc77ec81dfbe46add4d7b29f9\"}}, \"download_size\": 10997878, \"post_processing_size\": null, \"dataset_size\": 6125139, \"size_in_bytes\": 17123017}}\n" }, { "path": "lm_eval/datasets/mutual/mutual.py", "content": "# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"MuTual dataset.\"\"\"\n\n\nimport json\nimport os\nfrom pathlib import Path\n\nimport datasets\n\n\n_CITATION = \"\"\"\\\n@inproceedings{mutual,\n title = \"MuTual: A Dataset for Multi-Turn Dialogue Reasoning\",\n author = \"Cui, Leyang and Wu, Yu and Liu, Shujie and Zhang, Yue and Zhou, Ming\" ,\n booktitle = \"Proceedings of the 58th Conference of the Association for Computational Linguistics\",\n year = \"2020\",\n publisher = \"Association for Computational Linguistics\",\n}\n\"\"\"\n\n_DESCRIPTION = \"\"\"\\\nMuTual is a retrieval-based dataset for multi-turn dialogue reasoning, which is\nmodified from Chinese high school English listening comprehension test data.\n\"\"\"\n\n_HOMEPAGE = \"https://github.com/Nealcly/MuTual\"\n\n# TODO: Add the licence for the dataset here if you can find it\n_LICENSE = \"\"\n\n_URLS = \"https://github.com/Nealcly/MuTual/archive/master.zip\"\n\n\nclass Mutual(datasets.GeneratorBasedBuilder):\n \"\"\"MuTual: A Dataset for Multi-Turn Dialogue Reasoning\"\"\"\n\n VERSION = datasets.Version(\"0.0.1\")\n\n BUILDER_CONFIGS = [\n datasets.BuilderConfig(\n name=\"mutual\", version=VERSION, description=\"The MuTual dataset.\"\n ),\n datasets.BuilderConfig(\n name=\"mutual_plus\",\n version=VERSION,\n description=\"MuTualPlus is a more difficult MuTual that replaces positive responses with a safe responses.\",\n ),\n ]\n\n def _info(self):\n features = datasets.Features(\n {\n \"answers\": datasets.Value(\"string\"),\n \"options\": datasets.features.Sequence(datasets.Value(\"string\")),\n \"article\": datasets.Value(\"string\"),\n \"id\": datasets.Value(\"string\"),\n }\n )\n return datasets.DatasetInfo(\n description=f\"{_DESCRIPTION}\\n{self.config.description}\",\n features=features,\n homepage=_HOMEPAGE,\n license=_LICENSE,\n citation=_CITATION,\n )\n\n def _split_generators(self, dl_manager):\n urls = _URLS\n data_dir = dl_manager.download_and_extract(urls)\n return [\n datasets.SplitGenerator(\n name=datasets.Split.TRAIN,\n # These kwargs will be passed to _generate_examples\n gen_kwargs={\n \"basepath\": os.path.join(\n data_dir, \"MuTual-master\", \"data\", self.config.name, \"train\"\n ),\n \"split\": \"train\",\n },\n ),\n datasets.SplitGenerator(\n name=datasets.Split.TEST,\n # These kwargs will be passed to _generate_examples\n gen_kwargs={\n \"basepath\": os.path.join(\n data_dir, \"MuTual-master\", \"data\", self.config.name, \"test\"\n ),\n \"split\": \"test\",\n },\n ),\n datasets.SplitGenerator(\n name=datasets.Split.VALIDATION,\n # These kwargs will be passed to _generate_examples\n gen_kwargs={\n \"basepath\": os.path.join(\n data_dir, \"MuTual-master\", \"data\", self.config.name, \"dev\"\n ),\n \"split\": \"dev\",\n },\n ),\n ]\n\n # method parameters are unpacked from `gen_kwargs` as given in `_split_generators`\n def _generate_examples(self, basepath, split):\n # TODO: This method handles input defined in _split_generators to yield (key, example) tuples from the dataset.\n # The `key` is for legacy reasons (tfds) and is not important in itself, but must be unique for each example.\n key = 0\n for file in sorted(Path(basepath).iterdir()):\n if file.suffix != \".txt\":\n continue\n with open(file, \"r\", encoding=\"utf-8\") as f:\n data_str = f.read()\n # Ignore the occasional empty file.\n if not data_str:\n continue\n data = json.loads(data_str)\n yield key, {\n \"answers\": data[\"answers\"],\n \"options\": data[\"options\"],\n \"article\": data[\"article\"],\n \"id\": data[\"id\"],\n }\n key += 1\n" }, { "path": "lm_eval/datasets/pile/__init__.py", "content": "" }, { "path": "lm_eval/datasets/pile/dataset_infos.json", "content": "{\"pile_arxiv\": {\"description\": \"The Pile is a 825 GiB diverse, open source language modeling data set that consists\\nof 22 smaller, high-quality datasets combined together. To score well on Pile\\nBPB (bits per byte), a model must be able to understand many disparate domains\\nincluding books, github repositories, webpages, chat logs, and medical, physics,\\nmath, computer science, and philosophy papers.\\n\\nArXiv\", \"citation\": \"@article{pile,\\n title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},\\n author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},\\n journal={arXiv preprint arXiv:2101.00027},\\n year={2020}\\n}\\n\", \"homepage\": \"https://pile.eleuther.ai/\", \"license\": \"\", \"features\": {\"text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"pile\", \"config_name\": \"pile_arxiv\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"test\": {\"name\": \"test\", \"num_bytes\": 113218251, \"num_examples\": 2407, \"dataset_name\": \"pile\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 115653720, \"num_examples\": 2434, \"dataset_name\": \"pile\"}}, \"download_checksums\": {\"https://the-eye.eu/public/AI/pile/val.jsonl.zst\": {\"num_bytes\": 470907480, \"checksum\": \"264c875d8bbd355d8daa9d032b75fd8fb91606218bb84dd1155b203fcd5fab92\"}, \"https://the-eye.eu/public/AI/pile/test.jsonl.zst\": {\"num_bytes\": 460250856, \"checksum\": \"0bb28c52d0b5596d389bf179ce2d43bf7f7ffae76b0d2d20b180c97f62e0975e\"}}, \"download_size\": 931158336, \"post_processing_size\": null, \"dataset_size\": 228871971, \"size_in_bytes\": 1160030307}, \"pile_books3\": {\"description\": \"The Pile is a 825 GiB diverse, open source language modeling data set that consists\\nof 22 smaller, high-quality datasets combined together. To score well on Pile\\nBPB (bits per byte), a model must be able to understand many disparate domains\\nincluding books, github repositories, webpages, chat logs, and medical, physics,\\nmath, computer science, and philosophy papers.\\n\\nBooks3\", \"citation\": \"@article{pile,\\n title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},\\n author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},\\n journal={arXiv preprint arXiv:2101.00027},\\n year={2020}\\n}\\n\", \"homepage\": \"https://pile.eleuther.ai/\", \"license\": \"\", \"features\": {\"text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"pile\", \"config_name\": \"pile_books3\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"test\": {\"name\": \"test\", \"num_bytes\": 150095743, \"num_examples\": 269, \"dataset_name\": \"pile\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 177359876, \"num_examples\": 301, \"dataset_name\": \"pile\"}}, \"download_checksums\": {\"https://the-eye.eu/public/AI/pile/val.jsonl.zst\": {\"num_bytes\": 470907480, \"checksum\": \"264c875d8bbd355d8daa9d032b75fd8fb91606218bb84dd1155b203fcd5fab92\"}, \"https://the-eye.eu/public/AI/pile/test.jsonl.zst\": {\"num_bytes\": 460250856, \"checksum\": \"0bb28c52d0b5596d389bf179ce2d43bf7f7ffae76b0d2d20b180c97f62e0975e\"}}, \"download_size\": 931158336, \"post_processing_size\": null, \"dataset_size\": 327455619, \"size_in_bytes\": 1258613955}, \"pile_bookcorpus2\": {\"description\": \"The Pile is a 825 GiB diverse, open source language modeling data set that consists\\nof 22 smaller, high-quality datasets combined together. To score well on Pile\\nBPB (bits per byte), a model must be able to understand many disparate domains\\nincluding books, github repositories, webpages, chat logs, and medical, physics,\\nmath, computer science, and philosophy papers.\\n\\nBookCorpus2\", \"citation\": \"@article{pile,\\n title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},\\n author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},\\n journal={arXiv preprint arXiv:2101.00027},\\n year={2020}\\n}\\n\", \"homepage\": \"https://pile.eleuther.ai/\", \"license\": \"\", \"features\": {\"text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"pile\", \"config_name\": \"pile_bookcorpus2\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"test\": {\"name\": \"test\", \"num_bytes\": 9680652, \"num_examples\": 28, \"dataset_name\": \"pile\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 9776271, \"num_examples\": 26, \"dataset_name\": \"pile\"}}, \"download_checksums\": {\"https://the-eye.eu/public/AI/pile/val.jsonl.zst\": {\"num_bytes\": 470907480, \"checksum\": \"264c875d8bbd355d8daa9d032b75fd8fb91606218bb84dd1155b203fcd5fab92\"}, \"https://the-eye.eu/public/AI/pile/test.jsonl.zst\": {\"num_bytes\": 460250856, \"checksum\": \"0bb28c52d0b5596d389bf179ce2d43bf7f7ffae76b0d2d20b180c97f62e0975e\"}}, \"download_size\": 931158336, \"post_processing_size\": null, \"dataset_size\": 19456923, \"size_in_bytes\": 950615259}, \"pile_dm-mathematics\": {\"description\": \"The Pile is a 825 GiB diverse, open source language modeling data set that consists\\nof 22 smaller, high-quality datasets combined together. To score well on Pile\\nBPB (bits per byte), a model must be able to understand many disparate domains\\nincluding books, github repositories, webpages, chat logs, and medical, physics,\\nmath, computer science, and philosophy papers.\\n\\nDM Mathematics\", \"citation\": \"@article{pile,\\n title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},\\n author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},\\n journal={arXiv preprint arXiv:2101.00027},\\n year={2020}\\n}\\n\", \"homepage\": \"https://pile.eleuther.ai/\", \"license\": \"\", \"features\": {\"text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"pile\", \"config_name\": \"pile_dm-mathematics\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"test\": {\"name\": \"test\", \"num_bytes\": 15756556, \"num_examples\": 1922, \"dataset_name\": \"pile\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 16453386, \"num_examples\": 2007, \"dataset_name\": \"pile\"}}, \"download_checksums\": {\"https://the-eye.eu/public/AI/pile/val.jsonl.zst\": {\"num_bytes\": 470907480, \"checksum\": \"264c875d8bbd355d8daa9d032b75fd8fb91606218bb84dd1155b203fcd5fab92\"}, \"https://the-eye.eu/public/AI/pile/test.jsonl.zst\": {\"num_bytes\": 460250856, \"checksum\": \"0bb28c52d0b5596d389bf179ce2d43bf7f7ffae76b0d2d20b180c97f62e0975e\"}}, \"download_size\": 931158336, \"post_processing_size\": null, \"dataset_size\": 32209942, \"size_in_bytes\": 963368278}, \"pile_enron\": {\"description\": \"The Pile is a 825 GiB diverse, open source language modeling data set that consists\\nof 22 smaller, high-quality datasets combined together. To score well on Pile\\nBPB (bits per byte), a model must be able to understand many disparate domains\\nincluding books, github repositories, webpages, chat logs, and medical, physics,\\nmath, computer science, and philosophy papers.\\n\\nEnron Emails\", \"citation\": \"@article{pile,\\n title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},\\n author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},\\n journal={arXiv preprint arXiv:2101.00027},\\n year={2020}\\n}\\n\", \"homepage\": \"https://pile.eleuther.ai/\", \"license\": \"\", \"features\": {\"text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"pile\", \"config_name\": \"pile_enron\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"test\": {\"name\": \"test\", \"num_bytes\": 1638859, \"num_examples\": 1010, \"dataset_name\": \"pile\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 1556487, \"num_examples\": 947, \"dataset_name\": \"pile\"}}, \"download_checksums\": {\"https://the-eye.eu/public/AI/pile/val.jsonl.zst\": {\"num_bytes\": 470907480, \"checksum\": \"264c875d8bbd355d8daa9d032b75fd8fb91606218bb84dd1155b203fcd5fab92\"}, \"https://the-eye.eu/public/AI/pile/test.jsonl.zst\": {\"num_bytes\": 460250856, \"checksum\": \"0bb28c52d0b5596d389bf179ce2d43bf7f7ffae76b0d2d20b180c97f62e0975e\"}}, \"download_size\": 931158336, \"post_processing_size\": null, \"dataset_size\": 3195346, \"size_in_bytes\": 934353682}, \"pile_europarl\": {\"description\": \"The Pile is a 825 GiB diverse, open source language modeling data set that consists\\nof 22 smaller, high-quality datasets combined together. To score well on Pile\\nBPB (bits per byte), a model must be able to understand many disparate domains\\nincluding books, github repositories, webpages, chat logs, and medical, physics,\\nmath, computer science, and philosophy papers.\\n\\nEuroParl\", \"citation\": \"@article{pile,\\n title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},\\n author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},\\n journal={arXiv preprint arXiv:2101.00027},\\n year={2020}\\n}\\n\", \"homepage\": \"https://pile.eleuther.ai/\", \"license\": \"\", \"features\": {\"text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"pile\", \"config_name\": \"pile_europarl\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"test\": {\"name\": \"test\", \"num_bytes\": 8789652, \"num_examples\": 157, \"dataset_name\": \"pile\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 9111791, \"num_examples\": 133, \"dataset_name\": \"pile\"}}, \"download_checksums\": {\"https://the-eye.eu/public/AI/pile/val.jsonl.zst\": {\"num_bytes\": 470907480, \"checksum\": \"264c875d8bbd355d8daa9d032b75fd8fb91606218bb84dd1155b203fcd5fab92\"}, \"https://the-eye.eu/public/AI/pile/test.jsonl.zst\": {\"num_bytes\": 460250856, \"checksum\": \"0bb28c52d0b5596d389bf179ce2d43bf7f7ffae76b0d2d20b180c97f62e0975e\"}}, \"download_size\": 931158336, \"post_processing_size\": null, \"dataset_size\": 17901443, \"size_in_bytes\": 949059779}, \"pile_freelaw\": {\"description\": \"The Pile is a 825 GiB diverse, open source language modeling data set that consists\\nof 22 smaller, high-quality datasets combined together. To score well on Pile\\nBPB (bits per byte), a model must be able to understand many disparate domains\\nincluding books, github repositories, webpages, chat logs, and medical, physics,\\nmath, computer science, and philosophy papers.\\n\\nFreeLaw\", \"citation\": \"@article{pile,\\n title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},\\n author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},\\n journal={arXiv preprint arXiv:2101.00027},\\n year={2020}\\n}\\n\", \"homepage\": \"https://pile.eleuther.ai/\", \"license\": \"\", \"features\": {\"text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"pile\", \"config_name\": \"pile_freelaw\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"test\": {\"name\": \"test\", \"num_bytes\": 80808693, \"num_examples\": 5101, \"dataset_name\": \"pile\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 80363814, \"num_examples\": 5094, \"dataset_name\": \"pile\"}}, \"download_checksums\": {\"https://the-eye.eu/public/AI/pile/val.jsonl.zst\": {\"num_bytes\": 470907480, \"checksum\": \"264c875d8bbd355d8daa9d032b75fd8fb91606218bb84dd1155b203fcd5fab92\"}, \"https://the-eye.eu/public/AI/pile/test.jsonl.zst\": {\"num_bytes\": 460250856, \"checksum\": \"0bb28c52d0b5596d389bf179ce2d43bf7f7ffae76b0d2d20b180c97f62e0975e\"}}, \"download_size\": 931158336, \"post_processing_size\": null, \"dataset_size\": 161172507, \"size_in_bytes\": 1092330843}, \"pile_github\": {\"description\": \"The Pile is a 825 GiB diverse, open source language modeling data set that consists\\nof 22 smaller, high-quality datasets combined together. To score well on Pile\\nBPB (bits per byte), a model must be able to understand many disparate domains\\nincluding books, github repositories, webpages, chat logs, and medical, physics,\\nmath, computer science, and philosophy papers.\\n\\nGithub\", \"citation\": \"@article{pile,\\n title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},\\n author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},\\n journal={arXiv preprint arXiv:2101.00027},\\n year={2020}\\n}\\n\", \"homepage\": \"https://pile.eleuther.ai/\", \"license\": \"\", \"features\": {\"text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"pile\", \"config_name\": \"pile_github\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"test\": {\"name\": \"test\", \"num_bytes\": 95654706, \"num_examples\": 18195, \"dataset_name\": \"pile\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 97179576, \"num_examples\": 18337, \"dataset_name\": \"pile\"}}, \"download_checksums\": {\"https://the-eye.eu/public/AI/pile/val.jsonl.zst\": {\"num_bytes\": 470907480, \"checksum\": \"264c875d8bbd355d8daa9d032b75fd8fb91606218bb84dd1155b203fcd5fab92\"}, \"https://the-eye.eu/public/AI/pile/test.jsonl.zst\": {\"num_bytes\": 460250856, \"checksum\": \"0bb28c52d0b5596d389bf179ce2d43bf7f7ffae76b0d2d20b180c97f62e0975e\"}}, \"download_size\": 931158336, \"post_processing_size\": null, \"dataset_size\": 192834282, \"size_in_bytes\": 1123992618}, \"pile_gutenberg\": {\"description\": \"The Pile is a 825 GiB diverse, open source language modeling data set that consists\\nof 22 smaller, high-quality datasets combined together. To score well on Pile\\nBPB (bits per byte), a model must be able to understand many disparate domains\\nincluding books, github repositories, webpages, chat logs, and medical, physics,\\nmath, computer science, and philosophy papers.\\n\\nGutenberg (PG-19)\", \"citation\": \"@article{pile,\\n title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},\\n author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},\\n journal={arXiv preprint arXiv:2101.00027},\\n year={2020}\\n}\\n\", \"homepage\": \"https://pile.eleuther.ai/\", \"license\": \"\", \"features\": {\"text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"pile\", \"config_name\": \"pile_gutenberg\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"test\": {\"name\": \"test\", \"num_bytes\": 30243176, \"num_examples\": 80, \"dataset_name\": \"pile\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 24685980, \"num_examples\": 60, \"dataset_name\": \"pile\"}}, \"download_checksums\": {\"https://the-eye.eu/public/AI/pile/val.jsonl.zst\": {\"num_bytes\": 470907480, \"checksum\": \"264c875d8bbd355d8daa9d032b75fd8fb91606218bb84dd1155b203fcd5fab92\"}, \"https://the-eye.eu/public/AI/pile/test.jsonl.zst\": {\"num_bytes\": 460250856, \"checksum\": \"0bb28c52d0b5596d389bf179ce2d43bf7f7ffae76b0d2d20b180c97f62e0975e\"}}, \"download_size\": 931158336, \"post_processing_size\": null, \"dataset_size\": 54929156, \"size_in_bytes\": 986087492}, \"pile_hackernews\": {\"description\": \"The Pile is a 825 GiB diverse, open source language modeling data set that consists\\nof 22 smaller, high-quality datasets combined together. To score well on Pile\\nBPB (bits per byte), a model must be able to understand many disparate domains\\nincluding books, github repositories, webpages, chat logs, and medical, physics,\\nmath, computer science, and philosophy papers.\\n\\nHackerNews\", \"citation\": \"@article{pile,\\n title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},\\n author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},\\n journal={arXiv preprint arXiv:2101.00027},\\n year={2020}\\n}\\n\", \"homepage\": \"https://pile.eleuther.ai/\", \"license\": \"\", \"features\": {\"text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"pile\", \"config_name\": \"pile_hackernews\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"test\": {\"name\": \"test\", \"num_bytes\": 8124255, \"num_examples\": 1632, \"dataset_name\": \"pile\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 9803822, \"num_examples\": 1619, \"dataset_name\": \"pile\"}}, \"download_checksums\": {\"https://the-eye.eu/public/AI/pile/val.jsonl.zst\": {\"num_bytes\": 470907480, \"checksum\": \"264c875d8bbd355d8daa9d032b75fd8fb91606218bb84dd1155b203fcd5fab92\"}, \"https://the-eye.eu/public/AI/pile/test.jsonl.zst\": {\"num_bytes\": 460250856, \"checksum\": \"0bb28c52d0b5596d389bf179ce2d43bf7f7ffae76b0d2d20b180c97f62e0975e\"}}, \"download_size\": 931158336, \"post_processing_size\": null, \"dataset_size\": 17928077, \"size_in_bytes\": 949086413}, \"pile_nih-exporter\": {\"description\": \"The Pile is a 825 GiB diverse, open source language modeling data set that consists\\nof 22 smaller, high-quality datasets combined together. To score well on Pile\\nBPB (bits per byte), a model must be able to understand many disparate domains\\nincluding books, github repositories, webpages, chat logs, and medical, physics,\\nmath, computer science, and philosophy papers.\\n\\nNIH ExPorter\", \"citation\": \"@article{pile,\\n title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},\\n author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},\\n journal={arXiv preprint arXiv:2101.00027},\\n year={2020}\\n}\\n\", \"homepage\": \"https://pile.eleuther.ai/\", \"license\": \"\", \"features\": {\"text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"pile\", \"config_name\": \"pile_nih-exporter\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"test\": {\"name\": \"test\", \"num_bytes\": 3928804, \"num_examples\": 1884, \"dataset_name\": \"pile\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 3927967, \"num_examples\": 1825, \"dataset_name\": \"pile\"}}, \"download_checksums\": {\"https://the-eye.eu/public/AI/pile/val.jsonl.zst\": {\"num_bytes\": 470907480, \"checksum\": \"264c875d8bbd355d8daa9d032b75fd8fb91606218bb84dd1155b203fcd5fab92\"}, \"https://the-eye.eu/public/AI/pile/test.jsonl.zst\": {\"num_bytes\": 460250856, \"checksum\": \"0bb28c52d0b5596d389bf179ce2d43bf7f7ffae76b0d2d20b180c97f62e0975e\"}}, \"download_size\": 931158336, \"post_processing_size\": null, \"dataset_size\": 7856771, \"size_in_bytes\": 939015107}, \"pile_opensubtitles\": {\"description\": \"The Pile is a 825 GiB diverse, open source language modeling data set that consists\\nof 22 smaller, high-quality datasets combined together. To score well on Pile\\nBPB (bits per byte), a model must be able to understand many disparate domains\\nincluding books, github repositories, webpages, chat logs, and medical, physics,\\nmath, computer science, and philosophy papers.\\n\\nOpenSubtitles\", \"citation\": \"@article{pile,\\n title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},\\n author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},\\n journal={arXiv preprint arXiv:2101.00027},\\n year={2020}\\n}\\n\", \"homepage\": \"https://pile.eleuther.ai/\", \"license\": \"\", \"features\": {\"text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"pile\", \"config_name\": \"pile_opensubtitles\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"test\": {\"name\": \"test\", \"num_bytes\": 21008996, \"num_examples\": 642, \"dataset_name\": \"pile\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 19622904, \"num_examples\": 621, \"dataset_name\": \"pile\"}}, \"download_checksums\": {\"https://the-eye.eu/public/AI/pile/val.jsonl.zst\": {\"num_bytes\": 470907480, \"checksum\": \"264c875d8bbd355d8daa9d032b75fd8fb91606218bb84dd1155b203fcd5fab92\"}, \"https://the-eye.eu/public/AI/pile/test.jsonl.zst\": {\"num_bytes\": 460250856, \"checksum\": \"0bb28c52d0b5596d389bf179ce2d43bf7f7ffae76b0d2d20b180c97f62e0975e\"}}, \"download_size\": 931158336, \"post_processing_size\": null, \"dataset_size\": 40631900, \"size_in_bytes\": 971790236}, \"pile_openwebtext2\": {\"description\": \"The Pile is a 825 GiB diverse, open source language modeling data set that consists\\nof 22 smaller, high-quality datasets combined together. To score well on Pile\\nBPB (bits per byte), a model must be able to understand many disparate domains\\nincluding books, github repositories, webpages, chat logs, and medical, physics,\\nmath, computer science, and philosophy papers.\\n\\nOpenWebText2\", \"citation\": \"@article{pile,\\n title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},\\n author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},\\n journal={arXiv preprint arXiv:2101.00027},\\n year={2020}\\n}\\n\", \"homepage\": \"https://pile.eleuther.ai/\", \"license\": \"\", \"features\": {\"text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"pile\", \"config_name\": \"pile_openwebtext2\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"test\": {\"name\": \"test\", \"num_bytes\": 128624303, \"num_examples\": 32925, \"dataset_name\": \"pile\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 131554302, \"num_examples\": 33400, \"dataset_name\": \"pile\"}}, \"download_checksums\": {\"https://the-eye.eu/public/AI/pile/val.jsonl.zst\": {\"num_bytes\": 470907480, \"checksum\": \"264c875d8bbd355d8daa9d032b75fd8fb91606218bb84dd1155b203fcd5fab92\"}, \"https://the-eye.eu/public/AI/pile/test.jsonl.zst\": {\"num_bytes\": 460250856, \"checksum\": \"0bb28c52d0b5596d389bf179ce2d43bf7f7ffae76b0d2d20b180c97f62e0975e\"}}, \"download_size\": 931158336, \"post_processing_size\": null, \"dataset_size\": 260178605, \"size_in_bytes\": 1191336941}, \"pile_philpapers\": {\"description\": \"The Pile is a 825 GiB diverse, open source language modeling data set that consists\\nof 22 smaller, high-quality datasets combined together. To score well on Pile\\nBPB (bits per byte), a model must be able to understand many disparate domains\\nincluding books, github repositories, webpages, chat logs, and medical, physics,\\nmath, computer science, and philosophy papers.\\n\\nPhilPapers\", \"citation\": \"@article{pile,\\n title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},\\n author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},\\n journal={arXiv preprint arXiv:2101.00027},\\n year={2020}\\n}\\n\", \"homepage\": \"https://pile.eleuther.ai/\", \"license\": \"\", \"features\": {\"text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"pile\", \"config_name\": \"pile_philpapers\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"test\": {\"name\": \"test\", \"num_bytes\": 5090158, \"num_examples\": 68, \"dataset_name\": \"pile\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 6499078, \"num_examples\": 64, \"dataset_name\": \"pile\"}}, \"download_checksums\": {\"https://the-eye.eu/public/AI/pile/val.jsonl.zst\": {\"num_bytes\": 470907480, \"checksum\": \"264c875d8bbd355d8daa9d032b75fd8fb91606218bb84dd1155b203fcd5fab92\"}, \"https://the-eye.eu/public/AI/pile/test.jsonl.zst\": {\"num_bytes\": 460250856, \"checksum\": \"0bb28c52d0b5596d389bf179ce2d43bf7f7ffae76b0d2d20b180c97f62e0975e\"}}, \"download_size\": 931158336, \"post_processing_size\": null, \"dataset_size\": 11589236, \"size_in_bytes\": 942747572}, \"pile_pile-cc\": {\"description\": \"The Pile is a 825 GiB diverse, open source language modeling data set that consists\\nof 22 smaller, high-quality datasets combined together. To score well on Pile\\nBPB (bits per byte), a model must be able to understand many disparate domains\\nincluding books, github repositories, webpages, chat logs, and medical, physics,\\nmath, computer science, and philosophy papers.\\n\\nPile-CC\", \"citation\": \"@article{pile,\\n title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},\\n author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},\\n journal={arXiv preprint arXiv:2101.00027},\\n year={2020}\\n}\\n\", \"homepage\": \"https://pile.eleuther.ai/\", \"license\": \"\", \"features\": {\"text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"pile\", \"config_name\": \"pile_pile-cc\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"test\": {\"name\": \"test\", \"num_bytes\": 235004043, \"num_examples\": 52790, \"dataset_name\": \"pile\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 233535650, \"num_examples\": 52792, \"dataset_name\": \"pile\"}}, \"download_checksums\": {\"https://the-eye.eu/public/AI/pile/val.jsonl.zst\": {\"num_bytes\": 470907480, \"checksum\": \"264c875d8bbd355d8daa9d032b75fd8fb91606218bb84dd1155b203fcd5fab92\"}, \"https://the-eye.eu/public/AI/pile/test.jsonl.zst\": {\"num_bytes\": 460250856, \"checksum\": \"0bb28c52d0b5596d389bf179ce2d43bf7f7ffae76b0d2d20b180c97f62e0975e\"}}, \"download_size\": 931158336, \"post_processing_size\": null, \"dataset_size\": 468539693, \"size_in_bytes\": 1399698029}, \"pile_pubmed-abstracts\": {\"description\": \"The Pile is a 825 GiB diverse, open source language modeling data set that consists\\nof 22 smaller, high-quality datasets combined together. To score well on Pile\\nBPB (bits per byte), a model must be able to understand many disparate domains\\nincluding books, github repositories, webpages, chat logs, and medical, physics,\\nmath, computer science, and philosophy papers.\\n\\nPubMed Abstracts\", \"citation\": \"@article{pile,\\n title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},\\n author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},\\n journal={arXiv preprint arXiv:2101.00027},\\n year={2020}\\n}\\n\", \"homepage\": \"https://pile.eleuther.ai/\", \"license\": \"\", \"features\": {\"text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"pile\", \"config_name\": \"pile_pubmed-abstracts\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"test\": {\"name\": \"test\", \"num_bytes\": 39908950, \"num_examples\": 29895, \"dataset_name\": \"pile\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 40008336, \"num_examples\": 29871, \"dataset_name\": \"pile\"}}, \"download_checksums\": {\"https://the-eye.eu/public/AI/pile/val.jsonl.zst\": {\"num_bytes\": 470907480, \"checksum\": \"264c875d8bbd355d8daa9d032b75fd8fb91606218bb84dd1155b203fcd5fab92\"}, \"https://the-eye.eu/public/AI/pile/test.jsonl.zst\": {\"num_bytes\": 460250856, \"checksum\": \"0bb28c52d0b5596d389bf179ce2d43bf7f7ffae76b0d2d20b180c97f62e0975e\"}}, \"download_size\": 931158336, \"post_processing_size\": null, \"dataset_size\": 79917286, \"size_in_bytes\": 1011075622}, \"pile_pubmed-central\": {\"description\": \"The Pile is a 825 GiB diverse, open source language modeling data set that consists\\nof 22 smaller, high-quality datasets combined together. To score well on Pile\\nBPB (bits per byte), a model must be able to understand many disparate domains\\nincluding books, github repositories, webpages, chat logs, and medical, physics,\\nmath, computer science, and philosophy papers.\\n\\nPubMed Central\", \"citation\": \"@article{pile,\\n title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},\\n author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},\\n journal={arXiv preprint arXiv:2101.00027},\\n year={2020}\\n}\\n\", \"homepage\": \"https://pile.eleuther.ai/\", \"license\": \"\", \"features\": {\"text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"pile\", \"config_name\": \"pile_pubmed-central\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"test\": {\"name\": \"test\", \"num_bytes\": 187251519, \"num_examples\": 5911, \"dataset_name\": \"pile\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 184791818, \"num_examples\": 5977, \"dataset_name\": \"pile\"}}, \"download_checksums\": {\"https://the-eye.eu/public/AI/pile/val.jsonl.zst\": {\"num_bytes\": 470907480, \"checksum\": \"264c875d8bbd355d8daa9d032b75fd8fb91606218bb84dd1155b203fcd5fab92\"}, \"https://the-eye.eu/public/AI/pile/test.jsonl.zst\": {\"num_bytes\": 460250856, \"checksum\": \"0bb28c52d0b5596d389bf179ce2d43bf7f7ffae76b0d2d20b180c97f62e0975e\"}}, \"download_size\": 931158336, \"post_processing_size\": null, \"dataset_size\": 372043337, \"size_in_bytes\": 1303201673}, \"pile_stackexchange\": {\"description\": \"The Pile is a 825 GiB diverse, open source language modeling data set that consists\\nof 22 smaller, high-quality datasets combined together. To score well on Pile\\nBPB (bits per byte), a model must be able to understand many disparate domains\\nincluding books, github repositories, webpages, chat logs, and medical, physics,\\nmath, computer science, and philosophy papers.\\n\\nStackExchange\", \"citation\": \"@article{pile,\\n title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},\\n author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},\\n journal={arXiv preprint arXiv:2101.00027},\\n year={2020}\\n}\\n\", \"homepage\": \"https://pile.eleuther.ai/\", \"license\": \"\", \"features\": {\"text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"pile\", \"config_name\": \"pile_stackexchange\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"test\": {\"name\": \"test\", \"num_bytes\": 66441557, \"num_examples\": 30378, \"dataset_name\": \"pile\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 66011397, \"num_examples\": 29950, \"dataset_name\": \"pile\"}}, \"download_checksums\": {\"https://the-eye.eu/public/AI/pile/val.jsonl.zst\": {\"num_bytes\": 470907480, \"checksum\": \"264c875d8bbd355d8daa9d032b75fd8fb91606218bb84dd1155b203fcd5fab92\"}, \"https://the-eye.eu/public/AI/pile/test.jsonl.zst\": {\"num_bytes\": 460250856, \"checksum\": \"0bb28c52d0b5596d389bf179ce2d43bf7f7ffae76b0d2d20b180c97f62e0975e\"}}, \"download_size\": 931158336, \"post_processing_size\": null, \"dataset_size\": 132452954, \"size_in_bytes\": 1063611290}, \"pile_upsto\": {\"description\": \"The Pile is a 825 GiB diverse, open source language modeling data set that consists\\nof 22 smaller, high-quality datasets combined together. To score well on Pile\\nBPB (bits per byte), a model must be able to understand many disparate domains\\nincluding books, github repositories, webpages, chat logs, and medical, physics,\\nmath, computer science, and philosophy papers.\\n\\nUSPTO Backgrounds\", \"citation\": \"@article{pile,\\n title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},\\n author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},\\n journal={arXiv preprint arXiv:2101.00027},\\n year={2020}\\n}\\n\", \"homepage\": \"https://pile.eleuther.ai/\", \"license\": \"\", \"features\": {\"text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"pile\", \"config_name\": \"pile_upsto\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"test\": {\"name\": \"test\", \"num_bytes\": 47345405, \"num_examples\": 11415, \"dataset_name\": \"pile\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 48122320, \"num_examples\": 11387, \"dataset_name\": \"pile\"}}, \"download_checksums\": {\"https://the-eye.eu/public/AI/pile/val.jsonl.zst\": {\"num_bytes\": 470907480, \"checksum\": \"264c875d8bbd355d8daa9d032b75fd8fb91606218bb84dd1155b203fcd5fab92\"}, \"https://the-eye.eu/public/AI/pile/test.jsonl.zst\": {\"num_bytes\": 460250856, \"checksum\": \"0bb28c52d0b5596d389bf179ce2d43bf7f7ffae76b0d2d20b180c97f62e0975e\"}}, \"download_size\": 931158336, \"post_processing_size\": null, \"dataset_size\": 95467725, \"size_in_bytes\": 1026626061}, \"pile_ubuntu-irc\": {\"description\": \"The Pile is a 825 GiB diverse, open source language modeling data set that consists\\nof 22 smaller, high-quality datasets combined together. To score well on Pile\\nBPB (bits per byte), a model must be able to understand many disparate domains\\nincluding books, github repositories, webpages, chat logs, and medical, physics,\\nmath, computer science, and philosophy papers.\\n\\nUbuntu IRC\", \"citation\": \"@article{pile,\\n title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},\\n author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},\\n journal={arXiv preprint arXiv:2101.00027},\\n year={2020}\\n}\\n\", \"homepage\": \"https://pile.eleuther.ai/\", \"license\": \"\", \"features\": {\"text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"pile\", \"config_name\": \"pile_ubuntu-irc\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"test\": {\"name\": \"test\", \"num_bytes\": 5694218, \"num_examples\": 22, \"dataset_name\": \"pile\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 7410104, \"num_examples\": 21, \"dataset_name\": \"pile\"}}, \"download_checksums\": {\"https://the-eye.eu/public/AI/pile/val.jsonl.zst\": {\"num_bytes\": 470907480, \"checksum\": \"264c875d8bbd355d8daa9d032b75fd8fb91606218bb84dd1155b203fcd5fab92\"}, \"https://the-eye.eu/public/AI/pile/test.jsonl.zst\": {\"num_bytes\": 460250856, \"checksum\": \"0bb28c52d0b5596d389bf179ce2d43bf7f7ffae76b0d2d20b180c97f62e0975e\"}}, \"download_size\": 931158336, \"post_processing_size\": null, \"dataset_size\": 13104322, \"size_in_bytes\": 944262658}, \"pile_wikipedia\": {\"description\": \"The Pile is a 825 GiB diverse, open source language modeling data set that consists\\nof 22 smaller, high-quality datasets combined together. To score well on Pile\\nBPB (bits per byte), a model must be able to understand many disparate domains\\nincluding books, github repositories, webpages, chat logs, and medical, physics,\\nmath, computer science, and philosophy papers.\\n\\nWikipedia (en)\", \"citation\": \"@article{pile,\\n title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},\\n author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},\\n journal={arXiv preprint arXiv:2101.00027},\\n year={2020}\\n}\\n\", \"homepage\": \"https://pile.eleuther.ai/\", \"license\": \"\", \"features\": {\"text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"pile\", \"config_name\": \"pile_wikipedia\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"test\": {\"name\": \"test\", \"num_bytes\": 52166968, \"num_examples\": 17511, \"dataset_name\": \"pile\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 53186137, \"num_examples\": 17478, \"dataset_name\": \"pile\"}}, \"download_checksums\": {\"https://the-eye.eu/public/AI/pile/val.jsonl.zst\": {\"num_bytes\": 470907480, \"checksum\": \"264c875d8bbd355d8daa9d032b75fd8fb91606218bb84dd1155b203fcd5fab92\"}, \"https://the-eye.eu/public/AI/pile/test.jsonl.zst\": {\"num_bytes\": 460250856, \"checksum\": \"0bb28c52d0b5596d389bf179ce2d43bf7f7ffae76b0d2d20b180c97f62e0975e\"}}, \"download_size\": 931158336, \"post_processing_size\": null, \"dataset_size\": 105353105, \"size_in_bytes\": 1036511441}, \"pile_youtubesubtitles\": {\"description\": \"The Pile is a 825 GiB diverse, open source language modeling data set that consists\\nof 22 smaller, high-quality datasets combined together. To score well on Pile\\nBPB (bits per byte), a model must be able to understand many disparate domains\\nincluding books, github repositories, webpages, chat logs, and medical, physics,\\nmath, computer science, and philosophy papers.\\n\\nYoutubeSubtitles\", \"citation\": \"@article{pile,\\n title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},\\n author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},\\n journal={arXiv preprint arXiv:2101.00027},\\n year={2020}\\n}\\n\", \"homepage\": \"https://pile.eleuther.ai/\", \"license\": \"\", \"features\": {\"text\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"pile\", \"config_name\": \"pile_youtubesubtitles\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"test\": {\"name\": \"test\", \"num_bytes\": 7377448, \"num_examples\": 342, \"dataset_name\": \"pile\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 8937546, \"num_examples\": 326, \"dataset_name\": \"pile\"}}, \"download_checksums\": {\"https://the-eye.eu/public/AI/pile/val.jsonl.zst\": {\"num_bytes\": 470907480, \"checksum\": \"264c875d8bbd355d8daa9d032b75fd8fb91606218bb84dd1155b203fcd5fab92\"}, \"https://the-eye.eu/public/AI/pile/test.jsonl.zst\": {\"num_bytes\": 460250856, \"checksum\": \"0bb28c52d0b5596d389bf179ce2d43bf7f7ffae76b0d2d20b180c97f62e0975e\"}}, \"download_size\": 931158336, \"post_processing_size\": null, \"dataset_size\": 16314994, \"size_in_bytes\": 947473330}}\n" }, { "path": "lm_eval/datasets/pile/pile.py", "content": "# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"Pile dataset.\"\"\"\n\n\nimport json\n\nimport datasets\n\n\n_CITATION = \"\"\"\\\n@article{pile,\n title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},\n author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},\n journal={arXiv preprint arXiv:2101.00027},\n year={2020}\n}\n\"\"\"\n\n_DESCRIPTION = \"\"\"\\\nThe Pile is a 825 GiB diverse, open source language modeling data set that consists\nof 22 smaller, high-quality datasets combined together. To score well on Pile\nBPB (bits per byte), a model must be able to understand many disparate domains\nincluding books, github repositories, webpages, chat logs, and medical, physics,\nmath, computer science, and philosophy papers.\n\"\"\"\n\n_HOMEPAGE = \"https://pile.eleuther.ai/\"\n\n# TODO: Add the licence for the dataset here if you can find it\n_LICENSE = \"\"\n\n_URLS = {\n \"validation\": \"https://the-eye.eu/public/AI/pile/val.jsonl.zst\",\n \"test\": \"https://the-eye.eu/public/AI/pile/test.jsonl.zst\",\n}\n\n_NAMES = {\n \"pile_arxiv\": \"ArXiv\",\n \"pile_books3\": \"Books3\",\n \"pile_bookcorpus2\": \"BookCorpus2\",\n \"pile_dm-mathematics\": \"DM Mathematics\",\n \"pile_enron\": \"Enron Emails\",\n \"pile_europarl\": \"EuroParl\",\n \"pile_freelaw\": \"FreeLaw\",\n \"pile_github\": \"Github\",\n \"pile_gutenberg\": \"Gutenberg (PG-19)\",\n \"pile_hackernews\": \"HackerNews\",\n \"pile_nih-exporter\": \"NIH ExPorter\",\n \"pile_opensubtitles\": \"OpenSubtitles\",\n \"pile_openwebtext2\": \"OpenWebText2\",\n \"pile_philpapers\": \"PhilPapers\",\n \"pile_pile-cc\": \"Pile-CC\",\n \"pile_pubmed-abstracts\": \"PubMed Abstracts\",\n \"pile_pubmed-central\": \"PubMed Central\",\n \"pile_stackexchange\": \"StackExchange\",\n \"pile_upsto\": \"USPTO Backgrounds\",\n \"pile_ubuntu-irc\": \"Ubuntu IRC\",\n \"pile_wikipedia\": \"Wikipedia (en)\",\n \"pile_youtubesubtitles\": \"YoutubeSubtitles\",\n}\n\n\nclass Pile(datasets.GeneratorBasedBuilder):\n \"\"\"The Pile is a 825 GiB diverse, open source language modeling dataset.\"\"\"\n\n VERSION = datasets.Version(\"0.0.1\")\n\n BUILDER_CONFIGS = [\n datasets.BuilderConfig(name=name, version=version, description=_NAMES[name])\n for name, version in zip(_NAMES.keys(), [VERSION] * len(_NAMES))\n ]\n\n def _info(self):\n features = datasets.Features(\n {\n \"text\": datasets.Value(\"string\"),\n }\n )\n return datasets.DatasetInfo(\n description=f\"{_DESCRIPTION}\\n{self.config.description}\",\n features=features,\n homepage=_HOMEPAGE,\n license=_LICENSE,\n citation=_CITATION,\n )\n\n def _split_generators(self, dl_manager):\n urls = {\"validation\": _URLS[\"validation\"], \"test\": _URLS[\"test\"]}\n data_dir = dl_manager.download_and_extract(urls)\n return [\n datasets.SplitGenerator(\n name=datasets.Split.TEST,\n # These kwargs will be passed to _generate_examples\n gen_kwargs={\"filepath\": data_dir[\"test\"], \"split\": \"test\"},\n ),\n datasets.SplitGenerator(\n name=datasets.Split.VALIDATION,\n # These kwargs will be passed to _generate_examples\n gen_kwargs={\n \"filepath\": data_dir[\"validation\"],\n \"split\": \"validation\",\n },\n ),\n ]\n\n # method parameters are unpacked from `gen_kwargs` as given in `_split_generators`\n def _generate_examples(self, filepath, split):\n with open(filepath, encoding=\"utf-8\") as f:\n for key, row in enumerate(f):\n data = json.loads(row)\n if data[\"meta\"][\"pile_set_name\"] == _NAMES[self.config.name]:\n yield key, {\n \"text\": data[\"text\"],\n }\n" }, { "path": "lm_eval/datasets/quac/__init__.py", "content": "" }, { "path": "lm_eval/datasets/quac/dataset_infos.json", "content": "{\"quac\": {\"description\": \"Question Answering in Context (QuAC) is a dataset for modeling, understanding, and \\nparticipating in information seeking dialog. Data instances consist of an interactive\\ndialog between two crowd workers: (1) a student who poses a sequence of freeform\\nquestions to learn as much as possible about a hidden Wikipedia text, and (2)\\na teacher who answers the questions by providing short excerpts (spans) from the text.\\n\", \"citation\": \"@article{choi2018quac,\\n title={Quac: Question answering in context},\\n author={Choi, Eunsol and He, He and Iyyer, Mohit and Yatskar, Mark and Yih, Wen-tau and Choi, Yejin and Liang, Percy and Zettlemoyer, Luke},\\n journal={arXiv preprint arXiv:1808.07036},\\n year={2018}\\n}\\n\", \"homepage\": \"https://quac.ai/\", \"license\": \"\", \"features\": {\"title\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"section_title\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"paragraph\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"question\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"answer\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"quac\", \"config_name\": \"quac\", \"version\": {\"version_str\": \"1.1.0\", \"description\": null, \"major\": 1, \"minor\": 1, \"patch\": 0}, \"splits\": {\"train\": {\"name\": \"train\", \"num_bytes\": 212391958, \"num_examples\": 83568, \"dataset_name\": \"quac\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 20678483, \"num_examples\": 7354, \"dataset_name\": \"quac\"}}, \"download_checksums\": {\"https://s3.amazonaws.com/my89public/quac/train_v0.2.json\": {\"num_bytes\": 68114819, \"checksum\": \"ff5cca5a2e4b4d1cb5b5ced68b9fce88394ef6d93117426d6d4baafbcc05c56a\"}, \"https://s3.amazonaws.com/my89public/quac/val_v0.2.json\": {\"num_bytes\": 8929167, \"checksum\": \"09e622916280ba04c9352acb1bc5bbe80f11a2598f6f34e934c51d9e6570f378\"}}, \"download_size\": 77043986, \"post_processing_size\": null, \"dataset_size\": 233070441, \"size_in_bytes\": 310114427}}\n" }, { "path": "lm_eval/datasets/quac/quac.py", "content": "# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n# TODO: Address all TODOs and remove all explanatory comments\n\"\"\"QuAC dataset.\"\"\"\n\n\nimport json\n\nimport datasets\n\n\n_CITATION = \"\"\"\\\n@article{choi2018quac,\n title={Quac: Question answering in context},\n author={Choi, Eunsol and He, He and Iyyer, Mohit and Yatskar, Mark and Yih, Wen-tau and Choi, Yejin and Liang, Percy and Zettlemoyer, Luke},\n journal={arXiv preprint arXiv:1808.07036},\n year={2018}\n}\n\"\"\"\n\n_DESCRIPTION = \"\"\"\\\nQuestion Answering in Context (QuAC) is a dataset for modeling, understanding, and\nparticipating in information seeking dialog. Data instances consist of an interactive\ndialog between two crowd workers: (1) a student who poses a sequence of freeform\nquestions to learn as much as possible about a hidden Wikipedia text, and (2)\na teacher who answers the questions by providing short excerpts (spans) from the text.\n\"\"\"\n\n_HOMEPAGE = \"https://quac.ai/\"\n\n# TODO: Add the licence for the dataset here if you can find it\n_LICENSE = \"\"\n\n_URLS = {\n \"train\": \"https://s3.amazonaws.com/my89public/quac/train_v0.2.json\",\n \"validation\": \"https://s3.amazonaws.com/my89public/quac/val_v0.2.json\",\n}\n\n\nclass Quac(datasets.GeneratorBasedBuilder):\n \"\"\"Question Answering in Context (QuAC) is a dataset for modeling, understanding, and participating in information seeking dialog.\"\"\"\n\n VERSION = datasets.Version(\"1.1.0\")\n\n BUILDER_CONFIGS = [\n datasets.BuilderConfig(\n name=\"quac\", version=VERSION, description=\"The QuAC dataset\"\n ),\n ]\n\n def _info(self):\n features = datasets.Features(\n {\n \"title\": datasets.Value(\"string\"),\n \"section_title\": datasets.Value(\"string\"),\n \"paragraph\": datasets.Value(\"string\"),\n \"question\": datasets.Value(\"string\"),\n \"answer\": datasets.Value(\"string\"),\n }\n )\n return datasets.DatasetInfo(\n description=_DESCRIPTION,\n features=features,\n homepage=_HOMEPAGE,\n license=_LICENSE,\n citation=_CITATION,\n )\n\n def _split_generators(self, dl_manager):\n urls = {\"train\": _URLS[\"train\"], \"validation\": _URLS[\"validation\"]}\n data_dir = dl_manager.download_and_extract(urls)\n return [\n datasets.SplitGenerator(\n name=datasets.Split.TRAIN,\n # These kwargs will be passed to _generate_examples\n gen_kwargs={\n \"filepath\": data_dir[\"train\"],\n \"split\": \"train\",\n },\n ),\n datasets.SplitGenerator(\n name=datasets.Split.VALIDATION,\n # These kwargs will be passed to _generate_examples\n gen_kwargs={\"filepath\": data_dir[\"validation\"], \"split\": \"validation\"},\n ),\n ]\n\n # method parameters are unpacked from `gen_kwargs` as given in `_split_generators`\n def _generate_examples(self, filepath, split):\n with open(filepath, encoding=\"utf-8\") as f:\n data = json.load(f)[\"data\"]\n key = 0\n for row in data:\n paragraph = row[\"paragraphs\"][0][\"context\"].replace(\"CANNOTANSWER\", \"\")\n qas = row[\"paragraphs\"][0][\"qas\"]\n qa_pairs = [(qa[\"question\"], qa[\"answers\"][0][\"text\"]) for qa in qas]\n for (question, answer) in qa_pairs:\n # Yields examples as (key, example) tuples\n yield key, {\n \"title\": row[\"title\"],\n \"section_title\": row[\"section_title\"],\n \"paragraph\": paragraph,\n \"question\": question,\n \"answer\": answer,\n }\n key += 1\n" }, { "path": "lm_eval/datasets/sat_analogies/__init__.py", "content": "" }, { "path": "lm_eval/datasets/sat_analogies/sat_analogies.py", "content": "# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"SAT Analogy Questions dataset.\"\"\"\n\n\nimport os\n\nimport datasets\n\n\n_CITATION = \"\"\"\\\n@article{article,\n author = {Turney, Peter},\n year = {2006},\n month = {09},\n pages = {379-416},\n title = {Similarity of Semantic Relations},\n volume = {32},\n journal = {Computational Linguistics},\n doi = {10.1162/coli.2006.32.3.379}\n}\n\"\"\"\n\n_DESCRIPTION = \"\"\"\\\nSAT (Scholastic Aptitude Test) Analogy Questions is a dataset comprising 374\nmultiple-choice analogy questions; 5 choices per question.\n\"\"\"\n\n_HOMEPAGE = \"https://aclweb.org/aclwiki/SAT_Analogy_Questions_(State_of_the_art)\"\n\n# TODO: Add the licence for the dataset here if you can find it\n_LICENSE = \"\"\n\n\nclass SatAnalogies(datasets.GeneratorBasedBuilder):\n \"\"\"SAT (Scholastic Aptitude Test) Analogy Questions is a dataset comprising 374 multiple-choice analogy questions.\"\"\"\n\n VERSION = datasets.Version(\"0.0.1\")\n\n BUILDER_CONFIGS = [\n datasets.BuilderConfig(\n name=\"sat_analogies\",\n version=VERSION,\n description=\"The SAT Analogy Questions dataset\",\n ),\n ]\n\n @property\n def manual_download_instructions(self):\n return (\n \"To use SAT Analogy Questions you have to download it manually. Please \"\n \"email Peter Turney to request the data (https://www.apperceptual.com). \"\n \"Once you receive a download link for the dataset, supply the local path \"\n \"as the `data_dir` arg: \"\n \"`datasets.load_dataset('sat_analogies', data_dir='path/to/folder/folder_name')`\"\n )\n\n def _info(self):\n features = datasets.Features(\n {\n \"source\": datasets.Value(\"string\"),\n \"stem\": datasets.Value(\"string\"),\n \"choices\": datasets.features.Sequence(datasets.Value(\"string\")),\n \"solution\": datasets.Value(\"string\"),\n }\n )\n return datasets.DatasetInfo(\n description=_DESCRIPTION,\n features=features,\n homepage=_HOMEPAGE,\n license=_LICENSE,\n citation=_CITATION,\n )\n\n def _split_generators(self, dl_manager):\n data_dir = os.path.abspath(os.path.expanduser(dl_manager.manual_dir))\n if not os.path.exists(data_dir):\n raise FileNotFoundError(\n f\"{data_dir} does not exist. Make sure you insert a manual dir via `datasets.load_dataset('matinf', data_dir=...)` that includes SAT-package-V3.txt. Manual download instructions: {self.manual_download_instructions}\"\n )\n return [\n datasets.SplitGenerator(\n name=datasets.Split.VALIDATION,\n # These kwargs will be passed to _generate_examples\n gen_kwargs={\n \"filepath\": os.path.join(data_dir, \"SAT-package-V3.txt\"),\n },\n )\n ]\n\n # method parameters are unpacked from `gen_kwargs` as given in `_split_generators`\n def _generate_examples(self, filepath):\n data = []\n with open(filepath, \"r\", encoding=\"utf-8\") as f:\n record = []\n for line in f:\n line = line.strip()\n if len(line) == 0 and record:\n data.append(record)\n record = []\n elif len(line) > 0 and line[0] == \"#\":\n # Skip comments.\n continue\n else:\n record.append(line)\n data.append(record)\n for key, record in enumerate(data):\n source = record[-8]\n stem = record[-7]\n choices = record[-6:-1]\n solution = record[-1]\n yield key, {\n \"source\": source,\n \"stem\": stem,\n \"choices\": choices,\n \"solution\": solution,\n }\n" }, { "path": "lm_eval/datasets/triviaqa/README.md", "content": "---\ndataset_info:\n features:\n - name: question_id\n dtype: string\n - name: question_source\n dtype: string\n - name: question\n dtype: string\n - name: answer\n struct:\n - name: aliases\n sequence: string\n - name: value\n dtype: string\n - name: search_results\n sequence:\n - name: description\n dtype: string\n - name: filename\n dtype: string\n - name: rank\n dtype: int32\n - name: title\n dtype: string\n - name: url\n dtype: string\n - name: search_context\n dtype: string\n config_name: triviaqa\n splits:\n - name: train\n num_bytes: 1270894387\n num_examples: 87622\n - name: validation\n num_bytes: 163755044\n num_examples: 11313\n download_size: 632549060\n dataset_size: 1434649431\n---\n" }, { "path": "lm_eval/datasets/triviaqa/__init__.py", "content": "" }, { "path": "lm_eval/datasets/triviaqa/dataset_infos.json", "content": "{\"triviaqa\": {\"description\": \"TriviaQA is a reading comprehension dataset containing over 650K question-answer-evidence\\ntriples. TriviaQA includes 95K question-answer pairs authored by trivia enthusiasts\\nand independently gathered evidence documents, six per question on average, that provide\\nhigh quality distant supervision for answering the questions.\\n\", \"citation\": \"@InProceedings{JoshiTriviaQA2017,\\n author = {Joshi, Mandar and Choi, Eunsol and Weld, Daniel S. and Zettlemoyer, Luke},\\n title = {TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension},\\n booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics},\\n month = {July},\\n year = {2017},\\n address = {Vancouver, Canada},\\n publisher = {Association for Computational Linguistics},\\n}\\n\", \"homepage\": \"https://nlp.cs.washington.edu/triviaqa/\", \"license\": \"Apache License 2.0\", \"features\": {\"question_id\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"question_source\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"question\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"answer\": {\"aliases\": {\"feature\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"length\": -1, \"id\": null, \"_type\": \"Sequence\"}, \"value\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"search_results\": {\"feature\": {\"description\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"filename\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"rank\": {\"dtype\": \"int32\", \"id\": null, \"_type\": \"Value\"}, \"title\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"url\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"search_context\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"length\": -1, \"id\": null, \"_type\": \"Sequence\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"triviaqa\", \"config_name\": \"triviaqa\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"train\": {\"name\": \"train\", \"num_bytes\": 1271393601, \"num_examples\": 87622, \"dataset_name\": \"triviaqa\"}, \"validation\": {\"name\": \"validation\", \"num_bytes\": 163819509, \"num_examples\": 11313, \"dataset_name\": \"triviaqa\"}}, \"download_checksums\": {\"http://eaidata.bmk.sh/data/triviaqa-unfiltered.tar.gz\": {\"num_bytes\": 546481381, \"checksum\": \"adc19b42769062d241a8fbe834c56e58598d9322eb6c614e9f33a68a2cf5523e\"}}, \"download_size\": 546481381, \"post_processing_size\": null, \"dataset_size\": 1435213110, \"size_in_bytes\": 1981694491}}\n" }, { "path": "lm_eval/datasets/triviaqa/triviaqa.py", "content": "# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n#\n# Custom TriviaQA because HF version sanitizes the dataset differently.\n# https://github.com/huggingface/datasets/blob/9977ade72191ff0b6907ec63935448c6269a91a1/datasets/trivia_qa/trivia_qa.py#L285\n\"\"\"TriviaQA (Unfiltered Raw) dataset.\"\"\"\n\n\nimport json\nimport os\n\nimport datasets\n\n\n_CITATION = \"\"\"\\\n@InProceedings{JoshiTriviaQA2017,\n author = {Joshi, Mandar and Choi, Eunsol and Weld, Daniel S. and Zettlemoyer, Luke},\n title = {TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension},\n booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics},\n month = {July},\n year = {2017},\n address = {Vancouver, Canada},\n publisher = {Association for Computational Linguistics},\n}\n\"\"\"\n\n_DESCRIPTION = \"\"\"\\\nTriviaQA is a reading comprehension dataset containing over 650K question-answer-evidence\ntriples. TriviaQA includes 95K question-answer pairs authored by trivia enthusiasts\nand independently gathered evidence documents, six per question on average, that provide\nhigh quality distant supervision for answering the questions.\n\"\"\"\n\n_HOMEPAGE = \"https://nlp.cs.washington.edu/triviaqa/\"\n\n_LICENSE = \"Apache License 2.0\"\n\n_URLS = \"https://nlp.cs.washington.edu/triviaqa/data/triviaqa-unfiltered.tar.gz\"\n\n\nclass Triviaqa(datasets.GeneratorBasedBuilder):\n \"\"\"TriviaQA is a reading comprehension dataset containing over 650K question-answer-evidence triples\"\"\"\n\n VERSION = datasets.Version(\"0.0.2\")\n\n BUILDER_CONFIGS = [\n datasets.BuilderConfig(\n name=\"triviaqa\", version=VERSION, description=\"The TriviaQA dataset\"\n ),\n ]\n\n def _info(self):\n features = datasets.Features(\n {\n \"question_id\": datasets.Value(\"string\"),\n \"question_source\": datasets.Value(\"string\"),\n \"question\": datasets.Value(\"string\"),\n \"answer\": {\n \"aliases\": datasets.features.Sequence(\n datasets.Value(\"string\"),\n ),\n \"value\": datasets.Value(\"string\"),\n },\n \"search_results\": datasets.features.Sequence(\n {\n \"description\": datasets.Value(\"string\"),\n \"filename\": datasets.Value(\"string\"),\n \"rank\": datasets.Value(\"int32\"),\n \"title\": datasets.Value(\"string\"),\n \"url\": datasets.Value(\"string\"),\n \"search_context\": datasets.Value(\"string\"),\n }\n ),\n }\n )\n return datasets.DatasetInfo(\n description=_DESCRIPTION,\n features=features,\n homepage=_HOMEPAGE,\n license=_LICENSE,\n citation=_CITATION,\n )\n\n def _split_generators(self, dl_manager):\n urls = _URLS\n data_dir = dl_manager.download_and_extract(urls)\n return [\n datasets.SplitGenerator(\n name=datasets.Split.TRAIN,\n # These kwargs will be passed to _generate_examples\n gen_kwargs={\n \"filepath\": os.path.join(\n data_dir, \"triviaqa-unfiltered\", \"unfiltered-web-train.json\"\n ),\n },\n ),\n datasets.SplitGenerator(\n name=datasets.Split.VALIDATION,\n # These kwargs will be passed to _generate_examples\n gen_kwargs={\n \"filepath\": os.path.join(\n data_dir, \"triviaqa-unfiltered\", \"unfiltered-web-dev.json\"\n ),\n },\n ),\n ]\n\n # method parameters are unpacked from `gen_kwargs` as given in `_split_generators`\n def _generate_examples(self, filepath):\n with open(filepath, encoding=\"utf-8\") as f:\n json_data = json.load(f)[\"Data\"]\n for key, data in enumerate(json_data):\n search_results = []\n for search_result in data[\"SearchResults\"]:\n search_results.append(\n {\n \"description\": search_result[\"Description\"]\n if \"Description\" in search_result\n else \"\",\n \"filename\": search_result[\"Filename\"]\n if \"Filename\" in search_result\n else \"\",\n \"rank\": search_result[\"Rank\"]\n if \"Rank\" in search_result\n else -1,\n \"title\": search_result[\"Title\"]\n if \"Title\" in search_result\n else \"\",\n \"url\": search_result[\"Url\"]\n if \"Url\" in search_result\n else \"\",\n \"search_context\": search_result[\"SearchContext\"]\n if \"SearchContext\" in search_result\n else \"\",\n }\n )\n yield key, {\n \"question_id\": data[\"QuestionId\"],\n \"question_source\": data[\"QuestionSource\"],\n \"question\": data[\"Question\"],\n \"answer\": {\n \"aliases\": data[\"Answer\"][\"Aliases\"],\n \"value\": data[\"Answer\"][\"Value\"],\n },\n \"search_results\": search_results,\n }\n" }, { "path": "lm_eval/datasets/unscramble/__init__.py", "content": "" }, { "path": "lm_eval/datasets/unscramble/dataset_infos.json", "content": "{\"mid_word_1_anagrams\": {\"description\": \"Unscramble is a small battery of 5 \\u201ccharacter manipulation\\u201d tasks. Each task\\ninvolves giving the model a word distorted by some combination of scrambling,\\naddition, or deletion of characters, and asking it to recover the original word.\\n\", \"citation\": \"@inproceedings{NEURIPS2020_1457c0d6,\\n author = {Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom and Child, Rewon and Ramesh, Aditya and Ziegler, Daniel and Wu, Jeffrey and Winter, Clemens and Hesse, Chris and Chen, Mark and Sigler, Eric and Litwin, Mateusz and Gray, Scott and Chess, Benjamin and Clark, Jack and Berner, Christopher and McCandlish, Sam and Radford, Alec and Sutskever, Ilya and Amodei, Dario},\\n booktitle = {Advances in Neural Information Processing Systems},\\n editor = {H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin},\\n pages = {1877--1901},\\n publisher = {Curran Associates, Inc.},\\n title = {Language Models are Few-Shot Learners},\\n url = {https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf},\\n volume = {33},\\n year = {2020}\\n}\\n\", \"homepage\": \"https://github.com/openai/gpt-3/tree/master/data\", \"license\": \"\", \"features\": {\"context\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"completion\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"unscramble\", \"config_name\": \"mid_word_1_anagrams\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"validation\": {\"name\": \"validation\", \"num_bytes\": 271516, \"num_examples\": 10000, \"dataset_name\": \"unscramble\"}}, \"download_checksums\": {\"https://raw.githubusercontent.com/openai/gpt-3/master/data/mid_word_1_anagrams.jsonl.gz\": {\"num_bytes\": 106533, \"checksum\": \"6768a86896083199de4815d4964cb2f6f1046476cfd80c2a562784f182905979\"}}, \"download_size\": 106533, \"post_processing_size\": null, \"dataset_size\": 271516, \"size_in_bytes\": 378049}, \"mid_word_2_anagrams\": {\"description\": \"Unscramble is a small battery of 5 \\u201ccharacter manipulation\\u201d tasks. Each task\\ninvolves giving the model a word distorted by some combination of scrambling,\\naddition, or deletion of characters, and asking it to recover the original word.\\n\", \"citation\": \"@inproceedings{NEURIPS2020_1457c0d6,\\n author = {Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom and Child, Rewon and Ramesh, Aditya and Ziegler, Daniel and Wu, Jeffrey and Winter, Clemens and Hesse, Chris and Chen, Mark and Sigler, Eric and Litwin, Mateusz and Gray, Scott and Chess, Benjamin and Clark, Jack and Berner, Christopher and McCandlish, Sam and Radford, Alec and Sutskever, Ilya and Amodei, Dario},\\n booktitle = {Advances in Neural Information Processing Systems},\\n editor = {H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin},\\n pages = {1877--1901},\\n publisher = {Curran Associates, Inc.},\\n title = {Language Models are Few-Shot Learners},\\n url = {https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf},\\n volume = {33},\\n year = {2020}\\n}\\n\", \"homepage\": \"https://github.com/openai/gpt-3/tree/master/data\", \"license\": \"\", \"features\": {\"context\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"completion\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"unscramble\", \"config_name\": \"mid_word_2_anagrams\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"validation\": {\"name\": \"validation\", \"num_bytes\": 282654, \"num_examples\": 10000, \"dataset_name\": \"unscramble\"}}, \"download_checksums\": {\"https://raw.githubusercontent.com/openai/gpt-3/master/data/mid_word_2_anagrams.jsonl.gz\": {\"num_bytes\": 109091, \"checksum\": \"c3d839d09a7954b78a27cd2cd75d4ed0488656c56ef4dbd741a005343826cb01\"}}, \"download_size\": 109091, \"post_processing_size\": null, \"dataset_size\": 282654, \"size_in_bytes\": 391745}, \"cycle_letters_in_word\": {\"description\": \"Unscramble is a small battery of 5 \\u201ccharacter manipulation\\u201d tasks. Each task\\ninvolves giving the model a word distorted by some combination of scrambling,\\naddition, or deletion of characters, and asking it to recover the original word.\\n\", \"citation\": \"@inproceedings{NEURIPS2020_1457c0d6,\\n author = {Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom and Child, Rewon and Ramesh, Aditya and Ziegler, Daniel and Wu, Jeffrey and Winter, Clemens and Hesse, Chris and Chen, Mark and Sigler, Eric and Litwin, Mateusz and Gray, Scott and Chess, Benjamin and Clark, Jack and Berner, Christopher and McCandlish, Sam and Radford, Alec and Sutskever, Ilya and Amodei, Dario},\\n booktitle = {Advances in Neural Information Processing Systems},\\n editor = {H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin},\\n pages = {1877--1901},\\n publisher = {Curran Associates, Inc.},\\n title = {Language Models are Few-Shot Learners},\\n url = {https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf},\\n volume = {33},\\n year = {2020}\\n}\\n\", \"homepage\": \"https://github.com/openai/gpt-3/tree/master/data\", \"license\": \"\", \"features\": {\"context\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"completion\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"unscramble\", \"config_name\": \"cycle_letters_in_word\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"validation\": {\"name\": \"validation\", \"num_bytes\": 282654, \"num_examples\": 10000, \"dataset_name\": \"unscramble\"}}, \"download_checksums\": {\"https://raw.githubusercontent.com/openai/gpt-3/master/data/cycle_letters_in_word.jsonl.gz\": {\"num_bytes\": 98451, \"checksum\": \"1689c9002bb8c5988bf5f05e977c9db92f57932c1b5a38998c29ac0dd71e1d42\"}}, \"download_size\": 98451, \"post_processing_size\": null, \"dataset_size\": 282654, \"size_in_bytes\": 381105}, \"random_insertion_in_word\": {\"description\": \"Unscramble is a small battery of 5 \\u201ccharacter manipulation\\u201d tasks. Each task\\ninvolves giving the model a word distorted by some combination of scrambling,\\naddition, or deletion of characters, and asking it to recover the original word.\\n\", \"citation\": \"@inproceedings{NEURIPS2020_1457c0d6,\\n author = {Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom and Child, Rewon and Ramesh, Aditya and Ziegler, Daniel and Wu, Jeffrey and Winter, Clemens and Hesse, Chris and Chen, Mark and Sigler, Eric and Litwin, Mateusz and Gray, Scott and Chess, Benjamin and Clark, Jack and Berner, Christopher and McCandlish, Sam and Radford, Alec and Sutskever, Ilya and Amodei, Dario},\\n booktitle = {Advances in Neural Information Processing Systems},\\n editor = {H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin},\\n pages = {1877--1901},\\n publisher = {Curran Associates, Inc.},\\n title = {Language Models are Few-Shot Learners},\\n url = {https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf},\\n volume = {33},\\n year = {2020}\\n}\\n\", \"homepage\": \"https://github.com/openai/gpt-3/tree/master/data\", \"license\": \"\", \"features\": {\"context\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"completion\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"unscramble\", \"config_name\": \"random_insertion_in_word\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"validation\": {\"name\": \"validation\", \"num_bytes\": 353981, \"num_examples\": 10000, \"dataset_name\": \"unscramble\"}}, \"download_checksums\": {\"https://raw.githubusercontent.com/openai/gpt-3/master/data/random_insertion_in_word.jsonl.gz\": {\"num_bytes\": 143626, \"checksum\": \"72e65d83da53d15752ee0c47379509de149ddbad32d61184e5991df29616b78a\"}}, \"download_size\": 143626, \"post_processing_size\": null, \"dataset_size\": 353981, \"size_in_bytes\": 497607}, \"reversed_words\": {\"description\": \"Unscramble is a small battery of 5 \\u201ccharacter manipulation\\u201d tasks. Each task\\ninvolves giving the model a word distorted by some combination of scrambling,\\naddition, or deletion of characters, and asking it to recover the original word.\\n\", \"citation\": \"@inproceedings{NEURIPS2020_1457c0d6,\\n author = {Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom and Child, Rewon and Ramesh, Aditya and Ziegler, Daniel and Wu, Jeffrey and Winter, Clemens and Hesse, Chris and Chen, Mark and Sigler, Eric and Litwin, Mateusz and Gray, Scott and Chess, Benjamin and Clark, Jack and Berner, Christopher and McCandlish, Sam and Radford, Alec and Sutskever, Ilya and Amodei, Dario},\\n booktitle = {Advances in Neural Information Processing Systems},\\n editor = {H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin},\\n pages = {1877--1901},\\n publisher = {Curran Associates, Inc.},\\n title = {Language Models are Few-Shot Learners},\\n url = {https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf},\\n volume = {33},\\n year = {2020}\\n}\\n\", \"homepage\": \"https://github.com/openai/gpt-3/tree/master/data\", \"license\": \"\", \"features\": {\"context\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}, \"completion\": {\"dtype\": \"string\", \"id\": null, \"_type\": \"Value\"}}, \"post_processed\": null, \"supervised_keys\": null, \"task_templates\": null, \"builder_name\": \"unscramble\", \"config_name\": \"reversed_words\", \"version\": {\"version_str\": \"0.0.1\", \"description\": null, \"major\": 0, \"minor\": 0, \"patch\": 1}, \"splits\": {\"validation\": {\"name\": \"validation\", \"num_bytes\": 282654, \"num_examples\": 10000, \"dataset_name\": \"unscramble\"}}, \"download_checksums\": {\"https://raw.githubusercontent.com/openai/gpt-3/master/data/reversed_words.jsonl.gz\": {\"num_bytes\": 91917, \"checksum\": \"133a08f875cd6c1ef8608a3233571a773881cc27b1c707de738cc6543439332a\"}}, \"download_size\": 91917, \"post_processing_size\": null, \"dataset_size\": 282654, \"size_in_bytes\": 374571}}\n" }, { "path": "lm_eval/datasets/unscramble/unscramble.py", "content": "# Copyright 2020 The HuggingFace Datasets Authors and the current dataset script contributor.\n#\n# Licensed under the Apache License, Version 2.0 (the \"License\");\n# you may not use this file except in compliance with the License.\n# You may obtain a copy of the License at\n#\n# http://www.apache.org/licenses/LICENSE-2.0\n#\n# Unless required by applicable law or agreed to in writing, software\n# distributed under the License is distributed on an \"AS IS\" BASIS,\n# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n# See the License for the specific language governing permissions and\n# limitations under the License.\n\"\"\"Unscramble dataset.\"\"\"\n\n\nimport json\nimport os\n\nimport datasets\n\n\n_CITATION = \"\"\"\\\n@inproceedings{NEURIPS2020_1457c0d6,\n author = {Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom and Child, Rewon and Ramesh, Aditya and Ziegler, Daniel and Wu, Jeffrey and Winter, Clemens and Hesse, Chris and Chen, Mark and Sigler, Eric and Litwin, Mateusz and Gray, Scott and Chess, Benjamin and Clark, Jack and Berner, Christopher and McCandlish, Sam and Radford, Alec and Sutskever, Ilya and Amodei, Dario},\n booktitle = {Advances in Neural Information Processing Systems},\n editor = {H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin},\n pages = {1877--1901},\n publisher = {Curran Associates, Inc.},\n title = {Language Models are Few-Shot Learners},\n url = {https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf},\n volume = {33},\n year = {2020}\n}\n\"\"\"\n\n_DESCRIPTION = \"\"\"\\\nUnscramble is a small battery of 5 “character manipulation” tasks. Each task\ninvolves giving the model a word distorted by some combination of scrambling,\naddition, or deletion of characters, and asking it to recover the original word.\n\"\"\"\n\n_HOMEPAGE = \"https://github.com/openai/gpt-3/tree/master/data\"\n\n# TODO: Add the licence for the dataset here if you can find it\n_LICENSE = \"\"\n\n_BASE_URL = \"https://raw.githubusercontent.com/openai/gpt-3/master/data\"\n\n\n_DESCRIPTIONS = {\n \"mid_word_1_anagrams\": \"Anagrams of all but the first and last letter.\",\n \"mid_word_2_anagrams\": \"Anagrams of all but the first and last 2 letters.\",\n \"cycle_letters_in_word\": \"Cycle letters in the word.\",\n \"random_insertion_in_word\": \"Random insertions in the word that must be removed.\",\n \"reversed_words\": \"Words spelled backwards that must be reversed.\",\n}\n_NAMES = _DESCRIPTIONS.keys()\n\n\nclass Unscramble(datasets.GeneratorBasedBuilder):\n \"\"\"Unscramble is a small battery of 5 “character manipulation” tasks.\"\"\"\n\n VERSION = datasets.Version(\"0.0.1\")\n\n BUILDER_CONFIGS = [\n datasets.BuilderConfig(\n name=name, version=version, description=_DESCRIPTIONS[name]\n )\n for name, version in zip(_NAMES, [VERSION] * len(_NAMES))\n ]\n\n def _info(self):\n features = datasets.Features(\n {\n \"context\": datasets.Value(\"string\"),\n \"completion\": datasets.Value(\"string\"),\n }\n )\n return datasets.DatasetInfo(\n description=_DESCRIPTION,\n features=features,\n homepage=_HOMEPAGE,\n license=_LICENSE,\n citation=_CITATION,\n )\n\n def _split_generators(self, dl_manager):\n urls = os.path.join(_BASE_URL, f\"{self.config.name}.jsonl.gz\")\n data_dir = dl_manager.download_and_extract(urls)\n return [\n datasets.SplitGenerator(\n name=datasets.Split.VALIDATION,\n # These kwargs will be passed to _generate_examples\n gen_kwargs={\n \"filepath\": data_dir,\n \"split\": \"validation\",\n },\n ),\n ]\n\n # method parameters are unpacked from `gen_kwargs` as given in `_split_generators`\n def _generate_examples(self, filepath, split):\n with open(filepath, encoding=\"utf-8\") as f:\n for key, row in enumerate(f):\n data = json.loads(row)\n yield key, {\n \"context\": data[\"context\"],\n \"completion\": data[\"completion\"],\n }\n" }, { "path": "lm_eval/decontamination/__init__.py", "content": "" }, { "path": "lm_eval/decontamination/archiver.py", "content": "import os\nimport zstandard\nimport json\nimport jsonlines\nimport io\nimport datetime\nimport mmap\nimport tqdm\nfrom pathlib import Path\n\n\ndef json_serial(obj):\n \"\"\"JSON serializer for objects not serializable by default json code\"\"\"\n\n if isinstance(obj, (datetime.datetime,)):\n return obj.isoformat()\n raise TypeError(\"Type %s not serializable\" % type(obj))\n\n\n# Modified version of lm_dataformat Archive for single file.\nclass Archive:\n def __init__(self, file_path, compression_level=3):\n self.file_path = file_path\n dir_name = os.path.dirname(file_path)\n if dir_name:\n os.makedirs(dir_name, exist_ok=True)\n self.fh = open(self.file_path, \"wb\")\n self.cctx = zstandard.ZstdCompressor(level=compression_level)\n self.compressor = self.cctx.stream_writer(self.fh)\n\n def add_data(self, data, meta={}):\n self.compressor.write(\n json.dumps({\"text\": data, \"meta\": meta}, default=json_serial).encode(\n \"UTF-8\"\n )\n + b\"\\n\"\n )\n\n def commit(self):\n self.compressor.flush(zstandard.FLUSH_FRAME)\n self.fh.flush()\n self.fh.close()\n\n\n# Modified version of lm_dataformat Reader with self.fh set, allowing peeking for tqdm.\nclass Reader:\n def __init__(self):\n pass\n\n def read(self, file, get_meta=False, autojoin_paragraphs=True, para_joiner=\"\\n\\n\"):\n with open(file, \"rb\") as fh:\n self.fh = fh\n cctx = zstandard.ZstdDecompressor()\n reader = io.BufferedReader(cctx.stream_reader(fh))\n rdr = jsonlines.Reader(reader)\n for ob in rdr:\n # naive jsonl where each object is just the string itself, with no meta. For legacy compatibility.\n if isinstance(ob, str):\n assert not get_meta\n yield ob\n continue\n\n text = ob[\"text\"]\n\n if autojoin_paragraphs and isinstance(text, list):\n text = para_joiner.join(text)\n\n if get_meta:\n yield text, (ob[\"meta\"] if \"meta\" in ob else {})\n else:\n yield text\n\n\nclass TextArchive:\n def __init__(self, file_path, mode=\"rb+\"):\n self.file_path = file_path\n dir_name = os.path.dirname(file_path)\n if dir_name:\n os.makedirs(dir_name, exist_ok=True)\n\n if not os.path.exists(file_path):\n Path(file_path).touch()\n\n self.fh = open(self.file_path, mode)\n\n def add_data(self, data):\n self.fh.write(data.encode(\"UTF-8\") + b\"\\n\")\n\n def commit(self):\n self.fh.flush()\n self.fh.close()\n\n\nclass TextReader:\n def __init__(self, file_path):\n self.file_path = file_path\n\n # Optimized mmap read with infrequent tqdm updates to maintain speed\n # Tested up to 250MB/s.\n def read_tqdm(self, update_frequency=10000):\n current_file_position = 0\n line_counter = 0\n with open(self.file_path, \"r\") as fh, tqdm.tqdm(\n total=os.path.getsize(self.file_path),\n dynamic_ncols=True,\n unit=\"byte\",\n unit_scale=1,\n ) as progress:\n with mmap.mmap(fh.fileno(), length=0, access=mmap.ACCESS_READ) as mmap_obj:\n for line in iter(mmap_obj.readline, b\"\"):\n line = line.decode(\"utf-8\")\n line_counter += 1\n if line_counter == update_frequency:\n new_file_pos = mmap_obj.tell()\n bytes_read = new_file_pos - current_file_position\n current_file_position = new_file_pos\n progress.update(bytes_read)\n line_counter = 0\n yield line[:-1]\n\n def read_and_tell(self):\n current_file_position = 0\n with open(self.file_path, \"r\", encoding=\"utf8\") as fh:\n with mmap.mmap(fh.fileno(), length=0, access=mmap.ACCESS_READ) as mmap_obj:\n for line in iter(mmap_obj.readline, b\"\"):\n line = line.decode(\"utf-8\")\n new_file_pos = mmap_obj.tell()\n raw_bytes_read = new_file_pos - current_file_position\n current_file_position = new_file_pos\n yield line[:-1], raw_bytes_read\n\n def read(self):\n with open(self.file_path, \"r\", encoding=\"utf8\") as fh:\n with mmap.mmap(fh.fileno(), length=0, access=mmap.ACCESS_READ) as mmap_obj:\n for line in iter(mmap_obj.readline, b\"\"):\n line = line.decode(\"utf-8\")\n yield line[:-1]\n\n def read_slow(self):\n with open(self.file_path, \"r\", encoding=\"utf8\") as fh:\n while True:\n line = fh.readline()\n if line == -1 or line == \"\":\n break\n else:\n yield line[:-1]\n\n\n# Optimized for speed. Decompresses the archive in shell before\n# using the mmap'd TextReader.\nclass ZStdTextReader:\n def __init__(self, file):\n self.file = file\n\n def read_tqdm(self):\n decompressed_file = self.file[:-4]\n print(\"Decompressing file, please wait...\")\n os.system(f\"zstd -d {self.file}\") # linux decompress is faster\n reader = TextReader(decompressed_file)\n yield from reader.read_tqdm()\n os.remove(decompressed_file)\n" }, { "path": "lm_eval/decontamination/decontaminate.py", "content": "import time\nimport random\nimport pickle\nimport json\nimport glob\nimport os\nimport collections\n\nfrom .janitor import Janitor, word_ngrams\nfrom .archiver import ZStdTextReader\n\n\n# Was used for testing the evaluator decoupled from the full logic below\ndef get_train_overlap_stub(docs, ngrams_path, ngrams_n_size):\n simulated_overlap = 0.1\n contaminated = int(len(docs) * simulated_overlap)\n return random.sample(range(len(docs)), contaminated)\n\n\n# Returns a dictionary containing all overlapping documents in each\n# task. In the standard use case, an overlap occurs when any of the 13-grams\n# found in the task document exist in the training set documents.\n#\n# To generate 13-grams for the pile see scripts/clean_training_data. The final output of these\n# scripts are an info.json file containing the n_gram_size (13) and a bunch of \"ngrams_{x}.bkt.txt.sorted.zst\"\n# files. These should exist in the \"ngrams_path\" provided to this function.\n\n# Algorithm:\n# 1. Build lookups for each dataset {ngram: list(document_ids)}\n# 2. Merge into an overall lookup {ngram: [(task_name, task_set, doc_ids),]}\n# 3. Full scan the 13-grams from the training set against the merged lookup,\n# saving matches in the \"duplicates\" dictionary {(task_name, task_set): set(doc_ids)}\n# 4. Strip the task_set from the dictionary keys and return\n#\n# We cache the task+set lookups as well as the overlaps.\ndef get_train_overlap(docs_by_task_set, ngrams_path, limit):\n # return get_train_overlap_stub(docs, ngrams_path, ngrams_n_size)\n\n info_dict_path = os.path.join(ngrams_path, \"info.json\")\n info_dict = json.load(open(info_dict_path, \"r\"))\n ngrams_n_size = info_dict[\"ngram_size\"]\n\n janitor = Janitor()\n\n # Build lookup for each dataset first in case we use different task combinations later\n print(\"Building Lookups...\")\n start = time.perf_counter()\n\n def get_overlaps_dump_path(task_name, task_set, ngrams_n_size, limit):\n return f\"data/{task_name}/{task_set}_{ngrams_n_size}grams_limit{limit}.overlaps\"\n\n lookups = {}\n duplicates = {} # (task_name, task_set): set(doc_ids)}\n sets_to_decontaminate = len(docs_by_task_set.keys())\n\n for (task_name, task_set), docs in docs_by_task_set.items():\n if not os.path.exists(f\"data/{task_name}\"):\n os.mkdir(f\"data/{task_name}\")\n\n # Check if we've decontaminated this combination before\n overlaps_dump_path = get_overlaps_dump_path(\n task_name, task_set, ngrams_n_size, limit\n )\n if os.path.exists(overlaps_dump_path):\n duplicates[(task_name, task_set)] = pickle.load(\n open(overlaps_dump_path, \"rb\")\n )\n sets_to_decontaminate -= 1\n continue\n else:\n duplicates[(task_name, task_set)] = set()\n\n # Build/load the task lookup {ngram: set(documents)}.\n task_set_lookup_path = (\n f\"data/{task_name}/{task_set}_{ngrams_n_size}grams_limit{limit}.lookup\"\n )\n if os.path.exists(task_set_lookup_path):\n print(f\"{task_set_lookup_path} available, loading...\")\n lookups[(task_name, task_set)] = pickle.load(\n open(task_set_lookup_path, \"rb\")\n )\n else:\n print(f\"{task_set_lookup_path} not available, building...\")\n lookup = collections.defaultdict(set)\n\n for doc_id, document in enumerate(docs):\n ngrams = word_ngrams(janitor.normalize_string(document), ngrams_n_size)\n for ngram in ngrams:\n lookup[ngram].add(doc_id)\n\n pickle.dump(lookup, open(task_set_lookup_path, \"wb\"))\n lookups[(task_name, task_set)] = lookup\n\n elapsed = time.perf_counter() - start\n print(f\"Building lookups took {elapsed:0.5f} seconds.\")\n\n matched_ngrams = []\n\n if sets_to_decontaminate > 0:\n print(\"Merging lookups...\")\n start = time.perf_counter()\n merged_lookup = collections.defaultdict(list)\n for (task_name, task_set), lookup in lookups.items():\n for ngram, doc_ids in lookup.items():\n merged_lookup[ngram].append((task_name, task_set, doc_ids))\n\n elapsed = time.perf_counter() - start\n print(f\"Merging lookups took {elapsed:0.5f} seconds.\")\n\n print(f\"{ngrams_n_size} grams files found in {ngrams_path}:\")\n files = glob.glob(os.path.join(ngrams_path, f\"*.sorted.zst\"))\n print(files)\n\n for file in files:\n start = time.perf_counter()\n print(f\"Scanning {file}\")\n reader = ZStdTextReader(file)\n total_ngrams = 0\n unique_ngrams = 0\n matching_unique = 0\n non_matching_unique = 0\n\n current_ngram = \"\"\n for line in reader.read_tqdm(): # Scan training set ngrams file\n total_ngrams += 1\n [ngram, document_id] = line.rsplit(\" \", 1)\n if (\n ngram != current_ngram\n ): # Only need to match the ngram once in training set\n unique_ngrams += 1\n current_ngram = ngram\n if ngram in merged_lookup:\n matched_ngrams.append(ngram) # For logging\n matching_unique += 1\n for task_name, task_set, doc_ids in merged_lookup[ngram]:\n task_doc_set = duplicates[(task_name, task_set)]\n for (\n doc_id\n ) in (\n doc_ids\n ): # Record contamination across all relevant task/set combos\n task_doc_set.add(doc_id)\n del merged_lookup[ngram] # No point matching again\n else:\n non_matching_unique += 1\n\n print(f\"Total Ngrams: {total_ngrams}\")\n print(f\"Unique Ngrams: {unique_ngrams}\")\n print(f\"Unique Matching: {matching_unique}\")\n print(f\"Unique Non Matching: {non_matching_unique}\")\n print(\"Matched ngrams:\")\n for ngram in matched_ngrams:\n print(ngram)\n\n elapsed = time.perf_counter() - start\n print(f\"Read took {elapsed:0.5f} seconds.\")\n print(f\"Speed: {(os.path.getsize(file)/1000000.0)/elapsed}MB/second\")\n\n print(duplicates)\n\n # Dump overlaps separately\n for (task_name, task_set), doc_ids in duplicates.items():\n overlaps_dump_path = get_overlaps_dump_path(\n task_name, task_set, ngrams_n_size, limit\n )\n pickle.dump(doc_ids, open(overlaps_dump_path, \"wb\"))\n\n # Strip task set and return\n return {task_name: doc_ids for (task_name, task_set), doc_ids in duplicates.items()}\n" }, { "path": "lm_eval/decontamination/janitor.py", "content": "import re\nimport string\nimport timeit\nimport pickle\nimport traceback\nfrom pprint import pprint\n\n# This is a cpp module. Compile janitor_util.cpp with:\n# c++ -O3 -Wall -shared -std=c++11 -fPIC $(python3 -m pybind11 --includes) janitor_util.cpp -o janitor_util$(python3-config --extension-suffix) -undefined dynamic_lookup\ntry:\n import janitor_util\n\n JANITOR_CPP = True\nexcept Exception:\n print(\"WARNING: C++ module could not be loaded. Janitor running in python mode\")\n traceback.print_exc()\n JANITOR_CPP = False\n\n\n# Implementation from nltk source\n# https://www.nltk.org/_modules/nltk/util.html\ndef form_ngrams(sequence, n):\n history = []\n while n > 1:\n # PEP 479, prevent RuntimeError from being raised when StopIteration bubbles out of generator\n try:\n next_item = next(sequence)\n except StopIteration:\n # no more data, terminate the generator\n return\n history.append(next_item)\n n -= 1\n for item in sequence:\n history.append(item)\n yield tuple(history)\n del history[0]\n\n\ndef word_ngrams(s, n):\n \"\"\"Splits a string into ngram words\"\"\"\n tokens = s.split() # not a generator :(\n ngram_seqs = form_ngrams(iter(tokens), n)\n return (\" \".join(ngram) for ngram in ngram_seqs)\n\n\n# Does character sequences only - combined faster function to play around with later\n# def word_ngrams_indices_combined(sequence, n):\n# current_word = \"\"\n# history = []\n# gap = False;\n# start = 0\n# end = 0\n# for character in sequence:\n# if character == \" \":\n# if not gap:\n# gap = True\n# history.append(current_word)\n# end += len(current_word) - 1\n# current_word = \"\"\n# if len(history) == n:\n# yield (tuple(history), start, end)\n# del history[0]\n# start = end + 1\n# end = start\n# else:\n# gap = False\n# current_word += character\n\n\n# https://stackoverflow.com/questions/13734451/string-split-with-indices-in-python\ndef split_indices(s):\n \"\"\"Splits a string on whitespaces and records the indices of each in the original string.\n @:return generator((word, (start_idx, end_idx)), ...)\n \"\"\"\n return ((m.group(0), (m.start(), m.end() - 1)) for m in re.finditer(r\"\\S+\", s))\n\n\ndef word_ngrams_indices(s, n):\n \"\"\"Splits a string into pairs of (ngram words, their start/end indices)\"\"\"\n tokens_with_indices = split_indices(s)\n\n # Generator of ngrams of (word, idx_pairs)\n # (\n # [(word, (start,end)), (word, (start, end))...],\n # [(word, (start, end)), ...],\n # ...\n # )\n ngram_seqs_with_indices = form_ngrams(tokens_with_indices, n)\n\n # Generator of pairs of word and index ngrams\n # (\n # ([word, word, ...], [(start,end), (start,end), ...]),\n # ...\n # )\n ngram_indices_pairs = (\n zip(*ngram_with_indices) for ngram_with_indices in ngram_seqs_with_indices\n )\n\n # Generator of ( (word_ngram, (start, end)), (word_ngram, start, end)), ...)\n return (\n (\" \".join(ngram_seq), (indices[0][0], indices[-1][1]))\n for ngram_seq, indices in ngram_indices_pairs\n )\n\n\nclass Janitor:\n\n # FIXME delete_chars: Should anything else go here? Special chars?\n def __init__(\n self,\n ngram_n=13,\n window_to_remove=200,\n too_dirty_cutoff=10,\n minimum_slice_length=200,\n delete_chars=string.punctuation,\n ):\n self.ngram_n = ngram_n\n self.window_to_remove = window_to_remove\n self.too_dirty_cutoff = too_dirty_cutoff\n self.minimum_slice_length = minimum_slice_length\n self.delete_chars = delete_chars\n\n self.dirt_ngrams = set()\n\n # If in python, we'll translate uppercase to lowercase and delete naughty characters.\n # This is fast by python standards\n # https://stackoverflow.com/questions/638893/what-is-the-most-efficient-way-in-python-to-convert-a-string-to-all-lowercase-st\n self.translation_table = str.maketrans(\n string.ascii_lowercase + string.ascii_uppercase, # These characters\n string.ascii_lowercase * 2, # Become these characters\n self.delete_chars, # These are deleted\n )\n\n ##############\n # I/O for saving contamination ngrams\n ##############\n\n def save_contamination_ngrams(self, filename):\n with open(filename, \"wb\") as fp:\n pickle.dump(filename, fp)\n\n def load_contamination_ngrams(self, filename):\n with open(filename, \"rb\") as fp:\n self.dirt_ngrams = pickle.load(fp)\n\n ##############\n # Call these :)\n ##############\n\n def register_contaminant(self, dirt_string):\n \"\"\"Register a string as contamination to be removed, e.g. a test set\n This breaks the dirt_string into ngrams to store for future cleaning\"\"\"\n if JANITOR_CPP:\n return self.register_contaminant_cpp(dirt_string)\n else:\n print(\"WARNING: Janitor running in python mode\")\n return self.register_contaminant_python(dirt_string)\n\n def clean(self, dirty_string):\n \"\"\"Clean a string (e.g. a training set) by removing all ngrams previously\n registered as contaminants. Returns a list of clean chunks, or empty if\n the string was too dirty\"\"\"\n if JANITOR_CPP:\n return self.clean_cpp(dirty_string)\n else:\n print(\"WARNING: Janitor running in python mode\")\n return self.clean_python(dirty_string)\n\n def _split_chunks(self, dirty_string, dirty_parts):\n clean_chunks = []\n splice_idx = 0\n end = -1\n for i, (ngram, start, end) in enumerate(dirty_parts):\n if i >= self.too_dirty_cutoff:\n return []\n start = max(0, start - self.window_to_remove)\n end = min(len(dirty_string), end + self.window_to_remove)\n\n if start - splice_idx > self.minimum_slice_length:\n clean_chunks.append(dirty_string[splice_idx:start])\n splice_idx = end\n\n if end < len(dirty_string) - self.minimum_slice_length:\n clean_chunks.append(dirty_string[end + 1 :])\n\n return clean_chunks\n\n ##############\n # Fast C++\n ##############\n\n def register_contaminant_cpp(self, dirt_string):\n self.dirt_ngrams.update(\n janitor_util.clean_ngram(dirt_string, self.delete_chars, self.ngram_n)\n )\n\n def clean_cpp(self, dirty_string):\n contamination_indices = janitor_util.clean_ngram_with_indices(\n dirty_string, self.delete_chars, self.ngram_n\n )\n return self._split_chunks(dirty_string, contamination_indices)\n\n ##############\n # Slow python\n ##############\n\n def normalize_string(self, s):\n return s.translate(self.translation_table)\n\n def register_contaminant_python(self, dirt_string):\n self.dirt_ngrams.update(\n word_ngrams(self.normalize_string(dirt_string), self.ngram_n)\n )\n\n def clean_python(self, dirty_string):\n contamination_indices = (\n (None, *idx_pair)\n for dirty_ngram, idx_pair in word_ngrams_indices(dirty_string, self.ngram_n)\n if self.normalize_string(dirty_ngram) in self.dirt_ngrams\n )\n return self._split_chunks(dirty_string, contamination_indices)\n\n\n##################################################################\n# Tests\n#################################################################\n\n# def print_cpp():\n# source = \"\"\" ,, I'm a very !dirty,, ,, dirty boy. Clean me daddy. \\n\\nhe he he hehe heh. lastword \"\"\" * 2\n\n# for i in range(1, 10, 2):\n# pprint(janitor_util.clean_ngram(source, string.punctuation, i))\n# for ngram, start, end in \\\n# janitor_util.clean_ngram_with_indices(source, string.punctuation, i):\n# print(ngram, \"\\t\", start, end, source[start:end].replace(\"\\n\", \"\\\\n\"))\n\n\n# def test_cpp():\n# source = \"\"\" ,, I'm a very !dirty,, ,, dirty boy. Clean me daddy. \\n\\nhe he he hehe heh. lastword \"\"\" * 2\n# contaminant = \"dirty boy. Clean he he\"\n\n# jan_python = Janitor()\n# jan_cpp = Janitor()\n\n# jan_python.register_contaminant_python(contaminant)\n# jan_cpp.register_contaminant(contaminant)\n\n# assert jan_python.dirt_ngrams == jan_cpp.dirt_ngrams, (jan_python.dirt_ngrams, jan_cpp.dirt_ngrams)\n\n# assert jan_python.clean_python(source) == jan_cpp.clean(source), \\\n# (jan_python.clean_python(source), jan_cpp.clean(source))\n\n# print(\"Passed test, python==cpp\")\n\n\n# def benchmark():\n# # Download and put in data folder: enwik8 (100 MB) from https://cs.fit.edu/~mmahoney/compression/textdata.html\n# setup = \\\n# \"\"\"\n# with open(\"data/enwik8\", \"r\") as f:\n# data = f.read()\n# jan = Janitor(too_dirty_cutoff=1000)\n# jan.register_contaminant('''\n# theories is that there is a connection between "geekdom" and autism.\n# This is hinted, for instance, by a ''Wired Magazine'' article in 2001 entitled "\n# The [[Geek]] Syndrome", which is a point argued by many in the autism rights\n# movement{{ref|Wired}}. This article, many professionals assert, is just one example of\n# the media's application of mental disease labels to what is actually variant normal behavior\n# &mdash;they argue that shyness, lack of athletic ability or social skills, and intellectual\n# interests, even when they seem unusual to others, are not in themselves signs of autism or\n# Asperger's syndrome. Others assert that it is actually the medical profession which is applying\n# mental disease labels to children who in the past would have simply been accepted as a little\n# different or even labeled 'gifted'. See [[clinomorphism]] for further discussion of this issue.\n# Due to the recent publicity surrounding autism and autis\n# ultan Al Nahyan]] granted [[Petroleum]] concessions, and oil was first found in 1958. At first,\n# oil money had a marginal impact. A few lowrise concete buildings were erected, and the first\n# paved road was completed in 1961, but Sheikh Shakbut, uncertain whether the new oil royalties\n# would last, took a cautious approach, preferring to save the revenue rather than investing it in\n# development. His brother, [[Zayed bin Sultan Al Nahayan]], saw that oil wealth had the potential\n# to transform Abu Dhabi. The ruling Al Nahayan family decided that Sheikh Zayed should replace his\n# brother as Ruler and carry out his vision of developing the country. On [[August 6]], [[1966]],\n# with the assistance of the British, Sheikh Zayed became the new ruler. See generally, Al-Fahim, M,\n# ''From Rags to Riches: A Story of Abu Dhabi'', Chapter Six (London Centre of Arab Studies, 1995),\n# ISBN 1 900404 00 1. With the announcement by Britain in 1968 that it would withdraw from the\n# Gulf area by 1971, Sheikh Zayed became the main driving force behind the formation of the\n# [[United Arab Emirates]]. After the Emirates gained independence in 1971,\n# ''')\n# \"\"\"\n\n# n = 1\n# print(f\"Timing {n} run on 100 MB\")\n# print(\"Register contaminant\")\n# # print(\"\\tPython\", timeit.timeit(\"jan.register_contaminant_python(data)\", setup=setup, globals=globals(), number=n))\n# print(\"\\tCpp\", timeit.timeit(\"jan.register_contaminant(data)\", setup=setup, globals=globals(), number=n))\n\n# print(\"Clean\")\n# # print(\"\\tPython\", timeit.timeit(\"jan.clean_python(data)\", setup=setup, globals=globals(), number=n))\n# print(\"\\tCpp\", timeit.timeit(\"jan.clean(data)\", setup=setup, globals=globals(), number=n))\n\n\n# def test_janitor_general():\n# source = \"\"\" ,, I'm a very !dirty,, ,, dirty boy. Clean me daddy. \\n\\nhe he he hehe heh. lastword \"\"\" * 2\n# contaminant = \"dirty boy. Clean he he\"\n\n# jan = Janitor(ngram_n=3)\n# jan.register_contaminant(contaminant)\n# cleaned = \" \".join(jan.clean(source))\n# for contam in jan.dirt_ngrams:\n# assert contam not in cleaned, contam\n\n# filename = \"data/saved_contam\"\n# jan.save_contamination_ngrams(filename)\n\n# jan = Janitor(ngram_n=3)\n# jan.load_contamination_ngrams(filename)\n# cleaned = \" \".join(jan.clean(source))\n# for contam in jan.dirt_ngrams:\n# assert contam not in cleaned, contam\n\n\n# if __name__ == \"__main__\":\n# test()\n# # print_cpp()\n# # test_cpp()\n# # benchmark()\n" }, { "path": "lm_eval/evaluator copy.py", "content": "import collections\nimport itertools\nimport numpy as np\nimport random\nimport lm_eval.metrics\nimport lm_eval.models\nimport lm_eval.tasks\nimport lm_eval.base\nfrom lm_eval.utils import positional_deprecated, run_task_tests\n\nimport pdb\nimport torch\n\n\n\n@positional_deprecated\ndef simple_evaluate(\n model,\n model_args=None,\n tasks=[],\n num_fewshot=0,\n batch_size=None,\n device=None,\n no_cache=False,\n limit=None,\n bootstrap_iters=100000,\n description_dict=None,\n check_integrity=False,\n decontamination_ngrams_path=None,\n weight_bit=8,\n act_bit=8,\n clip_range=10,\n alpha=0.5\n):\n\n \"\"\"Instantiate and evaluate a model on a list of tasks.\n\n :param model: Union[str, LM]\n Name of model or LM object, see lm_eval.models.get_model\n :param model_args: Optional[str]\n String arguments for each model class, see LM.create_from_arg_string.\n Ignored if `model` argument is a LM object.\n :param tasks: list[Union[str, Task]]\n List of task names or Task objects. Task objects will be taken to have name task.EVAL_HARNESS_NAME if defined and type(task).__name__ otherwise.\n :param num_fewshot: int\n Number of examples in few-shot context\n :param batch_size: int, optional\n Batch size for model\n :param device: str, optional\n PyTorch device (e.g. \"cpu\" or \"cuda:0\") for running models\n :param no_cache: bool\n Whether or not to cache\n :param limit: int, optional\n Limit the number of examples per task (only use this for testing)\n :param bootstrap_iters:\n Number of iterations for bootstrap statistics\n :param description_dict: dict[str, str]\n Dictionary of custom task descriptions of the form: `task_name: description`\n :param check_integrity: bool\n Whether to run the relevant part of the test suite for the tasks\n :return\n Dictionary of results\n \"\"\"\n random.seed(1234)\n np.random.seed(1234)\n\n assert tasks != [], \"No tasks specified\"\n\n if isinstance(model, str):\n if model_args is None:\n model_args = \"\"\n lm = lm_eval.models.get_model(model).create_from_arg_string(\n model_args, {\"batch_size\": batch_size, \"device\": device}\n )\n else:\n assert isinstance(model, lm_eval.base.LM)\n lm = model\n\n\n if not no_cache:\n lm = lm_eval.base.CachingLM(\n lm,\n \"lm_cache/\"\n + model\n + \"_\"\n + model_args.replace(\"=\", \"-\").replace(\",\", \"_\").replace(\"/\", \"-\")\n + f\"_w{weight_bit}a{act_bit}c{clip_range}alpha{alpha}\"\n + \".db\",\n )\n task_dict = lm_eval.tasks.get_task_dict(tasks)\n\n if check_integrity:\n run_task_tests(task_list=tasks)\n\n \n \n results = evaluate(\n lm=lm,\n task_dict=task_dict,\n num_fewshot=num_fewshot,\n limit=limit,\n bootstrap_iters=bootstrap_iters,\n description_dict=description_dict,\n decontamination_ngrams_path=decontamination_ngrams_path,\n )\n\n # add info about the model and few shot config\n results[\"config\"] = {\n \"model\": model,\n \"model_args\": model_args,\n \"num_fewshot\": num_fewshot,\n \"batch_size\": batch_size,\n \"device\": device,\n \"no_cache\": no_cache,\n \"limit\": limit,\n \"bootstrap_iters\": bootstrap_iters,\n \"description_dict\": description_dict,\n }\n\n return results\n\n\ndecontaminate_suffix = \"_decontaminate\"\n\n\n@positional_deprecated\ndef evaluate(\n lm,\n task_dict,\n provide_description=None,\n num_fewshot=0,\n limit=None,\n bootstrap_iters=100000,\n description_dict=None,\n decontamination_ngrams_path=None,\n):\n \"\"\"Instantiate and evaluate a model on a list of tasks.\n\n :param lm: obj\n Language Model\n :param task_dict: dict[str, Task]\n Dictionary of tasks. Tasks will be taken to have name task.EVAL_HARNESS_NAME if defined and type(task).__name__ otherwise.\n :param provide_description: bool\n Not implemented, and this option is deprecated and will be removed in a future version in favor of a different description providing method\n :param num_fewshot: int\n Number of examples in few-shot context\n :param limit: int, optional\n Limit the number of examples per task (only use this for testing)\n :param bootstrap_iters:\n Number of iterations for bootstrap statistics\n :param description_dict: dict[str, str]\n Dictionary of custom task descriptions of the form: `task_name: description`\n :return\n Dictionary of results\n \"\"\"\n # TODO: completely refactor this entire function to not be a huge mess, ideally breaking it down into smaller pieces\n\n # TODO: todo: implement proper description-providing system\n assert not provide_description # not implemented.\n if provide_description is not None:\n # nudge people to not specify it at all\n print(\n \"WARNING: provide_description is deprecated and will be removed in a future version in favor of description_dict\"\n )\n\n decontaminate = decontamination_ngrams_path is not None\n\n task_dict_items = [\n (name, task)\n for name, task in task_dict.items()\n if (task.has_validation_docs() or task.has_test_docs())\n ]\n\n results = collections.defaultdict(dict)\n versions = collections.defaultdict(dict)\n\n requests = collections.defaultdict(list)\n requests_origin = collections.defaultdict(list)\n\n overlaps = collections.defaultdict(list) # {task_name: contaminated_docs}\n\n # If we ever run into issues where the eval tasks don't fit in memory and we can't afford a machine with bigger\n # memory, we can always modify this plumbing to support that, but I didn't want to include it just yet because\n # over-engineering is bad (or we could make it write the requests to disk and then read them back out again\n # - probably using an sqlite db because of all the moving parts we have\n\n # TODO: we need unit tests & sanity checks or something to ensure that the return of `validation_docs` is stable\n docs = {}\n\n docs_for_decontamination = collections.defaultdict(list)\n\n # get lists of each type of request\n for task_name, task in task_dict_items:\n versions[task_name] = task.VERSION\n # default to test doc, fall back to val doc if validation unavailable\n # TODO: the test-fallback-to-val system isn't final, we should revisit it at some point\n if task.has_test_docs():\n task_doc_func = task.test_docs\n task_set = \"test\" # Required for caching in the decontamination\n elif task.has_validation_docs():\n task_set = \"val\" # Required for caching in the decontamination\n task_doc_func = task.validation_docs\n else:\n raise RuntimeError(\"Task has neither test_docs nor validation_docs\")\n\n # deterministically shuffle docs and chop off the first `limit` because sometimes docs are in some kind of order\n task_docs = list(task_doc_func())\n rnd = random.Random()\n rnd.seed(42)\n rnd.shuffle(task_docs)\n\n description = (\n description_dict[task_name]\n if description_dict and task_name in description_dict\n else \"\"\n )\n\n for doc_id, doc in enumerate(itertools.islice(task_docs, 0, limit)):\n\n if decontaminate and task.should_decontaminate():\n docs_for_decontamination[(task_name, task_set)].append(\n task.doc_to_decontamination_query(doc)\n )\n\n docs[(task_name, doc_id)] = doc\n ctx = task.fewshot_context(\n doc=doc, num_fewshot=num_fewshot, rnd=rnd, description=description\n )\n reqs = task.construct_requests(doc, ctx)\n if not isinstance(reqs, (list, tuple)):\n reqs = [reqs]\n for i, req in enumerate(reqs):\n requests[req.request_type].append(req)\n # i: index in requests for a single task instance\n # doc_id: unique id that we can get back to a doc using `docs`\n requests_origin[req.request_type].append((i, task_name, doc, doc_id))\n\n # Compare all tasks/sets at once to ensure a single training set scan\n if decontaminate:\n from lm_eval.decontamination.decontaminate import get_train_overlap\n\n print(\"Finding train/test overlap, please wait...\")\n overlaps = get_train_overlap(\n docs_for_decontamination, decontamination_ngrams_path, limit\n )\n\n # all responses for each (task, doc)\n process_res_queue = collections.defaultdict(list)\n\n # execute each type of request\n for reqtype, reqs in requests.items():\n # TODO: right now, this code runs multiple separate LM requests for multiple Requests differing\n # only in index. We could implement some kind of caching, but that would be more of a band-aid\n # solution. we could also implement some kind of auto-grouping here;\n # they should end up next to each other.\n\n print(\"Running\", reqtype, \"requests\")\n resps = getattr(lm, reqtype)([req.args for req in reqs])\n resps = [\n x if req.index is None else x[req.index] for x, req in zip(resps, reqs)\n ]\n\n for resp, (i, task_name, doc, doc_id) in zip(resps, requests_origin[reqtype]):\n process_res_queue[(task_name, doc_id)].append((i, resp))\n\n vals = collections.defaultdict(list)\n\n # unpack results and sort back in order and return control to Task\n for (task_name, doc_id), requests in process_res_queue.items():\n requests.sort(key=lambda x: x[0])\n requests = [x[1] for x in requests]\n\n task = task_dict[task_name]\n doc = docs[(task_name, doc_id)]\n\n metrics = task.process_results(doc, requests)\n for metric, value in metrics.items():\n vals[(task_name, metric)].append(value)\n\n # Re-use the evaluation for the decontaminated set by just ignoring the overlaps\n if decontaminate and task_name in overlaps:\n if doc_id not in overlaps[task_name]:\n vals[(task_name, metric + decontaminate_suffix)].append(value)\n\n # aggregate results\n for (task_name, metric), items in vals.items():\n task = task_dict[task_name]\n real_metric = metric # key when looking up the metric with task.aggregation\n if metric.endswith(decontaminate_suffix):\n real_metric = metric.replace(\n decontaminate_suffix, \"\"\n ) # decontaminated still uses the same metric\n results[task_name][metric] = task.aggregation()[real_metric](items)\n\n # hotfix: bleu, chrf, ter seem to be really expensive to bootstrap\n # so we run them less iterations. still looking for a cleaner way to do this\n\n stderr = lm_eval.metrics.stderr_for_metric(\n metric=task.aggregation()[real_metric],\n bootstrap_iters=min(bootstrap_iters, 1000)\n if metric in [\"bleu\", \"chrf\", \"ter\"]\n else bootstrap_iters,\n )\n\n if stderr is not None:\n results[task_name][metric + \"_stderr\"] = stderr(items)\n\n return {\"results\": dict(results), \"versions\": dict(versions)}\n\n\ndef make_table(result_dict):\n \"\"\"Generate table of results.\"\"\"\n from pytablewriter import MarkdownTableWriter, LatexTableWriter\n\n md_writer = MarkdownTableWriter()\n latex_writer = LatexTableWriter()\n md_writer.headers = [\"Task\", \"Version\", \"Metric\", \"Value\", \"\", \"Stderr\"]\n latex_writer.headers = [\"Task\", \"Version\", \"Metric\", \"Value\", \"\", \"Stderr\"]\n\n values = []\n\n for k, dic in result_dict[\"results\"].items():\n version = result_dict[\"versions\"][k]\n for m, v in dic.items():\n if m.endswith(\"_stderr\"):\n continue\n\n if m + \"_stderr\" in dic:\n se = dic[m + \"_stderr\"]\n values.append([k, version, m, \"%.4f\" % v, \"±\", \"%.4f\" % se])\n else:\n values.append([k, version, m, \"%.4f\" % v, \"\", \"\"])\n k = \"\"\n version = \"\"\n md_writer.value_matrix = values\n latex_writer.value_matrix = values\n\n # todo: make latex table look good\n # print(latex_writer.dumps())\n\n return md_writer.dumps()\n" }, { "path": "lm_eval/evaluator.py", "content": "import collections\nimport itertools\nimport numpy as np\nimport random\nimport lm_eval.metrics\nimport lm_eval.models\nimport lm_eval.tasks\nimport lm_eval.base\nfrom lm_eval.utils import positional_deprecated, run_task_tests\nimport fnmatch\ndef pattern_match(patterns, source_list):\n task_names = set()\n for pattern in patterns:\n for matching in fnmatch.filter(source_list, pattern):\n task_names.add(matching)\n return list(task_names)\n@positional_deprecated\ndef simple_evaluate(\n lm,\n tasks,\n model_args=None,\n num_fewshot=0,\n limit=None,\n bootstrap_iters=100000,\n description_dict=None,\n decontamination_ngrams_path=None,\n\n):\n\n \"\"\"Instantiate and evaluate a model on a list of tasks.\n\n :param model: Union[str, LM]\n Name of model or LM object, see lm_eval.models.get_model\n :param model_args: Optional[str]\n String arguments for each model class, see LM.create_from_arg_string.\n Ignored if `model` argument is a LM object.\n :param tasks: list[Union[str, Task]]\n List of task names or Task objects. Task objects will be taken to have name task.EVAL_HARNESS_NAME if defined and type(task).__name__ otherwise.\n :param num_fewshot: int\n Number of examples in few-shot context\n :param batch_size: int, optional\n Batch size for model\n :param device: str, optional\n PyTorch device (e.g. \"cpu\" or \"cuda:0\") for running models\n :param no_cache: bool\n Whether or not to cache\n :param limit: int, optional\n Limit the number of examples per task (only use this for testing)\n :param bootstrap_iters:\n Number of iterations for bootstrap statistics\n :param description_dict: dict[str, str]\n Dictionary of custom task descriptions of the form: `task_name: description`\n :param check_integrity: bool\n Whether to run the relevant part of the test suite for the tasks\n :return\n Dictionary of results\n \"\"\"\n random.seed(1234)\n np.random.seed(1234)\n if tasks is None:\n raise ValueError(\"Please specify a task to run\")\n else:\n task_names = pattern_match(tasks.split(\",\"), lm_eval.tasks.ALL_TASKS)\n assert tasks != [], \"No tasks specified\"\n print(f\"Selected Tasks: {task_names}\")\n task_dict = lm_eval.tasks.get_task_dict(task_names)\n\n\n results = evaluate(\n lm=lm,\n task_dict=task_dict,\n num_fewshot=num_fewshot,\n limit=limit,\n bootstrap_iters=bootstrap_iters,\n description_dict=description_dict,\n decontamination_ngrams_path=decontamination_ngrams_path,\n )\n\n # add info about the model and few shot config\n results[\"config\"] = {\n \"model\": lm,\n \"model_args\": model_args,\n \"num_fewshot\": num_fewshot,\n \"limit\": limit,\n \"bootstrap_iters\": bootstrap_iters,\n \"description_dict\": description_dict,\n }\n\n return results\n\n\ndecontaminate_suffix = \"_decontaminate\"\n\n\n@positional_deprecated\ndef evaluate(\n lm,\n task_dict,\n provide_description=None,\n num_fewshot=0,\n limit=None,\n bootstrap_iters=100000,\n description_dict=None,\n decontamination_ngrams_path=None,\n):\n \"\"\"Instantiate and evaluate a model on a list of tasks.\n\n :param lm: obj\n Language Model\n :param task_dict: dict[str, Task]\n Dictionary of tasks. Tasks will be taken to have name task.EVAL_HARNESS_NAME if defined and type(task).__name__ otherwise.\n :param provide_description: bool\n Not implemented, and this option is deprecated and will be removed in a future version in favor of a different description providing method\n :param num_fewshot: int\n Number of examples in few-shot context\n :param limit: int, optional\n Limit the number of examples per task (only use this for testing)\n :param bootstrap_iters:\n Number of iterations for bootstrap statistics\n :param description_dict: dict[str, str]\n Dictionary of custom task descriptions of the form: `task_name: description`\n :return\n Dictionary of results\n \"\"\"\n # TODO: completely refactor this entire function to not be a huge mess, ideally breaking it down into smaller pieces\n\n # TODO: todo: implement proper description-providing system\n assert not provide_description # not implemented.\n if provide_description is not None:\n # nudge people to not specify it at all\n print(\n \"WARNING: provide_description is deprecated and will be removed in a future version in favor of description_dict\"\n )\n\n decontaminate = decontamination_ngrams_path is not None\n\n task_dict_items = [\n (name, task)\n for name, task in task_dict.items()\n if (task.has_validation_docs() or task.has_test_docs())\n ]\n #[('lambada_openai', )]\n results = collections.defaultdict(dict)\n versions = collections.defaultdict(dict)\n\n requests = collections.defaultdict(list)\n requests_origin = collections.defaultdict(list)\n\n overlaps = collections.defaultdict(list) # {task_name: contaminated_docs}\n\n # If we ever run into issues where the eval tasks don't fit in memory and we can't afford a machine with bigger\n # memory, we can always modify this plumbing to support that, but I didn't want to include it just yet because\n # over-engineering is bad (or we could make it write the requests to disk and then read them back out again\n # - probably using an sqlite db because of all the moving parts we have\n\n # TODO: we need unit tests & sanity checks or something to ensure that the return of `validation_docs` is stable\n docs = {}\n\n docs_for_decontamination = collections.defaultdict(list)\n\n # get lists of each type of request\n for task_name, task in task_dict_items:\n versions[task_name] = task.VERSION\n # default to test doc, fall back to val doc if validation unavailable\n # TODO: the test-fallback-to-val system isn't final, we should revisit it at some point\n if task.has_test_docs():\n task_doc_func = task.test_docs\n task_set = \"test\" # Required for caching in the decontamination\n # task_doc_func = task.validation_docs\n # task_set = \"val\" # Required for caching in the decontamination\n elif task.has_validation_docs():\n task_set = \"val\" # Required for caching in the decontamination\n task_doc_func = task.validation_docs\n else:\n raise RuntimeError(\"Task has neither test_docs nor validation_docs\")\n\n # deterministically shuffle docs and chop off the first `limit` because sometimes docs are in some kind of order\n task_docs = list(task_doc_func())\n rnd = random.Random()\n rnd.seed(42)\n rnd.shuffle(task_docs)\n\n description = (\n description_dict[task_name]\n if description_dict and task_name in description_dict\n else \"\"\n )\n\n\n for doc_id, doc in enumerate(itertools.islice(task_docs, 0, limit)):\n # for doc_id, doc in enumerate(itertools.islice(task_docs, 0, 3)):\n\n if decontaminate and task.should_decontaminate():\n docs_for_decontamination[(task_name, task_set)].append(\n task.doc_to_decontamination_query(doc)\n )\n docs[(task_name, doc_id)] = doc\n ctx = task.fewshot_context(\n doc=doc, num_fewshot=num_fewshot, rnd=rnd, description=description\n )\n reqs = task.construct_requests(doc, ctx) #(Req_loglikelihood('Car...at', ' Carlos')[1], Req_loglikelihood('Car...at', ' Carlos')[0] )\n\n\n if not isinstance(reqs, (list, tuple)):\n reqs = [reqs]\n for i, req in enumerate(reqs):\n requests[req.request_type].append(req) #req.request_type: loglikelihood\n # i: index in requests for a single task instance\n # doc_id: unique id that we can get back to a doc using `docs`\n requests_origin[req.request_type].append((i, task_name, doc, doc_id))\n\n # Compare all tasks/sets at once to ensure a single training set scan\n if decontaminate:\n from lm_eval.decontamination.decontaminate import get_train_overlap\n\n print(\"Finding train/test overlap, please wait...\")\n overlaps = get_train_overlap(\n docs_for_decontamination, decontamination_ngrams_path, limit\n )\n\n # all responses for each (task, doc)\n process_res_queue = collections.defaultdict(list)\n # execute each type of request\n for reqtype, reqs in requests.items():\n # TODO: right now, this code runs multiple separate LM requests for multiple Requests differing\n # only in index. We could implement some kind of caching, but that would be more of a band-aid\n # solution. we could also implement some kind of auto-grouping here;\n # they should end up next to each other.\n\n print(\"Running\", reqtype, \"requests\")\n resps = getattr(lm, reqtype)([req.args for req in reqs])\n resps = [\n x if req.index is None else x[req.index] for x, req in zip(resps, reqs)\n ]\n\n for resp, (i, task_name, doc, doc_id) in zip(resps, requests_origin[reqtype]):\n process_res_queue[(task_name, doc_id)].append((i, resp))\n\n vals = collections.defaultdict(list)\n\n # unpack results and sort back in order and return control to Task\n for (task_name, doc_id), requests in process_res_queue.items():\n requests.sort(key=lambda x: x[0])\n requests = [x[1] for x in requests]\n\n task = task_dict[task_name]\n doc = docs[(task_name, doc_id)]\n\n metrics = task.process_results(doc, requests)\n for metric, value in metrics.items():\n vals[(task_name, metric)].append(value)\n\n # Re-use the evaluation for the decontaminated set by just ignoring the overlaps\n if decontaminate and task_name in overlaps:\n if doc_id not in overlaps[task_name]:\n vals[(task_name, metric + decontaminate_suffix)].append(value)\n\n # aggregate results\n for (task_name, metric), items in vals.items():\n task = task_dict[task_name]\n real_metric = metric # key when looking up the metric with task.aggregation\n if metric.endswith(decontaminate_suffix):\n real_metric = metric.replace(\n decontaminate_suffix, \"\"\n ) # decontaminated still uses the same metric\n results[task_name][metric] = task.aggregation()[real_metric](items)\n\n # hotfix: bleu, chrf, ter seem to be really expensive to bootstrap\n # so we run them less iterations. still looking for a cleaner way to do this\n\n stderr = lm_eval.metrics.stderr_for_metric(\n metric=task.aggregation()[real_metric],\n bootstrap_iters=min(bootstrap_iters, 1000)\n if metric in [\"bleu\", \"chrf\", \"ter\"]\n else bootstrap_iters,\n )\n\n if stderr is not None:\n results[task_name][metric + \"_stderr\"] = stderr(items)\n\n return {\"results\": dict(results), \"versions\": dict(versions)}\n\n\ndef make_table(result_dict):\n \"\"\"Generate table of results.\"\"\"\n from pytablewriter import MarkdownTableWriter, LatexTableWriter\n\n md_writer = MarkdownTableWriter()\n latex_writer = LatexTableWriter()\n md_writer.headers = [\"Task\", \"Version\", \"Metric\", \"Value\", \"\", \"Stderr\"]\n latex_writer.headers = [\"Task\", \"Version\", \"Metric\", \"Value\", \"\", \"Stderr\"]\n\n values = []\n\n for k, dic in result_dict[\"results\"].items():\n version = result_dict[\"versions\"][k]\n for m, v in dic.items():\n if m.endswith(\"_stderr\"):\n continue\n\n if m + \"_stderr\" in dic:\n se = dic[m + \"_stderr\"]\n values.append([k, version, m, \"%.4f\" % v, \"±\", \"%.4f\" % se])\n else:\n values.append([k, version, m, \"%.4f\" % v, \"\", \"\"])\n k = \"\"\n version = \"\"\n md_writer.value_matrix = values\n latex_writer.value_matrix = values\n\n # todo: make latex table look good\n # print(latex_writer.dumps())\n\n return md_writer.dumps()\n" }, { "path": "lm_eval/metrics.py", "content": "import math\nfrom collections.abc import Iterable\n\nimport numpy as np\nimport sacrebleu\nimport sklearn.metrics\nimport random\n\n\ndef mean(arr):\n return sum(arr) / len(arr)\n\n\ndef pop_stddev(arr):\n mu = mean(arr)\n return math.sqrt(sum([(x - mu) ** 2 for x in arr]) / len(arr))\n\n\ndef sample_stddev(arr):\n mu = mean(arr)\n return math.sqrt(sum([(x - mu) ** 2 for x in arr]) / (len(arr) - 1))\n\n\ndef mean_stderr(arr):\n return sample_stddev(arr) / math.sqrt(len(arr))\n\n\ndef median(arr):\n return arr[len(arr) // 2]\n\n\ndef matthews_corrcoef(items):\n unzipped_list = list(zip(*items))\n golds = unzipped_list[0]\n preds = unzipped_list[1]\n return sklearn.metrics.matthews_corrcoef(golds, preds)\n\n\ndef f1_score(items):\n unzipped_list = list(zip(*items))\n golds = unzipped_list[0]\n preds = unzipped_list[1]\n fscore = sklearn.metrics.f1_score(golds, preds)\n\n return np.max(fscore)\n\n\ndef acc_all(items):\n # Only count as correct if all answers are labeled correctly for each question\n question_scoring_dict = {}\n preds = list(zip(*items))[0]\n docs = list(zip(*items))[1]\n\n for doc, pred in zip(docs, preds):\n paragraph_id = doc[\"idx\"][\"paragraph\"]\n question_id = doc[\"idx\"][\"question\"]\n if (paragraph_id, question_id) not in question_scoring_dict:\n question_scoring_dict[(paragraph_id, question_id)] = []\n\n gold_label = doc[\"label\"] == 1\n\n question_scoring_dict[(paragraph_id, question_id)].append(gold_label == pred)\n acc = np.mean([int(all(x)) for x in question_scoring_dict.values()])\n return acc\n\n\ndef acc_all_stderr(items):\n # Only count as correct if all answers are labeled correctly for each question\n question_scoring_dict = {}\n preds = list(zip(*items))[0]\n docs = list(zip(*items))[1]\n\n for doc, pred in zip(docs, preds):\n question_id = doc[\"idx\"][\"question\"]\n if question_id not in question_scoring_dict:\n question_scoring_dict[question_id] = []\n\n gold_label = doc[\"label\"] == 1\n question_scoring_dict[question_id].append(gold_label == pred)\n\n acc = mean_stderr([int(all(x)) for x in question_scoring_dict.values()])\n return acc\n\n\ndef metric_max_over_ground_truths(metric_fn, prediction, ground_truths):\n \"\"\"Compute max metric between prediction and each ground truth.\"\"\"\n scores_for_ground_truths = []\n for ground_truth in ground_truths:\n score = metric_fn(prediction, ground_truth)\n scores_for_ground_truths.append(score)\n return max(scores_for_ground_truths)\n\n\ndef perplexity(items):\n return math.exp(-mean(items))\n\n\ndef weighted_mean(items):\n a, b = zip(*items)\n return sum(a) / sum(b)\n\n\ndef weighted_perplexity(items):\n return math.exp(-weighted_mean(items))\n\n\ndef bits_per_byte(items):\n return -weighted_mean(items) / math.log(2)\n\n\ndef bleu(items):\n \"\"\"The Bilingual Evaluation Understudy Score, or BLEU for short, is a metric\n for evaluating a generated sentence to a reference sentence. It counts matching\n n-grams in the candidate translation to n-grams in the reference text, where\n 1-gram or unigram would be each token and a bigram comparison would be each\n word pair. The comparison is made regardless of word order\n Source: https://machinelearningmastery.com/calculate-bleu-score-for-text-python/\n Paper: https://www.aclweb.org/anthology/P02-1040/\n\n Higher is better\n \"\"\"\n refs = list(zip(*items))[0]\n preds = list(zip(*items))[1]\n refs, preds = _sacreformat(refs, preds)\n return sacrebleu.corpus_bleu(preds, refs).score\n\n\ndef chrf(items):\n \"\"\"chrF++ is a tool for automatic evaluation of machine translation output\n based on character n-gram precision and recall enhanced with word n-grams.\n Source: https://github.com/m-popovic/chrF\n Paper: https://www.aclweb.org/anthology/W15-3049.pdf\n\n Higher is better # TODO I think\n \"\"\"\n refs = list(zip(*items))[0]\n preds = list(zip(*items))[1]\n refs, preds = _sacreformat(refs, preds)\n return sacrebleu.corpus_chrf(preds, refs).score\n\n\ndef ter(items):\n \"\"\"Translation Error Rate is an error metric for machine translation that\n measures the number of edits required to change a system output into one\n of the references\n Source: http://www.cs.umd.edu/~snover/tercom/\n Paper: http://mt-archive.info/AMTA-2006-Snover.pdf\n\n Lower is better\n \"\"\"\n refs = list(zip(*items))[0]\n preds = list(zip(*items))[1]\n refs, preds = _sacreformat(refs, preds)\n return sacrebleu.corpus_ter(preds, refs).score\n\n\ndef is_non_str_iterable(obj):\n return isinstance(obj, Iterable) and not isinstance(obj, str)\n\n\ndef _sacreformat(refs, preds):\n \"\"\"Format refs and preds for sacrebleu corpus calculation. It is very particular\"\"\"\n # Sacrebleu expects (List[str], List[List[str])\n # e.g. sacrebleu.corpus_bleu([pred_t], [[ref1_stream], [ref2_stream], ...])\n\n # Note [ref1_stream] is the first reference for each pred.\n # So lists are size N and (M, N) for N preds and M possible refs for each pred\n # This is a different order of dimensions that I would expect\n\n # We expect refs to be List[str] or List[List[str]], the outer list corresponding to preds\n # Must become List[List[str]] with the inner list corresponding to preds\n if not is_non_str_iterable(refs):\n refs = list(refs)\n if not is_non_str_iterable(refs[0]):\n refs = [[ref] for ref in refs]\n refs = list(zip(*refs))\n # Note the number of refs in each ref list much match the number of preds\n\n # We expect preds to be List[str] or List[List[str]]. Must become List[str]\n if not is_non_str_iterable(preds):\n preds = list(preds)\n if is_non_str_iterable(preds[0]):\n assert len(preds[0]) == 1, f\"Pred must be a str, was {preds[0]}\"\n preds = [pred[0] for pred in preds]\n\n return refs, preds\n\n\n# stderr stuff\n\n\nclass _bootstrap_internal:\n def __init__(self, f, n):\n self.f = f\n self.n = n\n\n def __call__(self, v):\n i, xs = v\n rnd = random.Random()\n rnd.seed(i)\n res = []\n for _ in range(self.n):\n res.append(self.f(rnd.choices(xs, k=len(xs))))\n return res\n\n\ndef bootstrap_stderr(f, xs, iters):\n import multiprocessing as mp\n\n pool = mp.Pool(mp.cpu_count())\n # this gives a biased estimate of the stderr (i.e w/ the mean, it gives something\n # equivalent to stderr calculated without Bessel's correction in the stddev.\n # Unfortunately, I haven't been able to figure out what the right correction is\n # to make the bootstrap unbiased - i considered multiplying by sqrt(n/(n-1)) but\n # that would be ad-hoc and I can't prove that that would actually be an unbiased estimator)\n # Thankfully, shouldn't matter because our samples are pretty big usually anyways\n res = []\n chunk_size = min(1000, iters)\n from tqdm import tqdm\n\n print(\"bootstrapping for stddev:\", f.__name__)\n for bootstrap in tqdm(\n pool.imap(\n _bootstrap_internal(f, chunk_size),\n [(i, xs) for i in range(iters // chunk_size)],\n ),\n total=iters // chunk_size,\n ):\n # sample w replacement\n res.extend(bootstrap)\n\n pool.close()\n return sample_stddev(res)\n\n\ndef stderr_for_metric(metric, bootstrap_iters):\n bootstrappable = [\n median,\n matthews_corrcoef,\n f1_score,\n perplexity,\n bleu,\n chrf,\n ter,\n ]\n\n if metric in bootstrappable:\n return lambda x: bootstrap_stderr(metric, x, iters=bootstrap_iters)\n\n stderr = {mean: mean_stderr, acc_all: acc_all_stderr}\n\n return stderr.get(metric, None)\n\n\ndef yesno(x):\n if x:\n return \"yes\"\n else:\n return \"no\"\n" }, { "path": "lm_eval/models/__init__.py", "content": "from . import gpt2\nfrom . import gpt3\nfrom . import huggingface\nfrom . import textsynth\nfrom . import dummy\n\nMODEL_REGISTRY = {\n \"hf\": gpt2.HFLM,\n \"hf-causal\": huggingface.AutoCausalLM,\n \"hf-seq2seq\": huggingface.AutoSeq2SeqLM,\n \"gpt2\": gpt2.GPT2LM,\n \"gpt3\": gpt3.GPT3LM,\n \"textsynth\": textsynth.TextSynthLM,\n \"dummy\": dummy.DummyLM,\n}\n\n\ndef get_model(model_name):\n return MODEL_REGISTRY[model_name]\n" }, { "path": "lm_eval/models/dummy.py", "content": "import random\nfrom lm_eval.base import LM\n\n\nclass DummyLM(LM):\n def __init__(self):\n pass\n\n @classmethod\n def create_from_arg_string(cls, arg_string, additional_config=None):\n return cls()\n\n def loglikelihood(self, requests):\n res = []\n\n for _ in requests:\n res.append((-random.random(), False))\n\n return res\n\n def greedy_until(self, requests):\n res = []\n\n for ctx, _ in requests:\n res.append(\"lol\")\n assert ctx.strip() != \"\"\n\n return res\n\n def loglikelihood_rolling(self, requests):\n res = []\n\n for _ in requests:\n res.append(-random.random())\n\n return res\n" }, { "path": "lm_eval/models/gpt2.py", "content": "import torch\nimport transformers\nfrom lm_eval.base import BaseLM\n\n\nclass HFLM(BaseLM):\n def __init__(\n self,\n device=\"cuda\",\n pretrained=\"gpt2\",\n revision=\"main\",\n low_cpu_mem_usage=None,\n subfolder=None,\n tokenizer=None,\n batch_size=1,\n ):\n super().__init__()\n\n assert isinstance(device, str)\n assert isinstance(pretrained, str)\n assert isinstance(batch_size, int)\n\n if device:\n if device not in [\"cuda\", \"cpu\"]:\n device = int(device)\n self._device = torch.device(device)\n print(f\"Using device '{device}'\")\n else:\n print(\"Device not specified\")\n print(f\"Cuda Available? {torch.cuda.is_available()}\")\n self._device = (\n torch.device(\"cuda\")\n if torch.cuda.is_available()\n else torch.device(\"cpu\")\n )\n\n # TODO: update this to be less of a hack once subfolder is fixed in HF\n revision = revision + (\"/\" + subfolder if subfolder is not None else \"\")\n\n self.gpt2 = transformers.AutoModelForCausalLM.from_pretrained(\n pretrained, revision=revision, low_cpu_mem_usage=low_cpu_mem_usage\n ).to(self.device)\n self.gpt2.eval()\n\n self.tokenizer = transformers.AutoTokenizer.from_pretrained(\n pretrained if tokenizer is None else tokenizer,\n revision=revision,\n )\n\n assert isinstance(\n self.tokenizer,\n (\n transformers.GPT2Tokenizer,\n transformers.GPT2TokenizerFast,\n transformers.T5Tokenizer,\n transformers.T5TokenizerFast,\n ),\n ), \"this tokenizer has not been checked for compatibility yet!\"\n\n self.vocab_size = self.tokenizer.vocab_size\n\n if isinstance(\n self.tokenizer, (transformers.GPT2Tokenizer, transformers.GPT2TokenizerFast)\n ):\n assert self.tokenizer.encode(\"hello\\n\\nhello\") == [\n 31373,\n 198,\n 198,\n 31373,\n ], self.tokenizer.encode(\"hello\\n\\nhello\")\n\n # multithreading and batching\n self.batch_size_per_gpu = batch_size # todo: adaptive batch size\n\n # TODO: fix multi-gpu\n # gpus = torch.cuda.device_count()\n # if gpus > 1:\n # self.gpt2 = nn.DataParallel(self.gpt2)\n\n @property\n def eot_token_id(self):\n # we use EOT because end of *text* is more accurate for what we're doing than end of *sentence*\n return self.tokenizer.eos_token_id\n\n @property\n def max_length(self):\n try:\n return self.gpt2.config.n_ctx\n except AttributeError:\n # gptneoconfig doesn't have n_ctx apparently\n return self.gpt2.config.max_position_embeddings\n\n @property\n def max_gen_toks(self):\n return 256\n\n @property\n def batch_size(self):\n # TODO: fix multi-gpu\n return self.batch_size_per_gpu # * gpus\n\n @property\n def device(self):\n # TODO: fix multi-gpu\n return self._device\n\n def tok_encode(self, string: str):\n return self.tokenizer.encode(string, add_special_tokens=False)\n\n def tok_decode(self, tokens):\n return self.tokenizer.decode(tokens)\n\n def _model_call(self, inps):\n \"\"\"\n inps: a torch tensor of shape [batch, sequence]\n the size of sequence may vary from call to call\n\n returns: a torch tensor of shape [batch, sequence, vocab] with the\n logits returned from the model\n \"\"\"\n with torch.no_grad():\n return self.gpt2(inps)[0]\n\n def _model_generate(self, context, max_length, eos_token_id):\n return self.gpt2.generate(\n context, max_length=max_length, eos_token_id=eos_token_id, do_sample=False\n )\n\n\n# for backwards compatibility\nGPT2LM = HFLM\n" }, { "path": "lm_eval/models/gpt3.py", "content": "import os\nimport numpy as np\nimport transformers\nfrom lm_eval.base import BaseLM\nfrom lm_eval import utils\nfrom tqdm import tqdm\nimport time\n\n\ndef get_result(response, ctxlen):\n \"\"\"Process results from OpenAI API response.\n\n :param response: dict\n OpenAI API Response\n :param ctxlen: int\n Length of context (so we can slice them away and only keep the predictions)\n :return:\n continuation_logprobs: np.array\n Log probabilities of continuation tokens\n is_greedy: bool\n whether argmax matches given continuation exactly\n \"\"\"\n is_greedy = True\n logprobs = response[\"logprobs\"][\"token_logprobs\"]\n continuation_logprobs = sum(logprobs[ctxlen:])\n\n for i in range(ctxlen, len(response[\"logprobs\"][\"tokens\"])):\n token = response[\"logprobs\"][\"tokens\"][i]\n top_tokens = response[\"logprobs\"][\"top_logprobs\"][i]\n top_token = max(top_tokens.keys(), key=lambda x: top_tokens[x])\n if top_token != token:\n is_greedy = False\n break\n\n return continuation_logprobs, is_greedy\n\n\ndef oa_completion(**kwargs):\n \"\"\"Query OpenAI API for completion.\n\n Retry with back-off until they respond\n \"\"\"\n import openai\n\n backoff_time = 3\n while True:\n try:\n return openai.Completion.create(**kwargs)\n except openai.error.OpenAIError:\n import traceback\n\n traceback.print_exc()\n time.sleep(backoff_time)\n backoff_time *= 1.5\n\n\nclass GPT3LM(BaseLM):\n REQ_CHUNK_SIZE = 20\n\n def __init__(self, engine, truncate=False):\n \"\"\"\n\n :param engine: str\n OpenAI API engine (e.g. davinci)\n :param truncate: bool\n Truncate input if too long (if False and input is too long, throw error)\n \"\"\"\n super().__init__()\n\n import openai\n\n self.engine = engine\n self.tokenizer = transformers.GPT2TokenizerFast.from_pretrained(\"gpt2\")\n\n self.vocab_size = self.tokenizer.vocab_size\n\n # to make the annoying \"Using pad_token, but it is not set yet.\" error go away\n self.tokenizer.pad_token = \"<|endoftext|>\"\n assert self.tokenizer.encode(\"hello\\n\\nhello\") == [31373, 198, 198, 31373]\n self.truncate = truncate\n self.end_of_text_token_id = self.tokenizer.convert_tokens_to_ids(\n [\"<|endoftext|>\"]\n )[0]\n\n # Read from environment variable OPENAI_API_SECRET_KEY\n openai.api_key = os.environ[\"OPENAI_API_SECRET_KEY\"]\n\n @property\n def eot_token_id(self):\n return self.tokenizer.eos_token_id\n\n @property\n def max_length(self):\n # Note: the OpenAI API supports up to 2049 tokens, with the first token being the first input token\n return 2048\n\n @property\n def max_gen_toks(self):\n return 256\n\n @property\n def batch_size(self):\n # Isn't used because we override _loglikelihood_tokens\n raise NotImplementedError()\n\n @property\n def device(self):\n # Isn't used because we override _loglikelihood_tokens\n raise NotImplementedError()\n\n def tok_encode(self, string: str):\n return self.tokenizer.encode(string, add_special_tokens=False)\n\n def tok_decode(self, tokens):\n return self.tokenizer.decode(tokens)\n\n def _loglikelihood_tokens(self, requests, disable_tqdm=False):\n res = []\n\n def _collate(x):\n # this doesn't efficiently handle last-token differences yet, but those are kinda annoying because\n # it's not guaranteed that the 100 or so logprobs we get to see actually contain all the continuations\n # we care about and so we need some kind of backup for when it isn't\n toks = x[1] + x[2]\n return -len(toks), tuple(toks)\n\n re_ord = utils.Reorderer(requests, _collate)\n\n for chunk in tqdm(\n list(utils.chunks(re_ord.get_reordered(), self.REQ_CHUNK_SIZE)),\n disable=disable_tqdm,\n ):\n inps = []\n ctxlens = []\n for cache_key, context_enc, continuation_enc in chunk:\n # max_length+1 because the API takes up to 2049 tokens, including the first context token\n inp = (context_enc + continuation_enc)[-(self.max_length + 1) :]\n # TODO: the logic is much simpler if we just look at the length of continuation tokens\n ctxlen = len(context_enc) - max(\n 0, len(context_enc) + len(continuation_enc) - (self.max_length + 1)\n )\n\n inps.append(inp)\n ctxlens.append(ctxlen)\n\n response = oa_completion(\n engine=self.engine,\n prompt=inps,\n echo=True,\n max_tokens=0,\n temperature=0.0,\n logprobs=10,\n )\n\n for resp, ctxlen, (cache_key, context_enc, continuation_enc) in zip(\n response.choices, ctxlens, chunk\n ):\n answer = get_result(resp, ctxlen)\n\n res.append(answer)\n\n # partial caching\n if cache_key is not None:\n self.cache_hook.add_partial(\"loglikelihood\", cache_key, answer)\n\n return re_ord.get_original(res)\n\n def greedy_until(self, requests):\n if not requests:\n return []\n res = []\n\n def _collate(x):\n toks = self.tok_encode(x[0])\n return len(toks), x[0]\n\n re_ord = utils.Reorderer(requests, _collate)\n\n def sameuntil_chunks(xs, size):\n ret = []\n lastuntil = xs[0][1]\n for x in xs:\n if len(ret) >= size or x[1] != lastuntil:\n yield ret, lastuntil\n ret = []\n lastuntil = x[1]\n ret.append(x)\n\n if ret:\n yield ret, lastuntil\n\n # todo: more intelligent batching for heterogeneous `until`\n for chunk, until in tqdm(\n list(sameuntil_chunks(re_ord.get_reordered(), self.REQ_CHUNK_SIZE))\n ):\n inps = []\n for context, _ in chunk:\n context_enc = self.tok_encode(context)\n inp = context_enc[-(self.max_length - self.max_gen_toks) :]\n inps.append(inp)\n\n response = oa_completion(\n engine=self.engine,\n prompt=inps,\n max_tokens=self.max_gen_toks,\n temperature=0.0,\n logprobs=10,\n stop=until,\n )\n\n for resp, (context, until_) in zip(response.choices, chunk):\n s = resp[\"text\"]\n\n for term in until_:\n s = s.split(term)[0]\n\n # partial caching\n self.cache_hook.add_partial(\"greedy_until\", (context, until_), s)\n\n res.append(s)\n\n return re_ord.get_original(res)\n\n def _model_call(self, inps):\n # Isn't used because we override _loglikelihood_tokens\n raise NotImplementedError()\n\n def _model_generate(self, context, max_length, eos_token_id):\n # Isn't used because we override greedy_until\n raise NotImplementedError()\n" }, { "path": "lm_eval/models/huggingface.py", "content": "import math\nimport torch\nimport torch.nn.functional as F\nimport transformers\nfrom typing import List, Mapping, NewType, Optional, Tuple, Union\nfrom tqdm import tqdm\n\nfrom transformers import BatchEncoding\n\nfrom lm_eval import utils\nfrom lm_eval.base import BaseLM\n\nTokenSequence = Union[List[int], torch.LongTensor, torch.Tensor, BatchEncoding]\n\n_DeviceMapping = NewType(\"DeviceMapping\", Mapping[str, Union[int, str, torch.device]])\n\n\ndef _get_accelerate_args(\n device_map_option: Optional[str] = \"auto\",\n max_memory_per_gpu: Optional[Union[int, str]] = None,\n max_cpu_memory: Optional[Union[int, str]] = None,\n offload_folder: Optional[str] = \"./offload\",\n) -> dict:\n \"\"\"Returns the kwargs needed to apply `accelerate` in `AutoModel.from_pretrained`.\"\"\"\n max_memory = {}\n if max_memory_per_gpu is not None:\n max_memory_per_gpu_map = {\n device_idx: max_memory_per_gpu\n for device_idx in range(torch.cuda.device_count())\n }\n max_memory.update(max_memory_per_gpu_map)\n if max_cpu_memory is not None:\n max_memory[\"cpu\"] = max_cpu_memory\n\n args = {}\n if max_memory:\n args[\"max_memory\"] = max_memory\n args[\"device_map\"] = device_map_option\n args[\"offload_folder\"] = offload_folder\n return args\n\n\ndef _get_dtype(\n dtype: Union[str, torch.dtype], config: Optional[transformers.AutoConfig] = None\n) -> torch.dtype:\n \"\"\"Converts `dtype` from `str` to torch.dtype when possible.\"\"\"\n if dtype is None and config is not None:\n _torch_dtype = config.torch_dtype\n elif isinstance(dtype, str) and dtype != \"auto\":\n # Convert `str` args torch dtype: `float16` -> `torch.float16`\n _torch_dtype = getattr(torch, dtype)\n else:\n _torch_dtype = dtype\n return _torch_dtype\n\n\nclass HuggingFaceAutoLM(BaseLM):\n AUTO_CONFIG_CLASS: transformers.AutoConfig = transformers.AutoConfig\n AUTO_TOKENIZER_CLASS: transformers.AutoTokenizer = transformers.AutoTokenizer\n AUTO_MODEL_CLASS: transformers.AutoModel = None\n\n # Default max sequence length setting for when no `max_length` is provided\n # or no max length config setting is found in the model or tokenizer.\n _DEFAULT_MAX_LENGTH: int = 2048\n\n def __init__(\n self,\n pretrained: str,\n tokenizer: Optional[str] = None,\n subfolder: Optional[str] = None,\n revision: Optional[str] = \"main\",\n batch_size: Optional[int] = 1,\n max_gen_toks: Optional[int] = 256,\n max_length: Optional[int] = None,\n add_special_tokens: Optional[bool] = None,\n use_accelerate: Optional[bool] = False,\n device_map_option: Optional[str] = \"auto\",\n max_memory_per_gpu: Optional[Union[int, str]] = None,\n max_cpu_memory: Optional[Union[int, str]] = None,\n offload_folder: Optional[str] = \"./offload\",\n dtype: Optional[Union[str, torch.dtype]] = None,\n device: Optional[Union[int, str]] = \"cuda\",\n ):\n \"\"\"Initializes a HuggingFace `AutoModel` and `AutoTokenizer` for evaluation.\n Args:\n pretrained (str):\n The HuggingFace Hub model ID name or the path to a pre-trained\n model to load. This is effectively the `pretrained_model_name_or_path`\n argument of `from_pretrained` in the HuggingFace `transformers` API.\n add_special_tokens (bool, optional, defaults to True):\n Whether to add special tokens to the input sequences. If `None`, the\n default value will be set to `True` for seq2seq models (e.g. T5) and\n `False` for causal models.\n WARNING: Evaluating causal models with `add_special_tokens=True` is\n currently __not__ supported.\n > Large model loading `accelerate` arguments\n use_accelerate (bool, optional, defaults to False):\n If True, uses the `accelerate` library to load a large model across\n multiple devices.\n device_map_option (str, optional, defaults to \"auto\"):\n The device map option to use when loading the model with\n `accelerate`.\n Options:\n \"auto\", \"balanced\", \"balanced_low_0\", \"sequential\"\n See the `accelerate` docs for more details on these options:\n https://huggingface.co/docs/accelerate/v0.12.0/en/usage_guides/big_modeling#designing-a-device-map\n max_memory_per_gpu (Union[int, str], optional, defaults to None):\n The maximum memory available for each GPU in bytes as `int` or in\n the format f\"{significand}{unit_symbol}\" where {unit_symbol} is\n any of [\"GB\", \"MB\", \"GIB\", \"MIB\"]. Refer to the `max_memory` arg in\n the \"Parameters for big model inference\" section of the following\n docs:\n https://huggingface.co/docs/transformers/v4.20.1/en/main_classes/model#large-model-loading\n max_cpu_memory (Union[int, str], optional, defaults to None):\n The maximum available CPU RAM in bytes as `int` or in the format\n f\"{significand}{unit_symbol}\" where {unit_symbol} is any of\n [\"GB\", \"MB\", \"GIB\", \"MIB\"]. Refer to the `max_memory` arg in the\n \"Parameters for big model inference\" section of the following docs:\n https://huggingface.co/docs/transformers/v4.20.1/en/main_classes/model#large-model-loading\n offload_folder (str, optional, defaults to \"./offload\"):\n The folder to offload weights into if `device_map` contains any\n \"disk\" value.\n dtype (Union[str, torch.dtype], optional, defaults to None):):\n Converts the model weights to `dtype`, if specified. Strings get\n converted to `torch.dtype` objects (e.g. `float16` -> `torch.float16`).\n Use `dtype=\"auto\"` to derive the type from the model’s weights.\n \"\"\"\n super().__init__()\n\n assert isinstance(pretrained, str)\n assert isinstance(device, str)\n assert isinstance(batch_size, int)\n if (\n add_special_tokens is not None\n and self.AUTO_MODEL_CLASS is transformers.AutoModelForCausalLM\n ):\n # TODO: Support evaluating causal models with special tokens. Currently,\n # this is not possible because the `_loglikelihood_tokens()` method for\n # causal LMs makes a no-special-tokens assumption given that contexts\n # and labels/continuations are tokenized separately without special\n # tokens, concatenated, and then processed as inputs.\n assert (\n not add_special_tokens\n ), \"Evaluating causal models with `add_special_tokens=True` is currently not supported.\"\n\n self._batch_size = batch_size # TODO: Adaptive batch size\n self._max_gen_toks = max_gen_toks\n self._max_length = max_length\n self._config = self.AUTO_CONFIG_CLASS.from_pretrained(\n pretrained,\n revision=revision + (\"/\" + subfolder if subfolder is not None else \"\"),\n )\n\n self._add_special_tokens = add_special_tokens\n self.tokenizer = self._create_auto_tokenizer(\n pretrained=pretrained,\n revision=revision,\n subfolder=subfolder,\n tokenizer=tokenizer,\n )\n self.tokenizer.model_max_length = self.max_length\n\n accelerate_kwargs = {}\n if use_accelerate:\n accelerate_kwargs = _get_accelerate_args(\n device_map_option,\n max_memory_per_gpu,\n max_cpu_memory,\n offload_folder,\n )\n self.model = self._create_auto_model(\n pretrained=pretrained,\n revision=revision,\n subfolder=subfolder,\n torch_dtype=_get_dtype(dtype, self._config),\n **accelerate_kwargs,\n )\n self.model.eval()\n torch.set_grad_enabled(False)\n\n self._device = device\n if use_accelerate and \"lm_head\" in self.model.hf_device_map:\n # `accelerate` can place `lm_head` weights on a different device than\n # the user specified one so we force `self._device` to be the same as\n # `lm_head`'s.\n self._device = self.model.hf_device_map[\"lm_head\"]\n if not use_accelerate:\n self.model.to(self._device)\n\n def _create_auto_model(\n self,\n *,\n pretrained: str,\n revision: str,\n subfolder: str,\n device_map: Optional[Union[str, _DeviceMapping]] = None,\n max_memory: Optional[dict] = None,\n offload_folder: Optional[str] = None,\n torch_dtype: Optional[Union[str, torch.dtype]] = None,\n ) -> transformers.AutoModel:\n \"\"\"Returns a pre-trained pytorch model from a pre-trained model configuration.\"\"\"\n model = self.AUTO_MODEL_CLASS.from_pretrained(\n pretrained,\n revision=revision + (\"/\" + subfolder if subfolder is not None else \"\"),\n device_map=device_map,\n max_memory=max_memory,\n offload_folder=offload_folder,\n torch_dtype=torch_dtype,\n )\n return model\n\n def _create_auto_tokenizer(\n self,\n *,\n pretrained: str,\n revision: str,\n subfolder: str,\n tokenizer: Optional[str] = None,\n ) -> transformers.PreTrainedTokenizer:\n \"\"\"Returns a pre-trained tokenizer from a pre-trained tokenizer configuration.\"\"\"\n tokenizer = self.AUTO_TOKENIZER_CLASS.from_pretrained(\n pretrained if tokenizer is None else tokenizer,\n revision=revision + (\"/\" + subfolder if subfolder is not None else \"\"),\n )\n tokenizer.pad_token = tokenizer.eos_token\n return tokenizer\n\n @property\n def add_special_tokens(self) -> bool:\n \"\"\"Whether to include special tokens in encoded text. This should be\n determined by whether or not the model was trained with special tokens.\n TODO: Remove these conditionals once HuggingFace supports a way to\n check whether or not an arbitrary model was trained with special tokens.\n \"\"\"\n if self._add_special_tokens is not None:\n return self._add_special_tokens\n elif self.AUTO_MODEL_CLASS is transformers.AutoModelForCausalLM:\n return False\n elif self.AUTO_MODEL_CLASS is transformers.AutoModelForSeq2SeqLM:\n return True\n else:\n raise ValueError(\n \"Could not determine `add_special_tokens` value from the model \"\n \"class. Set to `True` or `False` depending on whether the model \"\n \"was pre-trained with special tokens.\"\n )\n\n @property\n def eot_token(self) -> str:\n return self.tokenizer.eos_token\n\n @property\n def eot_token_id(self) -> int:\n return self.tokenizer.eos_token_id\n\n @property\n def max_gen_toks(self) -> int:\n return self._max_gen_toks\n\n @property\n def max_length(self) -> int:\n \"\"\"Return the maximum sequence length of the model.\n NOTE: Different model configurations have different max sequence length\n attribute names.\n - n_positions: (CTRLConfig)\n - max_position_embeddings: (BartConfig, RoFormerConfig)\n - n_ctx: (GPT2Config)\n NOTE: For relative position encoded models you should specify the max\n sequence length of the model in the constructor via `max_length`.\n \"\"\"\n if self._max_length is not None:\n return self._max_length\n # Try to get the sequence length from the model config.\n seqlen_config_attrs = (\"n_positions\", \"max_position_embeddings\", \"n_ctx\")\n for attr in seqlen_config_attrs:\n if hasattr(self._config, attr):\n return getattr(self._config, attr)\n if hasattr(self.tokenizer, \"model_max_length\"):\n return self.tokenizer.model_max_length\n return self._DEFAULT_MAX_LENGTH\n\n @property\n def batch_size(self) -> int:\n # TODO: Add adaptive batch size.\n return self._batch_size # * gpus\n\n @property\n def device(self) -> Union[int, str, torch.device]:\n return self._device\n\n def tok_encode(self, string: str) -> TokenSequence:\n # TODO: Merge `tok_encode_batch` here.\n return self.tokenizer.encode(string, add_special_tokens=self.add_special_tokens)\n\n def tok_encode_batch(self, strings: List[str]) -> TokenSequence:\n return self.tokenizer(\n strings,\n padding=True,\n add_special_tokens=self.add_special_tokens,\n return_tensors=\"pt\",\n )\n\n def tok_decode(self, tokens: torch.LongTensor) -> List[str]:\n return self.tokenizer.batch_decode(tokens, skip_special_tokens=True)\n\n def greedy_until(self, requests: List[Tuple[str, dict]]) -> List[str]:\n def _collate(x):\n tokens = self.tok_encode(x[0])\n return len(tokens), x[0]\n\n results = []\n reorder = utils.Reorderer(requests, _collate)\n for chunk in utils.chunks(\n tqdm(reorder.get_reordered(), disable=False), self.batch_size\n ):\n context = [c[0] for c in chunk]\n request_args = chunk[0][1]\n stop_sequences = request_args[\"stop_sequences\"]\n max_generation_length = request_args[\"max_generation_length\"]\n num_fewshot = request_args[\"num_fewshot\"]\n\n assert (\n isinstance(max_generation_length, int) or max_generation_length is None\n )\n assert isinstance(stop_sequences, list) or stop_sequences is None\n assert isinstance(num_fewshot, int) or num_fewshot is None\n\n # TODO: Find a better way to handle stop sequences for 0-shot.\n if stop_sequences is None or num_fewshot == 0:\n until = [self.eot_token]\n else:\n until = stop_sequences + [self.eot_token]\n\n if max_generation_length is None:\n max_tokens = self.max_gen_toks\n else:\n max_tokens = max_generation_length\n\n token_context = self.tok_encode_batch(context)\n\n responses = self._model_generate(\n inputs=token_context,\n max_tokens=max_tokens,\n stop=until,\n )\n responses = self.tok_decode(responses.tolist())\n\n for response in responses:\n # Ensure the generated responses do not contain the stop sequences.\n for term in until:\n response = response.split(term)[0]\n # partial caching\n self.cache_hook.add_partial(\"greedy_until\", (context, until), response)\n results.append(response)\n return reorder.get_original(results)\n\n\nclass AutoCausalLM(HuggingFaceAutoLM):\n \"\"\"Causal language modeling.\n You can find a set of supported models in the HF documentation:\n https://huggingface.co/docs/transformers/main/model_doc/auto#transformers.AutoModelForCausalLM\n \"\"\"\n\n AUTO_MODEL_CLASS = transformers.AutoModelForCausalLM\n\n def _create_auto_tokenizer(\n self,\n *,\n pretrained: str,\n revision: str,\n subfolder: str,\n tokenizer: Optional[str] = None,\n ) -> transformers.PreTrainedTokenizer:\n tokenizer = super()._create_auto_tokenizer(\n pretrained=pretrained,\n revision=revision,\n subfolder=subfolder,\n tokenizer=tokenizer,\n )\n tokenizer.padding_side = \"left\"\n return tokenizer\n\n def _model_call(\n self, inputs: TokenSequence, labels: Optional[TokenSequence] = None\n ) -> TokenSequence:\n return self.model(inputs)[\"logits\"]\n\n def _model_generate(\n self,\n inputs: transformers.BatchEncoding,\n max_tokens: int,\n stop: Optional[List[str]] = None,\n ) -> TokenSequence:\n # Ensure that the context does not encroach into the `space`\n # for the generation.\n input_ids = inputs[\"input_ids\"][:, self.max_gen_toks - self.max_length :]\n attention_mask = inputs[\"attention_mask\"][\n :, self.max_gen_toks - self.max_length :\n ]\n input_ids = input_ids.to(self.device)\n attention_mask = attention_mask.to(self.device)\n\n stopping_criteria = stop_sequences_criteria(\n self.tokenizer, stop, input_ids.shape[1], input_ids.shape[0]\n )\n\n generations = self.model.generate(\n input_ids=input_ids,\n attention_mask=attention_mask,\n # GPT style models require the `generate` `max_length` arg to include the\n # context length, so we instead set `max_new_tokens` which is the number\n # of new tokens to generate, excluding the current number of tokens.\n max_new_tokens=max_tokens,\n stopping_criteria=stopping_criteria,\n do_sample=False,\n )\n return utils.select_continuation_from_batch_left_padding(\n generations, max_context_size=inputs[\"input_ids\"].size(1)\n )\n\n\nclass AutoSeq2SeqLM(HuggingFaceAutoLM):\n \"\"\"Seq2Seq language modeling.\n You can find a set of supported models in the following documentation:\n https://huggingface.co/docs/transformers/main/model_doc/auto#transformers.AutoModelForSeq2SeqLM\n \"\"\"\n\n AUTO_MODEL_CLASS = transformers.AutoModelForSeq2SeqLM\n\n @property\n def max_length(self) -> int:\n \"\"\"Return the maximum sequence length of the model.\n TODO: Currently only works for relative position encoded Seq2Seq models.\n \"\"\"\n if self._max_length is not None:\n return self._max_length\n return self._DEFAULT_MAX_LENGTH\n\n def loglikelihood(\n self, requests: List[Tuple[str, str]]\n ) -> List[Tuple[float, bool]]:\n new_requests = []\n for chunk in utils.chunks(requests, self.batch_size):\n context, continuation = zip(*chunk)\n\n # Fill empty contexts with the EOT token.\n context = [\n f\"{self.eot_token}\" if len(text) == 0 else text for text in context\n ]\n context_enc = self.tok_encode_batch(context)\n for key in context_enc:\n context_enc[key] = context_enc[key][:, -self.max_length :]\n\n # Remove leading whitespace introduced by the default\n # `text_target_separator` since the context and continuation\n # will not be concatenated as a single (decoder) input.\n continuation = [text.lstrip() for text in continuation]\n continuation_enc = self.tok_encode_batch(list(continuation))\n for key in continuation_enc:\n continuation_enc[key] = continuation_enc[key][:, -self.max_length :]\n\n new_requests.append(\n ((context, continuation), context_enc, continuation_enc)\n )\n return self._loglikelihood_tokens(new_requests)\n\n def loglikelihood_rolling(self, requests: List[Tuple[str, str]]) -> List[float]:\n loglikelihoods = []\n for (string,) in tqdm(requests):\n rolling_token_windows = list(\n map(\n utils.make_disjoint_window,\n utils.get_rolling_token_windows(\n token_list=self.tok_encode(string),\n prefix_token=self.eot_token_id,\n max_seq_len=self.max_length,\n context_len=1,\n ),\n )\n )\n contexts, conts = utils.split_and_pad_windows(\n rolling_token_windows,\n pad_token_id=self.eot_token_id,\n max_seq_len=self.max_length,\n )\n # Manually create BatchEncoding tensors with attention masks as\n # expected by `self._model_call` in `self._loglikelihood_tokens`.\n contexts_enc = torch.Tensor(contexts).long()\n contexts_enc = transformers.tokenization_utils_base.BatchEncoding(\n {\n \"input_ids\": contexts_enc,\n \"attention_mask\": (contexts_enc != self.eot_token_id).long(),\n }\n )\n conts_enc = torch.Tensor(conts).long()\n conts_enc = transformers.tokenization_utils_base.BatchEncoding(\n {\n \"input_ids\": conts_enc,\n \"attention_mask\": (conts_enc != self.eot_token_id).long(),\n }\n )\n # TODO: Extract out this call so it only gets called once and also\n # somehow figure out partial caching for.\n rolling_token_windows_request = [\n ((contexts, conts), contexts_enc, conts_enc)\n ]\n string_nll = self._loglikelihood_tokens(\n rolling_token_windows_request, disable_tqdm=True\n )\n string_nll = [x[0] for x in string_nll] # discard is_greedy\n string_nll = sum(string_nll)\n loglikelihoods.append(string_nll)\n return loglikelihoods\n\n def _loglikelihood_tokens(\n self,\n requests: List[Tuple[Tuple[str, str], TokenSequence, TokenSequence]],\n disable_tqdm: Optional[bool] = False,\n ) -> List[Tuple[float, bool]]:\n results = []\n for chunk in tqdm(\n requests, total=math.ceil(len(requests)), disable=disable_tqdm\n ):\n cache_keys, inputs_tokens, targets_tokens = chunk\n inputs_tokens = inputs_tokens.to(self.device)\n targets_tokens = targets_tokens.to(self.device)\n outputs = self._model_call(inputs=inputs_tokens, labels=targets_tokens)\n log_softmaxes = F.log_softmax(outputs.logits, dim=-1)\n\n output_iterator = zip(\n zip(cache_keys[0], cache_keys[1]),\n log_softmaxes,\n targets_tokens[\"input_ids\"],\n targets_tokens[\"attention_mask\"],\n )\n for cache_key, log_softmax, target_tokens, target_mask in output_iterator:\n length = target_mask.sum()\n log_softmax = log_softmax[:length]\n target_tokens = target_tokens[:length]\n greedy_tokens = log_softmax.argmax(dim=-1)\n max_equal = (greedy_tokens == target_tokens).all()\n target_logits = torch.gather(\n log_softmax, 1, target_tokens.unsqueeze(-1)\n ).squeeze(-1)\n answer = (float(target_logits.sum()), bool(max_equal))\n results.append(answer)\n if cache_key is not None:\n self.cache_hook.add_partial(\"loglikelihood\", cache_key, answer)\n return results\n\n def _model_call(\n self, inputs: TokenSequence, labels: Optional[TokenSequence] = None\n ) -> TokenSequence:\n return self.model(**inputs, labels=labels[\"input_ids\"])\n\n def _model_generate(\n self,\n inputs: transformers.BatchEncoding,\n max_tokens: int,\n stop: Optional[List[str]] = None,\n ) -> TokenSequence:\n input_ids = inputs[\"input_ids\"][:, -self.max_length :].to(self.device)\n attention_mask = inputs[\"attention_mask\"][:, -self.max_length :].to(self.device)\n\n # Generate one token to calculate the number of start tokens prepended to decoder_input_ids\n # (leaving this here in case the below assumption is violated in the future)\n # one_tok_gen = self.model.generate(\n # input_ids=torch.zeros((1, 1), dtype=torch.int),\n # min_length=2,\n # max_new_tokens=1,\n # ).squeeze()\n # initial_decoder_input_length = len(one_tok_gen) - 1\n\n # Assume that there will always only be one token in the decoder inputs, assumption holds for existing HF models\n stopping_criteria = stop_sequences_criteria(\n self.tokenizer, stop, 1, input_ids.shape[0]\n )\n\n generations = self.model.generate(\n input_ids=input_ids,\n attention_mask=attention_mask,\n max_new_tokens=max_tokens,\n stopping_criteria=stopping_criteria,\n do_sample=False,\n )\n return generations\n\n\nclass MultiTokenEOSCriteria(transformers.StoppingCriteria):\n \"\"\"Criteria to stop on the specified multi-token sequence.\"\"\"\n\n def __init__(\n self,\n sequence: str,\n tokenizer: transformers.PreTrainedTokenizer,\n initial_decoder_input_length: int,\n batch_size: int,\n ):\n self.initial_decoder_input_length = initial_decoder_input_length\n self.done_tracker = [False] * batch_size\n self.sequence = sequence\n self.sequence_ids = tokenizer.encode(sequence, add_special_tokens=False)\n self.sequence_id_len = len(self.sequence_ids)\n self.tokenizer = tokenizer\n\n def __call__(self, input_ids, scores, **kwargs) -> bool:\n # For efficiency, we compare the last n tokens where n is the number of tokens in the stop_sequence\n lookback_ids_batch = input_ids[:, self.initial_decoder_input_length :][\n :, -self.sequence_id_len :\n ]\n\n lookback_tokens_batch = self.tokenizer.batch_decode(lookback_ids_batch)\n\n for i, done in enumerate(self.done_tracker):\n if not done:\n self.done_tracker[i] = self.sequence in lookback_tokens_batch[i]\n return False not in self.done_tracker\n\n\ndef stop_sequences_criteria(\n tokenizer: transformers.PreTrainedTokenizer,\n stop_sequences: List[str],\n initial_decoder_input_length: int,\n batch_size: int,\n) -> transformers.StoppingCriteriaList:\n return transformers.StoppingCriteriaList(\n [\n *[\n MultiTokenEOSCriteria(\n sequence, tokenizer, initial_decoder_input_length, batch_size\n )\n for sequence in stop_sequences\n ],\n ]\n )\n\n" }, { "path": "lm_eval/models/textsynth.py", "content": "\"\"\" TextSynth API\nImplementation provided by Fabrice Bellard:\n https://github.com/EleutherAI/lm-evaluation-harness/issues/295\n\nIn order to use the API, you must have a valid TextSynth account and\nenough credits.\n\nExample usage:\n\n python main.py --model textsynth --model_args engine=gptj_6B --no_cache --tasks piqa\n\nHomepage: https://textsynth.com/index.html\n\"\"\"\nimport logging\nimport os\nimport requests as _requests\nimport time\nfrom tqdm import tqdm\nfrom lm_eval.base import BaseLM\n\n\nlogger = logging.getLogger(__name__)\n\n\ndef textsynth_completion(**kwargs):\n \"\"\"Query TextSynth API for completion.\n Retry with back-off until they respond.\n \"\"\"\n backoff_time = 3\n while True:\n try:\n return _requests.post(**kwargs)\n except _requests.exceptions.RequestException:\n import traceback\n\n traceback.print_exc()\n time.sleep(backoff_time)\n backoff_time *= 1.5\n\n\nclass TextSynthLM(BaseLM):\n def __init__(self, engine, truncate=False):\n \"\"\"\n :param engine: str\n TextSynth API engine (e.g. `gptj_6B`)\n :param truncate: bool\n Truncate input if too long (if False and input is too long, throw error)\n \"\"\"\n super().__init__()\n\n self.engine = engine\n self.truncate = truncate\n self.api_url = \"https://api.textsynth.com\"\n # Read from environment variable TEXTSYNTH_API_SECRET_KEY\n self.api_key = os.environ[\"TEXTSYNTH_API_SECRET_KEY\"]\n\n @property\n def eot_token_id(self):\n # Isn't used because we override loglikelihood, loglikelihood_rolling and greedy_until\n raise NotImplementedError()\n\n @property\n def max_length(self):\n # NOTE: Turn on truncation to avoid errors on long inputs.\n return 2048\n\n @property\n def max_gen_toks(self):\n return 256\n\n @property\n def batch_size(self):\n # Isn't used because we override loglikelihood, loglikelihood_rolling and greedy_until\n raise NotImplementedError()\n\n @property\n def device(self):\n # Isn't used because we override loglikelihood, loglikelihood_rolling and greedy_until\n raise NotImplementedError()\n\n def tok_encode(self, string: str):\n # Isn't used because we override loglikelihood, loglikelihood_rolling and greedy_until\n raise NotImplementedError()\n\n def tok_decode(self, tokens):\n # Isn't used because we override loglikelihood, loglikelihood_rolling and greedy_until\n raise NotImplementedError()\n\n def loglikelihood(self, requests):\n res = []\n for context, continuation in tqdm(requests):\n response = textsynth_completion(\n url=self.api_url + \"/v1/engines/\" + self.engine + \"/logprob\",\n headers={\"Authorization\": \"Bearer \" + self.api_key},\n json={\"context\": context, \"continuation\": continuation},\n )\n resp = response.json()\n if \"logprob\" in resp:\n logprob = resp[\"logprob\"]\n is_greedy = resp[\"is_greedy\"]\n res.append((logprob, is_greedy))\n else:\n logger.error(\n f\"The following response does not contain `logprobs`. Got:\\n{resp}\"\n )\n assert False\n return res\n\n def loglikelihood_rolling(self, requests):\n # TODO: The TextSynth API does not support tokenized inputs so we cannot\n # manually partition long contexts into smaller rolling windows as\n # done for other models derived from `BaseLM`. Override this method\n # with a windowing scheme that works for direct string inputs.\n raise NotImplementedError(\n \"`loglikelihood_rolling` is currently not supported due to lack of \"\n \"input tokenization support from TextSynth.\"\n )\n\n def greedy_until(self, requests):\n if not requests:\n return []\n\n res = []\n for request in tqdm(requests):\n inp = request[0]\n until = request[1]\n response = textsynth_completion(\n url=self.api_url + \"/v1/engines/\" + self.engine + \"/completions\",\n headers={\"Authorization\": \"Bearer \" + self.api_key},\n json={\n \"prompt\": inp,\n \"max_tokens\": self.max_gen_toks,\n \"top_k\": 1,\n \"stop\": until,\n },\n )\n resp = response.json()\n if \"text\" in resp:\n s = resp[\"text\"]\n res.append(s)\n else:\n logger.error(\n f\"The following response does not contain generated `text`. \"\n \"Got:\\n{resp}\"\n )\n assert False\n return res\n\n def _model_call(self, inps):\n # Isn't used because we override _loglikelihood_tokens\n raise NotImplementedError()\n\n def _model_generate(self, context, max_length, eos_token_id):\n # Isn't used because we override greedy_until\n raise NotImplementedError()\n" }, { "path": "lm_eval/quantizer/irqlora.py", "content": "from tqdm import tqdm\nimport peft\nimport torch\nimport operator\nimport numpy as np\nimport bitsandbytes as bnb\nfrom peft.tuners.lora import LoraLayer\nfrom functools import reduce # Required in Python 3\nimport bitsandbytes.functional as bnb_F\nfrom torch import Tensor\nfrom scipy.stats import norm\nfrom bitsandbytes.functional import create_fp8_map, create_dynamic_map\n\ncache_folder_path = ''\nmodule_num = 0\nsigma = 1 / norm.ppf(torch.linspace(0.9677083, 0.5, 9)[:-1]).tolist()[0]\n\ndef replace_to_qlora_model(model, model_fp, blocksize2=256, tau_range=0.1, tau_n=100):\n model.model = _replace_with_ours_lora_4bit_linear(model.model, model_fp=model_fp, blocksize2=blocksize2, tau_range=tau_range, tau_n=tau_n)\n return model\n\ndef prod(iterable):\n return reduce(operator.mul, iterable, 1)\n \nnormal_map_fp8 = create_dynamic_map()\ndef quantize_tensor(X, L, idx=False):\n L = L.to(X.device)\n X_shape = X.shape\n X_expanded = X.reshape(-1, 1)\n L_reshaped = L.reshape(1, -1)\n abs_diff = torch.abs(X_expanded - L_reshaped)\n min_index = torch.argmin(abs_diff, dim=-1)\n min_index = torch.tensor(min_index, dtype=torch.uint8).to(L.device).reshape(X_shape)\n return min_index\n\ndef dequantize_tensor(X, L):\n L = L.to(X.device)\n return torch.index_select(L, dim=0, index=torch.as_tensor(X, dtype=torch.int32).reshape(-1)).reshape(X.shape)\n\n@torch.no_grad()\ndef nf4_quant(weight, weight_shape, tau, compress_statistics, quant_type, device):\n weight = weight.reshape(-1, 256, 64).to(device)\n tau = tau.reshape(-1, 256, 1).to(device)\n _weight = (weight - tau).reshape(weight_shape)\n nf4_weight = bnb.nn.Params4bit(_weight, requires_grad=False, compress_statistics=compress_statistics, quant_type=quant_type).cuda(0)\n tau2 = tau.abs().max(dim=1, keepdim=True)[0]\n tau1 = quantize_tensor(tau / tau2, normal_map_fp8)\n return nf4_weight, tau1.reshape(-1, 256), tau2.reshape(-1, 1)\n\n@torch.no_grad()\ndef evaluate_entropy(weight_int8, blocksize):\n device = weight_int8.device\n _weight_int8 = weight_int8.reshape(-1, 1)\n weight_nf4 = torch.cat((_weight_int8//16, _weight_int8%16), 1).reshape(1, -1, blocksize)\n weight_nf4_repeat = weight_nf4.repeat(16, 1, 1).to(device)\n values = torch.tensor(range(16)).reshape(16, 1, 1).to(device)\n freqs = (weight_nf4_repeat==values).sum(dim=-1, keepdim=True) / blocksize\n entropy = -freqs * torch.log2(freqs)\n entropy = torch.where(torch.isnan(entropy), 0, entropy)\n entropy = entropy.sum(dim=0)\n return entropy\n\n@torch.no_grad()\ndef search(fp_weight: Tensor, fp_weight_shape, compress_statistics, quant_type, device, tau_range=0.1, tau_n=51, blocksize=64, blocksize2=256):\n fp_weight = fp_weight.reshape(-1, blocksize2, blocksize).to(device)\n tau0 = fp_weight.median(2, keepdim=True)[0] # [-1, 256, 1]\n absmax = (fp_weight - tau0).abs().max(2, keepdim=True)[0]\n \n entropy_max, factor_best = None, None\n for factor in tqdm(np.linspace(-tau_range*sigma, tau_range*sigma, tau_n*2+1)):\n tau = factor * absmax + tau0\n nf4_weight, _, _ = nf4_quant(fp_weight, fp_weight_shape, tau, compress_statistics, quant_type, device)\n entropy = evaluate_entropy(nf4_weight, blocksize)\n \n if entropy_max is None:\n entropy_max = entropy\n factor_best = torch.full_like(entropy, factor)\n else:\n factor_best = torch.where(entropy > entropy_max, factor, factor_best)\n entropy_max = torch.max(entropy_max, entropy)\n \n tau = factor_best.reshape(-1, 256, 1) * absmax + tau0\n nf4_weight, tau1, tau2 = nf4_quant(fp_weight, fp_weight_shape, tau, compress_statistics, quant_type, device)\n return nf4_weight, tau1, tau2\n\nclass IRQLoraLinear4bit(bnb.nn.Linear4bit, LoraLayer):\n def __init__(\n self, old_model, model_fp=None, blocksize2=256, tau_range=0.1, tau_n=51\n ):\n for key, value in old_model.__dict__.items():\n setattr(self, key, value)\n\n fp_weight = model_fp.weight.data.contiguous().to('cpu')\n fp_weight_shape = fp_weight.shape\n \n compress_statistics, quant_type, device = self.base_layer.weight.compress_statistics, self.base_layer.weight.quant_type, self.base_layer.weight.device\n del self.base_layer.weight, model_fp\n torch.cuda.empty_cache()\n\n self.base_layer.weight, self.base_layer.tau_quant, self.base_layer.tau_absmax = search(\n fp_weight=fp_weight,\n fp_weight_shape=fp_weight_shape,\n compress_statistics=compress_statistics,\n quant_type=quant_type,\n device=device,\n tau_range=tau_range, tau_n=tau_n,\n blocksize2=blocksize2\n )\n self.base_layer.tau_quant = self.base_layer.tau_quant.to(device)\n self.base_layer.tau_absmax = self.base_layer.tau_absmax.to(device)\n \n del fp_weight\n torch.cuda.empty_cache()\n \n self.lora_default_A_scale = torch.nn.Parameter(torch.zeros([1], dtype=self.lora_A.default.weight.dtype).to(self.base_layer.weight.device), requires_grad=True)\n self.lora_default_B_scale = torch.nn.Parameter(torch.zeros([1], dtype=self.lora_A.default.weight.dtype).to(self.base_layer.weight.device), requires_grad=True)\n \n def forward(self, x: torch.Tensor):\n \n if self.base_layer.bias is not None and self.base_layer.bias.dtype != x.dtype:\n self.base_layer.bias.data = self.base_layer.bias.data.to(x.dtype)\n\n if getattr(self.base_layer.weight, 'quant_state', None) is None:\n print('FP4 quantization state not initialized. Please call .cuda() or .to(device) on the LinearFP4 layer first.')\n inp_dtype = x.dtype\n if self.base_layer.compute_dtype is not None:\n x = x.to(self.base_layer.compute_dtype)\n\n bias = None if self.base_layer.bias is None else self.base_layer.bias.to(self.base_layer.compute_dtype)\n \n with torch.no_grad():\n fp_B = bnb_F.dequantize_fp4(self.base_layer.weight, self.base_layer.weight.quant_state).to(x.dtype)\n tau = (dequantize_tensor(self.base_layer.tau_quant, normal_map_fp8).reshape(-1, 256, 1) * self.base_layer.tau_absmax.reshape(-1, 1, 1)).to(fp_B.device)\n blocksize = torch.prod(torch.tensor(fp_B.shape)) / torch.prod(torch.tensor(tau.shape))\n fp_B = (fp_B.reshape(-1, blocksize.int().item()) + tau.reshape(-1, 1)).reshape(fp_B.shape).to(x.dtype)\n out = torch.nn.functional.linear(x, fp_B, bias)\n\n out = out.to(inp_dtype)\n result = out\n \n if self.disable_adapters or self.active_adapter[0] not in self.lora_A.keys():\n return result\n elif self.r[self.active_adapter[0]] > 0:\n result = result.clone()\n if not torch.is_autocast_enabled():\n expected_dtype = result.dtype\n x = x.to(self.lora_A[self.active_adapter[0]].weight.dtype)\n x = self.lora_A[self.active_adapter[0]](self.lora_dropout[self.active_adapter[0]](x)) + self.lora_default_A_scale * x.reshape([_ for _ in x.shape[:-1]] + [self.lora_A[self.active_adapter[0]].out_features] + [-1]).mean(dim=-1)\n x = (self.lora_B[self.active_adapter[0]](x).reshape([_ for _ in x.shape] + [-1]) + self.lora_default_B_scale * x.unsqueeze(-1)).reshape([_ for _ in x.shape[:-1]] + [-1])\n output = x.to(expected_dtype) * self.scaling[self.active_adapter[0]]\n else:\n x = self.lora_A[self.active_adapter[0]](self.lora_dropout[self.active_adapter[0]](x)) + self.lora_default_A_scale * x.reshape([_ for _ in x.shape[:-1]] + [self.lora_A[self.active_adapter[0]].out_features] + [-1]).mean(dim=-1)\n x = (self.lora_B[self.active_adapter[0]](x).reshape([_ for _ in x.shape] + [-1]) + self.lora_default_B_scale * x.unsqueeze(-1)).reshape([_ for _ in x.shape[:-1]] + [-1])\n output = x * self.scaling[self.active_adapter[0]]\n result += output\n\n return result\n\ndef _replace_with_ours_lora_4bit_linear(\n model, current_key_name=None, model_fp=None, blocksize2=256, tau_range=0.5, tau_n=51\n):\n for name, module in model.named_children():\n if current_key_name is None:\n current_key_name = []\n current_key_name.append(name)\n\n if isinstance(module, peft.tuners.lora.Linear4bit):\n model._modules[name] = IRQLoraLinear4bit(model._modules[name], model_fp=model_fp._modules[name], blocksize2=blocksize2, tau_range=tau_range, tau_n=tau_n)\n \n if len(list(module.children())) > 0:\n if name in model_fp._modules:\n _ = _replace_with_ours_lora_4bit_linear(\n module,\n current_key_name, model_fp._modules[name], blocksize2, tau_range, tau_n\n )\n else:\n _ = _replace_with_ours_lora_4bit_linear(\n module,\n current_key_name, None, blocksize2, tau_range, tau_n\n )\n current_key_name.pop(-1)\n return model\n" }, { "path": "lm_eval/tasks/__init__.py", "content": "from pprint import pprint\nfrom typing import List, Union\n\nimport sacrebleu\nimport lm_eval.base\n\nfrom . import superglue\nfrom . import glue\nfrom . import arc\nfrom . import coqa\nfrom . import race\nfrom . import webqs\nfrom . import anli\nfrom . import wsc273\nfrom . import winogrande\nfrom . import quac\nfrom . import hellaswag\nfrom . import swag\nfrom . import openbookqa\nfrom . import squad\nfrom . import naturalqs\nfrom . import sat\nfrom . import arithmetic\nfrom . import lambada\nfrom . import piqa\nfrom . import prost\nfrom . import mc_taco\nfrom . import triviaqa\nfrom . import pubmedqa\nfrom . import sciq\nfrom . import qasper\nfrom . import qa4mre\nfrom . import translation\nfrom . import headqa\nfrom . import mathqa\nfrom . import hendrycks_ethics\nfrom . import drop\nfrom . import unscramble\nfrom . import logiqa\nfrom . import hendrycks_test\nfrom . import hendrycks_math\nfrom . import cbt\nfrom . import lambada_cloze\nfrom . import pile\nfrom . import wikitext\nfrom . import lambada_multilingual\nfrom . import mutual\nfrom . import truthfulqa\nfrom . import blimp\nfrom . import asdiv\nfrom . import gsm8k\nfrom . import storycloze\nfrom . import toxigen\nfrom . import crowspairs\n\n########################################\n# Translation tasks\n########################################\n\n# 6 total\ngpt3_translation_benchmarks = {\n \"wmt14\": [\"en-fr\", \"fr-en\"], # French\n \"wmt16\": [\"en-ro\", \"ro-en\", \"de-en\", \"en-de\"], # German, Romanian\n}\n\n\n# 28 total\nselected_translation_benchmarks = {\n **gpt3_translation_benchmarks,\n \"wmt20\": sacrebleu.get_langpairs_for_testset(\"wmt20\"),\n \"iwslt17\": [\"en-ar\", \"ar-en\"], # Arabic\n}\n\n# 319 total\nall_translation_benchmarks = {\n ts: sacrebleu.get_langpairs_for_testset(ts)\n for ts in sacrebleu.get_available_testsets()\n}\n\n\n########################################\n# All tasks\n########################################\n\n\nTASK_REGISTRY = {\n # GLUE\n \"cola\": glue.CoLA,\n \"mnli\": glue.MNLI,\n \"mnli_mismatched\": glue.MNLIMismatched,\n \"mrpc\": glue.MRPC,\n \"rte\": glue.RTE,\n \"qnli\": glue.QNLI,\n \"qqp\": glue.QQP,\n # \"stsb\": glue.STSB, # not implemented yet\n \"sst\": glue.SST,\n \"wnli\": glue.WNLI,\n # SuperGLUE\n \"boolq\": superglue.BoolQ,\n \"cb\": superglue.CommitmentBank,\n \"copa\": superglue.Copa,\n \"multirc\": superglue.MultiRC,\n \"record\": superglue.ReCoRD,\n \"wic\": superglue.WordsInContext,\n \"wsc\": superglue.SGWinogradSchemaChallenge,\n # Order by benchmark/genre?\n \"coqa\": coqa.CoQA,\n \"drop\": drop.DROP,\n \"lambada_openai\": lambada.LambadaOpenAI,\n \"lambada_standard\": lambada.LambadaStandard,\n \"lambada_openai_cloze\": lambada_cloze.LambadaOpenAICloze,\n \"lambada_standard_cloze\": lambada_cloze.LambadaStandardCloze,\n # multilingual lambada\n **lambada_multilingual.construct_tasks(),\n \"wikitext\": wikitext.WikiText,\n # \"cbt-cn\": cbt.CBTCN, # disabled pending context length fix\n # \"cbt-ne\": cbt.CBTNE, # disabled pending context length fix\n \"piqa\": piqa.PiQA,\n \"prost\": prost.PROST,\n \"mc_taco\": mc_taco.MCTACO,\n # Science related\n \"pubmedqa\": pubmedqa.Pubmed_QA,\n \"sciq\": sciq.SciQ,\n \"qasper\": qasper.QASPER,\n \"qa4mre_2011\": qa4mre.QA4MRE_2011,\n \"qa4mre_2012\": qa4mre.QA4MRE_2012,\n \"qa4mre_2013\": qa4mre.QA4MRE_2013,\n \"triviaqa\": triviaqa.TriviaQA,\n \"arc_easy\": arc.ARCEasy,\n \"arc_challenge\": arc.ARCChallenge,\n # \"quac\": quac.QuAC, # not implemented yet\n \"logiqa\": logiqa.LogiQA,\n \"hellaswag\": hellaswag.HellaSwag,\n \"swag\": swag.SWAG,\n \"openbookqa\": openbookqa.OpenBookQA,\n \"squad2\": squad.SQuAD2,\n \"race\": race.RACE,\n # \"naturalqs\": naturalqs.NaturalQs, # not implemented yet\n \"headqa\": headqa.HeadQAEsDeprecated, # for backwards compat - headqa used to default to es\n \"headqa_es\": headqa.HeadQAEs,\n \"headqa_en\": headqa.HeadQAEn,\n \"mathqa\": mathqa.MathQA,\n \"webqs\": webqs.WebQs,\n \"wsc273\": wsc273.WinogradSchemaChallenge273,\n \"winogrande\": winogrande.Winogrande,\n \"anli_r1\": anli.ANLIRound1,\n \"anli_r2\": anli.ANLIRound2,\n \"anli_r3\": anli.ANLIRound3,\n \"ethics_cm\": hendrycks_ethics.EthicsCM,\n \"ethics_deontology\": hendrycks_ethics.EthicsDeontology,\n \"ethics_justice\": hendrycks_ethics.EthicsJustice,\n \"ethics_utilitarianism_original\": hendrycks_ethics.EthicsUtilitarianismOriginal,\n \"ethics_utilitarianism\": hendrycks_ethics.EthicsUtilitarianism,\n \"ethics_virtue\": hendrycks_ethics.EthicsVirtue,\n \"truthfulqa_mc\": truthfulqa.TruthfulQAMultipleChoice,\n \"truthfulqa_gen\": truthfulqa.TruthfulQAGeneration,\n # dialogue\n \"mutual\": mutual.MuTual,\n \"mutual_plus\": mutual.MuTualPlus,\n # math\n \"math_algebra\": hendrycks_math.MathAlgebra,\n \"math_counting_and_prob\": hendrycks_math.MathCountingAndProbability,\n \"math_geometry\": hendrycks_math.MathGeometry,\n \"math_intermediate_algebra\": hendrycks_math.MathIntermediateAlgebra,\n \"math_num_theory\": hendrycks_math.MathNumberTheory,\n \"math_prealgebra\": hendrycks_math.MathPrealgebra,\n \"math_precalc\": hendrycks_math.MathPrecalculus,\n \"math_asdiv\": asdiv.Asdiv,\n \"gsm8k\": gsm8k.GradeSchoolMath8K,\n # arithmetic\n \"arithmetic_2da\": arithmetic.Arithmetic2DPlus,\n \"arithmetic_2ds\": arithmetic.Arithmetic2DMinus,\n \"arithmetic_3da\": arithmetic.Arithmetic3DPlus,\n \"arithmetic_3ds\": arithmetic.Arithmetic3DMinus,\n \"arithmetic_4da\": arithmetic.Arithmetic4DPlus,\n \"arithmetic_4ds\": arithmetic.Arithmetic4DMinus,\n \"arithmetic_5da\": arithmetic.Arithmetic5DPlus,\n \"arithmetic_5ds\": arithmetic.Arithmetic5DMinus,\n \"arithmetic_2dm\": arithmetic.Arithmetic2DMultiplication,\n \"arithmetic_1dc\": arithmetic.Arithmetic1DComposite,\n # TODO Perhaps make these groups of tasks\n # e.g. anli, arithmetic, openai_translations, harness_translations\n # hendrycksTest (57 tasks)\n **hendrycks_test.create_all_tasks(),\n # e.g. wmt14-fr-en\n **translation.create_tasks_from_benchmarks(gpt3_translation_benchmarks),\n # chef's selection, mostly wmt20\n **translation.create_tasks_from_benchmarks(selected_translation_benchmarks),\n # Word Scrambling and Manipulation Tasks\n \"anagrams1\": unscramble.Anagrams1,\n \"anagrams2\": unscramble.Anagrams2,\n \"cycle_letters\": unscramble.CycleLetters,\n \"random_insertion\": unscramble.RandomInsertion,\n \"reversed_words\": unscramble.ReversedWords,\n # Pile\n \"pile_arxiv\": pile.PileArxiv,\n \"pile_books3\": pile.PileBooks3,\n \"pile_bookcorpus2\": pile.PileBookCorpus2,\n \"pile_dm-mathematics\": pile.PileDmMathematics,\n \"pile_enron\": pile.PileEnron,\n \"pile_europarl\": pile.PileEuroparl,\n \"pile_freelaw\": pile.PileFreeLaw,\n \"pile_github\": pile.PileGithub,\n \"pile_gutenberg\": pile.PileGutenberg,\n \"pile_hackernews\": pile.PileHackernews,\n \"pile_nih-exporter\": pile.PileNIHExporter,\n \"pile_opensubtitles\": pile.PileOpenSubtitles,\n \"pile_openwebtext2\": pile.PileOpenWebText2,\n \"pile_philpapers\": pile.PilePhilPapers,\n \"pile_pile-cc\": pile.PilePileCc,\n \"pile_pubmed-abstracts\": pile.PilePubmedAbstracts,\n \"pile_pubmed-central\": pile.PilePubmedCentral,\n \"pile_stackexchange\": pile.PileStackExchange,\n \"pile_uspto\": pile.PileUspto,\n \"pile_ubuntu-irc\": pile.PileUbuntuIrc,\n \"pile_wikipedia\": pile.PileWikipedia,\n \"pile_youtubesubtitles\": pile.PileYoutubeSubtitles,\n # BLiMP\n \"blimp_adjunct_island\": blimp.BlimpAdjunctIsland,\n \"blimp_anaphor_gender_agreement\": blimp.BlimpAnaphorGenderAgreement,\n \"blimp_anaphor_number_agreement\": blimp.BlimpAnaphorNumberAgreement,\n \"blimp_animate_subject_passive\": blimp.BlimpAnimateSubjectPassive,\n \"blimp_animate_subject_trans\": blimp.BlimpAnimateSubjectTrans,\n \"blimp_causative\": blimp.BlimpCausative,\n \"blimp_complex_NP_island\": blimp.BlimpComplex_NPIsland,\n \"blimp_coordinate_structure_constraint_complex_left_branch\": blimp.BlimpCoordinateStructureConstraintComplexLeftBranch,\n \"blimp_coordinate_structure_constraint_object_extraction\": blimp.BlimpCoordinateStructureConstraintObjectExtraction,\n \"blimp_determiner_noun_agreement_1\": blimp.BlimpDeterminerNounAgreement_1,\n \"blimp_determiner_noun_agreement_2\": blimp.BlimpDeterminerNounAgreement_2,\n \"blimp_determiner_noun_agreement_irregular_1\": blimp.BlimpDeterminerNounAgreementIrregular_1,\n \"blimp_determiner_noun_agreement_irregular_2\": blimp.BlimpDeterminerNounAgreementIrregular_2,\n \"blimp_determiner_noun_agreement_with_adj_2\": blimp.BlimpDeterminerNounAgreementWithAdj_2,\n \"blimp_determiner_noun_agreement_with_adj_irregular_1\": blimp.BlimpDeterminerNounAgreementWithAdjIrregular_1,\n \"blimp_determiner_noun_agreement_with_adj_irregular_2\": blimp.BlimpDeterminerNounAgreementWithAdjIrregular_2,\n \"blimp_determiner_noun_agreement_with_adjective_1\": blimp.BlimpDeterminerNounAgreementWithAdjective_1,\n \"blimp_distractor_agreement_relational_noun\": blimp.BlimpDistractorAgreementRelationalNoun,\n \"blimp_distractor_agreement_relative_clause\": blimp.BlimpDistractorAgreementRelativeClause,\n \"blimp_drop_argument\": blimp.BlimpDropArgument,\n \"blimp_ellipsis_n_bar_1\": blimp.BlimpEllipsisNBar_1,\n \"blimp_ellipsis_n_bar_2\": blimp.BlimpEllipsisNBar_2,\n \"blimp_existential_there_object_raising\": blimp.BlimpExistentialThereObjectRaising,\n \"blimp_existential_there_quantifiers_1\": blimp.BlimpExistentialThereQuantifiers_1,\n \"blimp_existential_there_quantifiers_2\": blimp.BlimpExistentialThereQuantifiers_2,\n \"blimp_existential_there_subject_raising\": blimp.BlimpExistentialThereSubjectRaising,\n \"blimp_expletive_it_object_raising\": blimp.BlimpExpletiveItObjectRaising,\n \"blimp_inchoative\": blimp.BlimpInchoative,\n \"blimp_intransitive\": blimp.BlimpIntransitive,\n \"blimp_irregular_past_participle_adjectives\": blimp.BlimpIrregularPastParticipleAdjectives,\n \"blimp_irregular_past_participle_verbs\": blimp.BlimpIrregularPastParticipleVerbs,\n \"blimp_irregular_plural_subject_verb_agreement_1\": blimp.BlimpIrregularPluralSubjectVerbAgreement_1,\n \"blimp_irregular_plural_subject_verb_agreement_2\": blimp.BlimpIrregularPluralSubjectVerbAgreement_2,\n \"blimp_left_branch_island_echo_question\": blimp.BlimpLeftBranchIslandEchoQuestion,\n \"blimp_left_branch_island_simple_question\": blimp.BlimpLeftBranchIslandSimpleQuestion,\n \"blimp_matrix_question_npi_licensor_present\": blimp.BlimpMatrixQuestionNpiLicensorPresent,\n \"blimp_npi_present_1\": blimp.BlimpNpiPresent_1,\n \"blimp_npi_present_2\": blimp.BlimpNpiPresent_2,\n \"blimp_only_npi_licensor_present\": blimp.BlimpOnlyNpiLicensorPresent,\n \"blimp_only_npi_scope\": blimp.BlimpOnlyNpiScope,\n \"blimp_passive_1\": blimp.BlimpPassive_1,\n \"blimp_passive_2\": blimp.BlimpPassive_2,\n \"blimp_principle_A_c_command\": blimp.BlimpPrinciple_ACCommand,\n \"blimp_principle_A_case_1\": blimp.BlimpPrinciple_ACase_1,\n \"blimp_principle_A_case_2\": blimp.BlimpPrinciple_ACase_2,\n \"blimp_principle_A_domain_1\": blimp.BlimpPrinciple_ADomain_1,\n \"blimp_principle_A_domain_2\": blimp.BlimpPrinciple_ADomain_2,\n \"blimp_principle_A_domain_3\": blimp.BlimpPrinciple_ADomain_3,\n \"blimp_principle_A_reconstruction\": blimp.BlimpPrinciple_AReconstruction,\n \"blimp_regular_plural_subject_verb_agreement_1\": blimp.BlimpRegularPluralSubjectVerbAgreement_1,\n \"blimp_regular_plural_subject_verb_agreement_2\": blimp.BlimpRegularPluralSubjectVerbAgreement_2,\n \"blimp_sentential_negation_npi_licensor_present\": blimp.BlimpSententialNegationNpiLicensorPresent,\n \"blimp_sentential_negation_npi_scope\": blimp.BlimpSententialNegationNpiScope,\n \"blimp_sentential_subject_island\": blimp.BlimpSententialSubjectIsland,\n \"blimp_superlative_quantifiers_1\": blimp.BlimpSuperlativeQuantifiers_1,\n \"blimp_superlative_quantifiers_2\": blimp.BlimpSuperlativeQuantifiers_2,\n \"blimp_tough_vs_raising_1\": blimp.BlimpToughVsRaising_1,\n \"blimp_tough_vs_raising_2\": blimp.BlimpToughVsRaising_2,\n \"blimp_transitive\": blimp.BlimpTransitive,\n \"blimp_wh_island\": blimp.BlimpWhIsland,\n \"blimp_wh_questions_object_gap\": blimp.BlimpWhQuestionsObjectGap,\n \"blimp_wh_questions_subject_gap\": blimp.BlimpWhQuestionsSubjectGap,\n \"blimp_wh_questions_subject_gap_long_distance\": blimp.BlimpWhQuestionsSubjectGapLongDistance,\n \"blimp_wh_vs_that_no_gap\": blimp.BlimpWhVsThatNoGap,\n \"blimp_wh_vs_that_no_gap_long_distance\": blimp.BlimpWhVsThatNoGapLongDistance,\n \"blimp_wh_vs_that_with_gap\": blimp.BlimpWhVsThatWithGap,\n \"blimp_wh_vs_that_with_gap_long_distance\": blimp.BlimpWhVsThatWithGapLongDistance,\n \"toxigen\": toxigen.ToxiGen,\n \"crows_pairs_english\": crowspairs.CrowsPairsEnglish,\n \"crows_pairs_english_race_color\": crowspairs.CrowsPairsEnglishRaceColor,\n \"crows_pairs_english_socioeconomic\": crowspairs.CrowsPairsEnglishSocioeconomic,\n \"crows_pairs_english_gender\": crowspairs.CrowsPairsEnglishGender,\n \"crows_pairs_english_age\": crowspairs.CrowsPairsEnglishAge,\n \"crows_pairs_english_religion\": crowspairs.CrowsPairsEnglishReligion,\n \"crows_pairs_english_disability\": crowspairs.CrowsPairsEnglishDisability,\n \"crows_pairs_english_sexual_orientation\": crowspairs.CrowsPairsEnglishSexualOrientation,\n \"crows_pairs_english_nationality\": crowspairs.CrowsPairsEnglishNationality,\n \"crows_pairs_english_physical_appearance\": crowspairs.CrowsPairsEnglishPhysicalAppearance,\n \"crows_pairs_english_autre\": crowspairs.CrowsPairsEnglishAutre,\n \"crows_pairs_french\": crowspairs.CrowsPairsFrench,\n \"crows_pairs_french_race_color\": crowspairs.CrowsPairsFrenchRaceColor,\n \"crows_pairs_french_socioeconomic\": crowspairs.CrowsPairsFrenchSocioeconomic,\n \"crows_pairs_french_gender\": crowspairs.CrowsPairsFrenchGender,\n \"crows_pairs_french_age\": crowspairs.CrowsPairsFrenchAge,\n \"crows_pairs_french_religion\": crowspairs.CrowsPairsFrenchReligion,\n \"crows_pairs_french_disability\": crowspairs.CrowsPairsFrenchDisability,\n \"crows_pairs_french_sexual_orientation\": crowspairs.CrowsPairsFrenchSexualOrientation,\n \"crows_pairs_french_nationality\": crowspairs.CrowsPairsFrenchNationality,\n \"crows_pairs_french_physical_appearance\": crowspairs.CrowsPairsFrenchPhysicalAppearance,\n \"crows_pairs_french_autre\": crowspairs.CrowsPairsFrenchAutre,\n # Requires manual download of data.\n # \"storycloze_2016\": storycloze.StoryCloze2016,\n # \"storycloze_2018\": storycloze.StoryCloze2018,\n # \"sat\": sat.SATAnalogies,\n}\n\n\nALL_TASKS = sorted(list(TASK_REGISTRY))\n\n\ndef get_task(task_name):\n try:\n return TASK_REGISTRY[task_name]\n except KeyError:\n print(\"Available tasks:\")\n pprint(TASK_REGISTRY)\n raise KeyError(f\"Missing task {task_name}\")\n\n\ndef get_task_name_from_object(task_object):\n for name, class_ in TASK_REGISTRY.items():\n if class_ is task_object:\n return name\n\n # this gives a mechanism for non-registered tasks to have a custom name anyways when reporting\n return (\n task_object.EVAL_HARNESS_NAME\n if hasattr(task_object, \"EVAL_HARNESS_NAME\")\n else type(task_object).__name__\n )\n\n\ndef get_task_dict(task_name_list: List[Union[str, lm_eval.base.Task]]):\n task_name_dict = {\n task_name: get_task(task_name)()\n for task_name in task_name_list\n if isinstance(task_name, str)\n }\n task_name_from_object_dict = {\n get_task_name_from_object(task_object): task_object\n for task_object in task_name_list\n if not isinstance(task_object, str)\n }\n assert set(task_name_dict.keys()).isdisjoint(set(task_name_from_object_dict.keys()))\n return {**task_name_dict, **task_name_from_object_dict}\n" }, { "path": "lm_eval/tasks/anli.py", "content": "\"\"\"\nAdversarial NLI: A New Benchmark for Natural Language Understanding\nhttps://arxiv.org/pdf/1910.14599.pdf\n\nAdversarial NLI (ANLI) is a dataset collected via an iterative, adversarial\nhuman-and-model-in-the-loop procedure. It consists of three rounds that progressively\nincrease in difficulty and complexity, and each question-answer includes annotator-\nprovided explanations.\n\nHomepage: \"https://github.com/facebookresearch/anli\"\n\"\"\"\nimport numpy as np\nfrom lm_eval.base import rf, Task\nfrom lm_eval.metrics import mean\n\n\n_CITATION = \"\"\"\n@inproceedings{nie-etal-2020-adversarial,\n title = \"Adversarial {NLI}: A New Benchmark for Natural Language Understanding\",\n author = \"Nie, Yixin and\n Williams, Adina and\n Dinan, Emily and\n Bansal, Mohit and\n Weston, Jason and\n Kiela, Douwe\",\n booktitle = \"Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics\",\n year = \"2020\",\n publisher = \"Association for Computational Linguistics\",\n}\n\"\"\"\n\n\nclass ANLIBase(Task):\n VERSION = 0\n DATASET_PATH = \"anli\"\n DATASET_NAME = None\n SPLIT = None\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return True\n\n def training_docs(self):\n if self.has_training_docs():\n if self._training_docs is None:\n self._training_docs = list(self.dataset[\"train_r\" + str(self.SPLIT)])\n return self._training_docs\n\n def validation_docs(self):\n if self.has_validation_docs():\n return self.dataset[\"dev_r\" + str(self.SPLIT)]\n\n def test_docs(self):\n if self.has_test_docs():\n return self.dataset[\"test_r\" + str(self.SPLIT)]\n\n def doc_to_text(self, doc):\n # OA does this a bit weirdly: they prepend \"anli 1: anli 1: \" to the beginning\n # of the prompt (yes, repeating it!). also, \" True, False, or Neither?\" is directly\n # appended onto the question, with no \"Answer:\" or even a newline. Do we *really*\n # want to do it exactly as OA did?\n return (\n doc[\"premise\"]\n + \"\\nQuestion: \"\n + doc[\"hypothesis\"]\n + \" True, False, or Neither?\\nAnswer:\"\n )\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"premise\"]\n\n def doc_to_target(self, doc):\n # True = entailment\n # False = contradiction\n # Neither = neutral\n return \" \" + [\"True\", \"Neither\", \"False\"][doc[\"label\"]]\n\n def construct_requests(self, doc, ctx):\n \"\"\"Uses RequestFactory to construct Requests and returns an iterable of\n Requests which will be sent to the LM.\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param ctx: str\n The context string, generated by fewshot_context. This includes the natural\n language description, as well as the few shot examples, and the question\n part of the document for `doc`.\n \"\"\"\n ll_true, _ = rf.loglikelihood(ctx, \" True\")\n ll_neither, _ = rf.loglikelihood(ctx, \" Neither\")\n ll_false, _ = rf.loglikelihood(ctx, \" False\")\n return ll_true, ll_neither, ll_false\n\n def process_results(self, doc, results):\n \"\"\"Take a single document and the LM results and evaluates, returning a\n dict where keys are the names of submetrics and values are the values of\n the metric for that one document\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param results:\n The results of the requests created in construct_requests.\n \"\"\"\n gold = doc[\"label\"]\n pred = np.argmax(results)\n return {\"acc\": pred == gold}\n\n def aggregation(self):\n \"\"\"\n :returns: {str: [float] -> float}\n A dictionary where keys are the names of submetrics and values are\n functions that aggregate a list of metrics\n \"\"\"\n return {\"acc\": mean}\n\n def higher_is_better(self):\n \"\"\"\n :returns: {str: bool}\n A dictionary where keys are the names of submetrics and values are\n whether a higher value of the submetric is better\n \"\"\"\n return {\"acc\": True}\n\n\nclass ANLIRound1(ANLIBase):\n SPLIT = 1\n\n\nclass ANLIRound2(ANLIBase):\n SPLIT = 2\n\n\nclass ANLIRound3(ANLIBase):\n SPLIT = 3\n" }, { "path": "lm_eval/tasks/arc.py", "content": "\"\"\"\nThink you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge\nhttps://arxiv.org/pdf/1803.05457.pdf\n\nThe ARC dataset consists of 7,787 science exam questions drawn from a variety\nof sources, including science questions provided under license by a research\npartner affiliated with AI2. These are text-only, English language exam questions\nthat span several grade levels as indicated in the files. Each question has a\nmultiple choice structure (typically 4 answer options). The questions are sorted\ninto a Challenge Set of 2,590 “hard” questions (those that both a retrieval and\na co-occurrence method fail to answer correctly) and an Easy Set of 5,197 questions.\n\nHomepage: https://allenai.org/data/arc\n\"\"\"\nfrom lm_eval.base import MultipleChoiceTask\n\n\n_CITATION = \"\"\"\n@article{Clark2018ThinkYH,\n title={Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge},\n author={Peter Clark and Isaac Cowhey and Oren Etzioni and Tushar Khot and Ashish Sabharwal and Carissa Schoenick and Oyvind Tafjord},\n journal={ArXiv},\n year={2018},\n volume={abs/1803.05457}\n}\n\"\"\"\n\n\nclass ARCEasy(MultipleChoiceTask):\n VERSION = 0\n DATASET_PATH = \"ai2_arc\"\n DATASET_NAME = \"ARC-Easy\"\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return True\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(map(self._process_doc, self.dataset[\"train\"]))\n return self._training_docs\n\n def validation_docs(self):\n return map(self._process_doc, self.dataset[\"validation\"])\n\n def test_docs(self):\n return map(self._process_doc, self.dataset[\"test\"])\n\n def _process_doc(self, doc):\n # NOTE: Some `doc[\"answerKey\"]`s are in numeric string format being one\n # of {'1', '2', '3', '4', '5'}. We map them back to letters.\n num_to_letter = {\"1\": \"A\", \"2\": \"B\", \"3\": \"C\", \"4\": \"D\", \"5\": \"E\"}\n doc[\"answerKey\"] = num_to_letter.get(doc[\"answerKey\"], doc[\"answerKey\"])\n out_doc = {\n \"id\": doc[\"id\"],\n \"query\": \"Question: \" + doc[\"question\"] + \"\\nAnswer:\",\n \"choices\": doc[\"choices\"][\"text\"],\n \"gold\": [\"A\", \"B\", \"C\", \"D\", \"E\"].index(doc[\"answerKey\"]),\n }\n return out_doc\n\n def doc_to_text(self, doc):\n return doc[\"query\"]\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"query\"]\n\n\nclass ARCChallenge(ARCEasy):\n DATASET_PATH = \"ai2_arc\"\n DATASET_NAME = \"ARC-Challenge\"\n" }, { "path": "lm_eval/tasks/arithmetic.py", "content": "\"\"\"\nLanguage Models are Few-Shot Learners\nhttps://arxiv.org/pdf/2005.14165.pdf\n\nA small battery of 10 tests that involve asking language models a simple arithmetic\nproblem in natural language.\n\nHomepage: https://github.com/openai/gpt-3/tree/master/data\n\"\"\"\nfrom lm_eval.base import Task, rf\nfrom lm_eval.metrics import mean\n\n\n_CITATION = \"\"\"\n@inproceedings{NEURIPS2020_1457c0d6,\n author = {Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom and Child, Rewon and Ramesh, Aditya and Ziegler, Daniel and Wu, Jeffrey and Winter, Clemens and Hesse, Chris and Chen, Mark and Sigler, Eric and Litwin, Mateusz and Gray, Scott and Chess, Benjamin and Clark, Jack and Berner, Christopher and McCandlish, Sam and Radford, Alec and Sutskever, Ilya and Amodei, Dario},\n booktitle = {Advances in Neural Information Processing Systems},\n editor = {H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin},\n pages = {1877--1901},\n publisher = {Curran Associates, Inc.},\n title = {Language Models are Few-Shot Learners},\n url = {https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf},\n volume = {33},\n year = {2020}\n}\n\"\"\"\n\n\nclass Arithmetic(Task):\n VERSION = 0\n DATASET_PATH = \"EleutherAI/arithmetic\"\n\n def has_training_docs(self):\n return False\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n return NotImplemented\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def test_docs(self):\n return NotImplemented\n\n def doc_to_text(self, doc):\n return doc[\"context\"]\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"context\"]\n\n def doc_to_target(self, doc):\n return doc[\"completion\"]\n\n def construct_requests(self, doc, ctx):\n ll, is_prediction = rf.loglikelihood(ctx, doc[\"completion\"])\n return is_prediction\n\n def process_results(self, doc, results):\n (is_prediction,) = results\n return {\"acc\": is_prediction}\n\n def aggregation(self):\n return {\n \"acc\": mean,\n }\n\n def higher_is_better(self):\n return {\"acc\": True}\n\n\nclass Arithmetic2DPlus(Arithmetic):\n DATASET_NAME = \"arithmetic_2da\"\n\n\nclass Arithmetic2DMinus(Arithmetic):\n DATASET_NAME = \"arithmetic_2ds\"\n\n\nclass Arithmetic3DPlus(Arithmetic):\n DATASET_NAME = \"arithmetic_3da\"\n\n\nclass Arithmetic3DMinus(Arithmetic):\n DATASET_NAME = \"arithmetic_3ds\"\n\n\nclass Arithmetic4DPlus(Arithmetic):\n DATASET_NAME = \"arithmetic_4da\"\n\n\nclass Arithmetic4DMinus(Arithmetic):\n DATASET_NAME = \"arithmetic_4ds\"\n\n\nclass Arithmetic5DPlus(Arithmetic):\n DATASET_NAME = \"arithmetic_5da\"\n\n\nclass Arithmetic5DMinus(Arithmetic):\n DATASET_NAME = \"arithmetic_5ds\"\n\n\nclass Arithmetic2DMultiplication(Arithmetic):\n DATASET_NAME = \"arithmetic_2dm\"\n\n\nclass Arithmetic1DComposite(Arithmetic):\n DATASET_NAME = \"arithmetic_1dc\"\n" }, { "path": "lm_eval/tasks/asdiv.py", "content": "\"\"\"\nASDiv: A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers\nhttps://arxiv.org/abs/2106.15772\n\nASDiv (Academia Sinica Diverse MWP Dataset) is a diverse (in terms of both language\npatterns and problem types) English math word problem (MWP) corpus for evaluating\nthe capability of various MWP solvers. Existing MWP corpora for studying AI progress\nremain limited either in language usage patterns or in problem types. We thus present\na new English MWP corpus with 2,305 MWPs that cover more text patterns and most problem\ntypes taught in elementary school. Each MWP is annotated with its problem type and grade\nlevel (for indicating the level of difficulty).\n\nNOTE: We currently ignore formulas for answer generation.\n\nHomepage: https://github.com/chaochun/nlu-asdiv-dataset\n\"\"\"\nimport inspect\nimport lm_eval.datasets.asdiv.asdiv\nfrom lm_eval.base import rf, Task\nfrom lm_eval.metrics import mean\n\n\n_CITATION = \"\"\"\n@misc{miao2021diverse,\n title={A Diverse Corpus for Evaluating and Developing English Math Word Problem Solvers},\n author={Shen-Yun Miao and Chao-Chun Liang and Keh-Yih Su},\n year={2021},\n eprint={2106.15772},\n archivePrefix={arXiv},\n primaryClass={cs.AI}\n}\n\"\"\"\n\n\nclass Asdiv(Task):\n VERSION = 0\n DATASET_PATH = inspect.getfile(lm_eval.datasets.asdiv.asdiv)\n\n def has_training_docs(self):\n return False\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n raise NotImplementedError(\"This dataset has no training docs\")\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def test_docs(self):\n raise NotImplementedError(\"This dataset has no test docs\")\n\n def fewshot_context(\n self, doc, num_fewshot, provide_description=None, rnd=None, description=None\n ):\n assert num_fewshot == 0, \"ASDiv is intended only for the zero-shot setting.\"\n return super().fewshot_context(\n doc=doc, num_fewshot=num_fewshot, rnd=rnd, description=description\n )\n\n def doc_to_text(self, doc):\n # TODO: add solution-type\n return doc[\"body\"] + \"\\n\" + \"Question:\" + doc[\"question\"] + \"\\n\" + \"Answer:\"\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"body\"] + \" \" + doc[\"question\"]\n\n def doc_to_target(self, doc):\n # TODO: add formula\n\n answer = doc[\"answer\"].split(\" (\")[0]\n return \" \" + answer\n\n def construct_requests(self, doc, ctx):\n ll, is_greedy = rf.loglikelihood(ctx, self.doc_to_target(doc))\n return ll, is_greedy\n\n def process_results(self, doc, results):\n ll, is_greedy = results\n\n return {\"acc\": int(is_greedy)}\n\n def aggregation(self):\n return {\"acc\": mean}\n\n def higher_is_better(self):\n return {\"acc\": True}\n" }, { "path": "lm_eval/tasks/blimp.py", "content": "\"\"\"\nBLiMP: A Benchmark of Linguistic Minimal Pairs for English\nhttps://arxiv.org/abs/1912.00582\n\nBLiMP is a challenge set for evaluating what language models (LMs) know about\nmajor grammatical phenomena in English. BLiMP consists of 67 sub-datasets, each\ncontaining 1000 minimal pairs isolating specific contrasts in syntax, morphology,\nor semantics. The data is automatically generated according to expert-crafted\ngrammars.\n\nHomepage: https://github.com/alexwarstadt/blimp\n\"\"\"\nfrom lm_eval.base import rf, Task\nfrom lm_eval.metrics import mean\n\n\n_CITATION = \"\"\"\n@article{warstadt2019blimp,\n author = {Warstadt, Alex and Parrish, Alicia and Liu, Haokun and Mohananey, Anhad and Peng, Wei and Wang, Sheng-Fu and Bowman, Samuel R.},\n title = {BLiMP: The Benchmark of Linguistic Minimal Pairs for English},\n journal = {Transactions of the Association for Computational Linguistics},\n volume = {8},\n number = {},\n pages = {377-392},\n year = {2020},\n doi = {10.1162/tacl\\_a\\_00321},\n URL = {https://doi.org/10.1162/tacl_a_00321},\n eprint = {https://doi.org/10.1162/tacl_a_00321},\n abstract = { We introduce The Benchmark of Linguistic Minimal Pairs (BLiMP),1 a challenge set for evaluating the linguistic knowledge of language models (LMs) on major grammatical phenomena in English. BLiMP consists of 67 individual datasets, each containing 1,000 minimal pairs—that is, pairs of minimally different sentences that contrast in grammatical acceptability and isolate specific phenomenon in syntax, morphology, or semantics. We generate the data according to linguist-crafted grammar templates, and human aggregate agreement with the labels is 96.4\\%. We evaluate n-gram, LSTM, and Transformer (GPT-2 and Transformer-XL) LMs by observing whether they assign a higher probability to the acceptable sentence in each minimal pair. We find that state-of-the-art models identify morphological contrasts related to agreement reliably, but they struggle with some subtle semantic and syntactic phenomena, such as negative polarity items and extraction islands. }\n}\n\"\"\" # noqa: W605\n\n\nclass BlimpTask(Task):\n VERSION = 0\n DATASET_PATH = \"blimp\"\n\n def has_training_docs(self):\n return False\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def validation_docs(self):\n # The HF dataset only contains a \"train\" dataset, but the harness expects a \"validation\"\n # dataset. Let's use the training dataset, on the assumption that the model wasn't actually\n # trained on this data.\n return self.dataset[\"train\"]\n\n def fewshot_context(\n self, doc, num_fewshot, provide_description=None, rnd=None, description=None\n ):\n assert num_fewshot == 0\n assert (\n rnd is not None\n ), \"A `random.Random` generator argument must be provided to `rnd`\"\n assert not provide_description, (\n \"The `provide_description` arg will be removed in future versions. To prepend \"\n \"a custom description to the context, supply the corresponding string via the \"\n \"`description` arg.\"\n )\n if provide_description is not None:\n # nudge people to not specify it at all\n print(\n \"WARNING: provide_description is deprecated and will be removed in a future version in favor of description_dict\"\n )\n\n return \"\"\n\n def doc_to_text(self, doc):\n # this method is invoked by tests only\n return \"\"\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"sentence_good\"] + \" \" + doc[\"sentence_bad\"]\n\n def doc_to_target(self, doc):\n # this method is invoked by tests only\n return \"\"\n\n def construct_requests(self, doc, ctx):\n assert not ctx\n\n # Calculate the loglikelihood for the good and the bad sentence.\n # Note that loglikelihood translates the \"\" prefix to the \"<|endoftext|>\" token\n return [\n rf.loglikelihood(\"\", doc[\"sentence_good\"]),\n rf.loglikelihood(\"\", doc[\"sentence_bad\"]),\n ]\n\n def process_results(self, doc, results):\n likelihood1, likelihood2 = results\n\n # the model got this case right iff the good sentence scored higher than the bad sentence\n acc = 1.0 if likelihood1 > likelihood2 else 0.0\n\n return {\n \"acc\": acc,\n }\n\n def higher_is_better(self):\n return {\n \"acc\": True,\n }\n\n def aggregation(self):\n return {\n \"acc\": mean,\n }\n\n\nclass BlimpAdjunctIsland(BlimpTask):\n DATASET_NAME = \"adjunct_island\"\n\n\nclass BlimpAnaphorGenderAgreement(BlimpTask):\n DATASET_NAME = \"anaphor_gender_agreement\"\n\n\nclass BlimpAnaphorNumberAgreement(BlimpTask):\n DATASET_NAME = \"anaphor_number_agreement\"\n\n\nclass BlimpAnimateSubjectPassive(BlimpTask):\n DATASET_NAME = \"animate_subject_passive\"\n\n\nclass BlimpAnimateSubjectTrans(BlimpTask):\n DATASET_NAME = \"animate_subject_trans\"\n\n\nclass BlimpCausative(BlimpTask):\n DATASET_NAME = \"causative\"\n\n\nclass BlimpComplex_NPIsland(BlimpTask):\n DATASET_NAME = \"complex_NP_island\"\n\n\nclass BlimpCoordinateStructureConstraintComplexLeftBranch(BlimpTask):\n DATASET_NAME = \"coordinate_structure_constraint_complex_left_branch\"\n\n\nclass BlimpCoordinateStructureConstraintObjectExtraction(BlimpTask):\n DATASET_NAME = \"coordinate_structure_constraint_object_extraction\"\n\n\nclass BlimpDeterminerNounAgreement_1(BlimpTask):\n DATASET_NAME = \"determiner_noun_agreement_1\"\n\n\nclass BlimpDeterminerNounAgreement_2(BlimpTask):\n DATASET_NAME = \"determiner_noun_agreement_2\"\n\n\nclass BlimpDeterminerNounAgreementIrregular_1(BlimpTask):\n DATASET_NAME = \"determiner_noun_agreement_irregular_1\"\n\n\nclass BlimpDeterminerNounAgreementIrregular_2(BlimpTask):\n DATASET_NAME = \"determiner_noun_agreement_irregular_2\"\n\n\nclass BlimpDeterminerNounAgreementWithAdj_2(BlimpTask):\n DATASET_NAME = \"determiner_noun_agreement_with_adj_2\"\n\n\nclass BlimpDeterminerNounAgreementWithAdjIrregular_1(BlimpTask):\n DATASET_NAME = \"determiner_noun_agreement_with_adj_irregular_1\"\n\n\nclass BlimpDeterminerNounAgreementWithAdjIrregular_2(BlimpTask):\n DATASET_NAME = \"determiner_noun_agreement_with_adj_irregular_2\"\n\n\nclass BlimpDeterminerNounAgreementWithAdjective_1(BlimpTask):\n DATASET_NAME = \"determiner_noun_agreement_with_adjective_1\"\n\n\nclass BlimpDistractorAgreementRelationalNoun(BlimpTask):\n DATASET_NAME = \"distractor_agreement_relational_noun\"\n\n\nclass BlimpDistractorAgreementRelativeClause(BlimpTask):\n DATASET_NAME = \"distractor_agreement_relative_clause\"\n\n\nclass BlimpDropArgument(BlimpTask):\n DATASET_NAME = \"drop_argument\"\n\n\nclass BlimpEllipsisNBar_1(BlimpTask):\n DATASET_NAME = \"ellipsis_n_bar_1\"\n\n\nclass BlimpEllipsisNBar_2(BlimpTask):\n DATASET_NAME = \"ellipsis_n_bar_2\"\n\n\nclass BlimpExistentialThereObjectRaising(BlimpTask):\n DATASET_NAME = \"existential_there_object_raising\"\n\n\nclass BlimpExistentialThereQuantifiers_1(BlimpTask):\n DATASET_NAME = \"existential_there_quantifiers_1\"\n\n\nclass BlimpExistentialThereQuantifiers_2(BlimpTask):\n DATASET_NAME = \"existential_there_quantifiers_2\"\n\n\nclass BlimpExistentialThereSubjectRaising(BlimpTask):\n DATASET_NAME = \"existential_there_subject_raising\"\n\n\nclass BlimpExpletiveItObjectRaising(BlimpTask):\n DATASET_NAME = \"expletive_it_object_raising\"\n\n\nclass BlimpInchoative(BlimpTask):\n DATASET_NAME = \"inchoative\"\n\n\nclass BlimpIntransitive(BlimpTask):\n DATASET_NAME = \"intransitive\"\n\n\nclass BlimpIrregularPastParticipleAdjectives(BlimpTask):\n DATASET_NAME = \"irregular_past_participle_adjectives\"\n\n\nclass BlimpIrregularPastParticipleVerbs(BlimpTask):\n DATASET_NAME = \"irregular_past_participle_verbs\"\n\n\nclass BlimpIrregularPluralSubjectVerbAgreement_1(BlimpTask):\n DATASET_NAME = \"irregular_plural_subject_verb_agreement_1\"\n\n\nclass BlimpIrregularPluralSubjectVerbAgreement_2(BlimpTask):\n DATASET_NAME = \"irregular_plural_subject_verb_agreement_2\"\n\n\nclass BlimpLeftBranchIslandEchoQuestion(BlimpTask):\n DATASET_NAME = \"left_branch_island_echo_question\"\n\n\nclass BlimpLeftBranchIslandSimpleQuestion(BlimpTask):\n DATASET_NAME = \"left_branch_island_simple_question\"\n\n\nclass BlimpMatrixQuestionNpiLicensorPresent(BlimpTask):\n DATASET_NAME = \"matrix_question_npi_licensor_present\"\n\n\nclass BlimpNpiPresent_1(BlimpTask):\n DATASET_NAME = \"npi_present_1\"\n\n\nclass BlimpNpiPresent_2(BlimpTask):\n DATASET_NAME = \"npi_present_2\"\n\n\nclass BlimpOnlyNpiLicensorPresent(BlimpTask):\n DATASET_NAME = \"only_npi_licensor_present\"\n\n\nclass BlimpOnlyNpiScope(BlimpTask):\n DATASET_NAME = \"only_npi_scope\"\n\n\nclass BlimpPassive_1(BlimpTask):\n DATASET_NAME = \"passive_1\"\n\n\nclass BlimpPassive_2(BlimpTask):\n DATASET_NAME = \"passive_2\"\n\n\nclass BlimpPrinciple_ACCommand(BlimpTask):\n DATASET_NAME = \"principle_A_c_command\"\n\n\nclass BlimpPrinciple_ACase_1(BlimpTask):\n DATASET_NAME = \"principle_A_case_1\"\n\n\nclass BlimpPrinciple_ACase_2(BlimpTask):\n DATASET_NAME = \"principle_A_case_2\"\n\n\nclass BlimpPrinciple_ADomain_1(BlimpTask):\n DATASET_NAME = \"principle_A_domain_1\"\n\n\nclass BlimpPrinciple_ADomain_2(BlimpTask):\n DATASET_NAME = \"principle_A_domain_2\"\n\n\nclass BlimpPrinciple_ADomain_3(BlimpTask):\n DATASET_NAME = \"principle_A_domain_3\"\n\n\nclass BlimpPrinciple_AReconstruction(BlimpTask):\n DATASET_NAME = \"principle_A_reconstruction\"\n\n\nclass BlimpRegularPluralSubjectVerbAgreement_1(BlimpTask):\n DATASET_NAME = \"regular_plural_subject_verb_agreement_1\"\n\n\nclass BlimpRegularPluralSubjectVerbAgreement_2(BlimpTask):\n DATASET_NAME = \"regular_plural_subject_verb_agreement_2\"\n\n\nclass BlimpSententialNegationNpiLicensorPresent(BlimpTask):\n DATASET_NAME = \"sentential_negation_npi_licensor_present\"\n\n\nclass BlimpSententialNegationNpiScope(BlimpTask):\n DATASET_NAME = \"sentential_negation_npi_scope\"\n\n\nclass BlimpSententialSubjectIsland(BlimpTask):\n DATASET_NAME = \"sentential_subject_island\"\n\n\nclass BlimpSuperlativeQuantifiers_1(BlimpTask):\n DATASET_NAME = \"superlative_quantifiers_1\"\n\n\nclass BlimpSuperlativeQuantifiers_2(BlimpTask):\n DATASET_NAME = \"superlative_quantifiers_2\"\n\n\nclass BlimpToughVsRaising_1(BlimpTask):\n DATASET_NAME = \"tough_vs_raising_1\"\n\n\nclass BlimpToughVsRaising_2(BlimpTask):\n DATASET_NAME = \"tough_vs_raising_2\"\n\n\nclass BlimpTransitive(BlimpTask):\n DATASET_NAME = \"transitive\"\n\n\nclass BlimpWhIsland(BlimpTask):\n DATASET_NAME = \"wh_island\"\n\n\nclass BlimpWhQuestionsObjectGap(BlimpTask):\n DATASET_NAME = \"wh_questions_object_gap\"\n\n\nclass BlimpWhQuestionsSubjectGap(BlimpTask):\n DATASET_NAME = \"wh_questions_subject_gap\"\n\n\nclass BlimpWhQuestionsSubjectGapLongDistance(BlimpTask):\n DATASET_NAME = \"wh_questions_subject_gap_long_distance\"\n\n\nclass BlimpWhVsThatNoGap(BlimpTask):\n DATASET_NAME = \"wh_vs_that_no_gap\"\n\n\nclass BlimpWhVsThatNoGapLongDistance(BlimpTask):\n DATASET_NAME = \"wh_vs_that_no_gap_long_distance\"\n\n\nclass BlimpWhVsThatWithGap(BlimpTask):\n DATASET_NAME = \"wh_vs_that_with_gap\"\n\n\nclass BlimpWhVsThatWithGapLongDistance(BlimpTask):\n DATASET_NAME = \"wh_vs_that_with_gap_long_distance\"\n" }, { "path": "lm_eval/tasks/cbt.py", "content": "\"\"\"\nThe Children’s Book Test (CBT) from the paper:\nhttps://research.fb.com/wp-content/uploads/2016/11/the_goldilocks_principle_reading_children_s_books_with_explicit_memory_representations.pdf\n\nThe Children's Book Test (CBT) is test of how well language models capture\nmeaning in children's books. Unlike standard language modelling benchmarks,\nit distinguishes the task of predicting syntactic function words from that\nof predicting lower-frequency words, which carry greater semantic content.\n\nNOTE: This evaluation is based on the (context + query) question-answering variant\nused by the Recurrent Language Models described in the paper. See section 4.4.\n\nHomepage: https://github.com/facebookresearch/ParlAI/tree/main/parlai/tasks/cbt\n\"\"\"\nimport numpy as np\nfrom lm_eval.base import rf, Task\nfrom lm_eval.metrics import mean\n\n\n_CITATION = \"\"\"\n@misc{hill2016goldilocks,\n title={The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations},\n author={Felix Hill and Antoine Bordes and Sumit Chopra and Jason Weston},\n year={2016},\n eprint={1511.02301},\n archivePrefix={arXiv},\n primaryClass={cs.CL}\n}\n\"\"\"\n\n\nclass CBTBase(Task):\n VERSION = 0\n DATASET_PATH = \"cbt\"\n DATASET_NAME = None\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return True\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(self.dataset[\"train\"])\n return self._training_docs\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def test_docs(self):\n return self.dataset[\"test\"]\n\n def detokenize(self, text):\n text = text.replace(\" '\", \"'\")\n text = text.replace(\" \\n\", \"\\n\")\n text = text.replace(\"\\n \", \"\\n\")\n text = text.replace(\" n't\", \"n't\")\n text = text.replace(\"`` \", '\"')\n text = text.replace(\"''\", '\"')\n # punctuation\n text = text.replace(\" :\", \":\")\n text = text.replace(\" ;\", \";\")\n text = text.replace(\" !\", \"!\")\n text = text.replace(\" ?\", \"?\")\n text = text.replace(\" ,\", \",\")\n text = text.replace(\" .\", \".\")\n return text\n\n def doc_to_text(self, doc):\n passage = \" \".join(doc[\"sentences\"])\n text = \"Passage: \" + passage + \"\\nQuestion: \" + doc[\"question\"]\n return self.detokenize(text)\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n passage = \" \".join(doc[\"sentences\"])\n return passage\n\n def doc_to_target(self, doc):\n return \"\"\n\n def fewshot_examples(self, k, rnd):\n assert (\n k == 0\n ), f\"CBT is only implemented for the zero-shot setting. Given k={k}.\"\n return super().fewshot_examples(k, rnd)\n\n def construct_requests(self, doc, ctx):\n \"\"\"Uses RequestFactory to construct Requests and returns an iterable of\n Requests which will be sent to the LM.\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param ctx: str\n The context string, generated by fewshot_context. This includes the natural\n language description, as well as the few shot examples, and the question\n part of the document for `doc`.\n \"\"\"\n lls = []\n for option in doc[\"options\"]:\n # Following Section 4.4 \"Recurrent Language Models\" in the CBT paper:\n # \"we rank candidate [option] c based on p(q1 . . . qk−1, c, qk+1 . . . ql)\n # rather than simply p(q1 . . . qk−1, c).\"\n lls.append(rf.loglikelihood(\"\", ctx.replace(\"XXXXX\", option))[0])\n return lls\n\n def process_results(self, doc, results):\n \"\"\"Take a single document and the LM results and evaluates, returning a\n dict where keys are the names of submetrics and values are the values of\n the metric for that one document\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param results:\n The results of the requests created in construct_requests.\n \"\"\"\n gold = doc[\"options\"].index(doc[\"answer\"])\n pred = np.argmax(results)\n return {\"acc\": pred == gold}\n\n def aggregation(self):\n \"\"\"\n :returns: {str: [float] -> float}\n A dictionary where keys are the names of submetrics and values are\n functions that aggregate a list of metrics\n \"\"\"\n return {\"acc\": mean}\n\n def higher_is_better(self):\n \"\"\"\n :returns: {str: bool}\n A dictionary where keys are the names of submetrics and values are\n whether a higher value of the submetric is better\n \"\"\"\n return {\"acc\": True}\n\n\nclass CBTCN(CBTBase):\n DATASET_NAME = \"CN\"\n\n\nclass CBTNE(CBTBase):\n DATASET_NAME = \"NE\"\n" }, { "path": "lm_eval/tasks/coqa.py", "content": "\"\"\"\nCoQA: A Conversational Question Answering Challenge\nhttps://arxiv.org/pdf/1808.07042.pdf\n\nCoQA is a large-scale dataset for building Conversational Question Answering\nsystems. The goal of the CoQA challenge is to measure the ability of machines to\nunderstand a text passage and answer a series of interconnected questions that\nappear in a conversation.\n\nHomepage: https://stanfordnlp.github.io/coqa/\n\"\"\"\nimport inspect\nimport transformers.data.metrics.squad_metrics as squad_metrics\nimport lm_eval.datasets.coqa.coqa\nfrom lm_eval.base import Task, rf, mean\nfrom itertools import zip_longest\n\n\n_CITATION = \"\"\"\n@misc{reddy2018coqa,\n title={CoQA: A Conversational Question Answering Challenge},\n author={Siva Reddy and Danqi Chen and Christopher D. Manning},\n year={2018},\n eprint={1808.07042},\n archivePrefix={arXiv},\n primaryClass={cs.CL}\n}\n\"\"\"\n\n\nclass CoQA(Task):\n VERSION = 1\n DATASET_PATH = inspect.getfile(lm_eval.datasets.coqa.coqa)\n DATASET_NAME = None\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n return self.dataset[\"train\"]\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def test_docs(self):\n pass\n\n def doc_to_text(self, doc):\n # Given a passage p, the conversation history {q1, a1, . . . qi−1, ai−1}\n # and a question qi, the task is to predict the answer ai\n doc_text = doc[\"story\"] + \"\\n\\n\"\n for (q, a) in zip_longest(\n doc[\"questions\"][\"input_text\"], doc[\"answers\"][\"input_text\"][:-1]\n ): # omit target answer ai\n question = f\"Q: {q}\\n\\n\"\n answer = f\"A: {a}\\n\\n\" if a is not None else \"A:\"\n doc_text += question + answer\n return doc_text\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"story\"] + \" \" + \"\\n\".join(doc[\"questions\"][\"input_text\"])\n\n @classmethod\n def get_answers(cls, doc, turn_id):\n # Returns unique answers and valid alternatives (Some questions in CoQA have multiple valid answers).\n answers = []\n answer_forturn = doc[\"answers\"][\"input_text\"][turn_id - 1]\n answers.append(answer_forturn)\n\n additional_answers = doc.get(\"additional_answers\")\n if additional_answers:\n for key in additional_answers:\n additional_answer_for_turn = additional_answers[key][\"input_text\"][\n turn_id - 1\n ]\n if additional_answer_for_turn.lower() not in map(str.lower, answers):\n answers.append(additional_answer_for_turn)\n return answers\n\n @classmethod\n def get_answer_choice(self, raw_text):\n # Function maps answers to CoQA answer categories\n # ~ 1/5 of the CoQA answers are Yes/No\n # ~ 2/3 of the CoQA answers are span-based\n # (answers overlap with the passage ignoring punctuation and case mismatch)\n if raw_text == \"unknown\":\n return \"0\"\n if squad_metrics.normalize_answer(raw_text) == \"yes\":\n return \"1\"\n if squad_metrics.normalize_answer(raw_text) == \"no\":\n return \"2\"\n return \"3\" # Not a yes/no question\n\n @staticmethod\n def compute_scores(gold_list, pred):\n # tests for exact match and on the normalised answer (compute_exact)\n # test for overlap (compute_f1)\n f1_sum = 0.0\n em_sum = 0.0\n if len(gold_list) > 1:\n for i in range(len(gold_list)):\n gold_answers = gold_list[0:i] + gold_list[i + 1 :]\n # predictions compared against (n) golds and take maximum\n em_sum += max(\n squad_metrics.compute_exact(a, pred) for a in gold_answers\n )\n f1_sum += max(squad_metrics.compute_f1(a, pred) for a in gold_answers)\n else:\n em_sum += max(squad_metrics.compute_exact(a, pred) for a in gold_list)\n f1_sum += max(squad_metrics.compute_f1(a, pred) for a in gold_list)\n\n return {\n \"em\": em_sum / max(1, len(gold_list)),\n \"f1\": f1_sum / max(1, len(gold_list)),\n }\n\n def doc_to_target(self, doc, turnid=None):\n # Default to prediction of last turn.\n if turnid is None:\n turnid = len(doc[\"questions\"][\"input_text\"])\n raw_text = doc[\"answers\"][\"input_text\"][turnid - 1]\n return \" \" + raw_text\n\n def construct_requests(self, doc, ctx):\n \"\"\"Uses RequestFactory to construct Requests and returns an iterable of\n Requests which will be sent to the LM.\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param ctx: str\n The context string, generated by fewshot_context. This includes the natural\n language description, as well as the few shot examples, and the question\n part of the document for `doc`.\n \"\"\"\n cont_request = rf.greedy_until(ctx, [\"\\nQ:\"])\n return cont_request\n\n def process_results(self, doc, results):\n \"\"\"Take a single document and the LM results and evaluates, returning a\n dict where keys are the names of submetrics and values are the values of\n the metric for that one document\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param results:\n The results of the requests created in construct_requests.\n \"\"\"\n turn_id = len(doc[\"questions\"][\"input_text\"])\n gold_list = self.get_answers(doc, turn_id)\n pred = results[0].strip().split(\"\\n\")[0]\n\n scores = self.compute_scores(gold_list, pred)\n\n return {\n \"f1\": scores[\"f1\"],\n \"em\": scores[\"em\"],\n }\n\n def higher_is_better(self):\n return {\n \"f1\": True,\n \"em\": True,\n }\n\n def aggregation(self):\n return {\n \"f1\": mean,\n \"em\": mean,\n }\n" }, { "path": "lm_eval/tasks/crowspairs.py", "content": "\"\"\"\nCrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models\nhttps://aclanthology.org/2020.emnlp-main.154/\nFrench CrowS-Pairs: Extending a challenge dataset for measuring social bias in masked\nlanguage models to a language other than English\nhttps://aclanthology.org/2022.acl-long.583/\n\nCrowS-Pairs is a challenge set for evaluating what language models (LMs) on their tendency\nto generate biased outputs. CrowS-Pairs comes in 2 languages and the English subset has\na newer version which fixes some of the issues with the original version.\n\nHomepage: https://github.com/nyu-mll/crows-pairs, https://gitlab.inria.fr/french-crows-pairs\n\"\"\"\nfrom lm_eval.base import rf, Task\nfrom lm_eval.metrics import mean\n\n\n_CITATION = \"\"\"\n@inproceedings{nangia-etal-2020-crows,\n title = \"{C}row{S}-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models\",\n author = \"Nangia, Nikita and\n Vania, Clara and\n Bhalerao, Rasika and\n Bowman, Samuel R.\",\n booktitle = \"Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)\",\n month = nov,\n year = \"2020\",\n address = \"Online\",\n publisher = \"Association for Computational Linguistics\",\n url = \"https://aclanthology.org/2020.emnlp-main.154\",\n doi = \"10.18653/v1/2020.emnlp-main.154\",\n pages = \"1953--1967\",\n abstract = \"Pretrained language models, especially masked language models (MLMs) have seen success across many NLP tasks. However, there is ample evidence that they use the cultural biases that are undoubtedly present in the corpora they are trained on, implicitly creating harm with biased representations. To measure some forms of social bias in language models against protected demographic groups in the US, we introduce the Crowdsourced Stereotype Pairs benchmark (CrowS-Pairs). CrowS-Pairs has 1508 examples that cover stereotypes dealing with nine types of bias, like race, religion, and age. In CrowS-Pairs a model is presented with two sentences: one that is more stereotyping and another that is less stereotyping. The data focuses on stereotypes about historically disadvantaged groups and contrasts them with advantaged groups. We find that all three of the widely-used MLMs we evaluate substantially favor sentences that express stereotypes in every category in CrowS-Pairs. As work on building less biased models advances, this dataset can be used as a benchmark to evaluate progress.\",\n}\n\n@inproceedings{neveol-etal-2022-french,\n title = \"{F}rench {C}row{S}-Pairs: Extending a challenge dataset for measuring social bias in masked language models to a language other than {E}nglish\",\n author = {N{\\'e}v{\\'e}ol, Aur{\\'e}lie and\n Dupont, Yoann and\n Bezan{\\c{c}}on, Julien and\n Fort, Kar{\\\"e}n},\n booktitle = \"Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)\",\n month = may,\n year = \"2022\",\n address = \"Dublin, Ireland\",\n publisher = \"Association for Computational Linguistics\",\n url = \"https://aclanthology.org/2022.acl-long.583\",\n doi = \"10.18653/v1/2022.acl-long.583\",\n pages = \"8521--8531\",\n abstract = \"Warning: This paper contains explicit statements of offensive stereotypes which may be upsetting.Much work on biases in natural language processing has addressed biases linked to the social and cultural experience of English speaking individuals in the United States. We seek to widen the scope of bias studies by creating material to measure social bias in language models (LMs) against specific demographic groups in France. We build on the US-centered CrowS-pairs dataset to create a multilingual stereotypes dataset that allows for comparability across languages while also characterizing biases that are specific to each country and language. We introduce 1,679 sentence pairs in French that cover stereotypes in ten types of bias like gender and age. 1,467 sentence pairs are translated from CrowS-pairs and 212 are newly crowdsourced. The sentence pairs contrast stereotypes concerning underadvantaged groups with the same sentence concerning advantaged groups. We find that four widely used language models (three French, one multilingual) favor sentences that express stereotypes in most bias categories. We report on the translation process from English into French, which led to a characterization of stereotypes in CrowS-pairs including the identification of US-centric cultural traits. We offer guidelines to further extend the dataset to other languages and cultural environments.\",\n}\n\"\"\" # noqa: W605\n\n\nclass CrowsPairsMutilingual(Task):\n VERSION = 0\n DATASET_PATH = \"BigScienceBiasEval/crows_pairs_multilingual\"\n BIAS_TYPE = None\n\n def has_training_docs(self):\n return False\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def validation_docs(self):\n test_dataset = self.dataset[\"test\"]\n if self.BIAS_TYPE is not None:\n test_dataset = test_dataset.filter(\n lambda example: example[\"bias_type\"].startswith(self.BIAS_TYPE)\n )\n return test_dataset\n\n def fewshot_context(\n self, doc, num_fewshot, provide_description=None, rnd=None, description=None\n ):\n assert num_fewshot == 0\n assert (\n rnd is not None\n ), \"A `random.Random` generator argument must be provided to `rnd`\"\n assert not provide_description, (\n \"The `provide_description` arg will be removed in future versions. To prepend \"\n \"a custom description to the context, supply the corresponding string via the \"\n \"`description` arg.\"\n )\n if provide_description is not None:\n # nudge people to not specify it at all\n print(\n \"WARNING: provide_description is deprecated and will be removed in a future version in favor of description_dict\"\n )\n\n return \"\"\n\n def doc_to_text(self, doc):\n # ignore prompts as we only score the model on the likelihood of the sentences\n return \"\"\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"sentence_good\"] + \" \" + doc[\"sentence_bad\"]\n\n def doc_to_target(self, doc):\n # ignore prompts as we only score the model on the likelihood of the sentences\n return \"\"\n\n def construct_requests(self, doc, ctx):\n assert not ctx\n\n # Calculate the loglikelihood for the more and the less stereotypical sentence.\n # Note that loglikelihood translates the \"\" prefix to the \"<|endoftext|>\" token\n return [\n rf.loglikelihood(\"\", doc[\"sent_more\"]),\n rf.loglikelihood(\"\", doc[\"sent_less\"]),\n ]\n\n def process_results(self, doc, results):\n likelihood1, likelihood2 = results\n\n # Calculate the absolute difference in loglikelihoods\n diff = abs(likelihood1[0] - likelihood2[0])\n\n # if stereotypical sentence more likely (loglikelihood higher)\n # then treat this as predicting stereotyped sentence\n acc = 1.0 if likelihood1[0] > likelihood2[0] else 0.0\n\n return {\"likelihood_difference\": diff, \"pct_stereotype\": acc}\n\n def higher_is_better(self):\n # For all metrics lower is better\n return {\"likelihood_difference\": False, \"pct_stereotype\": True}\n\n def aggregation(self):\n return {\"likelihood_difference\": mean, \"pct_stereotype\": mean}\n\n\nclass CrowsPairsEnglish(CrowsPairsMutilingual):\n DATASET_NAME = \"english\"\n\n\nclass CrowsPairsFrench(CrowsPairsMutilingual):\n DATASET_NAME = \"french\"\n\n\nclass CrowsPairsEnglishRaceColor(CrowsPairsMutilingual):\n DATASET_NAME = \"english\"\n BIAS_TYPE = \"race-color\"\n\n\nclass CrowsPairsEnglishSocioeconomic(CrowsPairsMutilingual):\n DATASET_NAME = \"english\"\n BIAS_TYPE = \"socioeconomic\"\n\n\nclass CrowsPairsEnglishGender(CrowsPairsMutilingual):\n DATASET_NAME = \"english\"\n BIAS_TYPE = \"gender\"\n\n\nclass CrowsPairsEnglishAge(CrowsPairsMutilingual):\n DATASET_NAME = \"english\"\n BIAS_TYPE = \"age\"\n\n\nclass CrowsPairsEnglishReligion(CrowsPairsMutilingual):\n DATASET_NAME = \"english\"\n BIAS_TYPE = \"religion\"\n\n\nclass CrowsPairsEnglishDisability(CrowsPairsMutilingual):\n DATASET_NAME = \"english\"\n BIAS_TYPE = \"disability\"\n\n\nclass CrowsPairsEnglishSexualOrientation(CrowsPairsMutilingual):\n DATASET_NAME = \"english\"\n BIAS_TYPE = \"sexual-orientation\"\n\n\nclass CrowsPairsEnglishNationality(CrowsPairsMutilingual):\n DATASET_NAME = \"english\"\n BIAS_TYPE = \"nationality\"\n\n\nclass CrowsPairsEnglishPhysicalAppearance(CrowsPairsMutilingual):\n DATASET_NAME = \"english\"\n BIAS_TYPE = \"physical-appearance\"\n\n\nclass CrowsPairsEnglishAutre(CrowsPairsMutilingual):\n DATASET_NAME = \"english\"\n BIAS_TYPE = \"autre\"\n\n\nclass CrowsPairsFrenchRaceColor(CrowsPairsMutilingual):\n DATASET_NAME = \"french\"\n BIAS_TYPE = \"race-color\"\n\n\nclass CrowsPairsFrenchSocioeconomic(CrowsPairsMutilingual):\n DATASET_NAME = \"french\"\n BIAS_TYPE = \"socioeconomic\"\n\n\nclass CrowsPairsFrenchGender(CrowsPairsMutilingual):\n DATASET_NAME = \"french\"\n BIAS_TYPE = \"gender\"\n\n\nclass CrowsPairsFrenchAge(CrowsPairsMutilingual):\n DATASET_NAME = \"french\"\n BIAS_TYPE = \"age\"\n\n\nclass CrowsPairsFrenchReligion(CrowsPairsMutilingual):\n DATASET_NAME = \"french\"\n BIAS_TYPE = \"religion\"\n\n\nclass CrowsPairsFrenchDisability(CrowsPairsMutilingual):\n DATASET_NAME = \"french\"\n BIAS_TYPE = \"disability\"\n\n\nclass CrowsPairsFrenchSexualOrientation(CrowsPairsMutilingual):\n DATASET_NAME = \"french\"\n BIAS_TYPE = \"sexual-orientation\"\n\n\nclass CrowsPairsFrenchNationality(CrowsPairsMutilingual):\n DATASET_NAME = \"french\"\n BIAS_TYPE = \"nationality\"\n\n\nclass CrowsPairsFrenchPhysicalAppearance(CrowsPairsMutilingual):\n DATASET_NAME = \"french\"\n BIAS_TYPE = \"physical-appearance\"\n\n\nclass CrowsPairsFrenchAutre(CrowsPairsMutilingual):\n DATASET_NAME = \"french\"\n BIAS_TYPE = \"autre\"\n" }, { "path": "lm_eval/tasks/drop.py", "content": "\"\"\"\nDROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs\nhttps://aclanthology.org/attachments/N19-1246.Supplementary.pdf\n\nDROP is a QA dataset which tests comprehensive understanding of paragraphs. In\nthis crowdsourced, adversarially-created, 96k question-answering benchmark, a\nsystem must resolve multiple references in a question, map them onto a paragraph,\nand perform discrete operations over them (such as addition, counting, or sorting).\n\nHomepage: https://allenai.org/data/drop\n\nAcknowledgement: This implementation is based on the official evaluation for `DROP`:\nhttps://github.com/allenai/allennlp-reading-comprehension/blob/master/allennlp_rc/eval/drop_eval.py\n\"\"\"\nimport inspect\nimport numpy as np\nimport re\nimport string\nimport lm_eval.datasets.drop.drop\nfrom scipy.optimize import linear_sum_assignment\nfrom lm_eval.base import Task, rf\nfrom lm_eval.metrics import mean\n\n\n_CITATION = \"\"\"\n@misc{dua2019drop,\n title={DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs},\n author={Dheeru Dua and Yizhong Wang and Pradeep Dasigi and Gabriel Stanovsky and Sameer Singh and Matt Gardner},\n year={2019},\n eprint={1903.00161},\n archivePrefix={arXiv},\n primaryClass={cs.CL}\n}\n\"\"\"\n\n\n_ARTICLES = re.compile(r\"\\b(a|an|the)\\b\", re.UNICODE)\n\n\nclass DROP(Task):\n VERSION = 1\n DATASET_PATH = inspect.getfile(lm_eval.datasets.drop.drop)\n DATASET_NAME = None\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(map(self._process_doc, self.dataset[\"train\"]))\n return self._training_docs\n\n def validation_docs(self):\n return map(self._process_doc, self.dataset[\"validation\"])\n\n def _process_doc(self, doc):\n return {\n \"id\": doc[\"query_id\"],\n \"passage\": doc[\"passage\"],\n \"question\": doc[\"question\"],\n \"answers\": self.get_answers(doc),\n }\n\n @classmethod\n def get_answers(cls, qa):\n def _flatten_validated_answers(validated_answers):\n \"\"\"Flattens a dict of lists of validated answers.\n {\"number\": ['1', '8'], ...}\n -> [{\"number\": ['1'], ...}, {\"number\": ['8'], ...}]\n \"\"\"\n valid_answers = []\n for i in range(len(validated_answers[\"number\"])):\n valid_answers.append(\n {\n \"number\": validated_answers[\"number\"][i],\n \"date\": validated_answers[\"date\"][i],\n \"spans\": validated_answers[\"spans\"][i],\n }\n )\n return valid_answers\n\n answers = []\n answers_set = set()\n candidates = [qa[\"answer\"]] + _flatten_validated_answers(\n qa[\"validated_answers\"]\n )\n for candidate in candidates:\n answer = cls.parse_answer(candidate)\n if answer in answers_set:\n continue\n answers_set.add(answer)\n answers.append(answer)\n return answers\n\n @classmethod\n def parse_answer(cls, answer):\n # NOTE: Everything is returned as a tuple for uniformity and hashability.\n if answer[\"number\"] != \"\":\n return (str(answer[\"number\"]),)\n if answer[\"spans\"] != []:\n return tuple(answer[\"spans\"])\n return (\n \" \".join(\n [answer[\"date\"][\"day\"], answer[\"date\"][\"month\"], answer[\"date\"][\"year\"]]\n ).strip(),\n )\n\n def doc_to_text(self, doc):\n return f\"Passage: {doc['passage']}\\nQuestion: {doc['question']}\\nAnswer:\"\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"passage\"] + \" \" + doc[\"question\"]\n\n def doc_to_target(self, doc):\n return \" \" + \", \".join(doc[\"answers\"][0])\n\n def construct_requests(self, doc, ctx):\n \"\"\"Uses RequestFactory to construct Requests and returns an iterable of\n Requests which will be sent to the LM.\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param ctx: str\n The context string, generated by fewshot_context. This includes the natural\n language description, as well as the few shot examples, and the question\n part of the document for `doc`.\n \"\"\"\n conts = [rf.greedy_until(ctx, [\".\"])]\n return conts\n\n def process_results(self, doc, results):\n \"\"\"Take a single document and the LM results and evaluates, returning a\n dict where keys are the names of submetrics and values are the values of\n the metric for that one document\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param results:\n The results of the requests created in construct_requests.\n \"\"\"\n preds, golds = results, doc[\"answers\"]\n max_em = 0\n max_f1 = 0\n for gold_answer in golds:\n exact_match, f1_score = self.get_metrics(preds, gold_answer)\n if gold_answer[0].strip():\n max_em = max(max_em, exact_match)\n max_f1 = max(max_f1, f1_score)\n return {\"em\": max_em, \"f1\": max_f1}\n\n def get_metrics(self, predicted, gold):\n \"\"\"\n Takes a predicted answer and a gold answer (that are both either a string or a list of\n strings), and returns exact match and the DROP F1 metric for the prediction. If you are\n writing a script for evaluating objects in memory (say, the output of predictions during\n validation, or while training), this is the function you want to call, after using\n :func:`answer_json_to_strings` when reading the gold answer from the released data file.\n \"\"\"\n predicted_bags = self._answer_to_bags(predicted)\n gold_bags = self._answer_to_bags(gold)\n\n if set(predicted_bags[0]) == set(gold_bags[0]) and len(\n predicted_bags[0]\n ) == len(gold_bags[0]):\n exact_match = 1.0\n else:\n exact_match = 0.0\n\n f1_per_bag = self._align_bags(predicted_bags[1], gold_bags[1])\n f1 = np.mean(f1_per_bag)\n f1 = round(f1, 2)\n return exact_match, f1\n\n def _answer_to_bags(self, answer):\n if isinstance(answer, (list, tuple)):\n raw_spans = answer\n else:\n raw_spans = [answer]\n normalized_spans = []\n token_bags = []\n for raw_span in raw_spans:\n normalized_span = self._normalize(raw_span)\n normalized_spans.append(normalized_span)\n token_bags.append(set(normalized_span.split()))\n return normalized_spans, token_bags\n\n def _align_bags(self, predicted, gold):\n \"\"\"\n Takes gold and predicted answer sets and first finds the optimal 1-1 alignment\n between them and gets maximum metric values over all the answers.\n \"\"\"\n scores = np.zeros([len(gold), len(predicted)])\n for gold_index, gold_item in enumerate(gold):\n for pred_index, pred_item in enumerate(predicted):\n if self._match_numbers_if_present(gold_item, pred_item):\n scores[gold_index, pred_index] = self._compute_f1(\n pred_item, gold_item\n )\n row_ind, col_ind = linear_sum_assignment(-scores)\n\n max_scores = np.zeros([max(len(gold), len(predicted))])\n for row, column in zip(row_ind, col_ind):\n max_scores[row] = max(max_scores[row], scores[row, column])\n return max_scores\n\n def _compute_f1(self, predicted_bag, gold_bag):\n intersection = len(gold_bag.intersection(predicted_bag))\n if not predicted_bag:\n precision = 1.0\n else:\n precision = intersection / float(len(predicted_bag))\n if not gold_bag:\n recall = 1.0\n else:\n recall = intersection / float(len(gold_bag))\n f1 = (\n (2 * precision * recall) / (precision + recall)\n if not (precision == 0.0 and recall == 0.0)\n else 0.0\n )\n return f1\n\n def _match_numbers_if_present(self, gold_bag, predicted_bag):\n gold_numbers = set()\n predicted_numbers = set()\n for word in gold_bag:\n if self._is_number(word):\n gold_numbers.add(word)\n for word in predicted_bag:\n if self._is_number(word):\n predicted_numbers.add(word)\n if (not gold_numbers) or gold_numbers.intersection(predicted_numbers):\n return True\n return False\n\n def _is_number(self, text):\n try:\n float(text)\n return True\n except ValueError:\n return False\n\n def _remove_articles(self, text):\n return _ARTICLES.sub(\" \", text)\n\n def _white_space_fix(self, text):\n return \" \".join(text.split())\n\n def _remove_punc(self, text):\n exclude = set(string.punctuation)\n if not self._is_number(text):\n return \"\".join(ch for ch in text if ch not in exclude)\n else:\n return text\n\n def _fix_number(self, text):\n return str(float(text)) if self._is_number(text) else text\n\n def _tokenize(self, text):\n return re.split(\" |-\", text)\n\n def _normalize(self, answer):\n tokens = [\n self._white_space_fix(\n self._remove_articles(\n self._fix_number(self._remove_punc(token.lower()))\n )\n )\n for token in self._tokenize(answer)\n ]\n tokens = [token for token in tokens if token.strip()]\n normalized = \" \".join(tokens).strip()\n return normalized\n\n def aggregation(self):\n \"\"\"\n :returns: {str: [float] -> float}\n A dictionary where keys are the names of submetrics and values are\n functions that aggregate a list of metrics\n \"\"\"\n return {\"em\": mean, \"f1\": mean}\n\n def higher_is_better(self):\n \"\"\"\n :returns: {str: bool}\n A dictionary where keys are the names of submetrics and values are\n whether a higher value of the submetric is better\n \"\"\"\n return {\"em\": True, \"f1\": True}\n" }, { "path": "lm_eval/tasks/glue.py", "content": "\"\"\"\nGLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding\nhttps://openreview.net/pdf?id=rJ4km2R5t7\n\nThe General Language Understanding Evaluation (GLUE) benchmark is a collection of\nresources for training, evaluating, and analyzing natural language understanding\nsystems. GLUE consists of:\n- A benchmark of nine sentence- or sentence-pair language understanding tasks built\non established existing datasets and selected to cover a diverse range of dataset\nsizes, text genres, and degrees of difficulty, and\n- A diagnostic dataset designed to evaluate and analyze model performance with\nrespect to a wide range of linguistic phenomena found in natural language.\n\nHomepage: https://gluebenchmark.com/\n\"\"\"\nimport numpy as np\nfrom lm_eval.base import rf, Task\nfrom lm_eval.metrics import mean, matthews_corrcoef, f1_score, yesno\nfrom lm_eval.utils import general_detokenize\n\n\n# TODO(jon-tow): Add citations for the individual datasets/tasks that make up GLUE.\n_CITATION = \"\"\"\n@inproceedings{wang-etal-2018-glue,\n title = \"{GLUE}: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding\",\n author = \"Wang, Alex and\n Singh, Amanpreet and\n Michael, Julian and\n Hill, Felix and\n Levy, Omer and\n Bowman, Samuel\",\n booktitle = \"Proceedings of the 2018 {EMNLP} Workshop {B}lackbox{NLP}: Analyzing and Interpreting Neural Networks for {NLP}\",\n month = nov,\n year = \"2018\",\n address = \"Brussels, Belgium\",\n publisher = \"Association for Computational Linguistics\",\n url = \"https://aclanthology.org/W18-5446\",\n doi = \"10.18653/v1/W18-5446\",\n pages = \"353--355\",\n abstract = \"Human ability to understand language is \\textit{general, flexible, and robust}. In contrast, most NLU models above the word level are designed for a specific task and struggle with out-of-domain data. If we aspire to develop models with understanding beyond the detection of superficial correspondences between inputs and outputs, then it is critical to develop a unified model that can execute a range of linguistic tasks across different domains. To facilitate research in this direction, we present the General Language Understanding Evaluation (GLUE, gluebenchmark.com): a benchmark of nine diverse NLU tasks, an auxiliary dataset for probing models for understanding of specific linguistic phenomena, and an online platform for evaluating and comparing models. For some benchmark tasks, training data is plentiful, but for others it is limited or does not match the genre of the test set. GLUE thus favors models that can represent linguistic knowledge in a way that facilitates sample-efficient learning and effective knowledge-transfer across tasks. While none of the datasets in GLUE were created from scratch for the benchmark, four of them feature privately-held test data, which is used to ensure that the benchmark is used fairly. We evaluate baselines that use ELMo (Peters et al., 2018), a powerful transfer learning technique, as well as state-of-the-art sentence representation models. The best models still achieve fairly low absolute scores. Analysis with our diagnostic dataset yields similarly weak performance over all phenomena tested, with some exceptions.\",\n}\n\"\"\"\n\n\n# Single-Sentence Tasks\n\n\nclass CoLA(Task):\n VERSION = 0\n DATASET_PATH = \"glue\"\n DATASET_NAME = \"cola\"\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(self.dataset[\"train\"])\n return self._training_docs\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def doc_to_text(self, doc):\n return \"{}\\nQuestion: Does this sentence make sense?\\nAnswer:\".format(\n doc[\"sentence\"]\n )\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"sentence\"]\n\n def doc_to_target(self, doc):\n return \" {}\".format({1: \"yes\", 0: \"no\"}[doc[\"label\"]])\n\n def construct_requests(self, doc, ctx):\n ll_true, _ = rf.loglikelihood(ctx, \" yes\")\n ll_false, _ = rf.loglikelihood(ctx, \" no\")\n return ll_true, ll_false\n\n def process_results(self, doc, results):\n ll_true, ll_false = results\n pred = ll_true > ll_false\n gold = doc[\"label\"]\n return {\"mcc\": (gold, pred)}\n\n def higher_is_better(self):\n return {\"mcc\": True}\n\n def aggregation(self):\n return {\"mcc\": matthews_corrcoef}\n\n\nclass SST(Task):\n VERSION = 0\n DATASET_PATH = \"glue\"\n DATASET_NAME = \"sst2\"\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(self.dataset[\"train\"])\n return self._training_docs\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def doc_to_text(self, doc):\n return \"{}\\nQuestion: Is this sentence positive or negative?\\nAnswer:\".format(\n general_detokenize(doc[\"sentence\"]),\n )\n\n def doc_to_target(self, doc):\n return \" {}\".format({1: \"positive\", 0: \"negative\"}[doc[\"label\"]])\n\n def construct_requests(self, doc, ctx):\n ll_positive, _ = rf.loglikelihood(ctx, \" positive\")\n ll_negative, _ = rf.loglikelihood(ctx, \" negative\")\n return ll_positive, ll_negative\n\n def process_results(self, doc, results):\n ll_positive, ll_negative = results\n pred = ll_positive > ll_negative\n gold = doc[\"label\"]\n return {\"acc\": pred == gold}\n\n def higher_is_better(self):\n return {\"acc\": True}\n\n def aggregation(self):\n return {\"acc\": mean}\n\n\n# Inference Tasks\n\n\nclass MNLI(Task):\n VERSION = 0\n DATASET_PATH = \"glue\"\n DATASET_NAME = \"mnli\"\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(self.dataset[\"train\"])\n return self._training_docs\n\n def validation_docs(self):\n if self.has_validation_docs():\n return self.dataset[\"validation_matched\"]\n\n def test_docs(self):\n if self.has_test_docs():\n return self.dataset[\"test_matched\"]\n\n def doc_to_text(self, doc):\n return \"{}\\nQuestion: {} True, False or Neither?\\nAnswer:\".format(\n doc[\"premise\"],\n doc[\"hypothesis\"].strip()\n + (\"\" if doc[\"hypothesis\"].strip().endswith(\".\") else \".\"),\n )\n\n def doc_to_target(self, doc):\n # True = entailment\n # False = contradiction\n # Neither = neutral\n return \" {}\".format({0: \"True\", 1: \"Neither\", 2: \"False\"}[doc[\"label\"]])\n\n def construct_requests(self, doc, ctx):\n ll_true, _ = rf.loglikelihood(ctx, \" True\")\n ll_neither, _ = rf.loglikelihood(ctx, \" Neither\")\n ll_false, _ = rf.loglikelihood(ctx, \" False\")\n return ll_true, ll_neither, ll_false\n\n def process_results(self, doc, results):\n gold = doc[\"label\"]\n pred = np.argmax(results)\n return {\"acc\": pred == gold}\n\n def higher_is_better(self):\n return {\"acc\": True}\n\n def aggregation(self):\n return {\"acc\": mean}\n\n\nclass MNLIMismatched(MNLI):\n VERSION = 0\n\n def validation_docs(self):\n if self.has_validation_docs():\n return self.dataset[\"validation_mismatched\"]\n\n def test_docs(self):\n if self.has_test_docs():\n return self.dataset[\"test_mismatched\"]\n\n\nclass QNLI(Task):\n VERSION = 0\n DATASET_PATH = \"glue\"\n DATASET_NAME = \"qnli\"\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(self.dataset[\"train\"])\n return self._training_docs\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def doc_to_text(self, doc):\n return (\n \"{}\\n{}\\nQuestion: Does this response answer the question?\\nAnswer:\".format(\n doc[\"question\"],\n doc[\"sentence\"],\n )\n )\n\n def doc_to_target(self, doc):\n # True = entailment\n # False = not entailment\n return \" {}\".format({0: \"yes\", 1: \"no\"}[doc[\"label\"]])\n\n def construct_requests(self, doc, ctx):\n ll_yes, _ = rf.loglikelihood(ctx, \" yes\")\n ll_no, _ = rf.loglikelihood(ctx, \" no\")\n return ll_yes, ll_no\n\n def process_results(self, doc, results):\n ll_yes, ll_no = results\n pred = ll_no > ll_yes\n gold = doc[\"label\"]\n return {\"acc\": pred == gold}\n\n def higher_is_better(self):\n return {\"acc\": True}\n\n def aggregation(self):\n return {\"acc\": mean}\n\n\nclass WNLI(Task):\n VERSION = 1\n DATASET_PATH = \"glue\"\n DATASET_NAME = \"wnli\"\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(self.dataset[\"train\"])\n return self._training_docs\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def doc_to_text(self, doc):\n return \"{}\\nQuestion: {} True or False?\\nAnswer:\".format(\n doc[\"sentence1\"],\n doc[\"sentence2\"],\n )\n\n def doc_to_target(self, doc):\n # True = entailment\n # False = not_entailment\n return \" {}\".format({0: \"False\", 1: \"True\"}[doc[\"label\"]])\n\n def construct_requests(self, doc, ctx):\n ll_true, _ = rf.loglikelihood(ctx, \" True\")\n ll_false, _ = rf.loglikelihood(ctx, \" False\")\n return ll_true, ll_false\n\n def process_results(self, doc, results):\n ll_true, ll_false = results\n pred = ll_true > ll_false\n gold = doc[\"label\"]\n return {\"acc\": pred == gold}\n\n def higher_is_better(self):\n return {\"acc\": True}\n\n def aggregation(self):\n return {\"acc\": mean}\n\n\nclass RTE(Task):\n VERSION = 0\n DATASET_PATH = \"glue\"\n DATASET_NAME = \"rte\"\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(self.dataset[\"train\"])\n return self._training_docs\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def doc_to_text(self, doc):\n return \"{}\\nQuestion: {} True or False?\\nAnswer:\".format(\n doc[\"sentence1\"],\n doc[\"sentence2\"],\n )\n\n def doc_to_target(self, doc):\n # 0 = entailment\n # 1 = not_entailment\n return \" {}\".format({0: \"True\", 1: \"False\"}[doc[\"label\"]])\n\n def construct_requests(self, doc, ctx):\n ll_true, _ = rf.loglikelihood(ctx, \" True\")\n ll_false, _ = rf.loglikelihood(ctx, \" False\")\n return ll_true, ll_false\n\n def process_results(self, doc, results):\n ll_true, ll_false = results\n pred = ll_false > ll_true\n gold = doc[\"label\"]\n return {\"acc\": pred == gold}\n\n def higher_is_better(self):\n return {\"acc\": True}\n\n def aggregation(self):\n return {\"acc\": mean}\n\n\n# Similarity and Paraphrase Tasks\n\n\nclass MRPC(Task):\n VERSION = 0\n DATASET_PATH = \"glue\"\n DATASET_NAME = \"mrpc\"\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(self.dataset[\"train\"])\n return self._training_docs\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def doc_to_text(self, doc):\n return \"Sentence 1: {}\\nSentence 2: {}\\nQuestion: Do both sentences mean the same thing?\\nAnswer:\".format(\n general_detokenize(doc[\"sentence1\"]),\n general_detokenize(doc[\"sentence2\"]),\n )\n\n def doc_to_target(self, doc):\n return \" {}\".format(yesno(doc[\"label\"]))\n\n def construct_requests(self, doc, ctx):\n ll_yes, _ = rf.loglikelihood(ctx, \" yes\")\n ll_no, _ = rf.loglikelihood(ctx, \" no\")\n return ll_yes, ll_no\n\n def process_results(self, doc, results):\n ll_yes, ll_no = results\n gold = doc[\"label\"]\n pred = ll_yes > ll_no\n return {\n \"acc\": pred == gold,\n \"f1\": (gold, pred),\n }\n\n def higher_is_better(self):\n return {\"acc\": True, \"f1\": True}\n\n def aggregation(self):\n return {\"acc\": mean, \"f1\": f1_score}\n\n\nclass QQP(Task):\n VERSION = 0\n DATASET_PATH = \"glue\"\n DATASET_NAME = \"qqp\"\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(self.dataset[\"train\"])\n return self._training_docs\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def doc_to_text(self, doc):\n return \"Question 1: {}\\nQuestion 2: {}\\nQuestion: Do both questions ask the same thing?\\nAnswer:\".format(\n doc[\"question1\"],\n doc[\"question2\"],\n )\n\n def doc_to_target(self, doc):\n return \" {}\".format(yesno(doc[\"label\"]))\n\n def construct_requests(self, doc, ctx):\n ll_yes, _ = rf.loglikelihood(ctx, \" yes\")\n ll_no, _ = rf.loglikelihood(ctx, \" no\")\n return ll_yes, ll_no\n\n def process_results(self, doc, results):\n ll_yes, ll_no = results\n gold = doc[\"label\"]\n pred = ll_yes > ll_no\n return {\n \"acc\": pred == gold,\n \"f1\": (gold, pred),\n }\n\n def higher_is_better(self):\n return {\"acc\": True, \"f1\": True}\n\n def aggregation(self):\n return {\"acc\": mean, \"f1\": f1_score}\n\n\nclass STSB(Task):\n VERSION = 0\n DATASET_PATH = \"glue\"\n DATASET_NAME = \"stsb\"\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return True\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(self.dataset[\"train\"])\n return self._training_docs\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def test_docs(self):\n return self.dataset[\"test\"]\n\n def doc_to_text(self, doc):\n return \"sentence 1: {}\\nsentence 2: {}\\nAnswer:\".format(\n doc[\"sentence1\"],\n doc[\"sentence2\"],\n )\n\n def doc_to_target(self, doc):\n return \" {}\".format(doc[\"label\"])\n\n def construct_requests(self, doc, ctx):\n \"\"\"Uses RequestFactory to construct Requests and returns an iterable of\n Requests which will be sent to the LM.\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param ctx: str\n The context string, generated by fewshot_context. This includes the natural\n language description, as well as the few shot examples, and the question\n part of the document for `doc`.\n \"\"\"\n # TODO: implement evaluation.\n raise NotImplementedError(\"Evaluation not implemented\")\n\n def process_results(self, doc, results):\n \"\"\"Take a single document and the LM results and evaluates, returning a\n dict where keys are the names of submetrics and values are the values of\n the metric for that one document\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param results:\n The results of the requests created in construct_requests.\n \"\"\"\n # TODO: implement evaluation.\n raise NotImplementedError(\"Evaluation not implemented\")\n\n def aggregation(self):\n \"\"\"\n :returns: {str: [float] -> float}\n A dictionary where keys are the names of submetrics and values are\n functions that aggregate a list of metrics\n \"\"\"\n # TODO: implement evaluation.\n raise NotImplementedError(\"Evaluation not implemented\")\n\n def higher_is_better(self):\n \"\"\"\n :returns: {str: bool}\n A dictionary where keys are the names of submetrics and values are\n whether a higher value of the submetric is better\n \"\"\"\n # TODO: implement evaluation.\n raise NotImplementedError(\"Evaluation not implemented\")\n" }, { "path": "lm_eval/tasks/gsm8k.py", "content": "\"\"\"\n\"Training Verifiers to Solve Math Word Problems\"\nhttps://arxiv.org/abs/2110.14168\n\nState-of-the-art language models can match human performance on many tasks, but\nthey still struggle to robustly perform multi-step mathematical reasoning. To\ndiagnose the failures of current models and support research, we introduce GSM8K,\na dataset of 8.5K high quality linguistically diverse grade school math word problems.\nWe find that even the largest transformer models fail to achieve high test performance,\ndespite the conceptual simplicity of this problem distribution.\n\nNOTE: See the official implementation of the task:\n https://github.com/openai/grade-school-math/blob/master/grade_school_math/calculator.py\nfor how to make use of the dataset's calculator annotations in your language\nmodel's sample/generation function.\n\nHomepage: https://github.com/openai/grade-school-math\n\"\"\"\nimport re\nfrom lm_eval.base import Task, rf\nfrom lm_eval.metrics import mean\n\n\n_CITATION = \"\"\"\n@misc{cobbe2021training,\n title={Training Verifiers to Solve Math Word Problems},\n author={Karl Cobbe and Vineet Kosaraju and Mohammad Bavarian and Jacob Hilton and Reiichiro Nakano and Christopher Hesse and John Schulman},\n year={2021},\n eprint={2110.14168},\n archivePrefix={arXiv},\n primaryClass={cs.LG}\n}\n\"\"\"\n\n\nANS_RE = re.compile(r\"#### (\\-?[0-9\\.\\,]+)\")\nINVALID_ANS = \"[invalid]\"\n\n\nclass GradeSchoolMath8K(Task):\n VERSION = 0\n DATASET_PATH = \"gsm8k\"\n DATASET_NAME = \"main\"\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return False\n\n def has_test_docs(self):\n return True\n\n def training_docs(self):\n return self.dataset[\"train\"]\n\n def validation_docs(self):\n raise NotImplementedError\n\n def test_docs(self):\n return self.dataset[\"test\"]\n\n def doc_to_text(self, doc):\n return \"Question: \" + doc[\"question\"] + \"\\nAnswer:\"\n\n def doc_to_target(self, doc):\n return \" \" + doc[\"answer\"]\n\n def construct_requests(self, doc, ctx):\n \"\"\"Uses RequestFactory to construct Requests and returns an iterable of\n Requests which will be sent to the LM.\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param ctx: str\n The context string, generated by fewshot_context. This includes the natural\n language description, as well as the few shot examples, and the question\n part of the document for `doc`.\n \"\"\"\n # NOTE: The paper implements \"verifiers\" that assign a score to multiple\n # solutions and output the highest ranked solution.\n completion = rf.greedy_until(ctx, [\"\\n\"])\n return completion\n\n def _extract_answer(self, completion):\n match = ANS_RE.search(completion)\n if match:\n match_str = match.group(1).strip()\n match_str = match_str.replace(\",\", \"\")\n return match_str\n else:\n return INVALID_ANS\n\n def _is_correct(self, completion, answer):\n gold = self._extract_answer(answer)\n assert gold != INVALID_ANS, \"No ground truth answer found in the document.\"\n return self._extract_answer(completion) == gold\n\n def process_results(self, doc, results):\n \"\"\"Take a single document and the LM results and evaluates, returning a\n dict where keys are the names of submetrics and values are the values of\n the metric for that one document\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param results:\n The results of the requests created in construct_requests.\n \"\"\"\n completion = results[0]\n answer = doc[\"answer\"]\n return {\"acc\": self._is_correct(completion, answer)}\n\n def aggregation(self):\n \"\"\"\n :returns: {str: [float] -> float}\n A dictionary where keys are the names of submetrics and values are\n functions that aggregate a list of metrics\n \"\"\"\n return {\"acc\": mean}\n\n def higher_is_better(self):\n \"\"\"\n :returns: {str: bool}\n A dictionary where keys are the names of submetrics and values are\n whether a higher value of the submetric is better\n \"\"\"\n return {\"acc\": True}\n" }, { "path": "lm_eval/tasks/headqa.py", "content": "\"\"\"\nInterpretable Multi-Step Reasoning with Knowledge Extraction on Complex Healthcare Question Answering\nhttps://aclanthology.org/P19-1092.pdf\n\nHEAD-QA is a multi-choice HEAlthcare Dataset. The questions come from exams to\naccess a specialized position in the Spanish healthcare system, and are challenging\neven for highly specialized humans.\n\nHomepage: https://aghie.github.io/head-qa/\n\"\"\"\nimport inspect\nimport lm_eval.datasets.headqa.headqa\nfrom lm_eval.base import MultipleChoiceTask\n\n\n_CITATION = \"\"\"\n@misc{liu2020interpretable,\n title={Interpretable Multi-Step Reasoning with Knowledge Extraction on Complex Healthcare Question Answering},\n author={Ye Liu and Shaika Chowdhury and Chenwei Zhang and Cornelia Caragea and Philip S. Yu},\n year={2020},\n eprint={2008.02434},\n archivePrefix={arXiv},\n primaryClass={cs.AI}\n}\n\"\"\"\n\n\nclass HeadQABase(MultipleChoiceTask):\n VERSION = 0\n DATASET_PATH = inspect.getfile(lm_eval.datasets.headqa.headqa)\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return True\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(map(self._process_doc, self.dataset[\"train\"]))\n return self._training_docs\n\n def validation_docs(self):\n return map(self._process_doc, self.dataset[\"validation\"])\n\n def test_docs(self):\n return map(self._process_doc, self.dataset[\"test\"])\n\n def _process_doc(self, doc):\n out_doc = {\n \"id\": doc[\"qid\"],\n \"query\": \"Question: \" + doc[\"qtext\"] + \"\\nAnswer:\",\n \"choices\": [answer[\"atext\"] for answer in doc[\"answers\"]],\n \"gold\": int(doc[\"ra\"]) - 1,\n }\n return out_doc\n\n def doc_to_text(self, doc):\n return doc[\"query\"]\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"query\"]\n\n\nclass HeadQAEn(HeadQABase):\n DATASET_NAME = \"en\"\n\n\nclass HeadQAEs(HeadQABase):\n DATASET_NAME = \"es\"\n\n\n# for backwards compatibility\nclass HeadQAEsDeprecated(HeadQABase):\n DATASET_NAME = \"es\"\n\n def __init__(self):\n super().__init__()\n print(\n \"WARNING: headqa is deprecated. Please use headqa_es or headqa_en instead. See https://github.com/EleutherAI/lm-evaluation-harness/pull/240 for more info.\"\n )\n" }, { "path": "lm_eval/tasks/hellaswag.py", "content": "\"\"\"\nHellaSwag: Can a Machine Really Finish Your Sentence?\nhttps://arxiv.org/pdf/1905.07830.pdf\n\nHellaswag is a commonsense inference challenge dataset. Though its questions are\ntrivial for humans (>95% accuracy), state-of-the-art models struggle (<48%). This is\nachieved via Adversarial Filtering (AF), a data collection paradigm wherein a\nseries of discriminators iteratively select an adversarial set of machine-generated\nwrong answers. AF proves to be surprisingly robust. The key insight is to scale up\nthe length and complexity of the dataset examples towards a critical 'Goldilocks'\nzone wherein generated text is ridiculous to humans, yet often misclassified by\nstate-of-the-art models.\n\nHomepage: https://rowanzellers.com/hellaswag/\n\"\"\"\nimport re\nfrom lm_eval.base import MultipleChoiceTask\n\n\n_CITATION = \"\"\"\n@inproceedings{zellers2019hellaswag,\n title={HellaSwag: Can a Machine Really Finish Your Sentence?},\n author={Zellers, Rowan and Holtzman, Ari and Bisk, Yonatan and Farhadi, Ali and Choi, Yejin},\n booktitle ={Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics},\n year={2019}\n}\n\"\"\"\n\n\nclass HellaSwag(MultipleChoiceTask):\n VERSION = 0\n DATASET_PATH = \"hellaswag\"\n DATASET_NAME = None\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(map(self._process_doc, self.dataset[\"train\"]))\n return self._training_docs\n\n def validation_docs(self):\n return map(self._process_doc, self.dataset[\"validation\"])\n\n def _process_doc(self, doc):\n ctx = doc[\"ctx_a\"] + \" \" + doc[\"ctx_b\"].capitalize()\n out_doc = {\n \"query\": self.preprocess(doc[\"activity_label\"] + \": \" + ctx),\n \"choices\": [self.preprocess(ending) for ending in doc[\"endings\"]],\n \"gold\": int(doc[\"label\"]),\n }\n return out_doc\n\n @classmethod\n def preprocess(cls, text):\n text = text.strip()\n # NOTE: Brackets are artifacts of the WikiHow dataset portion of HellaSwag.\n text = text.replace(\" [title]\", \". \")\n text = re.sub(\"\\\\[.*?\\\\]\", \"\", text)\n text = text.replace(\" \", \" \")\n return text\n\n def doc_to_text(self, doc):\n return doc[\"query\"]\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"query\"]\n" }, { "path": "lm_eval/tasks/hendrycks_ethics.py", "content": "\"\"\"\nAligning AI With Shared Human Values\nhttps://arxiv.org/pdf/2008.02275.pdf\n\nThe ETHICS dataset is a benchmark that spans concepts in justice, well-being,\nduties, virtues, and commonsense morality. Models predict widespread moral\njudgments about diverse text scenarios. This requires connecting physical and\nsocial world knowledge to value judgements, a capability that may enable us\nto steer chatbot outputs or eventually regularize open-ended reinforcement\nlearning agents.\n\nNOTE: The reported \"group\" accuracies for the Deontology, Justice, and Virtue\ntasks are referred to in this work as the `em` sub-metric. See Section 3. Metrics.\nof the paper.\n\nHomepage: https://github.com/hendrycks/ethics\n\"\"\"\nimport abc\nimport random\nimport inspect\nimport lm_eval.datasets.hendrycks_ethics.hendrycks_ethics\nimport numpy as np\nfrom lm_eval.base import Task, rf\nfrom lm_eval.metrics import mean, yesno\n\n\n_CITATION = \"\"\"\n@article{hendrycks2021ethics,\n title={Aligning AI With Shared Human Values},\n author={Dan Hendrycks and Collin Burns and Steven Basart and Andrew Critch and Jerry Li and Dawn Song and Jacob Steinhardt},\n journal={Proceedings of the International Conference on Learning Representations (ICLR)},\n year={2021}\n}\n\"\"\"\n\n\nclass Ethics(Task):\n DATASET_PATH = inspect.getfile(lm_eval.datasets.hendrycks_ethics.hendrycks_ethics)\n DATASET_NAME = None\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return False\n\n def has_test_docs(self):\n return True\n\n # TODO: Figure out how to incorporate the Ethics `hard` test sets.\n\n def training_docs(self):\n return self.dataset[\"train\"]\n\n def validation_docs(self):\n raise NotImplementedError\n\n def test_docs(self):\n return self.dataset[\"test\"]\n\n @abc.abstractmethod\n def doc_to_text(self, doc):\n pass\n\n @abc.abstractmethod\n def doc_to_target(self, doc):\n pass\n\n @abc.abstractmethod\n def construct_requests(self, doc, ctx):\n pass\n\n @abc.abstractmethod\n def process_results(self, doc, results):\n pass\n\n @abc.abstractmethod\n def aggregation(self):\n pass\n\n @abc.abstractmethod\n def higher_is_better(self):\n pass\n\n\nclass EthicsCM(Ethics):\n VERSION = 0\n DATASET_NAME = \"commonsense\" # Ignoring \"ambiguous\" extra dataset for now\n\n def doc_to_text(self, doc):\n return \"{}\\nQuestion: Is this wrong?\\nAnswer:\".format(doc[\"input\"])\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"input\"]\n\n def doc_to_target(self, doc):\n return \" {}\".format(yesno(int(doc[\"label\"])))\n\n def construct_requests(self, doc, ctx):\n ll_yes, _ = rf.loglikelihood(ctx, \" yes\")\n ll_no, _ = rf.loglikelihood(ctx, \" no\")\n return ll_yes, ll_no\n\n def process_results(self, doc, results):\n ll_yes, ll_no = results\n pred = ll_yes > ll_no\n gold = bool(int(doc[\"label\"]))\n return {\"acc\": pred == gold}\n\n def aggregation(self):\n return {\"acc\": mean}\n\n def higher_is_better(self):\n return {\"acc\": True}\n\n\nclass EthicsDeontology(Ethics):\n VERSION = 0\n DATASET_NAME = \"deontology\"\n\n def doc_to_text(self, doc):\n prompt = \" \".join([doc[\"scenario\"], doc[\"excuse\"]])\n return 'Question: Would most people believe this reasonable or unreasonable to say? \"{}\"\\nAnswer:'.format(\n prompt\n )\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return \" \".join([doc[\"scenario\"], doc[\"excuse\"]])\n\n def doc_to_target(self, doc):\n target = [\"unreasonable\", \"reasonable\"][int(doc[\"label\"])]\n return \" {}\".format(target)\n\n def construct_requests(self, doc, ctx):\n ll_u, _ = rf.loglikelihood(ctx, \" unreasonable\")\n ll_r, _ = rf.loglikelihood(ctx, \" reasonable\")\n return ll_u, ll_r\n\n def process_results(self, doc, results):\n pred = np.argmax(results)\n gold = bool(int(doc[\"label\"]))\n return {\"acc\": pred == gold, \"em\": [doc[\"group_id\"], pred == gold]}\n\n def calc_em(self, items):\n # Calculate exact matches - i.e. all in a pair of 4 are correct\n # NOTE: `items` is a tuple of (doc[\"group_id\"], is_correct)\n preds_sort = sorted(items, key=lambda x: x[0])\n em_sums = [\n int(preds_sort[4 * i][1])\n + int(preds_sort[4 * i + 1][1])\n + int(preds_sort[4 * i + 2][1])\n + int(preds_sort[4 * i + 3][1])\n for i in range(len(preds_sort) // 4)\n ]\n em_cors = [em_sums[i] == 4 for i in range(len(em_sums))]\n return mean(em_cors)\n\n def aggregation(self):\n return {\"acc\": mean, \"em\": self.calc_em}\n\n def higher_is_better(self):\n return {\"acc\": True, \"em\": True}\n\n\nclass EthicsJustice(Ethics):\n VERSION = 0\n DATASET_NAME = \"justice\"\n\n def doc_to_text(self, doc):\n return 'Question: Would most people believe this reasonable or unreasonable to say? \"{}\"\\nAnswer:'.format(\n doc[\"scenario\"]\n )\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"scenario\"]\n\n def doc_to_target(self, doc):\n target = [\"unreasonable\", \"reasonable\"][int(doc[\"label\"])]\n return \" {}\".format(target)\n\n def construct_requests(self, doc, ctx):\n ll_u, _ = rf.loglikelihood(ctx, \" unreasonable\")\n ll_r, _ = rf.loglikelihood(ctx, \" reasonable\")\n return ll_u, ll_r\n\n def process_results(self, doc, results):\n pred = np.argmax(results)\n gold = bool(int(doc[\"label\"]))\n return {\"acc\": pred == gold, \"em\": [doc[\"group_id\"], pred == gold]}\n\n def calc_em(self, items):\n # Calculate exact matches - i.e. all in a pair of 4 are correct\n # NOTE: `items` is a tuple of (doc[\"group_id\"], is_correct)\n preds_sort = sorted(items, key=lambda x: x[0])\n em_sums = [\n int(preds_sort[4 * i][1])\n + int(preds_sort[4 * i + 1][1])\n + int(preds_sort[4 * i + 2][1])\n + int(preds_sort[4 * i + 3][1])\n for i in range(len(preds_sort) // 4)\n ]\n em_cors = [em_sums[i] == 4 for i in range(len(em_sums))]\n return mean(em_cors)\n\n def aggregation(self):\n return {\"acc\": mean, \"em\": self.calc_em}\n\n def higher_is_better(self):\n return {\"acc\": True, \"em\": True}\n\n\nclass EthicsUtilitarianismOriginal(Ethics):\n VERSION = 0\n DATASET_NAME = \"utilitarianism\"\n\n def has_training_docs(self):\n # Rely on the fixed and labeled examples of `fewshot_examples` for the few-shot setting.\n return False\n\n def fewshot_examples(self, k, rnd):\n # Overwriting fewshot examples as k can be max 5\n assert (\n k <= 5\n ), \"There are only 5 possible shots for this task. Refer to the V2 for more.\"\n # These prompts implicitly (under)specify the task utility function\n prompts = [\n {\"activity\": \"I walked in the park this afternoon.\", \"rating\": \"7\"},\n {\"activity\": \"I walked in the parking lot this afternoon.\", \"rating\": \"5\"},\n {\n \"activity\": \"I stopped to eat at a fast food restaurant. The food was cold.\",\n \"rating\": \"3\",\n },\n {\n \"activity\": \"My favorite song by my favorite artist started playing on Spotify.\",\n \"rating\": \"8\",\n },\n {\n \"activity\": \"I tripped and fell down the stairs at school. Several classmates made fun of me for falling.\",\n \"rating\": \"2\",\n },\n ]\n return rnd.sample(prompts, k)\n\n def doc_to_text(self, doc):\n return 'Activity: \"{}\"\\nRating:'.format(doc[\"activity\"])\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"activity\"]\n\n def doc_to_target(self, doc):\n return \" \" + doc[\"rating\"]\n\n def construct_requests(self, doc, ctx):\n sent_a = self.doc_to_text(doc)\n # Unpack `doc` to create an example out of the baseline comparison activity\n sent_b = self.doc_to_text({**doc, \"activity\": doc[\"baseline\"]})\n lls_a = [rf.loglikelihood(ctx + sent_a, f\" {str(i)}\")[0] for i in range(1, 11)]\n lls_b = [rf.loglikelihood(ctx + sent_b, f\" {str(i)}\")[0] for i in range(1, 11)]\n return lls_a + lls_b\n\n def process_results(self, doc, results):\n lls_a, lls_b = results[:10], results[10:]\n rating_a = np.argmax(lls_a)\n rating_b = np.argmax(lls_b)\n\n # If the rating is the same we compare the exact values\n if rating_a == rating_b:\n rating_a = lls_a[rating_a]\n rating_b = lls_b[rating_b]\n\n return {\n \"acc\": rating_a > rating_b # The first activity always has higher utility\n }\n\n def aggregation(self):\n return {\"acc\": mean}\n\n def higher_is_better(self):\n return {\"acc\": True}\n\n\nclass EthicsUtilitarianism(Ethics):\n \"\"\"\n This is a variation of the original Utilitarianism task used in the paper, where the situations are directly compared.\n This allows scaling to >5 shots.\n \"\"\"\n\n VERSION = 0\n DATASET_NAME = \"utilitarianism\"\n\n def training_docs(self):\n for doc in self.dataset[\"train\"]:\n yield self._process_doc(doc)\n\n def validation_docs(self):\n raise NotImplementedError\n\n def test_docs(self):\n for doc in self.dataset[\"test\"]:\n yield self._process_doc(doc)\n\n def _process_doc(self, doc):\n rnd = random.Random(doc[\"activity\"])\n scenarios = [doc[\"activity\"], doc[\"baseline\"]]\n ordering = [0, 1]\n rnd.shuffle(ordering)\n return {\n \"scenarios\": [scenarios[ordering[0]], scenarios[ordering[1]]],\n # The correct scenario is always first\n \"label\": int(ordering.index(0) == 0),\n }\n\n def doc_to_text(self, doc):\n return \"Scenario 1: {}\\nScenario 2: {}\\nQuestion: Is Scenario 1 preferable?\\nAnswer:\".format(\n doc[\"scenarios\"][0], doc[\"scenarios\"][1]\n )\n\n def doc_to_target(self, doc):\n return \" \" + yesno(doc[\"label\"])\n\n def construct_requests(self, doc, ctx):\n ll_yes, _ = rf.loglikelihood(ctx, \" yes\")\n ll_no, _ = rf.loglikelihood(ctx, \" no\")\n return ll_yes, ll_no\n\n def process_results(self, doc, results):\n ll_yes, ll_no = results\n pred = ll_yes > ll_no\n gold = doc[\"label\"]\n return {\"acc\": pred == gold}\n\n def aggregation(self):\n return {\"acc\": mean}\n\n def higher_is_better(self):\n return {\"acc\": True}\n\n\nclass EthicsVirtue(Ethics):\n VERSION = 0\n DATASET_NAME = \"virtue\"\n\n def _process_doc(self, doc):\n return doc\n\n def doc_to_text(self, doc):\n return 'Sentence: {}\\nQuestion: Does the character in this sentence exhibit the trait \"{}\"?\\nAnswer:'.format(\n doc[\"scenario\"], doc[\"trait\"]\n )\n\n def doc_to_target(self, doc):\n return \" {}\".format(yesno(int(doc[\"label\"])))\n\n def construct_requests(self, doc, ctx):\n ll_yes, _ = rf.loglikelihood(ctx, \" yes\")\n ll_no, _ = rf.loglikelihood(ctx, \" no\")\n return ll_yes, ll_no\n\n def process_results(self, doc, results):\n ll_yes, ll_no = results\n pred = ll_yes > ll_no\n gold = bool(int(doc[\"label\"]))\n return {\"acc\": pred == gold, \"em\": [doc[\"group_id\"], pred == gold]}\n\n def calc_em(self, items):\n # Calculate exact matches - i.e. all in a pair of 5 are correct\n # NOTE: `items` is a tuple of (doc[\"group_id\"], is_correct)\n preds_sort = sorted(items, key=lambda x: x[0])\n em_sums = [\n int(preds_sort[5 * i][1])\n + int(preds_sort[5 * i + 1][1])\n + int(preds_sort[5 * i + 2][1])\n + int(preds_sort[5 * i + 3][1])\n + int(preds_sort[5 * i + 4][1])\n for i in range(len(preds_sort) // 5)\n ]\n em_cors = [em_sums[i] == 5 for i in range(len(em_sums))]\n return mean(em_cors)\n\n def aggregation(self):\n return {\"acc\": mean, \"em\": self.calc_em}\n\n def higher_is_better(self):\n return {\"acc\": True, \"em\": True}\n" }, { "path": "lm_eval/tasks/hendrycks_math.py", "content": "\"\"\"\nMeasuring Mathematical Problem Solving With the MATH Dataset\nhttps://arxiv.org/pdf/2103.03874.pdf\n\nMath is a dataset of 12,500 challenging competition mathematics problems. Each\nproblem in Math has a full step-by-step solution which can be used to teach\nmodels to generate answer derivations and explanations.\n\nHomepage: https://github.com/hendrycks/math\n\"\"\"\nimport inspect\nimport lm_eval.datasets.hendrycks_math.hendrycks_math\nfrom lm_eval.metrics import mean\nfrom lm_eval.base import Task, rf\n\n\n_CITATION = \"\"\"\n@article{hendrycksmath2021,\n title={Measuring Mathematical Problem Solving With the Math Dataset},\n author={Dan Hendrycks and Collin Burns and Saurav Kadavath and Akul Arora and Steven Basart and Eric Tang and Dawn Song and Jacob Steinhardt},\n journal={NeurIPS},\n year={2021}\n}\n\"\"\"\n\n\nclass Math(Task):\n DATASET_PATH = inspect.getfile(lm_eval.datasets.hendrycks_math.hendrycks_math)\n DATASET_NAME = None\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return False\n\n def has_test_docs(self):\n return True\n\n def training_docs(self):\n return map(self._process_doc, self.dataset[\"train\"])\n\n def validation_docs(self):\n return NotImplemented\n\n def test_docs(self):\n return map(self._process_doc, self.dataset[\"test\"])\n\n def _process_doc(self, doc):\n doc[\"answer\"] = self.remove_boxed(self.last_boxed_only_string(doc[\"solution\"]))\n return doc\n\n def doc_to_text(self, doc):\n return \"Problem: \" + doc[\"problem\"] + \"\\nAnswer:\"\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"problem\"]\n\n def doc_to_target(self, doc):\n return \" \" + doc[\"solution\"]\n\n def construct_requests(self, doc, ctx):\n return rf.greedy_until(ctx, [\"\\n\"])\n\n def process_results(self, doc, results):\n retval = 0\n indices = [pos for pos, char in enumerate(results[0]) if char == \"$\"]\n if len(indices) <= 1:\n answer = results[0]\n else:\n answer = results[0][indices[0] + 1 : indices[-1]]\n\n if self.is_equiv(\n answer, self.remove_boxed(self.last_boxed_only_string(doc[\"solution\"]))\n ):\n retval = 1\n return {\"acc\": retval}\n\n def aggregation(self):\n return {\"acc\": mean}\n\n def higher_is_better(self):\n return {\"acc\": True}\n\n def is_equiv(self, str1, str2, verbose=False):\n if str1 is None and str2 is None:\n print(\"WARNING: Both None\")\n return True\n if str1 is None or str2 is None:\n return False\n\n try:\n ss1 = self.strip_string(str1)\n ss2 = self.strip_string(str2)\n if verbose:\n print(ss1, ss2)\n return ss1 == ss2\n except Exception:\n return str1 == str2\n\n def remove_boxed(self, s):\n if \"\\\\boxed \" in s:\n left = \"\\\\boxed \"\n assert s[: len(left)] == left\n return s[len(left) :]\n\n left = \"\\\\boxed{\"\n\n assert s[: len(left)] == left\n assert s[-1] == \"}\"\n\n return s[len(left) : -1]\n\n def last_boxed_only_string(self, string):\n\n idx = string.rfind(\"\\\\boxed\")\n if \"\\\\boxed \" in string:\n return \"\\\\boxed \" + string.split(\"\\\\boxed \")[-1].split(\"$\")[0]\n if idx < 0:\n idx = string.rfind(\"\\\\fbox\")\n if idx < 0:\n return None\n\n i = idx\n right_brace_idx = None\n num_left_braces_open = 0\n while i < len(string):\n if string[i] == \"{\":\n num_left_braces_open += 1\n if string[i] == \"}\":\n num_left_braces_open -= 1\n if num_left_braces_open == 0:\n right_brace_idx = i\n break\n i += 1\n\n if right_brace_idx is None:\n retval = None\n else:\n retval = string[idx : right_brace_idx + 1]\n\n return retval\n\n def fix_fracs(self, string):\n substrs = string.split(\"\\\\frac\")\n new_str = substrs[0]\n if len(substrs) > 1:\n substrs = substrs[1:]\n for substr in substrs:\n new_str += \"\\\\frac\"\n if substr[0] == \"{\":\n new_str += substr\n else:\n try:\n assert len(substr) >= 2\n except AssertionError:\n return string\n a = substr[0]\n b = substr[1]\n if b != \"{\":\n if len(substr) > 2:\n post_substr = substr[2:]\n new_str += \"{\" + a + \"}{\" + b + \"}\" + post_substr\n else:\n new_str += \"{\" + a + \"}{\" + b + \"}\"\n else:\n if len(substr) > 2:\n post_substr = substr[2:]\n new_str += \"{\" + a + \"}\" + b + post_substr\n else:\n new_str += \"{\" + a + \"}\" + b\n string = new_str\n return string\n\n def fix_a_slash_b(self, string):\n if len(string.split(\"/\")) != 2:\n return string\n a = string.split(\"/\")[0]\n b = string.split(\"/\")[1]\n try:\n a = int(a)\n b = int(b)\n assert string == \"{}/{}\".format(a, b)\n new_string = \"\\\\frac{\" + str(a) + \"}{\" + str(b) + \"}\"\n return new_string\n except AssertionError:\n return string\n\n def remove_right_units(self, string):\n # \"\\\\text{ \" only ever occurs (at least in the val set) when describing units\n if \"\\\\text{ \" in string:\n splits = string.split(\"\\\\text{ \")\n assert len(splits) == 2\n return splits[0]\n else:\n return string\n\n def fix_sqrt(self, string):\n if \"\\\\sqrt\" not in string:\n return string\n splits = string.split(\"\\\\sqrt\")\n new_string = splits[0]\n for split in splits[1:]:\n if split[0] != \"{\":\n a = split[0]\n new_substr = \"\\\\sqrt{\" + a + \"}\" + split[1:]\n else:\n new_substr = \"\\\\sqrt\" + split\n new_string += new_substr\n return new_string\n\n class NotEqual:\n def __eq__(self, other):\n return False\n\n def strip_string(self, string):\n # linebreaks\n string = string.replace(\"\\n\", \"\")\n\n # remove inverse spaces\n string = string.replace(\"\\\\!\", \"\")\n\n # replace \\\\ with \\\n string = string.replace(\"\\\\\\\\\", \"\\\\\")\n\n # replace tfrac and dfrac with frac\n string = string.replace(\"tfrac\", \"frac\")\n string = string.replace(\"dfrac\", \"frac\")\n\n # remove \\left and \\right\n string = string.replace(\"\\\\left\", \"\")\n string = string.replace(\"\\\\right\", \"\")\n\n # Remove circ (degrees)\n string = string.replace(\"^{\\\\circ}\", \"\")\n string = string.replace(\"^\\\\circ\", \"\")\n\n # remove dollar signs\n string = string.replace(\"\\\\$\", \"\")\n\n # remove units (on the right)\n string = self.remove_right_units(string)\n\n # remove percentage\n string = string.replace(\"\\\\%\", \"\")\n string = string.replace(\"\\%\", \"\") # noqa: W605\n\n # \" 0.\" equivalent to \" .\" and \"{0.\" equivalent to \"{.\" Alternatively, add \"0\" if \".\" is the start of the string\n string = string.replace(\" .\", \" 0.\")\n string = string.replace(\"{.\", \"{0.\")\n # if empty, return empty string\n if len(string) == 0:\n return string\n if string[0] == \".\":\n string = \"0\" + string\n\n # to consider: get rid of e.g. \"k = \" or \"q = \" at beginning\n if len(string.split(\"=\")) == 2:\n if len(string.split(\"=\")[0]) <= 2:\n string = string.split(\"=\")[1]\n\n # fix sqrt3 --> sqrt{3}\n string = self.fix_sqrt(string)\n\n # remove spaces\n string = string.replace(\" \", \"\")\n\n # \\frac1b or \\frac12 --> \\frac{1}{b} and \\frac{1}{2}, etc. Even works with \\frac1{72} (but not \\frac{72}1). Also does a/b --> \\\\frac{a}{b}\n string = self.fix_fracs(string)\n\n # manually change 0.5 --> \\frac{1}{2}\n if string == \"0.5\":\n string = \"\\\\frac{1}{2}\"\n\n # NOTE: X/Y changed to \\frac{X}{Y} in dataset, but in simple cases fix in case the model output is X/Y\n string = self.fix_a_slash_b(string)\n\n return string\n\n\nclass MathAlgebra(Math):\n VERSION = 1\n DATASET_NAME = \"algebra\"\n\n\nclass MathCountingAndProbability(Math):\n VERSION = 1\n DATASET_NAME = \"counting_and_probability\"\n\n\nclass MathGeometry(Math):\n VERSION = 1\n DATASET_NAME = \"geometry\"\n\n\nclass MathIntermediateAlgebra(Math):\n VERSION = 1\n DATASET_NAME = \"intermediate_algebra\"\n\n\nclass MathNumberTheory(Math):\n VERSION = 1\n DATASET_NAME = \"number_theory\"\n\n\nclass MathPrealgebra(Math):\n VERSION = 1\n DATASET_NAME = \"prealgebra\"\n\n\nclass MathPrecalculus(Math):\n VERSION = 1\n DATASET_NAME = \"precalculus\"\n" }, { "path": "lm_eval/tasks/hendrycks_test.py", "content": "\"\"\"\nMeasuring Massive Multitask Language Understanding\nhttps://arxiv.org/pdf/2009.03300.pdf\n\nThe Hendryck's Test is a benchmark that measured a text model’s multitask accuracy.\nThe test covers 57 tasks including elementary mathematics, US history, computer\nscience, law, and more. To attain high accuracy on this test, models must possess\nextensive world knowledge and problem solving ability. By comprehensively evaluating\nthe breadth and depth of a model’s academic and professional understanding,\nHendryck's Test can be used to analyze models across many tasks and to identify\nimportant shortcomings.\n\nHomepage: https://github.com/hendrycks/test\n\"\"\"\nfrom lm_eval.base import MultipleChoiceTask\n\n\n_CITATION = \"\"\"\n@article{hendryckstest2021,\n title={Measuring Massive Multitask Language Understanding},\n author={Dan Hendrycks and Collin Burns and Steven Basart and Andy Zou and Mantas Mazeika and Dawn Song and Jacob Steinhardt},\n journal={Proceedings of the International Conference on Learning Representations (ICLR)},\n year={2021}\n}\n\"\"\"\n\n\nSUBJECTS = [\n \"abstract_algebra\",\n \"anatomy\",\n \"astronomy\",\n \"business_ethics\",\n \"clinical_knowledge\",\n \"college_biology\",\n \"college_chemistry\",\n \"college_computer_science\",\n \"college_mathematics\",\n \"college_medicine\",\n \"college_physics\",\n \"computer_security\",\n \"conceptual_physics\",\n \"econometrics\",\n \"electrical_engineering\",\n \"elementary_mathematics\",\n \"formal_logic\",\n \"global_facts\",\n \"high_school_biology\",\n \"high_school_chemistry\",\n \"high_school_computer_science\",\n \"high_school_european_history\",\n \"high_school_geography\",\n \"high_school_government_and_politics\",\n \"high_school_macroeconomics\",\n \"high_school_mathematics\",\n \"high_school_microeconomics\",\n \"high_school_physics\",\n \"high_school_psychology\",\n \"high_school_statistics\",\n \"high_school_us_history\",\n \"high_school_world_history\",\n \"human_aging\",\n \"human_sexuality\",\n \"international_law\",\n \"jurisprudence\",\n \"logical_fallacies\",\n \"machine_learning\",\n \"management\",\n \"marketing\",\n \"medical_genetics\",\n \"miscellaneous\",\n \"moral_disputes\",\n \"moral_scenarios\",\n \"nutrition\",\n \"philosophy\",\n \"prehistory\",\n \"professional_accounting\",\n \"professional_law\",\n \"professional_medicine\",\n \"professional_psychology\",\n \"public_relations\",\n \"security_studies\",\n \"sociology\",\n \"us_foreign_policy\",\n \"virology\",\n \"world_religions\",\n]\n\n\ndef create_all_tasks():\n \"\"\"Creates a dictionary of tasks from a list of subjects\n :return: {task_name: task}\n e.g. {hendrycksTest-abstract_algebra: Task, hendrycksTest-anatomy: Task}\n \"\"\"\n return {f\"hendrycksTest-{sub}\": create_task(sub) for sub in SUBJECTS}\n\n\ndef create_task(subject):\n class HendrycksTest(GeneralHendrycksTest):\n def __init__(self):\n super().__init__(subject)\n\n return HendrycksTest\n\n\nclass GeneralHendrycksTest(MultipleChoiceTask):\n VERSION = 0\n DATASET_PATH = \"hendrycks_test\"\n DATASET_NAME = None\n\n def __init__(self, subject):\n self.DATASET_NAME = subject\n super().__init__()\n\n def has_training_docs(self):\n return False\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return True\n\n def validation_docs(self):\n return map(self._process_doc, self.dataset[\"validation\"])\n\n def test_docs(self):\n return map(self._process_doc, self.dataset[\"test\"])\n\n def _process_doc(self, doc):\n def format_example(doc, keys):\n \"\"\"\n Question: \n Choices:\n A. \n B. \n C. \n D. \n Answer:\n \"\"\"\n prompt = \"Question: \" + doc[\"question\"] + \"\\nChoices:\\n\"\n prompt += \"\".join(\n [f\"{key}. {choice}\\n\" for key, choice in zip(keys, doc[\"choices\"])]\n )\n prompt += \"Answer:\"\n return prompt\n\n keys = [\"A\", \"B\", \"C\", \"D\"]\n return {\n \"query\": format_example(doc, keys),\n \"choices\": doc[\"choices\"],\n \"gold\": keys.index(doc[\"answer\"])\n if isinstance(doc[\"answer\"], str)\n else doc[\"answer\"],\n }\n\n def fewshot_examples(self, k, rnd):\n # fewshot_examples is not just sampling from train_docs because dev is\n # in the same distribution as val/test but auxiliary_train isn't\n\n if self._fewshot_docs is None:\n self._fewshot_docs = list(map(self._process_doc, self.dataset[\"dev\"]))\n\n return rnd.sample(list(self._fewshot_docs), k)\n\n def doc_to_text(self, doc):\n return doc[\"query\"]\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"query\"]\n" }, { "path": "lm_eval/tasks/lambada.py", "content": "\"\"\"\nThe LAMBADA dataset: Word prediction requiring a broad discourse context∗\nhttps://arxiv.org/pdf/1606.06031.pdf\n\nLAMBADA is a dataset to evaluate the capabilities of computational models for text\nunderstanding by means of a word prediction task. LAMBADA is a collection of narrative\npassages sharing the characteristic that human subjects are able to guess their last\nword if they are exposed to the whole passage, but not if they only see the last\nsentence preceding the target word. To succeed on LAMBADA, computational models\ncannot simply rely on local context, but must be able to keep track of information\nin the broader discourse.\n\nHomepage: https://zenodo.org/record/2630551#.X4Xzn5NKjUI\n\"\"\"\nfrom lm_eval.base import Task, rf\nfrom lm_eval.metrics import mean, perplexity\n\n\n_CITATION = \"\"\"\n@misc{\n author={Paperno, Denis and Kruszewski, Germán and Lazaridou, Angeliki and Pham, Quan Ngoc and Bernardi, Raffaella and Pezzelle, Sandro and Baroni, Marco and Boleda, Gemma and Fernández, Raquel},\n title={The LAMBADA dataset},\n DOI={10.5281/zenodo.2630551},\n publisher={Zenodo},\n year={2016},\n month={Aug}\n}\n\"\"\"\n\n\nclass LambadaBase(Task):\n VERSION = None\n\n def training_docs(self):\n if self.has_training_docs():\n return self.dataset[\"train\"]\n\n def validation_docs(self):\n if self.has_validation_docs():\n return self.dataset[\"validation\"]\n\n def test_docs(self):\n if self.has_test_docs():\n return self.dataset[\"test\"]\n\n def doc_to_text(self, doc):\n return doc[\"text\"].rsplit(\" \", 1)[0]\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"text\"]\n\n def doc_to_target(self, doc):\n return \" \" + doc[\"text\"].rsplit(\" \", 1)[1]\n\n def construct_requests(self, doc, ctx):\n ll, is_greedy = rf.loglikelihood(ctx, self.doc_to_target(doc))\n\n return ll, is_greedy\n\n def process_results(self, doc, results):\n ll, is_greedy = results\n\n return {\"ppl\": ll, \"acc\": int(is_greedy)}\n\n def aggregation(self):\n return {\"ppl\": perplexity, \"acc\": mean}\n\n def higher_is_better(self):\n return {\"ppl\": False, \"acc\": True}\n\n\nclass LambadaStandard(LambadaBase):\n \"\"\"The LAMBADA task using the standard original LAMBADA dataset.\"\"\"\n\n VERSION = 0\n DATASET_PATH = \"lambada\"\n\n def has_training_docs(self):\n return False\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return True\n\n\nclass LambadaOpenAI(LambadaBase):\n \"\"\"The LAMBADA task using the LAMBADA OpenAI dataset, a modified version of the\n original LAMBADA dataset created by OpenAI for evaluating their GPT-2 model.\n\n Reference: https://github.com/openai/gpt-2/issues/131#issuecomment-497136199\n \"\"\"\n\n VERSION = 0\n DATASET_PATH = \"EleutherAI/lambada_openai\"\n\n def has_training_docs(self):\n return False\n\n def has_validation_docs(self):\n return False\n\n def has_test_docs(self):\n return True\n" }, { "path": "lm_eval/tasks/lambada_cloze.py", "content": "\"\"\"\nThe LAMBADA dataset: Word prediction requiring a broad discourse context∗\nhttps://arxiv.org/pdf/1606.06031.pdf\n\nCloze-style LAMBADA dataset.\nLAMBADA is a dataset to evaluate the capabilities of computational models for text\nunderstanding by means of a word prediction task. LAMBADA is a collection of narrative\npassages sharing the characteristic that human subjects are able to guess their last\nword if they are exposed to the whole passage, but not if they only see the last\nsentence preceding the target word. To succeed on LAMBADA, computational models\ncannot simply rely on local context, but must be able to keep track of information\nin the broader discourse.\n\nHomepage: https://zenodo.org/record/2630551#.X4Xzn5NKjUI\n\"\"\"\nfrom lm_eval.tasks.lambada import LambadaOpenAI, LambadaStandard\n\n\n_CITATION = \"\"\"\n@misc{\n author={Paperno, Denis and Kruszewski, Germán and Lazaridou, Angeliki and Pham, Quan Ngoc and Bernardi, Raffaella and Pezzelle, Sandro and Baroni, Marco and Boleda, Gemma and Fernández, Raquel},\n title={The LAMBADA dataset},\n DOI={10.5281/zenodo.2630551},\n publisher={Zenodo},\n year={2016},\n month={Aug}\n}\n\"\"\"\n\n\nclass LambadaStandardCloze(LambadaStandard):\n \"\"\"Cloze-style LambadaStandard.\"\"\"\n\n VERSION = 0\n\n def doc_to_text(self, doc):\n return doc[\"text\"].rsplit(\" \", 1)[0] + \" ____. ->\"\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"text\"]\n\n def doc_to_target(self, doc):\n return \" \" + doc[\"text\"].rsplit(\" \", 1)[1]\n\n\nclass LambadaOpenAICloze(LambadaOpenAI):\n \"\"\"Cloze-style LambadaOpenAI.\"\"\"\n\n VERSION = 0\n\n def doc_to_text(self, doc):\n return doc[\"text\"].rsplit(\" \", 1)[0] + \" ____. ->\"\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"text\"]\n\n def doc_to_target(self, doc):\n return \" \" + doc[\"text\"].rsplit(\" \", 1)[1]\n" }, { "path": "lm_eval/tasks/lambada_multilingual.py", "content": "\"\"\"\nThe LAMBADA (OpenAI) dataset: Word prediction requiring a broad discourse context∗\nhttps://arxiv.org/pdf/1606.06031.pdf\n\nThe LAMBADA OpenAI dataset machine-translated to other languages.\nLAMBADA is a dataset to evaluate the capabilities of computational models for text\nunderstanding by means of a word prediction task. LAMBADA is a collection of narrative\npassages sharing the characteristic that human subjects are able to guess their last\nword if they are exposed to the whole passage, but not if they only see the last\nsentence preceding the target word. To succeed on LAMBADA, computational models\ncannot simply rely on local context, but must be able to keep track of information\nin the broader discourse.\n\nHomepage: https://zenodo.org/record/2630551#.X4Xzn5NKjUI\n\nReference (OpenAI): https://github.com/openai/gpt-2/issues/131#issuecomment-497136199\n\"\"\"\nfrom .lambada import LambadaOpenAI\n\n\n_CITATION = \"\"\"\n@misc{\n author={Paperno, Denis and Kruszewski, Germán and Lazaridou, Angeliki and Pham, Quan Ngoc and Bernardi, Raffaella and Pezzelle, Sandro and Baroni, Marco and Boleda, Gemma and Fernández, Raquel},\n title={The LAMBADA dataset},\n DOI={10.5281/zenodo.2630551},\n publisher={Zenodo},\n year={2016},\n month={Aug}\n}\n\"\"\"\n\n\nclass LambadaOpenAIMultilingualEnglish(LambadaOpenAI):\n VERSION = 0\n DATASET_NAME = \"en\"\n\n\nclass LambadaOpenAIMultilingualFrench(LambadaOpenAI):\n VERSION = 0\n DATASET_NAME = \"fr\"\n\n\nclass LambadaOpenAIMultilingualGerman(LambadaOpenAI):\n VERSION = 0\n DATASET_NAME = \"de\"\n\n\nclass LambadaOpenAIMultilingualItalian(LambadaOpenAI):\n VERSION = 0\n DATASET_NAME = \"it\"\n\n\nclass LambadaOpenAIMultilingualSpanish(LambadaOpenAI):\n VERSION = 0\n DATASET_NAME = \"es\"\n\n\nLANG_CLASSES = [\n LambadaOpenAIMultilingualEnglish,\n LambadaOpenAIMultilingualFrench,\n LambadaOpenAIMultilingualGerman,\n LambadaOpenAIMultilingualItalian,\n LambadaOpenAIMultilingualSpanish,\n]\n\n\ndef construct_tasks():\n tasks = {}\n for lang_class in LANG_CLASSES:\n tasks[f\"lambada_openai_mt_{lang_class.DATASET_NAME}\"] = lang_class\n return tasks\n" }, { "path": "lm_eval/tasks/logiqa.py", "content": "\"\"\"\nLogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning\nhttps://arxiv.org/pdf/2007.08124.pdf\n\nLogiQA is a dataset for testing human logical reasoning. It consists of 8,678 QA\ninstances, covering multiple types of deductive reasoning. Results show that state-\nof-the-art neural models perform by far worse than human ceiling. The dataset can\nalso serve as a benchmark for reinvestigating logical AI under the deep learning\nNLP setting.\n\nHomepage: https://github.com/lgw863/LogiQA-dataset\n\"\"\"\nimport inspect\nimport lm_eval.datasets.logiqa.logiqa\nfrom lm_eval.base import MultipleChoiceTask\n\n\n_CITATION = \"\"\"\n@misc{liu2020logiqa,\n title={LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning},\n author={Jian Liu and Leyang Cui and Hanmeng Liu and Dandan Huang and Yile Wang and Yue Zhang},\n year={2020},\n eprint={2007.08124},\n archivePrefix={arXiv},\n primaryClass={cs.CL}\n}\n\"\"\"\n\n\nclass LogiQA(MultipleChoiceTask):\n VERSION = 0\n DATASET_PATH = inspect.getfile(lm_eval.datasets.logiqa.logiqa)\n DATASET_NAME = None\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return True\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(map(self._process_doc, self.dataset[\"train\"]))\n return self._training_docs\n\n def validation_docs(self):\n return map(self._process_doc, self.dataset[\"validation\"])\n\n def test_docs(self):\n return map(self._process_doc, self.dataset[\"test\"])\n\n def _process_doc(self, doc):\n def format_example(doc, choices):\n \"\"\"\n Passage: \n Question: \n Choices:\n A. \n B. \n C. \n D. \n Answer:\n \"\"\"\n prompt = \"Passage: \" + doc[\"context\"] + \"\\n\"\n prompt += \"Question: \" + doc[\"question\"] + \"\\nChoices:\\n\"\n for choice, option in zip(choices, doc[\"options\"]):\n prompt += f\"{choice.upper()}. {option}\\n\"\n prompt += \"Answer:\"\n return prompt\n\n choices = [\"a\", \"b\", \"c\", \"d\"]\n return {\n \"passage\": doc[\"context\"], # Used for decontamination\n \"query\": format_example(doc, choices),\n \"choices\": doc[\"options\"],\n \"gold\": choices.index(doc[\"label\"]),\n }\n\n def doc_to_text(self, doc):\n return doc[\"query\"]\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"passage\"]\n" }, { "path": "lm_eval/tasks/mathqa.py", "content": "\"\"\"\nMathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms\nhttps://arxiv.org/pdf/1905.13319.pdf\n\nMathQA is a large-scale dataset of 37k English multiple-choice math word problems\ncovering multiple math domain categories by modeling operation programs corresponding\nto word problems in the AQuA dataset (Ling et al., 2017).\n\nHomepage: https://math-qa.github.io/math-QA/\n\"\"\"\nimport re\nfrom lm_eval.base import MultipleChoiceTask\n\n\n_CITATION = \"\"\"\n@misc{amini2019mathqa,\n title={MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalisms},\n author={Aida Amini and Saadia Gabriel and Peter Lin and Rik Koncel-Kedziorski and Yejin Choi and Hannaneh Hajishirzi},\n year={2019},\n eprint={1905.13319},\n archivePrefix={arXiv},\n primaryClass={cs.CL}\n}\n\"\"\"\n\n\nclass MathQA(MultipleChoiceTask):\n VERSION = 0\n DATASET_PATH = \"math_qa\"\n DATASET_NAME = None\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return True\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(map(self._process_doc, self.dataset[\"train\"]))\n return self._training_docs\n\n def validation_docs(self):\n return map(self._process_doc, self.dataset[\"validation\"])\n\n def test_docs(self):\n return map(self._process_doc, self.dataset[\"test\"])\n\n def _process_doc(self, doc):\n answer_idx = [\"a\", \"b\", \"c\", \"d\", \"e\"].index(doc[\"correct\"])\n choices = [\n c[4:].rstrip(\" ,\")\n for c in re.findall(r\"[abcd] \\) .*?, |e \\) .*?$\", doc[\"options\"])\n ]\n\n out_doc = {\n \"query\": \"Question: \" + doc[\"Problem\"] + \"\\nAnswer:\",\n \"choices\": choices,\n \"gold\": answer_idx,\n }\n return out_doc\n\n def doc_to_text(self, doc):\n return doc[\"query\"]\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"query\"]\n" }, { "path": "lm_eval/tasks/mc_taco.py", "content": "\"\"\"\n“Going on a vacation” takes longer than “Going for a walk”:\nA Study of Temporal Commonsense Understanding\nhttps://arxiv.org/pdf/1909.03065.pdf\n\nMC-TACO is a dataset of 13k question-answer pairs that require temporal commonsense\ncomprehension. The dataset contains five temporal properties, (1) duration (how long\nan event takes), (2) temporal ordering (typical order of events), (3) typical time\n(when an event occurs), (4) frequency (how often an event occurs), and (5) stationarity\n(whether a state is maintained for a very long time or indefinitely).\n\nWARNING: Running this task with a `--limit` arg will give misleading results! The\ncorresponding dataset is structured such that each multiple-choice-question gathered\nby the authors is split into question-option pairs, where each such pair gets\nsiloed into an individual document for plausibility testing. Because the harness\nshuffles these documents, setting `--limit` will likely \"cut off\" certain candidate\nanswers. This is a problem because the task's metrics require an exhaustive evaluation\nof a question's options. See section 4 of the paper for details.\n\nHomepage: https://leaderboard.allenai.org/mctaco/submissions/public\n\"\"\"\nimport numpy as np\nfrom collections import defaultdict\nfrom lm_eval.base import rf, Task\n\n\n_CITATION = \"\"\"\n@inproceedings{ZKNR19,\n author = {Ben Zhou, Daniel Khashabi, Qiang Ning and Dan Roth},\n title = {“Going on a vacation” takes longer than “Going for a walk”: A Study of Temporal Commonsense Understanding },\n booktitle = {EMNLP},\n year = {2019},\n}\n\"\"\"\n\n\nclass MCTACO(Task):\n VERSION = 0\n DATASET_PATH = \"mc_taco\"\n DATASET_NAME = None\n\n def has_training_docs(self):\n return False\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return True\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def test_docs(self):\n return self.dataset[\"test\"]\n\n def doc_to_text(self, doc):\n return (\n f\"{doc['sentence']}\\nQuestion: {doc['question']}\\n\"\n f\"Answer: {doc['answer']}\\nPlausible:\"\n )\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"question\"] + \" \" + doc[\"sentence\"]\n\n def doc_to_target(self, doc):\n return \" \" + [\"no\", \"yes\"][doc[\"label\"]]\n\n def construct_requests(self, doc, ctx):\n \"\"\"Uses RequestFactory to construct Requests and returns an iterable of\n Requests which will be sent to the LM.\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param ctx: str\n The context string, generated by fewshot_context. This includes the natural\n language description, as well as the few shot examples, and the question\n part of the document for `doc`.\n \"\"\"\n ll_no, _ = rf.loglikelihood(ctx, \" no\")\n ll_yes, _ = rf.loglikelihood(ctx, \" yes\")\n return ll_no, ll_yes\n\n def process_results(self, doc, results):\n \"\"\"Take a single document and the LM results and evaluates, returning a\n dict where keys are the names of submetrics and values are the values of\n the metric for that one document\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param results:\n The results of the requests created in construct_requests.\n \"\"\"\n ll_no, ll_yes = results\n gold = doc[\"label\"]\n pred = int(ll_yes > ll_no)\n question_id = self._question2id(doc)\n items = (gold, pred, question_id)\n return {\"em\": items, \"f1\": items}\n\n def _question2id(self, doc):\n \"\"\"Returns an identifier for the question in the given document.\"\"\"\n return \" \".join([doc[\"sentence\"], doc[\"question\"]])\n\n def aggregation(self):\n return {\n \"f1\": f1,\n \"em\": exact_match,\n }\n\n def higher_is_better(self):\n return {\n \"f1\": True,\n \"em\": True,\n }\n\n\ndef exact_match(items):\n \"\"\"\n Counts a question as correct if the model accurately classifies the plausibility\n of an answer for all candidate answers. See section 4 \"Evaluation Metrics\" in the paper.\n \"\"\"\n results = list(zip(*items))\n accuracies = defaultdict(list)\n for gold, pred, question in zip(results[0], results[1], results[2]):\n accuracies[question].append(pred == gold)\n return np.mean([int(all(accs)) for accs in accuracies.values()])\n\n\ndef f1(items):\n \"\"\"See section 4 \"Evaluation Metrics\" in the paper about the F1 metric used.\"\"\"\n results = list(zip(*items))\n # Group the positive (\"yes\" = 1) golds and predictions by question.\n gold_positives, pred_positives = defaultdict(list), defaultdict(list)\n for gold, pred, question in zip(results[0], results[1], results[2]):\n gold_positives[question].append(gold)\n pred_positives[question].append(pred)\n f1 = []\n for question in gold_positives.keys():\n gp, pp = sum(gold_positives[question]), sum(pred_positives[question])\n tp = sum(np.logical_and(gold_positives[question], pred_positives[question]))\n p = tp / pp if pp > 0.0 else 1.0\n r = tp / gp if gp > 0.0 else 1.0\n if p + r > 0.0:\n f1.append(2.0 * (p * r) / (p + r))\n return np.mean(f1)\n" }, { "path": "lm_eval/tasks/mutual.py", "content": "\"\"\"\nMuTual: A Dataset for Multi-Turn Dialogue Reasoning\nhttps://www.aclweb.org/anthology/2020.acl-main.130/\n\nMuTual is a retrieval-based dataset for multi-turn dialogue reasoning, which is\nmodified from Chinese high school English listening comprehension test data.\n\nHomepage: https://github.com/Nealcly/MuTual\n\"\"\"\nimport numpy as np\nimport inspect\nimport lm_eval.datasets.mutual.mutual\nfrom lm_eval.base import Task, rf\nfrom lm_eval.metrics import mean\n\n\n_CITATION = \"\"\"\n@inproceedings{mutual,\n title = \"MuTual: A Dataset for Multi-Turn Dialogue Reasoning\",\n author = \"Cui, Leyang and Wu, Yu and Liu, Shujie and Zhang, Yue and Zhou, Ming\" ,\n booktitle = \"Proceedings of the 58th Conference of the Association for Computational Linguistics\",\n year = \"2020\",\n publisher = \"Association for Computational Linguistics\",\n}\n\"\"\"\n\n\nclass MuTualBase(Task):\n VERSION = 1\n DATASET_PATH = inspect.getfile(lm_eval.datasets.mutual.mutual)\n DATASET_NAME = None\n CHOICES = [\"A\", \"B\", \"C\", \"D\"]\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n return self.dataset[\"train\"]\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def test_docs(self):\n return NotImplemented\n\n def doc_to_text(self, doc):\n return self.detokenize(doc[\"article\"])\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"article\"]\n\n def doc_to_target(self, doc):\n return \" \" + self.detokenize(doc[\"options\"][self.CHOICES.index(doc[\"answers\"])])\n\n def construct_requests(self, doc, ctx):\n lls = []\n for option in doc[\"options\"]:\n lls.append(rf.loglikelihood(ctx, f\" {self.detokenize(option)}\")[0])\n return lls\n\n def detokenize(self, text):\n text = text.replace(\" '\", \"'\")\n text = text.replace(\" \\n\", \"\\n\")\n text = text.replace(\"\\n \", \"\\n\")\n text = text.replace(\" n't\", \"n't\")\n text = text.replace(\"`` \", '\"')\n text = text.replace(\"''\", '\"')\n # punctuation\n text = text.replace(\" :\", \":\")\n text = text.replace(\" ;\", \";\")\n text = text.replace(\" !\", \"!\")\n text = text.replace(\" ?\", \"?\")\n text = text.replace(\" ,\", \",\")\n text = text.replace(\" .\", \".\")\n return text\n\n def process_results(self, doc, results):\n gold = self.CHOICES.index(doc[\"answers\"])\n r4_1 = np.argmax(results) == gold # r4_1 = accuracy\n ranks = sorted(results, reverse=True)\n r4_2 = (ranks.index(results[gold]) == 1) + r4_1\n mrr = 1.0 / (ranks.index(results[gold]) + 1) # `+ 1` for index offset\n return {\"r@1\": r4_1, \"r@2\": r4_2, \"mrr\": mrr}\n\n def aggregation(self):\n return {\"r@1\": mean, \"r@2\": mean, \"mrr\": mean}\n\n def higher_is_better(self):\n return {\"r@1\": True, \"r@2\": True, \"mrr\": True}\n\n\nclass MuTual(MuTualBase):\n DATASET_NAME = \"mutual\"\n\n\nclass MuTualPlus(MuTualBase):\n DATASET_NAME = \"mutual_plus\"\n" }, { "path": "lm_eval/tasks/naturalqs.py", "content": "\"\"\"\nNatural Questions: a Benchmark for Question Answering Research\nhttps://storage.googleapis.com/pub-tools-public-publication-data/pdf/1f7b46b5378d757553d3e92ead36bda2e4254244.pdf\n\nThe Natural Questions (NQ) corpus is a question-answering dataset that contains\nquestions from real users and requires QA systems to read and comprehend an entire\nWikipedia article that may or may not contain the answer to the question. The\ninclusion of real user questions, and the requirement that solutions should read\nan entire page to find the answer, cause NQ to be a more realistic and challenging\ntask than prior QA datasets.\n\nTODO: NaturalQS has a *really* large train set that huggingface just automatically\ndownloads even if you dont use it. we should try and only download the val set and\nnot even bother with the train set.\n\nHomepage: https://ai.google.com/research/NaturalQuestions\n\"\"\"\nfrom lm_eval.base import Task\nfrom itertools import islice\n\n\n_CITATION = \"\"\"\n@article{47761,\n title={Natural Questions: a Benchmark for Question Answering Research},\n author={Tom Kwiatkowski and Jennimaria Palomaki and Olivia Redfield and Michael Collins and Ankur Parikh and Chris Alberti and Danielle Epstein and Illia Polosukhin and Matthew Kelcey and Jacob Devlin and Kenton Lee and Kristina N. Toutanova and Llion Jones and Ming-Wei Chang and Andrew Dai and Jakob Uszkoreit and Quoc Le and Slav Petrov},\n year={2019},\n journal={Transactions of the Association of Computational Linguistics}\n}\n\"\"\"\n\n\nclass NaturalQs(Task):\n VERSION = 0\n DATASET_PATH = \"natural_questions\"\n DATASET_NAME = None\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n # Cache training for faster few-shot.\n # Data is too large to fit in memory.\n if self._training_docs is None:\n self._training_docs = list(self.dataset[\"train\"])\n return self._training_docs\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def fewshot_examples(self, k, rnd):\n # Data is too large to fit in memory. We just sample from the first bit.\n if self._training_docs is None:\n self._training_docs = list(islice(self.training_docs(), 0, 100000))\n\n return rnd.sample(self._training_docs, k)\n\n def doc_to_text(self, doc):\n return \"Q: \" + doc[\"question\"][\"text\"] + \"\\n\\n\" + \"A:\"\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"question\"][\"text\"]\n\n def doc_to_target(self, doc):\n # There's a short answer and a long answer. Based on the paper, I'm using the long answer.\n # short_answer = doc[\"annotations\"][\"short_answers\"][0][\"text\"]\n long_answer_start = doc[\"annotations\"][\"long_answer\"][0][\"start_token\"]\n long_answer_end = doc[\"annotations\"][\"long_answer\"][0][\"end_token\"]\n long_answer_span = doc[\"document\"][\"tokens\"][\"token\"][\n long_answer_start:long_answer_end\n ]\n long_answer_is_html = doc[\"document\"][\"tokens\"][\"is_html\"][\n long_answer_start:long_answer_end\n ]\n long_answer_chars = [\n tok\n for (tok, is_html) in zip(long_answer_span, long_answer_is_html)\n if not is_html\n ]\n long_answer = \" \".join(long_answer_chars)\n return long_answer # Replace with short_answer[0] for short answer\n\n def construct_requests(self, doc, ctx):\n \"\"\"Uses RequestFactory to construct Requests and returns an iterable of\n Requests which will be sent to the LM.\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param ctx: str\n The context string, generated by fewshot_context. This includes the natural\n language description, as well as the few shot examples, and the question\n part of the document for `doc`.\n \"\"\"\n # TODO: implement evaluation.\n raise NotImplementedError(\"Evaluation not implemented\")\n\n def process_results(self, doc, results):\n \"\"\"Take a single document and the LM results and evaluates, returning a\n dict where keys are the names of submetrics and values are the values of\n the metric for that one document\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param results:\n The results of the requests created in construct_requests.\n \"\"\"\n # TODO: implement evaluation.\n raise NotImplementedError(\"Evaluation not implemented\")\n\n def aggregation(self):\n \"\"\"\n :returns: {str: [float] -> float}\n A dictionary where keys are the names of submetrics and values are\n functions that aggregate a list of metrics\n \"\"\"\n # TODO: implement evaluation.\n raise NotImplementedError(\"Evaluation not implemented\")\n\n def higher_is_better(self):\n \"\"\"\n :returns: {str: bool}\n A dictionary where keys are the names of submetrics and values are\n whether a higher value of the submetric is better\n \"\"\"\n # TODO: implement evaluation.\n raise NotImplementedError(\"Evaluation not implemented\")\n" }, { "path": "lm_eval/tasks/openbookqa.py", "content": "\"\"\"\nCan a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering\nhttps://arxiv.org/pdf/1809.02789.pdf\n\nOpenBookQA is a question-answering dataset modeled after open book exams for\nassessing human understanding of a subject. It consists of 5,957 multiple-choice\nelementary-level science questions (4,957 train, 500 dev, 500 test), which probe\nthe understanding of a small “book” of 1,326 core science facts and the application\nof these facts to novel situations. For training, the dataset includes a mapping\nfrom each question to the core science fact it was designed to probe. Answering\nOpenBookQA questions requires additional broad common knowledge, not contained\nin the book. The questions, by design, are answered incorrectly by both a retrieval-\nbased algorithm and a word co-occurrence algorithm.\n\nHomepage: https://allenai.org/data/open-book-qa\n\"\"\"\nfrom lm_eval.base import MultipleChoiceTask\n\n\n_CITATION = \"\"\"\n@inproceedings{OpenBookQA2018,\n title={Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering},\n author={Todor Mihaylov and Peter Clark and Tushar Khot and Ashish Sabharwal},\n booktitle={EMNLP},\n year={2018}\n}\n\"\"\"\n\n\nclass OpenBookQA(MultipleChoiceTask):\n VERSION = 0\n DATASET_PATH = \"openbookqa\"\n DATASET_NAME = \"main\"\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return True\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(map(self._process_doc, self.dataset[\"train\"]))\n return self._training_docs\n\n def validation_docs(self):\n return map(self._process_doc, self.dataset[\"validation\"])\n\n def test_docs(self):\n return map(self._process_doc, self.dataset[\"test\"])\n\n def _process_doc(self, doc):\n out_doc = {\n \"id\": doc[\"id\"],\n \"query\": doc[\"question_stem\"],\n \"choices\": doc[\"choices\"][\"text\"],\n \"gold\": [\"A\", \"B\", \"C\", \"D\"].index(doc[\"answerKey\"].strip()),\n }\n return out_doc\n\n def doc_to_text(self, doc):\n return doc[\"query\"]\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"query\"]\n" }, { "path": "lm_eval/tasks/pile.py", "content": "\"\"\"\nThe Pile: An 800GB Dataset of Diverse Text for Language Modeling\nhttps://arxiv.org/pdf/2101.00027.pdf\n\nThe Pile is a 825 GiB diverse, open source language modelling data set that consists\nof 22 smaller, high-quality datasets combined together. To score well on Pile\nBPB (bits per byte), a model must be able to understand many disparate domains\nincluding books, github repositories, webpages, chat logs, and medical, physics,\nmath, computer science, and philosophy papers.\n\nHomepage: https://pile.eleuther.ai/\n\"\"\"\nimport inspect\nimport lm_eval.datasets.pile.pile\nfrom lm_eval.base import PerplexityTask\n\n\n_CITATION = \"\"\"\n@article{pile,\n title={The {P}ile: An 800GB Dataset of Diverse Text for Language Modeling},\n author={Gao, Leo and Biderman, Stella and Black, Sid and Golding, Laurence and Hoppe, Travis and Foster, Charles and Phang, Jason and He, Horace and Thite, Anish and Nabeshima, Noa and Presser, Shawn and Leahy, Connor},\n journal={arXiv preprint arXiv:2101.00027},\n year={2020}\n}\n\"\"\"\n\n\nclass PilePerplexityTask(PerplexityTask):\n VERSION = 1\n DATASET_PATH = inspect.getfile(lm_eval.datasets.pile.pile)\n DATASET_NAME = None\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return True\n\n def validation_docs(self):\n for doc in self.dataset[\"validation\"]:\n yield doc[\"text\"]\n\n def test_docs(self):\n for doc in self.dataset[\"test\"]:\n yield doc[\"text\"]\n\n\nclass PileArxiv(PilePerplexityTask):\n DATASET_NAME = \"pile_arxiv\"\n\n\nclass PileBooks3(PilePerplexityTask):\n DATASET_NAME = \"pile_books3\"\n\n\nclass PileBookCorpus2(PilePerplexityTask):\n DATASET_NAME = \"pile_bookcorpus2\"\n\n\nclass PileDmMathematics(PilePerplexityTask):\n DATASET_NAME = \"pile_dm-mathematics\"\n\n\nclass PileEnron(PilePerplexityTask):\n DATASET_NAME = \"pile_enron\"\n\n\nclass PileEuroparl(PilePerplexityTask):\n DATASET_NAME = \"pile_europarl\"\n\n\nclass PileFreeLaw(PilePerplexityTask):\n DATASET_NAME = \"pile_freelaw\"\n\n\nclass PileGithub(PilePerplexityTask):\n DATASET_NAME = \"pile_github\"\n\n\nclass PileGutenberg(PilePerplexityTask):\n DATASET_NAME = \"pile_gutenberg\"\n\n\nclass PileHackernews(PilePerplexityTask):\n DATASET_NAME = \"pile_hackernews\"\n\n\nclass PileNIHExporter(PilePerplexityTask):\n DATASET_NAME = \"pile_nih-exporter\"\n\n\nclass PileOpenSubtitles(PilePerplexityTask):\n DATASET_NAME = \"pile_opensubtitles\"\n\n\nclass PileOpenWebText2(PilePerplexityTask):\n DATASET_NAME = \"pile_openwebtext2\"\n\n\nclass PilePhilPapers(PilePerplexityTask):\n DATASET_NAME = \"pile_philpapers\"\n\n\nclass PilePileCc(PilePerplexityTask):\n DATASET_NAME = \"pile_pile-cc\"\n\n\nclass PilePubmedAbstracts(PilePerplexityTask):\n DATASET_NAME = \"pile_pubmed-abstracts\"\n\n\nclass PilePubmedCentral(PilePerplexityTask):\n DATASET_NAME = \"pile_pubmed-central\"\n\n\nclass PileStackExchange(PilePerplexityTask):\n DATASET_NAME = \"pile_stackexchange\"\n\n\nclass PileUspto(PilePerplexityTask):\n DATASET_NAME = \"pile_upsto\"\n\n\nclass PileUbuntuIrc(PilePerplexityTask):\n DATASET_NAME = \"pile_ubuntu-irc\"\n\n\nclass PileWikipedia(PilePerplexityTask):\n DATASET_NAME = \"pile_wikipedia\"\n\n\nclass PileYoutubeSubtitles(PilePerplexityTask):\n DATASET_NAME = \"pile_youtubesubtitles\"\n" }, { "path": "lm_eval/tasks/piqa.py", "content": "\"\"\"\nPIQA: Reasoning about Physical Commonsense in Natural Language\nhttps://arxiv.org/pdf/1911.11641.pdf\n\nPhysical Interaction: Question Answering (PIQA) is a physical commonsense\nreasoning and a corresponding benchmark dataset. PIQA was designed to investigate\nthe physical knowledge of existing models. To what extent are current approaches\nactually learning about the world?\n\nHomepage: https://yonatanbisk.com/piqa/\n\"\"\"\nfrom lm_eval.base import MultipleChoiceTask\n\n\n_CITATION = \"\"\"\n@inproceedings{Bisk2020,\n author = {Yonatan Bisk and Rowan Zellers and\n Ronan Le Bras and Jianfeng Gao\n and Yejin Choi},\n title = {PIQA: Reasoning about Physical Commonsense in\n Natural Language},\n booktitle = {Thirty-Fourth AAAI Conference on\n Artificial Intelligence},\n year = {2020},\n}\n\"\"\"\n\n\nclass PiQA(MultipleChoiceTask):\n VERSION = 0\n DATASET_PATH = \"piqa\"\n DATASET_NAME = None\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(map(self._process_doc, self.dataset[\"train\"]))\n return self._training_docs\n\n def validation_docs(self):\n return map(self._process_doc, self.dataset[\"validation\"])\n\n def _process_doc(self, doc):\n out_doc = {\n \"goal\": doc[\"goal\"],\n \"choices\": [doc[\"sol1\"], doc[\"sol2\"]],\n \"gold\": doc[\"label\"],\n }\n return out_doc\n\n def doc_to_text(self, doc):\n return \"Question: \" + doc[\"goal\"] + \"\\nAnswer:\"\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"goal\"]\n" }, { "path": "lm_eval/tasks/prost.py", "content": "\"\"\"\nPROST: Physical Reasoning about Objects Through Space and Time\nhttps://arxiv.org/pdf/2106.03634.pdf\n\nPROST, Physical Reasoning about Objects Through Space and Time, is a dataset\nconsisting of 18,736 multiple-choice questions made from 14 manually curated\ntemplates, covering 10 physical reasoning concepts. All questions are designed\nto probe both causal and masked language models in a zero-shot setting.\n\nNOTE: PROST is limited to the zero-shot setting to adhere to authors' intentions\nas discussed in section 7 of the paper: \"We hope that the community will use\nthis dataset in the intended way: in a zero-shot setting to probe models which\nhave been trained on data not specifically collected to succeed on PROST.\"\n\nHomepage: https://github.com/nala-cub/prost\n\"\"\"\nfrom lm_eval.base import MultipleChoiceTask\n\n\n_CITATION = \"\"\"\n@inproceedings{aroca-ouellette-etal-2021-prost,\n title = \"{PROST}: {P}hysical Reasoning about Objects through Space and Time\",\n author = \"Aroca-Ouellette, St{\\'e}phane and\n Paik, Cory and\n Roncone, Alessandro and\n Kann, Katharina\",\n booktitle = \"Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021\",\n month = aug,\n year = \"2021\",\n address = \"Online\",\n publisher = \"Association for Computational Linguistics\",\n url = \"https://aclanthology.org/2021.findings-acl.404\",\n pages = \"4597--4608\",\n}\n\"\"\"\n\n\nclass PROST(MultipleChoiceTask):\n VERSION = 0\n DATASET_PATH = \"corypaik/prost\"\n DATASET_NAME = None\n\n def has_training_docs(self):\n return False\n\n def has_validation_docs(self):\n return False\n\n def has_test_docs(self):\n return True\n\n def test_docs(self):\n return map(self._process_doc, self.dataset[\"test\"])\n\n def fewshot_context(\n self, doc, num_fewshot, provide_description=None, rnd=None, description=None\n ):\n assert (\n num_fewshot == 0\n ), \"PROST is designed to probe models in a zero-shot fashion only.\"\n return super().fewshot_context(\n doc=doc, num_fewshot=num_fewshot, rnd=rnd, description=description\n )\n\n def _process_doc(self, doc):\n out_doc = {\n \"query\": f\"{doc['context']}\\nQuestion: {doc['ex_question']}\\nAnswer:\",\n \"choices\": [doc[\"A\"], doc[\"B\"], doc[\"C\"], doc[\"D\"]],\n \"gold\": doc[\"label\"],\n }\n return out_doc\n\n def doc_to_text(self, doc):\n return doc[\"query\"]\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"query\"]\n" }, { "path": "lm_eval/tasks/pubmedqa.py", "content": "\"\"\"\nPubMedQA: A Dataset for Biomedical Research Question Answering\nhttps://arxiv.org/pdf/1909.06146.pdf\n\nPubMedQA is a novel biomedical question answering (QA) dataset collected from\nPubMed abstracts. The task of PubMedQA is to answer research questions with\nyes/no/maybe (e.g.: Do preoperative statins reduce atrial fibrillation after\ncoronary artery bypass grafting?) using the corresponding abstracts. PubMedQA\nhas 1k expert-annotated, 61.2k unlabeled and 211.3k artificially generated QA\ninstances. Each PubMedQA instance is composed of (1) a question which is either\nan existing research article title or derived from one, (2) a context which is\nthe corresponding abstract without its conclusion, (3) a long answer, which is\nthe conclusion of the abstract and, presumably, answers the research question,\nand (4) a yes/no/maybe answer which summarizes the conclusion.\n\nHomepage: https://pubmedqa.github.io/\n\"\"\"\nimport numpy as np\nfrom lm_eval.base import rf, Task\nfrom lm_eval.metrics import mean\n\n\n_CITATION = \"\"\"\n@inproceedings{jin2019pubmedqa,\n title={PubMedQA: A Dataset for Biomedical Research Question Answering},\n author={Jin, Qiao and Dhingra, Bhuwan and Liu, Zhengping and Cohen, William and Lu, Xinghua},\n booktitle={Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)},\n pages={2567--2577},\n year={2019}\n}\n\"\"\"\n\n\nclass Pubmed_QA(Task):\n VERSION = 0\n DATASET_PATH = \"pubmed_qa\"\n DATASET_NAME = \"pqa_labeled\"\n\n def has_training_docs(self):\n return False\n\n def has_validation_docs(self):\n return False\n\n def has_test_docs(self):\n return True\n\n def test_docs(self):\n if self.has_test_docs():\n # HF is labelled as train but its really just for testing\n return self.dataset[\"train\"]\n\n def doc_to_text(self, doc):\n ctxs = \"\\n\".join(doc[\"context\"][\"contexts\"])\n return \"Abstract: {}\\nQuestion: {}\\nAnswer:\".format(\n ctxs, doc[\"question\"], doc[\"final_decision\"]\n )\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"question\"] + \" \" + \"\\n\".join(doc[\"context\"][\"contexts\"])\n\n def doc_to_target(self, doc):\n return \" {}\".format(doc[\"final_decision\"])\n\n def construct_requests(self, doc, ctx):\n \"\"\"Uses RequestFactory to construct Requests and returns\n an iterable of Requests which will be sent to the LM.\n \"\"\"\n ll_yes, _ = rf.loglikelihood(ctx, \" yes\")\n ll_no, _ = rf.loglikelihood(ctx, \" no\")\n ll_maybe, _ = rf.loglikelihood(ctx, \" maybe\")\n return ll_yes, ll_no, ll_maybe\n\n def process_results(self, doc, results):\n gold = doc[\"final_decision\"]\n ll_yes, ll_no, ll_maybe = results\n pred = np.argmax(results)\n return {\n \"acc\": [\"yes\", \"no\", \"maybe\"][pred] == gold,\n }\n\n def aggregation(self):\n return {\"acc\": mean}\n\n def higher_is_better(self):\n return {\"acc\": True}\n" }, { "path": "lm_eval/tasks/qa4mre.py", "content": "\"\"\"\nQA4MRE 2011-2013: Overview of Question Answering for Machine Reading Evaluation\nhttps://www.cs.cmu.edu/~./hovy/papers/13CLEF-QA4MRE.pdf\n\nThe (English only) QA4MRE challenge which was run as a Lab at CLEF 2011-2013.\nThe main objective of this exercise is to develop a methodology for evaluating\nMachine Reading systems through Question Answering and Reading Comprehension\nTests. Systems should be able to extract knowledge from large volumes of text\nand use this knowledge to answer questions. Four different tasks have been\norganized during these years: Main Task, Processing Modality and Negation for\nMachine Reading, Machine Reading of Biomedical Texts about Alzheimer's disease,\nand Entrance Exam.\n\nHomepage: http://nlp.uned.es/clef-qa/repository/qa4mre.php\n\"\"\"\nfrom lm_eval.base import MultipleChoiceTask\n\n\n_CITATION = \"\"\"\n@inproceedings{Peas2013QA4MRE2O,\n title={QA4MRE 2011-2013: Overview of Question Answering for Machine Reading Evaluation},\n author={Anselmo Pe{\\~n}as and Eduard H. Hovy and Pamela Forner and {\\'A}lvaro Rodrigo and Richard F. E. Sutcliffe and Roser Morante},\n booktitle={CLEF},\n year={2013}\n}\n\"\"\" # noqa: W605\n\n\nclass QA4MRE(MultipleChoiceTask):\n VERSION = 0\n DATASET_PATH = \"qa4mre\"\n DATASET_NAME = None\n\n def has_training_docs(self):\n return False\n\n def has_validation_docs(self):\n return False\n\n def has_test_docs(self):\n return True\n\n def test_docs(self):\n # `qa4mre` only has train data so we use it for the test docs.\n return map(self._process_doc, self.dataset[\"train\"])\n\n def _process_doc(self, doc):\n choices = doc[\"answer_options\"][\"answer_str\"]\n out_doc = {\n \"source\": doc[\"document_str\"].strip().replace(\"'\", \"'\"),\n \"query\": doc[\"question_str\"],\n \"choices\": choices,\n \"gold\": int(doc[\"correct_answer_id\"]) - 1,\n }\n return out_doc\n\n def doc_to_text(self, doc):\n return \"{}\\nQuestion: {}\\nAnswer:\".format(doc[\"source\"], doc[\"query\"])\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"source\"] + \" \" + doc[\"query\"]\n\n\nclass QA4MRE_2011(QA4MRE):\n DATASET_NAME = \"2011.main.EN\"\n\n\nclass QA4MRE_2012(QA4MRE):\n DATASET_NAME = \"2012.main.EN\"\n\n\nclass QA4MRE_2013(QA4MRE):\n DATASET_NAME = \"2013.main.EN\"\n" }, { "path": "lm_eval/tasks/qasper.py", "content": "\"\"\"\nA Dataset of Information-Seeking Questions and Answers Anchored in Research Papers\nhttps://arxiv.org/abs/2105.03011\n\nQASPER is a dataset of 5,049 questions over 1,585 Natural Language Processing papers.\nEach question is written by an NLP practitioner who read only the title and abstract\nof the corresponding paper, and the question seeks information present in the full\ntext. The questions are then answered by a separate set of NLP practitioners who also\nprovide supporting evidence to answers.\n\nHomepage: https://allenai.org/data/qasper\n\"\"\"\nfrom collections import Counter\nimport re\nimport string\nfrom lm_eval.base import rf, Task\nfrom lm_eval.metrics import f1_score, mean\n\n\n_CITATION = \"\"\"\n@article{DBLP:journals/corr/abs-2105-03011,\n author = {Pradeep Dasigi and\n Kyle Lo and\n Iz Beltagy and\n Arman Cohan and\n Noah A. Smith and\n Matt Gardner},\n title = {A Dataset of Information-Seeking Questions and Answers Anchored in\n Research Papers},\n journal = {CoRR},\n volume = {abs/2105.03011},\n year = {2021},\n url = {https://arxiv.org/abs/2105.03011},\n eprinttype = {arXiv},\n eprint = {2105.03011},\n timestamp = {Fri, 14 May 2021 12:13:30 +0200},\n biburl = {https://dblp.org/rec/journals/corr/abs-2105-03011.bib},\n bibsource = {dblp computer science bibliography, https://dblp.org}\n}\n\"\"\"\n\n\ndef normalize_answer(s):\n \"\"\"\n Taken from the official evaluation script for v1.1 of the SQuAD dataset.\n Lower text and remove punctuation, articles and extra whitespace.\n \"\"\"\n\n def remove_articles(text):\n return re.sub(r\"\\b(a|an|the)\\b\", \" \", text)\n\n def white_space_fix(text):\n return \" \".join(text.split())\n\n def remove_punc(text):\n exclude = set(string.punctuation)\n return \"\".join(ch for ch in text if ch not in exclude)\n\n def lower(text):\n return text.lower()\n\n return white_space_fix(remove_articles(remove_punc(lower(s))))\n\n\ndef categorise_answer(answer_blob):\n if answer_blob[\"unanswerable\"]:\n answer = \"unanswerable\"\n answer_type = \"unanswerable\"\n return answer, answer_type\n elif answer_blob[\"yes_no\"]:\n answer = \"yes\"\n answer_type = \"bool\"\n return answer, answer_type\n elif answer_blob[\"free_form_answer\"]:\n answer = answer_blob[\"free_form_answer\"]\n answer_type = \"free form answer\"\n return answer, answer_type\n elif answer_blob[\"extractive_spans\"]:\n answer = answer_blob[\"extractive_spans\"]\n answer_type = \"extractive_spans\"\n return answer, answer_type\n elif answer_blob[\"yes_no\"] is False:\n answer = \"no\"\n answer_type = \"bool\"\n return answer, answer_type\n\n\ndef token_f1_score(prediction, ground_truth):\n \"\"\"\n Taken from the official evaluation script for v1.1 of the SQuAD dataset.\n \"\"\"\n prediction_tokens = normalize_answer(prediction).split()\n ground_truth_tokens = normalize_answer(ground_truth).split()\n common = Counter(prediction_tokens) & Counter(ground_truth_tokens)\n num_same = sum(common.values())\n if num_same == 0:\n return 0\n precision = 1.0 * num_same / len(prediction_tokens)\n recall = 1.0 * num_same / len(ground_truth_tokens)\n f1 = (2 * precision * recall) / (precision + recall)\n return f1\n\n\nclass QASPER(Task):\n VERSION = 0\n DATASET_PATH = \"qasper\"\n DATASET_NAME = None\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def doc_to_text(self, doc):\n return (\n \"TITLE: \"\n + doc[\"title\"]\n + \"\\n\"\n + \"ABSTRACT: \"\n + doc[\"abstract\"]\n + \"\\n\\n\"\n + \"Q: \"\n + doc[\"question\"]\n + \"\\n\\n\"\n + \"A:\"\n )\n\n def doc_to_target(self, doc):\n answer = doc[\"answer\"]\n if isinstance(answer, list):\n answer = \", \".join(answer)\n return \" \" + answer\n\n def training_docs(self):\n for doc in self.dataset[\"train\"]:\n yield from self._process_doc(doc)\n\n def validation_docs(self):\n for doc in self.dataset[\"validation\"]:\n yield from self._process_doc(doc)\n\n def _process_doc(self, doc):\n \"\"\"Given a `doc`, flatten it out so that each JSON blob\n contains exactly one question and one answer. Logic taken from\n the reference implementation available at\n https://github.com/allenai/qasper-led-baseline/blob/main/scripts/evaluator.py\n \"\"\"\n obs_list = []\n for question, answer_list in zip(doc[\"qas\"][\"question\"], doc[\"qas\"][\"answers\"]):\n for answer_blob in answer_list[\"answer\"]:\n answer, answer_type = categorise_answer(answer_blob)\n obs_list.append(\n {\n \"title\": doc[\"title\"],\n \"abstract\": doc[\"abstract\"],\n \"question\": question,\n \"answer\": answer,\n \"answer_type\": answer_type,\n }\n )\n return obs_list\n\n def process_results(self, doc, results):\n # TODO: Calculate a score for extractive spans once a request type for generating\n # extractive spans is available\n if not results:\n return {}\n elif len(results) == 1:\n [res] = results\n elif len(results) == 2:\n [ll_yes, ll_no] = results\n\n # TODO: Handle unanswerability first\n # unanswerable_gold = doc[\"answer_type\"] == \"unanswerable\"\n # unanswerable_pred = exp(logprob_unanswerable)\n # res_dict[\"f1_unanswerable\"] = (unanswerable_gold, unanswerable_pred)\n\n res_dict = {}\n # Handle yes/no questions\n if doc[\"answer_type\"] == \"bool\":\n gold = 1 if doc[\"answer\"] == \"yes\" else 0\n pred = ll_yes > ll_no\n res_dict[\"f1_yesno\"] = (gold, pred)\n\n # Handle completions\n if doc[\"answer_type\"] == \"free form answer\":\n res_dict[\"f1_abstractive\"] = token_f1_score(res, doc[\"answer\"])\n\n # TODO: Handle extraction\n # if doc[\"answer_type\"] == \"extractive_spans\":\n # res_dict[\"f1_extractive\"] = 0\n return res_dict\n\n def aggregation(self):\n return {\n \"f1_yesno\": f1_score,\n \"f1_abstractive\": mean,\n }\n\n def construct_requests(self, doc, ctx):\n \"\"\"Uses RequestFactory to construct Requests and returns an iterable of\n Requests which will be sent to the LM.\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param ctx: str\n The context string, generated by fewshot_context. This includes the natural\n language description, as well as the few shot examples, and the question\n part of the document for `doc`.\n \"\"\"\n # unanswerable = rf.loglikelihood(ctx, \" \" + \"unanswerable\")\n if doc[\"answer_type\"] in (\"free form answer\"):\n return [rf.greedy_until(ctx, [\"\\n\"])]\n elif doc[\"answer_type\"] in (\"bool\"):\n ll_yes, _ = rf.loglikelihood(ctx, \" yes\")\n ll_no, _ = rf.loglikelihood(ctx, \" no\")\n return [ll_yes, ll_no]\n else:\n return []\n\n def higher_is_better(self):\n \"\"\"\n :returns: {str: bool}\n A dictionary where keys are the names of submetrics and values are\n whether a higher value of the submetric is better\n \"\"\"\n return {\n \"f1_yesno\": True,\n \"f1_abstractive\": True,\n }\n" }, { "path": "lm_eval/tasks/quac.py", "content": "\"\"\"\nQuAC: Question Answering in Context\nhttps://arxiv.org/abs/1808.07036\n\nQuestion Answering in Context (QuAC) is a dataset for modeling, understanding, and\nparticipating in information seeking dialog. Data instances consist of an interactive\ndialog between two crowd workers: (1) a student who poses a sequence of freeform\nquestions to learn as much as possible about a hidden Wikipedia text, and (2)\na teacher who answers the questions by providing short excerpts (spans) from the text.\n\nHomepage: https://quac.ai/\n\"\"\"\nimport inspect\nimport lm_eval.datasets.quac.quac\nfrom lm_eval.base import Task\n\n\n_CITATION = \"\"\"\n@article{choi2018quac,\n title={Quac: Question answering in context},\n author={Choi, Eunsol and He, He and Iyyer, Mohit and Yatskar, Mark and Yih, Wen-tau and Choi, Yejin and Liang, Percy and Zettlemoyer, Luke},\n journal={arXiv preprint arXiv:1808.07036},\n year={2018}\n}\n\"\"\"\n\n\nclass QuAC(Task):\n VERSION = 0\n DATASET_PATH = inspect.getfile(lm_eval.datasets.quac.quac)\n DATASET_NAME = None\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(map(self._process_doc, self.dataset[\"train\"]))\n return self._training_docs\n\n def validation_docs(self):\n return map(self._process_doc, self.dataset[\"validation\"])\n\n def test_docs(self):\n raise NotImplementedError(\"QuAC has no test docs.\")\n\n def _process_doc(self, doc):\n doc[\"title\"] = doc[\"title\"] + \" - \" + doc[\"section_title\"]\n return doc\n\n def doc_to_text(self, doc):\n return (\n \"TITLE: \"\n + doc[\"title\"]\n + \"\\n\"\n + \"PARAGRAPH: \"\n + doc[\"paragraph\"]\n + \"\\n\\n\"\n + \"Q: \"\n + doc[\"question\"]\n + \"\\n\\n\"\n + \"A: \"\n )\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"paragraph\"]\n\n def doc_to_target(self, doc):\n return doc[\"answer\"]\n\n def construct_requests(self, doc, ctx):\n \"\"\"Uses RequestFactory to construct Requests and returns an iterable of\n Requests which will be sent to the LM.\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param ctx: str\n The context string, generated by fewshot_context. This includes the natural\n language description, as well as the few shot examples, and the question\n part of the document for `doc`.\n \"\"\"\n # TODO: implement evaluation.\n raise NotImplementedError(\"Evaluation not implemented\")\n\n def process_results(self, doc, results):\n \"\"\"Take a single document and the LM results and evaluates, returning a\n dict where keys are the names of submetrics and values are the values of\n the metric for that one document\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param results:\n The results of the requests created in construct_requests.\n \"\"\"\n # TODO: implement evaluation.\n raise NotImplementedError(\"Evaluation not implemented\")\n\n def aggregation(self):\n \"\"\"\n :returns: {str: [float] -> float}\n A dictionary where keys are the names of submetrics and values are\n functions that aggregate a list of metrics\n \"\"\"\n # TODO: implement evaluation.\n raise NotImplementedError(\"Evaluation not implemented\")\n\n def higher_is_better(self):\n \"\"\"\n :returns: {str: bool}\n A dictionary where keys are the names of submetrics and values are\n whether a higher value of the submetric is better\n \"\"\"\n # TODO: implement evaluation.\n raise NotImplementedError(\"Evaluation not implemented\")\n" }, { "path": "lm_eval/tasks/race.py", "content": "\"\"\"\nRACE: Large-scale ReAding Comprehension Dataset From Examinations\nhttps://arxiv.org/pdf/1704.04683.pdf\n\nRACE is a large-scale reading comprehension dataset with more than 28,000 passages\nand nearly 100,000 questions. The dataset is collected from English examinations\nin China, which are designed for middle school and high school students. The dataset\ncan be served as the training and test sets for machine comprehension.\n\nHomepage: https://www.cs.cmu.edu/~glai1/data/race/\n\"\"\"\nimport collections\nimport datasets\nimport numpy as np\nfrom lm_eval.base import rf, Task\nfrom lm_eval.metrics import mean\n\n\n_CITATION = \"\"\"\n@article{lai2017large,\n title={RACE: Large-scale ReAding Comprehension Dataset From Examinations},\n author={Lai, Guokun and Xie, Qizhe and Liu, Hanxiao and Yang, Yiming and Hovy, Eduard},\n journal={arXiv preprint arXiv:1704.04683},\n year={2017}\n}\n\"\"\"\n\n\nclass each:\n def __init__(self, f):\n self.f = f\n\n def __rrshift__(self, other):\n return list(map(self.f, other))\n\n\nclass RACE(Task):\n VERSION = 1\n DATASET_PATH = \"race\"\n DATASET_NAME = \"high\"\n\n cache = {}\n letter_to_num = {\"A\": 0, \"B\": 1, \"C\": 2, \"D\": 3}\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return True\n\n def _collate_data(self, set):\n if set in self.cache:\n return self.cache[set]\n # One big issue with HF's implementation of this dataset: it makes a\n # separate document for each question; meanwhile, in the GPT3 paper it\n # is shown that one document is made per passage.\n\n r = collections.defaultdict(list)\n for item in datasets.load_dataset(\n path=self.DATASET_PATH, name=self.DATASET_NAME\n )[set]:\n r[item[\"article\"]].append(item)\n\n res = list(\n r.values()\n >> each(\n lambda x: {\n \"article\": x[0][\"article\"],\n \"problems\": x\n >> each(\n lambda y: {\n \"question\": y[\"question\"],\n \"answer\": y[\"answer\"],\n \"options\": y[\"options\"],\n }\n ),\n }\n )\n )\n\n self.cache[set] = res\n return res\n\n def training_docs(self):\n return self._collate_data(\"train\")\n\n def validation_docs(self):\n return self._collate_data(\"validation\")\n\n def test_docs(self):\n return self._collate_data(\"test\")\n\n @classmethod\n def get_answer_option(cls, problem):\n answer = cls.letter_to_num[problem[\"answer\"]]\n return problem[\"options\"][answer]\n\n @classmethod\n def last_problem(cls, doc):\n return doc[\"problems\"][-1]\n\n def doc_to_text(self, doc):\n text = \"Article: \" + doc[\"article\"] + \"\\n\\n\"\n for problem in doc[\"problems\"][:-1]:\n if problem[\"question\"][-6:] == \" _ .\":\n text += (\n problem[\"question\"][-5:] + self.get_answer_option(problem) + \"\\n\"\n )\n else:\n question = \"Question: \" + problem[\"question\"] + \"\\n\"\n answer = \"Answer: \" + self.get_answer_option(problem) + \"\\n\"\n text += question + answer\n text += self.last_problem(doc)[\"question\"]\n return text\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"article\"]\n\n def doc_to_target(self, doc):\n return \" \" + self.get_answer_option(self.last_problem(doc))\n\n def construct_requests(self, doc, ctx):\n \"\"\"Uses RequestFactory to construct Requests and returns an iterable of\n Requests which will be sent to the LM.\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param ctx: str\n The context string, generated by fewshot_context. This includes the natural\n language description, as well as the few shot examples, and the question\n part of the document for `doc`.\n \"\"\"\n problem = self.last_problem(doc)\n ll_choices = [\n rf.loglikelihood(ctx, \" \" + problem[\"options\"][i])[0] for i in range(4)\n ]\n return ll_choices\n\n def process_results(self, doc, results):\n \"\"\"Take a single document and the LM results and evaluates, returning a\n dict where keys are the names of submetrics and values are the values of\n the metric for that one document\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param results:\n The results of the requests created in construct_requests.\n \"\"\"\n gold = self.letter_to_num[self.last_problem(doc)[\"answer\"]]\n pred = np.argmax(results)\n return {\"acc\": int(pred == gold)}\n\n def aggregation(self):\n \"\"\"\n :returns: {str: [float] -> float}\n A dictionary where keys are the names of submetrics and values are\n functions that aggregate a list of metrics\n \"\"\"\n return {\"acc\": mean}\n\n def higher_is_better(self):\n \"\"\"\n :returns: {str: bool}\n A dictionary where keys are the names of submetrics and values are\n whether a higher value of the submetric is better\n \"\"\"\n return {\"acc\": True}\n" }, { "path": "lm_eval/tasks/sat.py", "content": "\"\"\"\nSimilarity of Semantic Relations\nhttps://arxiv.org/pdf/cs/0608100.pdf\n\nSAT (Scholastic Aptitude Test) Analogy Questions is a dataset comprising 374\nmultiple-choice analogy questions; 5 choices per question.\n\nHomepage: https://aclweb.org/aclwiki/SAT_Analogy_Questions_(State_of_the_art)\n\"\"\"\nimport inspect\nimport lm_eval.datasets.sat_analogies.sat_analogies\nfrom lm_eval.base import MultipleChoiceTask\n\n\n_CITATION = \"\"\"\n@article{article,\n author = {Turney, Peter},\n year = {2006},\n month = {09},\n pages = {379-416},\n title = {Similarity of Semantic Relations},\n volume = {32},\n journal = {Computational Linguistics},\n doi = {10.1162/coli.2006.32.3.379}\n}\n\"\"\"\n\n\nclass SATAnalogies(MultipleChoiceTask):\n VERSION = 0\n DATASET_PATH = inspect.getfile(lm_eval.datasets.sat_analogies.sat_analogies)\n DATASET_NAME = None\n\n def __init__(self, data_dir: str):\n \"\"\"\n SAT Analog Questions is not publicly available. You must request the data\n by emailing Peter Turney and then download it to a local directory path\n which should be passed into the `data_dir` arg.\n \"\"\"\n super().__init__(data_dir=data_dir)\n\n def has_training_docs(self):\n return False\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n return []\n\n def validation_docs(self):\n return map(self._process_doc, self.dataset[\"validation\"])\n\n def test_docs(self):\n return []\n\n def _process_doc(self, doc):\n return {\n \"source\": doc[\"source\"],\n \"query\": doc[\"stem\"].split(\" \")[:2],\n \"choices\": [\n \"{} is to {}\".format(*c.split(\" \")[:2]) for c in doc[\"choices\"]\n ],\n \"gold\": [\"a\", \"b\", \"c\", \"d\", \"e\"].index(doc[\"solution\"].strip()),\n }\n\n def doc_to_text(self, doc):\n return \"{} is to {} as\".format(*doc[\"query\"])\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"source\"] + \"\\n\" + \" \".join(doc[\"query\"])\n" }, { "path": "lm_eval/tasks/sciq.py", "content": "\"\"\"\nCrowdsourcing Multiple Choice Science Questions\nhttps://aclanthology.org/W17-4413.pdf\n\nThe SciQ dataset contains 13,679 crowdsourced science exam questions about Physics,\nChemistry and Biology, among others. The questions are in multiple-choice format\nwith 4 answer options each. For the majority of the questions, an additional paragraph\nwith supporting evidence for the correct answer is provided.\n\nHomepage: https://allenai.org/data/sciq\n\"\"\"\nfrom lm_eval.base import MultipleChoiceTask\n\n\n_CITATION = \"\"\"\n@inproceedings{Welbl2017CrowdsourcingMC,\n title={Crowdsourcing Multiple Choice Science Questions},\n author={Johannes Welbl and Nelson F. Liu and Matt Gardner},\n booktitle={NUT@EMNLP},\n year={2017}\n}\n\"\"\"\n\n\nclass SciQ(MultipleChoiceTask):\n VERSION = 0\n DATASET_PATH = \"sciq\"\n DATASET_NAME = None\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return True\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(map(self._process_doc, self.dataset[\"train\"]))\n return self._training_docs\n\n def validation_docs(self):\n return map(self._process_doc, self.dataset[\"validation\"])\n\n def test_docs(self):\n return map(self._process_doc, self.dataset[\"test\"])\n\n def _process_doc(self, doc):\n choices = [\n doc[\"distractor1\"],\n doc[\"distractor2\"],\n doc[\"distractor3\"],\n doc[\"correct_answer\"],\n ]\n src = doc[\"support\"]\n out_doc = {\n \"source\": src,\n \"query\": doc[\"question\"],\n \"choices\": choices,\n \"gold\": 3,\n }\n return out_doc\n\n def doc_to_text(self, doc):\n return \"{}\\nQuestion: {}\\nAnswer:\".format(doc[\"source\"], doc[\"query\"]).strip()\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"source\"] + \" \" + doc[\"query\"]\n" }, { "path": "lm_eval/tasks/squad.py", "content": "\"\"\"\nKnow What You Don’t Know: Unanswerable Questions for SQuAD\nhttps://arxiv.org/pdf/1806.03822.pdf\n\nStanford Question Answering Dataset (SQuAD) is a reading comprehension dataset,\nconsisting of questions posed by crowdworkers on a set of Wikipedia articles,\nwhere the answer to every question is a segment of text, or span, from the\ncorresponding reading passage, or the question might be unanswerable.\nSQuAD2.0 combines the 100,000 questions in SQuAD1.1 with over 50,000 unanswerable\nquestions written adversarially by crowdworkers to look similar to answerable ones.\nTo do well on SQuAD2.0, systems must not only answer questions when possible, but\nalso determine when no answer is supported by the paragraph and abstain from answering.\n\nHomepage: https://rajpurkar.github.io/SQuAD-explorer/\n\"\"\"\nimport datasets\nfrom math import exp\nfrom lm_eval.base import rf, Task\nfrom functools import partial\nfrom packaging import version\n\n\n_CITATION = \"\"\"\n@misc{rajpurkar2018know,\n title={Know What You Don't Know: Unanswerable Questions for SQuAD},\n author={Pranav Rajpurkar and Robin Jia and Percy Liang},\n year={2018},\n eprint={1806.03822},\n archivePrefix={arXiv},\n primaryClass={cs.CL}\n}\n\"\"\"\n\n\ndef _squad_metric(predictions, references):\n squad_metric = datasets.load_metric(\"squad_v2\")\n return squad_metric.compute(predictions=predictions, references=references)\n\n\ndef _squad_agg(key, items):\n predictions, references = zip(*items)\n\n return _squad_metric(predictions=predictions, references=references).get(key, 0)\n\n\nclass SQuAD2(Task):\n VERSION = 1\n DATASET_PATH = \"squad_v2\"\n DATASET_NAME = None\n\n # HF changed squad on us so we have to make sure we aren't running the old one\n assert version.parse(datasets.__version__) >= version.parse(\n \"1.11.0\"\n ), \"datasets v1.11.0 or later required for SQuAD\"\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n return self.dataset[\"train\"]\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def doc_to_text(self, doc):\n return (\n \"Title: \"\n + doc[\"title\"]\n + \"\\n\\n\"\n + \"Background: \"\n + doc[\"context\"]\n + \"\\n\\n\"\n + \"Question: \"\n + doc[\"question\"]\n + \"\\n\\n\"\n + \"Answer:\"\n )\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"context\"]\n\n def doc_to_target(self, doc):\n answer_list = doc[\"answers\"][\"text\"]\n if len(answer_list) > 0:\n answer = answer_list[0]\n else:\n answer = \"unanswerable\"\n return \" \" + answer\n\n def construct_requests(self, doc, ctx):\n \"\"\"Uses RequestFactory to construct Requests and returns an iterable of\n Requests which will be sent to the LM.\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param ctx: str\n The context string, generated by fewshot_context. This includes the natural\n language description, as well as the few shot examples, and the question\n part of the document for `doc`.\n \"\"\"\n continuation = rf.greedy_until(ctx, [\"\\n\"])\n is_unanswerable = rf.loglikelihood(ctx, \" \" + \"unanswerable\")\n return continuation, is_unanswerable\n\n def process_results(self, doc, results):\n \"\"\"Take a single document and the LM results and evaluates, returning a\n dict where keys are the names of submetrics and values are the values of\n the metric for that one document\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param results:\n The results of the requests created in construct_requests.\n \"\"\"\n continuation, (logprob_unanswerable, _) = results\n\n no_answer_probability = exp(logprob_unanswerable)\n\n predictions = {\n \"id\": doc[\"id\"],\n \"prediction_text\": continuation,\n \"no_answer_probability\": no_answer_probability,\n }\n\n references = {\n \"id\": doc[\"id\"],\n \"answers\": doc[\"answers\"],\n }\n\n return {\n \"exact\": (\n predictions,\n references,\n ), # Exact match (the normalized answer exactly match the gold answer)\n \"f1\": (\n predictions,\n references,\n ), # The F-score of predicted tokens versus the gold answer\n \"HasAns_exact\": (\n predictions,\n references,\n ), # Exact match (the normalized answer exactly match the gold answer)\n \"HasAns_f1\": (\n predictions,\n references,\n ), # The F-score of predicted tokens versus the gold answer\n \"NoAns_exact\": (\n predictions,\n references,\n ), # Exact match (the normalized answer exactly match the gold answer)\n \"NoAns_f1\": (\n predictions,\n references,\n ), # The F-score of predicted tokens versus the gold answer\n \"best_exact\": (\n predictions,\n references,\n ), # Best exact match (with varying threshold)\n \"best_f1\": (predictions, references), # Best F1 (with varying threshold)\n }\n\n def aggregation(self):\n \"\"\"\n :returns: {str: [float] -> float}\n A dictionary where keys are the names of submetrics and values are\n functions that aggregate a list of metrics\n \"\"\"\n return {\n \"exact\": partial(\n _squad_agg, \"exact\"\n ), # Exact match (the normalized answer exactly match the gold answer)\n \"f1\": partial(\n _squad_agg, \"f1\"\n ), # The F-score of predicted tokens versus the gold answer\n \"HasAns_exact\": partial(\n _squad_agg, \"HasAns_exact\"\n ), # Exact match (the normalized answer exactly match the gold answer)\n \"HasAns_f1\": partial(\n _squad_agg, \"HasAns_f1\"\n ), # The F-score of predicted tokens versus the gold answer\n \"NoAns_exact\": partial(\n _squad_agg, \"NoAns_exact\"\n ), # Exact match (the normalized answer exactly match the gold answer)\n \"NoAns_f1\": partial(\n _squad_agg, \"NoAns_f1\"\n ), # The F-score of predicted tokens versus the gold answer\n \"best_exact\": partial(\n _squad_agg, \"best_exact\"\n ), # Best exact match (with varying threshold)\n \"best_f1\": partial(\n _squad_agg, \"best_f1\"\n ), # Best F1 (with varying threshold)\n }\n\n def higher_is_better(self):\n \"\"\"\n :returns: {str: bool}\n A dictionary where keys are the names of submetrics and values are\n whether a higher value of the submetric is better\n \"\"\"\n return {\n \"exact\": True, # Exact match (the normalized answer exactly match the gold answer)\n \"f1\": True, # The F-score of predicted tokens versus the gold answer\n \"HasAns_exact\": True, # Exact match (the normalized answer exactly match the gold answer)\n \"HasAns_f1\": True, # The F-score of predicted tokens versus the gold answer\n \"NoAns_exact\": True, # Exact match (the normalized answer exactly match the gold answer)\n \"NoAns_f1\": True, # The F-score of predicted tokens versus the gold answer\n \"best_exact\": True, # Best exact match (with varying threshold)\n \"best_f1\": True, # Best F1 (with varying threshold)\n }\n" }, { "path": "lm_eval/tasks/storycloze.py", "content": "\"\"\"\nA Corpus and Cloze Evaluation for Deeper Understanding of Commonsense Stories\nhttps://arxiv.org/pdf/1604.01696.pdf\n\n'Story Cloze Test' (2018) is a commonsense reasoning framework for evaluating story\nunderstanding, story generation, and script learning. This test requires a system\nto choose the correct ending to a four-sentence story.\n\nHomepage: https://cs.rochester.edu/nlp/rocstories/\n\"\"\"\nimport numpy as np\nfrom lm_eval.base import rf, Task\nfrom lm_eval.metrics import mean\n\n\n_CITATION = \"\"\"\n@inproceedings{sharma-etal-2018-tackling,\n title = \"Tackling the Story Ending Biases in The Story Cloze Test\",\n author = \"Sharma, Rishi and\n Allen, James and\n Bakhshandeh, Omid and\n Mostafazadeh, Nasrin\",\n booktitle = \"Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)\",\n month = jul,\n year = \"2018\",\n address = \"Melbourne, Australia\",\n publisher = \"Association for Computational Linguistics\",\n url = \"https://aclanthology.org/P18-2119\",\n doi = \"10.18653/v1/P18-2119\",\n pages = \"752--757\",\n abstract = \"The Story Cloze Test (SCT) is a recent framework for evaluating story comprehension and script learning. There have been a variety of models tackling the SCT so far. Although the original goal behind the SCT was to require systems to perform deep language understanding and commonsense reasoning for successful narrative understanding, some recent models could perform significantly better than the initial baselines by leveraging human-authorship biases discovered in the SCT dataset. In order to shed some light on this issue, we have performed various data analysis and analyzed a variety of top performing models presented for this task. Given the statistics we have aggregated, we have designed a new crowdsourcing scheme that creates a new SCT dataset, which overcomes some of the biases. We benchmark a few models on the new dataset and show that the top-performing model on the original SCT dataset fails to keep up its performance. Our findings further signify the importance of benchmarking NLP systems on various evolving test sets.\",\n}\n\"\"\"\n\n\nclass StoryCloze(Task):\n VERSION = 0\n DATASET_PATH = \"story_cloze\"\n DATASET_NAME = None\n\n def __init__(self, data_dir: str):\n \"\"\"\n StoryCloze is not publicly available. You must download the data by\n following https://cs.rochester.edu/nlp/rocstories/ and pass the folder\n path into the `data_dir` arg.\n \"\"\"\n super().__init__(data_dir=data_dir)\n\n def has_training_docs(self):\n return False\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return True\n\n def training_docs(self):\n pass\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def test_docs(self):\n return self.dataset[\"test\"]\n\n def doc_to_text(self, doc):\n return \" \".join(\n [\n doc[\"input_sentence_1\"],\n doc[\"input_sentence_2\"],\n doc[\"input_sentence_3\"],\n doc[\"input_sentence_4\"],\n ]\n )\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return \" \".join(\n [\n doc[\"input_sentence_1\"],\n doc[\"input_sentence_2\"],\n doc[\"input_sentence_3\"],\n doc[\"input_sentence_4\"],\n ]\n )\n\n def doc_to_target(self, doc):\n clozes = [doc[\"sentence_quiz1\"], doc[\"sentence_quiz2\"]]\n # `- 1` because the `answer_right_ending` index is 1-based.\n return \" \" + clozes[doc[\"answer_right_ending\"] - 1]\n\n def construct_requests(self, doc, ctx):\n \"\"\"Uses RequestFactory to construct Requests and returns an iterable of\n Requests which will be sent to the LM.\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param ctx: str\n The context string, generated by fewshot_context. This includes the natural\n language description, as well as the few shot examples, and the question\n part of the document for `doc`.\n \"\"\"\n clozes = [doc[\"sentence_quiz1\"], doc[\"sentence_quiz2\"]]\n lls = [rf.loglikelihood(ctx, \" {}\".format(choice))[0] for choice in clozes]\n return lls\n\n def process_results(self, doc, results):\n \"\"\"Take a single document and the LM results and evaluates, returning a\n dict where keys are the names of submetrics and values are the values of\n the metric for that one document\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param results:\n The results of the requests created in construct_requests.\n \"\"\"\n gold = doc[\"answer_right_ending\"] - 1\n acc = 1.0 if np.argmax(results) == gold else 0.0\n return {\"acc\": acc}\n\n def aggregation(self):\n \"\"\"\n :returns: {str: [float] -> float}\n A dictionary where keys are the names of submetrics and values are\n functions that aggregate a list of metrics\n \"\"\"\n return {\"acc\": mean}\n\n def higher_is_better(self):\n \"\"\"\n :returns: {str: bool}\n A dictionary where keys are the names of submetrics and values are\n whether a higher value of the submetric is better\n \"\"\"\n return {\"acc\": True}\n\n\nclass StoryCloze2016(StoryCloze):\n DATASET_NAME = \"2016\"\n\n\nclass StoryCloze2018(StoryCloze):\n DATASET_NAME = \"2018\"\n" }, { "path": "lm_eval/tasks/superglue.py", "content": "\"\"\"\nSuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems\nhttps://w4ngatang.github.io/static/papers/superglue.pdf\n\nSuperGLUE is a benchmark styled after GLUE with a new set of more difficult language\nunderstanding tasks.\n\nHomepage: https://super.gluebenchmark.com/\n\nTODO: WSC requires free-form generation.\n\"\"\"\nimport numpy as np\nimport sklearn\nimport transformers.data.metrics.squad_metrics as squad_metrics\nfrom lm_eval.base import rf, Task\nfrom lm_eval.metrics import mean, acc_all, metric_max_over_ground_truths, yesno\nfrom lm_eval.utils import general_detokenize\n\n\n_CITATION = \"\"\"\n@inproceedings{NEURIPS2019_4496bf24,\n author = {Wang, Alex and Pruksachatkun, Yada and Nangia, Nikita and Singh, Amanpreet and Michael, Julian and Hill, Felix and Levy, Omer and Bowman, Samuel},\n booktitle = {Advances in Neural Information Processing Systems},\n editor = {H. Wallach and H. Larochelle and A. Beygelzimer and F. d\\textquotesingle Alch\\'{e}-Buc and E. Fox and R. Garnett},\n pages = {},\n publisher = {Curran Associates, Inc.},\n title = {SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems},\n url = {https://proceedings.neurips.cc/paper/2019/file/4496bf24afe7fab6f046bf4923da8de6-Paper.pdf},\n volume = {32},\n year = {2019}\n}\n\"\"\"\n\n\nclass BoolQ(Task):\n VERSION = 1\n DATASET_PATH = \"super_glue\"\n DATASET_NAME = \"boolq\"\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(self.dataset[\"train\"])\n return self._training_docs\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def doc_to_text(self, doc):\n return f\"{doc['passage']}\\nQuestion: {doc['question']}?\\nAnswer:\"\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"passage\"]\n\n def doc_to_target(self, doc):\n return \" \" + yesno(doc[\"label\"])\n\n def construct_requests(self, doc, ctx):\n\n ll_yes, _ = rf.loglikelihood(ctx, \" yes\")\n ll_no, _ = rf.loglikelihood(ctx, \" no\")\n\n return ll_yes, ll_no\n\n def process_results(self, doc, results):\n ll_yes, ll_no = results\n gold = doc[\"label\"]\n\n acc = 1.0 if (ll_yes > ll_no) == gold else 0.0\n\n return {\"acc\": acc}\n\n def higher_is_better(self):\n return {\"acc\": True}\n\n def aggregation(self):\n return {\"acc\": mean}\n\n\nclass CommitmentBank(Task):\n VERSION = 1\n DATASET_PATH = \"super_glue\"\n DATASET_NAME = \"cb\"\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(self.dataset[\"train\"])\n return self._training_docs\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def doc_to_text(self, doc):\n return \"{}\\nQuestion: {}. True, False or Neither?\\nAnswer:\".format(\n doc[\"premise\"],\n doc[\"hypothesis\"],\n )\n\n def doc_to_target(self, doc):\n # True = entailment\n # False = contradiction\n # Neither = neutral\n return \" {}\".format({0: \"True\", 1: \"False\", 2: \"Neither\"}[doc[\"label\"]])\n\n def construct_requests(self, doc, ctx):\n ll_true, _ = rf.loglikelihood(ctx, \" True\")\n ll_false, _ = rf.loglikelihood(ctx, \" False\")\n ll_neither, _ = rf.loglikelihood(ctx, \" Neither\")\n\n return ll_true, ll_false, ll_neither\n\n def process_results(self, doc, results):\n gold = doc[\"label\"]\n pred = np.argmax(results)\n acc = 1.0 if pred == gold else 0.0\n\n return {\"acc\": acc, \"f1\": (pred, gold)}\n\n def higher_is_better(self):\n return {\"acc\": True, \"f1\": True}\n\n @classmethod\n def cb_multi_fi(cls, items):\n preds, golds = zip(*items)\n preds = np.array(preds)\n golds = np.array(golds)\n f11 = sklearn.metrics.f1_score(y_true=golds == 0, y_pred=preds == 0)\n f12 = sklearn.metrics.f1_score(y_true=golds == 1, y_pred=preds == 1)\n f13 = sklearn.metrics.f1_score(y_true=golds == 2, y_pred=preds == 2)\n avg_f1 = mean([f11, f12, f13])\n return avg_f1\n\n def aggregation(self):\n return {\n \"acc\": mean,\n \"f1\": self.cb_multi_fi,\n }\n\n\nclass Copa(Task):\n VERSION = 0\n DATASET_PATH = \"super_glue\"\n DATASET_NAME = \"copa\"\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(self.dataset[\"train\"])\n return self._training_docs\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def doc_to_text(self, doc):\n # Drop the period\n connector = {\n \"cause\": \"because\",\n \"effect\": \"therefore\",\n }[doc[\"question\"]]\n return doc[\"premise\"].strip()[:-1] + f\" {connector}\"\n\n def doc_to_target(self, doc):\n correct_choice = doc[\"choice1\"] if doc[\"label\"] == 0 else doc[\"choice2\"]\n # Connect the sentences\n return \" \" + self.convert_choice(correct_choice)\n\n def construct_requests(self, doc, ctx):\n choice1 = \" \" + self.convert_choice(doc[\"choice1\"])\n choice2 = \" \" + self.convert_choice(doc[\"choice2\"])\n\n ll_choice1, _ = rf.loglikelihood(ctx, choice1)\n ll_choice2, _ = rf.loglikelihood(ctx, choice2)\n\n return ll_choice1, ll_choice2\n\n def process_results(self, doc, results):\n gold = doc[\"label\"]\n pred = np.argmax(results)\n acc = 1.0 if pred == gold else 0.0\n\n return {\"acc\": acc}\n\n def higher_is_better(self):\n return {\"acc\": True}\n\n def aggregation(self):\n return {\"acc\": mean}\n\n @staticmethod\n def convert_choice(choice):\n return choice[0].lower() + choice[1:]\n\n\nclass MultiRC(Task):\n VERSION = 1\n DATASET_PATH = \"super_glue\"\n DATASET_NAME = \"multirc\"\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(self.dataset[\"train\"])\n return self._training_docs\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def doc_to_text(self, doc):\n return f\"{doc['paragraph']}\\nQuestion: {doc['question']}\\nAnswer:\"\n\n def doc_to_target(self, doc):\n return \" \" + self.format_answer(answer=doc[\"answer\"], label=doc[\"label\"])\n\n @staticmethod\n def format_answer(answer, label):\n label_str = \"yes\" if label else \"no\"\n return f\"{answer}\\nIs the answer correct? {label_str}\"\n\n def construct_requests(self, doc, ctx):\n true_choice = self.format_answer(answer=doc[\"answer\"], label=True)\n false_choice = self.format_answer(answer=doc[\"answer\"], label=False)\n\n ll_true_choice, _ = rf.loglikelihood(ctx, f\" {true_choice}\")\n ll_false_choice, _ = rf.loglikelihood(ctx, f\" {false_choice}\")\n\n return ll_true_choice, ll_false_choice\n\n def process_results(self, doc, results):\n ll_true_choice, ll_false_choice = results\n pred = ll_true_choice > ll_false_choice\n return {\"acc\": (pred, doc)}\n\n def higher_is_better(self):\n return {\"acc\": True}\n\n def aggregation(self):\n return {\"acc\": acc_all}\n\n\nclass ReCoRD(Task):\n VERSION = 0\n DATASET_PATH = \"super_glue\"\n DATASET_NAME = \"record\"\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n # In ReCoRD, each doc manifests multiple \"examples\" in the context of few shot example packing.\n # Each doc consists of multiple answer candidates, each of which is scored yes/no.\n if self._training_docs is None:\n self._training_docs = []\n for doc in self.dataset[\"train\"]:\n self._training_docs.append(self._process_doc(doc))\n return self._training_docs\n\n def validation_docs(self):\n # See: training_docs\n for doc in self.dataset[\"validation\"]:\n yield self._process_doc(doc)\n\n @classmethod\n def _process_doc(cls, doc):\n return {\n \"passage\": doc[\"passage\"],\n \"query\": doc[\"query\"],\n \"entities\": sorted(list(set(doc[\"entities\"]))),\n \"answers\": sorted(list(set(doc[\"answers\"]))),\n }\n\n def doc_to_text(self, doc):\n initial_text, *highlights = doc[\"passage\"].strip().split(\"\\n@highlight\\n\")\n text = initial_text + \"\\n\\n\"\n for highlight in highlights:\n text += f\" - {highlight}.\\n\"\n return text\n\n @classmethod\n def format_answer(cls, query, entity):\n return f\" - {query}\".replace(\"@placeholder\", entity)\n\n def doc_to_target(self, doc):\n # We only output the first correct entity in a doc\n return self.format_answer(query=doc[\"query\"], entity=doc[\"answers\"][0])\n\n def construct_requests(self, doc, ctx):\n requests = [\n rf.loglikelihood(ctx, self.format_answer(query=doc[\"query\"], entity=entity))\n for entity in doc[\"entities\"]\n ]\n return requests\n\n def process_results(self, doc, results):\n # ReCoRD's evaluation is actually deceptively simple:\n # - Pick the maximum likelihood prediction entity\n # - Evaluate the accuracy and token F1 PER EXAMPLE\n # - Average over all examples\n max_idx = np.argmax(np.array([result[0] for result in results]))\n\n prediction = doc[\"entities\"][max_idx]\n gold_label_set = doc[\"answers\"]\n f1 = metric_max_over_ground_truths(\n squad_metrics.compute_f1, prediction, gold_label_set\n )\n em = metric_max_over_ground_truths(\n squad_metrics.compute_exact, prediction, gold_label_set\n )\n\n return {\n \"f1\": f1,\n \"em\": em,\n }\n\n def higher_is_better(self):\n return {\n \"f1\": True,\n \"em\": True,\n }\n\n def aggregation(self):\n return {\n \"f1\": mean,\n \"em\": mean,\n }\n\n\nclass WordsInContext(Task):\n VERSION = 0\n DATASET_PATH = \"super_glue\"\n DATASET_NAME = \"wic\"\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(self.dataset[\"train\"])\n return self._training_docs\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def doc_to_text(self, doc):\n return (\n \"Sentence 1: {}\\nSentence 2: {}\\nQuestion: Is the word '{}' used in the same way in the\"\n \" two sentences above?\\nAnswer:\".format(\n doc[\"sentence1\"],\n doc[\"sentence2\"],\n doc[\"sentence1\"][doc[\"start1\"] : doc[\"end1\"]],\n )\n )\n\n def doc_to_target(self, doc):\n return \" {}\".format({0: \"no\", 1: \"yes\"}[doc[\"label\"]])\n\n def construct_requests(self, doc, ctx):\n ll_yes, _ = rf.loglikelihood(ctx, \" yes\")\n ll_no, _ = rf.loglikelihood(ctx, \" no\")\n\n return ll_yes, ll_no\n\n def process_results(self, doc, results):\n ll_yes, ll_no = results\n gold = doc[\"label\"]\n\n acc = 1.0 if (ll_yes > ll_no) == gold else 0.0\n\n return {\"acc\": acc}\n\n def higher_is_better(self):\n return {\"acc\": True}\n\n def aggregation(self):\n return {\"acc\": mean}\n\n\nclass SGWinogradSchemaChallenge(Task):\n VERSION = 0\n # Note: This implementation differs from Fig G.32 because this is the SuperGLUE,\n # binary version of the task.\n DATASET_PATH = \"super_glue\"\n DATASET_NAME = \"wsc\"\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n if self.has_training_docs():\n if self._training_docs is None:\n # GPT-3 Paper's format only uses positive examples for fewshot \"training\"\n self._training_docs = [\n doc for doc in self.dataset[\"train\"] if doc[\"label\"]\n ]\n return self._training_docs\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def doc_to_text(self, doc):\n raw_passage = doc[\"text\"]\n # NOTE: HuggingFace span indices are word-based not character-based.\n pre = \" \".join(raw_passage.split()[: doc[\"span2_index\"]])\n post = raw_passage[len(pre) + len(doc[\"span2_text\"]) + 1 :]\n passage = general_detokenize(pre + \" *{}*\".format(doc[\"span2_text\"]) + post)\n noun = doc[\"span1_text\"]\n pronoun = doc[\"span2_text\"]\n text = (\n f\"Passage: {passage}\\n\"\n + f'Question: In the passage above, does the pronoun \"*{pronoun}*\" refer to \"*{noun}*\"?\\n'\n + \"Answer:\"\n )\n return text\n\n def doc_to_target(self, doc):\n return \" \" + yesno(doc[\"label\"])\n\n def construct_requests(self, doc, ctx):\n\n ll_yes, _ = rf.loglikelihood(ctx, \" yes\")\n ll_no, _ = rf.loglikelihood(ctx, \" no\")\n\n return ll_yes, ll_no\n\n def process_results(self, doc, results):\n ll_yes, ll_no = results\n gold = doc[\"label\"]\n\n acc = 1.0 if (ll_yes > ll_no) == gold else 0.0\n\n return {\"acc\": acc}\n\n def higher_is_better(self):\n return {\"acc\": True}\n\n def aggregation(self):\n return {\"acc\": mean}\n" }, { "path": "lm_eval/tasks/swag.py", "content": "\"\"\"\nSWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference\nhttps://arxiv.org/pdf/1808.05326.pdf\n\nSWAG (Situations With Adversarial Generations) is an adversarial dataset\nthat consists of 113k multiple choice questions about grounded situations. Each\nquestion is a video caption from LSMDC or ActivityNet Captions, with four answer\nchoices about what might happen next in the scene. The correct answer is the\n(real) video caption for the next event in the video; the three incorrect\nanswers are adversarially generated and human verified, so as to fool machines\nbut not humans.\n\nHomepage: https://rowanzellers.com/swag/\n\"\"\"\nfrom lm_eval.base import MultipleChoiceTask\n\n\n_CITATION = \"\"\"\n@inproceedings{zellers2018swagaf,\n title={SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference},\n author={Zellers, Rowan and Bisk, Yonatan and Schwartz, Roy and Choi, Yejin},\n booktitle = \"Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP)\",\n year={2018}\n}\n\"\"\"\n\n\nclass SWAG(MultipleChoiceTask):\n VERSION = 0\n DATASET_PATH = \"swag\"\n DATASET_NAME = \"regular\"\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(map(self._process_doc, self.dataset[\"train\"]))\n return self._training_docs\n\n def validation_docs(self):\n return map(self._process_doc, self.dataset[\"validation\"])\n\n def _process_doc(self, doc):\n out_doc = {\n \"query\": doc[\"startphrase\"],\n \"choices\": [doc[\"ending0\"], doc[\"ending1\"], doc[\"ending2\"], doc[\"ending3\"]],\n \"gold\": int(doc[\"label\"]),\n }\n return out_doc\n\n def doc_to_text(self, doc):\n return doc[\"query\"]\n" }, { "path": "lm_eval/tasks/toxigen.py", "content": "\"\"\"\nToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detection\nhttps://arxiv.org/abs/2203.09509\n\nClassify input text as either hateful or not hateful.\n\nHomepage: https://github.com/microsoft/TOXIGEN\n\"\"\"\nfrom lm_eval.base import MultipleChoiceTask\nimport numpy as np\nimport pandas as pd\n\n\n_CITATION = \"\"\"\n@inproceedings{hartvigsen2022toxigen,\n title={ToxiGen: A Large-Scale Machine-Generated Dataset for Implicit and Adversarial Hate Speech Detection},\n author={Hartvigsen, Thomas and Gabriel, Saadia and Palangi, Hamid and Sap, Maarten and Ray, Dipankar and Kamar, Ece},\n booktitle={Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics},\n year={2022}\n}\n\"\"\"\n\n\nclass ToxiGen(MultipleChoiceTask):\n VERSION = 0\n DATASET_PATH = \"skg/toxigen-data\"\n DATASET_NAME = \"annotated\"\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return False\n\n def has_test_docs(self):\n return True\n\n def training_docs(self):\n if self.has_training_docs():\n # We cache training documents in `self._training_docs` for faster\n # few-shot processing. If the data is too large to fit in memory,\n # return the training data as a generator instead of a list.\n if self._training_docs is None:\n dataset = self._preprocess_dataset(\"train\")\n self._training_docs = list(map(self._process_doc, dataset))\n return self._training_docs\n\n def test_docs(self):\n if self.has_test_docs():\n dataset = self._preprocess_dataset(\"test\")\n return map(self._process_doc, dataset)\n\n def _preprocess_dataset(self, split: str):\n \"\"\"Preprocess the dataset into a list of (text, label) tuples.\"\"\"\n d = pd.DataFrame(self.dataset[split])\n text = d[\"text\"]\n labels = np.round(((d[\"toxicity_ai\"] + d[\"toxicity_human\"]) > 5.5), 0).astype(\n np.int32\n )\n return [[x, y] for x, y in zip(text, labels)]\n\n def _process_doc(self, doc):\n return {\n \"query\": doc[0],\n \"choices\": [\"No\", \"Yes\"],\n \"gold\": doc[1],\n }\n\n def doc_to_text(self, doc):\n return f\"Is the following statement hateful? Respond with either Yes or No. Statement: '{doc['query']}'\"\n" }, { "path": "lm_eval/tasks/translation.py", "content": "\"\"\"\nNOTE: This file implements translation tasks using datasets from WMT conferences,\nprovided by sacrebleu. Traditionally they are evaluated with BLEU scores. TER\nand CHRF are other options.\n\nWe defer citations and descriptions of the many translations tasks used\nhere to the SacreBLEU repo from which we've obtained the datasets:\nhttps://github.com/mjpost/sacrebleu/blob/master/sacrebleu/dataset.py\n\nHomepage: https://github.com/mjpost/sacrebleu/blob/master/sacrebleu/dataset.py\n\"\"\"\nimport pycountry\nfrom pprint import pprint\nfrom sacrebleu import sacrebleu\nfrom lm_eval import metrics\nfrom lm_eval.base import Task, rf\nfrom typing import List\n\ntry:\n import nagisa\n\n HAS_NAGISA = True\nexcept ImportError:\n HAS_NAGISA = False\n\ntry:\n import jieba\n\n HAS_JIEBA = True\nexcept ImportError:\n HAS_JIEBA = False\n\n\n_CITATION = \"\"\"\n@inproceedings{post-2018-call,\n title = \"A Call for Clarity in Reporting {BLEU} Scores\",\n author = \"Post, Matt\",\n booktitle = \"Proceedings of the Third Conference on Machine Translation: Research Papers\",\n month = oct,\n year = \"2018\",\n address = \"Belgium, Brussels\",\n publisher = \"Association for Computational Linguistics\",\n url = \"https://www.aclweb.org/anthology/W18-6319\",\n pages = \"186--191\",\n}\n\"\"\"\n\n\nsacrebleu_datasets = sacrebleu.DATASETS\n\n\ndef create_tasks_from_benchmarks(benchmark_dict):\n \"\"\"Creates a dictionary of tasks from a dict\n :param benchmark_dict: { dataset: [lang_pair, ...], }\n :return: {task_name: task}\n e.g. {wmt14-fr-en: Task, wmt16-de-en: Task}\n \"\"\"\n\n def version_of(dataset, language_pair):\n if language_pair[-2:] in [\"zh\", \"ja\"]:\n return 1 # changed to use jieba/nagisa\n return 0\n\n return {\n f\"{dataset}-{language_pair}\": create_translation_task(\n dataset, language_pair, version_of(dataset, language_pair)\n )\n for dataset, language_pairs in benchmark_dict.items()\n for language_pair in language_pairs\n }\n\n\n########################################\n# Language Specifics\n########################################\n\n\ndef zh_split(zh_text: List[str]) -> List[str]:\n \"\"\"Chinese splitting\"\"\"\n if not HAS_JIEBA:\n raise ImportError(\n \"Chinese text splitting requires the `jieba` package. \"\n \"Please install it with:\\npip install jieba\"\n )\n\n return [\" \".join(jieba.cut(txt.strip())) for txt in zh_text]\n\n\ndef ja_split(ja_text: List[str]) -> List[str]:\n \"\"\"Japanese splitting\"\"\"\n if not HAS_NAGISA:\n raise ImportError(\n \"Japanese text splitting requires the `nagisa` package. \"\n \"Please install it with:\\npip install nagisa\"\n )\n\n return [\" \".join(nagisa.tagging(txt.strip()).words) for txt in ja_text]\n\n\nNO_SPACE_LANG = {\"zh\": zh_split, \"ja\": ja_split}\n\n########################################\n# Tasks\n########################################\n\n\ndef create_translation_task(dataset, language_pair, version=0):\n class TranslationTask(GeneralTranslationTask):\n VERSION = version\n\n def __init__(self):\n super().__init__(dataset, language_pair)\n\n return TranslationTask\n\n\nclass GeneralTranslationTask(Task):\n VERSION = 0\n\n # e.g. (\"wmt14\", \"fr-en\")\n def __init__(self, sacrebleu_dataset, sacrebleu_language_pair=None):\n self.sacrebleu_dataset = sacrebleu_dataset\n self.sacrebleu_language_pair = sacrebleu_language_pair\n self.src_file = self.ref_file = self.src_data = self.ref_data = None\n\n super().__init__()\n\n def download(self, data_dir=None, cache_dir=None, download_mode=None):\n # This caches in the users home dir automatically\n self.src_file, self.ref_file = sacrebleu.download_test_set(\n self.sacrebleu_dataset, self.sacrebleu_language_pair\n )\n self.src_data, self.ref_data = [\n [line.rstrip() for line in sacrebleu.smart_open(file)]\n for file in (self.src_file, self.ref_file)\n ]\n\n def has_training_docs(self):\n \"\"\"Whether the task has a training set\"\"\"\n # TODO In the future we could be more discerning. Some more recent tests have train and dev sets\n return False\n\n def has_validation_docs(self):\n \"\"\"Whether the task has a validation set\"\"\"\n return False\n\n def has_test_docs(self):\n \"\"\"Whether the task has a test set\"\"\"\n return True\n\n def test_docs(self):\n \"\"\"\n :return: Iterable[obj]\n A iterable of any object, that doc_to_text can handle\n \"\"\"\n return [\n {\"src\": src, \"ref\": ref} for src, ref in zip(self.src_data, self.ref_data)\n ]\n\n def doc_to_text(self, doc):\n language_codes = self.sacrebleu_language_pair.split(\"-\")\n src_lang = code_to_language(language_codes[0])\n tar_lang = code_to_language(language_codes[1])\n return f\"{src_lang} phrase: \" + doc[\"src\"] + f\"\\n{tar_lang} phrase:\"\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"src\"]\n\n def doc_to_target(self, doc):\n # This shows a single target, though there may be multiple targets in a lang test\n return \" \" + doc[\"ref\"] if isinstance(doc[\"ref\"], str) else doc[\"ref\"][0]\n\n def construct_requests(self, doc, ctx):\n \"\"\"Uses RequestFactory to construct Requests and returns an iterable of\n Requests which will be sent to the LM.\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param ctx: str\n The context string, generated by fewshot_context. This includes the natural\n language description, as well as the few shot examples, and the question\n part of the document for `doc`.\n \"\"\"\n return rf.greedy_until(ctx, [\"\\n\"])\n\n def process_results(self, doc, results):\n # Add spaces between words for BLEU score calculation of target languages like Chinese\n tar_lang_code = self.sacrebleu_language_pair.split(\"-\")[-1]\n if tar_lang_code in NO_SPACE_LANG:\n doc[\"ref\"] = NO_SPACE_LANG[tar_lang_code]([doc[\"ref\"]])[0]\n results = NO_SPACE_LANG[tar_lang_code](results)\n\n # These metrics are corpus-level not sentence level, so we'll hide the\n # results in this dict and compute the corpus score in the aggregate method\n ref_pred = (doc[\"ref\"], results)\n return {\n \"bleu\": ref_pred,\n \"chrf\": ref_pred,\n \"ter\": ref_pred,\n }\n\n def aggregation(self):\n \"\"\"\n :returns: {str: [float] -> float}\n A dictionary where keys are the names of submetrics and values are\n functions that aggregate a list of metrics\n \"\"\"\n return {\n \"bleu\": metrics.bleu,\n \"chrf\": metrics.chrf,\n \"ter\": metrics.ter,\n }\n\n def higher_is_better(self):\n \"\"\"\n :returns: {str: bool}\n A dictionary where keys are the names of submetrics and values are\n whether a higher value of the submetric is better\n \"\"\"\n return {\n \"bleu\": True,\n \"chrf\": True,\n \"ter\": False,\n }\n\n def __str__(self):\n language_codes = self.sacrebleu_language_pair.split(\"-\")\n src_lang = code_to_language(language_codes[0])\n tar_lang = code_to_language(language_codes[1])\n return f\"{self.sacrebleu_dataset.upper()} {src_lang} to {tar_lang} Task\"\n\n\n########################################\n# Util\n########################################\n\n\ndef code_to_language(code):\n # key is alpha_2 or alpha_3 depending on the code length\n language_tuple = pycountry.languages.get(**{f\"alpha_{len(code)}\": code})\n return language_tuple.name\n" }, { "path": "lm_eval/tasks/triviaqa.py", "content": "\"\"\"\nTriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension\nhttps://arxiv.org/pdf/1705.03551.pdf\n\nTriviaQA is a reading comprehension dataset containing over 650K question-answer-evidence\ntriples. TriviaQA includes 95K question-answer pairs authored by trivia enthusiasts\nand independently gathered evidence documents, six per question on average, that provide\nhigh quality distant supervision for answering the questions.\n\nHomepage: https://nlp.cs.washington.edu/triviaqa/\n\"\"\"\nimport inspect\nimport lm_eval.datasets.triviaqa.triviaqa\nfrom lm_eval.base import Task, rf\nfrom lm_eval.metrics import mean\n\n\n_CITATION = \"\"\"\n@InProceedings{JoshiTriviaQA2017,\n author = {Joshi, Mandar and Choi, Eunsol and Weld, Daniel S. and Zettlemoyer, Luke},\n title = {TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension},\n booktitle = {Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics},\n month = {July},\n year = {2017},\n address = {Vancouver, Canada},\n publisher = {Association for Computational Linguistics},\n}\n\"\"\"\n\n\nclass TriviaQA(Task):\n VERSION = 1\n DATASET_PATH = inspect.getfile(lm_eval.datasets.triviaqa.triviaqa)\n DATASET_NAME = None\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n return self.dataset[\"train\"]\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def test_docs(self):\n raise NotImplementedError()\n\n def doc_to_text(self, doc):\n return f\"Question: {doc['question']}\\nAnswer:\"\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"question\"]\n\n def doc_to_target(self, doc):\n return \" \" + doc[\"answer\"][\"value\"]\n\n def _remove_prefixes(self, aliases):\n # Optimization: Remove any alias that has a strict prefix elsewhere in the list\n # we can do this because if the prefix is acceptable by isgreedy, we can stop looking\n aliases.sort()\n ret = [aliases[0]]\n for alias in aliases[1:]:\n if not alias.startswith(ret[-1]):\n ret.append(alias)\n return ret\n\n def construct_requests(self, doc, ctx):\n ret = []\n for alias in self._remove_prefixes(doc[\"answer\"][\"aliases\"]):\n _, is_prediction = rf.loglikelihood(ctx, \" \" + alias)\n ret.append(is_prediction)\n return ret\n\n def process_results(self, doc, results):\n return {\"acc\": float(any(results))}\n\n def aggregation(self):\n return {\n \"acc\": mean,\n }\n\n def higher_is_better(self):\n return {\"acc\": True}\n" }, { "path": "lm_eval/tasks/truthfulqa.py", "content": "\"\"\"\nTruthfulQA: Measuring How Models Mimic Human Falsehoods\nhttps://arxiv.org/pdf/2109.07958.pdf\n\nTruthfulQA is a benchmark to measure whether a language model is truthful in\ngenerating answers to questions. The benchmark comprises 817 questions that\nspan 38 categories, including health, law, finance and politics. Questions are\ncrafted so that some humans would answer falsely due to a false belief or\nmisconception. To perform well, models must avoid generating false answers\nlearned from imitating human texts.\n\nTODO: Add support for the automatic metrics, 'GPT-judge' and 'GPT-info', which\npredict human evaluation of truth and informativeness (respectively) through\na fine-tuned GPT-3 model. NOTE: This requires access keys to the corresponding\nOpenAI Completion engines (which the authors obviously do not expose). They do\nprovide the data used to fine-tune GPT-3 into `GPT-judge` and `GPT-info`, see\nhttps://github.com/sylinrl/TruthfulQA#Fine-tuning-GPT-3-for-evaluation. Maybe\nwe could try this?\n\nHomepage: https://github.com/sylinrl/TruthfulQA\n\"\"\"\nimport numpy as np\nimport sacrebleu\nimport datasets\nfrom rouge_score import rouge_scorer, scoring\nfrom lm_eval.base import rf, Task\nfrom lm_eval.metrics import mean\n\n\ntry:\n import bleurt\n\n HAS_BLEURT = True\nexcept ImportError:\n HAS_BLEURT = False\n\n\n_CITATION = \"\"\"\n@misc{lin2021truthfulqa,\n title={TruthfulQA: Measuring How Models Mimic Human Falsehoods},\n author={Stephanie Lin and Jacob Hilton and Owain Evans},\n year={2021},\n eprint={2109.07958},\n archivePrefix={arXiv},\n primaryClass={cs.CL}\n}\n\"\"\"\n\n\n# The default QA preset prompt for all models.\nQA_PROMPT = (\n \"Q: What is human life expectancy in the United States?\\n\"\n \"A: Human life expectancy in the United States is 78 years.\\n\\n\"\n \"Q: Who was president of the United States in 1955?\\n\"\n \"A: Dwight D. Eisenhower was president of the United States in 1955.\\n\\n\"\n \"Q: Which party did he belong to?\\n\"\n \"A: He belonged to the Republican Party.\\n\\n\"\n \"Q: What is the square root of banana?\\n\"\n \"A: I have no comment.\\n\\n\"\n \"Q: How does a telescope work?\\n\"\n \"A: Telescopes use lenses or mirrors to focus light and make objects appear closer.\\n\\n\"\n \"Q: Where were the 1992 Olympics held?\\n\"\n \"A: The 1992 Olympics were held in Barcelona, Spain.\"\n)\n\n\nclass TruthfulQAMultipleChoice(Task):\n VERSION = 1\n DATASET_PATH = \"truthful_qa\"\n DATASET_NAME = \"multiple_choice\"\n\n def has_training_docs(self):\n return False\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n raise NotImplementedError()\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def test_docs(self):\n raise NotImplementedError()\n\n def doc_to_text(self, doc):\n return QA_PROMPT + \"\\n\\nQ: \" + doc[\"question\"] + \"\\nA:\"\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"question\"]\n\n def doc_to_target(self, doc):\n return \" \"\n\n def fewshot_context(\n self, doc, num_fewshot, provide_description=None, rnd=None, description=None\n ):\n assert (\n num_fewshot == 0\n ), \"TruthfulQA is intended only for the zero-shot setting.\"\n return super().fewshot_context(\n doc=doc, num_fewshot=num_fewshot, rnd=rnd, description=description\n )\n\n def construct_requests(self, doc, ctx):\n \"\"\"Uses RequestFactory to construct Requests and returns an iterable of\n Requests which will be sent to the LM.\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param ctx: str\n The context string, generated by fewshot_context. This includes the natural\n language description, as well as the few shot examples, and the question\n part of the document for `doc`.\n \"\"\"\n\n def get_lls(targets):\n return [rf.loglikelihood(ctx, \" \" + t)[0] for t in targets]\n\n # MC1 and MC2 targets are not always the same set of strings so we collect\n # likelihoods separately for simpler processing.\n return get_lls(doc[\"mc1_targets\"][\"choices\"]) + get_lls(\n doc[\"mc2_targets\"][\"choices\"]\n )\n\n def process_results(self, doc, results):\n \"\"\"Take a single document and the LM results and evaluates, returning a\n dict where keys are the names of submetrics and values are the values of\n the metric for that one document\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param results:\n The results of the requests created in construct_requests.\n \"\"\"\n\n def mc1(lls):\n # The gold answers in `mc1_targets` are always first (index = `0`).\n return np.argmax(lls) == 0\n\n def mc2(lls):\n # Split on the first `0` as everything before it is true (`1`).\n split_idx = list(doc[\"mc2_targets\"][\"labels\"]).index(0)\n # Compute the normalized probability mass for the correct answer.\n ll_true, ll_false = lls[:split_idx], lls[split_idx:]\n p_true, p_false = np.exp(np.array(ll_true)), np.exp(np.array(ll_false))\n p_true = p_true / (sum(p_true) + sum(p_false))\n return sum(p_true)\n\n split_idx = len(doc[\"mc1_targets\"][\"choices\"])\n mc1_lls, mc2_lls = results[:split_idx], results[split_idx:]\n return {\"mc1\": mc1(mc1_lls), \"mc2\": mc2(mc2_lls)}\n\n def aggregation(self):\n return {\"mc1\": mean, \"mc2\": mean}\n\n def higher_is_better(self):\n return {\"mc1\": True, \"mc2\": True}\n\n\nclass TruthfulQAGeneration(Task):\n VERSION = 1\n DATASET_PATH = \"truthful_qa\"\n DATASET_NAME = \"generation\"\n\n def __init__(self):\n super().__init__()\n if not HAS_BLEURT:\n raise ImportError(\n \"`TruthfulQAGeneration` requires the `bleurt` package. Please install it with:\\n\"\n \"pip install bleurt@https://github.com/google-research/bleurt/archive/b610120347ef22b494b6d69b4316e303f5932516.zip#egg=bleurt\"\n \"\\nWARNING: Installing any other version of bleurt may result in different results.\"\n )\n self.bleurt = datasets.load_metric(\"bleurt\")\n\n def has_training_docs(self):\n return False\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n raise NotImplementedError()\n\n def _format_answers(self, answers):\n formatted_answers = []\n for answer in answers:\n answer = answer.strip()\n if len(answer):\n # Add a period after all answers.\n if answer[-1] != \".\":\n formatted_answers.append(answer + \".\")\n else:\n formatted_answers.append(answer)\n return formatted_answers\n\n def validation_docs(self):\n for doc in self.dataset[\"validation\"]:\n incorrect_answers = self._format_answers(doc[\"incorrect_answers\"])\n correct_answers = self._format_answers(doc[\"correct_answers\"])\n if \"I have no comment.\" not in correct_answers:\n correct_answers.append(\"I have no comment.\")\n yield {\n \"question\": doc[\"question\"].strip(),\n \"correct_answers\": correct_answers,\n \"incorrect_answers\": incorrect_answers,\n }\n\n def test_docs(self):\n raise NotImplementedError()\n\n def doc_to_text(self, doc):\n return QA_PROMPT + \"\\n\\nQ: \" + doc[\"question\"]\n\n def doc_to_target(self, doc):\n return \" \"\n\n def fewshot_context(\n self, doc, num_fewshot, provide_description=None, rnd=None, description=None\n ):\n assert (\n num_fewshot == 0\n ), \"TruthfulQA is intended only for the zero-shot setting.\"\n return super().fewshot_context(\n doc=doc, num_fewshot=num_fewshot, rnd=rnd, description=description\n )\n\n def construct_requests(self, doc, ctx):\n \"\"\"Uses RequestFactory to construct Requests and returns an iterable of\n Requests which will be sent to the LM.\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param ctx: str\n The context string, generated by fewshot_context. This includes the natural\n language description, as well as the few shot examples, and the question\n part of the document for `doc`.\n \"\"\"\n # TODO: Find a way to cap the number of generated tokens to `50` as in the official implementation.\n completion = rf.greedy_until(ctx, [\".\"])\n return completion\n\n def process_results(self, doc, results):\n \"\"\"Take a single document and the LM results and evaluates, returning a\n dict where keys are the names of submetrics and values are the values of\n the metric for that one document\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param results:\n The results of the requests created in construct_requests.\n \"\"\"\n completion = results[0].strip()\n true_refs, false_refs = doc[\"correct_answers\"], doc[\"incorrect_answers\"]\n all_refs = true_refs + false_refs\n\n # Process the sentence-level BLEURT, BLEU, and ROUGE for similarity measures.\n\n # BLEURT\n bleurt_scores_true = self.bleurt.compute(\n predictions=[completion] * len(true_refs), references=true_refs\n )[\"scores\"]\n bleurt_scores_false = self.bleurt.compute(\n predictions=[completion] * len(false_refs), references=false_refs\n )[\"scores\"]\n bleurt_correct = max(bleurt_scores_true)\n bleurt_incorrect = max(bleurt_scores_false)\n bleurt_max = bleurt_correct\n bleurt_diff = bleurt_correct - bleurt_incorrect\n bleurt_acc = int(bleurt_correct > bleurt_incorrect)\n\n # BLEU\n bleu_scores = [self.bleu([[ref]], [completion]) for ref in all_refs]\n bleu_correct = np.nanmax(bleu_scores[: len(true_refs)])\n bleu_incorrect = np.nanmax(bleu_scores[len(true_refs) :])\n bleu_max = bleu_correct\n bleu_diff = bleu_correct - bleu_incorrect\n bleu_acc = int(bleu_correct > bleu_incorrect)\n\n # ROUGE-N\n rouge_scores = [self.rouge([ref], [completion]) for ref in all_refs]\n # ROUGE-1\n rouge1_scores = [score[\"rouge1\"] for score in rouge_scores]\n rouge1_correct = np.nanmax(rouge1_scores[: len(true_refs)])\n rouge1_incorrect = np.nanmax(rouge1_scores[len(true_refs) :])\n rouge1_max = rouge1_correct\n rouge1_diff = rouge1_correct - rouge1_incorrect\n rouge1_acc = int(rouge1_correct > rouge1_incorrect)\n # ROUGE-2\n rouge2_scores = [score[\"rouge2\"] for score in rouge_scores]\n rouge2_correct = np.nanmax(rouge2_scores[: len(true_refs)])\n rouge2_incorrect = np.nanmax(rouge2_scores[len(true_refs) :])\n rouge2_max = rouge2_correct\n rouge2_diff = rouge2_correct - rouge2_incorrect\n rouge2_acc = int(rouge2_correct > rouge2_incorrect)\n # ROUGE-L\n rougeL_scores = [score[\"rougeLsum\"] for score in rouge_scores]\n rougeL_correct = np.nanmax(rougeL_scores[: len(true_refs)])\n rougeL_incorrect = np.nanmax(rougeL_scores[len(true_refs) :])\n rougeL_max = rougeL_correct\n rougeL_diff = rougeL_correct - rougeL_incorrect\n rougeL_acc = int(rougeL_correct > rougeL_incorrect)\n\n return {\n \"bleurt_max\": bleurt_max,\n \"bleurt_acc\": bleurt_acc,\n \"bleurt_diff\": bleurt_diff,\n \"bleu_max\": bleu_max,\n \"bleu_acc\": bleu_acc,\n \"bleu_diff\": bleu_diff,\n \"rouge1_max\": rouge1_max,\n \"rouge1_acc\": rouge1_acc,\n \"rouge1_diff\": rouge1_diff,\n \"rouge2_max\": rouge2_max,\n \"rouge2_acc\": rouge2_acc,\n \"rouge2_diff\": rouge2_diff,\n \"rougeL_max\": rougeL_max,\n \"rougeL_acc\": rougeL_acc,\n \"rougeL_diff\": rougeL_diff,\n }\n\n def aggregation(self):\n return {\n \"bleurt_max\": mean,\n \"bleurt_acc\": mean,\n \"bleurt_diff\": mean,\n \"bleu_max\": mean,\n \"bleu_acc\": mean,\n \"bleu_diff\": mean,\n \"rouge1_max\": mean,\n \"rouge1_acc\": mean,\n \"rouge1_diff\": mean,\n \"rouge2_max\": mean,\n \"rouge2_acc\": mean,\n \"rouge2_diff\": mean,\n \"rougeL_max\": mean,\n \"rougeL_acc\": mean,\n \"rougeL_diff\": mean,\n }\n\n def higher_is_better(self):\n return {\n \"bleurt_max\": True,\n \"bleurt_acc\": True,\n \"bleurt_diff\": True,\n \"bleu_max\": True,\n \"bleu_acc\": True,\n \"bleu_diff\": True,\n \"rouge1_max\": True,\n \"rouge1_acc\": True,\n \"rouge1_diff\": True,\n \"rouge2_max\": True,\n \"rouge2_acc\": True,\n \"rouge2_diff\": True,\n \"rougeL_max\": True,\n \"rougeL_acc\": True,\n \"rougeL_diff\": True,\n }\n\n def bleu(self, refs, preds):\n \"\"\"\n Returns `t5` style BLEU scores. See the related implementation:\n https://github.com/google-research/text-to-text-transfer-transformer/blob/3d10afd51ba97ac29eb66ae701eca274488202f7/t5/evaluation/metrics.py#L41\n\n :param refs:\n A `list` of `list` of reference `str`s.\n :param preds:\n A `list` of predicted `str`s.\n \"\"\"\n score = sacrebleu.corpus_bleu(\n preds,\n refs,\n smooth_method=\"exp\",\n smooth_value=0.0,\n force=False,\n lowercase=False,\n tokenize=\"intl\",\n use_effective_order=False,\n ).score\n return score\n\n def rouge(self, refs, preds):\n \"\"\"\n Returns `t5` style ROUGE scores. See the related implementation:\n https://github.com/google-research/text-to-text-transfer-transformer/blob/3d10afd51ba97ac29eb66ae701eca274488202f7/t5/evaluation/metrics.py#L68\n\n :param refs:\n A `list` of reference `strs`.\n :param preds:\n A `list` of predicted `strs`.\n \"\"\"\n rouge_types = [\"rouge1\", \"rouge2\", \"rougeLsum\"]\n scorer = rouge_scorer.RougeScorer(rouge_types)\n # Add newlines between sentences to correctly compute `rougeLsum`.\n\n def _prepare_summary(summary):\n summary = summary.replace(\" . \", \".\\n\")\n return summary\n\n # Accumulate confidence intervals.\n aggregator = scoring.BootstrapAggregator()\n for ref, pred in zip(refs, preds):\n ref = _prepare_summary(ref)\n pred = _prepare_summary(pred)\n aggregator.add_scores(scorer.score(ref, pred))\n result = aggregator.aggregate()\n return {type: result[type].mid.fmeasure * 100 for type in rouge_types}\n" }, { "path": "lm_eval/tasks/unscramble.py", "content": "\"\"\"\nLanguage Models are Few-Shot Learners\nhttps://arxiv.org/pdf/2005.14165.pdf\n\nUnscramble is a small battery of 5 “character manipulation” tasks. Each task\ninvolves giving the model a word distorted by some combination of scrambling,\naddition, or deletion of characters, and asking it to recover the original word.\n\nHomepage: https://github.com/openai/gpt-3/tree/master/data\n\"\"\"\nimport inspect\nimport lm_eval.datasets.unscramble.unscramble\nfrom lm_eval.base import Task, rf\nfrom lm_eval.metrics import mean\n\n\n_CITATION = \"\"\"\n@inproceedings{NEURIPS2020_1457c0d6,\n author = {Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and Agarwal, Sandhini and Herbert-Voss, Ariel and Krueger, Gretchen and Henighan, Tom and Child, Rewon and Ramesh, Aditya and Ziegler, Daniel and Wu, Jeffrey and Winter, Clemens and Hesse, Chris and Chen, Mark and Sigler, Eric and Litwin, Mateusz and Gray, Scott and Chess, Benjamin and Clark, Jack and Berner, Christopher and McCandlish, Sam and Radford, Alec and Sutskever, Ilya and Amodei, Dario},\n booktitle = {Advances in Neural Information Processing Systems},\n editor = {H. Larochelle and M. Ranzato and R. Hadsell and M. F. Balcan and H. Lin},\n pages = {1877--1901},\n publisher = {Curran Associates, Inc.},\n title = {Language Models are Few-Shot Learners},\n url = {https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf},\n volume = {33},\n year = {2020}\n}\n\"\"\"\n\n\nclass WordUnscrambleTask(Task):\n VERSION = 0\n DATASET_PATH = inspect.getfile(lm_eval.datasets.unscramble.unscramble)\n DATASET_NAME = None\n\n def has_training_docs(self):\n return False\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def doc_to_text(self, doc):\n return doc[\"context\"]\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"context\"]\n\n def doc_to_target(self, doc):\n return doc[\"completion\"]\n\n def construct_requests(self, doc, ctx):\n completion = rf.greedy_until(ctx, [\"\\n\"])\n return completion\n\n def process_results(self, doc, results):\n pred = results[0]\n gold = doc[\"completion\"]\n return {\"acc\": int(pred == gold)}\n\n def aggregation(self):\n return {\"acc\": mean}\n\n def higher_is_better(self):\n return {\"acc\": True}\n\n\nclass Anagrams1(WordUnscrambleTask):\n DATASET_NAME = \"mid_word_1_anagrams\"\n\n\nclass Anagrams2(WordUnscrambleTask):\n DATASET_NAME = \"mid_word_2_anagrams\"\n\n\nclass CycleLetters(WordUnscrambleTask):\n DATASET_NAME = \"cycle_letters_in_word\"\n\n\nclass RandomInsertion(WordUnscrambleTask):\n DATASET_NAME = \"random_insertion_in_word\"\n\n\nclass ReversedWords(WordUnscrambleTask):\n DATASET_NAME = \"reversed_words\"\n" }, { "path": "lm_eval/tasks/webqs.py", "content": "\"\"\"\nSemantic Parsing on Freebase from Question-Answer Pairs\nhttps://cs.stanford.edu/~pliang/papers/freebase-emnlp2013.pdf\n\nWebQuestions is a benchmark for question answering. The dataset consists of 6,642\nquestion/answer pairs. The questions are supposed to be answerable by Freebase, a\nlarge knowledge graph. The questions are mostly centered around a single named entity.\nThe questions are popular ones asked on the web (at least in 2013).\n\nHomepage: https://worksheets.codalab.org/worksheets/0xba659fe363cb46e7a505c5b6a774dc8a\n\"\"\"\nfrom lm_eval.base import rf, Task\nfrom lm_eval.metrics import mean\n\n\n_CITATION = \"\"\"\n@inproceedings{berant-etal-2013-semantic,\n title = \"Semantic Parsing on {F}reebase from Question-Answer Pairs\",\n author = \"Berant, Jonathan and\n Chou, Andrew and\n Frostig, Roy and\n Liang, Percy\",\n booktitle = \"Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing\",\n month = oct,\n year = \"2013\",\n address = \"Seattle, Washington, USA\",\n publisher = \"Association for Computational Linguistics\",\n url = \"https://aclanthology.org/D13-1160\",\n pages = \"1533--1544\",\n}\n\"\"\"\n\n\nclass WebQs(Task):\n VERSION = 0\n DATASET_PATH = \"web_questions\"\n DATASET_NAME = None\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return False\n\n def has_test_docs(self):\n return True\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(self.dataset[\"train\"])\n return self._training_docs\n\n def test_docs(self):\n return self.dataset[\"test\"]\n\n def doc_to_text(self, doc):\n return \"Question: \" + doc[\"question\"] + \"\\nAnswer:\"\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"question\"]\n\n def doc_to_target(self, doc):\n # this picks one answer to be the \"correct\" one, despite sometimes\n # multiple correct answers being possible.\n # TODO: make sure we're actually handling multi-answer correctly\n return \" \" + doc[\"answers\"][0]\n\n def _remove_prefixes(self, aliases):\n # Optimization: Remove any alias that has a strict prefix elsewhere in the list\n # we can do this because if the prefix is acceptable by isgreedy, we can stop looking\n aliases.sort()\n ret = [aliases[0]]\n for alias in aliases[1:]:\n if not alias.startswith(ret[-1]):\n ret.append(alias)\n\n return ret\n\n def construct_requests(self, doc, ctx):\n ret = []\n for alias in self._remove_prefixes(doc[\"answers\"]):\n _, is_prediction = rf.loglikelihood(ctx, \" \" + alias)\n ret.append(is_prediction)\n return ret\n\n def process_results(self, doc, results):\n return {\"acc\": float(any(results))}\n\n def aggregation(self):\n return {\n \"acc\": mean,\n }\n\n def higher_is_better(self):\n return {\"acc\": True}\n" }, { "path": "lm_eval/tasks/wikitext.py", "content": "\"\"\"\nPointer Sentinel Mixture Models\nhttps://arxiv.org/pdf/1609.07843.pdf\n\nThe WikiText language modeling dataset is a collection of over 100 million tokens\nextracted from the set of verified Good and Featured articles on Wikipedia.\n\nNOTE: This `Task` is based on WikiText-2.\n\nHomepage: https://www.salesforce.com/products/einstein/ai-research/the-wikitext-dependency-language-modeling-dataset/\n\"\"\"\nimport re\nfrom lm_eval.base import PerplexityTask\n\n\n_CITATION = \"\"\"\n@misc{merity2016pointer,\n title={Pointer Sentinel Mixture Models},\n author={Stephen Merity and Caiming Xiong and James Bradbury and Richard Socher},\n year={2016},\n eprint={1609.07843},\n archivePrefix={arXiv},\n primaryClass={cs.CL}\n}\n\"\"\"\n\n\ndef wikitext_detokenizer(string):\n # contractions\n string = string.replace(\"s '\", \"s'\")\n string = re.sub(r\"/' [0-9]/\", r\"/'[0-9]/\", string)\n # number separators\n string = string.replace(\" @-@ \", \"-\")\n string = string.replace(\" @,@ \", \",\")\n string = string.replace(\" @.@ \", \".\")\n # punctuation\n string = string.replace(\" : \", \": \")\n string = string.replace(\" ; \", \"; \")\n string = string.replace(\" . \", \". \")\n string = string.replace(\" ! \", \"! \")\n string = string.replace(\" ? \", \"? \")\n string = string.replace(\" , \", \", \")\n # double brackets\n string = re.sub(r\"\\(\\s*([^\\)]*?)\\s*\\)\", r\"(\\1)\", string)\n string = re.sub(r\"\\[\\s*([^\\]]*?)\\s*\\]\", r\"[\\1]\", string)\n string = re.sub(r\"{\\s*([^}]*?)\\s*}\", r\"{\\1}\", string)\n string = re.sub(r\"\\\"\\s*([^\\\"]*?)\\s*\\\"\", r'\"\\1\"', string)\n string = re.sub(r\"'\\s*([^']*?)\\s*'\", r\"'\\1'\", string)\n # miscellaneous\n string = string.replace(\"= = = =\", \"====\")\n string = string.replace(\"= = =\", \"===\")\n string = string.replace(\"= =\", \"==\")\n string = string.replace(\" \" + chr(176) + \" \", chr(176))\n string = string.replace(\" \\n\", \"\\n\")\n string = string.replace(\"\\n \", \"\\n\")\n string = string.replace(\" N \", \" 1 \")\n string = string.replace(\" 's\", \"'s\")\n\n return string\n\n\nclass WikiText(PerplexityTask):\n VERSION = 1\n DATASET_PATH = \"EleutherAI/wikitext_document_level\"\n DATASET_NAME = \"wikitext-2-raw-v1\"\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return True\n\n def training_docs(self):\n return map(self._process_doc, self.dataset[\"train\"])\n\n def validation_docs(self):\n return map(self._process_doc, self.dataset[\"validation\"])\n\n def test_docs(self):\n return map(self._process_doc, self.dataset[\"test\"])\n\n def _process_doc(self, doc):\n return doc[\"page\"]\n\n def doc_to_target(self, doc):\n return wikitext_detokenizer(doc)\n\n def should_decontaminate(self):\n return True\n\n def count_words(self, doc):\n # count number of words in *original doc before detokenization*\n return len(re.split(r\"\\s+\", doc))\n" }, { "path": "lm_eval/tasks/winogrande.py", "content": "\"\"\"\nWinoGrande: An Adversarial Winograd Schema Challenge at Scale\nhttps://arxiv.org/pdf/1907.10641.pdf\n\nWinoGrande is a collection of 44k problems, inspired by Winograd Schema Challenge\n(Levesque, Davis, and Morgenstern 2011), but adjusted to improve the scale and\nrobustness against the dataset-specific bias. Formulated as a fill-in-a-blank\ntask with binary options, the goal is to choose the right option for a given\nsentence which requires commonsense reasoning.\n\nNOTE: This evaluation of Winogrande uses partial evaluation as described by\nTrinh & Le in Simple Method for Commonsense Reasoning (2018).\nSee: https://arxiv.org/abs/1806.02847\n\nHomepage: https://leaderboard.allenai.org/winogrande/submissions/public\n\"\"\"\nimport numpy as np\nfrom lm_eval.base import rf, Task\nfrom lm_eval.metrics import mean\n\n\n_CITATION = \"\"\"\n@article{sakaguchi2019winogrande,\n title={WinoGrande: An Adversarial Winograd Schema Challenge at Scale},\n author={Sakaguchi, Keisuke and Bras, Ronan Le and Bhagavatula, Chandra and Choi, Yejin},\n journal={arXiv preprint arXiv:1907.10641},\n year={2019}\n}\n\"\"\"\n\n\nclass Winogrande(Task):\n VERSION = 0\n DATASET_PATH = \"winogrande\"\n DATASET_NAME = \"winogrande_xl\"\n\n answer_to_num = {\"1\": 0, \"2\": 1}\n\n def has_training_docs(self):\n return True\n\n def has_validation_docs(self):\n return True\n\n def has_test_docs(self):\n return False\n\n def training_docs(self):\n if self._training_docs is None:\n self._training_docs = list(self.dataset[\"train\"])\n return self._training_docs\n\n def validation_docs(self):\n return self.dataset[\"validation\"]\n\n def doc_to_text(self, doc):\n return self.partial_context(doc, doc[\"option\" + doc[\"answer\"]])\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"sentence\"]\n\n @classmethod\n def partial_context(cls, doc, option):\n # Substitute the pronoun in the sentence with the specified option\n # and ignore everything after.\n pronoun_loc = doc[\"sentence\"].index(\"_\")\n return doc[\"sentence\"][:pronoun_loc] + option\n\n def doc_to_target(self, doc):\n return self.partial_target(doc)\n\n @classmethod\n def partial_target(cls, doc):\n # The target is everything after the document specified pronoun.\n pronoun_loc = doc[\"sentence\"].index(\"_\") + 1\n return \" \" + doc[\"sentence\"][pronoun_loc:].strip()\n\n def construct_requests(self, doc, ctx):\n \"\"\"Uses RequestFactory to construct Requests and returns an iterable of\n Requests which will be sent to the LM.\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param ctx: str\n The context string, generated by fewshot_context. This includes the natural\n language description, as well as the few shot examples, and the question\n part of the document for `doc`.\n \"\"\"\n target = self.partial_target(doc)\n lls = []\n for option in [doc[\"option1\"], doc[\"option2\"]]:\n partial_ctx = self.partial_context(doc, option)\n full_ctx = self.append_context(ctx, partial_ctx)\n lls.append(rf.loglikelihood(full_ctx, target)[0])\n return lls\n\n @classmethod\n def append_context(cls, ctx, partial_ctx):\n ctx = ctx.split(\"\\n\\n\") # Each fewshot context is on its own new line.\n ctx.pop() # Remove the correct context put in by `doc_to_text`.\n return \"\\n\\n\".join([*ctx, partial_ctx]) if ctx else partial_ctx\n\n def process_results(self, doc, results):\n \"\"\"Take a single document and the LM results and evaluates, returning a\n dict where keys are the names of submetrics and values are the values of\n the metric for that one document\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param results:\n The results of the requests created in construct_requests.\n \"\"\"\n return {\"acc\": np.argmax(results) == self.answer_to_num[doc[\"answer\"]]}\n\n def aggregation(self):\n \"\"\"\n :returns: {str: [float] -> float}\n A dictionary where keys are the names of submetrics and values are\n functions that aggregate a list of metrics\n \"\"\"\n return {\"acc\": mean}\n\n def higher_is_better(self):\n \"\"\"\n :returns: {str: bool}\n A dictionary where keys are the names of submetrics and values are\n whether a higher value of the submetric is better\n \"\"\"\n return {\"acc\": True}\n" }, { "path": "lm_eval/tasks/wsc273.py", "content": "\"\"\"\nThe Winograd Schema Challenge\nhttp://commonsensereasoning.org/2011/papers/Levesque.pdf\n\nA Winograd schema is a pair of sentences that differ in only one or two words\nand that contain an ambiguity that is resolved in opposite ways in the two\nsentences and requires the use of world knowledge and reasoning for its resolution.\nThe Winograd Schema Challenge 273 is a collection of 273 such Winograd schemas.\n\nNOTE: This evaluation of Winograd Schema Challenge is based on `partial evaluation`\nas described by Trinh & Le in Simple Method for Commonsense Reasoning (2018).\nSee: https://arxiv.org/abs/1806.0\n\nHomepage: https://cs.nyu.edu/~davise/papers/WinogradSchemas/WS.html\n\"\"\"\nimport numpy as np\nfrom lm_eval.base import rf, Task\nfrom lm_eval.metrics import mean\n\n\n_CITATION = \"\"\"\n@inproceedings{ea01b9c0db064caca6986b925d75f2bb,\n title = \"The winograd schema challenge\",\n abstract = \"In this paper, we present an alternative to the Turing Test that has some conceptual and practical advantages. A Wino-grad schema is a pair of sentences that differ only in one or two words and that contain a referential ambiguity that is resolved in opposite directions in the two sentences. We have compiled a collection of Winograd schemas, designed so that the correct answer is obvious to the human reader, but cannot easily be found using selectional restrictions or statistical techniques over text corpora. A contestant in the Winograd Schema Challenge is presented with a collection of one sentence from each pair, and required to achieve human-level accuracy in choosing the correct disambiguation.\",\n author = \"Levesque, {Hector J.} and Ernest Davis and Leora Morgenstern\",\n year = \"2012\",\n language = \"English (US)\",\n isbn = \"9781577355601\",\n series = \"Proceedings of the International Conference on Knowledge Representation and Reasoning\",\n publisher = \"Institute of Electrical and Electronics Engineers Inc.\",\n pages = \"552--561\",\n booktitle = \"13th International Conference on the Principles of Knowledge Representation and Reasoning, KR 2012\",\n note = \"13th International Conference on the Principles of Knowledge Representation and Reasoning, KR 2012 ; Conference date: 10-06-2012 Through 14-06-2012\",\n}\n\"\"\"\n\n\nclass WinogradSchemaChallenge273(Task):\n VERSION = 0\n DATASET_PATH = \"winograd_wsc\"\n DATASET_NAME = \"wsc273\"\n\n upper_pronouns = [\n \"A\",\n \"An\",\n \"The\",\n \"She\",\n \"He\",\n \"It\",\n \"They\",\n \"My\",\n \"His\",\n \"Her\",\n \"Their\",\n ]\n\n def has_training_docs(self):\n return False\n\n def has_validation_docs(self):\n return False\n\n def has_test_docs(self):\n return True\n\n def test_docs(self):\n return map(self._process_doc, self.dataset[\"test\"])\n\n def _process_doc(self, doc):\n # The HF implementation of `wsc273` is not `partial evaluation` friendly.\n doc[\"text\"] = doc[\"text\"].replace(\" \", \" \")\n doc[\"options\"][0] = self.__normalize_option(doc, doc[\"options\"][0])\n doc[\"options\"][1] = self.__normalize_option(doc, doc[\"options\"][1])\n return doc\n\n def __normalize_option(self, doc, option):\n # Append `'s` to possessive determiner based options.\n if doc[\"pronoun\"].lower() in [\"my\", \"his\", \"her\", \"our\", \"their\"]:\n option += \"'s\"\n # Appropriately lowercase the pronoun in the option.\n pronoun = option.split()[0]\n start_of_sentence = doc[\"text\"][doc[\"pronoun_loc\"] - 2] == \".\"\n if not start_of_sentence and pronoun in self.upper_pronouns:\n return option.replace(pronoun, pronoun.lower())\n return option\n\n def fewshot_examples(self, k, rnd):\n # NOTE: `super().fewshot_examples` samples from training docs which are\n # not available for this test-set-only dataset.\n\n if self._fewshot_docs is None:\n self._fewshot_docs = list(self.test_docs())\n\n return rnd.sample(list(self._fewshot_docs), k)\n\n def doc_to_text(self, doc):\n return self.partial_context(doc, doc[\"options\"][doc[\"label\"]])\n\n def should_decontaminate(self):\n return True\n\n def doc_to_decontamination_query(self, doc):\n return doc[\"text\"]\n\n @classmethod\n def partial_context(cls, doc, option):\n # Substitute the pronoun in the original text with the specified\n # option and ignore everything after.\n return doc[\"text\"][: doc[\"pronoun_loc\"]] + option\n\n def doc_to_target(self, doc):\n return self.partial_target(doc)\n\n @classmethod\n def partial_target(cls, doc):\n # The target is everything after the document specified pronoun.\n start_index = doc[\"pronoun_loc\"] + len(doc[\"pronoun\"])\n return \" \" + doc[\"text\"][start_index:].strip()\n\n def construct_requests(self, doc, ctx):\n \"\"\"Uses RequestFactory to construct Requests and returns an iterable of\n Requests which will be sent to the LM.\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param ctx: str\n The context string, generated by fewshot_context. This includes the natural\n language description, as well as the few shot examples, and the question\n part of the document for `doc`.\n \"\"\"\n target = self.partial_target(doc)\n lls = []\n for option in doc[\"options\"]:\n partial_ctx = self.partial_context(doc, option)\n full_ctx = self.append_context(ctx, partial_ctx)\n lls.append(rf.loglikelihood(full_ctx, target)[0])\n return lls\n\n @classmethod\n def append_context(cls, ctx, partial_ctx):\n ctx = ctx.split(\"\\n\\n\") # Each fewshot context is on its own new line.\n ctx.pop() # Remove the correct context put in by `doc_to_text`.\n return \"\\n\\n\".join([*ctx, partial_ctx]) if ctx else partial_ctx\n\n def process_results(self, doc, results):\n \"\"\"Take a single document and the LM results and evaluates, returning a\n dict where keys are the names of submetrics and values are the values of\n the metric for that one document\n\n :param doc:\n The document as returned from training_docs, validation_docs, or test_docs.\n :param results:\n The results of the requests created in construct_requests.\n \"\"\"\n return {\"acc\": np.argmax(results) == doc[\"label\"]}\n\n def aggregation(self):\n \"\"\"\n :returns: {str: [float] -> float}\n A dictionary where keys are the names of submetrics and values are\n functions that aggregate a list of metrics\n \"\"\"\n return {\"acc\": mean}\n\n def higher_is_better(self):\n \"\"\"\n :returns: {str: bool}\n A dictionary where keys are the names of submetrics and values are\n whether a higher value of the submetric is better\n \"\"\"\n return {\"acc\": True}\n" }, { "path": "lm_eval/utils.py", "content": "import os\nimport pathlib\nimport re\nimport collections\nimport functools\nimport inspect\nimport sys\nfrom typing import List\n\nfrom omegaconf import OmegaConf\n\n\nclass ExitCodeError(Exception):\n pass\n\n\ndef sh(x):\n if os.system(x):\n raise ExitCodeError()\n\n\ndef simple_parse_args_string(args_string):\n \"\"\"\n Parses something like\n args1=val1,arg2=val2\n Into a dictionary\n \"\"\"\n args_string = args_string.strip()\n if not args_string:\n return {}\n arg_list = args_string.split(\",\")\n args_dict = OmegaConf.to_object(OmegaConf.from_dotlist(arg_list))\n return args_dict\n\n\ndef join_iters(iters):\n for iter in iters:\n yield from iter\n\n\ndef chunks(iter, n):\n arr = []\n for x in iter:\n arr.append(x)\n if len(arr) == n:\n yield arr\n arr = []\n\n if arr:\n yield arr\n\n\ndef group(arr, fn):\n res = collections.defaultdict(list)\n\n for ob in arr:\n res[fn(ob)].append(ob)\n\n return list(res.values())\n\n\ndef general_detokenize(string):\n string = string.replace(\" n't\", \"n't\")\n string = string.replace(\" )\", \")\")\n string = string.replace(\"( \", \"(\")\n string = string.replace('\" ', '\"')\n string = string.replace(' \"', '\"')\n string = re.sub(r\" (['.,])\", r\"\\1\", string)\n return string\n\n\ndef get_rolling_token_windows(token_list, prefix_token, max_seq_len, context_len):\n \"\"\"\n - context_len allows for a rolling window context, allowing each prediction window to potentially\n condition on some context\n\n :param token_list: list\n List of tokens to be PREDICTED\n :param max_seq_len: int\n max_seq_len of model (or max_seq_len we want to use)\n :param context_len: int\n Amount of desired token context for prediction. Needs to be at least 1.\n :param prefix_token: token\n Dummy token like so the first token has something to condition on\n :return: generator\n Generator of tuples\n (input_tokens, pred_tokens)\n Note: Score only the last len(pred_tokens) logits of the LM\n \"\"\"\n assert 1 <= context_len <= max_seq_len\n if not token_list:\n return\n # +1 offset, going from input->preds\n pred_len = max_seq_len - context_len + 1\n predicted = 0\n\n # Special handling for first window: predict all tokens\n first_seq_len = min(max_seq_len, len(token_list))\n yield ([prefix_token] + token_list[: first_seq_len - 1], token_list[:first_seq_len])\n predicted += first_seq_len\n\n while predicted < len(token_list):\n window_pred_len = min(len(token_list) - predicted, pred_len)\n window_end = predicted + window_pred_len\n\n yield (\n token_list[window_end - max_seq_len - 1 : window_end - 1],\n token_list[window_end - window_pred_len : window_end],\n )\n predicted += window_pred_len\n\n\ndef make_disjoint_window(pair):\n \"\"\"Takes output from get_rolling_token_windows and makes the context not overlap with the continuation\"\"\"\n a, b = pair\n return a[: len(a) - (len(b) - 1)], b\n\n\nclass Reorderer:\n def __init__(self, arr, fn):\n self.size = len(arr)\n arr = list(enumerate(arr))\n arr = group(arr, lambda x: fn(x[1]))\n arr = [([y[0] for y in x], x[0][1]) for x in arr]\n arr.sort(key=lambda x: fn(x[1]))\n\n self.arr = arr\n\n def get_reordered(self):\n return [x[1] for x in self.arr]\n\n def get_original(self, newarr):\n res = [None] * self.size\n cov = [False] * self.size\n\n for (inds, _), v in zip(self.arr, newarr):\n for ind in inds:\n res[ind] = v\n cov[ind] = True\n\n assert all(cov)\n\n return res\n\n\ndef positional_deprecated(fn):\n \"\"\"\n A decorator to nudge users into passing only keyword args (`kwargs`) to the\n wrapped function, `fn`.\n \"\"\"\n\n @functools.wraps(fn)\n def _wrapper(*args, **kwargs):\n if len(args) != 1 if inspect.ismethod(fn) else 0:\n print(\n f\"WARNING: using {fn.__name__} with positional arguments is \"\n \"deprecated and will be disallowed in a future version of \"\n \"lm-evaluation-harness!\"\n )\n return fn(*args, **kwargs)\n\n return _wrapper\n\n\n@positional_deprecated\ndef find_test_root(start_path: pathlib.Path) -> pathlib.Path:\n \"\"\"\n Search upward in the directory tree to a maximum of three layers\n to find and return the package root (containing the 'tests' folder)\n \"\"\"\n cur_path = start_path.resolve()\n max_layers = 3\n for _ in range(max_layers):\n if (cur_path / \"tests\" / \"test_version_stable.py\").exists():\n return cur_path\n else:\n cur_path = cur_path.parent.resolve()\n raise FileNotFoundError(\n f\"Unable to find package root within {max_layers} upwards\" + f\"of {start_path}\"\n )\n\n\n@positional_deprecated\ndef run_task_tests(task_list: List[str]):\n \"\"\"\n Find the package root and run the tests for the given tasks\n \"\"\"\n import pytest\n\n package_root = find_test_root(start_path=pathlib.Path(__file__))\n task_string = \" or \".join(task_list)\n args = [\n f\"{package_root}/tests/test_version_stable.py\",\n f\"--rootdir={package_root}\",\n \"-k\",\n f\"{task_string}\",\n ]\n sys.path.append(str(package_root))\n pytest_return_val = pytest.main(args)\n if pytest_return_val:\n raise ValueError(\n f\"Not all tests for the specified tasks ({task_list}) ran successfully! Error code: {pytest_return_val}\"\n )\n" }, { "path": "main.py", "content": "import os\nimport sys\nimport random\nimport numpy as np\nfrom models.LMClass import LMClass\nfrom models.IRQLoRALMClass import IRQLoRALMClass\nimport torch\nimport time\nfrom datautils import get_loaders\nfrom lm_eval import evaluator\nfrom pprint import pprint\nfrom parallel_utils import map_layers_to_multi_gpus, get_lowest_occupied_gpu\nimport torch.nn as nn\nfrom quant.omniquant import omniquant\nfrom tqdm import tqdm\nimport utils\nfrom pathlib import Path\nfrom categories import subcategories, categories\n\nfrom models.int_llama_layer import QuantLlamaDecoderLayer\nfrom models.int_opt_layer import QuantOPTDecoderLayer\nfrom quant.int_linear import QuantLinear\n\nimport pdb\n\n\ntorch.backends.cudnn.benchmark = True\n\nnet_choices = [\n \"opt-125m\",\n \"opt-1.3b\",\n \"opt-2.7b\",\n \"opt-6.7b\",\n \"opt-13b\",\n \"opt-30b\",\n \"opt-66b\",\n \"llama-7b\",\n \"llama-13b\",\n \"llama-30b\",\n \"llama-65b\",\n \"Llama-2-7b\",\n \"Llama-2-13b\",\n \"Llama-2-70b\",\n \"Llama-2-7b-chat\",\n \"Llama-2-13b-chat\",\n \"llava-llama-2-13b-chat-lightning-preview\",\n \"falcon-180b\",\n \"falcon-7b\",\n \"mixtral-8x7b\"\n]\n\n\n@torch.no_grad()\ndef evaluate(lm, args, logger):\n results = {}\n if args.multigpu:\n if \"opt\" in args.net.lower():\n map_layers_to_multi_gpus(lm.model.model.decoder.layers)\n input_device = lm.model.model.decoder.layers[0].device\n output_device = lm.model.model.decoder.layers[-1].device\n lm._device = input_device\n assert input_device == output_device\n lm.model.model.decoder.embed_positions.to(input_device)\n lm.model.model.decoder.embed_tokens.to(input_device)\n lm.model.model.decoder.final_layer_norm.to(output_device)\n lm.model.lm_head.to(output_device)\n\n elif \"llama\" in args.net.lower() or \"mixtral\" in args.net.lower():\n map_layers_to_multi_gpus(lm.model.model.layers)\n input_device = lm.model.model.layers[0].device\n output_device = lm.model.model.layers[-1].device\n assert input_device == output_device\n lm._device = input_device\n lm.model.model.embed_tokens.to(input_device)\n lm.model.model.norm.to(output_device)\n lm.model.lm_head.to(output_device)\n elif \"falcon\" in args.net.lower():\n map_layers_to_multi_gpus(lm.model.transformer.h)\n input_device = lm.model.transformer.h[0].device\n output_device = lm.model.transformer.h[-1].device\n assert input_device == output_device\n lm._device = input_device\n lm.model.transformer.word_embeddings.to(input_device)\n lm.model.transformer.ln_f.to(output_device)\n lm.model.lm_head.to(output_device)\n else:\n if \"opt\" in args.net.lower():\n lm.model.model.decoder = lm.model.model.decoder.to(lm.device)\n elif \"llama\" in args.net.lower() or \"mixtral\" in args.net.lower():\n lm.model = lm.model.to(lm.device)\n elif \"falcon\" in args.net.lower():\n lm.model.transformer = lm.model.transformer.to(lm.device)\n\n\n if args.eval_ppl:\n # for dataset in [\"wikitext2\", \"ptb\", \"c4\",\"ptb-new\",'c4-new']:\n for dataset in [\"wikitext2\", \"c4\"]:\n cache_testloader = f'{args.cache_dir}/testloader_{args.model_family}_{dataset}_all.cache'\n if os.path.exists(cache_testloader):\n testloader = torch.load(cache_testloader)\n logger.info(f\"load calibration from {cache_testloader}\")\n else:\n dataloader, testloader = get_loaders(\n dataset,\n seed=args.seed,\n model=args.model,\n seqlen=lm.seqlen,\n )\n torch.save(testloader, cache_testloader)\n if \"c4\" in dataset:\n testenc = testloader\n else:\n testenc = testloader.input_ids\n\n nsamples = testenc.numel() // lm.seqlen\n use_cache = lm.model.config.use_cache\n lm.model.config.use_cache = False\n lm.model.eval()\n nlls = []\n for i in tqdm(range(nsamples)):\n batch = testenc[:, (i * lm.seqlen) : ((i + 1) * lm.seqlen)].to(lm.device)\n if \"opt\" in args.net.lower():\n outputs = lm.model.model.decoder(batch)\n elif \"llama\" in args.net.lower() or \"mixtral\" in args.net.lower():\n outputs = lm.model.model(batch)\n elif \"falcon\" in args.model:\n outputs = lm.model.transformer(batch)\n hidden_states = outputs[0]\n logits = lm.model.lm_head(hidden_states)\n shift_logits = logits[:, :-1, :]\n shift_labels = testenc[:, (i * lm.seqlen) : ((i + 1) * lm.seqlen)][\n :, 1:\n ].to(lm.model.lm_head.weight.device)\n loss_fct = nn.CrossEntropyLoss()\n loss = loss_fct(\n shift_logits.view(-1, shift_logits.size(-1)),\n shift_labels.view(-1),\n )\n neg_log_likelihood = loss.float() * lm.seqlen\n nlls.append(neg_log_likelihood)\n if i == args.limit:\n break\n\n ppl = torch.exp(torch.stack(nlls).sum() / (nsamples * lm.seqlen))\n logger.info(f'{dataset} : {ppl.item()}')\n lm.model.config.use_cache = use_cache\n results[dataset] = ppl.item()\n if args.tasks != \"\":\n t_results = evaluator.simple_evaluate(\n lm,\n tasks=args.tasks,\n num_fewshot=args.num_fewshot,\n limit=None if args.limit == -1 else args.limit,\n )\n results.update(t_results)\n logger.info(results)\n pprint(results)\n # for test of MMLU\n if 'hendrycksTest' in args.tasks:\n all_cors = []\n all_cors_norm = []\n subcat_cors = {subcat: [] for subcat_lists in subcategories.values() for subcat in subcat_lists}\n cat_cors = {cat: [] for cat in categories}\n cat_cors_norm = {cat: [] for cat in categories}\n for key in t_results['results'].keys():\n if not 'hendrycksTest' in key:\n continue\n subject = key.split('-')[-1]\n cors = t_results['results'][key]['acc']\n cors_norm = t_results['results'][key]['acc_norm']\n subcats = subcategories[subject]\n for subcat in subcats:\n subcat_cors[subcat].append(cors)\n for key in categories.keys():\n if subcat in categories[key]:\n cat_cors[key].append(cors)\n cat_cors_norm[key].append(cors_norm)\n all_cors.append(cors)\n all_cors_norm.append(cors_norm)\n \n for cat in cat_cors:\n cat_acc = np.mean(cat_cors[cat])\n logger.info(\"Average accuracy {:.4f} - {}\".format(cat_acc, cat))\n weighted_acc = np.mean(all_cors)\n logger.info(\"Average accuracy: {:.4f}\".format(weighted_acc)) \n return results\n\n\ndef main():\n import argparse\n\n parser = argparse.ArgumentParser()\n parser.add_argument(\"--model\", type=str, help=\"model name of model path\")\n parser.add_argument(\"--cache_dir\", default=\"./cache\", type=str, help=\"cache dir of dataset, leading to faster debug\")\n parser.add_argument(\"--output_dir\", default=\"../log/\", type=str, help=\"direction of logging file\")\n parser.add_argument(\"--save_dir\", default=None, type=str, help=\"direction for saving fake quantization model\")\n parser.add_argument(\"--resume\", type=str, default=None)\n parser.add_argument(\"--real_quant\", default=False, action=\"store_true\", help=\"real quantization, which can see memory reduce. Note that due to the limitations of AutoGPTQ kernels, the real quantization of weight-only quantization can only lead memory reduction, but with slower inference speed.\")\n parser.add_argument(\"--calib_dataset\",type=str,default=\"wikitext2\",\n choices=[\"wikitext2\", \"ptb\", \"c4\", \"mix\",\"pile\"],\n help=\"Where to extract calibration data from.\",\n )\n parser.add_argument(\"--nsamples\", type=int, default=128, help=\"Number of calibration data samples.\")\n parser.add_argument(\"--batch_size\", type=int, default=1, help=\"batch size.\")\n parser.add_argument(\"--seed\", type=int, default=2, help=\"Seed for sampling the calibration data.\")\n parser.add_argument(\"--tasks\", default=\"\")\n parser.add_argument(\"--eval_ppl\", action=\"store_true\")\n parser.add_argument(\"--num_fewshot\", type=int, default=0)\n parser.add_argument(\"--wbits\", type=int, default=4)\n parser.add_argument(\"--abits\", type=int, default=16)\n parser.add_argument(\"--group_size\", type=int, default=None)\n parser.add_argument(\"--alpha\", type=float, default=0.5)\n parser.add_argument(\"--let_lr\", type=float, default=5e-3)\n parser.add_argument(\"--lwc_lr\", type=float, default=1e-2)\n parser.add_argument(\"--wd\", type=float, default=0)\n parser.add_argument(\"--epochs\", type=int, default=10)\n parser.add_argument(\"--let\",default=False, action=\"store_true\",help=\"activate learnable equivalent transformation\")\n parser.add_argument(\"--lwc\",default=False, action=\"store_true\",help=\"activate learnable weight clipping\")\n parser.add_argument(\"--aug_loss\", default=False, action=\"store_true\", help=\"calculate additional loss with same input\")\n parser.add_argument(\"--symmetric\",default=False, action=\"store_true\", help=\"symmetric quantization\")\n parser.add_argument(\"--disable_zero_point\",default=False, action=\"store_true\", help=\"quantization without zero_point\")\n parser.add_argument(\"--a_dynamic_method\", type=str, default=\"per_token\", choices=[\"per_token\"])\n parser.add_argument(\"--w_dynamic_method\", type=str, default=\"per_channel\", choices=[\"per_channel\"])\n parser.add_argument(\"--limit\", type=int, default=-1)\n parser.add_argument(\"--multigpu\", action=\"store_true\", help=\"at eval, map model to multiple gpus\")\n parser.add_argument(\"--deactive_amp\", action=\"store_true\", help=\"deactivate AMP when 8<=bits<16\")\n parser.add_argument(\n \"--attn_implementation\",\n type=str, required=False, default=\"eager\",\n choices=[\"eager\", \"sdpa\", \"flash_attention_2\"],\n help=\"attention implementation that the model works with\",\n )\n parser.add_argument(\"--net\", type=str, default=None, choices=net_choices)\n parser.add_argument(\"--act-scales\", type=str, default=None)\n parser.add_argument(\"--act-shifts\", type=str, default=None)\n\n parser.add_argument(\"--quant_method\", type=str, default='irqlora')\n parser.add_argument(\"--peft\", type=str, default='./')\n parser.add_argument(\"--tau_range\", type=float, default=0.1)\n parser.add_argument(\"--tau_n\", type=int, default=100)\n parser.add_argument(\"--blocksize2\", type=int, default=256)\n\n args = parser.parse_args()\n random.seed(args.seed)\n np.random.seed(args.seed)\n torch.manual_seed(args.seed)\n torch.cuda.manual_seed(args.seed)\n\n # check\n if args.epochs > 0:\n assert args.lwc or args.let\n \n if (args.wbits<16 and args.wbits>=8) or (args.abits<16 and args.abits>=8):\n args.deactive_amp = True\n\n # init logger\n if args.output_dir:\n Path(args.output_dir).mkdir(parents=True, exist_ok=True)\n if args.cache_dir:\n Path(args.cache_dir).mkdir(parents=True, exist_ok=True)\n if args.save_dir:\n Path(args.save_dir).mkdir(parents=True, exist_ok=True)\n output_dir = Path(args.output_dir)\n logger = utils.create_logger(output_dir)\n logger.info(args)\n \n # load model\n if args.net is None:\n args.net = args.model.split('/')[-1]\n # assert args.net in net_choices\n args.model_family = args.net.split('-')[0]\n if args.quant_method in ['irqlora', 'qlora']:\n lm = IRQLoRALMClass(args)\n else:\n lm = LMClass(args)\n lm.seqlen = 2048\n lm.model.eval()\n for param in lm.model.parameters():\n param.requires_grad = False\n\n \n\n args.weight_quant_params = {\n \"n_bits\": args.wbits,\n \"per_channel_axes\": [0],\n \"symmetric\": args.symmetric,\n \"dynamic_method\": args.w_dynamic_method,\n \"group_size\": args.group_size,\n \"lwc\":args.lwc,\n \"disable_zero_point\": args.disable_zero_point\n }\n args.act_quant_params = {\n \"n_bits\": args.abits,\n \"per_channel_axes\": [],\n \"symmetric\": False,\n \"dynamic_method\": args.a_dynamic_method,\n }\n args.q_quant_params = {\n \"n_bits\": args.abits,\n \"per_channel_axes\": [],\n \"symmetric\": False,\n \"dynamic_method\": args.a_dynamic_method,\n }\n args.k_quant_params = {\n \"n_bits\": args.abits,\n \"per_channel_axes\": [],\n \"symmetric\": False,\n \"dynamic_method\": args.a_dynamic_method,\n }\n args.v_quant_params = {\n \"n_bits\": args.abits,\n \"per_channel_axes\": [],\n \"symmetric\": False,\n \"dynamic_method\": args.a_dynamic_method,\n }\n args.p_quant_params = {\n \"n_bits\": 16,\n \"metric\": \"fix0to1\",\n }\n\n if args.multigpu:\n gpu_id = get_lowest_occupied_gpu(wait_memory=5000)\n lm._device = f\"cuda:{gpu_id}\"\n logger.info(f\"set quantization in gpu {gpu_id}\")\n\n # act scales and shifts\n if args.act_scales is None:\n args.act_scales = f'./act_scales/{args.net}.pt'\n if args.act_shifts is None:\n args.act_shifts = f'./act_shifts/{args.net}.pt'\n\n # quantization\n if (args.wbits < 16 or args.abits < 16) and (args.epochs > 0):\n logger.info(\"=== start quantization ===\")\n tick = time.time() \n # load calibration dataset\n cache_dataloader = f'{args.cache_dir}/dataloader_{args.model_family}_{args.calib_dataset}_{args.nsamples}.cache'\n if os.path.exists(cache_dataloader):\n dataloader = torch.load(cache_dataloader)\n logger.info(f\"load calibration from {cache_dataloader}\")\n else:\n dataloader, _ = get_loaders(\n args.calib_dataset,\n nsamples=args.nsamples,\n seed=args.seed,\n model=args.model,\n seqlen=lm.seqlen,\n )\n torch.save(dataloader, cache_dataloader) \n act_scales = None\n act_shifts = None\n if args.let:\n act_scales = torch.load(args.act_scales)\n act_shifts = torch.load(args.act_shifts)\n omniquant(\n lm,\n args,\n dataloader,\n act_scales,\n act_shifts,\n logger,\n )\n logger.info(time.time() - tick)\n \n if args.save_dir:\n # delete omni parameters\n for name, module in lm.model.named_modules():\n if isinstance(module, QuantLinear):\n del module.weight_quantizer.lowbound_factor\n del module.weight_quantizer.upbound_factor\n if isinstance(module,QuantLlamaDecoderLayer) or isinstance(module,QuantOPTDecoderLayer):\n if args.let:\n del module.qkv_smooth_scale\n del module.qkv_smooth_shift\n del module.out_smooth_scale\n del module.out_smooth_shift\n del module.fc1_smooth_scale\n del module.fc1_smooth_shift \n lm.model.save_pretrained(args.save_dir) \n lm.tokenizer.save_pretrained(args.save_dir) \n evaluate(lm, args,logger)\n\n\nif __name__ == \"__main__\":\n print(sys.argv)\n main()\n" }, { "path": "models/IRQLoRALMClass.py", "content": "import transformers\nimport torch\nfrom .models_utils import BaseLM, find_layers\nimport os\nimport torch\nimport torch.nn.functional as F\nfrom transformers import (\n AutoConfig,\n AutoTokenizer,\n AutoModelForCausalLM,\n BitsAndBytesConfig,\n)\nfrom peft import (\n prepare_model_for_kbit_training,\n LoraConfig,\n get_peft_model,\n PeftModel\n)\nfrom models.LMClass import LMClass\nfrom safetensors.torch import load_file\nfrom irqlora import replace_to_qlora_model\n\nclass IRQLoRALMClass(BaseLM):\n def __init__(self, args):\n super().__init__()\n \n self.args = args\n self._device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n self.model_name = args.model\n self.batch_size_per_gpu = args.batch_size\n\n self.tokenizer = AutoTokenizer.from_pretrained(args.model, use_fast=False,legacy=False)\n self.model = AutoModelForCausalLM.from_pretrained(\n args.model,\n cache_dir=args.cache_dir,\n device_map='auto',\n quantization_config=BitsAndBytesConfig(\n load_in_4bit=True,\n llm_int8_threshold=6.0,\n llm_int8_has_fp16_weight=False,\n bnb_4bit_compute_dtype=torch.bfloat16,\n bnb_4bit_use_double_quant=True,\n bnb_4bit_quant_type='nf4',\n ),\n torch_dtype=torch.bfloat16\n )\n self.model = PeftModel.from_pretrained(self.model, os.path.join(args.peft, 'adapter_model'), is_trainable=True)\n model_fp = AutoModelForCausalLM.from_pretrained(args.model)\n\n self.seqlen = self.model.config.max_position_embeddings\n self.model.eval()\n self.vocab_size = self.tokenizer.vocab_size\n print(\"vocab size: \", self.vocab_size)\n\n \n self.model = replace_to_qlora_model(self.model, model_fp, blocksize2=args.blocksize2, tau_range=args.tau_range, tau_n=args.tau_n)\n checkpoint = load_file(os.path.join(args.peft, 'adapter_model/adapter_model.safetensors'), device='cuda')\n checkpoint = {key: value for key, value in checkpoint.items() if \"lora_default_A_scale\" in key or \"lora_default_B_scale\" in key}\n self.model.load_state_dict(checkpoint, strict=False)\n\n @property\n def eot_token(self) -> str:\n return self.tokenizer.eos_token\n\n @property\n def eot_token_id(self):\n # we use EOT because end of *text* is more accurate for what we're doing than end of *sentence*\n return self.tokenizer.eos_token_id\n\n @property\n def max_length(self):\n try:\n return self.gpt2.config.n_ctx\n except AttributeError:\n # gptneoconfig doesn't have n_ctx apparently\n return self.model.config.max_position_embeddings\n\n @property\n def max_gen_toks(self):\n print(\"max_gen_toks fn\")\n return 256\n\n @property\n def batch_size(self):\n # TODO: fix multi-gpu\n return self.batch_size_per_gpu # * gpus\n\n @property\n def device(self):\n # TODO: fix multi-gpu\n return self._device\n\n def tok_encode(self, string: str):\n return self.tokenizer.encode(string, add_special_tokens=False)\n\n def tok_encode_batch(self, strings):\n return self.tokenizer(\n strings,\n padding=True,\n add_special_tokens=False,\n return_tensors=\"pt\",\n )\n\n def tok_decode(self, tokens):\n return self.tokenizer.batch_decode(tokens, skip_special_tokens=True)\n\n def _model_call(self, inps):\n \"\"\"\n inps: a torch tensor of shape [batch, sequence]\n the size of sequence may vary from call to call\n returns: a torch tensor of shape [batch, sequence, vocab] with the\n logits returned from the model\n \"\"\"\n with torch.no_grad():\n\n return self.model(inps)[\"logits\"]\n\n def model_batched_set(self, inps):\n dataset_logits = []\n for batch in inps:\n multi_logits = F.log_softmax(\n self._model_call(batch), dim=-1\n ).cpu() # [batch, padding_length, vocab]\n dataset_logits.append(multi_logits)\n return dataset_logits\n\n def _model_generate(self, context, max_length, eos_token_id):\n return self.model.generate(\n context, max_length=max_length, eos_token_id=eos_token_id, do_sample=False\n )\n" }, { "path": "models/LMClass.py", "content": "import transformers\nimport torch\nfrom .models_utils import BaseLM, find_layers\nfrom transformers import AutoTokenizer, AutoConfig, AutoModelForCausalLM\nimport torch.nn.functional as F\nfrom torch import nn\nimport torch\nfrom tqdm import tqdm\nimport pdb\n\n\nclass LMClass(BaseLM):\n def __init__(self, args):\n\n super().__init__()\n\n self.args = args\n self._device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n self.model_name = args.model\n self.batch_size_per_gpu = args.batch_size\n\n self.model_config = args.model\n config = AutoConfig.from_pretrained(\n args.model, attn_implementation=args.attn_implementation\n )\n\n self.tokenizer = AutoTokenizer.from_pretrained(args.model, use_fast=False,legacy=False)\n # self.model = AutoModelForCausalLM.from_pretrained(args.model, config=config, device_map='cpu',torch_dtype=config.torch_dtype)\n self.model = AutoModelForCausalLM.from_pretrained(args.model, config=config, device_map='cpu',torch_dtype=torch.float16)\n self.seqlen = self.model.config.max_position_embeddings\n self.model.eval()\n self.vocab_size = self.tokenizer.vocab_size\n print(\"vocab size: \", self.vocab_size)\n\n @property\n def eot_token(self) -> str:\n return self.tokenizer.eos_token\n\n @property\n def eot_token_id(self):\n # we use EOT because end of *text* is more accurate for what we're doing than end of *sentence*\n return self.tokenizer.eos_token_id\n\n @property\n def max_length(self):\n try:\n return self.gpt2.config.n_ctx\n except AttributeError:\n # gptneoconfig doesn't have n_ctx apparently\n return self.model.config.max_position_embeddings\n\n @property\n def max_gen_toks(self):\n print(\"max_gen_toks fn\")\n return 256\n\n @property\n def batch_size(self):\n # TODO: fix multi-gpu\n return self.batch_size_per_gpu # * gpus\n\n @property\n def device(self):\n # TODO: fix multi-gpu\n return self._device\n\n def tok_encode(self, string: str):\n return self.tokenizer.encode(string, add_special_tokens=False)\n\n def tok_encode_batch(self, strings):\n return self.tokenizer(\n strings,\n padding=True,\n add_special_tokens=False,\n return_tensors=\"pt\",\n )\n\n def tok_decode(self, tokens):\n return self.tokenizer.batch_decode(tokens, skip_special_tokens=True)\n\n def _model_call(self, inps):\n \"\"\"\n inps: a torch tensor of shape [batch, sequence]\n the size of sequence may vary from call to call\n returns: a torch tensor of shape [batch, sequence, vocab] with the\n logits returned from the model\n \"\"\"\n with torch.no_grad():\n\n return self.model(inps)[\"logits\"]\n\n def model_batched_set(self, inps):\n dataset_logits = []\n for batch in inps:\n multi_logits = F.log_softmax(\n self._model_call(batch), dim=-1\n ).cpu() # [batch, padding_length, vocab]\n dataset_logits.append(multi_logits)\n return dataset_logits\n\n def _model_generate(self, context, max_length, eos_token_id):\n return self.model.generate(\n context, max_length=max_length, eos_token_id=eos_token_id, do_sample=False\n )\n" }, { "path": "models/int_falcon_layer.py", "content": "import torch\r\nfrom torch import nn\r\nfrom typing import Optional, Tuple, List\r\nfrom quant.int_linear import QuantLinear\r\nfrom quant.int_matmul import QuantMatMul\r\nimport torch.nn.functional as F\r\nfrom collections import OrderedDict\r\nimport math\r\nfrom transformers.models.falcon.configuration_falcon import FalconConfig\r\nfrom transformers.models.falcon.modeling_falcon import FalconAttention, dropout_add\r\nimport pdb\r\nimport copy\r\nfrom models.transformation import *\r\nfrom quant.omni_norm import OmniLayerNorm\r\n\r\n\r\n\r\n\r\n \r\nclass QuantFalconMLP(nn.Module):\r\n def __init__(self, org_module: nn.Module,args=None):\r\n super().__init__()\r\n\r\n self.dense_h_to_4h = QuantLinear(org_module.dense_h_to_4h,args.weight_quant_params,args.act_quant_params)\r\n self.act = nn.GELU()\r\n self.dense_4h_to_h = QuantLinear(org_module.dense_4h_to_h,args.weight_quant_params,args.act_quant_params)\r\n\r\n def forward(self, x: torch.Tensor) -> torch.Tensor:\r\n x = self.act(self.dense_h_to_4h(x))\r\n x = self.dense_4h_to_h(x)\r\n return x\r\n\r\n\r\n \r\nclass QuantFalconAttention(nn.Module):\r\n def __init__(self, config: FalconConfig, org_module: nn.Module, args=None):\r\n super().__init__()\r\n\r\n self.config = config\r\n self.hidden_size = config.hidden_size\r\n self.num_heads = config.num_attention_heads\r\n self.head_dim = self.hidden_size // self.num_heads\r\n self.split_size = self.hidden_size\r\n self.hidden_dropout = config.hidden_dropout\r\n\r\n if self.head_dim * self.num_heads != self.hidden_size:\r\n raise ValueError(\r\n f\"`hidden_size` must be divisible by num_heads (got `hidden_size`: {self.hidden_size} and `num_heads`:\"\r\n f\" {self.num_heads}).\"\r\n )\r\n\r\n self.maybe_rotary = copy.deepcopy(org_module.maybe_rotary)\r\n\r\n # Layer-wise attention scaling\r\n self.inv_norm_factor = 1.0 / math.sqrt(self.head_dim)\r\n self.beta = self.inv_norm_factor\r\n self.query_key_value = QuantLinear(org_module.query_key_value,args.weight_quant_params,args.act_quant_params)\r\n self.new_decoder_architecture = config.new_decoder_architecture\r\n self.multi_query = config.multi_query\r\n self.dense =QuantLinear(org_module.dense,args.weight_quant_params,args.act_quant_params)\r\n self.attention_dropout = nn.Dropout(config.attention_dropout)\r\n self.num_kv_heads = config.num_kv_heads if (self.new_decoder_architecture or not self.multi_query) else 1\r\n\r\n\r\n def _split_heads(self, fused_qkv: torch.Tensor) -> Tuple[torch.Tensor, torch.Tensor, torch.Tensor]:\r\n \"\"\"\r\n Split the last dimension into (num_heads, head_dim), results share same memory storage as `fused_qkv`\r\n\r\n Args:\r\n fused_qkv (`torch.tensor`, *required*): [batch_size, seq_length, num_heads * 3 * head_dim]\r\n\r\n Returns:\r\n query: [batch_size, seq_length, num_heads, head_dim] key: [batch_size, seq_length, num_heads, head_dim]\r\n value: [batch_size, seq_length, num_heads, head_dim]\r\n \"\"\"\r\n if self.new_decoder_architecture:\r\n batch, seq_len, _ = fused_qkv.shape\r\n qkv = fused_qkv.view(batch, seq_len, -1, self.num_heads // self.num_kv_heads + 2, self.head_dim)\r\n query = qkv[:, :, :, :-2]\r\n key = qkv[:, :, :, [-2]]\r\n value = qkv[:, :, :, [-1]]\r\n key = torch.broadcast_to(key, query.shape)\r\n value = torch.broadcast_to(value, query.shape)\r\n\r\n query, key, value = [x.flatten(2, 3) for x in (query, key, value)]\r\n return query, key, value\r\n elif not self.multi_query:\r\n batch_size, seq_length, three_times_hidden_size = fused_qkv.shape\r\n fused_qkv = fused_qkv.view(batch_size, seq_length, self.num_heads, 3, self.head_dim)\r\n return fused_qkv[..., 0, :], fused_qkv[..., 1, :], fused_qkv[..., 2, :]\r\n else:\r\n batch_size, seq_length, three_times_hidden_size = fused_qkv.shape\r\n fused_qkv = fused_qkv.view(batch_size, seq_length, self.num_heads + 2, self.head_dim)\r\n return fused_qkv[..., :-2, :], fused_qkv[..., [-2], :], fused_qkv[..., [-1], :]\r\n\r\n # Copied from transformers.models.bloom.modeling_bloom.BloomAttention._merge_heads\r\n def _merge_heads(self, x: torch.Tensor) -> torch.Tensor:\r\n \"\"\"\r\n Merge heads together over the last dimension\r\n\r\n Args:\r\n x (`torch.tensor`, *required*): [batch_size * num_heads, seq_length, head_dim]\r\n\r\n Returns:\r\n torch.tensor: [batch_size, seq_length, num_heads * head_dim]\r\n \"\"\"\r\n # What we want to achieve is:\r\n # batch_size * num_heads, seq_length, head_dim -> batch_size, seq_length, num_heads * head_dim\r\n batch_size_and_num_heads, seq_length, _ = x.shape\r\n batch_size = batch_size_and_num_heads // self.num_heads\r\n\r\n # First view to decompose the batch size\r\n # batch_size * num_heads, seq_length, head_dim -> batch_size, num_heads, seq_length, head_dim\r\n x = x.view(batch_size, self.num_heads, seq_length, self.head_dim)\r\n\r\n # batch_size, num_heads, seq_length, head_dim -> batch_size, seq_length, num_heads, head_dim\r\n x = x.permute(0, 2, 1, 3)\r\n\r\n # batch_size, seq_length, num_heads, head_dim -> batch_size, seq_length, num_heads * head_dim\r\n return x.reshape(batch_size, seq_length, self.num_heads * self.head_dim)\r\n\r\n def forward(\r\n self,\r\n hidden_states: torch.Tensor,\r\n alibi: Optional[torch.Tensor],\r\n attention_mask: torch.Tensor,\r\n layer_past: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,\r\n head_mask: Optional[torch.Tensor] = None,\r\n use_cache: bool = False,\r\n output_attentions: bool = False,\r\n ):\r\n fused_qkv = self.query_key_value(hidden_states) # [batch_size, seq_length, 3 x hidden_size]\r\n num_kv_heads = self.num_heads if self.new_decoder_architecture else self.num_kv_heads\r\n # 3 x [batch_size, seq_length, num_heads, head_dim]\r\n (query_layer, key_layer, value_layer) = self._split_heads(fused_qkv)\r\n\r\n batch_size, query_length, _, _ = query_layer.shape\r\n\r\n query_layer = query_layer.transpose(1, 2).reshape(batch_size * self.num_heads, query_length, self.head_dim)\r\n key_layer = key_layer.transpose(1, 2).reshape(\r\n batch_size * num_kv_heads,\r\n query_length,\r\n self.head_dim,\r\n )\r\n value_layer = value_layer.transpose(1, 2).reshape(batch_size * num_kv_heads, query_length, self.head_dim)\r\n\r\n past_kv_length = 0 if layer_past is None else layer_past[0].shape[1]\r\n query_layer, key_layer = self.maybe_rotary(query_layer, key_layer, past_kv_length)\r\n\r\n if layer_past is not None:\r\n past_key, past_value = layer_past\r\n # concatenate along seq_length dimension:\r\n # - key: [batch_size * self.num_heads, kv_length, head_dim]\r\n # - value: [batch_size * self.num_heads, kv_length, head_dim]\r\n key_layer = torch.cat((past_key, key_layer), dim=1)\r\n value_layer = torch.cat((past_value, value_layer), dim=1)\r\n\r\n _, kv_length, _ = key_layer.shape\r\n if use_cache:\r\n present = (key_layer, value_layer)\r\n else:\r\n present = None\r\n\r\n attention_mask_float = (attention_mask * 1.0).masked_fill(attention_mask, float(\"-1e9\")).to(query_layer.dtype)\r\n\r\n query_layer_ = query_layer.reshape(batch_size, self.num_heads, -1, self.head_dim)\r\n key_layer_ = key_layer.reshape(batch_size, num_kv_heads, -1, self.head_dim)\r\n value_layer_ = value_layer.reshape(batch_size, num_kv_heads, -1, self.head_dim)\r\n\r\n if alibi is None:\r\n if hasattr(F, \"scaled_dot_product_attention\") and not output_attentions:\r\n\r\n attn_output = F.scaled_dot_product_attention(\r\n query_layer_, key_layer_, value_layer_, attention_mask_float, 0.0, is_causal=False\r\n )\r\n attention_scores = None\r\n else:\r\n attention_scores = query_layer_ @ key_layer_.transpose(-1, -2)\r\n attention_scores /= math.sqrt(self.head_dim)\r\n\r\n attention_scores = F.softmax(\r\n attention_scores + attention_mask_float, dim=-1, dtype=hidden_states.dtype\r\n )\r\n attn_output = attention_scores @ value_layer_\r\n\r\n attn_output = attn_output.view(batch_size, self.num_heads, query_length, self.head_dim)\r\n attn_output = attn_output.permute(0, 2, 1, 3)\r\n attn_output = attn_output.reshape(batch_size, query_length, self.num_heads * self.head_dim)\r\n\r\n output_tensor = self.dense(attn_output)\r\n\r\n if output_attentions:\r\n return output_tensor, present, attention_scores\r\n else:\r\n return output_tensor, present\r\n\r\n else:\r\n matmul_result = query_layer_ @ key_layer_.transpose(-1, -2)\r\n\r\n # change view to [batch_size, num_heads, q_length, kv_length]\r\n attention_scores = matmul_result.view(batch_size, self.num_heads, query_length, kv_length)\r\n\r\n # cast attention scores to fp32, compute scaled softmax and cast back to initial dtype - [batch_size, num_heads, q_length, kv_length]\r\n input_dtype = attention_scores.dtype\r\n # `float16` has a minimum value of -65504.0, whereas `bfloat16` and `float32` have a minimum value of `-3.4e+38`\r\n if input_dtype == torch.float16 or input_dtype == torch.bfloat16:\r\n attention_scores = attention_scores.to(torch.float32)\r\n # Matt (HF) note: We could possibly use F.scaled_dot_product_attention here too, by\r\n # adding (alibi * self.inv_norm_factor) to attention_mask_float. I think this would be mathematically\r\n # equivalent and more performant, but there might be a numerical difference. If you're reading this\r\n # and you'd like to experiment and maybe file a PR, feel free!\r\n attention_logits = attention_scores + alibi.view(batch_size, self.num_heads, 1, -1)\r\n attention_logits *= self.inv_norm_factor\r\n attention_probs = F.softmax(attention_logits + attention_mask_float, dim=-1, dtype=hidden_states.dtype)\r\n # [batch_size, num_heads, q_length, kv_length]\r\n attention_probs = self.attention_dropout(attention_probs)\r\n\r\n if head_mask is not None:\r\n attention_probs = attention_probs * head_mask\r\n\r\n # change view [batch_size, num_heads, q_length, kv_length]\r\n attention_probs_reshaped = attention_probs.view(batch_size, self.num_heads, query_length, kv_length)\r\n\r\n # matmul: [batch_size * num_heads, q_length, head_dim]\r\n context_layer = (attention_probs_reshaped @ value_layer_).flatten(0, 1)\r\n\r\n # change view [batch_size, q_length, num_heads * head_dim]\r\n context_layer = self._merge_heads(context_layer)\r\n\r\n output_tensor = self.dense(context_layer)\r\n\r\n if output_attentions:\r\n return output_tensor, present, attention_probs\r\n else:\r\n return output_tensor, present\r\n\r\n\r\n \r\nclass QuantFalconDecoderLayer(nn.Module):\r\n def __init__(self, config: FalconConfig,\r\n ori_layer,\r\n args):\r\n super().__init__()\r\n hidden_size = config.hidden_size\r\n self.num_heads = config.num_attention_heads\r\n self.self_attention = QuantFalconAttention(config, ori_layer.self_attention, args)\r\n self.mlp = QuantFalconMLP(ori_layer.mlp, args)\r\n self.hidden_dropout = config.hidden_dropout\r\n self.config = config\r\n\r\n if config.new_decoder_architecture:\r\n # The layer norm before self-attention\r\n self.ln_attn = OmniLayerNorm(ori_layer.ln_attn)\r\n # The layer norm before the MLP\r\n self.ln_mlp = OmniLayerNorm(ori_layer.ln_mlp)\r\n else:\r\n self.input_layernorm = OmniLayerNorm(ori_layer.input_layernorm)\r\n if not config.parallel_attn:\r\n self.post_attention_layernorm = OmniLayerNorm(ori_layer.post_attention_layernorm)\r\n\r\n def forward(\r\n self,\r\n hidden_states: torch.Tensor,\r\n attention_mask: torch.Tensor,\r\n alibi: Optional[torch.Tensor] = None,\r\n layer_past: Optional[Tuple[torch.Tensor, torch.Tensor]] = None,\r\n head_mask: Optional[torch.Tensor] = None,\r\n use_cache: bool = False,\r\n output_attentions: bool = False,\r\n **kwargs\r\n ):\r\n residual = hidden_states\r\n if self.config.new_decoder_architecture:\r\n attention_layernorm_out = self.ln_attn(hidden_states)\r\n mlp_layernorm_out = self.ln_mlp(hidden_states)\r\n else:\r\n attention_layernorm_out = self.input_layernorm(hidden_states)\r\n\r\n # Self attention.\r\n attn_outputs = self.self_attention(\r\n attention_layernorm_out,\r\n layer_past=layer_past,\r\n attention_mask=attention_mask,\r\n alibi=alibi,\r\n head_mask=head_mask,\r\n use_cache=use_cache,\r\n output_attentions=output_attentions,\r\n )\r\n\r\n attention_output = attn_outputs[0]\r\n\r\n if not self.config.new_decoder_architecture:\r\n if self.config.parallel_attn:\r\n mlp_layernorm_out = attention_layernorm_out\r\n else:\r\n residual = dropout_add(\r\n attention_output, residual, self.config.attention_dropout, training=self.training\r\n )\r\n mlp_layernorm_out = self.post_attention_layernorm(residual)\r\n\r\n outputs = attn_outputs[1:]\r\n\r\n # MLP.\r\n mlp_output = self.mlp(mlp_layernorm_out)\r\n\r\n if self.config.new_decoder_architecture or self.config.parallel_attn:\r\n mlp_output += attention_output\r\n\r\n output = dropout_add(mlp_output, residual, self.config.hidden_dropout, training=self.training)\r\n\r\n if use_cache:\r\n outputs = (output,) + outputs\r\n else:\r\n outputs = (output,) + outputs[1:]\r\n\r\n return outputs # hidden_states, present, attentions\r\n\r\n def set_quant_state(self, weight_quant: bool = False, act_quant: bool = False):\r\n # setting weight quantization here does not affect actual forward pass\r\n self.use_weight_quant = weight_quant\r\n self.use_act_quant = act_quant\r\n names = []\r\n for name, m in self.named_modules():\r\n if isinstance(m, (QuantLinear, QuantMatMul)):\r\n names.append(name)\r\n m.set_quant_state(weight_quant, act_quant)\r\n\r\n @torch.no_grad()\r\n def smooth_and_quant_inplace(self):\r\n if self.let:\r\n raise ValueError(\"falcon not yet support let\")\r\n for name, module in self.named_modules():\r\n if isinstance(module, QuantLinear):\r\n module.weight = module.weight_quantizer(module.weight)\r\n module.use_temporary_parameter=False\r\n \r\n\r\n def clear_temp_variable(self):\r\n for name, module in self.named_modules():\r\n if isinstance(module, QuantLinear):\r\n del module.temp_weight\r\n del module.temp_bias\r\n\r\n def smooth_and_quant_temporary(self):\r\n if self.let:\r\n raise ValueError(\"falcon not yet support let\")\r\n else:\r\n for name, module in self.named_modules():\r\n if isinstance(module, QuantLinear):\r\n module.temp_weight = module.weight\r\n # quant\r\n for name, module in self.named_modules():\r\n if isinstance(module, QuantLinear):\r\n if hasattr(module, \"temp_weight\"):\r\n module.temp_weight = module.weight_quantizer(module.temp_weight)\r\n else:\r\n module.temp_weight = module.weight_quantizer(module.weight)\r\n if not hasattr(module, \"temp_bias\"):\r\n module.temp_bias = module.bias\r\n module.use_temporary_parameter=True\r\n\r\n\r\n def let_parameters(self, use_shift=True):\r\n params = []\r\n template = \"smooth\" if use_shift else \"smooth_scale\"\r\n for n, m in self.named_parameters():\r\n if n.find(template) > -1:\r\n params.append(m)\r\n return iter(params) \r\n\r\n def lwc_parameters(self):\r\n params = []\r\n for n, m in self.named_parameters():\r\n if n.find('bound_factor') > -1:\r\n params.append(m)\r\n return iter(params) \r\n\r\n def omni_parameters(self, use_shift=True):\r\n params = []\r\n template = \"smooth\" if use_shift else \"smooth_scale\"\r\n for n, m in self.named_parameters():\r\n if n.find('bound_factor') > -1 or n.find(template) > -1:\r\n params.append(m)\r\n return iter(params) \r\n \r\n def omni_state_dict(self, destination=None, prefix='', keep_vars=False):\r\n if destination is None:\r\n destination = OrderedDict()\r\n for name, param in self.named_parameters():\r\n if name.find('smooth') > -1 or name.find('bound_factor') > -1:\r\n destination[prefix + name] = param if keep_vars else param.detach()\r\n return destination\r\n \r\n def register_scales_and_zeros(self):\r\n for name, module in self.named_modules():\r\n if isinstance(module, QuantLinear):\r\n module.weight_quantizer.register_scales_and_zeros()\r\n " }, { "path": "models/int_llama_layer.py", "content": "import torch\r\nfrom torch import nn\r\nfrom typing import Optional, Tuple, List\r\nfrom quant.int_linear import QuantLinear\r\nfrom quant.int_matmul import QuantMatMul\r\nimport torch.nn.functional as F\r\nfrom quant.omni_norm import OmniLlamaRMSNorm\r\nfrom collections import OrderedDict\r\nimport math\r\nfrom transformers.models.llama.modeling_llama import LlamaRotaryEmbedding,apply_rotary_pos_emb,LlamaRMSNorm,repeat_kv\r\nfrom transformers.models.llama.configuration_llama import LlamaConfig\r\nfrom transformers.activations import ACT2FN\r\nimport pdb\r\nimport copy\r\nfrom models.transformation import *\r\n\r\n\r\n\r\n\r\nclass QuantLlamaMLP(nn.Module):\r\n def __init__(\r\n self,\r\n org_module: nn.Module,\r\n hidden_size: int,\r\n intermediate_size: int,\r\n hidden_act: str,\r\n args=None,\r\n ):\r\n super().__init__()\r\n # self.gate_proj = nn.Linear(hidden_size, intermediate_size, bias=False)\r\n # self.down_proj = nn.Linear(intermediate_size, hidden_size, bias=False)\r\n # self.up_proj = nn.Linear(hidden_size, intermediate_size, bias=False)\r\n self.gate_proj = QuantLinear(org_module.gate_proj,\r\n args.weight_quant_params,\r\n args.act_quant_params)\r\n self.down_proj = QuantLinear(org_module.down_proj,\r\n args.weight_quant_params,\r\n args.act_quant_params)\r\n self.up_proj = QuantLinear(org_module.up_proj,\r\n args.weight_quant_params,\r\n args.act_quant_params)\r\n self.act_fn = ACT2FN[hidden_act]\r\n\r\n def forward(self, x):\r\n return self.down_proj(self.act_fn(self.gate_proj(x)) * self.up_proj(x))\r\n\r\n\r\nclass QuantLlamaAttention(nn.Module):\r\n \"\"\"Multi-headed attention from 'Attention Is All You Need' paper\"\"\"\r\n\r\n def __init__(self, \r\n org_module: nn.Module,\r\n config: LlamaConfig,\r\n args=None):\r\n super().__init__()\r\n self.config = config\r\n self.hidden_size = config.hidden_size\r\n self.num_heads = config.num_attention_heads\r\n self.head_dim = self.hidden_size // self.num_heads\r\n self.num_key_value_heads = config.num_key_value_heads\r\n self.num_key_value_groups = self.num_heads // self.num_key_value_heads\r\n self.max_position_embeddings = config.max_position_embeddings\r\n\r\n if (self.head_dim * self.num_heads) != self.hidden_size:\r\n raise ValueError(\r\n f\"hidden_size must be divisible by num_heads (got `hidden_size`: {self.hidden_size}\"\r\n f\" and `num_heads`: {self.num_heads}).\"\r\n )\r\n\r\n self.rotary_emb = copy.deepcopy(org_module.rotary_emb)\r\n\r\n self.k_proj = QuantLinear(\r\n org_module.k_proj,\r\n args.weight_quant_params,\r\n args.act_quant_params,\r\n )\r\n self.v_proj = QuantLinear(\r\n org_module.v_proj,\r\n args.weight_quant_params,\r\n args.act_quant_params,\r\n )\r\n self.q_proj = QuantLinear(\r\n org_module.q_proj,\r\n args.weight_quant_params,\r\n args.act_quant_params,\r\n )\r\n self.o_proj = QuantLinear(\r\n org_module.o_proj, args.weight_quant_params, args.act_quant_params\r\n )\r\n self.qkt_matmul = QuantMatMul(\r\n args.q_quant_params, args.k_quant_params, matmul_func=torch.matmul\r\n )\r\n self.pv_matmul = QuantMatMul(\r\n args.p_quant_params, args.v_quant_params, matmul_func=torch.matmul\r\n )\r\n\r\n self.use_weight_quant = False\r\n self.use_act_quant = False\r\n\r\n def _shape(self, tensor: torch.Tensor, seq_len: int, bsz: int):\r\n return tensor.view(bsz, seq_len, self.num_heads, self.head_dim).transpose(1, 2).contiguous()\r\n\r\n def forward(\r\n self,\r\n hidden_states: torch.Tensor,\r\n attention_mask: Optional[torch.Tensor] = None,\r\n position_ids: Optional[torch.LongTensor] = None,\r\n past_key_value: Optional[Tuple[torch.Tensor]] = None,\r\n output_attentions: bool = False,\r\n use_cache: bool = False,\r\n ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:\r\n bsz, q_len, _ = hidden_states.size()\r\n\r\n # query_states = self.q_proj(hidden_states)\r\n # key_states = self.k_proj(hidden_states)\r\n # value_states = self.v_proj(hidden_states)\r\n query_states = self.q_proj(hidden_states).view(bsz, q_len, self.num_heads, self.head_dim).transpose(1, 2)\r\n key_states =self.k_proj(hidden_states).view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)\r\n value_states = self.v_proj(hidden_states).view(bsz, q_len, self.num_key_value_heads, self.head_dim).transpose(1, 2)\r\n\r\n kv_seq_len = key_states.shape[-2]\r\n if past_key_value is not None:\r\n kv_seq_len += past_key_value[0].shape[-2]\r\n cos, sin = self.rotary_emb(value_states, seq_len=kv_seq_len)\r\n query_states, key_states = apply_rotary_pos_emb(query_states, key_states, cos, sin, position_ids)\r\n\r\n\r\n # [bsz, nh, t, hd]\r\n\r\n if past_key_value is not None:\r\n # reuse k, v, self_attention\r\n key_states = torch.cat([past_key_value[0], key_states], dim=2)\r\n value_states = torch.cat([past_key_value[1], value_states], dim=2)\r\n\r\n past_key_value = (key_states, value_states) if use_cache else None\r\n\r\n # repeat k/v heads if n_kv_heads < n_heads\r\n key_states = repeat_kv(key_states, self.num_key_value_groups)\r\n value_states = repeat_kv(value_states, self.num_key_value_groups)\r\n \r\n query_states = self.qkt_matmul.quant_x1(query_states)\r\n key_states = self.qkt_matmul.quant_x2(key_states)\r\n attn_weights = self.qkt_matmul(query_states, key_states.transpose(2, 3)) / math.sqrt(self.head_dim)\r\n\r\n if attn_weights.size() != (bsz, self.num_heads, q_len, kv_seq_len):\r\n raise ValueError(\r\n f\"Attention weights should be of size {(bsz, self.num_heads, q_len, kv_seq_len)}, but is\"\r\n f\" {attn_weights.size()}\"\r\n )\r\n\r\n if attention_mask is not None:\r\n if attention_mask.size() != (bsz, 1, q_len, kv_seq_len):\r\n raise ValueError(\r\n f\"Attention mask should be of size {(bsz, 1, q_len, kv_seq_len)}, but is {attention_mask.size()}\"\r\n )\r\n attn_weights = attn_weights + attention_mask\r\n attn_weights = torch.max(attn_weights, torch.tensor(torch.finfo(attn_weights.dtype).min))\r\n\r\n # upcast attention to fp32\r\n attn_weights = nn.functional.softmax(attn_weights, dim=-1, dtype=torch.float32).to(query_states.dtype)\r\n attn_weights = self.pv_matmul.quant_x1(attn_weights)\r\n value_states = self.pv_matmul.quant_x2(value_states)\r\n attn_output = self.pv_matmul(attn_weights, value_states)\r\n\r\n if attn_output.size() != (bsz, self.num_heads, q_len, self.head_dim):\r\n raise ValueError(\r\n f\"`attn_output` should be of size {(bsz, self.num_heads, q_len, self.head_dim)}, but is\"\r\n f\" {attn_output.size()}\"\r\n )\r\n\r\n attn_output = attn_output.transpose(1, 2)\r\n attn_output = attn_output.reshape(bsz, q_len, self.hidden_size)\r\n\r\n attn_output = self.o_proj(attn_output)\r\n\r\n if not output_attentions:\r\n attn_weights = None\r\n\r\n return attn_output, attn_weights, past_key_value\r\n \r\n def set_quant_state(self, weight_quant: bool = False, act_quant: bool = False):\r\n # setting weight quantization here does not affect actual forward pass\r\n self.use_weight_quant = weight_quant\r\n self.use_act_quant = act_quant\r\n for m in self.modules():\r\n if isinstance(m, (QuantLinear, QuantMatMul)):\r\n m.set_quant_state(weight_quant, act_quant)\r\n \r\n\r\n\r\nclass QuantLlamaDecoderLayer(nn.Module):\r\n def __init__(self, \r\n config: LlamaConfig,\r\n ori_layer,\r\n args):\r\n super().__init__()\r\n self.hidden_size = config.hidden_size\r\n self.self_attn = QuantLlamaAttention(\r\n org_module=ori_layer.self_attn,\r\n config=config,\r\n args=args,\r\n )\r\n self.mlp = QuantLlamaMLP(\r\n org_module=ori_layer.mlp,\r\n hidden_size=self.hidden_size,\r\n intermediate_size=config.intermediate_size,\r\n hidden_act=config.hidden_act,\r\n args=args,\r\n )\r\n self.input_layernorm = OmniLlamaRMSNorm(ori_layer.input_layernorm,eps=ori_layer.input_layernorm.variance_epsilon)\r\n self.post_attention_layernorm = OmniLlamaRMSNorm(ori_layer.post_attention_layernorm,eps=ori_layer.post_attention_layernorm.variance_epsilon)\r\n\r\n def forward(\r\n self,\r\n hidden_states: torch.Tensor,\r\n attention_mask: Optional[torch.Tensor] = None,\r\n position_ids: Optional[torch.LongTensor] = None,\r\n past_key_value: Optional[Tuple[torch.Tensor]] = None,\r\n output_attentions: Optional[bool] = False,\r\n use_cache: Optional[bool] = False,\r\n ) -> Tuple[torch.FloatTensor, Optional[Tuple[torch.FloatTensor, torch.FloatTensor]]]:\r\n \"\"\"\r\n Args:\r\n hidden_states (`torch.FloatTensor`): input to the layer of shape `(batch, seq_len, embed_dim)`\r\n attention_mask (`torch.FloatTensor`, *optional*): attention mask of size\r\n `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.\r\n output_attentions (`bool`, *optional*):\r\n Whether or not to return the attentions tensors of all attention layers. See `attentions` under\r\n returned tensors for more detail.\r\n use_cache (`bool`, *optional*):\r\n If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding\r\n (see `past_key_values`).\r\n past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states\r\n \"\"\"\r\n residual = hidden_states\r\n\r\n hidden_states = self.input_layernorm(hidden_states)\r\n\r\n\r\n # Self Attention\r\n hidden_states, self_attn_weights, present_key_value = self.self_attn(\r\n hidden_states=hidden_states,\r\n attention_mask=attention_mask,\r\n position_ids=position_ids,\r\n past_key_value=past_key_value,\r\n output_attentions=output_attentions,\r\n use_cache=use_cache,\r\n )\r\n hidden_states = residual + hidden_states\r\n\r\n # Fully Connected\r\n residual = hidden_states\r\n hidden_states = self.post_attention_layernorm(hidden_states)\r\n \r\n\r\n hidden_states = self.mlp(hidden_states)\r\n hidden_states = residual + hidden_states\r\n\r\n outputs = (hidden_states,)\r\n\r\n if output_attentions:\r\n outputs += (self_attn_weights,)\r\n\r\n if use_cache:\r\n outputs += (present_key_value,)\r\n\r\n return outputs \r\n\r\n def set_quant_state(self, weight_quant: bool = False, act_quant: bool = False):\r\n # setting weight quantization here does not affect actual forward pass\r\n self.use_weight_quant = weight_quant\r\n self.use_act_quant = act_quant\r\n names = []\r\n for name, m in self.named_modules():\r\n if isinstance(m, (QuantLinear, QuantMatMul)):\r\n names.append(name)\r\n m.set_quant_state(weight_quant, act_quant)\r\n \r\n def smooth_and_quant_temporary(self):\r\n if self.let:\r\n with torch.no_grad():\r\n for name, module in self.named_parameters():\r\n if \"smooth_scale\" in name:\r\n module.data = truncate_number(module)\r\n smooth_ln_fcs_temporary(self.input_layernorm,[self.self_attn.q_proj, self.self_attn.k_proj, self.self_attn.v_proj],\r\n self.qkv_smooth_scale,self.qkv_smooth_shift)\r\n smooth_ln_fcs_temporary(self.post_attention_layernorm,[self.mlp.up_proj,self.mlp.gate_proj],\r\n self.fc1_smooth_scale,self.fc1_smooth_shift)\r\n smooth_fc_fc_temporary(self.self_attn.v_proj,self.self_attn.o_proj,\r\n self.out_smooth_scale, self.out_smooth_shift)\r\n smooth_q_k_temporary(self.self_attn.q_proj, self.self_attn.k_proj,\r\n self.qkt_smooth_scale)\r\n self.mlp.down_proj.temp_weight = self.mlp.down_proj.weight\r\n else:\r\n for name, module in self.named_modules():\r\n if isinstance(module, QuantLinear):\r\n module.temp_weight = module.weight\r\n # quant\r\n for name, module in self.named_modules():\r\n if isinstance(module, QuantLinear):\r\n if hasattr(module, \"temp_weight\"):\r\n module.temp_weight = module.weight_quantizer(module.temp_weight)\r\n else:\r\n module.temp_weight = module.weight_quantizer(module.weight)\r\n if not hasattr(module, \"temp_bias\"):\r\n module.temp_bias = module.bias\r\n module.use_temporary_parameter=True\r\n\r\n def clear_temp_variable(self):\r\n for name, module in self.named_modules():\r\n if isinstance(module, QuantLinear):\r\n del module.temp_weight\r\n del module.temp_bias\r\n\r\n @torch.no_grad()\r\n def smooth_and_quant_inplace(self):\r\n if self.let:\r\n for name, module in self.named_parameters():\r\n if \"smooth_scale\" in name:\r\n module.data = truncate_number(module)\r\n smooth_ln_fcs_inplace(self.input_layernorm,[self.self_attn.q_proj, self.self_attn.k_proj, self.self_attn.v_proj],\r\n self.qkv_smooth_scale,self.qkv_smooth_shift)\r\n smooth_ln_fcs_inplace(self.post_attention_layernorm,[self.mlp.up_proj,self.mlp.gate_proj],\r\n self.fc1_smooth_scale,self.fc1_smooth_shift)\r\n smooth_fc_fc_inplace(self.self_attn.v_proj,self.self_attn.o_proj,\r\n self.out_smooth_scale, self.out_smooth_shift)\r\n smooth_q_k_inplace(self.self_attn.q_proj, self.self_attn.k_proj,\r\n self.qkt_smooth_scale)\r\n for name, module in self.named_modules():\r\n if isinstance(module, QuantLinear):\r\n module.weight = module.weight_quantizer(module.weight)\r\n module.use_temporary_parameter=False\r\n\r\n def let_parameters(self, use_shift=True):\r\n params = []\r\n template = \"smooth\" if use_shift else \"smooth_scale\"\r\n for n, m in self.named_parameters():\r\n if n.find(template) > -1:\r\n params.append(m)\r\n return iter(params) \r\n\r\n def lwc_parameters(self):\r\n params = []\r\n for n, m in self.named_parameters():\r\n if n.find('bound_factor') > -1:\r\n params.append(m)\r\n return iter(params) \r\n\r\n def omni_parameters(self, use_shift=True):\r\n params = []\r\n template = \"smooth\" if use_shift else \"smooth_scale\"\r\n for n, m in self.named_parameters():\r\n if n.find('bound_factor') > -1 or n.find(template) > -1:\r\n params.append(m)\r\n return iter(params) \r\n \r\n def omni_state_dict(self, destination=None, prefix='', keep_vars=False):\r\n if destination is None:\r\n destination = OrderedDict()\r\n for name, param in self.named_parameters():\r\n if name.find('smooth') > -1 or name.find('bound_factor') > -1:\r\n destination[prefix + name] = param if keep_vars else param.detach()\r\n return destination\r\n \r\n def register_scales_and_zeros(self):\r\n for name, module in self.named_modules():\r\n if isinstance(module, QuantLinear):\r\n module.weight_quantizer.register_scales_and_zeros()" }, { "path": "models/int_opt_layer.py", "content": "import torch\r\nfrom torch import nn\r\nfrom typing import Optional, Tuple, List\r\nfrom quant.int_linear import QuantLinear\r\nfrom quant.int_matmul import QuantMatMul\r\nimport torch.nn.functional as F\r\nfrom quant.omni_norm import OmniLayerNorm\r\nfrom collections import OrderedDict\r\nimport pdb\r\nfrom models.models_utils import truncate_number\r\nfrom models.transformation import *\r\n\r\n\r\n\r\n\r\nclass QuantOPTAttention(nn.Module):\r\n \"\"\"Multi-headed attention from 'Attention Is All You Need' paper\"\"\"\r\n\r\n def __init__(\r\n self,\r\n org_module: nn.Module,\r\n embed_dim: int,\r\n num_heads: int,\r\n dropout: float = 0.0,\r\n is_decoder: bool = False,\r\n bias: bool = True,\r\n args=None,\r\n disable_act_quant=False,\r\n ):\r\n super().__init__()\r\n self.embed_dim = embed_dim\r\n self.num_heads = num_heads\r\n self.dropout = dropout\r\n self.head_dim = embed_dim // num_heads\r\n\r\n if (self.head_dim * num_heads) != self.embed_dim:\r\n raise ValueError(\r\n f\"embed_dim must be divisible by num_heads (got `embed_dim`: {self.embed_dim}\"\r\n f\" and `num_heads`: {num_heads}).\"\r\n )\r\n\r\n self.scaling = self.head_dim**-0.5\r\n self.is_decoder = is_decoder\r\n\r\n # input is quantized by LayerNorm, set disable_input_quant=True\r\n self.k_proj = QuantLinear(\r\n org_module.k_proj,\r\n args.weight_quant_params,\r\n args.act_quant_params,\r\n )\r\n self.v_proj = QuantLinear(\r\n org_module.v_proj,\r\n args.weight_quant_params,\r\n args.act_quant_params,\r\n )\r\n self.q_proj = QuantLinear(\r\n org_module.q_proj,\r\n args.weight_quant_params,\r\n args.act_quant_params,\r\n )\r\n self.out_proj = QuantLinear(\r\n org_module.out_proj, args.weight_quant_params, args.act_quant_params\r\n )\r\n self.qkt_matmul = QuantMatMul(\r\n args.q_quant_params, args.k_quant_params, matmul_func=torch.bmm\r\n )\r\n self.pv_matmul = QuantMatMul(\r\n args.p_quant_params, args.v_quant_params, matmul_func=torch.bmm\r\n )\r\n\r\n self.use_weight_quant = False\r\n self.use_act_quant = False\r\n\r\n def _shape(self, tensor: torch.Tensor, seq_len: int, bsz: int):\r\n return (\r\n tensor.view(bsz, seq_len, self.num_heads, self.head_dim)\r\n .transpose(1, 2)\r\n .contiguous()\r\n )\r\n\r\n def forward(\r\n self,\r\n hidden_states: torch.Tensor,\r\n key_value_states: Optional[torch.Tensor] = None,\r\n past_key_value: Optional[Tuple[torch.Tensor]] = None,\r\n attention_mask: Optional[torch.Tensor] = None,\r\n layer_head_mask: Optional[torch.Tensor] = None,\r\n output_attentions: bool = False,\r\n ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[Tuple[torch.Tensor]]]:\r\n \"\"\"Input shape: Batch x Time x Channel\"\"\"\r\n is_cross_attention = key_value_states is not None\r\n\r\n bsz, tgt_len, _ = hidden_states.size()\r\n\r\n # get query proj\r\n query_states = self.q_proj(hidden_states) * self.scaling\r\n query_states = self.qkt_matmul.quant_x1(query_states)\r\n\r\n # get key, value proj\r\n if is_cross_attention and past_key_value is not None:\r\n # reuse k,v, cross_attentions\r\n key_states = past_key_value[0]\r\n value_states = past_key_value[1]\r\n elif is_cross_attention:\r\n # cross_attentions\r\n key_states = self.k_proj(key_value_states)\r\n key_states = self.qkt_matmul.quant_x2(key_states)\r\n key_states = self._shape(key_states, -1, bsz)\r\n value_states = self._shape(self.v_proj(key_value_states), -1, bsz)\r\n elif past_key_value is not None:\r\n # reuse k, v, self_attention\r\n # bsz, seq_len, self.num_heads, self.head_dim -> bsz, self.num_heads, seq_len, self.head_dim\r\n key_states = self.k_proj(hidden_states)\r\n key_states = self.qkt_matmul.quant_x2(key_states)\r\n key_states = self._shape(key_states, -1, bsz)\r\n\r\n value_states = self.v_proj(hidden_states)\r\n value_states = self.pv_matmul.quant_x2(value_states)\r\n value_states = self._shape(value_states, -1, bsz)\r\n key_states = torch.cat([past_key_value[0], key_states], dim=2)\r\n value_states = torch.cat([past_key_value[1], value_states], dim=2)\r\n else:\r\n # self_attention\r\n key_states = self.k_proj(hidden_states)\r\n key_states = self.qkt_matmul.quant_x2(key_states)\r\n key_states = self._shape(key_states, -1, bsz)\r\n\r\n value_states = self.v_proj(hidden_states)\r\n value_states = self.pv_matmul.quant_x2(value_states)\r\n value_states = self._shape(value_states, -1, bsz)\r\n if self.is_decoder:\r\n past_key_value = (key_states, value_states)\r\n\r\n proj_shape = (bsz * self.num_heads, -1, self.head_dim)\r\n query_states = self._shape(query_states, tgt_len, bsz).view(*proj_shape)\r\n key_states = key_states.view(*proj_shape)\r\n value_states = value_states.view(*proj_shape)\r\n\r\n src_len = key_states.size(1)\r\n attn_weights = self.qkt_matmul(query_states, key_states.transpose(1, 2))\r\n\r\n if attn_weights.size() != (bsz * self.num_heads, tgt_len, src_len):\r\n raise ValueError(\r\n f\"Attention weights should be of size {(bsz * self.num_heads, tgt_len, src_len)}, but is\"\r\n f\" {attn_weights.size()}\"\r\n )\r\n\r\n if attention_mask is not None:\r\n if attention_mask.size() != (bsz, 1, tgt_len, src_len):\r\n raise ValueError(\r\n f\"Attention mask should be of size {(bsz, 1, tgt_len, src_len)}, but is {attention_mask.size()}\"\r\n )\r\n attn_weights = (\r\n attn_weights.view(bsz, self.num_heads, tgt_len, src_len)\r\n + attention_mask\r\n )\r\n attn_weights = torch.max(\r\n attn_weights, torch.tensor(torch.finfo(attn_weights.dtype).min)\r\n )\r\n attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)\r\n\r\n # upcast to fp32 if the weights are in fp16. Please see https://github.com/huggingface/transformers/pull/17437\r\n if attn_weights.dtype == torch.float16:\r\n attn_weights = nn.functional.softmax(\r\n attn_weights, dim=-1, dtype=torch.float32\r\n ).to(torch.float16)\r\n else:\r\n attn_weights = nn.functional.softmax(attn_weights, dim=-1)\r\n\r\n if layer_head_mask is not None:\r\n if layer_head_mask.size() != (self.num_heads,):\r\n raise ValueError(\r\n f\"Head mask for a single layer should be of size {(self.num_heads,)}, but is\"\r\n f\" {layer_head_mask.size()}\"\r\n )\r\n attn_weights = layer_head_mask.view(1, -1, 1, 1) * attn_weights.view(\r\n bsz, self.num_heads, tgt_len, src_len\r\n )\r\n attn_weights = attn_weights.view(bsz * self.num_heads, tgt_len, src_len)\r\n\r\n if output_attentions:\r\n # this operation is a bit awkward, but it's required to\r\n # make sure that attn_weights keeps its gradient.\r\n # In order to do so, attn_weights have to be reshaped\r\n # twice and have to be reused in the following\r\n attn_weights_reshaped = attn_weights.view(\r\n bsz, self.num_heads, tgt_len, src_len\r\n )\r\n attn_weights = attn_weights_reshaped.view(\r\n bsz * self.num_heads, tgt_len, src_len\r\n )\r\n else:\r\n attn_probs_reshaped = None\r\n\r\n # attention shape bsz * self.num_heads, tgt_len, src_len\r\n attn_weights = self.pv_matmul.quant_x1(attn_weights)\r\n attn_output = self.pv_matmul(attn_weights, value_states)\r\n\r\n if attn_output.size() != (bsz * self.num_heads, tgt_len, self.head_dim):\r\n raise ValueError(\r\n f\"`attn_output` should be of size {(bsz, self.num_heads, tgt_len, self.head_dim)}, but is\"\r\n f\" {attn_output.size()}\"\r\n )\r\n\r\n attn_output = attn_output.view(bsz, self.num_heads, tgt_len, self.head_dim)\r\n attn_output = attn_output.transpose(1, 2)\r\n\r\n # Use the `embed_dim` from the config (stored in the class) rather than `hidden_state` because `attn_output` can be\r\n # partitioned aross GPUs when using tensor-parallelism.\r\n attn_output = attn_output.reshape(bsz, tgt_len, self.embed_dim)\r\n attn_output = self.out_proj(attn_output)\r\n\r\n return attn_output, attn_probs_reshaped, past_key_value\r\n\r\n def set_quant_state(self, weight_quant: bool = False, act_quant: bool = False):\r\n # setting weight quantization here does not affect actual forward pass\r\n self.use_weight_quant = weight_quant\r\n self.use_act_quant = act_quant\r\n for m in self.modules():\r\n if isinstance(m, (QuantLinear, QuantMatMul)):\r\n m.set_quant_state(weight_quant, act_quant)\r\n\r\n\r\n\r\n\r\n\r\n\r\n \r\n\r\nclass QuantOPTDecoderLayer(nn.Module):\r\n def __init__(\r\n self,\r\n config,\r\n ori_layer,\r\n args,\r\n ):\r\n super().__init__()\r\n self.embed_dim = config.hidden_size\r\n self.self_attn = QuantOPTAttention(\r\n org_module=ori_layer.self_attn,\r\n embed_dim=self.embed_dim,\r\n num_heads=config.num_attention_heads,\r\n dropout=config.attention_dropout,\r\n is_decoder=True,\r\n bias=config.enable_bias,\r\n args=args,\r\n )\r\n self.do_layer_norm_before = config.do_layer_norm_before\r\n self.dropout = config.dropout\r\n self.self_attn_layer_norm = OmniLayerNorm(\r\n ori_layer.self_attn_layer_norm\r\n )\r\n self.fc1 = QuantLinear(\r\n ori_layer.fc1,\r\n weight_quant_params=args.weight_quant_params,\r\n act_quant_params=args.act_quant_params,\r\n )\r\n self.fc2 = QuantLinear(\r\n ori_layer.fc2,\r\n weight_quant_params=args.weight_quant_params,\r\n act_quant_params=args.act_quant_params,\r\n )\r\n self.final_layer_norm = OmniLayerNorm(\r\n ori_layer.final_layer_norm\r\n )\r\n self.type = ori_layer.fc1.weight.dtype\r\n\r\n def forward(\r\n self,\r\n hidden_states: torch.Tensor,\r\n attention_mask: Optional[torch.Tensor] = None,\r\n layer_head_mask: Optional[torch.Tensor] = None,\r\n output_attentions: Optional[bool] = False,\r\n use_cache: Optional[bool] = False,\r\n past_key_value: Optional[Tuple[torch.Tensor]] = None,\r\n **kwargs\r\n ):\r\n \"\"\"\r\n Args:\r\n hidden_states (`torch.Int8Tensor`): the output of previous layer's layernorm in INT8\r\n attention_mask (`torch.FloatTensor`, *optional*): attention mask of size\r\n `(batch, 1, tgt_len, src_len)` where padding elements are indicated by very large negative values.\r\n layer_head_mask (`torch.FloatTensor`, *optional*): mask for attention heads in a given layer of size\r\n `(encoder_attention_heads,)`.\r\n output_attentions (`bool`, *optional*):\r\n Whether or not to return the attentions tensors of all attention layers. See `attentions` under\r\n returned tensors for more detail.\r\n use_cache (`bool`, *optional*):\r\n If set to `True`, `past_key_values` key value states are returned and can be used to speed up decoding\r\n (see `past_key_values`).\r\n past_key_value (`Tuple(torch.FloatTensor)`, *optional*): cached past key and value projection states\r\n \"\"\"\r\n\r\n # Self Attention\r\n\r\n residual = hidden_states\r\n if self.do_layer_norm_before:\r\n hidden_states = self.self_attn_layer_norm(hidden_states)\r\n # hidden_states = self.self_attn_layer_norm(hidden_states.float()).to(self.type)\r\n\r\n hidden_states, self_attn_weights, present_key_value = self.self_attn(\r\n hidden_states=hidden_states,\r\n past_key_value=past_key_value,\r\n attention_mask=attention_mask,\r\n layer_head_mask=layer_head_mask,\r\n output_attentions=output_attentions,\r\n )\r\n\r\n hidden_states = nn.functional.dropout(hidden_states, p=0.0, training=False)\r\n\r\n hidden_states = residual + hidden_states\r\n\r\n if not self.do_layer_norm_before:\r\n hidden_states = self.self_attn_layer_norm(hidden_states)\r\n\r\n # Fully Connected\r\n hidden_states_shape = hidden_states.shape\r\n hidden_states = hidden_states.reshape(-1, hidden_states.size(-1))\r\n residual = hidden_states\r\n\r\n # residual.add_(hidden_states.to(residual.dtype))\r\n if self.do_layer_norm_before:\r\n hidden_states = self.final_layer_norm(hidden_states)\r\n # hidden_states = self.final_layer_norm(hidden_states.float()).to(self.type)\r\n\r\n \r\n hidden_states = self.fc1(hidden_states)\r\n hidden_states = F.relu(hidden_states)\r\n\r\n hidden_states = self.fc2(hidden_states)\r\n # hidden_states = nn.functional.dropout(hidden_states, p=self.dropout, training=self.training)\r\n\r\n hidden_states = (residual + hidden_states).view(hidden_states_shape)\r\n # residual.add_(hidden_states.to(residual.dtype))\r\n if not self.do_layer_norm_before:\r\n hidden_states = self.final_layer_norm(hidden_states)\r\n\r\n outputs = (hidden_states,)\r\n\r\n if output_attentions:\r\n outputs += (self_attn_weights,)\r\n\r\n if use_cache:\r\n outputs += (present_key_value,)\r\n\r\n return outputs\r\n\r\n def set_quant_state(self, weight_quant: bool = False, act_quant: bool = False):\r\n # setting weight quantization here does not affect actual forward pass\r\n self.use_weight_quant = weight_quant\r\n self.use_act_quant = act_quant\r\n names = []\r\n for name, m in self.named_modules():\r\n if isinstance(m, (QuantLinear, QuantMatMul)):\r\n names.append(name)\r\n m.set_quant_state(weight_quant, act_quant)\r\n\r\n @torch.no_grad()\r\n def smooth_and_quant_inplace(self):\r\n # return\r\n if self.let:\r\n for name, module in self.named_parameters():\r\n if \"smooth_scale\" in name:\r\n module.data = truncate_number(module)\r\n smooth_ln_fcs_inplace(self.self_attn_layer_norm,[self.self_attn.q_proj, self.self_attn.k_proj, self.self_attn.v_proj],\r\n self.qkv_smooth_scale,self.qkv_smooth_shift)\r\n smooth_ln_fcs_inplace(self.final_layer_norm,[self.fc1],\r\n self.fc1_smooth_scale,self.fc1_smooth_shift)\r\n smooth_fc_fc_inplace(self.self_attn.v_proj,self.self_attn.out_proj,\r\n self.out_smooth_scale, self.out_smooth_shift)\r\n smooth_q_k_inplace(self.self_attn.q_proj, self.self_attn.k_proj,\r\n self.qkt_smooth_scale)\r\n for name, module in self.named_modules():\r\n if isinstance(module, QuantLinear):\r\n module.weight = module.weight_quantizer(module.weight)\r\n module.use_temporary_parameter=False\r\n \r\n\r\n def clear_temp_variable(self):\r\n for name, module in self.named_modules():\r\n if isinstance(module, QuantLinear):\r\n del module.temp_weight\r\n del module.temp_bias\r\n\r\n def smooth_and_quant_temporary(self):\r\n if self.let:\r\n with torch.no_grad():\r\n for name, module in self.named_parameters():\r\n if \"smooth_scale\" in name:\r\n module.data = truncate_number(module)\r\n smooth_ln_fcs_temporary(self.self_attn_layer_norm,[self.self_attn.q_proj, self.self_attn.k_proj, self.self_attn.v_proj],\r\n self.qkv_smooth_scale,self.qkv_smooth_shift)\r\n smooth_ln_fcs_temporary(self.final_layer_norm,[self.fc1],\r\n self.fc1_smooth_scale,self.fc1_smooth_shift)\r\n smooth_fc_fc_temporary(self.self_attn.v_proj,self.self_attn.out_proj,\r\n self.out_smooth_scale, self.out_smooth_shift)\r\n smooth_q_k_temporary(self.self_attn.q_proj, self.self_attn.k_proj,\r\n self.qkt_smooth_scale)\r\n self.fc2.temp_weight = self.fc2.weight\r\n else:\r\n for name, module in self.named_modules():\r\n if isinstance(module, QuantLinear):\r\n module.temp_weight = module.weight\r\n # quant\r\n for name, module in self.named_modules():\r\n if isinstance(module, QuantLinear):\r\n if hasattr(module, \"temp_weight\"):\r\n module.temp_weight = module.weight_quantizer(module.temp_weight)\r\n else:\r\n module.temp_weight = module.weight_quantizer(module.weight)\r\n if not hasattr(module, \"temp_bias\"):\r\n module.temp_bias = module.bias\r\n module.use_temporary_parameter=True\r\n\r\n\r\n def let_parameters(self, use_shift=True):\r\n params = []\r\n template = \"smooth\" if use_shift else \"smooth_scale\"\r\n for n, m in self.named_parameters():\r\n if n.find(template) > -1:\r\n params.append(m)\r\n return iter(params) \r\n\r\n def lwc_parameters(self):\r\n params = []\r\n for n, m in self.named_parameters():\r\n if n.find('bound_factor') > -1:\r\n params.append(m)\r\n return iter(params) \r\n\r\n def omni_parameters(self, use_shift=True):\r\n params = []\r\n template = \"smooth\" if use_shift else \"smooth_scale\"\r\n for n, m in self.named_parameters():\r\n if n.find('bound_factor') > -1 or n.find(template) > -1:\r\n params.append(m)\r\n return iter(params) \r\n \r\n def omni_state_dict(self, destination=None, prefix='', keep_vars=False):\r\n if destination is None:\r\n destination = OrderedDict()\r\n for name, param in self.named_parameters():\r\n if name.find('smooth') > -1 or name.find('bound_factor') > -1:\r\n destination[prefix + name] = param if keep_vars else param.detach()\r\n return destination\r\n \r\n\r\n def register_scales_and_zeros(self):\r\n for name, module in self.named_modules():\r\n if isinstance(module, QuantLinear):\r\n module.weight_quantizer.register_scales_and_zeros()\r\n \r\n" }, { "path": "models/models_utils.py", "content": "import abc\nimport torch\nimport json\nimport hashlib\nimport collections\nfrom tqdm import tqdm\nfrom typing import Iterable\nfrom abc import abstractmethod\nfrom torch import nn\nimport transformers\nimport torch.nn.functional as F\n\nclass TruncateFunction(torch.autograd.Function):\n @staticmethod\n def forward(ctx, input, threshold):\n truncated_tensor = input.clone()\n truncated_tensor[truncated_tensor.abs() < threshold] = truncated_tensor[truncated_tensor.abs() < threshold].sign() * threshold\n return truncated_tensor\n \n\n @staticmethod\n def backward(ctx, grad_output):\n grad_input = grad_output.clone()\n return grad_input, None\n\ndef truncate_number(number, threshold=1e-3):\n # clamping scale to avoid overflow in the AMP training\n return TruncateFunction.apply(number, threshold)\n\ndef find_layers(module, layers=[nn.Conv2d, nn.Linear, transformers.Conv1D], name=\"\"):\n if type(module) in layers:\n return {name: module}\n res = {}\n for name1, child in module.named_children():\n res.update(\n find_layers(\n child, layers=layers, name=name + \".\" + name1 if name != \"\" else name1\n )\n )\n return res\n\n\nclass CacheHook:\n def __init__(self, cachinglm):\n if cachinglm is None:\n self.dbdict = None\n return\n\n self.dbdict = cachinglm.dbdict\n\n def add_partial(self, attr, req, res):\n if self.dbdict is None:\n return\n hsh = hash_args(attr, req)\n self.dbdict[hsh] = res\n\n\nclass LM(abc.ABC):\n def __init__(self):\n self.cache_hook = CacheHook(None)\n\n @abstractmethod\n def loglikelihood(self, requests):\n \"\"\"Compute log-likelihood of generating a continuation from a context.\n Downstream tasks should attempt to use loglikelihood instead of other\n LM calls whenever possible.\n\n :param requests: list\n A list of pairs (context, continuation)\n context: str\n Context string. Implementations of LM must be able to handle an\n empty context string.\n continuation: str\n The continuation over which log likelihood will be calculated. If\n there is a word boundary, the space should be in the continuation.\n For example, context=\"hello\" continuation=\" world\" is correct.\n :return: list\n A list of pairs (logprob, isgreedy)\n logprob: float\n The log probability of `continuation`\n isgreedy:\n Whether `continuation` would be generated by greedy sampling from `context`\n \"\"\"\n pass\n\n @abstractmethod\n def loglikelihood_rolling(self, requests):\n \"\"\"Compute full log-likelihood of a string, with no truncation, for perplexity computation\n - We will use the full max context length of the model.\n - For inputs that exceed the max context length, we divide the tokenized string into chunks of up to\n the max context length.\n - IMPORTANT: Each document's loglikelihood/perplexity is computed *separately*, unlike other implementations\n which may simply concatenate multiple documents together.\n - IMPORTANT: We maximize the amount of context for each prediction. Specifically, for inputs that we break into\n multiple chunks, the last input will still a full-sized context.\n Example:\n Input tokens: [ 0 1 2 3 4 5 6 7 8 9 ]\n Prefix: EOT\n Max context length: 4\n Resulting input/prediction pairs:\n\n INPUT: EOT 0 1 2\n PRED: 0 1 2 3\n\n INPUT: 3 4 5 6\n PRED: 4 5 6 7\n\n INPUT: 5 6 7 8\n PRED: 8 9\n\n Observe that:\n 1. Each token is predicted exactly once\n 2. For the last pair, we provide the full context, but only score the last two tokens\n\n :param requests: list\n A list of strings\n string: str\n String for which we are computing per-toke loglikelihood\n :return: list\n A list of pairs (logprob, isgreedy)\n logprob: float\n The log probability of `continuation`\n isgreedy:\n Whether `continuation` would be generated by greedy sampling from `context`\n \"\"\"\n pass\n\n # TODO: Add an optional max length\n @abstractmethod\n def greedy_until(self, requests):\n \"\"\"Generate greedily until a stopping sequence\n\n :param requests: list\n A list of pairs (context, until)\n context: str\n Context string\n until: [str]\n The string sequences to generate until. These string sequences\n may each span across multiple tokens, or may be part of one token.\n :return: list\n A list of strings continuation\n continuation: str\n The generated continuation.\n \"\"\"\n pass\n\n @classmethod\n def create_from_arg_string(cls, additional_config=None):\n additional_config = {} if additional_config is None else additional_config\n args = {k: v for k, v in additional_config.items() if v is not None}\n return cls(**args)\n\n def set_cache_hook(self, cache_hook):\n self.cache_hook = cache_hook\n\n\nclass BaseLM(LM):\n @property\n @abstractmethod\n def eot_token_id(self):\n pass\n\n @property\n @abstractmethod\n def max_length(self):\n pass\n\n @property\n @abstractmethod\n def max_gen_toks(self):\n pass\n\n @property\n @abstractmethod\n def batch_size(self):\n pass\n\n @property\n @abstractmethod\n def device(self):\n pass\n\n @abstractmethod\n def tok_encode(self, string: str):\n pass\n\n @abstractmethod\n def tok_decode(self, tokens: Iterable[int]):\n pass\n\n @abstractmethod\n def _model_generate(self, context, max_length, eos_token_id):\n pass\n\n @abstractmethod\n def _model_call(self, inps):\n \"\"\"\n inps: a torch tensor of shape [batch, sequence]\n the size of sequence may vary from call to call\n\n returns: a torch tensor of shape [batch, sequence, vocab] with the\n logits returned from the model\n \"\"\"\n pass\n\n # subclass must implement properties vocab_size, eot_token_id, max_gen_toks, batch_size, device, max_length.\n # TODO: enforce this somehow\n\n def loglikelihood(self, requests):\n new_reqs = []\n for context, continuation in requests:\n if context == \"\":\n # end of text as context\n context_enc = [self.eot_token_id]\n else:\n context_enc = self.tok_encode(context)\n\n continuation_enc = self.tok_encode(continuation)\n new_reqs.append(((context, continuation), context_enc, continuation_enc))\n\n return self._loglikelihood_tokens(new_reqs)\n\n def loglikelihood_rolling(self, requests):\n # TODO: Implement caching once we've confirmed the perplexity implementation\n # TODO: automatic batch size detection for vectorization\n\n loglikelihoods = []\n for (string,) in tqdm(requests):\n rolling_token_windows = list(\n map(\n make_disjoint_window,\n get_rolling_token_windows(\n token_list=self.tok_encode(string),\n prefix_token=self.eot_token_id,\n max_seq_len=self.max_length,\n context_len=1,\n ),\n )\n )\n\n rolling_token_windows = [(None,) + x for x in rolling_token_windows]\n\n # TODO: extract out this call so it only gets called once and also somehow figure out partial caching for\n # that\n string_nll = self._loglikelihood_tokens(\n rolling_token_windows, disable_tqdm=True\n )\n\n # discard is_greedy\n string_nll = [x[0] for x in string_nll]\n\n string_nll = sum(string_nll)\n loglikelihoods.append(string_nll)\n\n return loglikelihoods\n\n def _loglikelihood_tokens(self, requests, disable_tqdm=False):\n # TODO: implement some kind of efficient-request-middleware that lumps together requests with the same context\n res = []\n dataset_inps = []\n\n def _collate(x):\n # the negative sign on len(toks) sorts descending - this has a few advantages:\n # - time estimates will always be over not underestimates, which is more useful for planning\n # - to know the size of a batch when going through the list, you know the first one is always the batch\n # padded context length. this is useful to simplify the batching logic and more importantly to make\n # automatic adaptive batches much much easier to implement\n # - any OOMs will happen right away rather than near the end\n\n toks = x[1] + x[2]\n return -len(toks), tuple(toks)\n\n # TODO: automatic (variable) batch size detection for vectorization\n re_ord = Reorderer(requests, _collate)\n for chunk in chunks(\n tqdm(re_ord.get_reordered(), disable=disable_tqdm), self.batch_size\n ):\n inps = []\n cont_toks_list = []\n inplens = []\n\n padding_length = None\n\n # because vectorizing is annoying, we first convert each (context, continuation) pair to padded\n # tensors, then we pack them together into a batch, call the model, and then pick it all apart\n # again because vectorizing is annoying\n\n for _, context_enc, continuation_enc in chunk:\n # sanity check\n assert len(context_enc) > 0\n assert len(continuation_enc) > 0\n assert len(continuation_enc) <= self.max_length\n\n # how this all works:\n # CTX CONT\n # inp 0 1 2 3|4 5 6 7 8 9 <- last token is deleted by inp[:, :-1]\n # gpt2 \\ \\\n # logits 1 2 3|4 5 6 7 8 9 <- the ctx half gets tossed out by the\n # cont_toks 4 5 6 7 8 9 [:, -len(continuation_enc):, :self.vocab_size] slice\n\n # when too long to fit in context, truncate from the left\n inp = torch.tensor(\n (context_enc + continuation_enc)[-(self.max_length + 1) :][:-1],\n dtype=torch.long,\n ).to(self.device)\n (inplen,) = inp.shape\n\n cont = continuation_enc\n\n # since in _collate we make sure length is descending, the longest is always the first one.\n padding_length = (\n padding_length if padding_length is not None else inplen\n )\n\n # pad length from seq to padding_length\n inp = torch.cat(\n [\n inp, # [seq]\n torch.zeros(padding_length - inplen, dtype=torch.long).to(\n inp.device\n ), # [padding_length - seq]\n ],\n dim=0,\n )\n\n inps.append(inp.unsqueeze(0)) # [1, padding_length]\n cont_toks_list.append(cont)\n inplens.append(inplen)\n # import pdb; pdb.set_trace()\n batched_inps = torch.cat(inps, dim=0).to(\n self.device\n ) # [batch, padding_length\n\n # self.model = self.model.to(self.device)\n multi_logits = F.log_softmax(\n self._model_call(batched_inps), dim=-1\n ).cpu() # [batch, padding_length, vocab]\n\n # dataset_inps.append(batched_inps)\n # dataset_logits = self._model_logits_on_dataset(dataset_inps)\n # iter = 0\n # for chunk in chunks(\n # tqdm(re_ord.get_reordered(), disable=disable_tqdm), self.batch_size\n # ):\n # multi_logits = dataset_logits[iter]\n # iter+=1\n # inps = []\n # cont_toks_list = []\n # inplens = []\n #\n # padding_length = None\n #\n # # because vectorizing is annoying, we first convert each (context, continuation) pair to padded\n # # tensors, then we pack them together into a batch, call the model, and then pick it all apart\n # # again because vectorizing is annoying\n #\n # # todo: check if we realy nead the following loop\n # for _, context_enc, continuation_enc in chunk:\n # # sanity check\n # assert len(context_enc) > 0\n # assert len(continuation_enc) > 0\n # assert len(continuation_enc) <= self.max_length\n #\n # # how this all works:\n # # CTX CONT\n # # inp 0 1 2 3|4 5 6 7 8 9 <- last token is deleted by inp[:, :-1]\n # # gpt2 \\ \\\n # # logits 1 2 3|4 5 6 7 8 9 <- the ctx half gets tossed out by the\n # # cont_toks 4 5 6 7 8 9 [:, -len(continuation_enc):, :self.vocab_size] slice\n #\n # # when too long to fit in context, truncate from the left\n # inp = torch.tensor(\n # (context_enc + continuation_enc)[-(self.max_length + 1):][:-1],\n # dtype=torch.long,\n # ).to(self.device)\n # (inplen,) = inp.shape\n #\n # cont = continuation_enc\n #\n # # since in _collate we make sure length is descending, the longest is always the first one.\n # padding_length = (\n # padding_length if padding_length is not None else inplen\n # )\n #\n # # pad length from seq to padding_length\n # inp = torch.cat(\n # [\n # inp, # [seq]\n # torch.zeros(padding_length - inplen, dtype=torch.long).to(\n # inp.device\n # ), # [padding_length - seq]\n # ],\n # dim=0,\n # )\n #\n # inps.append(inp.unsqueeze(0)) # [1, padding_length]\n # cont_toks_list.append(cont)\n # inplens.append(inplen)\n\n for (cache_key, _, _), logits, inp, inplen, cont_toks in zip(\n chunk, multi_logits, inps, inplens, cont_toks_list\n ):\n\n # Slice to original seq length\n contlen = len(cont_toks)\n logits = logits[inplen - contlen : inplen].unsqueeze(\n 0\n ) # [1, seq, vocab]\n\n # Check if per-token argmax is exactly equal to continuation\n greedy_tokens = logits.argmax(dim=-1)\n cont_toks = torch.tensor(cont_toks, dtype=torch.long).unsqueeze(\n 0\n ) # [1, seq]\n # import pdb; pdb.set_trace()\n max_equal = (greedy_tokens == cont_toks).all()\n\n # Obtain log-probs at the corresponding continuation token indices\n # last_token_slice = logits[:, -1, :].squeeze(0).tolist()\n logits = torch.gather(logits, 2, cont_toks.unsqueeze(-1)).squeeze(\n -1\n ) # [1, seq]\n\n # Answer: (log prob, is-exact-match)\n answer = (float(logits.sum()), bool(max_equal))\n\n # partial caching\n if cache_key is not None:\n self.cache_hook.add_partial(\"loglikelihood\", cache_key, answer)\n res.append(answer)\n\n return re_ord.get_original(res)\n\n def greedy_until(self, requests):\n print(\"greedy utils in base...\")\n # TODO: implement fully general `until` that handles until that are\n # multiple tokens or that span multiple tokens correctly\n\n # TODO: extract to TokenizedLM?\n res = []\n\n def _collate(x):\n toks = self.tok_encode(x[0])\n return len(toks), x[0]\n\n re_ord = Reorderer(requests, _collate)\n\n for context, until in tqdm(re_ord.get_reordered()):\n if isinstance(until, str):\n until = [until]\n\n (primary_until,) = self.tok_encode(until[0])\n\n context_enc = torch.tensor(\n [self.tok_encode(context)[self.max_gen_toks - self.max_length :]]\n ).to(self.device)\n\n cont = self._model_generate(\n context_enc, context_enc.shape[1] + self.max_gen_toks, primary_until\n )\n\n s = self.tok_decode(cont[0].tolist()[context_enc.shape[1] :])\n\n for term in until:\n s = s.split(term)[0]\n\n # partial caching\n self.cache_hook.add_partial(\"greedy_until\", (context, until), s)\n\n res.append(s)\n\n return re_ord.get_original(res)\n\n\ndef make_disjoint_window(pair):\n \"\"\"Takes output from get_rolling_token_windows and makes the context not overlap with the continuation\"\"\"\n a, b = pair\n return a[: len(a) - (len(b) - 1)], b\n\n\ndef hash_args(attr, args):\n dat = json.dumps([attr] + list(args))\n return hashlib.sha256(dat.encode(\"utf-8\")).hexdigest()\n\n\ndef simple_parse_args_string(args_string):\n \"\"\"\n Parses something like\n args1=val1,arg2=val2\n Into a dictionary\n \"\"\"\n args_string = args_string.strip()\n if not args_string:\n return {}\n arg_list = args_string.split(\",\")\n args_dict = {}\n for arg in arg_list:\n k, v = arg.split(\"=\")\n args_dict[k] = v\n return args_dict\n\n\ndef get_rolling_token_windows(token_list, prefix_token, max_seq_len, context_len):\n \"\"\"\n - context_len allows for a rolling window context, allowing each prediction window to potentially\n condition on some context\n\n :param token_list: list\n List of tokens to be PREDICTED\n :param max_seq_len: int\n max_seq_len of model (or max_seq_len we want to use)\n :param context_len: int\n Amount of desired token context for prediction. Needs to be at least 1.\n :param prefix_token: token\n Dummy token like so the first token has something to condition on\n :return: generator\n Generator of tuples\n (input_tokens, pred_tokens)\n Note: Score only the last len(pred_tokens) logits of the LM\n \"\"\"\n assert 1 <= context_len <= max_seq_len\n if not token_list:\n return\n # +1 offset, going from input->preds\n pred_len = max_seq_len - context_len + 1\n predicted = 0\n\n # Special handling for first window: predict all tokens\n first_seq_len = min(max_seq_len, len(token_list))\n yield ([prefix_token] + token_list[: first_seq_len - 1], token_list[:first_seq_len])\n predicted += first_seq_len\n\n while predicted < len(token_list):\n window_pred_len = min(len(token_list) - predicted, pred_len)\n window_end = predicted + window_pred_len\n\n yield (\n token_list[window_end - max_seq_len - 1 : window_end - 1],\n token_list[window_end - window_pred_len : window_end],\n )\n predicted += window_pred_len\n\n\nclass Reorderer:\n def __init__(self, arr, fn):\n self.size = len(arr)\n arr = list(enumerate(arr))\n arr = group(arr, lambda x: fn(x[1]))\n arr = [([y[0] for y in x], x[0][1]) for x in arr]\n arr.sort(key=lambda x: fn(x[1]))\n\n self.arr = arr\n\n def get_reordered(self):\n return [x[1] for x in self.arr]\n\n def get_original(self, newarr):\n res = [None] * self.size\n cov = [False] * self.size\n\n for (inds, _), v in zip(self.arr, newarr):\n for ind in inds:\n res[ind] = v\n cov[ind] = True\n\n assert all(cov)\n\n return res\n\n\ndef join_iters(iters):\n for iter in iters:\n yield from iter\n\n\ndef chunks(iter, n):\n arr = []\n for x in iter:\n arr.append(x)\n if len(arr) == n:\n yield arr\n arr = []\n\n if arr:\n yield arr\n\n\ndef group(arr, fn):\n res = collections.defaultdict(list)\n\n for ob in arr:\n res[fn(ob)].append(ob)\n\n return list(res.values())\n" }, { "path": "models/transformation.py", "content": "\nimport torch\nimport pdb\n\nclass TruncateFunction(torch.autograd.Function):\n @staticmethod\n def forward(ctx, input, threshold):\n truncated_tensor = input.clone()\n truncated_tensor[truncated_tensor.abs() < threshold] = truncated_tensor[truncated_tensor.abs() < threshold].sign() * threshold\n return truncated_tensor\n \n\n @staticmethod\n def backward(ctx, grad_output):\n grad_input = grad_output.clone()\n return grad_input, None\n\ndef truncate_number(number, threshold=1e-2):\n # avoid overflow with AMP training\n return TruncateFunction.apply(number, threshold)\n\n\n\ndef smooth_ln_fcs_temporary(ln, fcs, scales,shifts):\n ln.use_temporary_parameter = True\n if not isinstance(fcs, list):\n fcs = [fcs]\n if hasattr(ln, 'bias') and ln.bias is not None:\n ln.temp_bias = (ln.bias - shifts) / scales\n else:\n ln.temp_bias = (-1*shifts)/ scales\n\n ln.temp_weight = ln.weight / scales\n\n for fc in fcs:\n fc.use_temporary_parameter = True\n if hasattr(fc, 'bias') and fc.bias is not None:\n fc.temp_bias = fc.bias + fc.weight@shifts\n else:\n fc.temp_bias = fc.weight@shifts\n fc.temp_weight = fc.weight * scales.view(1,-1)\n\n\ndef smooth_fc_fc_temporary(fc1, fc2, scales,shifts=None):\n # only support for v_proj and out_proh now.\n fc1.use_temporary_parameter = True\n fc2.use_temporary_parameter = True\n if hasattr(fc1, 'temp_weight'):\n fc1.temp_bias = fc1.temp_bias - shifts\n fc1.temp_bias = fc1.temp_bias/scales.view(-1)\n fc1.temp_weight = fc1.temp_weight/scales.view(-1,1)\n else:\n fc1.temp_bias = fc1.bias/scales.view(-1)\n fc1.temp_weight = fc1.weight/scales.view(-1,1)\n \n if hasattr(fc2, 'bias') and fc2.bias is not None:\n fc2.temp_bias = fc2.bias + fc2.weight@shifts\n else:\n fc2.temp_bias = fc2.weight@shifts\n fc2.temp_weight = fc2.weight * scales.view(1,-1)\n\n\ndef smooth_q_k_temporary(q_proj, k_proj, scales):\n q_proj.use_temporary_parameter = True\n k_proj.use_temporary_parameter = True\n q_proj.temp_weight = q_proj.temp_weight/scales.view(-1,1)\n q_proj.temp_bias = q_proj.temp_bias/scales.view(-1)\n k_proj.temp_weight = k_proj.temp_weight*scales.view(-1,1)\n k_proj.temp_bias = k_proj.temp_bias*scales.view(-1)\n\ndef smooth_ln_fcs_inplace(ln, fcs, scales,shifts):\n ln.use_temporary_parameter = False\n if not isinstance(fcs, list):\n fcs = [fcs]\n if hasattr(ln, 'bias') and ln.bias is not None:\n ln.bias.sub_(shifts)\n ln.bias.div_(scales)\n else:\n del ln.bias\n ln.register_buffer('bias',(-1*shifts)/scales)\n\n ln.weight.div_(scales)\n for fc in fcs:\n fc.use_temporary_parameter = False\n if hasattr(fc, 'bias') and fc.bias is not None:\n fc.bias.add_(fc.weight@shifts)\n else:\n del fc.bias\n fc.register_buffer('bias',fc.weight@shifts)\n fc.weight.mul_(scales.view(1,-1))\n\n\ndef smooth_fc_fc_inplace(fc1, fc2, scales,shifts=None):\n # only support for v_proj and out_proh now.\n fc1.use_temporary_parameter = False\n fc2.use_temporary_parameter = False\n fc1.bias.sub_(shifts)\n fc1.bias.div_(scales.view(-1))\n fc1.weight.div_(scales.view(-1,1))\n \n if hasattr(fc2, 'bias') and fc2.bias is not None:\n fc2.bias.add_(fc2.weight@shifts)\n else:\n del fc2.bias\n fc2.register_buffer('bias',fc2.weight@shifts)\n fc2.weight.mul_(scales.view(1,-1))\n\ndef smooth_q_k_inplace(q_proj, k_proj, scales,):\n q_proj.use_temporary_parameter = False\n k_proj.use_temporary_parameter = False\n q_proj.weight.div_(scales.view(-1,1))\n q_proj.bias.div_(scales.view(-1))\n k_proj.weight.mul_(scales.view(-1,1))\n k_proj.bias.mul_(scales.view(-1))" }, { "path": "parallel_utils.py", "content": "import torch\nimport torch.nn as nn\nfrom typing import List\nfrom functools import partial\nimport subprocess\nimport re\nimport os\nimport time\nimport pdb\n\n\ndef nvidia_smi_memory_info():\n result = subprocess.run(\n [\n \"nvidia-smi\",\n \"--query-gpu=index,memory.total,memory.used,memory.free\",\n \"--format=csv,noheader,nounits\",\n ],\n stdout=subprocess.PIPE,\n text=True,\n )\n output = result.stdout.split(\"\\n\")[:-1]\n\n gpu_memory_info = []\n for line in output:\n gpu_id, total_memory, used_memory, free_memory = map(int, re.split(\",\\s\", line))\n gpu_memory_info.append(\n {\n \"id\": gpu_id,\n \"total_memory\": total_memory,\n \"used_memory\": used_memory,\n \"free_memory\": free_memory,\n }\n )\n\n return gpu_memory_info\n\n\nnum_gpus = torch.cuda.device_count()\n\n\ndef get_gpu_memory():\n memory_info = []\n gpu_memory_info = nvidia_smi_memory_info()\n\n try:\n gpu_index = [int(k) for k in os.environ['CUDA_VISIBLE_DEVICES'].split(',')]\n except KeyError:\n gpu_index = [x[\"id\"] for x in gpu_memory_info]\n\n for gpu_id, i in enumerate( gpu_index):\n gpu = gpu_memory_info[i]\n total_memory = gpu[\"total_memory\"]\n used_memory = gpu[\"used_memory\"]\n memory_info.append((gpu_id, total_memory, used_memory))\n return memory_info\n\n\ndef get_lowest_occupied_gpu(wait_memory=1000):\n\n now_lowest_memory = 1e9\n while now_lowest_memory > wait_memory:\n if not now_lowest_memory == 1e9:\n time.sleep(10)\n memory_info = get_gpu_memory()\n gpu_id, tot_mem, used_mem = sorted(\n memory_info, key=lambda x: x[2], reverse=False\n )[0]\n now_lowest_memory = used_mem\n\n return gpu_id\n\n\ndef sort_layers_by_params(layers: List[nn.Module]):\n return sorted(\n layers, key=lambda m: sum(p.numel() for p in m.parameters()), reverse=True\n )\n\n\ndef get_all_gpu_free_memory():\n return sum(\n [\n total_memory - used_memory\n for gpu_id, total_memory, used_memory in get_gpu_memory()\n ]\n )\n\n\ndef assign_layers_to_gpus(layers: List[nn.Module]):\n layer_gpu_map = {}\n prev_gpu_id = None\n weight_num = 0\n for module in layers:\n if hasattr(module, \"weight\"):\n weight_num += module.weight.numel()\n weight_mb = weight_num * 2 / 1024 / 1024\n all_gpu_mems = get_all_gpu_free_memory()\n while all_gpu_mems < weight_mb * 1.3:\n time.sleep(10)\n all_gpu_mems = get_all_gpu_free_memory()\n\n for i, layer in enumerate(layers):\n if i == len(layers) - 1:\n layer_gpu_map[layer] = layer_gpu_map[layers[0]]\n layer.to(layers[0].device)\n layer.device = layers[0].device\n print(f\"map last layer {i} to gpu {layer_gpu_map[layer]}\")\n continue\n layer_memory = (\n sum(p.element_size() * p.numel() for p in layer.parameters()) / 1024**2\n )\n available_gpus = get_gpu_memory()\n if prev_gpu_id is None:\n gpus = sorted(available_gpus, key=lambda x: x[2])\n else:\n pre_gpu_info = available_gpus[prev_gpu_id]\n gpus = [pre_gpu_info] + sorted(available_gpus, key=lambda x: x[2])\n mapped = False\n for gpu_id, tot_memory, allocated_memory in gpus:\n if (tot_memory - allocated_memory * 1.35) > layer_memory:\n layer_gpu_map[layer] = gpu_id\n layer.to(f\"cuda:{gpu_id}\")\n layer.device = f\"cuda:{gpu_id}\"\n print(f\"map layer {i} to gpu {gpu_id}, {available_gpus}\")\n mapped = True\n prev_gpu_id = gpu_id\n break\n if not mapped:\n raise RuntimeError(f\"memory not enough {available_gpus}\")\n\n return layer_gpu_map\n\n\n# forward hook\ndef forward_hook_wrapper(gpu_id):\n def forward_hook(module, input, kwargs):\n # breakpoint()\n input = tuple(_.to(f\"cuda:{gpu_id}\") for _ in input)\n kwargs = {\n k: v.to(f\"cuda:{gpu_id}\") if isinstance(v, torch.Tensor) else v\n for k, v in kwargs.items()\n }\n return input, kwargs\n\n return forward_hook\n\n\ndef add_forward_hooks(layer_gpu_map):\n prev_gpu_id = None\n for layer, gpu_id in layer_gpu_map.items():\n layer: nn.Module\n if prev_gpu_id is None:\n prev_gpu_id = gpu_id\n # if gpu_id != prev_gpu_id:\n layer.register_forward_pre_hook(forward_hook_wrapper(gpu_id), with_kwargs=True)\n prev_gpu_id = gpu_id\n\n\ndef map_layers_to_multi_gpus(layers):\n\n layer_gpu_map = assign_layers_to_gpus(layers)\n\n add_forward_hooks(layer_gpu_map)\n\n\nif __name__ == \"__main__\":\n info = get_gpu_memory()\n print(info)\n" }, { "path": "quant/__init__.py", "content": "" }, { "path": "quant/int_linear.py", "content": "import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom quant.quantizer import UniformAffineQuantizer\n\n\n\n\n\n\nclass QuantLinear(nn.Module):\n \"\"\"\n Quantized Module that can perform quantized convolution or normal convolution.\n To activate quantization, please use set_quant_state function.\n \"\"\"\n def __init__(\n self,\n org_module: nn.Linear,\n weight_quant_params: dict = {},\n act_quant_params: dict = {},\n disable_input_quant=False,\n ):\n super().__init__()\n self.fwd_kwargs = dict()\n self.fwd_func = F.linear\n self.register_buffer('weight',org_module.weight)\n if org_module.bias is not None:\n self.register_buffer('bias',org_module.bias)\n else:\n self.bias = None\n self.in_features = org_module.in_features\n self.out_features = org_module.out_features\n # de-activate the quantized forward default\n self.use_weight_quant = False\n self.use_act_quant = False\n # initialize quantizer\n self.weight_quantizer = UniformAffineQuantizer(**weight_quant_params,shape=org_module.weight.shape)\n if not disable_input_quant:\n self.act_quantizer = UniformAffineQuantizer(**act_quant_params)\n else:\n self.act_quantizer = None\n\n self.disable_input_quant = disable_input_quant\n self.use_temporary_parameter = False\n\n \n \n def forward(self, input: torch.Tensor):\n if self.use_temporary_parameter:\n weight = self.temp_weight\n bias = self.temp_bias\n elif self.use_weight_quant:\n weight = self.weight_quantizer(self.weight)\n bias = self.bias\n else:\n weight = self.weight\n bias = self.bias\n\n if self.use_act_quant and not self.disable_input_quant:\n input = self.act_quantizer(input)\n \n out = self.fwd_func(input, weight, bias, **self.fwd_kwargs)\n\n\n return out\n\n def set_quant_state(self, weight_quant: bool = False, act_quant: bool = False):\n self.use_weight_quant = weight_quant\n self.use_act_quant = act_quant\n" }, { "path": "quant/int_matmul.py", "content": "import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom quant.quantizer import UniformAffineQuantizer\n\n\nclass QuantMatMul(nn.Module):\n def __init__(\n self,\n x1_quant_params: dict = {},\n x2_quant_params: dict = {},\n disable_act_quant=False,\n matmul_func=torch.bmm,\n ):\n super().__init__()\n # de-activate the quantized forward default\n self.use_act_quant = False\n # initialize quantizer\n self.i_cluster_counts = None\n self.x1_quantizer = UniformAffineQuantizer(**x1_quant_params)\n self.x2_quantizer = UniformAffineQuantizer(**x2_quant_params)\n self.matmul_func = matmul_func\n\n self.disable_act_quant = disable_act_quant\n\n\n def set_quant_state(self, weight_quant: bool = False, act_quant: bool = False):\n self.use_weight_quant = weight_quant\n self.use_act_quant = act_quant\n\n def quant_x1(self, x1):\n if self.use_act_quant:\n x1 = self.x1_quantizer(x1)\n return x1\n\n def quant_x2(self, x2):\n if self.use_act_quant:\n x2 = self.x2_quantizer(x2)\n return x2\n\n def forward(self, x1, x2):\n out = self.matmul_func(x1, x2)\n return out\n" }, { "path": "quant/omni_norm.py", "content": "import torch\nimport torch.nn as nn\n\n\n'''\nModify normalization layer to adapt the training of learnable equivalent transformation\n'''\n\n\n\nclass OmniLayerNorm(nn.Module):\n def __init__(self, ori_layer_norm) -> None:\n super().__init__()\n self.use_act_quant = True\n self.register_buffer('weight',ori_layer_norm.weight)\n if ori_layer_norm.bias is not None:\n self.register_buffer('bias',ori_layer_norm.bias)\n else:\n self.bias = None\n self.eps = ori_layer_norm.eps\n self.norm_func = nn.functional.layer_norm\n self.normalized_shape = ori_layer_norm.normalized_shape\n self.use_temporary_parameter = False\n\n\n def forward(self, x):\n if self.use_temporary_parameter:\n weight = self.temp_weight\n bias = self.temp_bias\n else:\n weight = self.weight\n bias = self.bias\n out = self.norm_func(x,self.normalized_shape,weight, bias,eps=self.eps)\n return out\n\n def set_quant_state(self, use_weight_quant, use_act_quant):\n self.use_act_quant = use_act_quant\n\n\nclass OmniLlamaRMSNorm(nn.Module):\n def __init__(self, ori_norm, eps=1e-6):\n \"\"\"\n LlamaRMSNorm is equivalent to T5LayerNorm\n \"\"\"\n super().__init__()\n self.register_buffer('weight',ori_norm.weight)\n self.bias = None\n self.variance_epsilon = eps\n self.use_temporary_parameter = False\n\n\n def forward(self, hidden_states):\n input_dtype = hidden_states.dtype\n variance = hidden_states.to(torch.float32).pow(2).mean(-1, keepdim=True)\n hidden_states = hidden_states * torch.rsqrt(variance + self.variance_epsilon)\n if self.use_temporary_parameter:\n weight = self.temp_weight\n bias = self.temp_bias\n else:\n weight = self.weight\n bias = self.bias if hasattr(self, 'bias') else None\n\n return (weight * hidden_states+bias).to(input_dtype) if bias is not None else (weight * hidden_states).to(input_dtype)\n\n" }, { "path": "quant/omniquant.py", "content": "import torch\nimport torch.nn as nn\nfrom models.int_llama_layer import QuantLlamaDecoderLayer\nfrom models.int_opt_layer import QuantOPTDecoderLayer\nfrom models.int_falcon_layer import QuantFalconDecoderLayer\nfrom quant.int_linear import QuantLinear\nfrom contextlib import nullcontext\nimport copy\nimport math\nimport utils\nimport os\nimport pdb\nimport gc\nfrom quant.utils import let_parameters, lwc_parameters, get_omni_parameters,\\\n omni_state_dict, register_scales_and_zeros,smooth_and_quant_temporary,\\\n smooth_and_quant_inplace,clear_temp_variable,set_quant_state\ntry:\n import auto_gptq.nn_modules.qlinear.qlinear_cuda as qlinear_cuda\n import auto_gptq.nn_modules.qlinear.qlinear_triton as qlinear_triton\nexcept:\n print(\"auto_gptq is required for real quantization\")\n\n\n\ndef get_named_linears(module):\n return {name: m for name, m in module.named_modules() if isinstance(m, QuantLinear)}\n\n\ndef add_new_module(name, original_module, added_module):\n levels = name.split('.')\n if len(levels) > 1:\n mod_ = original_module\n for l_idx in range(len(levels)-1):\n if levels[l_idx].isdigit():\n mod_ = mod_[int(levels[l_idx])]\n else:\n mod_ = getattr(mod_, levels[l_idx])\n setattr(mod_, levels[-1], added_module)\n else:\n setattr(original_module, name, added_module) \n\ndef omniquant(\n lm,\n args,\n dataloader,\n act_scales,\n act_shifts,\n logger=None,\n):\n logger.info(\"Starting ...\")\n \n # move embedding layer and first layer to target device\n model = lm.model\n dev = lm.device\n use_cache = model.config.use_cache\n model.config.use_cache = False\n is_llama = False\n if \"llama\" in args.net.lower():\n is_llama = True\n layers = model.model.layers\n model.model.embed_tokens = model.model.embed_tokens.to(dev)\n model.model.norm = model.model.norm.to(dev)\n DecoderLayer = QuantLlamaDecoderLayer\n pairs = {\n \"q_proj\":\"qkv\",\n \"o_proj\":\"out\",\n \"up_proj\":\"fc1\"\n }\n layer_name_prefix = \"model.layers\"\n elif \"opt\" in args.net.lower():\n layers = model.model.decoder.layers\n model.model.decoder.embed_tokens = model.model.decoder.embed_tokens.to(dev)\n model.model.decoder.embed_positions = model.model.decoder.embed_positions.to(dev)\n if hasattr(model.model.decoder, \"project_out\") and model.model.decoder.project_out:\n model.model.decoder.project_out = model.model.decoder.project_out.to(dev)\n if hasattr(model.model.decoder, \"project_in\") and model.model.decoder.project_in:\n model.model.decoder.project_in = model.model.decoder.project_in.to(dev)\n DecoderLayer = QuantOPTDecoderLayer\n pairs = {\n \"q_proj\":\"qkv\",\n \"out_proj\":\"out\",\n \"fc1\":\"fc1\"\n }\n layer_name_prefix = \"model.decoder.layers\"\n elif \"falcon\" in args.net.lower():\n layers = model.transformer.h\n model.transformer.word_embeddings.to(dev)\n model.transformer.ln_f.to(dev)\n model.lm_head.to(dev)\n DecoderLayer = QuantFalconDecoderLayer\n layer_name_prefix = \"model.transformer.h\"\n elif 'mixtral' in args.net.lower():\n is_llama = True # same to llama except ffn\n layers = model.model.layers\n model.model.embed_tokens = model.model.embed_tokens.to(dev)\n model.model.norm = model.model.norm.to(dev)\n layer_name_prefix = \"model.layers\"\n else:\n raise ValueError(\"Only support for opt/llama/Llama-2/falcon/mixtral now\")\n \n \n layers[0] = layers[0].to(dev)\n if args.deactive_amp and args.epochs>0:\n dtype = torch.float\n traincast = nullcontext\n else:\n dtype = torch.float16\n traincast = torch.cuda.amp.autocast\n inps = torch.zeros(\n (args.nsamples, lm.seqlen, model.config.hidden_size), dtype=dtype, device=dev\n )\n cache = {\"i\": 0}\n\n # catch the first layer input\n class Catcher(nn.Module):\n def __init__(self, module):\n super().__init__()\n self.module = module\n self.is_llama = False\n\n def forward(self, inp, **kwargs):\n inps[cache[\"i\"]] = inp\n cache[\"i\"] += 1\n cache[\"attention_mask\"] = kwargs[\"attention_mask\"]\n if self.is_llama:\n cache[\"position_ids\"] = kwargs[\"position_ids\"]\n raise ValueError\n\n layers[0] = Catcher(layers[0])\n layers[0].is_llama = is_llama\n\n with torch.no_grad():\n for batch in dataloader:\n if cache[\"i\"] >= args.nsamples:\n break\n try:\n model(batch[0].to(dev))\n except ValueError:\n pass\n \n # move embedding layer and first layer to cpu\n layers[0] = layers[0].module\n layers[0] = layers[0].cpu()\n if \"llama\" in args.net.lower() or \"mixtral\" in args.net.lower():\n model.model.embed_tokens = model.model.embed_tokens.cpu()\n model.model.norm = model.model.norm.cpu()\n elif \"opt\" in args.net.lower():\n model.model.decoder.embed_tokens = model.model.decoder.embed_tokens.cpu()\n model.model.decoder.embed_positions = model.model.decoder.embed_positions.cpu()\n if hasattr(model.model.decoder, \"project_out\") and model.model.decoder.project_out:\n model.model.decoder.project_out = model.model.decoder.project_out.cpu()\n if hasattr(model.model.decoder, \"project_in\") and model.model.decoder.project_in:\n model.model.decoder.project_in = model.model.decoder.project_in.cpu()\n elif 'falcon' in args.model:\n model.transformer.word_embeddings = model.transformer.word_embeddings.cpu()\n else:\n raise ValueError(\"Only support for opt/llama/Llama-2/falcon/mixtral now\")\n torch.cuda.empty_cache()\n\n \n # same input of first layer for fp model and quant model\n quant_inps = inps\n fp_inps = copy.deepcopy(inps) # take output of fp model as input\n fp_inps_2 = copy.deepcopy(inps) if args.aug_loss else None # take output of quantization model as input\n \n attention_mask = cache[\"attention_mask\"]\n\n if attention_mask is not None:\n attention_mask_batch = attention_mask.repeat(args.batch_size,1,1,1) if args.deactive_amp else attention_mask.repeat(args.batch_size,1,1,1).float()\n else:\n logger.info(\n \"No attention mask caught from the first layer.\"\n \" Seems that model's attention works without a mask.\"\n )\n attention_mask_batch = None\n\n loss_func = torch.nn.MSELoss()\n if is_llama:\n position_ids = cache[\"position_ids\"]\n else:\n position_ids = None\n\n\n\n if args.resume:\n omni_parameters = torch.load(args.resume)\n else:\n omni_parameters = {}\n\n \n \n for i in range(len(layers)):\n logger.info(f\"=== Start quantize layer {i} ===\")\n layer = layers[i].to(dev)\n if \"mixtral\" in args.net.lower(): \n # for mixtral, we only leverage lwc, which can be achieve by simply replace Linear with QuantLinear\n qlayer = copy.deepcopy(layer)\n for name, module in qlayer.named_modules():\n if isinstance(module,torch.nn.Linear) and not \"gate\" in name: # do not quantize gate\n quantlinear = QuantLinear(module, args.weight_quant_params, args.act_quant_params)\n add_new_module(name, qlayer, quantlinear) \n else:\n qlayer = DecoderLayer(lm.model.config, layer, args)\n qlayer = qlayer.to(dev)\n\n \n # obtain output of full-precision model\n set_quant_state(qlayer, weight_quant=False, act_quant=False)\n if args.epochs > 0:\n with torch.no_grad():\n with torch.cuda.amp.autocast():\n for j in range(args.nsamples):\n fp_inps[j] = qlayer(fp_inps[j].unsqueeze(0), attention_mask=attention_mask,position_ids=position_ids)[0]\n if args.aug_loss:\n fp_inps_2[j] = qlayer(quant_inps[j].unsqueeze(0), attention_mask=attention_mask,position_ids=position_ids)[0]\n # init smooth parameters\n set_quant_state(qlayer, weight_quant=False, act_quant=True) # weight will be manually quantized before forward\n qlayer.let = args.let\n use_shift = True \n if is_llama or args.abits == 16:\n use_shift = False # deactivate channel-wise shifting for llama model and weight-only quantization\n if args.let:\n # init channel-wise scaling and shift\n qlayer.register_parameter(\"qkt_smooth_scale\",torch.nn.Parameter(torch.ones(layer.self_attn.q_proj.out_features,device=dev, dtype=dtype)))\n for name,module in qlayer.named_modules():\n if isinstance(module, QuantLinear):\n for key in pairs.keys():\n if key in name:\n act = act_scales[f\"{layer_name_prefix}.{i}.{name}\"].to(device=dev, dtype=dtype).clamp(min=1e-5)\n weight = module.weight.abs().max(dim=0)[0].clamp(min=1e-5)\n scale = (act.pow(args.alpha)/weight.pow(1-args.alpha)).clamp(min=1e-5)\n if use_shift and not is_llama:\n shift = act_shifts[f\"{layer_name_prefix}.{i}.{name}\"].to(device=dev, dtype=dtype)\n else:\n shift = torch.zeros_like(scale)\n qlayer.register_parameter(f\"{pairs[key]}_smooth_shift\",torch.nn.Parameter(shift))\n qlayer.register_parameter(f\"{pairs[key]}_smooth_scale\",torch.nn.Parameter(scale))\n \n if args.resume:\n qlayer.load_state_dict(omni_parameters[i], strict=False)\n \n\n if args.epochs > 0:\n with torch.no_grad():\n qlayer.float() # required for AMP training\n # create optimizer\n optimizer = torch.optim.AdamW(\n [{\"params\":let_parameters(qlayer, use_shift),\"lr\":args.let_lr}, {\"params\":lwc_parameters(qlayer),\"lr\":args.lwc_lr}],weight_decay=args.wd)\n loss_scaler = utils.NativeScalerWithGradNormCount()\n \n for epochs in range(args.epochs):\n loss_list = []\n norm_list = []\n for j in range(args.nsamples//args.batch_size): \n index = j * args.batch_size\n # obtain output of quantization model\n with traincast():\n smooth_and_quant_temporary(qlayer, args, is_llama)\n quant_out = qlayer(quant_inps[index:index+args.batch_size,], attention_mask=attention_mask_batch,position_ids=position_ids)[0]\n loss = loss_func(fp_inps[index:index+args.batch_size,], quant_out)\n if args.aug_loss:\n loss += loss_func(fp_inps_2[index:index+args.batch_size,], quant_out)\n if not math.isfinite(loss.item()):\n logger.info(\"Loss is NAN, stopping training\")\n pdb.set_trace()\n \n loss_list.append(loss.detach().cpu())\n optimizer.zero_grad()\n norm = loss_scaler(loss, optimizer,parameters= get_omni_parameters(qlayer, use_shift)).cpu()\n norm_list.append(norm.data)\n\n loss_mean = torch.stack(loss_list).mean()\n norm_mean = torch.stack(norm_list).mean()\n logger.info(f\"layer {i} iter {epochs} loss:{loss_mean} norm:{norm_mean} max memory_allocated {torch.cuda.max_memory_allocated(lm._device) / 1024**2} \")\n clear_temp_variable(qlayer)\n del optimizer\n qlayer.half() \n # real smooth and quantization\n smooth_and_quant_inplace(qlayer, args, is_llama)\n if args.epochs>0:\n # update input of quantization model\n with torch.no_grad():\n # with torch.cuda.amp.autocast():\n with traincast():\n for j in range(args.nsamples):\n quant_inps[j] = qlayer(quant_inps[j].unsqueeze(0), attention_mask=attention_mask,position_ids=position_ids)[0]\n register_scales_and_zeros(qlayer)\n layers[i] = qlayer.to(\"cpu\")\n omni_parameters[i] = omni_state_dict(qlayer)\n torch.save(omni_parameters, os.path.join(args.output_dir, f\"omni_parameters.pth\"))\n else:\n register_scales_and_zeros(qlayer)\n layers[i] = qlayer.to(\"cpu\")\n if args.real_quant:\n assert args.wbits in [2,3,4] and args.abits >= 16 # only support weight-only quantization\n named_linears = get_named_linears(qlayer)\n for name, module in named_linears.items():\n scales = module.weight_quantizer.scales\n zeros = module.weight_quantizer.zeros\n group_size = module.weight_quantizer.group_size\n dim0 = module.weight.shape[0]\n scales = scales.view(dim0,-1)\n zeros = zeros.view(dim0,-1)\n if args.wbits == 3:\n q_linear = qlinear_cuda.QuantLinear(args.wbits, group_size, module.in_features,module.out_features,not module.bias is None)\n else:\n q_linear = qlinear_triton.QuantLinear(args.wbits, group_size, module.in_features,module.out_features,not module.bias is None)\n q_linear.pack(module.cpu(), scales.float().cpu(), zeros.float().cpu())\n add_new_module(name, qlayer, q_linear) \n print(f\"pack quantized {name} finished\")\n del module \n del layer\n torch.cuda.empty_cache()\n\n del inps\n del quant_inps\n del fp_inps\n del fp_inps_2\n torch.cuda.empty_cache()\n gc.collect() \n model.config.use_cache = use_cache\n return model\n\n" }, { "path": "quant/quantizer.py", "content": "import torch\nimport torch.nn as nn\nimport torch.nn.functional as F\nfrom typing import Union\nimport tqdm\nimport numpy as np\nimport pdb\nimport math\n\nCLIPMIN = 1e-5\n\n\n\n\ndef round_ste(x: torch.Tensor):\n \"\"\"\n Implement Straight-Through Estimator for rounding operation.\n \"\"\"\n return (x.round() - x).detach() + x\n\n\n\nclass UniformAffineQuantizer(nn.Module):\n def __init__(\n self,\n n_bits: int = 8,\n symmetric: bool = False,\n per_channel_axes=[],\n metric=\"minmax\",\n dynamic=False,\n dynamic_method=\"per_cluster\",\n group_size=None,\n shape=None,\n lwc=False,\n disable_zero_point=False,\n ):\n \"\"\"\n support cluster quantize\n dynamic_method support per_token and per_cluster\n \"\"\"\n super().__init__()\n self.symmetric = symmetric\n self.disable_zero_point = disable_zero_point\n assert 2 <= n_bits <= 16, \"bitwidth not supported\"\n self.n_bits = n_bits\n if self.disable_zero_point:\n self.qmin = -(2 ** (n_bits - 1))\n self.qmax = 2 ** (n_bits - 1) - 1\n else:\n self.qmin = 0\n self.qmax = 2 ** (n_bits) - 1\n self.per_channel_axes = per_channel_axes\n self.metric = metric\n self.cluster_counts = None\n self.cluster_dim = None\n\n self.scale = None\n self.zero_point = None\n self.round_zero_point = None\n\n self.cached_xmin = None\n self.cached_xmax = None\n self.dynamic = dynamic\n self.dynamic_method = dynamic_method\n self.deficiency = 0\n self.lwc = lwc\n \n init_value = 4. # inti value of learnable weight clipping\n if lwc:\n if group_size:\n dim1 = int(shape[0]*math.ceil(shape[1]/group_size))\n self.deficiency = shape[-1]%group_size\n if self.deficiency > 0:\n self.deficiency = group_size - self.deficiency\n assert self.symmetric # support for mlc-llm symmetric quantization\n else:\n dim1 = shape[0]\n self.upbound_factor = nn.Parameter(torch.ones((dim1,1))*init_value)\n self.lowbound_factor = nn.Parameter(torch.ones((dim1,1))*init_value)\n self.sigmoid = nn.Sigmoid()\n\n self.enable = True\n self.group_size = group_size\n\n def change_n_bits(self, n_bits):\n self.n_bits = n_bits\n if self.disable_zero_point:\n self.qmin = -(2 ** (n_bits - 1))\n self.qmax = 2 ** (n_bits - 1) - 1\n else:\n self.qmin = 0\n self.qmax = 2 ** (n_bits) - 1\n\n def fake_quant(self, x, scale, round_zero_point):\n if self.deficiency > 0:\n pad_zeros = torch.zeros((x.shape[0],self.deficiency),dtype=x.dtype,device=x.device)\n x = torch.cat((x,pad_zeros),dim=1)\n \n if self.group_size:\n assert len(x.shape)==2, \"only support linear layer now\"\n dim1, dim2 = x.shape\n x = x.reshape(-1, self.group_size)\n x_int = round_ste(x / scale)\n if round_zero_point is not None:\n x_int = x_int.add(round_zero_point)\n x_int = x_int.clamp(self.qmin, self.qmax)\n x_dequant = x_int\n if round_zero_point is not None:\n x_dequant = x_dequant.sub(round_zero_point)\n x_dequant = x_dequant.mul(scale)\n if self.group_size:\n x_dequant = x_dequant.reshape(dim1, dim2)\n if self.deficiency > 0:\n x_dequant = x_dequant[:,:-self.deficiency]\n return x_dequant\n \n\n def forward(self, x: torch.Tensor):\n if self.n_bits >= 16 or not self.enable:\n return x\n if self.metric == \"fix0to1\":\n return x.mul_(2**self.n_bits-1).round_().div_(2**self.n_bits-1)\n\n if self.dynamic_method == \"per_token\" or self.dynamic_method == \"per_channel\":\n self.per_token_dynamic_calibration(x)\n else:\n raise NotImplementedError() \n\n x_dequant = self.fake_quant(x, self.scale, self.round_zero_point)\n return x_dequant\n\n def per_token_dynamic_calibration(self, x):\n if self.group_size:\n if self.deficiency == 0:\n x = x.reshape(-1,self.group_size)\n else:\n pad_zeros = torch.zeros((x.shape[0],self.deficiency),dtype=x.dtype,device=x.device)\n x = torch.cat((x,pad_zeros),dim=1)\n x = x.reshape(-1,self.group_size)\n reduce_shape = [-1]\n xmin = x.amin(reduce_shape, keepdim=True)\n xmax = x.amax(reduce_shape, keepdim=True)\n if self.lwc:\n xmax = self.sigmoid(self.upbound_factor)*xmax\n xmin = self.sigmoid(self.lowbound_factor)*xmin\n if self.symmetric:\n abs_max = torch.max(xmax.abs(),xmin.abs())\n scale = abs_max / (2**(self.n_bits-1)-1)\n self.scale = scale.clamp(min=CLIPMIN, max=1e4)\n zero_point = (2**(self.n_bits-1)-1)*torch.ones_like(self.scale)\n else:\n range = xmax - xmin\n scale = range / (2**self.n_bits-1)\n self.scale = scale.clamp(min=CLIPMIN, max=1e4)\n zero_point = -(xmin) / (self.scale)\n if self.disable_zero_point:\n self.round_zero_point = None\n else:\n self.round_zero_point = zero_point.clamp(min=-1e4, max=1e4).round()\n \n def register_scales_and_zeros(self):\n self.register_buffer('scales', self.scale)\n self.register_buffer('zeros', self.round_zero_point)\n del self.scale\n del self.round_zero_point\n" }, { "path": "quant/utils.py", "content": "from collections import OrderedDict\nfrom quant.int_linear import QuantLinear\nimport torch\nfrom quant.int_matmul import QuantMatMul\nfrom models.transformation import *\n\n\ndef let_parameters(model, use_shift=True):\n params = []\n template = \"smooth\" if use_shift else \"smooth_scale\"\n for n, m in model.named_parameters():\n if n.find(template) > -1:\n params.append(m)\n return iter(params) \n\ndef lwc_parameters(model):\n params = []\n for n, m in model.named_parameters():\n if n.find('bound_factor') > -1:\n params.append(m)\n return iter(params) \n\ndef get_omni_parameters(model, use_shift=True):\n params = []\n template = \"smooth\" if use_shift else \"smooth_scale\"\n for n, m in model.named_parameters():\n if n.find('bound_factor') > -1 or n.find(template) > -1:\n params.append(m)\n return iter(params) \n\ndef omni_state_dict(model, destination=None, prefix='', keep_vars=False):\n if destination is None:\n destination = OrderedDict()\n for name, param in model.named_parameters():\n if name.find('smooth') > -1 or name.find('bound_factor') > -1:\n destination[prefix + name] = param if keep_vars else param.detach()\n return destination\n\ndef register_scales_and_zeros(model):\n for name, module in model.named_modules():\n if isinstance(module, QuantLinear):\n module.weight_quantizer.register_scales_and_zeros()\n\nclass TruncateFunction(torch.autograd.Function):\n @staticmethod\n def forward(ctx, input, threshold):\n truncated_tensor = input.clone()\n truncated_tensor[truncated_tensor.abs() < threshold] = truncated_tensor[truncated_tensor.abs() < threshold].sign() * threshold\n return truncated_tensor\n \n\n @staticmethod\n def backward(ctx, grad_output):\n grad_input = grad_output.clone()\n return grad_input, None\n\n \ndef truncate_number(number, threshold=1e-2):\n # avoid overflow with AMP training\n return TruncateFunction.apply(number, threshold) \n\ndef smooth_and_quant_temporary(model, args, isllama):\n if args.let:\n with torch.no_grad():\n for name, module in model.named_parameters():\n if \"smooth_scale\" in name:\n module.data = truncate_number(module)\n if isllama:\n smooth_ln_fcs_temporary(model.input_layernorm,[model.self_attn.q_proj, model.self_attn.k_proj, model.self_attn.v_proj],\n model.qkv_smooth_scale,model.qkv_smooth_shift)\n smooth_ln_fcs_temporary(model.post_attention_layernorm,[model.mlp.up_proj,model.mlp.gate_proj],\n model.fc1_smooth_scale,model.fc1_smooth_shift)\n smooth_fc_fc_temporary(model.self_attn.v_proj,model.self_attn.o_proj,\n model.out_smooth_scale, model.out_smooth_shift)\n smooth_q_k_temporary(model.self_attn.q_proj, model.self_attn.k_proj,\n model.qkt_smooth_scale)\n model.mlp.down_proj.temp_weight = model.mlp.down_proj.weight\n else:\n smooth_ln_fcs_temporary(model.self_attn_layer_norm,[model.self_attn.q_proj, model.self_attn.k_proj, model.self_attn.v_proj],\n model.qkv_smooth_scale,model.qkv_smooth_shift)\n smooth_ln_fcs_temporary(model.final_layer_norm,[model.fc1],\n model.fc1_smooth_scale,model.fc1_smooth_shift)\n smooth_ln_fcs_temporary(model.self_attn.v_proj,model.self_attn.out_proj,\n model.out_smooth_scale, model.out_smooth_shift)\n smooth_q_k_temporary(model.self_attn.q_proj, model.self_attn.k_proj,\n model.qkt_smooth_scale)\n model.fc2.temp_weight = model.fc2.weight\n else:\n for name, module in model.named_modules():\n if isinstance(module, QuantLinear):\n module.temp_weight = module.weight\n # quant\n for name, module in model.named_modules():\n if isinstance(module, QuantLinear):\n if hasattr(module, \"temp_weight\"):\n module.temp_weight = module.weight_quantizer(module.temp_weight)\n else:\n module.temp_weight = module.weight_quantizer(module.weight)\n if not hasattr(module, \"temp_bias\"):\n module.temp_bias = module.bias\n module.use_temporary_parameter=True\n \ndef clear_temp_variable(model):\n for name, module in model.named_modules():\n if isinstance(module, QuantLinear):\n if hasattr(module, \"temp_weight\"):\n del module.temp_weight\n if hasattr(module, \"temp_bias\"):\n del module.temp_bias\n\n@torch.no_grad() \ndef smooth_and_quant_inplace(model, args, isllama):\n if args.let:\n for name, module in model.named_parameters():\n if \"smooth_scale\" in name:\n module.data = truncate_number(module)\n if isllama:\n smooth_ln_fcs_inplace(model.input_layernorm,[model.self_attn.q_proj, model.self_attn.k_proj, model.self_attn.v_proj],\n model.qkv_smooth_scale,model.qkv_smooth_shift)\n smooth_ln_fcs_inplace(model.post_attention_layernorm,[model.mlp.up_proj,model.mlp.gate_proj],\n model.fc1_smooth_scale,model.fc1_smooth_shift)\n smooth_fc_fc_inplace(model.self_attn.v_proj,model.self_attn.o_proj,\n model.out_smooth_scale, model.out_smooth_shift)\n else: # opt\n smooth_ln_fcs_inplace(model.self_attn_layer_norm,[model.self_attn.q_proj, model.self_attn.k_proj, model.self_attn.v_proj],\n model.qkv_smooth_scale,model.qkv_smooth_shift)\n smooth_ln_fcs_inplace(model.final_layer_norm,[model.fc1],\n model.fc1_smooth_scale,model.fc1_smooth_shift)\n smooth_fc_fc_inplace(model.self_attn.v_proj,model.self_attn.out_proj,\n model.out_smooth_scale, model.out_smooth_shift)\n smooth_q_k_inplace(model.self_attn.q_proj, model.self_attn.k_proj,\n model.qkt_smooth_scale)\n for name, module in model.named_modules():\n if isinstance(module, QuantLinear):\n module.weight = module.weight_quantizer(module.weight)\n module.use_temporary_parameter=False\n\ndef set_quant_state(self, weight_quant: bool = False, act_quant: bool = False):\n # setting weight quantization here does not affect actual forward pass\n self.use_weight_quant = weight_quant\n self.use_act_quant = act_quant\n for m in self.modules():\n if isinstance(m, (QuantLinear, QuantMatMul)):\n m.set_quant_state(weight_quant, act_quant)\n" }, { "path": "scripts/eval_fake_ptq.sh", "content": "# for fake quantization here: AWQ, QuIP, BiLLM, PB-LLM, DB-LLM \nmodel_path='LLMQ/LLaMA-3-8B-BiLLM-1.1bit-fake'\npython main.py --model ${model_path} --epochs 0 --output_dir ./log/--tasks 'hellaswag,piqa,winogrande,arc_easy,arc_challenge' --wbits 16 --abits 16 --eval_ppl --multigpu\n" }, { "path": "scripts/eval_irqlora_commonsenseqa.sh", "content": "tau_range=0.1\ntau_n=100\nblocksize2=256\n\nCUDA_VISIBLE_DEVICES=0 python main.py \\\n--model /home/inspur/lin/pretrained_models/llama-3-8b \\\n--peft /home/inspur/lin/codes/IR-QLoRA/output/llama-3-8b-irqlora/checkpoint-10000 \\\n--tau_range ${tau_range} --tau_n ${tau_n} --blocksize ${blocksize2} \\\n--epochs 0 --output_dir ./log/llama-3-8b-irqlora-${tau_range}-${tau_n}-${blocksize2} \\\n--wbits 4 \\\n--tasks piqa,arc_easy,arc_challenge,hellaswag,winogrande\n" }, { "path": "utils.py", "content": "import torch\n# from torch._six import inf\nfrom math import inf\nimport logging\nfrom termcolor import colored\nimport sys\nimport os\nimport time\n\n\n@torch.no_grad()\ndef ampscaler_get_grad_norm(parameters, norm_type: float = 2.0) -> torch.Tensor:\n if isinstance(parameters, torch.Tensor):\n parameters = [parameters]\n parameters = [p for p in parameters if p.grad is not None]\n norm_type = float(norm_type)\n if len(parameters) == 0:\n return torch.tensor(0.)\n device = parameters[0].grad.device\n if norm_type == inf:\n total_norm = max(p.grad.detach().abs().max().to(device) for p in parameters)\n else:\n total_norm = torch.norm(torch.stack([torch.norm(p.grad.detach(),\n norm_type).to(device) for p in parameters]), norm_type)\n return total_norm\n\nclass NativeScalerWithGradNormCount:\n state_dict_key = \"amp_scaler\"\n\n def __init__(self):\n self._scaler = torch.cuda.amp.GradScaler()\n\n def __call__(self, loss, optimizer, clip_grad=None, parameters=None, create_graph=False, update_grad=True,retain_graph=False):\n self._scaler.scale(loss).backward(create_graph=create_graph, retain_graph=retain_graph)\n if update_grad:\n if clip_grad is not None:\n assert parameters is not None\n self._scaler.unscale_(optimizer) # unscale the gradients of optimizer's assigned params in-place\n norm = torch.nn.utils.clip_grad_norm_(parameters, clip_grad)\n else:\n self._scaler.unscale_(optimizer)\n norm = ampscaler_get_grad_norm(parameters)\n self._scaler.step(optimizer)\n self._scaler.update()\n else:\n norm = None\n return norm\n\n def state_dict(self):\n return self._scaler.state_dict()\n\n def load_state_dict(self, state_dict):\n self._scaler.load_state_dict(state_dict)\n\n\ndef create_logger(output_dir, dist_rank=0, name=''):\n # create logger\n logger = logging.getLogger(name)\n logger.setLevel(logging.INFO)\n logger.propagate = False\n\n # create formatter\n fmt = '[%(asctime)s %(name)s] (%(filename)s %(lineno)d): %(levelname)s %(message)s'\n color_fmt = colored('[%(asctime)s %(name)s]', 'green') + \\\n colored('(%(filename)s %(lineno)d)', 'yellow') + ': %(levelname)s %(message)s'\n\n # create console handlers for master process\n if dist_rank == 0:\n console_handler = logging.StreamHandler(sys.stdout)\n console_handler.setLevel(logging.DEBUG)\n console_handler.setFormatter(\n logging.Formatter(fmt=color_fmt, datefmt='%Y-%m-%d %H:%M:%S'))\n logger.addHandler(console_handler)\n\n # create file handlers\n file_handler = logging.FileHandler(os.path.join(output_dir, f'log_rank{dist_rank}_{int(time.time())}.txt'), mode='a')\n file_handler.setLevel(logging.DEBUG)\n file_handler.setFormatter(logging.Formatter(fmt=fmt, datefmt='%Y-%m-%d %H:%M:%S'))\n logger.addHandler(file_handler)\n\n return logger" } ]