\n✅ From scratch implementations ✅ No abstractions ✅ Beginner friendly\n ✅ Flash attention ✅ FSDP ✅ LoRA, QLoRA, Adapter\n✅ Reduce GPU memory (fp4/8/16/32) ✅ 1-1000+ GPUs/TPUs ✅ 20+ LLMs \n\n\n\n---\n\n\n\n [](https://github.com/Lightning-AI/lit-stablelm/blob/master/LICENSE) [](https://discord.gg/VptPCZkGNa)\n\n
\n Quick start •\n Models •\n Finetune •\n Deploy •\n All workflows •\n Features •\n Recipes (YAML) •\n Lightning AI •\n Tutorials\n
\n\n \n\n\n![\"Get](\"https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/get-started-badge.svg\"){kind=icon}
\n\n\n \n\n![\"Run](\"https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/run-on-studio.svg\")
\n![\"Deploy](\"https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/deploy-on-studios.svg\")
\n![](\"images/0_to_litgpt/usage.webp\")
\n\n \n\n![](\"images/0_to_litgpt/commands.webp\")
\n\n \n\nHowever, it is also possible, and even common, to use and deploy models with LitGPT without pretraining and finetuning. So, if you are not interested in pretraining and finetuning, please feel free to skip these sections.\n\n\n\n\n\n \n## Install LitGPT\n\nLitGPT is available as a Python library from the PyPI package repository, and we recommend installing it using Python's `pip` installer module, including all required package dependencies:\n\n```bash\npip install 'litgpt[all]'\n```\n\nAlternatively, if you are a researcher or developer planning to make changes to LitGPT, you can clone the GitHub repository and install it from a local folder as follows:\n\n```\ngit clone https://github.com/Lightning-AI/litgpt.git\ncd litgpt\npip install -e '.[all]'\n```\n\n\n \n## Pretrain LLMs\n\nPretraining LLMs requires substantial compute resources and time commitment. For that reason, most researchers and practitioners prefer to skip this step and continue with the *Download pretrained model weights* section instead.\n\nHowever, if you feel adventurous and want to pretrain your own LLM, here's how.\n\nFirst, we have to decide which type of model architecture we want to use. We list the available architectures by using the `pretrain` command without any additional arguments:\n\n```bash\nlitgpt pretrain list\n```\n\nThis prints a list of all available model architectures in alphabetical order:\n\n```\nCamel-Platypus2-13B\nCamel-Platypus2-70B\nCodeLlama-13b-Python-hf\n...\nEleutherAI/pythia-410m\n...\nvicuna-13b-v1.3\nvicuna-13b-v1.5\nvicuna-13b-v1.5-16k\nvicuna-33b-v1.3\nvicuna-7b-v1.3\nvicuna-7b-v1.5\nvicuna-7b-v1.5-16k\n```\n\nSuppose we want to pretraining the 1.1B parameter small `tiny-llama-1.1b` model. Before starting finetuning, we must also choose and download a tokenizer.\n\nWe can download a tokenizer via the `download` command. Note that running `litgpt download list` will also print a list of all available models and tokenizers to download.\n\nTo filter for specific models, e.g., TinyLlama, we can use the `grep` command in our terminal:\n\n```bash\nlitgpt download list | grep TinyLlama\n```\n\nThis prints\n\n```\nTinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T\nTinyLlama/TinyLlama-1.1B-Chat-v1.0\n```\n\nLet's now download the tokenizer corresponding to `TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T` that we can then use to pretrain the TinyLlama model:\n\n```\nlitgpt download \\\n TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T \\\n --tokenizer_only true\n```\n\n(when specified)\n\n \n\n![](\"images/0_to_litgpt/pretrain.webp\")
\n\n \n\nNext, we can pretrain the model on the OpenWebText dataset with the default setting as follows:\n\n```bash\nlitgpt pretrain tiny-llama-1.1b \\\n --data OpenWebText \\\n --tokenizer_dir TinyLlama/TinyLlama-1.1B-intermediate-step-1431k-3T\n```\n\nIf you are interested in additional settings, you can use the help command as follows:\n\n```\nlitgpt pretrain --help\n```\n\n \n\n> [!TIP]\n> Above, we only covered the most basic commands for pretraining a model using LitGPT. We highly recommend checking the resources below if you are interested in pretraining a model.\n\n \n\n**More information and additional resources**\n\n- [tutorials/pretrain](./pretrain.md): General information about pretraining in LitGPT\n- [tutorials/pretrain_tinyllama](./pretrain_tinyllama.md): A tutorial for finetuning a 1.1B TinyLlama model on 3 trillion tokens\n- [config_hub/pretrain](../config_hub/pretrain): Pre-made config files for pretraining that work well out of the box\n- Project templates in reproducible environments with multi-GPU and multi-node support:\n - [Prepare the TinyLlama 1T token dataset](https://lightning.ai/lightning-ai/studios/prepare-the-tinyllama-1t-token-dataset)\n - [Pretrain LLMs - TinyLlama 1.1B](https://lightning.ai/lightning-ai/studios/pretrain-llms-tinyllama-1-1b)\n - [Continued Pretraining with TinyLlama 1.1B](https://lightning.ai/lightning-ai/studios/continued-pretraining-with-tinyllama-1-1b)\n\n\n \n## Download pretrained model weights\n\nMost practical use cases, like LLM inference (/chat) or finetuning, involve using pretrained model weights. LitGPT supports a large number of model weights, which can be listed by executing the `download` with `list` as an argument:\n\n```bash\nlitgpt download list\n```\n\nThis will print a (long) list of all supported pretrained models (abbreviated for readability below):\n\n```\n..\ngoogle/gemma-2b\n...\nmeta-llama/Llama-2-7b-hf\n...\nmicrosoft/phi-2\n...\nmistralai/Mixtral-8x7B-Instruct-v0.1\n...\n```\n\nTo download the model weights, provide one of the model strings above as input argument:\n\n```bash\nlitgpt download microsoft/phi-2\n```\n\n```\nmodel-00001-of-00002.safetensors: 100%|████████████████████████████████| 5.00G/5.00G [00:40<00:00, 124MB/s]\nmodel-00002-of-00002.safetensors: 100%|████████████████████████████████| 564M/564M [00:01<00:00, 330MB/s]\ntokenizer.json: 100%|██████████████████████████████████████████████████| 2.11M/2.11M [00:00<00:00, 54.0MB/s]\n...\nConverting checkpoint files to LitGPT format.\nProcessing checkpoints/microsoft/phi-2/model-00001-of-00002.bin\n...\nSaving converted checkpoint to checkpoints/microsoft/phi-2\n```\n\n\n \n\n> [!TIP]\n> Note that some models, such as Llama 2, require that you accept Meta AI's terms of service for this model, and you need to use a special access token via the `litgpt download ... --access_token ...` option. For more information, visit the respective Model Hub website, e.g., [meta-llama/Llama-2-7b-hf](https://huggingface.co/meta-llama/Llama-2-7b-hf). The access token can be created under your Model Hub in the `Profile > Access Tokens` menu.\n\n \n\n\nBy default, the weights are going to be stored in a `./checkpoints` subdirectory:\n\n```bash\nls -lh checkpoints/microsoft/phi-2/\n```\n\n```\ntotal 11G\n-rw-r--r-- 1 sebastian sebastian 863 Mar 19 21:14 config.json\n-rw-r--r-- 1 sebastian sebastian 124 Mar 19 21:14 generation_config.json\n-rw-r--r-- 1 sebastian sebastian 5.2G Mar 19 21:15 lit_model.pth\n-rw-r--r-- 1 sebastian sebastian 4.7G Mar 19 21:15 model-00001-of-00002.bin\n-rw-r--r-- 1 sebastian sebastian 538M Mar 19 21:15 model-00002-of-00002.bin\n-rw-r--r-- 1 sebastian sebastian 528 Mar 19 21:15 model_config.yaml\n-rw-r--r-- 1 sebastian sebastian 2.1M Mar 19 21:14 tokenizer.json\n-rw-r--r-- 1 sebastian sebastian 7.2K Mar 19 21:14 tokenizer_config.json\n```\n\nThe model is now ready for inference and chat, for example, using the `chat` command on the checkpoint directory:\n\n```bash\nlitgpt chat microsoft/phi-2\n```\n\n```\nNow chatting with phi-2.\nTo exit, press 'Enter' on an empty prompt.\n\nSeed set to 1234\n>> Prompt: Why are LLMs so useful?\n>> Reply: When building applications or operating systems, you can use LLMs to know how a computer should respond to your commands. This can make your programs run faster and more efficiently.\n\nTime for inference: 1.26 sec total, 27.81 tokens/sec, 35 tokens\n\n>> Prompt:\n```\n \n\n> [!TIP]\n> Use `--multiline true` to support prompts that require multiple input lines.\n\n![](\"images/0_to_litgpt/instruction-1.webp\")
\n\n \n\nAlternatively, datasets for instruction finetuning can also contain an `'input'` field:\n\nIn an instruction-finetuning context, \"full\" finetuning means updating all model parameters as opposed to only a subset. Adapter and LoRA (short for low-rank adaptation) are methods for parameter-efficient finetuning that only require updating a small fraction of the model weights.\n\n \n\n![](\"images/0_to_litgpt/finetune.webp\")
\n\n \n\nParameter-efficient finetuning is much more resource-efficient and cheaper than full finetuning, and it often results in the same good performance on downstream tasks.\n\nIn the following example, we will use LoRA for finetuning, which is one of the most popular LLM finetuning methods. (For more information on how LoRA works, please see [Code LoRA from Scratch](https://lightning.ai/lightning-ai/studios/code-lora-from-scratch).)\n\nBefore we start, we have to download a model as explained in the previous \"Download pretrained model\" section above:\n\n```bash\nlitgpt download microsoft/phi-2\n```\n\nThe LitGPT interface can be used via command line arguments and configuration files. We recommend starting with the configuration files from the [config_hub](../config_hub) and either modifying them directly or overriding specific settings via the command line. For example, we can use the following setting to train the downloaded 2.7B parameter `microsoft/phi-2` model, where we set `--max_steps 5` for a quick test run.\n\nIf you have downloaded or cloned the LitGPT repository, you can provide the `config` file via a relative path:\n\n```bash\nlitgpt finetune_lora microsoft/phi-2\\\n --config config_hub/finetune/phi-2/lora.yaml \\\n --train.max_steps 5\n```\n\nAlternatively, you can provide a URL:\n\n```bash\nlitgpt finetune_lora microsoft/phi-2\\\n --config https://raw.githubusercontent.com/Lightning-AI/litgpt/main/config_hub/finetune/phi-2/lora.yaml \\\n --train.max_steps 5\n```\n\n\n \n\n\n> [!TIP]\n> Note that the config file above will finetune the model on the `Alpaca2k` dataset on 1 GPU and save the resulting files in an `out/finetune/lora-phi-2` directory. All of these settings can be changed via a respective command line argument or by changing the config file.\n> To see more options, execute `litgpt finetune_lora --help`.\n\n \n\nRunning the previous finetuning command will initiate the finetuning process, which should only take about a minute on a GPU due to the `--train.max_steps 5` setting.\n\n```\n{'checkpoint_dir': PosixPath('checkpoints/microsoft/phi-2'), # TODO\n 'data': Alpaca2k(mask_prompt=False,\n val_split_fraction=0.03847,\n prompt_style=![\"Open](\"https://pl-bolts-doc-images.s3.us-east-2.amazonaws.com/app-2/studio-badge.svg\"){kind=icon}
\n![](\"images/prepare_dataset/alpaca.jpg\")
\n\nIn this case, a cut-off of 256 may be a reasonable choice:\n\n```bash\nlitgpt finetune_lora tiiuae/falcon-7b \\\n --data Alpaca \\\n --train.max_seq_length 256\n```\n\nFor comparison, the Falcon 7B model requires 23.52 GB of memory for the original Alpaca dataset and 15.73 GB of memory for the truncated Alpaca dataset when finetuning with LoRA using a micro batchsize of 1 and bfloat-16 precision.\n\n \n\n### Alpaca-2k\n\n[Alpaca-2k](https://huggingface.co/datasets/mhenrichsen/alpaca_2k_test) is a smaller, 2000-sample subset of Alpaca described above.\n\n```bash\nlitgpt finetune_lora \"tiiuae/falcon-7b\" \\\n --data Alpaca2k\n```\n\n \n\n> [!TIP]\n> Use `litgpt_finetune --data.help Alpaca2k` to list additional dataset-specific command line options.\n\n \n\nThe Alpaca-2k dataset distribution is shown below.\n\n![](\"images/prepare_dataset/alpaca-2k.jpg\")
\n\n\n### Alpaca-GPT4\n\nThe Alpaca-GPT4 was built by using the prompts of the original Alpaca dataset and generate the responses via GPT 4. The\ndataset consists of 52,000 instructions and responses.\n\nThe original [Alpaca-GPT4](https://github.com/Instruction-Tuning-with-GPT-4/GPT-4-LLM) dataset can be used as follows:\n\n```bash\nlitgpt finetune lora \"tiiuae/falcon-7b\" \\\n --data AlpacaGPT4\n```\n\n \n\n> [!TIP]\n> Use `litgpt_finetune --data.help AlpacaGPT4` to list additional dataset-specific command line options.\n\n \n\nThe Alpaca-GPT4 dataset distribution is shown below.\n\n![](\"images/prepare_dataset/alpacagpt4.jpg\")
\n\n \n\n### Alpaca Libre\n\n[Alpaca Libre](https://github.com/mobarski/alpaca-libre) is a reimplementation or alternative to Alpaca using the same formatting.\n\nTo use Alpaca Libre instead of the original Alpaca dataset, use the following command:\n\n```bash\nlitgpt finetune_lora tiiuae/falcon-7b \\\n --data Alpaca \\\n --data.file_url \"https://raw.githubusercontent.com/mobarski/alpaca-libre/main/data/output/alpaca_libre_ok_tasks_v4.json\" \\\n --data.file_name \"alpaca_libre_data_cleaned_archive.json\"\n```\n\n \n\n> [!TIP]\n> Use `litgpt finetune --data.help Alpaca` to list additional dataset-specific command line options.\n\n \n\nThe Alpaca Libre dataset distribution is shown below.\n\n![](\"images/prepare_dataset/alpaca_libre.jpg\")
\n\nYou may want to consider truncating the dataset (see the *Truncating datasets* discussion in the Alpaca section for more information.) For this dataset, a cut-off of 256 may be a good choice:\n\n```bash\nlitgpt finetune_lora tiiuae/falcon-7b \\\n --data Alpaca \\\n --data.file_url \"https://raw.githubusercontent.com/mobarski/alpaca-libre/main/data/output/alpaca_libre_ok_tasks_v4.json\" \\\n --data.file_name \"alpaca_libre_data_cleaned_archive.json\" \\\n --train.max_seq_length 256\n```\n\n \n\n### Deita\n\nThe Deita dataset (short for Data-Efficient Instruction Tuning for Alignment) is a collection of 9500 prompts and responses, as described in the [What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning](https://arxiv.org/abs/2312.15685) paper.\nUsing Falcon 7b as an example, we can use the dataset as follows:\n\n```bash\nlitgpt finetune_lora tiiuae/falcon-7b \\\n --data Deita\n```\n\n \n\n\n> [!TIP]\n> Use `litgpt finetune --data.help Deita` to list additional dataset-specific command line options.\n\n \n\nDeita contains multiturn conversations. By default, only the first instruction-response pairs from\neach of these multiturn conversations are included. If you want to override this behavior and include the follow-up instructions\nand responses, set `--data.include_multiturn_conversations True`, which will include all multiturn conversations as regular\nprompt-response pairs. Considering the multiturn-answers, the dataset consists of 209,272 prompt-response pairs.\n\nThe Deita dataset distribution without including multit-turn conversations is shown below.\n\n![](\"images/prepare_dataset/deita.jpg\")
\n\nThe Deita dataset distribution including multit-turn conversations is depicted in the following histogram.\n\n![](\"images/prepare_dataset/deita-multiturn.jpg\")
\n\nYou may want to consider truncating the dataset (see the *Truncating datasets* discussion in the Alpaca section for more information.) For this dataset, a cut-off of 512 may be a good choice:\n\n```bash\nlitgpt finetune_lora tiiuae/falcon-7b \\\n --data Deita \\\n --train.max_seq_length 512\n```\n\n \n\n### Dolly\n\nThe Dolly dataset is a publicly available collection of 15k instruction-following entries created by Databricks. It spans multiple behavioral domains, as described in the [InstructGPT paper](https://arxiv.org/abs/2203.02155) paper. These include areas like brainstorming, classification, closed QA, content creation, information retrieval, open QA, and summary generation.\n\nThe usage is similar to the Alpaca dataset described above. Using Falcon 7b as an example, we can use the dataset as follows:\n\n```bash\nlitgpt finetune_lora tiiuae/falcon-7b \\\n --data Dolly\n```\n\n \n\n> [!TIP]\n> Use `litgpt finetune --data.help Dolly` to list additional dataset-specific command line options.\n\n \n\nThe Dolly dataset distribution is shown below.\n\n![](\"images/prepare_dataset/dolly.jpg\")
\n\nYou may want to consider truncating the dataset (see the *Truncating datasets* discussion in the Alpaca section for more information.) For this dataset, a cut-off of 512 may be a good choice:\n\n```bash\nlitgpt finetune_lora tiiuae/falcon-7b \\\n --data Dolly \\\n --train.max_seq_length 256\n```\n\n \n\n### LongForm\n\nLongForm is a semi-synthetic dataset based on raw text corpora for which the instructions were generated via an LLM. For more details about the instruction-generation process, please refer to the [LongForm research paper](https://arxiv.org/abs/2304.08460) by Köksal et al. According to the research paper, a Llama 7B model trained on LongForm achieves substantially better performance than the same Llama model trained on the 2x larger Alpaca dataset.\n\nLongForm consists of 23,652 training samples, 2,042 validation samples, and 2,045 test samples. (In LitGPT, the validation samples are currently not used.)\n\nThe more detailed dataset composition is as follows based on a table taken from the [dataset repository](https://github.com/akoksal/LongForm):\n\n| **Type** | **Source** | **Number of Examples** |\n|------------------------|----------------|------------------------|\n| **Corpora** | C4 | 10,000 |\n| | Wikipedia | 5,000 |\n| **Structured Corpora** | Stack Exchange | 4,380 |\n| | WikiHow | 2,500 |\n| **Tasks** | NIv2 | 3,684 |\n| | Big Bench | 600 |\n| | BEA-GEC | 1,203 |\n| | Enron | 372 |\n| **Total** | | 27,739 |\n| | | |\n| **Train** | | 23,652 |\n| **Validation** | | 2,042 |\n| **Test** | | 2,045 |\n\nLicense information is not provided but would depend on the individual subsets listed above.\n\nThe LongForm dataset distribution is shown below.\n\n![](\"images/prepare_dataset/longform.jpg\")
\n\nYou may want to consider truncating the dataset (see the *Truncating datasets* discussion in the Alpaca section for more information.) For this dataset, a cut-off of 1500 may be a good choice:\n\n```bash\nlitgpt finetune_lora tiiuae/falcon-7b \\\n --data LongForm \\\n --train.max_seq_length 1500\n```\n\n \n\n> [!TIP]\n> Use `litgpt finetune --data.help LongForm` to list additional dataset-specific command line options.\n\n \n\n \n\n### LIMA\n\nThe LIMA dataset is a collection of 1,000 carefully curated prompts and responses, as described in the [LIMA: Less Is More for Alignment](https://arxiv.org/abs/2305.11206) paper. The dataset is sourced from three community Q&A websites: Stack Exchange, wikiHow, and the Pushshift Reddit Dataset. In addition, it also contains prompts and answers written and collected by the authors of the LIMA paper.\n\nThe usage is similar to the Dolly dataset described above except that it requires an Hugging Face access token that you need to copy & paste from your Hugging Face account. Using Falcon 7b as an example, we can use the dataset as follows:\n\n```bash\nexport HF_TOKEN=\"insert_your_huggingface_token_here\"\n\nlitgpt finetune lora \\\n --data LIMA \\\n --checkpoint_dir \"tiiuae/falcon-7b\"\n```\n\n \n\n> [!TIP]\n> Use `litgpt finetune --data.help LIMA` to list additional dataset-specific command line options.\n\n \n\nLIMA contains a handful of multiturn conversations. By default, only the first instruction-response pairs from\neach of these multiturn conversations are included. If you want to override this behavior and include the follow-up instructions\nand responses, set `--data.include_multiturn_conversations True`.\n\nThe LIMA dataset distribution is shown below.\n\n![](\"images/prepare_dataset/lima.jpg\")
\n\nYou may want to consider truncating the dataset (see the *Truncating datasets* discussion in the Alpaca section for more information.) For this dataset, a cut-off of 512 may be a good choice:\n\n```bash\nlitgpt finetune_lora tiiuae/falcon-7b \\\n --data LIMA \\\n --train.max_seq_length 512\n```\n\n \n\n### FLAN\n\nFLAN is a collection of several dataset subsets by Google. In particular, the provided script in LitGPT loads the subsets from\n[here](https://huggingface.co/datasets/Muennighoff/flan).\n\nBy default, all subsets (1,386,050 samples) and validations sets (367,190 subsets) are combined into a single dataset:\n\n```bash\nlitgpt finetune_lora tiiuae/falcon-7b \\\n --data FLAN\n```\n\nHowever, you can also select individual subsets via comma-separated strings as follows:\n\n```bash\nlitgpt finetune lora tiiuae/falcon-7b \\\n --data FLAN \\\n --data.subsets \"aeslc_10templates,ag_news_subset_10templates,anli_r1_10templates\"\n```\n\n \n\n> [!TIP]\n> Use `litgpt finetune --data.help FLAN` to list additional dataset-specific command line options.\n\n \n\nYou can find a list of all 66 supported subsets [here](https://huggingface.co/datasets/Muennighoff/flan).\n\n \n\n## Preparing Custom Datasets for Instruction Finetuning\n\nThe models in LitGPT expect datasets for instruction finetuning in the following format:\n\n```text\n[\n {\n \"instruction\": \"Write a limerick about a\n pelican.”,\n \"input\": \"\",\n \"output\": \"There once was a pelican so fine,\n \\nHis beak was as colorful as\n sunshine,\\nHe would fish all day,\\nIn\n a very unique way,\\nThis pelican was\n truly divine!\\n\\n\\n\"\n },\n {\n \"instruction\": \"Identify the odd one out from\n the group.\",\n \"input\": \"Carrot, Apple, Banana, Grape\",\n \"output\": \"Carrot\\n\\n\"\n },\n]\n```\n\n(Note that depending on the task, the `\"input\"` text can be an empty string, as shown above.)\n\nYou can use your own data in LitGPT by either reading in a JSON file in the format shown above or by implementing a custom `DataModule`.\n\n \n\n### Preparing Custom Datasets From a JSON File\n\nYou can prepare custom dataset using a JSON file where each row is a dictionary with these keys:\n\n- `instruction`: Column which will describe the task.\n- `input`: A string holding a special input value for the instruction. This applies to some samples, and in others, this is empty (empty string).\n- `output`: The expected response\n\n> If any of the fields are missing, then the script will fail to read the dataset.\n\nThen simply run any of the finetuning scripts with this input:\n\n```bash\nlitgpt finetune_lora tiiuae/falcon-7b \\\n --data JSON \\\n --data.json_path path/to/your/data.json \\\n --data.val_split_fraction 0.1\n```\n\nYou can also customize how the dataset is read by using these additional parameters\n\n- `val_split_fraction`: The fraction of the data to split. Defaults to `0.1`\n\n- `seed`: The seed value to reproduce the same random splits for train and test data.\n\n- `mask_inputs`: Whether to mask the prompt section from the label (with `ignore_index`).\n\n- `ignore_index`: The index to use for labels that should be ignored. Defaults to `-100` (used when `mask_inputs` is `True`).\n\nTo use the settings described above, you can add the respective command line arguments when calling the finetuning scripts as shown in the example below:\n\n```bash\nlitgpt finetune_lora tiiuae/falcon-7b \\\n --data JSON \\\n --data.json_path path/to/your/data.json \\\n --data.val_split_fraction 0.1 \\\n --data.seed 42 \\\n --data.mask_inputs False \\\n --data.ignore_index -100\n```\n\nYou can also pass a directory containing a `train.json` and `val.json` to `--data.json_path` to define a fixed train/val split.\n\n \n\n> [!TIP]\n> Use `litgpt finetune --data.help JSON` to list additional dataset-specific command line options.\n\n \n\n### Preparing Custom Datasets Using DataModule\n\nIf you don't have a JSON file following the format described in the previous section, the easiest way to prepare a new dataset is to copy and modify one of the existing data modules in LitGPT:\n\n- [`litgpt/data/alpaca.py`](https://github.com/Lightning-AI/litgpt/blob/main/litgpt/data/alpaca.py) (if you plan to load a dataset from a JSON file);\n- [`litgpt/data/lima.py`](https://github.com/Lightning-AI/litgpt/blob/main/litgpt/data/lima.py) (if you plan to load a dataset using the `datasets` Python library).\n\nNote that you only need to modify a small fraction of the code file, namely the portion that downloads and formats the training data (see the `prepare_data` and `setup()` methods).\n\n \n\n## Preparing Pretraining Datasets\n\nIn addition to the finetuning dataset described above, LitGPT also supports several datasets for pretraining. The pretraining datasets are described in more detail in the following separate tutorial documents:\n\n- [Pretrain TinyLlama on Slimpajama and Starcoder](./pretrain_tinyllama.md)\n"
},
{
"path": "tutorials/pretrain.md",
"content": "# Pretrain LLMs with LitGPT\n\n\nThis document explains how to pretrain LLMs using LitGPT.\n\n \n## Using the `litgpt pretrain` command\n\nYou can pretrain models in LitGPT using the `litgpt pretrain` API starting with any of the available architectures listed by calling `litgpt pretrain list` without any additional arguments:\n\n \n> [!TIP]\n> To install all required dependencies before pretraining, first run `pip install \"litgpt[all]\"`.\n \n\n```bash\nlitgpt pretrain list\n```\n\nShown below is an abbreviated list:\n\n```\nValueError: Please specify --model_name ![](\"https://pl-public-data.s3.amazonaws.com/assets_litgpt/readme/3.webp\")
![](\"https://pl-public-data.s3.amazonaws.com/assets_litgpt/readme/4.webp\")
![](\"https://pl-public-data.s3.amazonaws.com/assets_litgpt/readme/1.webp\")