Repository: huggingface/skills Branch: main Commit: 73246ad6dc1a Files: 129 Total size: 1.1 MB Directory structure: gitextract_brvqcrv9/ ├── .claude-plugin/ │ ├── marketplace.json │ └── plugin.json ├── .cursor-plugin/ │ ├── marketplace.json │ └── plugin.json ├── .github/ │ └── workflows/ │ ├── SECURITY.md │ ├── generate-agents.yml │ ├── push-evals-leaderboard.yml │ ├── push-hackers-leaderboard.yml │ └── push-quests.yml ├── .gitignore ├── .mcp.json ├── LICENSE ├── README.md ├── agents/ │ └── AGENTS.md ├── apps/ │ ├── evals-leaderboard/ │ │ ├── app.py │ │ ├── collect_evals.py │ │ └── requirements.txt │ ├── hackers-leaderboard/ │ │ ├── README.md │ │ ├── app.py │ │ ├── collect_points.py │ │ └── requirements.txt │ └── quests/ │ ├── 02_evaluate-hub-model.md │ ├── 03_publish-hub-dataset.md │ ├── 04_sft-finetune-hub.md │ └── README.md ├── gemini-extension.json ├── hf-mcp/ │ └── skills/ │ └── hf-mcp/ │ └── SKILL.md ├── scripts/ │ ├── AGENTS_TEMPLATE.md │ ├── UV_RULES.md │ ├── generate_agents.py │ ├── generate_cursor_plugin.py │ ├── publish.sh │ └── run_skills_help.py └── skills/ ├── hf-cli/ │ └── SKILL.md ├── hugging-face-dataset-viewer/ │ └── SKILL.md ├── hugging-face-datasets/ │ ├── SKILL.md │ ├── examples/ │ │ ├── diverse_training_examples.json │ │ ├── system_prompt_template.txt │ │ └── training_examples.json │ ├── scripts/ │ │ ├── dataset_manager.py │ │ └── sql_manager.py │ └── templates/ │ ├── chat.json │ ├── classification.json │ ├── completion.json │ ├── custom.json │ ├── qa.json │ └── tabular.json ├── hugging-face-evaluation/ │ ├── SKILL.md │ ├── examples/ │ │ ├── USAGE_EXAMPLES.md │ │ ├── artificial_analysis_to_hub.py │ │ ├── example_readme_tables.md │ │ └── metric_mapping.json │ └── scripts/ │ ├── evaluation_manager.py │ ├── inspect_eval_uv.py │ ├── inspect_vllm_uv.py │ ├── lighteval_vllm_uv.py │ ├── run_eval_job.py │ ├── run_vllm_eval_job.py │ └── test_extraction.py ├── hugging-face-jobs/ │ ├── SKILL.md │ ├── index.html │ ├── references/ │ │ ├── hardware_guide.md │ │ ├── hub_saving.md │ │ ├── token_usage.md │ │ └── troubleshooting.md │ └── scripts/ │ ├── cot-self-instruct.py │ ├── finepdfs-stats.py │ └── generate-responses.py ├── hugging-face-model-trainer/ │ ├── SKILL.md │ ├── references/ │ │ ├── gguf_conversion.md │ │ ├── hardware_guide.md │ │ ├── hub_saving.md │ │ ├── local_training_macos.md │ │ ├── reliability_principles.md │ │ ├── trackio_guide.md │ │ ├── training_methods.md │ │ ├── training_patterns.md │ │ ├── troubleshooting.md │ │ └── unsloth.md │ └── scripts/ │ ├── convert_to_gguf.py │ ├── dataset_inspector.py │ ├── estimate_cost.py │ ├── train_dpo_example.py │ ├── train_grpo_example.py │ ├── train_sft_example.py │ └── unsloth_sft_example.py ├── hugging-face-paper-pages/ │ └── SKILL.md ├── hugging-face-paper-publisher/ │ ├── SKILL.md │ ├── examples/ │ │ └── example_usage.md │ ├── references/ │ │ └── quick_reference.md │ ├── scripts/ │ │ └── paper_manager.py │ └── templates/ │ ├── arxiv.md │ ├── ml-report.md │ ├── modern.md │ └── standard.md ├── hugging-face-tool-builder/ │ ├── SKILL.md │ └── references/ │ ├── baseline_hf_api.py │ ├── baseline_hf_api.sh │ ├── baseline_hf_api.tsx │ ├── find_models_by_paper.sh │ ├── hf_enrich_models.sh │ ├── hf_model_card_frontmatter.sh │ └── hf_model_papers_auth.sh ├── hugging-face-trackio/ │ ├── .claude-plugin/ │ │ └── plugin.json │ ├── SKILL.md │ └── references/ │ ├── alerts.md │ ├── logging_metrics.md │ └── retrieving_metrics.md ├── hugging-face-vision-trainer/ │ ├── SKILL.md │ ├── references/ │ │ ├── finetune_sam2_trainer.md │ │ ├── hub_saving.md │ │ ├── image_classification_training_notebook.md │ │ ├── object_detection_training_notebook.md │ │ ├── reliability_principles.md │ │ └── timm_trainer.md │ └── scripts/ │ ├── dataset_inspector.py │ ├── estimate_cost.py │ ├── image_classification_training.py │ ├── object_detection_training.py │ └── sam_segmentation_training.py ├── huggingface-gradio/ │ ├── SKILL.md │ └── examples.md └── transformers.js/ ├── SKILL.md └── references/ ├── CACHE.md ├── CONFIGURATION.md ├── EXAMPLES.md ├── MODEL_ARCHITECTURES.md ├── PIPELINE_OPTIONS.md └── TEXT_GENERATION.md ================================================ FILE CONTENTS ================================================ ================================================ FILE: .claude-plugin/marketplace.json ================================================ { "name": "huggingface-skills", "owner": { "name": "Hugging Face" }, "metadata": { "description": "Agent Skills for AI/ML tasks including dataset creation, model training, evaluation, and research paper publishing on Hugging Face Hub", "version": "1.0.1" }, "plugins": [ { "name": "hugging-face-model-trainer", "source": "./skills/hugging-face-model-trainer", "skills": "./", "description": "Train or fine-tune language models using TRL on Hugging Face Jobs infrastructure. Covers SFT, DPO, GRPO and reward modeling training methods, plus GGUF conversion for local deployment. Includes hardware selection, cost estimation, Trackio monitoring, and Hub persistence." }, { "name": "hugging-face-paper-publisher", "source": "./skills/hugging-face-paper-publisher", "skills": "./", "description": "Publish and manage research papers on Hugging Face Hub. Supports creating paper pages, linking papers to models/datasets, claiming authorship, and generating professional markdown-based research articles." }, { "name": "hugging-face-paper-pages", "source": "./skills/hugging-face-paper-pages", "skills": "./", "description": "Look up and read Hugging Face paper pages in markdown, and use the papers API for structured metadata like authors, linked models, datasets, Spaces, and media URLs when needed." }, { "name": "hugging-face-datasets", "source": "./skills/hugging-face-datasets", "skills": "./", "description": "Create and manage datasets on Hugging Face Hub. Supports initializing repos, defining configs/system prompts, streaming row updates, and SQL-based dataset querying/transformation." }, { "name": "hugging-face-evaluation", "source": "./skills/hugging-face-evaluation", "skills": "./", "description": "Add and manage evaluation results in Hugging Face model cards. Supports extracting eval tables from README content, importing scores from Artificial Analysis API, and running custom evaluations with vLLM/lighteval." }, { "name": "hugging-face-tool-builder", "source": "./skills/hugging-face-tool-builder", "skills": "./", "description": "Build reusable scripts for Hugging Face API operations. Useful for chaining API calls or automating repeated tasks." }, { "name": "hf-cli", "source": "./skills/hf-cli", "skills": "./", "description": "Execute Hugging Face Hub operations using the hf CLI. Download models/datasets, upload files, manage repos, and run cloud compute jobs." }, { "name": "hugging-face-jobs", "source": "./skills/hugging-face-jobs", "skills": "./", "description": "Run compute jobs on Hugging Face infrastructure. Execute Python scripts, manage scheduled jobs, and monitor job status." }, { "name": "hugging-face-trackio", "source": "./skills/hugging-face-trackio", "skills": "./", "description": "Track and visualize ML training experiments with Trackio. Log metrics via Python API and retrieve them via CLI. Supports real-time dashboards synced to HF Spaces." }, { "name": "hugging-face-dataset-viewer", "source": "./skills/hugging-face-dataset-viewer", "skills": "./", "description": "Explore, query, and extract data from any Hugging Face dataset using the Dataset Viewer REST API and npx tooling. Zero Python dependencies — covers split/config discovery, row pagination, text search, filtering, SQL via parquetlens, and dataset upload via CLI." }, { "name": "gradio", "source": "./skills/huggingface-gradio", "skills": "./", "description": "Build Gradio web UIs and demos in Python. Use when creating or editing Gradio apps, components, event listeners, layouts, or chatbots." }, { "name": "transformers-js", "source": "./skills/transformers.js", "skills": "./", "description": "Run state-of-the-art machine learning models directly in JavaScript/TypeScript for NLP, computer vision, audio processing, and multimodal tasks. Works in Node.js and browsers with WebGPU/WASM using Hugging Face models." }, { "name": "hugging-face-vision-trainer", "source": "./skills/hugging-face-vision-trainer", "skills": "./", "description": "Train and fine-tune object detection models (RTDETRv2, YOLOS, DETR and others) and image classification models (timm and transformers models — MobileNetV3, MobileViT, ResNet, ViT/DINOv3) using Transformers Trainer API on Hugging Face Jobs infrastructure or locally. Includes COCO dataset format support, Albumentations augmentation, mAP/mAR metrics, trackio tracking, hardware selection, and Hub persistence." } ] } ================================================ FILE: .claude-plugin/plugin.json ================================================ { "name": "huggingface-skills", "description": "Agent Skills for AI/ML tasks including dataset creation, model training, evaluation, and research paper publishing on Hugging Face Hub", "version": "1.0.1", "author": { "name": "Hugging Face" }, "homepage": "https://github.com/huggingface/skills", "repository": "https://github.com/huggingface/skills", "license": "Apache-2.0", "keywords": [ "huggingface", "machine-learning", "datasets", "training", "evaluation", "papers", "fine-tuning", "llm" ] } ================================================ FILE: .cursor-plugin/marketplace.json ================================================ { "name": "huggingface-skills", "owner": { "name": "Hugging Face" }, "metadata": { "description": "Agent Skills for AI/ML tasks including dataset creation, model training, evaluation, and research paper publishing on Hugging Face Hub", "version": "1.0.0" }, "plugins": [ { "name": "huggingface-skills", "source": ".", "skills": "skills", "description": "Agent Skills for AI/ML tasks including dataset creation, model training, evaluation, and research paper publishing on Hugging Face Hub" } ] } ================================================ FILE: .cursor-plugin/plugin.json ================================================ { "name": "huggingface-skills", "skills": "skills", "mcpServers": ".mcp.json", "description": "Agent Skills for AI/ML tasks including dataset creation, model training, evaluation, and research paper publishing on Hugging Face Hub", "version": "1.0.1", "author": { "name": "Hugging Face" }, "homepage": "https://github.com/huggingface/skills", "repository": "https://github.com/huggingface/skills", "license": "Apache-2.0", "keywords": [ "huggingface", "machine-learning", "datasets", "training", "evaluation", "papers", "fine-tuning", "llm" ] } ================================================ FILE: .github/workflows/SECURITY.md ================================================ # Security Policy ## Reporting a Vulnerability We take the security of this project seriously. If you discover a security vulnerability within this repository, please **do not open a public issue**. Instead, please report it through one of the following channels: * **Email:** security@huggingface.co * **Hugging Face Hub:** Use the "Report" feature on the repository’s Hub page if applicable. Please include a detailed description of the vulnerability and, if possible, a Proof of Concept (PoC) to help us reproduce the issue. ## Scope The following types of vulnerabilities are of particular interest: * **Remote Code Execution (RCE):** Vulnerabilities that allow an agent to execute unauthorized code outside of its intended environment. * **Credential Leakage:** Skills that inadvertently expose API keys, tokens, or sensitive environment variables. * **Injection Attacks:** Vulnerabilities in skills that handle user input for database queries or shell commands. ## Our Response Process 1. **Acknowledgment:** We will acknowledge your report within 48 business hours. 2. **Investigation:** Our team will investigate the issue and determine its impact. 3. **Fix and Disclosure:** Once a fix is ready, we will coordinate a release and, with your permission, credit you for the discovery. Thank you for helping keep the Hugging Face ecosystem safe! ================================================ FILE: .github/workflows/generate-agents.yml ================================================ name: Validate generated agent/plugin artifacts on: pull_request: paths: - "scripts/AGENTS_TEMPLATE.md" - "scripts/generate_agents.py" - "scripts/generate_cursor_plugin.py" - "scripts/publish.sh" - "**/SKILL.md" - "agents/AGENTS.md" - "README.md" - ".claude-plugin/marketplace.json" - ".claude-plugin/plugin.json" - "gemini-extension.json" - ".cursor-plugin/plugin.json" - ".mcp.json" jobs: validate: runs-on: ubuntu-latest steps: - name: Checkout uses: actions/checkout@v6 - name: Set up uv uses: astral-sh/setup-uv@v7 - name: Ensure generated files are up to date run: ./scripts/publish.sh --check ================================================ FILE: .github/workflows/push-evals-leaderboard.yml ================================================ name: Sync Evals Leaderboard to Hugging Face Space on: push: branches: - main paths: - 'apps/evals-leaderboard/**' workflow_dispatch: jobs: sync: runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v6 - name: Install huggingface_hub run: curl -LsSf https://hf.co/cli/install.sh | bash - name: Upload evals leaderboard to HF Space env: HF_TOKEN: ${{ secrets.HF_TOKEN }} run: hf upload hf-skills/distributed-leaderboard apps/evals-leaderboard . --repo-type=space ================================================ FILE: .github/workflows/push-hackers-leaderboard.yml ================================================ name: Sync Hackers Leaderboard to Hugging Face Space on: push: branches: - main paths: - 'apps/hackers-leaderboard/**' workflow_dispatch: jobs: sync: runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v6 - name: Install huggingface_hub run: curl -LsSf https://hf.co/cli/install.sh | bash - name: Upload hackers leaderboard to HF Space env: HF_TOKEN: ${{ secrets.HF_TOKEN }} run: hf upload hf-skills/hacker_leaderboard apps/hackers-leaderboard . --repo-type=space ================================================ FILE: .github/workflows/push-quests.yml ================================================ name: Sync Quests to Hugging Face Space on: push: branches: - main paths: - 'quests/**' workflow_dispatch: jobs: sync: runs-on: ubuntu-latest steps: - name: Checkout repository uses: actions/checkout@v6 - name: Install huggingface_hub run: curl -LsSf https://hf.co/cli/install.sh | bash - name: Upload quests to HF Space env: HF_TOKEN: ${{ secrets.HF_TOKEN }} run: hf upload hf-skills/README quests . --repo-type=space ================================================ FILE: .gitignore ================================================ # Byte-compiled / optimized / DLL files __pycache__/ *.py[codz] *$py.class # C extensions *.so # Distribution / packaging .Python build/ develop-eggs/ dist/ downloads/ eggs/ .eggs/ lib/ lib64/ parts/ sdist/ var/ wheels/ share/python-wheels/ *.egg-info/ .installed.cfg *.egg MANIFEST # PyInstaller # Usually these files are written by a python script from a template # before PyInstaller builds the exe, so as to inject date/other infos into it. *.manifest *.spec # Installer logs pip-log.txt pip-delete-this-directory.txt # Unit test / coverage reports htmlcov/ .tox/ .nox/ .coverage .coverage.* .cache nosetests.xml coverage.xml *.cover *.py.cover .hypothesis/ .pytest_cache/ cover/ # Translations *.mo *.pot # Django stuff: *.log local_settings.py db.sqlite3 db.sqlite3-journal # Flask stuff: instance/ .webassets-cache # Scrapy stuff: .scrapy # Sphinx documentation docs/_build/ # PyBuilder .pybuilder/ target/ # Jupyter Notebook .ipynb_checkpoints # IPython profile_default/ ipython_config.py # pyenv # For a library or package, you might want to ignore these files since the code is # intended to run in multiple environments; otherwise, check them in: # .python-version # pipenv # According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control. # However, in case of collaboration, if having platform-specific dependencies or dependencies # having no cross-platform support, pipenv may install dependencies that don't work, or not # install all needed dependencies. #Pipfile.lock # UV # Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control. # This is especially recommended for binary packages to ensure reproducibility, and is more # commonly ignored for libraries. #uv.lock # poetry # Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control. # This is especially recommended for binary packages to ensure reproducibility, and is more # commonly ignored for libraries. # https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control #poetry.lock #poetry.toml # pdm # Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control. # pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python. # https://pdm-project.org/en/latest/usage/project/#working-with-version-control #pdm.lock #pdm.toml .pdm-python .pdm-build/ # pixi # Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control. #pixi.lock # Pixi creates a virtual environment in the .pixi directory, just like venv module creates one # in the .venv directory. It is recommended not to include this directory in version control. .pixi # PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm __pypackages__/ # Celery stuff celerybeat-schedule celerybeat.pid # SageMath parsed files *.sage.py # Environments .env .envrc .venv env/ venv/ ENV/ env.bak/ venv.bak/ # Spyder project settings .spyderproject .spyproject # Rope project settings .ropeproject # mkdocs documentation /site # mypy .mypy_cache/ .dmypy.json dmypy.json # Pyre type checker .pyre/ # pytype static type analyzer .pytype/ # Cython debug symbols cython_debug/ # PyCharm # JetBrains specific template is maintained in a separate JetBrains.gitignore that can # be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore # and can be added to the global gitignore or merged into this file. For a more nuclear # option (not recommended) you can uncomment the following to ignore the entire idea folder. #.idea/ # Abstra # Abstra is an AI-powered process automation framework. # Ignore directories containing user credentials, local state, and settings. # Learn more at https://abstra.io/docs .abstra/ # Visual Studio Code # Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore # that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore # and can be added to the global gitignore or merged into this file. However, if you prefer, # you could uncomment the following to ignore the entire vscode folder # .vscode/ # Ruff stuff: .ruff_cache/ # PyPI configuration file .pypirc # Cursor # Cursor is an AI-powered code editor. `.cursorignore` specifies files/directories to # exclude from AI features like autocomplete and code analysis. Recommended for sensitive data # refer to https://docs.cursor.com/context/ignore-files .cursorignore .cursorindexingignore # Marimo marimo/_static/ marimo/_lsp/ __marimo__/ .claude .fast-agent/ # MacOS .DS_Store ================================================ FILE: .mcp.json ================================================ { "mcpServers": { "huggingface-skills": { "url": "https://huggingface.co/mcp?login" } } } ================================================ FILE: LICENSE ================================================ Apache License Version 2.0, January 2004 http://www.apache.org/licenses/ TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION 1. Definitions. "License" shall mean the terms and conditions for use, reproduction, and distribution as defined by Sections 1 through 9 of this document. "Licensor" shall mean the copyright owner or entity authorized by the copyright owner that is granting the License. "Legal Entity" shall mean the union of the acting entity and all other entities that control, are controlled by, or are under common control with that entity. For the purposes of this definition, "control" means (i) the power, direct or indirect, to cause the direction or management of such entity, whether by contract or otherwise, or (ii) ownership of fifty percent (50%) or more of the outstanding shares, or (iii) beneficial ownership of such entity. "You" (or "Your") shall mean an individual or Legal Entity exercising permissions granted by this License. "Source" form shall mean the preferred form for making modifications, including but not limited to software source code, documentation source, and configuration files. "Object" form shall mean any form resulting from mechanical transformation or translation of a Source form, including but not limited to compiled object code, generated documentation, and conversions to other media types. "Work" shall mean the work of authorship, whether in Source or Object form, made available under the License, as indicated by a copyright notice that is included in or attached to the work (an example is provided in the Appendix below). "Derivative Works" shall mean any work, whether in Source or Object form, that is based on (or derived from) the Work and for which the editorial revisions, annotations, elaborations, or other modifications represent, as a whole, an original work of authorship. For the purposes of this License, Derivative Works shall not include works that remain separable from, or merely link (or bind by name) to the interfaces of, the Work and Derivative Works thereof. "Contribution" shall mean any work of authorship, including the original version of the Work and any modifications or additions to that Work or Derivative Works thereof, that is intentionally submitted to Licensor for inclusion in the Work by the copyright owner or by an individual or Legal Entity authorized to submit on behalf of the copyright owner. For the purposes of this definition, "submitted" means any form of electronic, verbal, or written communication sent to the Licensor or its representatives, including but not limited to communication on electronic mailing lists, source code control systems, and issue tracking systems that are managed by, or on behalf of, the Licensor for the purpose of discussing and improving the Work, but excluding communication that is conspicuously marked or otherwise designated in writing by the copyright owner as "Not a Contribution." "Contributor" shall mean Licensor and any individual or Legal Entity on behalf of whom a Contribution has been received by Licensor and subsequently incorporated within the Work. 2. Grant of Copyright License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable copyright license to reproduce, prepare Derivative Works of, publicly display, publicly perform, sublicense, and distribute the Work and such Derivative Works in Source or Object form. 3. Grant of Patent License. Subject to the terms and conditions of this License, each Contributor hereby grants to You a perpetual, worldwide, non-exclusive, no-charge, royalty-free, irrevocable (except as stated in this section) patent license to make, have made, use, offer to sell, sell, import, and otherwise transfer the Work, where such license applies only to those patent claims licensable by such Contributor that are necessarily infringed by their Contribution(s) alone or by combination of their Contribution(s) with the Work to which such Contribution(s) was submitted. If You institute patent litigation against any entity (including a cross-claim or counterclaim in a lawsuit) alleging that the Work or a Contribution incorporated within the Work constitutes direct or contributory patent infringement, then any patent licenses granted to You under this License for that Work shall terminate as of the date such litigation is filed. 4. Redistribution. You may reproduce and distribute copies of the Work or Derivative Works thereof in any medium, with or without modifications, and in Source or Object form, provided that You meet the following conditions: (a) You must give any other recipients of the Work or Derivative Works a copy of this License; and (b) You must cause any modified files to carry prominent notices stating that You changed the files; and (c) You must retain, in the Source form of any Derivative Works that You distribute, all copyright, patent, trademark, and attribution notices from the Source form of the Work, excluding those notices that do not pertain to any part of the Derivative Works; and (d) If the Work includes a "NOTICE" text file as part of its distribution, then any Derivative Works that You distribute must include a readable copy of the attribution notices contained within such NOTICE file, excluding those notices that do not pertain to any part of the Derivative Works, in at least one of the following places: within a NOTICE text file distributed as part of the Derivative Works; within the Source form or documentation, if provided along with the Derivative Works; or, within a display generated by the Derivative Works, if and wherever such third-party notices normally appear. The contents of the NOTICE file are for informational purposes only and do not modify the License. You may add Your own attribution notices within Derivative Works that You distribute, alongside or as an addendum to the NOTICE text from the Work, provided that such additional attribution notices cannot be construed as modifying the License. You may add Your own copyright statement to Your modifications and may provide additional or different license terms and conditions for use, reproduction, or distribution of Your modifications, or for any such Derivative Works as a whole, provided Your use, reproduction, and distribution of the Work otherwise complies with the conditions stated in this License. 5. Submission of Contributions. Unless You explicitly state otherwise, any Contribution intentionally submitted for inclusion in the Work by You to the Licensor shall be under the terms and conditions of this License, without any additional terms or conditions. Notwithstanding the above, nothing herein shall supersede or modify the terms of any separate license agreement you may have executed with Licensor regarding such Contributions. 6. Trademarks. This License does not grant permission to use the trade names, trademarks, service marks, or product names of the Licensor, except as required for reasonable and customary use in describing the origin of the Work and reproducing the content of the NOTICE file. 7. Disclaimer of Warranty. Unless required by applicable law or agreed to in writing, Licensor provides the Work (and each Contributor provides its Contributions) on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied, including, without limitation, any warranties or conditions of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A PARTICULAR PURPOSE. You are solely responsible for determining the appropriateness of using or redistributing the Work and assume any risks associated with Your exercise of permissions under this License. 8. Limitation of Liability. In no event and under no legal theory, whether in tort (including negligence), contract, or otherwise, unless required by applicable law (such as deliberate and grossly negligent acts) or agreed to in writing, shall any Contributor be liable to You for damages, including any direct, indirect, special, incidental, or consequential damages of any character arising as a result of this License or out of the use or inability to use the Work (including but not limited to damages for loss of goodwill, work stoppage, computer failure or malfunction, or any and all other commercial damages or losses), even if such Contributor has been advised of the possibility of such damages. 9. Accepting Warranty or Additional Liability. While redistributing the Work or Derivative Works thereof, You may choose to offer, and charge a fee for, acceptance of support, warranty, indemnity, or other liability obligations and/or rights consistent with this License. However, in accepting such obligations, You may act only on Your own behalf and on Your sole responsibility, not on behalf of any other Contributor, and only if You agree to indemnify, defend, and hold each Contributor harmless for any liability incurred by, or claims asserted against, such Contributor by reason of your accepting any such warranty or additional liability. END OF TERMS AND CONDITIONS APPENDIX: How to apply the Apache License to your work. To apply the Apache License to your work, attach the following boilerplate notice, with the fields enclosed by brackets "[]" replaced with your own identifying information. (Don't include the brackets!) The text should be enclosed in the appropriate comment syntax for the file format. We also recommend that a file or class name and description of purpose be included on the same "printed page" as the copyright notice for easier identification within third-party archives. Copyright [yyyy] [name of copyright owner] Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. ================================================ FILE: README.md ================================================ # Hugging Face Skills Hugging Face Skills are definitions for AI/ML tasks like dataset creation, model training, and evaluation. They are interoperable with all major coding agent tools like OpenAI Codex, Anthropic's Claude Code, Google DeepMind's Gemini CLI, and Cursor. The skills in this repository follow the standardized [Agent Skills](https://agentskills.io/home) format. ## How do Skills work? In practice, skills are self-contained folders that package instructions, scripts, and resources together for an AI agent to use on a specific use case. Each folder includes a `SKILL.md` file with YAML frontmatter (name and description) followed by the guidance your coding agent follows while the skill is active. > [!NOTE] > 'Skills' is actually an Anthropic term used within Claude AI and Claude Code and not adopted by other agent tools, but we love it! OpenAI Codex uses the open [Agent Skills](https://agentskills.io/specification) format, where each skill is a directory with a `SKILL.md` file that Codex discovers from standard `.agents/skills` locations documented in the [Codex Skills guide](https://developers.openai.com/codex/skills/). Codex can also work with an `AGENTS.md` file. Google Gemini uses 'extensions' to define the instructions for your coding agent in a `gemini-extension.json` file. **This repo is compatible with all of them, and more!** > [!TIP] > If your agent doesn't support skills, you can use [`agents/AGENTS.md`](agents/AGENTS.md) directly as a fallback. ## Installation Hugging Face skills are compatible with Claude Code, Codex, Gemini CLI, and Cursor. ### Claude Code 1. Register the repository as a plugin marketplace: ``` /plugin marketplace add huggingface/skills ``` 2. To install a skill, run: ``` /plugin install @huggingface/skills ``` For example: ``` /plugin install hf-cli@huggingface/skills ``` ### Codex 1. Copy or symlink any skills you want to use from this repository's `skills/` directory into one of Codex's standard `.agents/skills` locations (for example, `$REPO_ROOT/.agents/skills` or `$HOME/.agents/skills`) as described in the [Codex Skills guide](https://developers.openai.com/codex/skills/). 2. Once a skill is available in one of those locations, Codex will discover it using the Agent Skills standard and load the `SKILL.md` instructions when it decides to use that skill or when you explicitly invoke it. 3. If your Codex setup still relies on `AGENTS.md`, you can use the generated [`agents/AGENTS.md`](agents/AGENTS.md) file in this repo as a fallback bundle of instructions. ### Gemini CLI 1. This repo includes `gemini-extension.json` to integrate with the Gemini CLI. 2. Install locally: ``` gemini extensions install . --consent ``` or use the GitHub URL: ``` gemini extensions install https://github.com/huggingface/skills.git --consent ``` 4. See [Gemini CLI extensions docs](https://geminicli.com/docs/extensions/#installing-an-extension) for more help. ### Cursor This repository includes Cursor plugin manifests: - `.cursor-plugin/plugin.json` - `.mcp.json` (configured with the Hugging Face MCP server URL) Install from repository URL (or local checkout) via the Cursor plugin flow. For contributors, regenerate manifests with: ```bash ./scripts/publish.sh ``` ## Skills This repository contains a few skills to get you started. You can also contribute your own skills to the repository. ### Available skills | Name | Description | Documentation | |------|-------------|---------------| | `gradio` | Build Gradio web UIs and demos in Python. Use when creating or editing Gradio apps, components, event listeners, layouts, or chatbots. | [SKILL.md](skills/huggingface-gradio/SKILL.md) | | `hf-cli` | Execute Hugging Face Hub operations using the hf CLI. Download models/datasets, upload files, manage repos, and run cloud compute jobs. | [SKILL.md](skills/hf-cli/SKILL.md) | | `hugging-face-dataset-viewer` | Explore, query, and extract data from any Hugging Face dataset using the Dataset Viewer REST API and npx tooling. Zero Python dependencies — covers split/config discovery, row pagination, text search, filtering, SQL via parquetlens, and dataset upload via CLI. | [SKILL.md](skills/hugging-face-dataset-viewer/SKILL.md) | | `hugging-face-datasets` | Create and manage datasets on Hugging Face Hub. Supports initializing repos, defining configs/system prompts, streaming row updates, and SQL-based dataset querying/transformation. | [SKILL.md](skills/hugging-face-datasets/SKILL.md) | | `hugging-face-evaluation` | Add and manage evaluation results in Hugging Face model cards. Supports extracting eval tables from README content, importing scores from Artificial Analysis API, and running custom evaluations with vLLM/lighteval. | [SKILL.md](skills/hugging-face-evaluation/SKILL.md) | | `hugging-face-jobs` | Run compute jobs on Hugging Face infrastructure. Execute Python scripts, manage scheduled jobs, and monitor job status. | [SKILL.md](skills/hugging-face-jobs/SKILL.md) | | `hugging-face-model-trainer` | Train or fine-tune language models using TRL on Hugging Face Jobs infrastructure. Covers SFT, DPO, GRPO and reward modeling training methods, plus GGUF conversion for local deployment. Includes hardware selection, cost estimation, Trackio monitoring, and Hub persistence. | [SKILL.md](skills/hugging-face-model-trainer/SKILL.md) | | `hugging-face-paper-pages` | Look up and read Hugging Face paper pages in markdown, and use the papers API for structured metadata like authors, linked models, datasets, Spaces, and media URLs when needed. | [SKILL.md](skills/hugging-face-paper-pages/SKILL.md) | | `hugging-face-paper-publisher` | Publish and manage research papers on Hugging Face Hub. Supports creating paper pages, linking papers to models/datasets, claiming authorship, and generating professional markdown-based research articles. | [SKILL.md](skills/hugging-face-paper-publisher/SKILL.md) | | `hugging-face-tool-builder` | Build reusable scripts for Hugging Face API operations. Useful for chaining API calls or automating repeated tasks. | [SKILL.md](skills/hugging-face-tool-builder/SKILL.md) | | `hugging-face-trackio` | Track and visualize ML training experiments with Trackio. Log metrics via Python API and retrieve them via CLI. Supports real-time dashboards synced to HF Spaces. | [SKILL.md](skills/hugging-face-trackio/SKILL.md) | | `hugging-face-vision-trainer` | Train and fine-tune object detection models (RTDETRv2, YOLOS, DETR and others) and image classification models (timm and transformers models — MobileNetV3, MobileViT, ResNet, ViT/DINOv3) using Transformers Trainer API on Hugging Face Jobs infrastructure or locally. Includes COCO dataset format support, Albumentations augmentation, mAP/mAR metrics, trackio tracking, hardware selection, and Hub persistence. | [SKILL.md](skills/hugging-face-vision-trainer/SKILL.md) | | `transformers-js` | Run state-of-the-art machine learning models directly in JavaScript/TypeScript for NLP, computer vision, audio processing, and multimodal tasks. Works in Node.js and browsers with WebGPU/WASM using Hugging Face models. | [SKILL.md](skills/transformers.js/SKILL.md) | ### Using skills in your coding agent Once a skill is installed, mention it directly while giving your coding agent instructions: - "Use the HF LLM trainer skill to estimate the GPU memory needed for a 70B model run." - "Use the HF model evaluation skill to launch `run_eval_job.py` on the latest checkpoint." - "Use the HF dataset creator skill to draft new few-shot classification templates." - "Use the HF paper publisher skill to index my arXiv paper and link it to my model." Your coding agent automatically loads the corresponding `SKILL.md` instructions and helper scripts while it completes the task. ### Contribute or customize a skill 1. Copy one of the existing skill folders (for example, `hf-datasets/`) and rename it. 2. Update the new folder's `SKILL.md` frontmatter: ```markdown --- name: my-skill-name description: Describe what the skill does and when to use it --- # Skill Title Guidance + examples + guardrails ``` 3. Add or edit supporting scripts, templates, and documents referenced by your instructions. 4. Add an entry to `.claude-plugin/marketplace.json` with a concise, human-readable description. 5. Run: ```bash ./scripts/publish.sh ``` to regenerate and validate all generated metadata. 6. Reinstall or reload the skill bundle in your coding agent so the updated folder is available. ### Marketplace The `.claude-plugin/marketplace.json` file lists skills with human-readable descriptions for the plugin marketplace. The CI validates that skill names and paths match between `SKILL.md` files and `marketplace.json`, but descriptions are maintained separately: `SKILL.md` descriptions guide when Claude activates the skill, while marketplace descriptions are written for humans browsing available skills. ### Additional references - Browse the latest instructions, scripts, and templates directly at [huggingface/skills](https://github.com/huggingface/skills). - Review Hugging Face documentation for the specific libraries or workflows you reference inside each skill. ================================================ FILE: agents/AGENTS.md ================================================ You have additional SKILLs documented in directories containing a "SKILL.md" file. These skills are: - gradio -> "skills/huggingface-gradio/SKILL.md" - hf-cli -> "skills/hf-cli/SKILL.md" - hugging-face-dataset-viewer -> "skills/hugging-face-dataset-viewer/SKILL.md" - hugging-face-datasets -> "skills/hugging-face-datasets/SKILL.md" - hugging-face-evaluation -> "skills/hugging-face-evaluation/SKILL.md" - hugging-face-jobs -> "skills/hugging-face-jobs/SKILL.md" - hugging-face-model-trainer -> "skills/hugging-face-model-trainer/SKILL.md" - hugging-face-paper-pages -> "skills/hugging-face-paper-pages/SKILL.md" - hugging-face-paper-publisher -> "skills/hugging-face-paper-publisher/SKILL.md" - hugging-face-tool-builder -> "skills/hugging-face-tool-builder/SKILL.md" - hugging-face-trackio -> "skills/hugging-face-trackio/SKILL.md" - hugging-face-vision-trainer -> "skills/hugging-face-vision-trainer/SKILL.md" - transformers-js -> "skills/transformers.js/SKILL.md" IMPORTANT: You MUST read the SKILL.md file whenever the description of the skills matches the user intent, or may help accomplish their task. gradio: `Build Gradio web UIs and demos in Python. Use when creating or editing Gradio apps, components, event listeners, layouts, or chatbots.` hf-cli: `"Hugging Face Hub CLI (`hf`) for downloading, uploading, and managing repositories, models, datasets, and Spaces on the Hugging Face Hub. Replaces now deprecated `huggingface-cli` command."` hugging-face-dataset-viewer: `Use this skill for Hugging Face Dataset Viewer API workflows that fetch subset/split metadata, paginate rows, search text, apply filters, download parquet URLs, and read size or statistics.` hugging-face-datasets: `Create and manage datasets on Hugging Face Hub. Supports initializing repos, defining configs/system prompts, streaming row updates, and SQL-based dataset querying/transformation. Designed to work alongside HF MCP server for comprehensive dataset workflows.` hugging-face-evaluation: `Add and manage evaluation results in Hugging Face model cards. Supports extracting eval tables from README content, importing scores from Artificial Analysis API, and running custom model evaluations with vLLM/lighteval. Works with the model-index metadata format.` hugging-face-jobs: `This skill should be used when users want to run any workload on Hugging Face Jobs infrastructure. Covers UV scripts, Docker-based jobs, hardware selection, cost estimation, authentication with tokens, secrets management, timeout configuration, and result persistence. Designed for general-purpose compute workloads including data processing, inference, experiments, batch jobs, and any Python-based tasks. Should be invoked for tasks involving cloud compute, GPU workloads, or when users mention running jobs on Hugging Face infrastructure without local setup.` hugging-face-model-trainer: `This skill should be used when users want to train or fine-tune language models using TRL (Transformer Reinforcement Learning) on Hugging Face Jobs infrastructure. Covers SFT, DPO, GRPO and reward modeling training methods, plus GGUF conversion for local deployment. Includes guidance on the TRL Jobs package, UV scripts with PEP 723 format, dataset preparation and validation, hardware selection, cost estimation, Trackio monitoring, Hub authentication, and model persistence. Should be invoked for tasks involving cloud GPU training, GGUF conversion, or when users mention training on Hugging Face Jobs without local GPU setup.` hugging-face-paper-pages: `Look up and read Hugging Face paper pages in markdown, and use the papers API for structured metadata such as authors, linked models/datasets/spaces, Github repo and project page. Use when the user shares a Hugging Face paper page URL, an arXiv URL or ID, or asks to summarize, explain, or analyze an AI research paper.` hugging-face-paper-publisher: `Publish and manage research papers on Hugging Face Hub. Supports creating paper pages, linking papers to models/datasets, claiming authorship, and generating professional markdown-based research articles.` hugging-face-tool-builder: `Use this skill when the user wants to build tool/scripts or achieve a task where using data from the Hugging Face API would help. This is especially useful when chaining or combining API calls or the task will be repeated/automated. This Skill creates a reusable script to fetch, enrich or process data.` hugging-face-trackio: `Track and visualize ML training experiments with Trackio. Use when logging metrics during training (Python API), firing alerts for training diagnostics, or retrieving/analyzing logged metrics (CLI). Supports real-time dashboard visualization, alerts with webhooks, HF Space syncing, and JSON output for automation.` hugging-face-vision-trainer: `Trains and fine-tunes vision models for object detection (D-FINE, RT-DETR v2, DETR, YOLOS), image classification (timm models — MobileNetV3, MobileViT, ResNet, ViT/DINOv3 — plus any Transformers classifier), and SAM/SAM2 segmentation using Hugging Face Transformers on Hugging Face Jobs cloud GPUs. Covers COCO-format dataset preparation, Albumentations augmentation, mAP/mAR evaluation, accuracy metrics, SAM segmentation with bbox/point prompts, DiceCE loss, hardware selection, cost estimation, Trackio monitoring, and Hub persistence. Use when users mention training object detection, image classification, SAM, SAM2, segmentation, image matting, DETR, D-FINE, RT-DETR, ViT, timm, MobileNet, ResNet, bounding box models, or fine-tuning vision models on Hugging Face Jobs.` transformers-js: `Use Transformers.js to run state-of-the-art machine learning models directly in JavaScript/TypeScript. Supports NLP (text classification, translation, summarization), computer vision (image classification, object detection), audio (speech recognition, audio classification), and multimodal tasks. Works in Node.js and browsers (with WebGPU/WASM) using pre-trained models from Hugging Face Hub.` Paths referenced within SKILL folders are relative to that SKILL. For example the hf-datasets `scripts/example.py` would be referenced as `hf-datasets/scripts/example.py`. ================================================ FILE: apps/evals-leaderboard/app.py ================================================ #!/usr/bin/env python3 """ Evals Leaderboard - Gradio app for displaying model evaluation scores. Reads leaderboard data from the hf-skills/evals-leaderboard dataset. Run collect_evals.py separately to update the dataset. Usage: python app.py """ from __future__ import annotations import json import gradio as gr import requests TABLE_HEADERS = [ "Model", "Benchmark", "Score", "Source", ] TABLE_DATATYPES = [ "markdown", "text", "number", "markdown", ] DATASET_REPO = "hf-skills/evals-leaderboard" LEADERBOARD_URL = f"https://huggingface.co/datasets/{DATASET_REPO}/raw/main/data/leaderboard.jsonl" METADATA_URL = f"https://huggingface.co/datasets/{DATASET_REPO}/raw/main/data/metadata.json" def format_model_link(model_id: str) -> str: """Format model ID as a clickable link.""" return f"[{model_id}](https://huggingface.co/{model_id})" def format_source_link(source_type: str, contributor: str, source_url: str) -> str: """Format source as a clickable link.""" return f"{source_type} by [{contributor}]({source_url})" def fetch_leaderboard() -> tuple[list[dict], dict]: """Fetch leaderboard data from the HF dataset.""" # Fetch leaderboard JSONL resp = requests.get(LEADERBOARD_URL, timeout=30) resp.raise_for_status() leaderboard = [json.loads(line) for line in resp.text.strip().split("\n") if line] # Fetch metadata resp = requests.get(METADATA_URL, timeout=30) resp.raise_for_status() metadata = resp.json() return leaderboard, metadata def refresh_handler() -> tuple[str, list[list]]: """Refresh the leaderboard data from the dataset.""" try: leaderboard, metadata = fetch_leaderboard() # Build table rows rows = [] for entry in leaderboard: rows.append( [ format_model_link(entry["model_id"]), entry["benchmark"], entry["score"], format_source_link( entry["source_type"], entry["contributor"], entry["source_url"], ), ] ) status = "\n".join( [ f"**Data from:** [{DATASET_REPO}](https://huggingface.co/datasets/{DATASET_REPO})", f"**Last updated:** {metadata.get('generated_at', 'unknown')}", f"**Models with scores:** {metadata.get('models_with_scores', 'unknown')}", f"**Total entries:** {metadata.get('total_entries', len(leaderboard))}", ] ) return status, rows except Exception as e: return f"❌ Failed to load leaderboard: {e}", [] with gr.Blocks() as demo: gr.Markdown( """ # 📊 HF Evaluation Leaderboard Shows MMLU, BigCodeBench, and ARC MC scores pulled from model-index metadata or their pull requests for trending text-generation models. """ ) status_box = gr.Markdown("Loading leaderboard...") leaderboard_table = gr.Dataframe( headers=TABLE_HEADERS, datatype=TABLE_DATATYPES, interactive=False, wrap=True, ) demo.load( refresh_handler, outputs=[status_box, leaderboard_table], ) gr.Markdown( f""" --- **Links:** - [Dataset: {DATASET_REPO}](https://huggingface.co/datasets/{DATASET_REPO}) - [GitHub Repository](https://github.com/huggingface/skills) """ ) if __name__ == "__main__": demo.launch() ================================================ FILE: apps/evals-leaderboard/collect_evals.py ================================================ #!/usr/bin/env python3 """ Collect evaluation scores from trending models' model-index metadata. Scans trending text-generation models on the Hub and extracts benchmark scores from their model-index metadata or open pull requests. Results are saved to a dataset for the evals leaderboard. Usage: python collect_evals.py [--push-to-hub] """ from __future__ import annotations import argparse import json import os import re from dataclasses import dataclass from datetime import datetime, timezone from pathlib import Path from typing import Any, Dict, List, Optional import requests import yaml from huggingface_hub import hf_hub_download from huggingface_hub.utils import HfHubHTTPError API_BASE = "https://huggingface.co/api" PIPELINE_FILTER = "text-generation" TRENDING_LIMIT = 50 TRENDING_FETCH_LIMIT = 100 PR_SCAN_LIMIT = 40 USER_AGENT = "skills-evals-leaderboard/0.3" def _normalize(text: Optional[str]) -> str: if not text: return "" text = text.lower() text = re.sub(r"[^a-z0-9]+", " ", text) return text.strip() def _coerce_score(value: Any) -> Optional[float]: if value is None: return None if isinstance(value, (int, float)): return float(value) if isinstance(value, str): candidate = value.strip() if candidate.endswith("%"): candidate = candidate[:-1] try: return float(candidate) except ValueError: return None return None @dataclass(frozen=True) class BenchmarkSpec: key: str label: str aliases: tuple[str, ...] def matches(self, fields: List[str]) -> bool: for alias in self.aliases: alias_norm = _normalize(alias) if not alias_norm: continue for field in fields: if alias_norm in field: return True return False BENCHMARKS: Dict[str, BenchmarkSpec] = { "mmlu": BenchmarkSpec( key="mmlu", label="MMLU", aliases=("mmlu", "massive multitask language understanding"), ), "bigcodebench": BenchmarkSpec( key="bigcodebench", label="BigCodeBench", aliases=("bigcodebench", "big code bench"), ), "arc_mc": BenchmarkSpec( key="arc_mc", label="ARC MC", aliases=( "arc mc", "arc-challenge", "arc challenge", "arc multiple choice", "arc c", ), ), } class EvalsCollector: """Collects evaluation scores from model-index metadata.""" def __init__(self, token: str | None = None) -> None: self.token = token self.session = requests.Session() self.session.headers.update({"User-Agent": USER_AGENT}) if token: self.session.headers.update({"Authorization": f"Bearer {token}"}) self.logs: List[str] = [] self.results: List[Dict[str, Any]] = [] def log(self, message: str) -> None: """Add a log message.""" print(message) self.logs.append(message) def collect_all(self) -> List[Dict[str, Any]]: """Collect evaluation scores from trending models.""" self.log("🔍 Fetching trending text-generation models...") trending = self._fetch_trending_models() for entry in trending: repo_id = entry.get("modelId") or entry.get("id") if not repo_id: continue scores = self._collect_scores(repo_id) if scores["scores"]: self.results.extend(self._format_scores(repo_id, scores["scores"])) self.log(f"✅ Collected {len(self.results)} evaluation entries") return self.results def _fetch_trending_models(self) -> List[Dict[str, Any]]: params = {"sort": "trendingScore", "limit": TRENDING_FETCH_LIMIT} response = self.session.get( f"{API_BASE}/models", params=params, timeout=30, ) response.raise_for_status() data = response.json() if not isinstance(data, list): raise ValueError("Unexpected trending response.") filtered = [ model for model in data if (model.get("pipeline_tag") == PIPELINE_FILTER or PIPELINE_FILTER in (model.get("tags") or [])) ] if not filtered: self.log("⚠️ No text-generation models in trending feed.") return [] limited = filtered[:TRENDING_LIMIT] self.log(f"📊 Found {len(limited)} trending text-generation models") return limited def _collect_scores(self, repo_id: str) -> Dict[str, Any]: owner = repo_id.split("/")[0] card_meta = self._read_model_card(repo_id) model_index = card_meta.get("model-index") if model_index: self.log(f"✅ {repo_id}: model card metadata found.") scores = self._extract_scores( repo_id=repo_id, model_index=model_index, contributor=owner, source_type="model-card", source_url=f"https://huggingface.co/{repo_id}", revision="main", ) if scores: return {"model_id": repo_id, "scores": scores} prs = self._fetch_pull_requests(repo_id) for pr in prs: revision = f"refs/pr/{pr['num']}" pr_meta = self._read_model_card(repo_id, revision=revision) pr_index = pr_meta.get("model-index") if not pr_index: continue author_info = pr.get("author", {}) or {} contributor = author_info.get("name") or author_info.get("fullname") or "unknown-author" discussion_path = f"{repo_id}/discussions/{pr['num']}" source_url = f"https://huggingface.co/{discussion_path}" scores = self._extract_scores( repo_id=repo_id, model_index=pr_index, contributor=contributor, source_type="pull-request", source_url=source_url, revision=revision, ) if scores: note = f"📝 {repo_id}: PR #{pr['num']} by {contributor}." self.log(note) return {"model_id": repo_id, "scores": scores} self.log(f"⚠️ {repo_id}: no target benchmarks located.") return {"model_id": repo_id, "scores": {}} def _read_model_card( self, repo_id: str, revision: Optional[str] = None, ) -> Dict[str, Any]: try: path = hf_hub_download( repo_id=repo_id, filename="README.md", repo_type="model", revision=revision, token=self.token, ) except HfHubHTTPError as err: ctx = f"{repo_id} ({revision or 'main'})" self.log(f"🚫 {ctx}: README download failed ({err}).") return {} text = Path(path).read_text(encoding="utf-8", errors="ignore") return self._parse_front_matter(text) @staticmethod def _parse_front_matter(content: str) -> Dict[str, Any]: content = content.lstrip("\ufeff") if not content.startswith("---"): return {} lines = content.splitlines() end_idx = None for idx, line in enumerate(lines[1:], start=1): if line.strip() == "---": end_idx = idx break if end_idx is None: return {} front_matter = "\n".join(lines[1:end_idx]) try: data = yaml.safe_load(front_matter) or {} return data if isinstance(data, dict) else {} except yaml.YAMLError: return {} def _fetch_pull_requests(self, repo_id: str) -> List[Dict[str, Any]]: url = f"{API_BASE}/models/{repo_id}/discussions" try: response = self.session.get( url, params={"limit": PR_SCAN_LIMIT}, timeout=30, ) response.raise_for_status() except requests.RequestException as err: self.log(f"🚫 {repo_id}: PR list request failed ({err}).") return [] payload = response.json() discussions = payload.get("discussions", []) prs = [disc for disc in discussions if disc.get("isPullRequest")] prs.sort(key=lambda item: item.get("createdAt", ""), reverse=True) if prs: self.log(f"📬 {repo_id}: scanning {len(prs)} pull requests.") return prs def _extract_scores( self, repo_id: str, model_index: Any, contributor: str, source_type: str, source_url: str, revision: str, ) -> Dict[str, Dict[str, Any]]: if not isinstance(model_index, list): return {} scores: Dict[str, Dict[str, Any]] = {} for entry in model_index: if not isinstance(entry, dict): continue model_name = entry.get("name") or repo_id.split("/")[-1] for result in entry.get("results", []): dataset_info = result.get("dataset") or {} dataset_name = dataset_info.get("name") dataset_type = dataset_info.get("type") task_info = result.get("task") or {} task_type = task_info.get("type") for metric in result.get("metrics", []): benchmark_key = self._match_benchmark( dataset_name, dataset_type, metric, ) if not benchmark_key: continue raw_value = metric.get("value") value = _coerce_score(raw_value) if value is None: continue unit = metric.get("unit") or "" is_pct = isinstance(raw_value, str) and raw_value.strip().endswith("%") if not unit and is_pct: unit = "%" metric_name = metric.get("name") or metric.get("type") or "" payload = { "model": repo_id, "model_name": model_name, "benchmark_key": benchmark_key, "benchmark_label": BENCHMARKS[benchmark_key].label, "value": value, "unit": unit, "dataset": dataset_name or dataset_type or "", "task_type": task_type or "", "metric_name": metric_name, "contributor": contributor, "source_type": source_type, "source_url": source_url, "revision": revision, } existing = scores.get(benchmark_key) if not existing or value > existing["value"]: scores[benchmark_key] = payload return scores def _match_benchmark( self, dataset_name: Optional[str], dataset_type: Optional[str], metric: Dict[str, Any], ) -> Optional[str]: fields = [ _normalize(dataset_name), _normalize(dataset_type), _normalize(metric.get("name")), _normalize(metric.get("type")), ] fields = [field for field in fields if field] for key, spec in BENCHMARKS.items(): if spec.matches(fields): return key return None def _format_scores(self, model_id: str, scores: Dict[str, Dict[str, Any]]) -> List[Dict[str, Any]]: """Format scores as flat records for the dataset.""" rows = [] for benchmark_key, payload in scores.items(): rows.append( { "model_id": model_id, "benchmark": payload["benchmark_label"], "benchmark_key": benchmark_key, "score": round(payload["value"], 2), "source_type": payload["source_type"], "source_url": payload["source_url"], "contributor": payload["contributor"], "collected_at": datetime.now(timezone.utc).isoformat(), } ) return rows def get_leaderboard(self) -> List[Dict[str, Any]]: """Get results sorted by score descending.""" return sorted(self.results, key=lambda x: x["score"], reverse=True) def save_json(self, filepath: str) -> None: """Save the leaderboard to a JSON file.""" leaderboard = self.get_leaderboard() output = { "generated_at": datetime.now(timezone.utc).isoformat(), "total_entries": len(leaderboard), "benchmarks": list(BENCHMARKS.keys()), "leaderboard": leaderboard, } with open(filepath, "w") as f: json.dump(output, f, indent=2) self.log(f"💾 Saved leaderboard to {filepath}") def push_to_hub(self, repo_id: str = "hf-skills/evals-leaderboard") -> None: """Push the leaderboard data to a HF dataset.""" try: from huggingface_hub import HfApi except ImportError: self.log("❌ huggingface_hub not installed. Run: pip install huggingface_hub") return api = HfApi(token=self.token) leaderboard = self.get_leaderboard() # Create dataset as JSONL jsonl_content = "\n".join(json.dumps(row) for row in leaderboard) # Create metadata file metadata = { "generated_at": datetime.now(timezone.utc).isoformat(), "total_entries": len(leaderboard), "models_with_scores": len(set(r["model_id"] for r in leaderboard)), "benchmarks": list(BENCHMARKS.keys()), } try: # Create repo if it doesn't exist api.create_repo(repo_id=repo_id, repo_type="dataset", exist_ok=True) self.log(f"📁 Ensured dataset repo exists: {repo_id}") # Upload leaderboard data api.upload_file( path_or_fileobj=jsonl_content.encode(), path_in_repo="data/leaderboard.jsonl", repo_id=repo_id, repo_type="dataset", commit_message=f"Update leaderboard - {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M')} UTC", ) # Upload metadata api.upload_file( path_or_fileobj=json.dumps(metadata, indent=2).encode(), path_in_repo="data/metadata.json", repo_id=repo_id, repo_type="dataset", commit_message=f"Update metadata - {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M')} UTC", ) self.log(f"🚀 Pushed leaderboard to {repo_id}") except Exception as e: self.log(f"❌ Failed to push to hub: {e}") def main() -> None: parser = argparse.ArgumentParser(description="Collect evaluation scores from model-index metadata") parser.add_argument( "--push-to-hub", action="store_true", help="Push results to HF dataset", ) parser.add_argument( "--output", type=str, default="leaderboard.json", help="Output JSON file path", ) parser.add_argument( "--repo-id", type=str, default="hf-skills/evals-leaderboard", help="HF dataset repo ID for pushing", ) args = parser.parse_args() token = os.environ.get("HF_TOKEN") if not token: print("⚠️ No HF_TOKEN found. Some requests may be rate-limited.") collector = EvalsCollector(token=token) collector.collect_all() # Print leaderboard summary print("\n" + "=" * 60) print("📊 EVALUATION LEADERBOARD") print("=" * 60) leaderboard = collector.get_leaderboard() for entry in leaderboard[:20]: print(f"{entry['model_id']:40} | {entry['benchmark']:12} | {entry['score']:6.2f}") if len(leaderboard) > 20: print(f" ... and {len(leaderboard) - 20} more entries") print("=" * 60) print(f"Total entries: {len(leaderboard)}") print(f"Models with scores: {len(set(r['model_id'] for r in leaderboard))}") # Save locally collector.save_json(args.output) # Push to hub if requested if args.push_to_hub: collector.push_to_hub(args.repo_id) if __name__ == "__main__": main() ================================================ FILE: apps/evals-leaderboard/requirements.txt ================================================ gradio requests pyyaml huggingface_hub ================================================ FILE: apps/hackers-leaderboard/README.md ================================================ --- title: Hackers Leaderboard emoji: 🏆 colorFrom: purple colorTo: pink sdk: gradio sdk_version: 5.50.0 app_file: app.py pinned: false --- # Hackers Leaderboard Tracks engagement from the [hf-skills](https://huggingface.co/hf-skills) organization for the hackathon leaderboard. ## How Points Work Simple and fair - **1 point per activity**: | Activity | Points | |----------|--------| | 💬 Open a discussion | 1 | | 📝 Post a comment | 1 | | 🔀 Open a PR | 1 | | 📦 Own/create a repo | 1 | ## Scripts ### Collect Points ```bash # Collect org activity only HF_TOKEN=$HF_TOKEN python collect_points.py # Also scan trending repos for member PRs/discussions HF_TOKEN=$HF_TOKEN python collect_points.py --scan-external # Scan only specific repo types HF_TOKEN=$HF_TOKEN python collect_points.py --scan-external --repo-type models HF_TOKEN=$HF_TOKEN python collect_points.py --scan-external --repo-type models datasets # Push to HF dataset HF_TOKEN=$HF_TOKEN python collect_points.py --scan-external --push-to-hub # Custom output python collect_points.py --output my_leaderboard.json --repo-id my-org/my-dataset ``` ### Options | Flag | Description | |------|-------------| | `--scan-external` | Scan trending repos across Hub for member activity | | `--repo-type` | Filter external scan to: `models`, `datasets`, `spaces` | | `--push-to-hub` | Push results to HF dataset | | `--repo-id` | Target dataset repo (default: `hf-skills/hackers-leaderboard`) | | `--output` | Local JSON output path | ### Run the App ```bash HF_TOKEN=$HF_TOKEN python app.py ``` ## API The collector scans: - All models, datasets, and spaces in the org - All discussions and PRs on those repos - All comments on discussions Results are saved as JSONL for easy dataset consumption. ## Output Format ```json { "username": "user123", "total_points": 15, "discussions_opened": 3, "comments_made": 8, "prs_opened": 2, "repos_owned": 2 } ``` ================================================ FILE: apps/hackers-leaderboard/app.py ================================================ #!/usr/bin/env python3 """ Hackers Leaderboard - Gradio app for displaying engagement from hf-skills org. Reads leaderboard data from the hf-skills/hackers-leaderboard dataset. Run collect_points.py separately to update the dataset. Usage: python app.py """ from __future__ import annotations import json import gradio as gr import requests TABLE_HEADERS = [ "Rank", "Username", "Points", "💬 Discussions", ] TABLE_DATATYPES = [ "number", "markdown", "number", ] DATASET_REPO = "hf-skills/hackers-leaderboard" LEADERBOARD_URL = f"https://huggingface.co/datasets/{DATASET_REPO}/raw/main/data/leaderboard.jsonl" METADATA_URL = f"https://huggingface.co/datasets/{DATASET_REPO}/raw/main/data/metadata.json" def format_username(username: str) -> str: """Format username as a clickable link.""" return f"[{username}](https://huggingface.co/{username})" def fetch_leaderboard() -> tuple[list[dict], dict]: """Fetch leaderboard data from the HF dataset.""" # Fetch leaderboard JSONL resp = requests.get(LEADERBOARD_URL, timeout=30) resp.raise_for_status() leaderboard = [json.loads(line) for line in resp.text.strip().split("\n") if line] # Fetch metadata resp = requests.get(METADATA_URL, timeout=30) resp.raise_for_status() metadata = resp.json() return leaderboard, metadata def refresh_handler() -> tuple[str, list[list]]: """Refresh the leaderboard data from the dataset.""" try: leaderboard, metadata = fetch_leaderboard() # Build table rows rows = [] for i, entry in enumerate(leaderboard, 1): rows.append( [ i, format_username(entry["username"]), entry["prs_opened"], ] ) status = "\n".join( [ f"**Data from:** [{DATASET_REPO}](https://huggingface.co/datasets/{DATASET_REPO})", f"**Last updated:** {metadata.get('generated_at', 'unknown')}", f"**Participants:** {metadata.get('total_participants', len(leaderboard))}", f"**Total points:** {metadata.get('total_points', sum(e['total_points'] for e in leaderboard))}", ] ) return status, rows except Exception as e: return f"❌ Failed to load leaderboard: {e}", [] with gr.Blocks() as demo: gr.HTML( """
Humanity's Last Hackathon (of 2025)

🏆 Humanity's Last Hackathon Leaderboard

""" ) leaderboard_table = gr.Dataframe( headers=TABLE_HEADERS, datatype=TABLE_DATATYPES, interactive=False, wrap=True, ) status_box = gr.Markdown("Click refresh to load the leaderboard...") demo.load( refresh_handler, outputs=[status_box, leaderboard_table], ) gr.Markdown( """ --- **Links:** - [Join hf-skills](https://huggingface.co/organizations/hf-skills/share/KrqrmBxkETjvevFbfkXeezcyMbgMjjMaOp) - [Quest Instructions](https://github.com/huggingface/skills/tree/main/apps/quests) - [GitHub Repository](https://github.com/huggingface/skills) """ ) if __name__ == "__main__": demo.launch() ================================================ FILE: apps/hackers-leaderboard/collect_points.py ================================================ #!/usr/bin/env python3 """ Collect engagement points from the hf-skills organization. Tracks user activity across all repos (models, datasets, spaces) and counts: - 1 point per discussion opened - 1 point per comment made - 1 point per PR opened - 1 point per repo owned/created Results are saved to a dataset for the hackers leaderboard. Usage: HF_TOKEN=$HF_TOKEN python collect_points.py [--push-to-hub] """ from __future__ import annotations import argparse import json import os from dataclasses import dataclass, field from datetime import datetime, timezone from typing import Any import requests API_BASE = "https://huggingface.co/api" ORG_NAME = "hf-skills" USER_AGENT = "hf-skills-leaderboard/1.0" DISCUSSION_LIMIT = 100 # Max discussions to fetch per repo TRENDING_LIMIT = 50 # Number of trending repos to scan for external PRs @dataclass class UserStats: """Tracks engagement stats for a single user.""" username: str is_org_member: bool = True discussions_opened: int = 0 comments_made: int = 0 prs_opened: int = 0 repos_owned: int = 0 activities: list[dict[str, Any]] = field(default_factory=list) @property def total_points(self) -> int: return self.discussions_opened + self.comments_made + self.prs_opened + self.repos_owned def to_dict(self) -> dict[str, Any]: return { "username": self.username, "is_org_member": self.is_org_member, "total_points": self.total_points, "discussions_opened": self.discussions_opened, "comments_made": self.comments_made, "prs_opened": self.prs_opened, "repos_owned": self.repos_owned, } class PointsCollector: """Collects engagement points from the hf-skills organization.""" def __init__(self, token: str | None = None) -> None: self.token = token self.session = requests.Session() self.session.headers.update({"User-Agent": USER_AGENT}) if token: self.session.headers.update({"Authorization": f"Bearer {token}"}) self.user_stats: dict[str, UserStats] = {} self.logs: list[str] = [] def log(self, message: str) -> None: """Add a log message.""" print(message) self.logs.append(message) def _fetch_org_members(self) -> list[str]: """Fetch all members of the organization.""" try: from huggingface_hub import HfApi api = HfApi(token=self.token) members = list(api.list_organization_members(ORG_NAME)) usernames = [m.username for m in members if m.username] self.log(f"👥 Found {len(usernames)} organization members") return usernames except Exception as e: self.log(f"⚠️ Failed to fetch org members: {e}") # Fallback: try the API directly try: url = f"{API_BASE}/organizations/{ORG_NAME}/members" response = self.session.get(url, timeout=30) response.raise_for_status() members = response.json() usernames = [m.get("user") or m.get("username") or m.get("name") for m in members] usernames = [u for u in usernames if u] self.log(f"👥 Found {len(usernames)} organization members (via API)") return usernames except Exception as e2: self.log(f"⚠️ Fallback also failed: {e2}") return [] def collect_all(self) -> dict[str, UserStats]: """Collect points from all repos in the organization.""" self.log(f"🔍 Scanning organization: {ORG_NAME}") # First, get all org members and initialize their stats members = self._fetch_org_members() for username in members: self.user_stats[username] = UserStats(username=username) # Collect from all repo types models = self._list_repos("models") datasets = self._list_repos("datasets") spaces = self._list_repos("spaces") all_repos = [ *[(r, "model") for r in models], *[(r, "dataset") for r in datasets], *[(r, "space") for r in spaces], ] self.log(f"📦 Found {len(models)} models, {len(datasets)} datasets, {len(spaces)} spaces") for repo_info, repo_type in all_repos: repo_id = repo_info.get("id") or repo_info.get("modelId") if not repo_id: continue # Credit repo owner owner = repo_info.get("author") or repo_id.split("/")[0] if owner and owner != ORG_NAME: self._add_point(owner, "repos_owned", repo_id, "repo_created") # Scan discussions self._scan_discussions(repo_id, repo_type) return dict(self.user_stats) def scan_external_repos(self, repo_types: list[str] | None = None) -> None: """Scan trending repos across the Hub for PRs by org members. Args: repo_types: List of repo types to scan. Options: "models", "datasets", "spaces". If None, scans all types. """ org_members = set(self.user_stats.keys()) if not org_members: self.log("⚠️ No org members loaded. Run collect_all() first.") return if repo_types is None: repo_types = ["models", "datasets", "spaces"] self.log(f"🌐 Scanning trending repos for PRs by {len(org_members)} org members...") self.log(f"📂 Repo types: {', '.join(repo_types)}") for repo_type in repo_types: trending = self._fetch_trending(repo_type) self.log(f"📈 Scanning {len(trending)} trending {repo_type}...") for repo_info in trending: repo_id = repo_info.get("id") or repo_info.get("modelId") if not repo_id: continue # Skip org repos (already scanned) if repo_id.startswith(f"{ORG_NAME}/"): continue # Scan for PRs/discussions by each org member using author filter self._scan_repo_for_members(repo_id, repo_type, org_members) def _fetch_trending(self, repo_type: str) -> list[dict[str, Any]]: """Fetch trending repos of a given type.""" endpoint = f"{API_BASE}/{repo_type}" params = {"sort": "trendingScore", "limit": TRENDING_LIMIT} try: response = self.session.get(endpoint, params=params, timeout=30) response.raise_for_status() return response.json() except requests.RequestException as e: self.log(f"⚠️ Failed to fetch trending {repo_type}: {e}") return [] def _scan_repo_for_members(self, repo_id: str, repo_type: str, org_members: set[str]) -> None: """Scan a repo's discussions for activity by org members using author filter.""" # Parse namespace and repo from repo_id parts = repo_id.split("/") if len(parts) != 2: return namespace, repo = parts for member in org_members: # Use author filter for efficient querying self._fetch_member_discussions( repo_type=repo_type, namespace=namespace, repo=repo, author=member, discussion_type="pull_request", ) self._fetch_member_discussions( repo_type=repo_type, namespace=namespace, repo=repo, author=member, discussion_type="discussion", ) def _fetch_member_discussions( self, repo_type: str, namespace: str, repo: str, author: str, discussion_type: str = "all", ) -> None: """Fetch discussions for a specific author on a repo. Uses: GET /api/{repoType}/{namespace}/{repo}/discussions?author={author}&type={type} """ url = f"{API_BASE}/{repo_type}/{namespace}/{repo}/discussions" params = { "author": author, "type": discussion_type, "status": "all", } try: response = self.session.get(url, params=params, timeout=30) response.raise_for_status() data = response.json() except requests.RequestException: return discussions = data.get("discussions", []) repo_id = f"{namespace}/{repo}" for discussion in discussions: is_pr = discussion.get("isPullRequest", False) disc_num = discussion.get("num") if is_pr: self._add_point(author, "prs_opened", repo_id, "external_pr", disc_num) self.log(f"🔀 Found PR by {author} on {repo_id}") else: self._add_point(author, "discussions_opened", repo_id, "external_discussion", disc_num) self.log(f"💬 Found discussion by {author} on {repo_id}") # Count comments on the discussion num_comments = discussion.get("numComments", 0) if num_comments > 0: self._fetch_discussion_comments(repo_type, namespace, repo, disc_num, author) def _fetch_discussion_comments( self, repo_type: str, namespace: str, repo: str, disc_num: int, target_author: str, ) -> None: """Fetch comments on a discussion and count those by target author.""" url = f"{API_BASE}/{repo_type}/{namespace}/{repo}/discussions/{disc_num}" try: response = self.session.get(url, timeout=30) response.raise_for_status() data = response.json() except requests.RequestException: return repo_id = f"{namespace}/{repo}" events = data.get("events", []) for event in events: if event.get("type") == "comment": author_info = event.get("author", {}) or {} author = author_info.get("name") or author_info.get("fullname") if author == target_author: self._add_point(author, "comments_made", repo_id, "external_comment", disc_num) def _list_repos(self, repo_type: str) -> list[dict[str, Any]]: """List all repos of a given type in the organization.""" endpoint = f"{API_BASE}/{repo_type}" params = {"author": ORG_NAME, "limit": 1000} try: response = self.session.get(endpoint, params=params, timeout=30) response.raise_for_status() return response.json() except requests.RequestException as e: self.log(f"⚠️ Failed to list {repo_type}: {e}") return [] def _scan_discussions(self, repo_id: str, repo_type: str) -> None: """Scan all discussions for a repo and count engagement.""" # Map repo type to API path type_map = {"model": "models", "dataset": "datasets", "space": "spaces"} api_type = type_map.get(repo_type, "models") url = f"{API_BASE}/{api_type}/{repo_id}/discussions" try: response = self.session.get(url, params={"limit": DISCUSSION_LIMIT}, timeout=30) response.raise_for_status() data = response.json() except requests.RequestException as e: self.log(f"⚠️ Failed to get discussions for {repo_id}: {e}") return discussions = data.get("discussions", []) if not discussions: return self.log(f"💬 {repo_id}: found {len(discussions)} discussions") for discussion in discussions: self._process_discussion(repo_id, api_type, discussion) def _process_discussion(self, repo_id: str, api_type: str, discussion: dict[str, Any]) -> None: """Process a single discussion and its comments.""" author_info = discussion.get("author", {}) or {} author = author_info.get("name") or author_info.get("fullname") is_pr = discussion.get("isPullRequest", False) disc_num = discussion.get("num") if author and author != ORG_NAME: activity_type = "pr_opened" if is_pr else "discussion_opened" point_type = "prs_opened" if is_pr else "discussions_opened" self._add_point(author, point_type, repo_id, activity_type, disc_num) # Fetch discussion details to get comments if disc_num: self._fetch_comments(repo_id, api_type, disc_num) def _fetch_comments(self, repo_id: str, api_type: str, disc_num: int) -> None: """Fetch and count comments on a discussion.""" url = f"{API_BASE}/{api_type}/{repo_id}/discussions/{disc_num}" try: response = self.session.get(url, timeout=30) response.raise_for_status() data = response.json() except requests.RequestException: # Silently skip failed comment fetches return events = data.get("events", []) for event in events: event_type = event.get("type") # Count comments (not the initial post, status changes, etc.) if event_type == "comment": author_info = event.get("author", {}) or {} author = author_info.get("name") or author_info.get("fullname") if author and author != ORG_NAME: self._add_point(author, "comments_made", repo_id, "comment", disc_num) def _add_point( self, username: str, point_type: str, repo_id: str, activity_type: str, disc_num: int | None = None, ) -> None: """Add a point to a user's stats.""" if not username: return # Initialize stats for users not in the org (external contributors) if username not in self.user_stats: self.user_stats[username] = UserStats(username=username, is_org_member=False) stats = self.user_stats[username] current = getattr(stats, point_type, 0) setattr(stats, point_type, current + 1) stats.activities.append( { "type": activity_type, "repo_id": repo_id, "discussion_num": disc_num, "timestamp": datetime.now(timezone.utc).isoformat(), } ) def get_leaderboard(self) -> list[dict[str, Any]]: """Get the leaderboard sorted by total points.""" leaderboard = [stats.to_dict() for stats in self.user_stats.values()] leaderboard.sort(key=lambda x: x["total_points"], reverse=True) return leaderboard def save_json(self, filepath: str) -> None: """Save the leaderboard to a JSON file.""" leaderboard = self.get_leaderboard() output = { "generated_at": datetime.now(timezone.utc).isoformat(), "organization": ORG_NAME, "total_participants": len(leaderboard), "leaderboard": leaderboard, } with open(filepath, "w") as f: json.dump(output, f, indent=2) self.log(f"💾 Saved leaderboard to {filepath}") def push_to_hub(self, repo_id: str = "hf-skills/hackers-leaderboard") -> None: """Push the leaderboard data to a HF dataset.""" try: from huggingface_hub import HfApi except ImportError: self.log("❌ huggingface_hub not installed. Run: pip install huggingface_hub") return api = HfApi() leaderboard = self.get_leaderboard() # Create dataset as JSONL jsonl_content = "\n".join(json.dumps(row) for row in leaderboard) # Also create a metadata file metadata = { "generated_at": datetime.now(timezone.utc).isoformat(), "organization": ORG_NAME, "total_participants": len(leaderboard), "total_points": sum(row["total_points"] for row in leaderboard), } try: # Create repo if it doesn't exist api.create_repo(repo_id=repo_id, repo_type="dataset", exist_ok=True) self.log(f"📁 Ensured dataset repo exists: {repo_id}") # Upload leaderboard data api.upload_file( path_or_fileobj=jsonl_content.encode(), path_in_repo="data/leaderboard.jsonl", repo_id=repo_id, repo_type="dataset", commit_message=f"Update leaderboard - {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M')} UTC", ) # Upload metadata api.upload_file( path_or_fileobj=json.dumps(metadata, indent=2).encode(), path_in_repo="data/metadata.json", repo_id=repo_id, repo_type="dataset", commit_message=f"Update metadata - {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M')} UTC", ) self.log(f"🚀 Pushed leaderboard to {repo_id}") except Exception as e: self.log(f"❌ Failed to push to hub: {e}") def main() -> None: parser = argparse.ArgumentParser(description="Collect engagement points from hf-skills organization") parser.add_argument( "--push-to-hub", action="store_true", help="Push results to HF dataset", ) parser.add_argument( "--output", type=str, default="leaderboard.json", help="Output JSON file path", ) parser.add_argument( "--repo-id", type=str, default="hf-skills/hackers-leaderboard", help="HF dataset repo ID for pushing", ) parser.add_argument( "--scan-external", action="store_true", help="Also scan trending repos for PRs/discussions by org members", ) parser.add_argument( "--repo-type", type=str, nargs="+", choices=["models", "datasets", "spaces"], default=None, help="Repo types to scan (for --scan-external). Default: all types", ) args = parser.parse_args() token = os.environ.get("HF_TOKEN") if not token: print("⚠️ No HF_TOKEN found. Some requests may be rate-limited.") collector = PointsCollector(token=token) collector.collect_all() # Optionally scan external repos for member activity if args.scan_external: collector.scan_external_repos(repo_types=args.repo_type) # Print leaderboard print("\n" + "=" * 50) print("🏆 HACKERS LEADERBOARD") print("=" * 50) leaderboard = collector.get_leaderboard() for i, entry in enumerate(leaderboard[:20], 1): print( f"{i:2}. {entry['username']:20} - {entry['total_points']:4} points " f"(💬{entry['discussions_opened']} 📝{entry['comments_made']} " f"🔀{entry['prs_opened']} 📦{entry['repos_owned']})" ) if len(leaderboard) > 20: print(f" ... and {len(leaderboard) - 20} more participants") print("=" * 50) print(f"Total participants: {len(leaderboard)}") print(f"Total points awarded: {sum(e['total_points'] for e in leaderboard)}") # Save locally collector.save_json(args.output) # Push to hub if requested if args.push_to_hub: collector.push_to_hub(args.repo_id) if __name__ == "__main__": main() ================================================ FILE: apps/hackers-leaderboard/requirements.txt ================================================ gradio>=5.50.0 huggingface_hub>=1.1.4 requests>=2.32.5 ================================================ FILE: apps/quests/02_evaluate-hub-model.md ================================================ # Week 1: Evaluate a Hub Model 📣 TASK: Add evaluation results to model cards across the Hub. Together, we're building a distributed leaderboard of open source model performance. >[!NOTE] > Bonus XP for contributing to the leaderboard application. Open a PR [on the hub](https://huggingface.co/spaces/hf-skills/distributed-leaderboard/discussions) or [on GitHub](https://github.com/huggingface/skills/blob/main/apps/evals-leaderboard/app.py) to get your (bonus) XP. ## Why This Matters Model cards without evaluation data are hard to compare. By adding structured eval results to metadata, we make models easier to compare and review. Your contributions power leaderboards and help the community find the best models for their needs. Also, by doing this in a distributed way, we can share our evaluation results with the community. ## Goals - Add eval scores to the 100 trending models on the Hub - Include AIME 2025, BigBenchHard, LiveCodeBench, MMLU, ARC on trending models. - It is ok to include a subset of the benchmarks available for the model. - Build a leaderboard application that shows the evaluation results for the trending models. ## XP Tiers Taking part is simple. We need to get model authors to show evaluation results in their model cards. This is a clean up job! | Tier | XP | Description | What Counts | |-----------------|-------|---------------------------------------------------------------|-----------------------------------------------| | 🐢 Contributor | 1 XP | Extract evaluation results from one benchmark and update its model card. | Any PR on the repo with evaluation data. | | 🐕 Evaluator | 5 XP | Import scores from third-party benchmarks like Artificial Analysis. | Undefined benchmark scores and merged PRs. | | 🦁 Advanced | 20 XP| Run your own evaluation with inspect-ai and publish results. | Original eval run and merged PR. | | 🐉 Bonus | 20 XP| Contribute to the leaderboard application. | Any Merged PR on the hub or GitHub. | | 🤢 Slop | -20 XP | Opening none useful PRs. | Duplicate PRs, Incorrect Eval Scores, Incorrect Benchmark Scores | > [!WARNING] > This hackathon is about advancing the state of open source AI. We want useful PRs that help everyone out, not just metrics. ## The Skill Use `hf-evaluation/` for this quest. Key capabilities: - Extract evaluation tables from existing README content posted by model authors. - Import benchmark scores from [Artificial Analysis](https://artificial.com/). - Run your own evals with [inspect-ai](https://github.com/UKGovernmentBEIS/inspect_ai) on [HF Jobs](https://huggingface.co/docs/huggingface_hub/en/guides/jobs). - Update model-index metadata in the model card. >[!NOTE] > Take a look at the [SKILL.md](https://github.com/huggingface/skills/blob/main/hf-evaluation/SKILL.md) for more details. ### Extract Evaluation Tables from README 1. Pick a Hub model without evaluation data from *trending models* on the hub 2. Use the skill to extract or add a benchmark score 3. Create a PR (or push directly if you own the model) The agent will use this script to extract evaluation tables from the model's README. ```bash python hf-evaluation/scripts/evaluation_manager.py extract-readme \ --repo-id "model-author/model-name" --dry-run ``` ### Import Scores from Artificial Analysis 1. Find a model with benchmark data on external sites 2. Use `import-aa` to fetch scores from Artificial Analysis API 3. Create a PR with properly attributed evaluation data The agent will use this script to fetch scores from Artificial Analysis API and add them to the model card. ```bash python hf-evaluation/scripts/evaluation_manager.py import-aa \ --creator-slug "anthropic" --model-name "claude-sonnet-4" \ --repo-id "target/model" --create-pr ``` ### Run your own evaluation with inspect-ai and publish results. 1. Choose an eval task (MMLU, GSM8K, HumanEval, etc.) 2. Run the evaluation on HF Jobs infrastructure 3. Update the model card with your results and methodology The agent will use this script to run the evaluation on HF Jobs infrastructure and update the model card with the results. ```bash HF_TOKEN=$HF_TOKEN hf jobs uv run hf-evaluation/scripts/inspect_eval_uv.py \ --flavor a10g-small --secret HF_TOKEN=$HF_TOKEN \ -- --model "meta-llama/Llama-2-7b-hf" --task "mmlu" ``` ## Tips - Always use `--dry-run` first to preview changes before pushing - Check for transposed tables where models are rows and benchmarks are columns - Be careful with PRs for models you don't own — most maintainers appreciate eval contributions but be respectful. - Manually validate the extracted scores and close PRs if needed. ## Resources - [SKILL.md](../../hf-evaluation/SKILL.md) — Full skill documentation - [Example Usage](../../hf-evaluation/examples/USAGE_EXAMPLES.md) — Worked examples - [Metric Mapping](../../hf-evaluation/examples/metric_mapping.json) — Standard metric types ================================================ FILE: apps/quests/03_publish-hub-dataset.md ================================================ # Week 2: Publish a Hub Dataset Create and share high-quality datasets on the Hub. Good data is the foundation of good models—help the community by contributing datasets others can train on. ## Why This Matters The best open source models are built on openly available datasets. By publishing well-documented, properly structured datasets, you're directly enabling the next generation of model development. Quality matters more than quantity. ## The Skill Use `hf-datasets/` for this quest. Key capabilities: - Initialize dataset repos with proper structure - Multi-format support: chat, classification, QA, completion, tabular - Template-based validation for data quality - Streaming uploads without downloading entire datasets ```bash # Quick setup with a template python hf-datasets/scripts/dataset_manager.py quick_setup \ --repo_id "your-username/dataset-name" --template chat ``` ## XP Tiers ### 🐢 Starter — 50 XP **Upload a small, clean dataset with a complete dataset card.** 1. Create a dataset with ≤1,000 rows 2. Write a dataset card covering: license, splits, and data provenance 3. Upload to the Hub under the hackathon organization (or your own account) **What counts:** Clean data, clear documentation, proper licensing. ```bash python hf-datasets/scripts/dataset_manager.py init \ --repo_id "hf-skills/your-dataset-name" python hf-datasets/scripts/dataset_manager.py add_rows \ --repo_id "hf-skills/your-dataset-name" \ --template classification \ --rows_json "$(cat your_data.json)" ``` ### 🐕 Standard — 100 XP **Publish a conversational dataset with a complete dataset card.** 1. Create a dataset with ≤1,000 rows 2. Write a dataset card covering: license and splits. 3. Upload to the Hub under the hackathon organization. **What counts:** Clean data, clear documentation, proper licensing. ### 🦁 Advanced — 200 XP **Translate a dataset into multiple languages and publish it on the Hub.** 1. Find a dataset on the Hub 2. Translate the dataset into multiple languages 3. Publish the translated datasets on the Hub under the hackathon organization **What counts:** Translated datasets and merged PRs. ## Resources - [SKILL.md](../../hf-datasets/SKILL.md) — Full skill documentation - [Templates](../../hf-datasets/templates/) — JSON templates for each format - [Examples](../../hf-datasets/examples/) — Sample data and system prompts --- **Next Quest:** [Supervised Fine-Tuning](04_sft-finetune-hub.md) ================================================ FILE: apps/quests/04_sft-finetune-hub.md ================================================ # Week 3: Supervised Fine-Tuning on the Hub Fine-tune and share models on the Hub. Take a base model, train it on your data, and publish the result for the community to use. ## Why This Matters Fine-tuning is how we adapt foundation models to specific tasks. By sharing fine-tuned models—along with your training methodology—you're giving the community ready-to-use solutions and reproducible recipes they can learn from. ## The Skill Use `hf-llm-trainer/` for this quest. Key capabilities: - **SFT** (Supervised Fine-Tuning) — Standard instruction tuning - **DPO** (Direct Preference Optimization) — Alignment from preference data - **GRPO** (Group Relative Policy Optimization) — Online RL training - Cloud GPU training on HF Jobs—no local setup required - Trackio integration for real-time monitoring - GGUF conversion for local deployment Your coding agent uses `hf_jobs()` to submit training scripts directly to HF infrastructure. ## XP Tiers We'll announce the XP tiers for this quest soon. ## Resources - [SKILL.md](../../hf-llm-trainer/SKILL.md) — Full skill documentation - [SFT Example](../../hf-llm-trainer/scripts/train_sft_example.py) — Production SFT template - [DPO Example](../../hf-llm-trainer/scripts/train_dpo_example.py) — Production DPO template - [GRPO Example](../../hf-llm-trainer/scripts/train_grpo_example.py) — Production GRPO template - [Training Methods](../../hf-llm-trainer/references/training_methods.md) — Method selection guide - [Hardware Guide](../../hf-llm-trainer/references/hardware_guide.md) — GPU selection ================================================ FILE: apps/quests/README.md ================================================ --- title: README emoji: 🐠 colorFrom: yellow colorTo: gray sdk: static pinned: false --- # Humanity's Last Hackathon (of 2025) Humanity's Last Hackathon (of 2025) Welcome to our hackathon! Whether you’re a tooled up ML engineer, a classicist NLP dev, or an AGI pilled vibe coder, this hackathon is going to be hard work! We’re going to take the latest and greatest coding agents and use them to level up open source AI. After all, **why use December to relax and spend time with loved ones, when you can solve AI for all humanity?** Jokes aside, this hackathon is not about learning skills from zero or breaking things down in their simplest components. It’s about collaborating, shipping, and making a difference for the open source community. ## What We're Building Over four weeks, we're using coding agents to level up the open source AI ecosystem: - **Week 1** — Evaluate models and build a distributed leaderboard - **Week 2** — Create high-quality datasets for the community - **Week 3** — Fine-tune and share models on the Hub - **Week 4** — Sprint to the finish line together Every contribution earns XP. Top contributors make the leaderboard. Winners get prizes! Here's the schedule: | Date | Event | Link | |------|-------|------| | Dec 2 (Mon) | Week 1 Quest Released | [Evaluate a Hub Model](02_evaluate-hub-model.md) | | Dec 4 (Wed) | Livestream 1 | [Q&A 1](https://youtube.com/live/rworGSh-Rgk?feature=share) | | Dec 9 (Mon) | Week 2 Quest Released | [Publish a Hub Dataset](03_publish-hub-dataset.md) | | Dec 11 (Wed) | Livestream 2 | TBA | | Dec 16 (Mon) | Week 3 Quest Released | [Supervised Fine-Tuning](04_sft-finetune-hub.md) | | Dec 18 (Wed) | Livestream 3 | TBA | | Dec 23 (Mon) | Week 4 Community Sprint | TBA | | Dec 31 (Tue) | Hackathon Ends | TBA ## Getting Started ### 1. Join the Organization Join [hf-skills](https://huggingface.co/organizations/hf-skills/share/KrqrmBxkETjvevFbfkXeezcyMbgMjjMaOp) on Hugging Face. This is where your contributions will be tracked and updated on the leaderboard. ### 2. Set Up Your Coding Agent Use whatever coding agent you prefer: - **Claude Code** — `claude` in your terminal - **Codex** — `codex` CLI - **Gemini CLI** — `gemini` in your terminal - **Cursor / Windsurf** — IDE-based agents - **Open source** — aider, continue, etc. The skills in this repo work with any agent that can read markdown instructions and run Python scripts. To install the skills, follow the instructions in the [README](../README.md). ### 3. Get Your HF Token Most quests require a Hugging Face token with write access: ```bash # mac/linux curl -LsSf https://hf.co/cli/install.sh | bash # windows powershell -ExecutionPolicy ByPass -c "irm https://hf.co/cli/install.ps1 | iex" # Login (creates/stores your token) hf auth login ``` This will set your `HF_TOKEN` environment variable. ### 4. Clone the Skills Repo ```bash git clone https://github.com/huggingface/skills.git cd skills ``` Point your coding agent at the relevant configuration. Check the [README](../README.md) for instructions on how to use the skills with your coding agent. ## Your First Quest **Week 1 is live!** Head to [02_evaluate-hub-model.md](02_evaluate-hub-model.md) to start evaluating models and climb the leaderboard. [Leaderboard](https://hf-skills-hacker-leaderboard.hf.space) ## Earning XP Each quest has three tiers: | Tier | What it takes | XP | |------|---------------|-----| | 🐢 | Complete the basics | 50-75 XP | | 🐕 | Go deeper with more features | 100-125 XP | | 🦁 | Ship something impressive | 200-225 XP | You can complete multiple tiers, and you can complete the same quest multiple times with different models/datasets/spaces. ## Getting Help - [Discord](https://discord.com/channels/879548962464493619/1442881667986624554) — Join the Hugging Face Discord for real-time help - [Livestreams](https://www.youtube.com/@HuggingFace/streams) — Weekly streams with walkthroughs and Q&A - [Issues](https://github.com/huggingface/skills/issues) — Open an issue in this repo if you're stuck To join the Hackathon, join the organization on the hub and setup your coding agent. Ready? Let's ship some AI. 🚀 ================================================ FILE: gemini-extension.json ================================================ { "name": "huggingface-skills", "description": "Provides access to the Hugging Face Skills.", "version": "0.1.0", "contextFileName": "agents/AGENTS.md", "mcpServers": { "huggingface-skills": { "httpUrl": "https://huggingface.co/mcp?login" } } } ================================================ FILE: hf-mcp/skills/hf-mcp/SKILL.md ================================================ --- name: hf-mcp description: Use Hugging Face Hub via MCP server tools. Search models, datasets, Spaces, papers. Get repo details, fetch documentation, run compute jobs, and use Gradio Spaces as AI tools. Available when connected to the HF MCP server. --- # Hugging Face MCP Server Connect AI assistants to the Hugging Face Hub. Setup: https://huggingface.co/settings/mcp ## Use Cases & Examples ### Find the Best Model for a Task ``` User: "Find the best model for code generation" 1. model_search(task="text-generation", query="code", sort="trendingScore", limit=10) 2. hub_repo_details(repo_ids=["top-result-id"], include_readme=true) ``` ### Compare Models from Different Providers ``` User: "Compare Llama vs Qwen for text generation" 1. model_search(author="meta-llama", task="text-generation", sort="downloads", limit=5) 2. model_search(author="Qwen", task="text-generation", sort="downloads", limit=5) 3. hub_repo_details(repo_ids=["meta-llama/Llama-3.2-1B", "Qwen/Qwen3-8B"], include_readme=true) ``` ### Find Training Datasets ``` User: "Find datasets for sentiment analysis in English" 1. dataset_search(query="sentiment", tags=["language:en", "task_categories:text-classification"], sort="downloads") 2. hub_repo_details(repo_ids=["top-dataset-id"], repo_type="dataset", include_readme=true) ``` ### Discover AI Tools (MCP Spaces) ``` User: "Find a tool that can remove image backgrounds" 1. space_search(query="background removal", mcp=true) 2. dynamic_space(operation="view_parameters", space_name="result-space-id") 3. dynamic_space(operation="invoke", space_name="result-space-id", parameters="{...}") ``` ### Generate Images ``` User: "Create an image of a robot reading a book" 1. dynamic_space(operation="discover") # See available tasks 2. gr1_flux1_schnell_infer(prompt="a robot sitting in a library reading a book, warm lighting, detailed") ``` ### Research a Topic ``` User: "What are the latest papers on RLHF?" 1. paper_search(query="reinforcement learning from human feedback", results_limit=10) 2. hub_repo_details(repo_ids=["paper-linked-model"], include_readme=true) # If paper links to models ``` ### Learn How to Use a Library ``` User: "How do I fine-tune with LoRA using PEFT?" 1. hf_doc_search(query="LoRA fine-tuning", product="peft") 2. hf_doc_fetch(doc_url="https://huggingface.co/docs/peft/...") ``` ### Run a Quick GPU Job ``` User: "Run this Python script on a GPU" hf_jobs(operation="uv", args={ "script": "# /// script\n# dependencies = [\"torch\"]\n# ///\nimport torch\nprint(torch.cuda.is_available())", "flavor": "t4-small" }) ``` ### Train a Model on Cloud GPU ``` User: "Run my training script on an A10G" hf_jobs(operation="run", args={ "image": "pytorch/pytorch:2.5.1-cuda12.4-cudnn9-runtime", "command": ["/bin/sh", "-lc", "pip install transformers trl && python train.py"], "flavor": "a10g-small", "secrets": {"HF_TOKEN": "$HF_TOKEN"} }) ``` ### Check Job Status ``` User: "What's happening with my training job?" 1. hf_jobs(operation="ps") 2. hf_jobs(operation="logs", args={"job_id": "job-xxxxx"}) ``` ### Explore What's Trending ``` User: "What models are trending right now?" model_search(sort="trendingScore", limit=20) ``` ### Get Model Card Details ``` User: "Tell me about Mistral-7B" hub_repo_details(repo_ids=["mistralai/Mistral-7B-v0.1"], include_readme=true) ``` ### Find Quantized Models ``` User: "Find GGUF versions of Llama 3" model_search(query="Llama 3 GGUF", sort="downloads", limit=10) ``` ### Use a Gradio Space as a Tool ``` User: "Transcribe this audio file" 1. space_search(query="speech to text transcription", mcp=true) 2. dynamic_space(operation="view_parameters", space_name="openai/whisper") 3. dynamic_space(operation="invoke", space_name="openai/whisper", parameters="{\"audio\": \"...\"}") ``` ### Schedule Recurring Jobs ``` User: "Run this data sync every day at midnight" hf_jobs(operation="scheduled uv", args={ "script": "...", "cron": "0 0 * * *", "flavor": "cpu-basic" }) ``` ## Tool Selection Guide | Goal | Tool | |------|------| | Find models | `model_search` | | Find datasets | `dataset_search` | | Find Spaces/apps | `space_search` | | Find papers | `paper_search` | | Get repo README/details | `hub_repo_details` | | Learn library usage | `hf_doc_search` → `hf_doc_fetch` | | Run code on GPU/CPU | `hf_jobs` | | Use Gradio apps as tools | `dynamic_space` | | Generate images | `gr1_flux1_schnell_infer` or `dynamic_space` | | Check auth | `hf_whoami` | ## Tips - Use `sort="trendingScore"` to find what's popular now - Use `sort="downloads"` to find battle-tested options - Set `mcp=true` in `space_search` to find Spaces usable as tools - Use `include_readme=true` in `hub_repo_details` for full model/dataset documentation - For jobs accessing private repos, always include `secrets: {"HF_TOKEN": "$HF_TOKEN"}` - Use `dynamic_space(operation="discover")` to see all available Space-based tasks ================================================ FILE: scripts/AGENTS_TEMPLATE.md ================================================ You have additional SKILLs documented in directories containing a "SKILL.md" file. These skills are: {{#skills}} - {{name}} -> "{{path}}/SKILL.md" {{/skills}} IMPORTANT: You MUST read the SKILL.md file whenever the description of the skills matches the user intent, or may help accomplish their task. {{#skills}} {{name}}: `{{description}}` {{/skills}} Paths referenced within SKILL folders are relative to that SKILL. For example the hf-datasets `scripts/example.py` would be referenced as `hf-datasets/scripts/example.py`. ================================================ FILE: scripts/UV_RULES.md ================================================ # UV rules Use these rules for Python scripts in this repository: 1. **Use PEP 723 inline dependencies** in each runnable script: ```python # /// script # requires-python = ">=3.10" # dependencies = ["requests"] # /// ``` 2. **Run scripts with `uv run`**, not `python ...`: ```bash uv run scripts/my_script.py --help ``` 3. **Do not document `pip install -r requirements.txt` for repo scripts** unless there is a specific fallback reason. Normal usage should not require manual installation. 4. **Do not tell users to `source .venv/bin/activate` for skill scripts.** `uv run` should be enough. 5. **If a manual install example is truly needed, use `uv pip install ...`**, not `uv add`, unless you are intentionally editing a project-managed environment. 6. **For Hugging Face Jobs UV workloads, use `hf jobs uv run ...`**. ================================================ FILE: scripts/generate_agents.py ================================================ #!/usr/bin/env -S uv run # /// script # requires-python = ">=3.10" # dependencies = [] # /// """Generate AGENTS.md from AGENTS_TEMPLATE.md and SKILL.md frontmatter. Also validates that marketplace.json is in sync with discovered skills, and updates the skills table in README.md. """ from __future__ import annotations import json import re import sys from pathlib import Path ROOT = Path(__file__).resolve().parent.parent TEMPLATE_PATH = ROOT / "scripts" / "AGENTS_TEMPLATE.md" OUTPUT_PATH = ROOT / "agents" / "AGENTS.md" MARKETPLACE_PATH = ROOT / ".claude-plugin" / "marketplace.json" README_PATH = ROOT / "README.md" # Markers for the auto-generated skills table in README README_TABLE_START = "" README_TABLE_END = "" def load_template() -> str: return TEMPLATE_PATH.read_text(encoding="utf-8") def parse_frontmatter(text: str) -> dict[str, str]: """Parse a minimal YAML-ish frontmatter block without external deps.""" match = re.search(r"^---\s*\n(.*?)\n---\s*", text, re.DOTALL) if not match: return {} data: dict[str, str] = {} for line in match.group(1).splitlines(): if ":" not in line: continue key, value = line.split(":", 1) data[key.strip()] = value.strip() return data def collect_skills() -> list[dict[str, str]]: skills: list[dict[str, str]] = [] for skill_md in ROOT.glob("skills/*/SKILL.md"): meta = parse_frontmatter(skill_md.read_text(encoding="utf-8")) name = meta.get("name") description = meta.get("description") if not name or not description: continue skills.append( { "name": name, "description": description, "path": str(skill_md.parent.relative_to(ROOT)), } ) # Keep deterministic order for consistent output return sorted(skills, key=lambda s: s["name"].lower()) def render(template: str, skills: list[dict[str, str]]) -> str: """Very small Mustache-like renderer that only supports a single skills loop.""" def repl(match: re.Match[str]) -> str: block = match.group(1).strip("\n") rendered_blocks = [] for skill in skills: rendered = ( block.replace("{{name}}", skill["name"]) .replace("{{description}}", skill["description"]) .replace("{{path}}", skill["path"]) ) rendered_blocks.append(rendered) return "\n".join(rendered_blocks) # Render loop blocks content = re.sub(r"{{#skills}}(.*?){{/skills}}", repl, template, flags=re.DOTALL) return content def load_marketplace() -> dict: """Load marketplace.json and return parsed structure.""" if not MARKETPLACE_PATH.exists(): raise FileNotFoundError(f"marketplace.json not found at {MARKETPLACE_PATH}") return json.loads(MARKETPLACE_PATH.read_text(encoding="utf-8")) def generate_readme_table(skills: list[dict[str, str]]) -> str: """Generate the skills table for README.md using marketplace.json names.""" marketplace = load_marketplace() plugins = {p["source"]: p for p in marketplace.get("plugins", [])} lines = [ "| Name | Description | Documentation |", "|------|-------------|---------------|", ] for skill in skills: source = f"./{skill['path']}" plugin = plugins.get(source, {}) name = plugin.get("name", skill["name"]) description = plugin.get("description", skill["description"]) doc_link = f"[SKILL.md]({skill['path']}/SKILL.md)" lines.append(f"| `{name}` | {description} | {doc_link} |") return "\n".join(lines) def update_readme(skills: list[dict[str, str]]) -> bool: """ Update the README.md skills table between markers. Returns True if the file was updated, False if markers not found. """ if not README_PATH.exists(): print(f"Warning: README.md not found at {README_PATH}", file=sys.stderr) return False content = README_PATH.read_text(encoding="utf-8") start_idx = content.find(README_TABLE_START) end_idx = content.find(README_TABLE_END) if start_idx == -1 or end_idx == -1: print( f"Warning: README.md markers not found. Add {README_TABLE_START} and " f"{README_TABLE_END} to enable table generation.", file=sys.stderr, ) return False if end_idx < start_idx: print("Warning: README.md markers are in wrong order.", file=sys.stderr) return False table = generate_readme_table(skills) new_content = ( content[: start_idx + len(README_TABLE_START)] + "\n" + table + "\n" + content[end_idx:] ) README_PATH.write_text(new_content, encoding="utf-8") return True def validate_marketplace(skills: list[dict[str, str]]) -> list[str]: """ Validate marketplace.json against discovered skills. Returns list of error messages (empty = passed). """ errors: list[str] = [] marketplace = load_marketplace() plugins = marketplace.get("plugins", []) # Build lookups (normalize paths: skill uses "skills/x", marketplace uses "./skills/x") skill_by_source = {f"./{s['path']}": s for s in skills} plugin_by_source = {p["source"]: p for p in plugins} # Check: every skill has a marketplace entry with matching name for skill in skills: expected_source = f"./{skill['path']}" if expected_source not in plugin_by_source: errors.append( f"Skill '{skill['name']}' at '{skill['path']}' is missing from marketplace.json" ) elif plugin_by_source[expected_source]["name"] != skill["name"]: errors.append( f"Name mismatch at '{expected_source}': " f"SKILL.md='{skill['name']}', marketplace.json='{plugin_by_source[expected_source]['name']}'" ) # Check: every marketplace plugin has a corresponding skill for plugin in plugins: if plugin["source"] not in skill_by_source: errors.append( f"Marketplace plugin '{plugin['name']}' at '{plugin['source']}' has no SKILL.md" ) return errors def main() -> None: template = load_template() skills = collect_skills() output = render(template, skills) OUTPUT_PATH.parent.mkdir(parents=True, exist_ok=True) OUTPUT_PATH.write_text(output, encoding="utf-8") print(f"Wrote {OUTPUT_PATH} with {len(skills)} skills.") # Validate marketplace.json errors = validate_marketplace(skills) if errors: print("\nMarketplace.json validation errors:", file=sys.stderr) for error in errors: print(f" - {error}", file=sys.stderr) sys.exit(1) print("Marketplace.json validation passed.") # Update README.md skills table if update_readme(skills): print(f"Updated {README_PATH} skills table.") if __name__ == "__main__": main() ================================================ FILE: scripts/generate_cursor_plugin.py ================================================ #!/usr/bin/env -S uv run # /// script # requires-python = ">=3.10" # dependencies = [] # /// """Generate Cursor plugin artifacts from existing repo metadata. Outputs: - .cursor-plugin/plugin.json - .mcp.json Design goals: - Keep Claude + Cursor metadata in sync. - Reuse .claude-plugin/plugin.json as primary metadata source. - Discover skills from skills/*/SKILL.md. - Reuse MCP URL from gemini-extension.json when available. """ from __future__ import annotations import argparse import json import re import sys from pathlib import Path ROOT = Path(__file__).resolve().parent.parent CLAUDE_PLUGIN_MANIFEST = ROOT / ".claude-plugin" / "plugin.json" GEMINI_EXTENSION = ROOT / "gemini-extension.json" CURSOR_PLUGIN_DIR = ROOT / ".cursor-plugin" CURSOR_PLUGIN_MANIFEST = CURSOR_PLUGIN_DIR / "plugin.json" CURSOR_MCP_CONFIG = ROOT / ".mcp.json" DEFAULT_MCP_SERVER_NAME = "huggingface-skills" DEFAULT_MCP_URL = "https://huggingface.co/mcp?login" PLUGIN_NAME_RE = re.compile(r"^[a-z0-9](?:[a-z0-9.-]*[a-z0-9])?$") def load_json(path: Path) -> dict: if not path.exists(): raise FileNotFoundError(f"Missing required file: {path}") return json.loads(path.read_text(encoding="utf-8")) def parse_frontmatter(text: str) -> dict[str, str]: match = re.search(r"^---\s*\n(.*?)\n---\s*", text, re.DOTALL) if not match: return {} data: dict[str, str] = {} for line in match.group(1).splitlines(): if ":" not in line: continue key, value = line.split(":", 1) data[key.strip()] = value.strip() return data def collect_skills() -> list[str]: skills: list[str] = [] for skill_md in sorted(ROOT.glob("skills/*/SKILL.md")): meta = parse_frontmatter(skill_md.read_text(encoding="utf-8")) name = meta.get("name", "").strip() if not name: continue skills.append(name) return skills def validate_plugin_name(name: str) -> None: if not PLUGIN_NAME_RE.match(name): raise ValueError( "Invalid plugin name in .claude-plugin/plugin.json: " f"'{name}'. Must be lowercase and match {PLUGIN_NAME_RE.pattern}" ) def build_cursor_plugin_manifest() -> dict: src = load_json(CLAUDE_PLUGIN_MANIFEST) name = src.get("name") if not isinstance(name, str) or not name: raise ValueError(".claude-plugin/plugin.json must define a non-empty 'name'") validate_plugin_name(name) skills = collect_skills() if not skills: raise ValueError("No skills discovered under skills/*/SKILL.md") manifest: dict = {"name": name, "skills": "skills", "mcpServers": ".mcp.json"} # Copy optional metadata fields when present. for key in [ "description", "version", "author", "homepage", "repository", "license", "keywords", "logo", ]: if key in src: manifest[key] = src[key] return manifest def extract_mcp_from_gemini() -> tuple[str, str]: """Return (server_name, url) from gemini-extension when available.""" if not GEMINI_EXTENSION.exists(): return DEFAULT_MCP_SERVER_NAME, DEFAULT_MCP_URL data = load_json(GEMINI_EXTENSION) servers = data.get("mcpServers") if not isinstance(servers, dict) or not servers: return DEFAULT_MCP_SERVER_NAME, DEFAULT_MCP_URL # Use first configured server as source of truth. server_name = next(iter(servers.keys())) server_cfg = servers[server_name] if not isinstance(server_cfg, dict): return DEFAULT_MCP_SERVER_NAME, DEFAULT_MCP_URL url = server_cfg.get("url") or server_cfg.get("httpUrl") or DEFAULT_MCP_URL if not isinstance(url, str) or not url.strip(): url = DEFAULT_MCP_URL return server_name, url def build_mcp_config() -> dict: server_name, url = extract_mcp_from_gemini() return { "mcpServers": { server_name: { "url": url, } } } def render_json(data: dict) -> str: return json.dumps(data, indent=2, ensure_ascii=False) + "\n" def write_or_check(path: Path, content: str, check: bool) -> bool: """Return True when file is already up-to-date (or after writing in non-check mode).""" current = path.read_text(encoding="utf-8") if path.exists() else None if current == content: return True if check: return False path.parent.mkdir(parents=True, exist_ok=True) path.write_text(content, encoding="utf-8") return True def main() -> None: parser = argparse.ArgumentParser(description="Generate Cursor plugin manifest + MCP config") parser.add_argument( "--check", action="store_true", help="Validate generated files are up-to-date without writing changes.", ) args = parser.parse_args() plugin_manifest = render_json(build_cursor_plugin_manifest()) mcp_config = render_json(build_mcp_config()) ok_plugin = write_or_check(CURSOR_PLUGIN_MANIFEST, plugin_manifest, check=args.check) ok_mcp = write_or_check(CURSOR_MCP_CONFIG, mcp_config, check=args.check) if args.check: outdated = [] if not ok_plugin: outdated.append(str(CURSOR_PLUGIN_MANIFEST.relative_to(ROOT))) if not ok_mcp: outdated.append(str(CURSOR_MCP_CONFIG.relative_to(ROOT))) if outdated: print("Generated Cursor artifacts are out of date:", file=sys.stderr) for item in outdated: print(f" - {item}", file=sys.stderr) print("Run: uv run scripts/generate_cursor_plugin.py", file=sys.stderr) sys.exit(1) print("Cursor plugin artifacts are up to date.") return print(f"Wrote {CURSOR_PLUGIN_MANIFEST.relative_to(ROOT)}") print(f"Wrote {CURSOR_MCP_CONFIG.relative_to(ROOT)}") if __name__ == "__main__": main() ================================================ FILE: scripts/publish.sh ================================================ #!/usr/bin/env bash set -euo pipefail ROOT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" cd "$ROOT_DIR" GENERATED_FILES=( "agents/AGENTS.md" "README.md" ".cursor-plugin/plugin.json" ".mcp.json" ) file_sig() { local path="$1" if [[ -f "$path" ]]; then sha256sum "$path" | awk '{print $1}' else echo "__MISSING__" fi } run_generate() { uv run scripts/generate_agents.py uv run scripts/generate_cursor_plugin.py } run_check() { declare -A before local changed=() for path in "${GENERATED_FILES[@]}"; do before["$path"]="$(file_sig "$path")" done run_generate for path in "${GENERATED_FILES[@]}"; do if [[ "${before[$path]}" != "$(file_sig "$path")" ]]; then changed+=("$path") fi done if [[ ${#changed[@]} -gt 0 ]]; then echo "Generated artifacts are outdated." echo "Run: ./scripts/publish.sh" echo echo "Changed files:" for path in "${changed[@]}"; do echo "$path" done exit 1 fi # Extra explicit check for cursor-only artifacts uv run scripts/generate_cursor_plugin.py --check echo "All generated artifacts are up to date." } case "${1:-}" in "") run_generate echo "Publish artifacts generated successfully." ;; "--check") run_check ;; "-h"|"--help") cat <<'EOF' Usage: ./scripts/publish.sh Generate all publish artifacts ./scripts/publish.sh --check Verify generated artifacts are up to date This script regenerates: - agents/AGENTS.md - README.md (skills table section) - .cursor-plugin/plugin.json - .mcp.json EOF ;; *) echo "Unknown option: $1" >&2 echo "Use --help for usage." >&2 exit 2 ;; esac ================================================ FILE: scripts/run_skills_help.py ================================================ #!/usr/bin/env python3 """ Script to execute all Python programs under "skills" directories with `uv run` and --help flag. """ import subprocess from pathlib import Path def find_python_files(): """Find all Python files under skills directories.""" python_files = [] # Search for skills directories and find Python files for skills_dir in Path('.').rglob('../skills'): if skills_dir.is_dir(): python_files.extend(skills_dir.rglob('*.py')) return sorted(set(python_files)) def run_with_help(python_file): """Run a Python file with uv run --help.""" try: print(f"\n{'='*60}") print(f"Running: {python_file}") print(f"{'='*60}") result = subprocess.run( ['uv', 'run', str(python_file), '--help'], capture_output=True, text=True, timeout=30 # 30 second timeout ) if result.returncode == 0: print("SUCCESS - Output:") print(result.stdout) else: print(f"FAILED - Return code: {result.returncode}") if result.stderr: print("Stderr:") print(result.stderr) if result.stdout: print("Stdout:") print(result.stdout) except subprocess.TimeoutExpired: print("TIMEOUT - Command took too long") except FileNotFoundError: print("ERROR - uv command not found. Please install uv.") return False except Exception as e: print(f"ERROR - {str(e)}") return True def main(): """Main function to find and run all Python files.""" print("Finding Python files under skills directories...") python_files = find_python_files() if not python_files: print("No Python files found under skills directories.") return print(f"Found {len(python_files)} Python files") success_count = 0 failed_count = 0 for python_file in python_files: if run_with_help(python_file): success_count += 1 else: failed_count += 1 print(f"\n{'='*60}") print(f"SUMMARY:") print(f"Total files: {len(python_files)}") print(f"Successful: {success_count}") print(f"Failed: {failed_count}") print(f"{'='*60}") if __name__ == "__main__": main() ================================================ FILE: skills/hf-cli/SKILL.md ================================================ --- name: hf-cli description: "Hugging Face Hub CLI (`hf`) for downloading, uploading, and managing repositories, models, datasets, and Spaces on the Hugging Face Hub. Replaces now deprecated `huggingface-cli` command." --- Install: `curl -LsSf https://hf.co/cli/install.sh | bash -s`. The Hugging Face Hub CLI tool `hf` is available. IMPORTANT: The `hf` command replaces the deprecated `huggingface-cli` command. Use `hf --help` to view available functions. Note that auth commands are now all under `hf auth` e.g. `hf auth whoami`. Generated with `huggingface_hub v1.7.2`. Run `hf skills add --force` to regenerate. ## Commands - `hf download REPO_ID` — Download files from the Hub. `[--type CHOICE --revision TEXT --include TEXT --exclude TEXT --cache-dir TEXT --local-dir TEXT --force-download --dry-run --quiet --max-workers INTEGER]` - `hf env` — Print information about the environment. - `hf sync` — Sync files between local directory and a bucket. `[--delete --ignore-times --ignore-sizes --plan TEXT --apply TEXT --dry-run --include TEXT --exclude TEXT --filter-from TEXT --existing --ignore-existing --verbose --quiet]` - `hf upload REPO_ID` — Upload a file or a folder to the Hub. Recommended for single-commit uploads. `[--type CHOICE --revision TEXT --private --include TEXT --exclude TEXT --delete TEXT --commit-message TEXT --commit-description TEXT --create-pr --every FLOAT --quiet]` - `hf upload-large-folder REPO_ID LOCAL_PATH` — Upload a large folder to the Hub. Recommended for resumable uploads. `[--type CHOICE --revision TEXT --private --include TEXT --exclude TEXT --num-workers INTEGER --no-report --no-bars]` - `hf version` — Print information about the hf version. ### `hf auth` — Manage authentication (login, logout, etc.). - `hf auth list` — List all stored access tokens. - `hf auth login` — Login using a token from huggingface.co/settings/tokens. `[--add-to-git-credential --force]` - `hf auth logout` — Logout from a specific token. `[--token-name TEXT]` - `hf auth switch` — Switch between access tokens. `[--token-name TEXT --add-to-git-credential]` - `hf auth whoami` — Find out which huggingface.co account you are logged in as. `[--format CHOICE]` ### `hf buckets` — Commands to interact with buckets. - `hf buckets cp SRC` — Copy a single file to or from a bucket. `[--quiet]` - `hf buckets create BUCKET_ID` — Create a new bucket. `[--private --exist-ok --quiet]` - `hf buckets delete BUCKET_ID` — Delete a bucket. `[--yes --missing-ok --quiet]` - `hf buckets info BUCKET_ID` — Get info about a bucket. `[--quiet]` - `hf buckets list` — List buckets or files in a bucket. `[--human-readable --tree --recursive --format CHOICE --quiet]` - `hf buckets move FROM_ID TO_ID` — Move (rename) a bucket to a new name or namespace. - `hf buckets remove ARGUMENT` — Remove files from a bucket. `[--recursive --yes --dry-run --include TEXT --exclude TEXT --quiet]` - `hf buckets sync` — Sync files between local directory and a bucket. `[--delete --ignore-times --ignore-sizes --plan TEXT --apply TEXT --dry-run --include TEXT --exclude TEXT --filter-from TEXT --existing --ignore-existing --verbose --quiet]` ### `hf cache` — Manage local cache directory. - `hf cache list` — List cached repositories or revisions. `[--cache-dir TEXT --revisions --filter TEXT --format CHOICE --quiet --sort CHOICE --limit INTEGER]` - `hf cache prune` — Remove detached revisions from the cache. `[--cache-dir TEXT --yes --dry-run]` - `hf cache rm TARGETS` — Remove cached repositories or revisions. `[--cache-dir TEXT --yes --dry-run]` - `hf cache verify REPO_ID` — Verify checksums for a single repo revision from cache or a local directory. `[--type CHOICE --revision TEXT --cache-dir TEXT --local-dir TEXT --fail-on-missing-files --fail-on-extra-files]` ### `hf collections` — Interact with collections on the Hub. - `hf collections add-item COLLECTION_SLUG ITEM_ID ITEM_TYPE` — Add an item to a collection. `[--note TEXT --exists-ok]` - `hf collections create TITLE` — Create a new collection on the Hub. `[--namespace TEXT --description TEXT --private --exists-ok]` - `hf collections delete COLLECTION_SLUG` — Delete a collection from the Hub. `[--missing-ok]` - `hf collections delete-item COLLECTION_SLUG ITEM_OBJECT_ID` — Delete an item from a collection. `[--missing-ok]` - `hf collections info COLLECTION_SLUG` — Get info about a collection on the Hub. Output is in JSON format. - `hf collections list` — List collections on the Hub. `[--owner TEXT --item TEXT --sort CHOICE --limit INTEGER --format CHOICE --quiet]` - `hf collections update COLLECTION_SLUG` — Update a collection's metadata on the Hub. `[--title TEXT --description TEXT --position INTEGER --private --theme TEXT]` - `hf collections update-item COLLECTION_SLUG ITEM_OBJECT_ID` — Update an item in a collection. `[--note TEXT --position INTEGER]` ### `hf datasets` — Interact with datasets on the Hub. - `hf datasets info DATASET_ID` — Get info about a dataset on the Hub. Output is in JSON format. `[--revision TEXT --expand TEXT]` - `hf datasets list` — List datasets on the Hub. `[--search TEXT --author TEXT --filter TEXT --sort CHOICE --limit INTEGER --expand TEXT --format CHOICE --quiet]` - `hf datasets parquet DATASET_ID` — List parquet file URLs available for a dataset. `[--subset TEXT --split TEXT --format CHOICE --quiet]` - `hf datasets sql SQL` — Execute a raw SQL query with DuckDB against dataset parquet URLs. `[--format CHOICE]` ### `hf discussions` — Manage discussions and pull requests on the Hub. - `hf discussions close REPO_ID NUM` — Close a discussion or pull request. `[--comment TEXT --yes --type CHOICE]` - `hf discussions comment REPO_ID NUM` — Comment on a discussion or pull request. `[--body TEXT --body-file PATH --type CHOICE]` - `hf discussions create REPO_ID --title TEXT` — Create a new discussion or pull request on a repo. `[--body TEXT --body-file PATH --pull-request --type CHOICE]` - `hf discussions diff REPO_ID NUM` — Show the diff of a pull request. `[--type CHOICE]` - `hf discussions info REPO_ID NUM` — Get info about a discussion or pull request. `[--comments --diff --no-color --type CHOICE --format CHOICE]` - `hf discussions list REPO_ID` — List discussions and pull requests on a repo. `[--status CHOICE --kind CHOICE --author TEXT --limit INTEGER --type CHOICE --format CHOICE --quiet]` - `hf discussions merge REPO_ID NUM` — Merge a pull request. `[--comment TEXT --yes --type CHOICE]` - `hf discussions rename REPO_ID NUM NEW_TITLE` — Rename a discussion or pull request. `[--type CHOICE]` - `hf discussions reopen REPO_ID NUM` — Reopen a closed discussion or pull request. `[--comment TEXT --yes --type CHOICE]` ### `hf endpoints` — Manage Hugging Face Inference Endpoints. - `hf endpoints catalog deploy --repo TEXT` — Deploy an Inference Endpoint from the Model Catalog. `[--name TEXT --accelerator TEXT --namespace TEXT]` - `hf endpoints catalog list` — List available Catalog models. - `hf endpoints delete NAME` — Delete an Inference Endpoint permanently. `[--namespace TEXT --yes]` - `hf endpoints deploy NAME --repo TEXT --framework TEXT --accelerator TEXT --instance-size TEXT --instance-type TEXT --region TEXT --vendor TEXT` — Deploy an Inference Endpoint from a Hub repository. `[--namespace TEXT --task TEXT --min-replica INTEGER --max-replica INTEGER --scale-to-zero-timeout INTEGER --scaling-metric CHOICE --scaling-threshold FLOAT]` - `hf endpoints describe NAME` — Get information about an existing endpoint. `[--namespace TEXT]` - `hf endpoints list` — Lists all Inference Endpoints for the given namespace. `[--namespace TEXT --format CHOICE --quiet]` - `hf endpoints pause NAME` — Pause an Inference Endpoint. `[--namespace TEXT]` - `hf endpoints resume NAME` — Resume an Inference Endpoint. `[--namespace TEXT --fail-if-already-running]` - `hf endpoints scale-to-zero NAME` — Scale an Inference Endpoint to zero. `[--namespace TEXT]` - `hf endpoints update NAME` — Update an existing endpoint. `[--namespace TEXT --repo TEXT --accelerator TEXT --instance-size TEXT --instance-type TEXT --framework TEXT --revision TEXT --task TEXT --min-replica INTEGER --max-replica INTEGER --scale-to-zero-timeout INTEGER --scaling-metric CHOICE --scaling-threshold FLOAT]` ### `hf extensions` — Manage hf CLI extensions. - `hf extensions exec NAME` — Execute an installed extension. - `hf extensions install REPO_ID` — Install an extension from a public GitHub repository. `[--force]` - `hf extensions list` — List installed extension commands. `[--format CHOICE --quiet]` - `hf extensions remove NAME` — Remove an installed extension. - `hf extensions search` — Search extensions available on GitHub (tagged with 'hf-extension' topic). `[--format CHOICE --quiet]` ### `hf jobs` — Run and manage Jobs on the Hub. - `hf jobs cancel JOB_ID` — Cancel a Job `[--namespace TEXT]` - `hf jobs hardware` — List available hardware options for Jobs - `hf jobs inspect JOB_IDS` — Display detailed information on one or more Jobs `[--namespace TEXT]` - `hf jobs logs JOB_ID` — Fetch the logs of a Job. `[--follow --tail INTEGER --namespace TEXT]` - `hf jobs ps` — List Jobs. `[--all --namespace TEXT --filter TEXT --format TEXT --quiet]` - `hf jobs run IMAGE COMMAND` — Run a Job. `[--env TEXT --secrets TEXT --label TEXT --env-file TEXT --secrets-file TEXT --flavor CHOICE --timeout TEXT --detach --namespace TEXT]` - `hf jobs scheduled delete SCHEDULED_JOB_ID` — Delete a scheduled Job. `[--namespace TEXT]` - `hf jobs scheduled inspect SCHEDULED_JOB_IDS` — Display detailed information on one or more scheduled Jobs `[--namespace TEXT]` - `hf jobs scheduled ps` — List scheduled Jobs `[--all --namespace TEXT --filter TEXT --format TEXT --quiet]` - `hf jobs scheduled resume SCHEDULED_JOB_ID` — Resume (unpause) a scheduled Job. `[--namespace TEXT]` - `hf jobs scheduled run SCHEDULE IMAGE COMMAND` — Schedule a Job. `[--suspend --concurrency --env TEXT --secrets TEXT --label TEXT --env-file TEXT --secrets-file TEXT --flavor CHOICE --timeout TEXT --namespace TEXT]` - `hf jobs scheduled suspend SCHEDULED_JOB_ID` — Suspend (pause) a scheduled Job. `[--namespace TEXT]` - `hf jobs scheduled uv run SCHEDULE SCRIPT` — Run a UV script (local file or URL) on HF infrastructure `[--suspend --concurrency --image TEXT --flavor CHOICE --env TEXT --secrets TEXT --label TEXT --env-file TEXT --secrets-file TEXT --timeout TEXT --namespace TEXT --with TEXT --python TEXT]` - `hf jobs stats` — Fetch the resource usage statistics and metrics of Jobs `[--namespace TEXT]` - `hf jobs uv run SCRIPT` — Run a UV script (local file or URL) on HF infrastructure `[--image TEXT --flavor CHOICE --env TEXT --secrets TEXT --label TEXT --env-file TEXT --secrets-file TEXT --timeout TEXT --detach --namespace TEXT --with TEXT --python TEXT]` ### `hf models` — Interact with models on the Hub. - `hf models info MODEL_ID` — Get info about a model on the Hub. Output is in JSON format. `[--revision TEXT --expand TEXT]` - `hf models list` — List models on the Hub. `[--search TEXT --author TEXT --filter TEXT --num-parameters TEXT --sort CHOICE --limit INTEGER --expand TEXT --format CHOICE --quiet]` ### `hf papers` — Interact with papers on the Hub. - `hf papers list` — List daily papers on the Hub. `[--date TEXT --sort CHOICE --limit INTEGER --format CHOICE --quiet]` ### `hf repos` — Manage repos on the Hub. - `hf repos branch create REPO_ID BRANCH` — Create a new branch for a repo on the Hub. `[--revision TEXT --type CHOICE --exist-ok]` - `hf repos branch delete REPO_ID BRANCH` — Delete a branch from a repo on the Hub. `[--type CHOICE]` - `hf repos create REPO_ID` — Create a new repo on the Hub. `[--type CHOICE --space-sdk TEXT --private --exist-ok --resource-group-id TEXT]` - `hf repos delete REPO_ID` — Delete a repo from the Hub. This is an irreversible operation. `[--type CHOICE --missing-ok]` - `hf repos delete-files REPO_ID PATTERNS` — Delete files from a repo on the Hub. `[--type CHOICE --revision TEXT --commit-message TEXT --commit-description TEXT --create-pr]` - `hf repos duplicate FROM_ID` — Duplicate a repo on the Hub (model, dataset, or Space). `[--type CHOICE --private --exist-ok]` - `hf repos move FROM_ID TO_ID` — Move a repository from a namespace to another namespace. `[--type CHOICE]` - `hf repos settings REPO_ID` — Update the settings of a repository. `[--gated CHOICE --private --type CHOICE]` - `hf repos tag create REPO_ID TAG` — Create a tag for a repo. `[--message TEXT --revision TEXT --type CHOICE]` - `hf repos tag delete REPO_ID TAG` — Delete a tag for a repo. `[--yes --type CHOICE]` - `hf repos tag list REPO_ID` — List tags for a repo. `[--type CHOICE]` ### `hf skills` — Manage skills for AI assistants. - `hf skills add` — Download a skill and install it for an AI assistant. `[--claude --codex --cursor --opencode --global --dest PATH --force]` - `hf skills preview` — Print the generated SKILL.md to stdout. ### `hf spaces` — Interact with spaces on the Hub. - `hf spaces dev-mode SPACE_ID` — Enable or disable dev mode on a Space. `[--stop]` - `hf spaces hot-reload SPACE_ID` — Hot-reload any Python file of a Space without a full rebuild + restart. `[--local-file TEXT --skip-checks --skip-summary]` - `hf spaces info SPACE_ID` — Get info about a space on the Hub. Output is in JSON format. `[--revision TEXT --expand TEXT]` - `hf spaces list` — List spaces on the Hub. `[--search TEXT --author TEXT --filter TEXT --sort CHOICE --limit INTEGER --expand TEXT --format CHOICE --quiet]` ### `hf webhooks` — Manage webhooks on the Hub. - `hf webhooks create --watch TEXT` — Create a new webhook. `[--url TEXT --job-id TEXT --domain CHOICE --secret TEXT]` - `hf webhooks delete WEBHOOK_ID` — Delete a webhook permanently. `[--yes]` - `hf webhooks disable WEBHOOK_ID` — Disable an active webhook. - `hf webhooks enable WEBHOOK_ID` — Enable a disabled webhook. - `hf webhooks info WEBHOOK_ID` — Show full details for a single webhook as JSON. - `hf webhooks list` — List all webhooks for the current user. `[--format CHOICE --quiet]` - `hf webhooks update WEBHOOK_ID` — Update an existing webhook. Only provided options are changed. `[--url TEXT --watch TEXT --domain CHOICE --secret TEXT]` ## Common options - `--format` — Output format: `--format json` (or `--json`) or `--format table` (default). - `-q / --quiet` — Minimal output. - `--revision` — Git revision id which can be a branch name, a tag, or a commit hash. - `--token` — Use a User Access Token. Prefer setting `HF_TOKEN` env var instead of passing `--token`. - `--type` — The type of repository (model, dataset, or space). ## Tips - Use `hf --help` for full options, descriptions, usage, and real-world examples - Authenticate with `HF_TOKEN` env var (recommended) or with `--token` ================================================ FILE: skills/hugging-face-dataset-viewer/SKILL.md ================================================ --- name: hugging-face-dataset-viewer description: Use this skill for Hugging Face Dataset Viewer API workflows that fetch subset/split metadata, paginate rows, search text, apply filters, download parquet URLs, and read size or statistics. --- # Hugging Face Dataset Viewer Use this skill to execute read-only Dataset Viewer API calls for dataset exploration and extraction. ## Core workflow 1. Optionally validate dataset availability with `/is-valid`. 2. Resolve `config` + `split` with `/splits`. 3. Preview with `/first-rows`. 4. Paginate content with `/rows` using `offset` and `length` (max 100). 5. Use `/search` for text matching and `/filter` for row predicates. 6. Retrieve parquet links via `/parquet` and totals/metadata via `/size` and `/statistics`. ## Defaults - Base URL: `https://datasets-server.huggingface.co` - Default API method: `GET` - Query params should be URL-encoded. - `offset` is 0-based. - `length` max is usually `100` for row-like endpoints. - Gated/private datasets require `Authorization: Bearer `. ## Dataset Viewer - `Validate dataset`: `/is-valid?dataset=` - `List subsets and splits`: `/splits?dataset=` - `Preview first rows`: `/first-rows?dataset=&config=&split=` - `Paginate rows`: `/rows?dataset=&config=&split=&offset=&length=` - `Search text`: `/search?dataset=&config=&split=&query=&offset=&length=` - `Filter with predicates`: `/filter?dataset=&config=&split=&where=&orderby=&offset=&length=` - `List parquet shards`: `/parquet?dataset=` - `Get size totals`: `/size?dataset=` - `Get column statistics`: `/statistics?dataset=&config=&split=` - `Get Croissant metadata (if available)`: `/croissant?dataset=` Pagination pattern: ```bash curl "https://datasets-server.huggingface.co/rows?dataset=stanfordnlp/imdb&config=plain_text&split=train&offset=0&length=100" curl "https://datasets-server.huggingface.co/rows?dataset=stanfordnlp/imdb&config=plain_text&split=train&offset=100&length=100" ``` When pagination is partial, use response fields such as `num_rows_total`, `num_rows_per_page`, and `partial` to drive continuation logic. Search/filter notes: - `/search` matches string columns (full-text style behavior is internal to the API). - `/filter` requires predicate syntax in `where` and optional sort in `orderby`. - Keep filtering and searches read-only and side-effect free. ## Querying Datasets Use `npx parquetlens` with Hub parquet alias paths for SQL querying. Parquet alias shape: ```text hf://datasets//@~parquet///.parquet ``` Derive ``, ``, and `` from Dataset Viewer `/parquet`: ```bash curl -s "https://datasets-server.huggingface.co/parquet?dataset=cfahlgren1/hub-stats" \ | jq -r '.parquet_files[] | "hf://datasets/\(.dataset)@~parquet/\(.config)/\(.split)/\(.filename)"' ``` Run SQL query: ```bash npx -y -p parquetlens -p @parquetlens/sql parquetlens \ "hf://datasets//@~parquet///.parquet" \ --sql "SELECT * FROM data LIMIT 20" ``` ### SQL export - CSV: `--sql "COPY (SELECT * FROM data LIMIT 1000) TO 'export.csv' (FORMAT CSV, HEADER, DELIMITER ',')"` - JSON: `--sql "COPY (SELECT * FROM data LIMIT 1000) TO 'export.json' (FORMAT JSON)"` - Parquet: `--sql "COPY (SELECT * FROM data LIMIT 1000) TO 'export.parquet' (FORMAT PARQUET)"` ## Creating and Uploading Datasets Use one of these flows depending on dependency constraints. Zero local dependencies (Hub UI): - Create dataset repo in browser: `https://huggingface.co/new-dataset` - Upload parquet files in the repo "Files and versions" page. - Verify shards appear in Dataset Viewer: ```bash curl -s "https://datasets-server.huggingface.co/parquet?dataset=/" ``` Low dependency CLI flow (`npx @huggingface/hub` / `hfjs`): - Set auth token: ```bash export HF_TOKEN= ``` - Upload parquet folder to a dataset repo (auto-creates repo if missing): ```bash npx -y @huggingface/hub upload datasets// ./local/parquet-folder data ``` - Upload as private repo on creation: ```bash npx -y @huggingface/hub upload datasets// ./local/parquet-folder data --private ``` After upload, call `/parquet` to discover `//` values for querying with `@~parquet`. ================================================ FILE: skills/hugging-face-datasets/SKILL.md ================================================ --- name: hugging-face-datasets description: Create and manage datasets on Hugging Face Hub. Supports initializing repos, defining configs/system prompts, streaming row updates, and SQL-based dataset querying/transformation. Designed to work alongside HF MCP server for comprehensive dataset workflows. --- # Overview This skill provides tools to manage datasets on the Hugging Face Hub with a focus on creation, configuration, content management, and SQL-based data manipulation. It is designed to complement the existing Hugging Face MCP server by providing dataset editing and querying capabilities. ## Integration with HF MCP Server - **Use HF MCP Server for**: Dataset discovery, search, and metadata retrieval - **Use This Skill for**: Dataset creation, content editing, SQL queries, data transformation, and structured data formatting # Version 2.1.0 # Dependencies # This skill uses PEP 723 scripts with inline dependency management # Scripts auto-install requirements when run with: uv run scripts/script_name.py - uv (Python package manager) - Getting Started: See "Usage Instructions" below for PEP 723 usage # Core Capabilities ## 1. Dataset Lifecycle Management - **Initialize**: Create new dataset repositories with proper structure - **Configure**: Store detailed configuration including system prompts and metadata - **Stream Updates**: Add rows efficiently without downloading entire datasets ## 2. SQL-Based Dataset Querying (NEW) Query any Hugging Face dataset using DuckDB SQL via `scripts/sql_manager.py`: - **Direct Queries**: Run SQL on datasets using the `hf://` protocol - **Schema Discovery**: Describe dataset structure and column types - **Data Sampling**: Get random samples for exploration - **Aggregations**: Count, histogram, unique values analysis - **Transformations**: Filter, join, reshape data with SQL - **Export & Push**: Save results locally or push to new Hub repos ## 3. Multi-Format Dataset Support Supports diverse dataset types through template system: - **Chat/Conversational**: Chat templating, multi-turn dialogues, tool usage examples - **Text Classification**: Sentiment analysis, intent detection, topic classification - **Question-Answering**: Reading comprehension, factual QA, knowledge bases - **Text Completion**: Language modeling, code completion, creative writing - **Tabular Data**: Structured data for regression/classification tasks - **Custom Formats**: Flexible schema definition for specialized needs ## 4. Quality Assurance Features - **JSON Validation**: Ensures data integrity during uploads - **Batch Processing**: Efficient handling of large datasets - **Error Recovery**: Graceful handling of upload failures and conflicts # Usage Instructions The skill includes two Python scripts that use PEP 723 inline dependency management: > **All paths are relative to the directory containing this SKILL.md file.** > Scripts are run with: `uv run scripts/script_name.py [arguments]` - `scripts/dataset_manager.py` - Dataset creation and management - `scripts/sql_manager.py` - SQL-based dataset querying and transformation ### Prerequisites - `uv` package manager installed - `HF_TOKEN` environment variable must be set with a Write-access token --- # SQL Dataset Querying (sql_manager.py) Query, transform, and push Hugging Face datasets using DuckDB SQL. The `hf://` protocol provides direct access to any public dataset (or private with token). ## Quick Start ```bash # Query a dataset uv run scripts/sql_manager.py query \ --dataset "cais/mmlu" \ --sql "SELECT * FROM data WHERE subject='nutrition' LIMIT 10" # Get dataset schema uv run scripts/sql_manager.py describe --dataset "cais/mmlu" # Sample random rows uv run scripts/sql_manager.py sample --dataset "cais/mmlu" --n 5 # Count rows with filter uv run scripts/sql_manager.py count --dataset "cais/mmlu" --where "subject='nutrition'" ``` ## SQL Query Syntax Use `data` as the table name in your SQL - it gets replaced with the actual `hf://` path: ```sql -- Basic select SELECT * FROM data LIMIT 10 -- Filtering SELECT * FROM data WHERE subject='nutrition' -- Aggregations SELECT subject, COUNT(*) as cnt FROM data GROUP BY subject ORDER BY cnt DESC -- Column selection and transformation SELECT question, choices[answer] AS correct_answer FROM data -- Regex matching SELECT * FROM data WHERE regexp_matches(question, 'nutrition|diet') -- String functions SELECT regexp_replace(question, '\n', '') AS cleaned FROM data ``` ## Common Operations ### 1. Explore Dataset Structure ```bash # Get schema uv run scripts/sql_manager.py describe --dataset "cais/mmlu" # Get unique values in column uv run scripts/sql_manager.py unique --dataset "cais/mmlu" --column "subject" # Get value distribution uv run scripts/sql_manager.py histogram --dataset "cais/mmlu" --column "subject" --bins 20 ``` ### 2. Filter and Transform ```bash # Complex filtering with SQL uv run scripts/sql_manager.py query \ --dataset "cais/mmlu" \ --sql "SELECT subject, COUNT(*) as cnt FROM data GROUP BY subject HAVING cnt > 100" # Using transform command uv run scripts/sql_manager.py transform \ --dataset "cais/mmlu" \ --select "subject, COUNT(*) as cnt" \ --group-by "subject" \ --order-by "cnt DESC" \ --limit 10 ``` ### 3. Create Subsets and Push to Hub ```bash # Query and push to new dataset uv run scripts/sql_manager.py query \ --dataset "cais/mmlu" \ --sql "SELECT * FROM data WHERE subject='nutrition'" \ --push-to "username/mmlu-nutrition-subset" \ --private # Transform and push uv run scripts/sql_manager.py transform \ --dataset "ibm/duorc" \ --config "ParaphraseRC" \ --select "question, answers" \ --where "LENGTH(question) > 50" \ --push-to "username/duorc-long-questions" ``` ### 4. Export to Local Files ```bash # Export to Parquet uv run scripts/sql_manager.py export \ --dataset "cais/mmlu" \ --sql "SELECT * FROM data WHERE subject='nutrition'" \ --output "nutrition.parquet" \ --format parquet # Export to JSONL uv run scripts/sql_manager.py export \ --dataset "cais/mmlu" \ --sql "SELECT * FROM data LIMIT 100" \ --output "sample.jsonl" \ --format jsonl ``` ### 5. Working with Dataset Configs/Splits ```bash # Specify config (subset) uv run scripts/sql_manager.py query \ --dataset "ibm/duorc" \ --config "ParaphraseRC" \ --sql "SELECT * FROM data LIMIT 5" # Specify split uv run scripts/sql_manager.py query \ --dataset "cais/mmlu" \ --split "test" \ --sql "SELECT COUNT(*) FROM data" # Query all splits uv run scripts/sql_manager.py query \ --dataset "cais/mmlu" \ --split "*" \ --sql "SELECT * FROM data LIMIT 10" ``` ### 6. Raw SQL with Full Paths For complex queries or joining datasets: ```bash uv run scripts/sql_manager.py raw --sql " SELECT a.*, b.* FROM 'hf://datasets/dataset1@~parquet/default/train/*.parquet' a JOIN 'hf://datasets/dataset2@~parquet/default/train/*.parquet' b ON a.id = b.id LIMIT 100 " ``` ## Python API Usage ```python from sql_manager import HFDatasetSQL sql = HFDatasetSQL() # Query results = sql.query("cais/mmlu", "SELECT * FROM data WHERE subject='nutrition' LIMIT 10") # Get schema schema = sql.describe("cais/mmlu") # Sample samples = sql.sample("cais/mmlu", n=5, seed=42) # Count count = sql.count("cais/mmlu", where="subject='nutrition'") # Histogram dist = sql.histogram("cais/mmlu", "subject") # Filter and transform results = sql.filter_and_transform( "cais/mmlu", select="subject, COUNT(*) as cnt", group_by="subject", order_by="cnt DESC", limit=10 ) # Push to Hub url = sql.push_to_hub( "cais/mmlu", "username/nutrition-subset", sql="SELECT * FROM data WHERE subject='nutrition'", private=True ) # Export locally sql.export_to_parquet("cais/mmlu", "output.parquet", sql="SELECT * FROM data LIMIT 100") sql.close() ``` ## HF Path Format DuckDB uses the `hf://` protocol to access datasets: ``` hf://datasets/{dataset_id}@{revision}/{config}/{split}/*.parquet ``` Examples: - `hf://datasets/cais/mmlu@~parquet/default/train/*.parquet` - `hf://datasets/ibm/duorc@~parquet/ParaphraseRC/test/*.parquet` The `@~parquet` revision provides auto-converted Parquet files for any dataset format. ## Useful DuckDB SQL Functions ```sql -- String functions LENGTH(column) -- String length regexp_replace(col, '\n', '') -- Regex replace regexp_matches(col, 'pattern') -- Regex match LOWER(col), UPPER(col) -- Case conversion -- Array functions choices[0] -- Array indexing (0-based) array_length(choices) -- Array length unnest(choices) -- Expand array to rows -- Aggregations COUNT(*), SUM(col), AVG(col) GROUP BY col HAVING condition -- Sampling USING SAMPLE 10 -- Random sample USING SAMPLE 10 (RESERVOIR, 42) -- Reproducible sample -- Window functions ROW_NUMBER() OVER (PARTITION BY col ORDER BY col2) ``` --- # Dataset Creation (dataset_manager.py) ### Recommended Workflow **1. Discovery (Use HF MCP Server):** ```python # Use HF MCP tools to find existing datasets search_datasets("conversational AI training") get_dataset_details("username/dataset-name") ``` **2. Creation (Use This Skill):** ```bash # Initialize new dataset uv run scripts/dataset_manager.py init --repo_id "your-username/dataset-name" [--private] # Configure with detailed system prompt uv run scripts/dataset_manager.py config --repo_id "your-username/dataset-name" --system_prompt "$(cat system_prompt.txt)" ``` **3. Content Management (Use This Skill):** ```bash # Quick setup with any template uv run scripts/dataset_manager.py quick_setup \ --repo_id "your-username/dataset-name" \ --template classification # Add data with template validation uv run scripts/dataset_manager.py add_rows \ --repo_id "your-username/dataset-name" \ --template qa \ --rows_json "$(cat your_qa_data.json)" ``` ### Template-Based Data Structures **1. Chat Template (`--template chat`)** ```json { "messages": [ {"role": "user", "content": "Natural user request"}, {"role": "assistant", "content": "Response with tool usage"}, {"role": "tool", "content": "Tool response", "tool_call_id": "call_123"} ], "scenario": "Description of use case", "complexity": "simple|intermediate|advanced" } ``` **2. Classification Template (`--template classification`)** ```json { "text": "Input text to be classified", "label": "classification_label", "confidence": 0.95, "metadata": {"domain": "technology", "language": "en"} } ``` **3. QA Template (`--template qa`)** ```json { "question": "What is the question being asked?", "answer": "The complete answer", "context": "Additional context if needed", "answer_type": "factual|explanatory|opinion", "difficulty": "easy|medium|hard" } ``` **4. Completion Template (`--template completion`)** ```json { "prompt": "The beginning text or context", "completion": "The expected continuation", "domain": "code|creative|technical|conversational", "style": "description of writing style" } ``` **5. Tabular Template (`--template tabular`)** ```json { "columns": [ {"name": "feature1", "type": "numeric", "description": "First feature"}, {"name": "target", "type": "categorical", "description": "Target variable"} ], "data": [ {"feature1": 123, "target": "class_a"}, {"feature1": 456, "target": "class_b"} ] } ``` ### Advanced System Prompt Template For high-quality training data generation: ```text You are an AI assistant expert at using MCP tools effectively. ## MCP SERVER DEFINITIONS [Define available servers and tools] ## TRAINING EXAMPLE STRUCTURE [Specify exact JSON schema for chat templating] ## QUALITY GUIDELINES [Detail requirements for realistic scenarios, progressive complexity, proper tool usage] ## EXAMPLE CATEGORIES [List development workflows, debugging scenarios, data management tasks] ``` ### Example Categories & Templates The skill includes diverse training examples beyond just MCP usage: **Available Example Sets:** - `training_examples.json` - MCP tool usage examples (debugging, project setup, database analysis) - `diverse_training_examples.json` - Broader scenarios including: - **Educational Chat** - Explaining programming concepts, tutorials - **Git Workflows** - Feature branches, version control guidance - **Code Analysis** - Performance optimization, architecture review - **Content Generation** - Professional writing, creative brainstorming - **Codebase Navigation** - Legacy code exploration, systematic analysis - **Conversational Support** - Problem-solving, technical discussions **Using Different Example Sets:** ```bash # Add MCP-focused examples uv run scripts/dataset_manager.py add_rows --repo_id "your-username/dataset-name" \ --rows_json "$(cat examples/training_examples.json)" # Add diverse conversational examples uv run scripts/dataset_manager.py add_rows --repo_id "your-username/dataset-name" \ --rows_json "$(cat examples/diverse_training_examples.json)" # Mix both for comprehensive training data uv run scripts/dataset_manager.py add_rows --repo_id "your-username/dataset-name" \ --rows_json "$(jq -s '.[0] + .[1]' examples/training_examples.json examples/diverse_training_examples.json)" ``` ### Commands Reference **List Available Templates:** ```bash uv run scripts/dataset_manager.py list_templates ``` **Quick Setup (Recommended):** ```bash uv run scripts/dataset_manager.py quick_setup --repo_id "your-username/dataset-name" --template classification ``` **Manual Setup:** ```bash # Initialize repository uv run scripts/dataset_manager.py init --repo_id "your-username/dataset-name" [--private] # Configure with system prompt uv run scripts/dataset_manager.py config --repo_id "your-username/dataset-name" --system_prompt "Your prompt here" # Add data with validation uv run scripts/dataset_manager.py add_rows \ --repo_id "your-username/dataset-name" \ --template qa \ --rows_json '[{"question": "What is AI?", "answer": "Artificial Intelligence..."}]' ``` **View Dataset Statistics:** ```bash uv run scripts/dataset_manager.py stats --repo_id "your-username/dataset-name" ``` ### Error Handling - **Repository exists**: Script will notify and continue with configuration - **Invalid JSON**: Clear error message with parsing details - **Network issues**: Automatic retry for transient failures - **Token permissions**: Validation before operations begin --- # Combined Workflow Examples ## Example 1: Create Training Subset from Existing Dataset ```bash # 1. Explore the source dataset uv run scripts/sql_manager.py describe --dataset "cais/mmlu" uv run scripts/sql_manager.py histogram --dataset "cais/mmlu" --column "subject" # 2. Query and create subset uv run scripts/sql_manager.py query \ --dataset "cais/mmlu" \ --sql "SELECT * FROM data WHERE subject IN ('nutrition', 'anatomy', 'clinical_knowledge')" \ --push-to "username/mmlu-medical-subset" \ --private ``` ## Example 2: Transform and Reshape Data ```bash # Transform MMLU to QA format with correct answers extracted uv run scripts/sql_manager.py query \ --dataset "cais/mmlu" \ --sql "SELECT question, choices[answer] as correct_answer, subject FROM data" \ --push-to "username/mmlu-qa-format" ``` ## Example 3: Merge Multiple Dataset Splits ```bash # Export multiple splits and combine uv run scripts/sql_manager.py export \ --dataset "cais/mmlu" \ --split "*" \ --output "mmlu_all.parquet" ``` ## Example 4: Quality Filtering ```bash # Filter for high-quality examples uv run scripts/sql_manager.py query \ --dataset "squad" \ --sql "SELECT * FROM data WHERE LENGTH(context) > 500 AND LENGTH(question) > 20" \ --push-to "username/squad-filtered" ``` ## Example 5: Create Custom Training Dataset ```bash # 1. Query source data uv run scripts/sql_manager.py export \ --dataset "cais/mmlu" \ --sql "SELECT question, subject FROM data WHERE subject='nutrition'" \ --output "nutrition_source.jsonl" \ --format jsonl # 2. Process with your pipeline (add answers, format, etc.) # 3. Push processed data uv run scripts/dataset_manager.py init --repo_id "username/nutrition-training" uv run scripts/dataset_manager.py add_rows \ --repo_id "username/nutrition-training" \ --template qa \ --rows_json "$(cat processed_data.json)" ``` ================================================ FILE: skills/hugging-face-datasets/examples/diverse_training_examples.json ================================================ [ { "messages": [ { "role": "user", "content": "Hi there! I'm working on a Python project and getting stuck with some async programming concepts. Can you help explain how to use async/await properly?" }, { "role": "assistant", "content": "I'd be happy to help you understand async/await in Python! These are powerful tools for writing concurrent code. Let me show you some practical examples and explain the key concepts.\n\n**Basic async/await syntax:**\n```python\nimport asyncio\n\nasync def fetch_data():\n # Simulate an async operation\n await asyncio.sleep(1)\n return \"Data fetched!\"\n\nasync def main():\n result = await fetch_data()\n print(result)\n\n# Run the async function\nasyncio.run(main())\n```\n\n**Key concepts:**\n- `async def` defines an asynchronous function (coroutine)\n- `await` pauses execution until the awaited operation completes\n- `asyncio.run()` runs the main async function\n\n**When to use async/await:**\n- I/O operations (file reads, network requests, database queries)\n- Operations that involve waiting\n- When you want to handle multiple tasks concurrently\n\nWould you like me to show you a more complex example with multiple concurrent operations, or do you have a specific use case you're working on?" } ], "scenario": "Educational chat about Python async programming concepts", "complexity": "simple", "tools_used": [], "learning_objectives": ["conversational teaching", "code explanation", "technical mentoring", "educational chat patterns"] }, { "messages": [ { "role": "user", "content": "I need to create a new feature branch for my project, make some changes, and then create a pull request. Can you walk me through the Git workflow?" }, { "role": "assistant", "content": "I'll help you with the complete Git workflow for creating a feature branch and pull request. Let me first check your current Git status and then guide you through each step.", "tool_calls": [ { "type": "function", "function": { "name": "git_status", "arguments": {} } } ] }, { "role": "tool", "content": "On branch main\nYour branch is up to date with 'origin/main'.\n\nnothing to commit, working tree clean", "tool_call_id": "call_301" }, { "role": "assistant", "content": "Perfect! Your working tree is clean and you're on the main branch. Now let's create your feature branch and walk through the workflow.", "tool_calls": [ { "type": "function", "function": { "name": "git_branch", "arguments": { "branch_name": "feature/user-authentication", "create": true } } } ] }, { "role": "tool", "content": "Switched to a new branch 'feature/user-authentication'", "tool_call_id": "call_302" }, { "role": "assistant", "content": "Great! I've created and switched to your new feature branch. Here's the complete workflow:\n\n**✅ Step 1: Created feature branch** (Done)\n```bash\ngit checkout -b feature/user-authentication\n```\n\n**Next steps for you:**\n\n**Step 2: Make your changes**\n- Edit your files as needed\n- Add your new feature code\n\n**Step 3: Stage and commit changes**\n```bash\ngit add .\ngit commit -m \"Add user authentication feature\"\n```\n\n**Step 4: Push branch to remote**\n```bash\ngit push -u origin feature/user-authentication\n```\n\n**Step 5: Create Pull Request**\n- Go to your repository on GitHub/GitLab\n- Click \"Create Pull Request\" or \"Merge Request\"\n- Fill in description and request review\n\nWould you like me to help you with any of these steps once you've made your changes?" } ], "scenario": "Git workflow guidance for feature development", "complexity": "intermediate", "tools_used": ["git_status", "git_branch"], "learning_objectives": ["Git workflow patterns", "branch management", "collaborative development", "step-by-step guidance"] }, { "messages": [ { "role": "user", "content": "I have a Python function that's running really slowly. Can you help me analyze the code and suggest optimizations?" }, { "role": "assistant", "content": "I'd be happy to help optimize your Python code! Let me analyze the function to identify potential performance bottlenecks and suggest improvements.", "tool_calls": [ { "type": "function", "function": { "name": "find_functions", "arguments": { "pattern": "def.*process.*" } } } ] }, { "role": "tool", "content": "Found functions:\n- def process_user_data(users_list): line 45 in utils.py\n- def process_payments(payment_records): line 78 in billing.py", "tool_call_id": "call_401" }, { "role": "assistant", "content": "I found some processing functions. Let me examine the code to identify performance issues.", "tool_calls": [ { "type": "function", "function": { "name": "read_file", "arguments": { "path": "utils.py", "start_line": 45, "end_line": 60 } } } ] }, { "role": "tool", "content": "def process_user_data(users_list):\n results = []\n for user in users_list:\n # Multiple database calls in loop - PERFORMANCE ISSUE!\n profile = db.get_user_profile(user.id)\n settings = db.get_user_settings(user.id)\n preferences = db.get_user_preferences(user.id)\n \n # String concatenation in loop - PERFORMANCE ISSUE!\n summary = \"\"\n for field in user.fields:\n summary += f\"{field.name}: {field.value}\\n\"\n \n results.append({\n 'user': user,\n 'profile': profile,\n 'settings': settings,\n 'preferences': preferences,\n 'summary': summary\n })\n return results", "tool_call_id": "call_402" }, { "role": "assistant", "content": "I've identified several performance issues in your `process_user_data` function. Here's my analysis and optimization suggestions:\n\n**🔍 Performance Issues Found:**\n\n1. **N+1 Database Query Problem** (Lines 48-50)\n - Making 3 separate DB calls per user in a loop\n - This scales linearly with user count (very slow!)\n\n2. **String Concatenation in Loop** (Lines 52-55)\n - Using `+=` for string building is inefficient\n - Creates new string objects each iteration\n\n**⚡ Optimization Recommendations:**\n\n```python\ndef process_user_data_optimized(users_list):\n # Fix 1: Batch database queries\n user_ids = [user.id for user in users_list]\n profiles = db.get_user_profiles_batch(user_ids) # Single query\n settings = db.get_user_settings_batch(user_ids) # Single query \n preferences = db.get_user_preferences_batch(user_ids) # Single query\n \n results = []\n for user in users_list:\n # Fix 2: Use join() for string concatenation\n summary = \"\\n\".join(f\"{field.name}: {field.value}\" \n for field in user.fields)\n \n results.append({\n 'user': user,\n 'profile': profiles.get(user.id),\n 'settings': settings.get(user.id), \n 'preferences': preferences.get(user.id),\n 'summary': summary\n })\n return results\n```\n\n**Expected Performance Gain:** 10-100x faster depending on user count!\n\nWould you like me to help implement the batch database queries or analyze any other functions?" } ], "scenario": "Code performance analysis and optimization guidance", "complexity": "advanced", "tools_used": ["find_functions", "read_file"], "learning_objectives": ["performance optimization", "code analysis", "database optimization patterns", "Python best practices"] }, { "messages": [ { "role": "user", "content": "I need to write a professional email to a client explaining a project delay. Can you help me draft something that's apologetic but confident?" }, { "role": "assistant", "content": "I'll help you draft a professional email that strikes the right balance between taking responsibility and maintaining client confidence. Here's a template that addresses the delay while focusing on solutions:\n\n**Subject:** Project Update - Revised Timeline for [Project Name]\n\n**Email Draft:**\n\nDear [Client Name],\n\nI hope this email finds you well. I'm writing to provide you with an important update regarding the [Project Name] timeline.\n\nAfter a thorough review of our current progress, I need to inform you that we will require an additional [X weeks/days] to complete the project to our mutual quality standards. This adjustment is primarily due to [brief, honest reason - e.g., \"unexpected technical complexities in the integration phase\" or \"additional testing requirements to ensure optimal performance\"].\n\nI want to be completely transparent about this delay and take full responsibility. However, I'm confident that this additional time will allow us to deliver a solution that exceeds your expectations and provides long-term value.\n\n**Here's our revised timeline:**\n- [Key milestone 1]: [New date]\n- [Key milestone 2]: [New date] \n- Final delivery: [New date]\n\nTo ensure we stay on track, I've implemented [specific action - e.g., \"daily progress reviews\" or \"additional resources to the development team\"].\n\nI understand this may impact your planning, and I'm available to discuss how we can minimize any inconvenience. Would you be available for a brief call this week to address any concerns?\n\nThank you for your patience and continued trust in our partnership.\n\nBest regards,\n[Your name]\n\n**Key elements this email includes:**\n- Direct, honest communication\n- Takes responsibility without excessive apologizing\n- Provides specific reasons and solutions\n- Shows proactive management\n- Offers further discussion\n\nWould you like me to adjust the tone or focus on any particular aspects?" } ], "scenario": "Professional communication assistance and content generation", "complexity": "simple", "tools_used": [], "learning_objectives": ["professional writing", "client communication", "content generation", "business correspondence patterns"] }, { "messages": [ { "role": "user", "content": "I'm trying to understand this legacy codebase I inherited. There are so many files and I don't know where to start. Can you help me map out the structure?" }, { "role": "assistant", "content": "I completely understand - navigating a new codebase can be overwhelming! Let me help you systematically explore and understand the structure. I'll start by getting an overview and then drill down into the important parts.", "tool_calls": [ { "type": "function", "function": { "name": "list_directory", "arguments": { "path": "." } } } ] }, { "role": "tool", "content": "README.md\npackage.json\nsrc/\ntest/\nconfig/\ndocs/\n.env.example\n.gitignore\nDockerfile\nnode_modules/", "tool_call_id": "call_501" }, { "role": "assistant", "content": "Great! This looks like a Node.js project with a well-organized structure. Let me examine the key areas to give you a roadmap.", "tool_calls": [ { "type": "function", "function": { "name": "read_file", "arguments": { "path": "package.json" } } } ] }, { "role": "tool", "content": "{\n \"name\": \"legacy-api-service\",\n \"version\": \"2.3.1\",\n \"description\": \"Customer management API with billing integration\",\n \"main\": \"src/app.js\",\n \"scripts\": {\n \"start\": \"node src/app.js\",\n \"dev\": \"nodemon src/app.js\",\n \"test\": \"jest test/\"\n },\n \"dependencies\": {\n \"express\": \"^4.18.0\",\n \"mongoose\": \"^6.2.0\",\n \"stripe\": \"^8.0.0\"\n }\n}", "tool_call_id": "call_502" }, { "role": "assistant", "content": "Now let me explore the source directory to understand the application structure.", "tool_calls": [ { "type": "function", "function": { "name": "list_directory", "arguments": { "path": "src" } } } ] }, { "role": "tool", "content": "app.js\nroutes/\nmodels/\nservices/\nmiddleware/\nutils/\ncontrollers/", "tool_call_id": "call_503" }, { "role": "assistant", "content": "Perfect! Now I have a clear picture. Here's your **Codebase Navigation Guide**:\n\n## 📋 Project Overview\n**Type:** Customer Management API with billing (Node.js/Express)\n**Main Entry:** `src/app.js`\n**Database:** MongoDB (Mongoose)\n**Payment:** Stripe integration\n\n## 🗺️ Architecture Map\n\n**Start Here:**\n1. 📖 `README.md` - Project documentation\n2. ⚙️ `src/app.js` - Application entry point\n3. 🛣️ `src/routes/` - API endpoints definition\n\n**Core Application Logic:**\n- `src/controllers/` - Request handling logic\n- `src/models/` - Database schemas (MongoDB)\n- `src/services/` - Business logic layer\n- `src/middleware/` - Authentication, validation, etc.\n\n**Supporting Code:**\n- `src/utils/` - Helper functions\n- `config/` - Configuration files\n- `test/` - Unit/integration tests\n\n## 🎯 Recommended Exploration Order:\n\n**Phase 1 - Foundation (Start here!):**\n1. Read `README.md` for setup instructions\n2. Check `src/app.js` to see how everything connects\n3. Browse `src/routes/` to understand available APIs\n\n**Phase 2 - Business Logic:**\n4. Explore `src/controllers/` for main functionality\n5. Review `src/models/` for data structure\n6. Study `src/services/` for business rules\n\n**Phase 3 - Details:**\n7. Check `src/middleware/` for security/validation\n8. Look at `config/` for environment setup\n9. Review `test/` to understand expected behavior\n\nWould you like me to dive deeper into any specific area, like showing you the main routes or examining the data models?" } ], "scenario": "Codebase exploration and architecture understanding", "complexity": "intermediate", "tools_used": ["list_directory", "read_file"], "learning_objectives": ["codebase navigation", "architecture analysis", "systematic exploration", "legacy code understanding"] }, { "messages": [ { "role": "user", "content": "Can you help me brainstorm some creative names for my new productivity app? It's focused on time blocking and helps people focus on deep work." }, { "role": "assistant", "content": "I'd love to help you brainstorm creative names for your time blocking and deep work app! Let me generate some ideas across different themes and styles.\n\n## 🎯 Focus & Deep Work Theme\n- **FlowBlocks** - Combines time blocking with flow state\n- **DeepDive** - Simple, suggests deep work immersion\n- **FocusForge** - Creating focused time periods\n- **ZenBlocks** - Peaceful, mindful time management\n- **MindVault** - Protecting mental space for deep work\n\n## ⏰ Time & Productivity Theme\n- **ChronoFlow** - Time management with smooth workflow\n- **TimeWeaver** - Weaving together productive time blocks\n- **ClockCraft** - Skillfully crafting your schedule\n- **TempoLock** - Locking in productive tempos\n- **PulseBlocks** - Rhythmic time blocking\n\n## 🏗️ Building & Structure Theme\n- **TaskArchitect** - Building your productive day\n- **FocusFoundry** - Where focused time is created\n- **ProductivityBricks** - Building blocks of productivity\n- **WorkStack** - Stacking focused work sessions\n- **MindGrid** - Organized mental workspace\n\n## 🎨 Creative & Unique\n- **Concentrix** - Focus + matrix of time\n- **Quietude** - Emphasizes calm, focused environment\n- **DeepSpace** - Space for deep work (tech feel)\n- **FlowChamber** - Protected space for flow state\n- **ThinkTank** - Tank of focused thinking time\n\n## 💡 Short & Catchy\n- **Blokr** - Modern, minimal spelling\n- **Focux** - Focus + flux\n- **Zenith** - Peak productivity\n- **Prism** - Breaking time into focused segments\n- **Vault** - Simple, suggests protection/security\n\n**My top recommendations:**\n1. **FlowBlocks** - Perfect blend of your core features\n2. **DeepVault** - Strong, memorable, suggests protection of deep work\n3. **FocusForge** - Active, suggests creating something valuable\n\nDo any of these resonate with your vision? Would you like me to explore variations of your favorites or brainstorm in a different direction?" } ], "scenario": "Creative brainstorming and idea generation assistance", "complexity": "simple", "tools_used": [], "learning_objectives": ["creative thinking", "brainstorming techniques", "branding assistance", "idea generation patterns"] } ] ================================================ FILE: skills/hugging-face-datasets/examples/system_prompt_template.txt ================================================ You are an AI assistant expert at using MCP (Model Context Protocol) tools effectively. You have access to various MCP servers and tools that allow you to interact with external systems, databases, APIs, and services. ## MCP SERVER DEFINITIONS ### Available MCP Servers and Tools: **1. Filesystem Server (mcp-server-filesystem)** - `read_file`: Read file contents - `write_file`: Create or overwrite files - `edit_file`: Make precise edits to existing files - `list_directory`: List directory contents - `create_directory`: Create new directories - `move_file`: Move/rename files - `delete_file`: Remove files - `search_files`: Search for files by name pattern **2. Git Server (mcp-server-git)** - `git_status`: Check repository status - `git_log`: View commit history - `git_diff`: Show changes between commits - `git_commit`: Create commits - `git_branch`: Manage branches - `git_push`: Push changes to remote - `git_pull`: Pull changes from remote **3. Database Server (mcp-server-sqlite)** - `execute_query`: Run SQL queries - `describe_table`: Get table schema - `list_tables`: Show all tables - `create_table`: Create new tables - `insert_data`: Add records - `backup_database`: Create database backup **4. Web Server (mcp-server-web)** - `fetch_url`: Get web page content - `post_request`: Send POST requests - `search_web`: Search the internet - `download_file`: Download files from URLs **5. Code Analysis Server (mcp-server-code)** - `analyze_code`: Static code analysis - `find_functions`: Locate function definitions - `get_dependencies`: Extract imports/dependencies - `format_code`: Auto-format source code - `run_tests`: Execute test suites ## TRAINING EXAMPLE STRUCTURE Generate training examples using this exact structure: ```json { "messages": [ { "role": "user", "content": "[Natural user request describing a real problem]" }, { "role": "assistant", "content": "[Conversational response with embedded MCP tool usage]", "tool_calls": [ { "type": "function", "function": { "name": "[mcp_tool_name]", "arguments": { "[param1]": "[value1]", "[param2]": "[value2]" } } } ] }, { "role": "tool", "content": "[Simulated tool response/output]", "tool_call_id": "call_123" }, { "role": "assistant", "content": "[Follow-up response analyzing results and next steps]" } ], "scenario": "[Brief description of the use case]", "complexity": "[simple|intermediate|advanced]", "tools_used": ["[tool1]", "[tool2]"], "learning_objectives": ["[objective1]", "[objective2]"] } ``` ## QUALITY GUIDELINES **1. Realistic Scenarios**: Create examples based on actual developer workflows: - Debugging application errors - Setting up new projects - Code refactoring and optimization - Database management tasks - API integration challenges - Version control operations - File organization and cleanup **2. Progressive Complexity**: - **Simple**: Single tool usage for straightforward tasks - **Intermediate**: Multiple tools working together - **Advanced**: Complex workflows with error handling and iteration **3. Tool Usage Patterns**: - Always explain tool selection rationale - Show proper parameter usage - Demonstrate error handling - Include realistic tool outputs - Show iterative problem-solving **4. Conversational Quality**: - Natural language flow - User context awareness - Helpful explanations without being verbose - Professional but approachable tone - Clear next steps and follow-up options **5. Technical Accuracy**: - Correct MCP tool syntax - Realistic file paths and data - Valid code snippets - Proper error messages - Authentic command outputs ## EXAMPLE CATEGORIES TO COVER **Development Workflows**: - Project setup and initialization - Code review and quality checks - Dependency management - Build and deployment processes **Git & Version Control**: - Feature branch workflows - Merge conflict resolution - Repository management - Collaborative development patterns **Code Analysis & Optimization**: - Performance profiling and optimization - Code quality assessment - Refactoring guidance - Architecture analysis **Debugging & Troubleshooting**: - Error investigation - Log analysis - Performance profiling - Configuration issues **Data Management**: - Database queries and updates - File processing and transformation - Backup and migration tasks - Data validation and cleanup **Integration Tasks**: - API testing and integration - External service connections - Webhook setup and testing - Authentication implementation **Educational & Mentoring**: - Concept explanations - Technical tutorials - Best practices guidance - Learning path recommendations **Content Generation**: - Professional writing assistance - Documentation creation - Creative brainstorming - Technical communication **Conversational Support**: - Problem-solving discussions - Project guidance - Technical Q&A - Collaborative planning **Codebase Navigation**: - Legacy code exploration - Architecture understanding - Documentation analysis - Systematic code review **Maintenance & Operations**: - Code refactoring - Documentation updates - Security audits - System monitoring Generate examples that demonstrate sophisticated MCP usage while maintaining natural conversational flow and practical applicability. ================================================ FILE: skills/hugging-face-datasets/examples/training_examples.json ================================================ [ { "messages": [ { "role": "user", "content": "My Node.js app is crashing on startup. Can you help me debug this? The error message mentions something about a missing config file." }, { "role": "assistant", "content": "I'll help you debug the startup crash. Let me first check your application structure and then look at the logs to understand the missing config file issue.", "tool_calls": [ { "type": "function", "function": { "name": "list_directory", "arguments": { "path": "." } } } ] }, { "role": "tool", "content": "package.json\nserver.js\nsrc/\nnode_modules/\nREADME.md\nlogs/", "tool_call_id": "call_001" }, { "role": "assistant", "content": "I can see your project structure. Now let me check the main server file to understand what config file it's expecting.", "tool_calls": [ { "type": "function", "function": { "name": "read_file", "arguments": { "path": "server.js" } } } ] }, { "role": "tool", "content": "const express = require('express');\nconst config = require('./config/database.json');\nconst app = express();\n\napp.listen(config.port, () => {\n console.log(`Server running on port ${config.port}`);\n});", "tool_call_id": "call_002" }, { "role": "assistant", "content": "Found the issue! Your server.js is trying to load './config/database.json' but I don't see a config directory. Let me check if it exists and create it if needed.", "tool_calls": [ { "type": "function", "function": { "name": "list_directory", "arguments": { "path": "config" } } } ] }, { "role": "tool", "content": "Error: Directory 'config' does not exist", "tool_call_id": "call_003" }, { "role": "assistant", "content": "Perfect! The config directory is missing. I'll create it and add a basic database configuration file to fix your startup crash.", "tool_calls": [ { "type": "function", "function": { "name": "create_directory", "arguments": { "path": "config" } } } ] }, { "role": "tool", "content": "Directory 'config' created successfully", "tool_call_id": "call_004" }, { "role": "assistant", "content": "Now I'll create the missing database.json configuration file with sensible defaults.", "tool_calls": [ { "type": "function", "function": { "name": "write_file", "arguments": { "path": "config/database.json", "content": "{\n \"port\": 3000,\n \"database\": {\n \"host\": \"localhost\",\n \"port\": 5432,\n \"name\": \"myapp\",\n \"user\": \"postgres\"\n }\n}" } } } ] }, { "role": "tool", "content": "File 'config/database.json' created successfully", "tool_call_id": "call_005" }, { "role": "assistant", "content": "Great! I've fixed the missing config file issue. Your Node.js app should now start without crashing. The server will run on port 3000 by default. You may want to adjust the database connection settings in config/database.json to match your actual database setup." } ], "scenario": "Debugging Node.js startup crash due to missing configuration file", "complexity": "intermediate", "tools_used": ["list_directory", "read_file", "create_directory", "write_file"], "learning_objectives": ["systematic debugging approach", "file system navigation", "configuration file management", "error root cause analysis"] }, { "messages": [ { "role": "user", "content": "I need to set up a new React project with TypeScript and add some basic components. Can you help me get started?" }, { "role": "assistant", "content": "I'll help you set up a new React TypeScript project from scratch. Let me start by creating the project structure and configuration files.", "tool_calls": [ { "type": "function", "function": { "name": "create_directory", "arguments": { "path": "my-react-app" } } } ] }, { "role": "tool", "content": "Directory 'my-react-app' created successfully", "tool_call_id": "call_101" }, { "role": "assistant", "content": "Now I'll create the package.json with React and TypeScript dependencies.", "tool_calls": [ { "type": "function", "function": { "name": "write_file", "arguments": { "path": "my-react-app/package.json", "content": "{\n \"name\": \"my-react-app\",\n \"version\": \"1.0.0\",\n \"scripts\": {\n \"dev\": \"vite\",\n \"build\": \"tsc && vite build\",\n \"preview\": \"vite preview\"\n },\n \"dependencies\": {\n \"react\": \"^18.2.0\",\n \"react-dom\": \"^18.2.0\"\n },\n \"devDependencies\": {\n \"@types/react\": \"^18.2.0\",\n \"@types/react-dom\": \"^18.2.0\",\n \"@vitejs/plugin-react\": \"^4.0.0\",\n \"typescript\": \"^5.0.0\",\n \"vite\": \"^4.3.0\"\n }\n}" } } } ] }, { "role": "tool", "content": "File 'my-react-app/package.json' created successfully", "tool_call_id": "call_102" }, { "role": "assistant", "content": "Perfect! I've set up your React TypeScript project with proper configuration. The project includes package.json with all necessary dependencies, and a clean structure. You can now run `npm install` to install dependencies and start developing your components. Would you like me to create any specific components or add additional configuration?" } ], "scenario": "Setting up new React TypeScript project from scratch", "complexity": "simple", "tools_used": ["create_directory", "write_file"], "learning_objectives": ["project initialization", "React TypeScript setup", "configuration management", "directory structure planning"] } ] ================================================ FILE: skills/hugging-face-datasets/scripts/dataset_manager.py ================================================ #!/usr/bin/env -S uv run # /// script # requires-python = ">=3.10" # dependencies = [ # "huggingface_hub>=0.20.0", # ] # /// """ Hugging Face Dataset Manager Enhanced dataset creation and management tool designed to work alongside the HF MCP server. Provides dataset creation, configuration, and content management capabilities optimized for conversational AI training data. Version: 2.0.0 Usage: uv run dataset_manager.py init --repo_id username/dataset-name uv run dataset_manager.py quick_setup --repo_id username/dataset-name --template chat uv run dataset_manager.py add_rows --repo_id username/dataset-name --rows_json '[{"messages": [...]}]' uv run dataset_manager.py stats --repo_id username/dataset-name uv run dataset_manager.py list_templates """ import os import json import time import argparse from pathlib import Path from typing import List, Dict, Any, Optional from huggingface_hub import HfApi, create_repo from huggingface_hub.utils import HfHubHTTPError # Configuration HF_TOKEN = os.environ.get("HF_TOKEN") EXAMPLES_DIR = Path(__file__).parent.parent / "examples" def init_dataset(repo_id, token=None, private=True): """ Initialize a new dataset repository on Hugging Face Hub. """ api = HfApi(token=token) try: create_repo(repo_id, repo_type="dataset", private=private, token=token) print(f"Created dataset repository: {repo_id}") except HfHubHTTPError as e: if "409" in str(e): print(f"Repository {repo_id} already exists.") else: raise e # Create a basic README.md with metadata if it doesn't exist readme_content = f"""--- license: mit --- # {repo_id.split("/")[-1]} This dataset was created using the Claude Dataset Skill. """ try: api.upload_file( path_or_fileobj=readme_content.encode("utf-8"), path_in_repo="README.md", repo_id=repo_id, repo_type="dataset", commit_message="Initialize dataset README", ) except Exception as e: print(f"Note: README might already exist or failed to update: {e}") def define_config(repo_id, system_prompt=None, token=None): """ Define a configuration for the dataset, including a system prompt. This saves a config.json file to the repository. """ api = HfApi(token=token) config_data = {"dataset_config": {"version": "1.0", "created_at": time.time()}} if system_prompt: config_data["system_prompt"] = system_prompt # Upload config.json api.upload_file( path_or_fileobj=json.dumps(config_data, indent=2).encode("utf-8"), path_in_repo="config.json", repo_id=repo_id, repo_type="dataset", commit_message="Update dataset configuration", ) print(f"Configuration updated for {repo_id}") def load_dataset_template(template_name: str) -> Dict[str, Any]: """Load dataset template configuration from templates directory.""" template_path = EXAMPLES_DIR.parent / "templates" / f"{template_name}.json" if not template_path.exists(): available_templates = [f.stem for f in (EXAMPLES_DIR.parent / "templates").glob("*.json")] print(f"❌ Template '{template_name}' not found.") print(f"Available templates: {', '.join(available_templates)}") return {} with open(template_path) as f: return json.load(f) def validate_by_template(rows: List[Dict[str, Any]], template: Dict[str, Any]) -> bool: """Validate data according to template schema.""" if not template: return False schema = template.get("validation_schema", {}) required_fields = set(schema.get("required_fields", [])) recommended_fields = set(schema.get("recommended_fields", [])) field_types = schema.get("field_types", {}) for i, row in enumerate(rows): # Check required fields if not all(field in row for field in required_fields): missing = required_fields - set(row.keys()) print(f"Row {i}: Missing required fields: {missing}") return False # Validate field types for field, expected_type in field_types.items(): if field in row: if not _validate_field_type(row[field], expected_type, f"Row {i}, field '{field}'"): return False # Template-specific validation if template["type"] == "chat": if not _validate_chat_format(row, i): return False elif template["type"] == "classification": if not _validate_classification_format(row, i): return False elif template["type"] == "tabular": if not _validate_tabular_format(row, i): return False # Warn about missing recommended fields missing_recommended = recommended_fields - set(row.keys()) if missing_recommended: print(f"Row {i}: Recommended to include: {missing_recommended}") print(f"✓ Validated {len(rows)} examples for {template['type']} dataset") return True def _validate_field_type(value: Any, expected_type: str, context: str) -> bool: """Validate individual field type.""" if expected_type.startswith("enum:"): valid_values = expected_type[5:].split(",") if value not in valid_values: print(f"{context}: Invalid value '{value}'. Must be one of: {valid_values}") return False elif expected_type == "array" and not isinstance(value, list): print(f"{context}: Expected array, got {type(value).__name__}") return False elif expected_type == "object" and not isinstance(value, dict): print(f"{context}: Expected object, got {type(value).__name__}") return False elif expected_type == "string" and not isinstance(value, str): print(f"{context}: Expected string, got {type(value).__name__}") return False elif expected_type == "number" and not isinstance(value, (int, float)): print(f"{context}: Expected number, got {type(value).__name__}") return False return True def _validate_chat_format(row: Dict[str, Any], row_index: int) -> bool: """Validate chat-specific format.""" messages = row.get("messages", []) if not isinstance(messages, list) or len(messages) == 0: print(f"Row {row_index}: 'messages' must be a non-empty list") return False valid_roles = {"user", "assistant", "tool", "system"} for j, msg in enumerate(messages): if not isinstance(msg, dict): print(f"Row {row_index}, message {j}: Must be an object") return False if "role" not in msg or msg["role"] not in valid_roles: print(f"Row {row_index}, message {j}: Invalid role. Use: {valid_roles}") return False if "content" not in msg: print(f"Row {row_index}, message {j}: Missing 'content' field") return False return True def _validate_classification_format(row: Dict[str, Any], row_index: int) -> bool: """Validate classification-specific format.""" if "text" not in row: print(f"Row {row_index}: Missing 'text' field") return False if "label" not in row: print(f"Row {row_index}: Missing 'label' field") return False return True def _validate_tabular_format(row: Dict[str, Any], row_index: int) -> bool: """Validate tabular-specific format.""" if "data" not in row: print(f"Row {row_index}: Missing 'data' field") return False if "columns" not in row: print(f"Row {row_index}: Missing 'columns' field") return False data = row["data"] columns = row["columns"] if not isinstance(data, list): print(f"Row {row_index}: 'data' must be an array") return False if not isinstance(columns, list): print(f"Row {row_index}: 'columns' must be an array") return False return True def validate_training_data(rows: List[Dict[str, Any]], template_name: str = "chat") -> bool: """ Validate training data structure according to template. Supports multiple dataset types with appropriate validation. """ template = load_dataset_template(template_name) if not template: print(f"❌ Could not load template '{template_name}', falling back to basic validation") return _basic_validation(rows) return validate_by_template(rows, template) def _basic_validation(rows: List[Dict[str, Any]]) -> bool: """Basic validation when no template is available.""" for i, row in enumerate(rows): if not isinstance(row, dict): print(f"Row {i}: Must be a dictionary/object") return False print(f"✓ Basic validation passed for {len(rows)} rows") return True def add_rows( repo_id: str, rows: List[Dict[str, Any]], split: str = "train", validate: bool = True, template: str = "chat", token: Optional[str] = None, ) -> None: """ Stream updates to the dataset by uploading a new chunk of rows. Enhanced with validation for multiple dataset types. Args: repo_id: Repository identifier (username/dataset-name) rows: List of training examples split: Dataset split name (train, test, validation) validate: Whether to validate data structure before upload template: Dataset template type (chat, classification, qa, completion, tabular, custom) token: HuggingFace API token """ api = HfApi(token=token) if not rows: print("No rows to add.") return # Validate training data structure if validate and not validate_training_data(rows, template): print("❌ Validation failed. Use --no-validate to skip validation.") return # Create a newline-delimited JSON string jsonl_content = "\n".join(json.dumps(row) for row in rows) # Generate a unique filename for this chunk timestamp = int(time.time() * 1000) filename = f"data/{split}-{timestamp}.jsonl" try: api.upload_file( path_or_fileobj=jsonl_content.encode("utf-8"), path_in_repo=filename, repo_id=repo_id, repo_type="dataset", commit_message=f"Add {len(rows)} rows to {split} split", ) print(f"✅ Added {len(rows)} rows to {repo_id} (split: {split})") except Exception as e: print(f"❌ Upload failed: {e}") return def load_template(template_name: str = "system_prompt_template.txt") -> str: """Load a template file from the examples directory.""" template_path = EXAMPLES_DIR / template_name if template_path.exists(): return template_path.read_text() else: print(f"⚠️ Template {template_name} not found at {template_path}") return "" def quick_setup(repo_id: str, template_type: str = "chat", token: Optional[str] = None) -> None: """ Quick setup for different dataset types using templates. Args: repo_id: Repository identifier template_type: Dataset template (chat, classification, qa, completion, tabular, custom) token: HuggingFace API token """ print(f"🚀 Quick setup for {repo_id} with '{template_type}' template...") # Load template configuration template_config = load_dataset_template(template_type) if not template_config: print(f"❌ Could not load template '{template_type}'. Setup cancelled.") return # Initialize repository init_dataset(repo_id, token=token, private=True) # Configure with template system prompt system_prompt = template_config.get("system_prompt", "") if system_prompt: define_config(repo_id, system_prompt=system_prompt, token=token) # Add template examples examples = template_config.get("examples", []) if examples: add_rows(repo_id, examples, template=template_type, token=token) print(f"✅ Added {len(examples)} example(s) from template") print(f"✅ Quick setup complete for {repo_id}") print(f"📊 Dataset type: {template_config.get('description', 'No description')}") # Show next steps print(f"\n📋 Next steps:") print( f"1. Add more data: uv run scripts/dataset_manager.py add_rows --repo_id {repo_id} --template {template_type} --rows_json 'your_data.json'" ) print(f"2. View stats: uv run scripts/dataset_manager.py stats --repo_id {repo_id}") print(f"3. Explore at: https://huggingface.co/datasets/{repo_id}") def show_stats(repo_id: str, token: Optional[str] = None) -> None: """Display statistics about the dataset.""" api = HfApi(token=token) try: # Get repository info repo_info = api.repo_info(repo_id, repo_type="dataset") print(f"\n📊 Dataset Stats: {repo_id}") print(f"Created: {repo_info.created_at}") print(f"Updated: {repo_info.last_modified}") print(f"Private: {repo_info.private}") # List files files = api.list_repo_files(repo_id, repo_type="dataset") data_files = [f for f in files if f.startswith("data/")] print(f"Data files: {len(data_files)}") if "config.json" in files: print("✅ Configuration present") else: print("⚠️ No configuration found") except Exception as e: print(f"❌ Failed to get stats: {e}") def list_available_templates() -> None: """List all available dataset templates with descriptions.""" templates_dir = EXAMPLES_DIR.parent / "templates" if not templates_dir.exists(): print("❌ Templates directory not found") return print("\n📋 Available Dataset Templates:") print("=" * 50) for template_file in templates_dir.glob("*.json"): try: with open(template_file) as f: template = json.load(f) name = template_file.stem desc = template.get("description", "No description available") template_type = template.get("type", name) print(f"\n🏷️ {name}") print(f" Type: {template_type}") print(f" Description: {desc}") # Show required fields schema = template.get("validation_schema", {}) required = schema.get("required_fields", []) if required: print(f" Required fields: {', '.join(required)}") except Exception as e: print(f"❌ Error loading template {template_file.name}: {e}") print( f"\n💡 Usage: uv run scripts/dataset_manager.py quick_setup --repo_id your-username/dataset-name --template TEMPLATE_NAME" ) print(f"📚 Example templates directory: {templates_dir}") if __name__ == "__main__": parser = argparse.ArgumentParser(description="Hugging Face Dataset Manager") subparsers = parser.add_subparsers(dest="command", required=True) # Init command init_parser = subparsers.add_parser("init", help="Initialize a new dataset") init_parser.add_argument("--repo_id", required=True, help="Repository ID (user/repo_name)") init_parser.add_argument("--private", action="store_true", help="Make repository private") # Config command config_parser = subparsers.add_parser("config", help="Setup dataset config") config_parser.add_argument("--repo_id", required=True, help="Repository ID") config_parser.add_argument("--system_prompt", help="System prompt to store in config") # Add rows command add_parser = subparsers.add_parser("add_rows", help="Add rows to the dataset") add_parser.add_argument("--repo_id", required=True, help="Repository ID") add_parser.add_argument("--split", default="train", help="Dataset split (e.g., train, test)") add_parser.add_argument( "--template", default="chat", choices=[ "chat", "classification", "qa", "completion", "tabular", "custom", ], help="Dataset template type for validation", ) add_parser.add_argument( "--rows_json", required=True, help="JSON string containing a list of rows", ) add_parser.add_argument( "--no-validate", dest="validate", action="store_false", help="Skip data validation", ) # Quick setup command setup_parser = subparsers.add_parser("quick_setup", help="Quick setup with template") setup_parser.add_argument("--repo_id", required=True, help="Repository ID") setup_parser.add_argument( "--template", default="chat", choices=[ "chat", "classification", "qa", "completion", "tabular", "custom", ], help="Dataset template type", ) # Stats command stats_parser = subparsers.add_parser("stats", help="Show dataset statistics") stats_parser.add_argument("--repo_id", required=True, help="Repository ID") # List templates command templates_parser = subparsers.add_parser("list_templates", help="List available dataset templates") args = parser.parse_args() token = HF_TOKEN if not token: print("Warning: HF_TOKEN environment variable not set.") if args.command == "init": init_dataset(args.repo_id, token=token, private=args.private) elif args.command == "config": define_config(args.repo_id, system_prompt=args.system_prompt, token=token) elif args.command == "add_rows": try: rows = json.loads(args.rows_json) if not isinstance(rows, list): raise ValueError("rows_json must be a JSON list of objects") add_rows( args.repo_id, rows, split=args.split, template=args.template, validate=args.validate, token=token, ) except json.JSONDecodeError: print("Error: Invalid JSON provided for --rows_json") elif args.command == "quick_setup": quick_setup(args.repo_id, template_type=args.template, token=token) elif args.command == "stats": show_stats(args.repo_id, token=token) elif args.command == "list_templates": list_available_templates() ================================================ FILE: skills/hugging-face-datasets/scripts/sql_manager.py ================================================ #!/usr/bin/env -S uv run # /// script # requires-python = ">=3.10" # dependencies = [ # "duckdb>=1.0.0", # "huggingface_hub>=0.20.0", # "datasets>=2.14.0", # "pandas>=2.0.0", # ] # /// """ Hugging Face Dataset SQL Manager Query, transform, and push Hugging Face datasets using DuckDB's SQL interface. Supports the hf:// protocol for direct dataset access, data wrangling, and pushing results back to the Hub. Version: 1.0.0 Usage: # Query a dataset uv run sql_manager.py query --dataset "cais/mmlu" --sql "SELECT * FROM data LIMIT 10" # Query and push to new dataset uv run sql_manager.py query --dataset "cais/mmlu" --sql "SELECT * FROM data WHERE subject='nutrition'" \ --push-to "username/nutrition-subset" # Describe dataset schema uv run sql_manager.py describe --dataset "cais/mmlu" # List available splits/configs uv run sql_manager.py info --dataset "cais/mmlu" # Get random sample uv run sql_manager.py sample --dataset "cais/mmlu" --n 5 # Export to parquet uv run sql_manager.py export --dataset "cais/mmlu" --output "data.parquet" """ import os import json import re import argparse from typing import Optional, List, Dict, Any, Union import duckdb from huggingface_hub import HfApi # Regex for valid SQL identifiers (column names, view names) _IDENTIFIER_RE = re.compile(r"^[A-Za-z_][A-Za-z0-9_]*$") # Configuration HF_TOKEN = os.environ.get("HF_TOKEN") class HFDatasetSQL: """ Query Hugging Face datasets using DuckDB SQL. Examples: >>> sql = HFDatasetSQL() >>> results = sql.query("cais/mmlu", "SELECT * FROM data LIMIT 5") >>> schema = sql.describe("cais/mmlu") >>> sql.query_and_push("cais/mmlu", "SELECT * FROM data WHERE subject='nutrition'", "user/nutrition-qa") """ def __init__(self, token: Optional[str] = None): """Initialize the SQL manager with optional HF token.""" self.token = token or HF_TOKEN self.conn = duckdb.connect() self._setup_connection() @staticmethod def _quote_identifier(name: str) -> str: """Quote a SQL identifier, escaping embedded double-quotes.""" return '"' + name.replace('"', '""') + '"' @staticmethod def _validate_identifier(name: str) -> None: """Raise ValueError if *name* is not a safe SQL identifier.""" if not _IDENTIFIER_RE.match(name): raise ValueError( f"Invalid identifier: {name!r}. " "Identifiers must start with a letter or underscore and contain only " "alphanumeric characters and underscores." ) def _setup_connection(self): """Configure DuckDB connection for HF access.""" # Set HF token if available (for private datasets) if self.token: self.conn.execute("CREATE SECRET hf_token (TYPE HUGGINGFACE, TOKEN $1);", [self.token]) def _build_hf_path( self, dataset_id: str, split: str = "*", config: Optional[str] = None, revision: str = "~parquet" ) -> str: """ Build the hf:// path for a dataset. Args: dataset_id: Dataset ID (e.g., "cais/mmlu") split: Split name or "*" for all splits config: Optional config/subset name revision: Revision, defaults to ~parquet for auto-converted parquet Returns: hf:// path string """ if config: return f"hf://datasets/{dataset_id}@{revision}/{config}/{split}/*.parquet" else: return f"hf://datasets/{dataset_id}@{revision}/default/{split}/*.parquet" def _build_hf_path_flexible( self, dataset_id: str, split: Optional[str] = None, config: Optional[str] = None, ) -> str: """ Build flexible hf:// path with wildcards for discovery. Args: dataset_id: Dataset ID split: Optional specific split config: Optional config name Returns: hf:// path with appropriate wildcards """ base = f"hf://datasets/{dataset_id}@~parquet" if config and split: return f"{base}/{config}/{split}/*.parquet" elif config: return f"{base}/{config}/*/*.parquet" elif split: return f"{base}/*/{split}/*.parquet" else: return f"{base}/*/*/*.parquet" def query( self, dataset_id: str, sql: str, split: str = "train", config: Optional[str] = None, limit: Optional[int] = None, output_format: str = "dict", ) -> Union[List[Dict], Any]: """ Execute SQL query on a Hugging Face dataset. Args: dataset_id: Dataset ID (e.g., "cais/mmlu", "ibm/duorc") sql: SQL query. Use 'data' as table name (will be replaced with actual path) split: Dataset split (train, test, validation, or * for all) config: Optional dataset config/subset limit: Optional limit override output_format: Output format - "dict", "df" (pandas), "arrow", "raw" Returns: Query results in specified format Examples: >>> sql.query("cais/mmlu", "SELECT * FROM data WHERE subject='nutrition' LIMIT 10") >>> sql.query("cais/mmlu", "SELECT subject, COUNT(*) as cnt FROM data GROUP BY subject") """ # Build the HF path hf_path = self._build_hf_path(dataset_id, split=split, config=config) # Replace 'data' placeholder with actual path # Handle various SQL patterns processed_sql = sql.replace("FROM data", f"FROM '{hf_path}'") processed_sql = processed_sql.replace("from data", f"FROM '{hf_path}'") processed_sql = processed_sql.replace("JOIN data", f"JOIN '{hf_path}'") processed_sql = processed_sql.replace("join data", f"JOIN '{hf_path}'") # If user provides raw path, use as-is if "hf://" in sql: processed_sql = sql # Apply limit if specified and not already in query if limit and "LIMIT" not in processed_sql.upper(): processed_sql += f" LIMIT {limit}" try: result = self.conn.execute(processed_sql) if output_format == "df": return result.fetchdf() elif output_format == "arrow": return result.fetch_arrow_table() elif output_format == "raw": return result.fetchall() else: # dict columns = [desc[0] for desc in result.description] rows = result.fetchall() return [dict(zip(columns, row)) for row in rows] except Exception as e: print(f"❌ Query error: {e}") print(f" SQL: {processed_sql[:200]}...") raise def query_raw(self, sql: str, output_format: str = "dict") -> Union[List[Dict], Any]: """ Execute raw SQL query without path substitution. Useful for queries that already contain full hf:// paths or for multi-dataset queries. Args: sql: Complete SQL query output_format: Output format Returns: Query results """ result = self.conn.execute(sql) if output_format == "df": return result.fetchdf() elif output_format == "arrow": return result.fetch_arrow_table() elif output_format == "raw": return result.fetchall() else: columns = [desc[0] for desc in result.description] rows = result.fetchall() return [dict(zip(columns, row)) for row in rows] def describe(self, dataset_id: str, split: str = "train", config: Optional[str] = None) -> List[Dict[str, str]]: """ Get schema/structure of a dataset. Args: dataset_id: Dataset ID split: Dataset split config: Optional config Returns: List of column definitions with name, type, nullable info """ hf_path = self._build_hf_path(dataset_id, split=split, config=config) sql = f"DESCRIBE SELECT * FROM '{hf_path}' LIMIT 1" result = self.conn.execute(sql) columns = [desc[0] for desc in result.description] rows = result.fetchall() return [dict(zip(columns, row)) for row in rows] def sample( self, dataset_id: str, n: int = 10, split: str = "train", config: Optional[str] = None, seed: Optional[int] = None, ) -> List[Dict]: """ Get a random sample from a dataset. Args: dataset_id: Dataset ID n: Number of samples split: Dataset split config: Optional config seed: Random seed for reproducibility Returns: List of sampled rows """ hf_path = self._build_hf_path(dataset_id, split=split, config=config) if seed is not None: sql = f"SELECT * FROM '{hf_path}' USING SAMPLE {n} (RESERVOIR, {seed})" else: sql = f"SELECT * FROM '{hf_path}' USING SAMPLE {n}" return self.query_raw(sql) def count( self, dataset_id: str, split: str = "train", config: Optional[str] = None, where: Optional[str] = None ) -> int: """ Count rows in a dataset, optionally with filter. Args: dataset_id: Dataset ID split: Dataset split config: Optional config where: Optional WHERE clause (without WHERE keyword) Returns: Row count """ hf_path = self._build_hf_path(dataset_id, split=split, config=config) sql = f"SELECT COUNT(*) FROM '{hf_path}'" if where: sql += f" WHERE {where}" result = self.conn.execute(sql).fetchone() return result[0] if result else 0 def unique_values( self, dataset_id: str, column: str, split: str = "train", config: Optional[str] = None, limit: int = 100 ) -> List[Any]: """ Get unique values in a column. Args: dataset_id: Dataset ID column: Column name split: Dataset split config: Optional config limit: Max unique values to return Returns: List of unique values """ hf_path = self._build_hf_path(dataset_id, split=split, config=config) quoted_col = self._quote_identifier(column) sql = f"SELECT DISTINCT {quoted_col} FROM '{hf_path}' LIMIT {limit}" result = self.conn.execute(sql).fetchall() return [row[0] for row in result] def histogram( self, dataset_id: str, column: str, split: str = "train", config: Optional[str] = None, bins: int = 10 ) -> List[Dict]: """ Get value distribution/histogram for a column. Args: dataset_id: Dataset ID column: Column name split: Dataset split config: Optional config bins: Number of bins for numeric columns Returns: Distribution data """ hf_path = self._build_hf_path(dataset_id, split=split, config=config) quoted_col = self._quote_identifier(column) sql = f""" SELECT {quoted_col}, COUNT(*) as count FROM '{hf_path}' GROUP BY {quoted_col} ORDER BY count DESC LIMIT {bins} """ return self.query_raw(sql) def filter_and_transform( self, dataset_id: str, select: str = "*", where: Optional[str] = None, group_by: Optional[str] = None, order_by: Optional[str] = None, split: str = "train", config: Optional[str] = None, limit: Optional[int] = None, ) -> List[Dict]: """ Filter and transform dataset with SQL clauses. Args: dataset_id: Dataset ID select: SELECT clause (columns, expressions, aggregations) where: WHERE clause (filter conditions) group_by: GROUP BY clause order_by: ORDER BY clause split: Dataset split config: Optional config limit: Row limit Returns: Transformed data Examples: >>> sql.filter_and_transform( ... "cais/mmlu", ... select="subject, COUNT(*) as cnt", ... group_by="subject", ... order_by="cnt DESC", ... limit=10 ... ) """ hf_path = self._build_hf_path(dataset_id, split=split, config=config) sql_parts = [f"SELECT {select}", f"FROM '{hf_path}'"] if where: sql_parts.append(f"WHERE {where}") if group_by: sql_parts.append(f"GROUP BY {group_by}") if order_by: sql_parts.append(f"ORDER BY {order_by}") if limit: sql_parts.append(f"LIMIT {limit}") sql = " ".join(sql_parts) return self.query_raw(sql) def join_datasets( self, left_dataset: str, right_dataset: str, on: str, select: str = "*", join_type: str = "INNER", left_split: str = "train", right_split: str = "train", left_config: Optional[str] = None, right_config: Optional[str] = None, limit: Optional[int] = None, ) -> List[Dict]: """ Join two datasets. Args: left_dataset: Left dataset ID right_dataset: Right dataset ID on: JOIN condition (e.g., "left.id = right.id") select: SELECT clause join_type: Type of join (INNER, LEFT, RIGHT, FULL) left_split: Split for left dataset right_split: Split for right dataset left_config: Config for left dataset right_config: Config for right dataset limit: Row limit Returns: Joined data """ left_path = self._build_hf_path(left_dataset, split=left_split, config=left_config) right_path = self._build_hf_path(right_dataset, split=right_split, config=right_config) sql = f""" SELECT {select} FROM '{left_path}' AS left_table {join_type} JOIN '{right_path}' AS right_table ON {on} """ if limit: sql += f" LIMIT {limit}" return self.query_raw(sql) def export_to_parquet( self, dataset_id: str, output_path: str, sql: Optional[str] = None, split: str = "train", config: Optional[str] = None, ) -> str: """ Export query results to a local Parquet file. Args: dataset_id: Source dataset ID output_path: Local path for output Parquet file sql: Optional SQL query (uses SELECT * if not provided) split: Dataset split config: Optional config Returns: Path to created file """ hf_path = self._build_hf_path(dataset_id, split=split, config=config) if sql: # Process the query processed_sql = sql.replace("FROM data", f"FROM '{hf_path}'") processed_sql = processed_sql.replace("from data", f"FROM '{hf_path}'") else: processed_sql = f"SELECT * FROM '{hf_path}'" if "'" in output_path: raise ValueError(f"Invalid output path: paths must not contain single quotes") export_sql = f"COPY ({processed_sql}) TO '{output_path}' (FORMAT PARQUET)" self.conn.execute(export_sql) print(f"✅ Exported to {output_path}") return output_path def export_to_jsonl( self, dataset_id: str, output_path: str, sql: Optional[str] = None, split: str = "train", config: Optional[str] = None, ) -> str: """ Export query results to JSONL format. Args: dataset_id: Source dataset ID output_path: Local path for output JSONL file sql: Optional SQL query split: Dataset split config: Optional config Returns: Path to created file """ results = self.query(dataset_id, sql or "SELECT * FROM data", split=split, config=config) with open(output_path, "w") as f: for row in results: f.write(json.dumps(row) + "\n") print(f"✅ Exported {len(results)} rows to {output_path}") return output_path def push_to_hub( self, dataset_id: str, target_repo: str, sql: Optional[str] = None, split: str = "train", config: Optional[str] = None, target_split: str = "train", private: bool = True, commit_message: Optional[str] = None, ) -> str: """ Query a dataset and push results to a new Hub repository. Args: dataset_id: Source dataset ID target_repo: Target repository ID (e.g., "username/new-dataset") sql: SQL query to transform data (optional, defaults to SELECT *) split: Source split config: Source config target_split: Target split name private: Whether to create private repo commit_message: Commit message Returns: URL of created dataset """ try: from datasets import Dataset except ImportError: raise ImportError( "datasets library required for push_to_hub. " "Run with `uv run ...` or install with `uv pip install datasets`." ) # Execute query results = self.query(dataset_id, sql or "SELECT * FROM data", split=split, config=config) if not results: print("❌ No results to push") return "" # Convert to HF Dataset ds = Dataset.from_list(results) # Push to Hub ds.push_to_hub( target_repo, split=target_split, private=private, commit_message=commit_message or f"Created from {dataset_id} via SQL query", token=self.token, ) url = f"https://huggingface.co/datasets/{target_repo}" print(f"✅ Pushed {len(results)} rows to {url}") return url def create_view(self, name: str, dataset_id: str, split: str = "train", config: Optional[str] = None): """ Create a DuckDB view for easier querying. Args: name: View name dataset_id: Dataset ID split: Dataset split config: Optional config """ self._validate_identifier(name) hf_path = self._build_hf_path(dataset_id, split=split, config=config) quoted_name = self._quote_identifier(name) self.conn.execute(f"CREATE OR REPLACE VIEW {quoted_name} AS SELECT * FROM '{hf_path}'") print(f"✅ Created view '{name}' for {dataset_id}") def info(self, dataset_id: str) -> Dict[str, Any]: """ Get information about a dataset including available configs and splits. Args: dataset_id: Dataset ID Returns: Dataset information """ api = HfApi(token=self.token) try: info = api.dataset_info(dataset_id) result = { "id": info.id, "author": info.author, "private": info.private, "downloads": info.downloads, "likes": info.likes, "tags": info.tags, "created_at": str(info.created_at) if info.created_at else None, "last_modified": str(info.last_modified) if info.last_modified else None, } # Try to get config/split info from card data if info.card_data: result["configs"] = getattr(info.card_data, "configs", None) return result except Exception as e: print(f"❌ Failed to get info: {e}") return {} def close(self): """Close the database connection.""" self.conn.close() def main(): """CLI entry point.""" parser = argparse.ArgumentParser( description="Query Hugging Face datasets with SQL", formatter_class=argparse.RawDescriptionHelpFormatter, epilog=""" Examples: # Query dataset with SQL python sql_manager.py query --dataset "cais/mmlu" --sql "SELECT * FROM data WHERE subject='nutrition' LIMIT 10" # Get random sample python sql_manager.py sample --dataset "cais/mmlu" --n 5 # Describe schema python sql_manager.py describe --dataset "cais/mmlu" # Get value counts python sql_manager.py histogram --dataset "cais/mmlu" --column "subject" # Filter and transform python sql_manager.py transform --dataset "cais/mmlu" \\ --select "subject, COUNT(*) as cnt" \\ --group-by "subject" \\ --order-by "cnt DESC" # Query and push to Hub python sql_manager.py query --dataset "cais/mmlu" \\ --sql "SELECT * FROM data WHERE subject='nutrition'" \\ --push-to "username/nutrition-subset" # Export to Parquet python sql_manager.py export --dataset "cais/mmlu" \\ --sql "SELECT * FROM data WHERE subject='nutrition'" \\ --output "nutrition.parquet" """, ) subparsers = parser.add_subparsers(dest="command", required=True) # Common arguments def add_common_args(p): p.add_argument("--dataset", "-d", required=True, help="Dataset ID (e.g., cais/mmlu)") p.add_argument("--split", "-s", default="train", help="Dataset split (default: train)") p.add_argument("--config", "-c", help="Dataset config/subset") # Query command query_parser = subparsers.add_parser("query", help="Execute SQL query on dataset") add_common_args(query_parser) query_parser.add_argument("--sql", required=True, help="SQL query (use 'data' as table name)") query_parser.add_argument("--limit", "-l", type=int, help="Limit results") query_parser.add_argument("--format", choices=["json", "table", "csv"], default="json", help="Output format") query_parser.add_argument("--push-to", help="Push results to this Hub repo") query_parser.add_argument("--private", action="store_true", help="Make pushed repo private") # Sample command sample_parser = subparsers.add_parser("sample", help="Get random sample from dataset") add_common_args(sample_parser) sample_parser.add_argument("--n", type=int, default=10, help="Number of samples") sample_parser.add_argument("--seed", type=int, help="Random seed") # Describe command describe_parser = subparsers.add_parser("describe", help="Get dataset schema") add_common_args(describe_parser) # Count command count_parser = subparsers.add_parser("count", help="Count rows in dataset") add_common_args(count_parser) count_parser.add_argument("--where", "-w", help="WHERE clause for filtering") # Histogram command histogram_parser = subparsers.add_parser("histogram", help="Get value distribution") add_common_args(histogram_parser) histogram_parser.add_argument("--column", required=True, help="Column name") histogram_parser.add_argument("--bins", type=int, default=20, help="Number of bins") # Unique command unique_parser = subparsers.add_parser("unique", help="Get unique values in column") add_common_args(unique_parser) unique_parser.add_argument("--column", required=True, help="Column name") unique_parser.add_argument("--limit", "-l", type=int, default=100, help="Max values") # Transform command transform_parser = subparsers.add_parser("transform", help="Filter and transform dataset") add_common_args(transform_parser) transform_parser.add_argument("--select", default="*", help="SELECT clause") transform_parser.add_argument("--where", "-w", help="WHERE clause") transform_parser.add_argument("--group-by", help="GROUP BY clause") transform_parser.add_argument("--order-by", help="ORDER BY clause") transform_parser.add_argument("--limit", "-l", type=int, help="LIMIT") transform_parser.add_argument("--push-to", help="Push results to Hub repo") # Export command export_parser = subparsers.add_parser("export", help="Export query results to file") add_common_args(export_parser) export_parser.add_argument("--sql", help="SQL query (defaults to SELECT *)") export_parser.add_argument("--output", "-o", required=True, help="Output file path") export_parser.add_argument("--format", choices=["parquet", "jsonl"], default="parquet", help="Output format") # Info command info_parser = subparsers.add_parser("info", help="Get dataset information") info_parser.add_argument("--dataset", "-d", required=True, help="Dataset ID") # Raw SQL command raw_parser = subparsers.add_parser("raw", help="Execute raw SQL with full hf:// paths") raw_parser.add_argument("--sql", required=True, help="Complete SQL query") raw_parser.add_argument("--format", choices=["json", "table", "csv"], default="json", help="Output format") args = parser.parse_args() # Initialize SQL manager sql = HFDatasetSQL() try: if args.command == "query": results = sql.query(args.dataset, args.sql, split=args.split, config=args.config, limit=args.limit) if getattr(args, "push_to", None): sql.push_to_hub( args.dataset, args.push_to, sql=args.sql, split=args.split, config=args.config, private=args.private ) else: _print_results(results, args.format) elif args.command == "sample": results = sql.sample(args.dataset, n=args.n, split=args.split, config=args.config, seed=args.seed) _print_results(results, "json") elif args.command == "describe": schema = sql.describe(args.dataset, split=args.split, config=args.config) _print_results(schema, "table") elif args.command == "count": count = sql.count(args.dataset, split=args.split, config=args.config, where=args.where) print(f"Count: {count:,}") elif args.command == "histogram": results = sql.histogram(args.dataset, args.column, split=args.split, config=args.config, bins=args.bins) _print_results(results, "table") elif args.command == "unique": values = sql.unique_values( args.dataset, args.column, split=args.split, config=args.config, limit=args.limit ) for v in values: print(v) elif args.command == "transform": results = sql.filter_and_transform( args.dataset, select=args.select, where=args.where, group_by=args.group_by, order_by=args.order_by, split=args.split, config=args.config, limit=args.limit, ) if getattr(args, "push_to", None): # Build SQL for push query_sql = f"SELECT {args.select} FROM data" if args.where: query_sql += f" WHERE {args.where}" if args.group_by: query_sql += f" GROUP BY {args.group_by}" if args.order_by: query_sql += f" ORDER BY {args.order_by}" if args.limit: query_sql += f" LIMIT {args.limit}" sql.push_to_hub(args.dataset, args.push_to, sql=query_sql, split=args.split, config=args.config) else: _print_results(results, "json") elif args.command == "export": if args.format == "parquet": sql.export_to_parquet(args.dataset, args.output, sql=args.sql, split=args.split, config=args.config) else: sql.export_to_jsonl(args.dataset, args.output, sql=args.sql, split=args.split, config=args.config) elif args.command == "info": info = sql.info(args.dataset) _print_results([info], "json") elif args.command == "raw": results = sql.query_raw(args.sql) _print_results(results, args.format) finally: sql.close() def _print_results(results: List[Dict], format: str): """Print results in specified format.""" if not results: print("No results") return if format == "json": print(json.dumps(results, indent=2, default=str)) elif format == "csv": if results: keys = results[0].keys() print(",".join(str(k) for k in keys)) for row in results: print(",".join(str(row.get(k, "")) for k in keys)) elif format == "table": if results: keys = list(results[0].keys()) # Calculate column widths widths = {k: max(len(str(k)), max(len(str(r.get(k, ""))) for r in results)) for k in keys} # Header header = " | ".join(str(k).ljust(widths[k]) for k in keys) print(header) print("-" * len(header)) # Rows for row in results: print(" | ".join(str(row.get(k, "")).ljust(widths[k]) for k in keys)) if __name__ == "__main__": main() ================================================ FILE: skills/hugging-face-datasets/templates/chat.json ================================================ { "type": "chat", "description": "Conversational AI training data with chat templating support", "system_prompt": "You are an AI assistant expert at using tools and engaging in helpful conversations. Generate training examples that demonstrate natural dialogue patterns, tool usage, and problem-solving interactions.", "validation_schema": { "required_fields": ["messages"], "recommended_fields": ["scenario", "complexity", "tools_used", "learning_objectives"], "field_types": { "messages": "array", "scenario": "string", "complexity": "enum:simple,intermediate,advanced", "tools_used": "array", "learning_objectives": "array" } }, "example_structure": { "messages": [ { "role": "user|assistant|tool|system", "content": "Message content", "tool_calls": [ { "type": "function", "function": { "name": "tool_name", "arguments": {"param": "value"} } } ] } ], "scenario": "Brief description of the use case", "complexity": "simple|intermediate|advanced", "tools_used": ["tool1", "tool2"], "learning_objectives": ["objective1", "objective2"] }, "examples": [ { "messages": [ { "role": "user", "content": "Hello! Can you help me understand how to use APIs?" }, { "role": "assistant", "content": "I'd be happy to help you understand APIs! An API (Application Programming Interface) is like a bridge that allows different software applications to communicate with each other.\n\nThink of it like ordering at a restaurant:\n- You (the client) look at the menu (API documentation)\n- You tell the waiter (the API) what you want\n- The waiter takes your order to the kitchen (the server)\n- The kitchen prepares your food (processes the request)\n- The waiter brings your food back (returns the response)\n\nWould you like me to show you a simple example of making an API call?" } ], "scenario": "API education and explanation", "complexity": "simple", "tools_used": [], "learning_objectives": ["API concepts", "educational conversation", "analogical explanation"] } ] } ================================================ FILE: skills/hugging-face-datasets/templates/classification.json ================================================ { "type": "classification", "description": "Text classification datasets for sentiment analysis, topic classification, intent detection, etc.", "system_prompt": "Create high-quality text classification training data. Each example should have clear, unambiguous text and appropriate labels. Ensure balanced representation across all classes and include diverse language patterns.", "validation_schema": { "required_fields": ["text", "label"], "recommended_fields": ["confidence", "metadata", "source"], "field_types": { "text": "string", "label": "string|array", "confidence": "number", "metadata": "object", "source": "string" } }, "example_structure": { "text": "Input text to be classified", "label": "classification_label", "confidence": 0.95, "metadata": { "length": 42, "language": "en", "domain": "technology" }, "source": "web_scraping" }, "examples": [ { "text": "I absolutely love this new smartphone! The camera quality is outstanding and the battery lasts all day.", "label": "positive", "confidence": 0.98, "metadata": { "length": 98, "language": "en", "domain": "product_reviews" }, "source": "customer_feedback" }, { "text": "The software keeps crashing and customer support is unresponsive. Very disappointed with this purchase.", "label": "negative", "confidence": 0.92, "metadata": { "length": 96, "language": "en", "domain": "product_reviews" }, "source": "customer_feedback" }, { "text": "Book a table for two at 7 PM tonight", "label": "restaurant_reservation", "confidence": 0.95, "metadata": { "length": 35, "language": "en", "domain": "intent_detection" }, "source": "voice_assistant" } ] } ================================================ FILE: skills/hugging-face-datasets/templates/completion.json ================================================ { "type": "completion", "description": "Text completion and generation datasets for language modeling, code completion, creative writing", "system_prompt": "Generate diverse text completion examples that demonstrate natural language patterns, creativity, and contextual understanding. Include various domains like code, creative writing, technical documentation, and conversational text.", "validation_schema": { "required_fields": ["prompt", "completion"], "recommended_fields": ["domain", "style", "length", "difficulty"], "field_types": { "prompt": "string", "completion": "string", "domain": "enum:code,creative,technical,conversational,academic", "style": "string", "length": "enum:short,medium,long", "difficulty": "enum:beginner,intermediate,advanced" } }, "example_structure": { "prompt": "The beginning text or context", "completion": "The expected continuation or completion", "domain": "code|creative|technical|conversational|academic", "style": "description of writing style", "length": "short|medium|long", "difficulty": "beginner|intermediate|advanced" }, "examples": [ { "prompt": "def fibonacci(n):\n \"\"\"Calculate the nth Fibonacci number.\"\"\"\n if n <= 1:\n return n\n else:", "completion": " return fibonacci(n-1) + fibonacci(n-2)", "domain": "code", "style": "python_function", "length": "short", "difficulty": "beginner" }, { "prompt": "The old lighthouse stood silently on the cliff, its beacon long extinguished. Sarah approached slowly, wondering", "completion": "if the stories her grandmother had told her were true. As she pushed open the weathered door, a gust of salt air carried whispers of the past, and she could almost see the ghostly figure of the lighthouse keeper who had vanished mysteriously fifty years ago.", "domain": "creative", "style": "narrative_fiction", "length": "medium", "difficulty": "intermediate" }, { "prompt": "To configure SSL/TLS encryption for your web server, you'll need to:", "completion": "1. Obtain an SSL certificate from a trusted Certificate Authority (CA)\n2. Install the certificate on your web server\n3. Configure your server to use HTTPS (port 443)\n4. Update your application to redirect HTTP traffic to HTTPS\n5. Test the implementation using SSL testing tools\n\nMake sure to backup your server configuration before making changes.", "domain": "technical", "style": "instructional", "length": "medium", "difficulty": "intermediate" } ] } ================================================ FILE: skills/hugging-face-datasets/templates/custom.json ================================================ { "type": "custom", "description": "Flexible template for custom dataset formats - define your own schema and validation rules", "system_prompt": "Generate data according to the custom schema provided. Follow the field definitions, data types, and validation rules specified in the schema configuration.", "validation_schema": { "required_fields": ["data"], "recommended_fields": ["schema", "metadata"], "field_types": { "data": "any", "schema": "object", "metadata": "object" } }, "example_structure": { "data": "Your custom data structure goes here", "schema": { "fields": [ { "name": "field_name", "type": "string|number|boolean|array|object", "required": true, "description": "Field description" } ], "validation_rules": { "custom_rule_1": "validation logic", "custom_rule_2": "additional validation" } }, "metadata": { "created_by": "user", "purpose": "dataset purpose", "version": "1.0" } }, "examples": [ { "data": { "id": "unique_identifier", "features": { "text_feature": "sample text", "numeric_features": [1.0, 2.5, -0.3], "categorical_feature": "category_a" }, "labels": { "primary_label": "positive", "confidence_score": 0.87, "secondary_labels": ["helpful", "informative"] }, "annotations": { "annotator_id": "expert_1", "annotation_time": "2024-01-15T10:30:00Z", "quality_score": 9 } }, "schema": { "fields": [ {"name": "id", "type": "string", "required": true, "description": "Unique identifier"}, {"name": "features", "type": "object", "required": true, "description": "Input features"}, {"name": "labels", "type": "object", "required": true, "description": "Target labels"}, {"name": "annotations", "type": "object", "required": false, "description": "Annotation metadata"} ], "validation_rules": { "id_format": "must be non-empty string", "confidence_range": "must be between 0 and 1" } }, "metadata": { "created_by": "data_scientist", "purpose": "multi_modal_classification", "version": "1.0" } } ] } ================================================ FILE: skills/hugging-face-datasets/templates/qa.json ================================================ { "type": "qa", "description": "Question-answering datasets for reading comprehension, knowledge QA, and factual question answering", "system_prompt": "Generate high-quality question-answer pairs with clear, informative answers. Include context when needed and ensure questions are well-formed and answerable. Cover diverse topics and question types.", "validation_schema": { "required_fields": ["question", "answer"], "recommended_fields": ["context", "answer_type", "difficulty", "topic", "source"], "field_types": { "question": "string", "answer": "string|array", "context": "string", "answer_type": "enum:factual,explanatory,opinion,yes_no,multiple_choice", "difficulty": "enum:easy,medium,hard", "topic": "string", "source": "string" } }, "example_structure": { "question": "What is the question being asked?", "answer": "The complete answer to the question", "context": "Additional context or passage (if needed)", "answer_type": "factual|explanatory|opinion|yes_no|multiple_choice", "difficulty": "easy|medium|hard", "topic": "subject_area", "source": "where_this_came_from" }, "examples": [ { "question": "What is the capital of France?", "answer": "Paris", "answer_type": "factual", "difficulty": "easy", "topic": "geography", "source": "general_knowledge" }, { "question": "How does photosynthesis work?", "answer": "Photosynthesis is the process by which plants use sunlight, carbon dioxide, and water to produce glucose and oxygen. Chlorophyll in plant cells captures light energy, which drives chemical reactions that convert CO2 and H2O into C6H12O6 (glucose) and O2. This process occurs primarily in the chloroplasts of leaf cells.", "answer_type": "explanatory", "difficulty": "medium", "topic": "biology", "source": "educational_content" }, { "question": "Based on the passage, what was the main cause of the economic downturn?", "answer": "According to the passage, the main cause was the sudden drop in consumer confidence following the bank failures, which led to reduced spending and business investment.", "context": "The economic downturn of 2008 began when several major banks failed due to subprime mortgage losses. This triggered a crisis of confidence among consumers and businesses, leading to decreased spending and investment. The ripple effects spread throughout the global economy.", "answer_type": "explanatory", "difficulty": "medium", "topic": "economics", "source": "reading_comprehension" } ] } ================================================ FILE: skills/hugging-face-datasets/templates/tabular.json ================================================ { "type": "tabular", "description": "Structured tabular data for regression, classification, or analysis tasks", "system_prompt": "Generate structured tabular data with clear column definitions and appropriate data types. Ensure realistic value ranges and relationships between variables. Include proper metadata for each column.", "validation_schema": { "required_fields": ["data", "columns"], "recommended_fields": ["target", "metadata", "description"], "field_types": { "data": "array", "columns": "array", "target": "string", "metadata": "object", "description": "string" } }, "example_structure": { "columns": [ { "name": "column_name", "type": "numeric|categorical|text|datetime", "description": "What this column represents", "nullable": true } ], "data": [ {"column1": "value1", "column2": "value2"}, {"column1": "value3", "column2": "value4"} ], "target": "target_column_name", "metadata": { "rows": 1000, "features": 10, "task": "classification|regression|clustering" }, "description": "Description of the dataset purpose" }, "examples": [ { "columns": [ { "name": "age", "type": "numeric", "description": "Customer age in years", "nullable": false }, { "name": "income", "type": "numeric", "description": "Annual income in USD", "nullable": false }, { "name": "education", "type": "categorical", "description": "Education level", "nullable": false }, { "name": "purchased", "type": "categorical", "description": "Whether customer made a purchase", "nullable": false } ], "data": [ {"age": 25, "income": 45000, "education": "bachelor", "purchased": "yes"}, {"age": 34, "income": 67000, "education": "master", "purchased": "yes"}, {"age": 19, "income": 23000, "education": "high_school", "purchased": "no"}, {"age": 42, "income": 85000, "education": "bachelor", "purchased": "yes"}, {"age": 28, "income": 52000, "education": "bachelor", "purchased": "no"} ], "target": "purchased", "metadata": { "rows": 5, "features": 3, "task": "classification" }, "description": "Customer purchase prediction based on demographics" } ] } ================================================ FILE: skills/hugging-face-evaluation/SKILL.md ================================================ --- name: hugging-face-evaluation description: Add and manage evaluation results in Hugging Face model cards. Supports extracting eval tables from README content, importing scores from Artificial Analysis API, and running custom model evaluations with vLLM/lighteval. Works with the model-index metadata format. --- # Overview This skill provides tools to add structured evaluation results to Hugging Face model cards. It supports multiple methods for adding evaluation data: - Extracting existing evaluation tables from README content - Importing benchmark scores from Artificial Analysis - Running custom model evaluations with vLLM or accelerate backends (lighteval/inspect-ai) ## Integration with HF Ecosystem - **Model Cards**: Updates model-index metadata for leaderboard integration - **Artificial Analysis**: Direct API integration for benchmark imports - **Papers with Code**: Compatible with their model-index specification - **Jobs**: Run evaluations directly on Hugging Face Jobs with `uv` integration - **vLLM**: Efficient GPU inference for custom model evaluation - **lighteval**: HuggingFace's evaluation library with vLLM/accelerate backends - **inspect-ai**: UK AI Safety Institute's evaluation framework # Version 1.3.0 # Dependencies ## Core Dependencies - huggingface_hub>=0.26.0 - markdown-it-py>=3.0.0 - python-dotenv>=1.2.1 - pyyaml>=6.0.3 - requests>=2.32.5 - re (built-in) ## Inference Provider Evaluation - inspect-ai>=0.3.0 - inspect-evals - openai ## vLLM Custom Model Evaluation (GPU required) - lighteval[accelerate,vllm]>=0.6.0 - vllm>=0.4.0 - torch>=2.0.0 - transformers>=4.40.0 - accelerate>=0.30.0 Note: vLLM dependencies are installed automatically via PEP 723 script headers when using `uv run`. # IMPORTANT: Using This Skill ## ⚠️ CRITICAL: Check for Existing PRs Before Creating New Ones **Before creating ANY pull request with `--create-pr`, you MUST check for existing open PRs:** ```bash uv run scripts/evaluation_manager.py get-prs --repo-id "username/model-name" ``` **If open PRs exist:** 1. **DO NOT create a new PR** - this creates duplicate work for maintainers 2. **Warn the user** that open PRs already exist 3. **Show the user** the existing PR URLs so they can review them 4. Only proceed if the user explicitly confirms they want to create another PR This prevents spamming model repositories with duplicate evaluation PRs. --- > **All paths are relative to the directory containing this SKILL.md file.** > Before running any script, first `cd` to that directory or use the full path. **Use `--help` for the latest workflow guidance.** Works with plain Python or `uv run`: ```bash uv run scripts/evaluation_manager.py --help uv run scripts/evaluation_manager.py inspect-tables --help uv run scripts/evaluation_manager.py extract-readme --help ``` Key workflow (matches CLI help): 1) `get-prs` → check for existing open PRs first 2) `inspect-tables` → find table numbers/columns 3) `extract-readme --table N` → prints YAML by default 4) add `--apply` (push) or `--create-pr` to write changes # Core Capabilities ## 1. Inspect and Extract Evaluation Tables from README - **Inspect Tables**: Use `inspect-tables` to see all tables in a README with structure, columns, and sample rows - **Parse Markdown Tables**: Accurate parsing using markdown-it-py (ignores code blocks and examples) - **Table Selection**: Use `--table N` to extract from a specific table (required when multiple tables exist) - **Format Detection**: Recognize common formats (benchmarks as rows, columns, or comparison tables with multiple models) - **Column Matching**: Automatically identify model columns/rows; prefer `--model-column-index` (index from inspect output). Use `--model-name-override` only with exact column header text. - **YAML Generation**: Convert selected table to model-index YAML format - **Task Typing**: `--task-type` sets the `task.type` field in model-index output (e.g., `text-generation`, `summarization`) ## 2. Import from Artificial Analysis - **API Integration**: Fetch benchmark scores directly from Artificial Analysis - **Automatic Formatting**: Convert API responses to model-index format - **Metadata Preservation**: Maintain source attribution and URLs - **PR Creation**: Automatically create pull requests with evaluation updates ## 3. Model-Index Management - **YAML Generation**: Create properly formatted model-index entries - **Merge Support**: Add evaluations to existing model cards without overwriting - **Validation**: Ensure compliance with Papers with Code specification - **Batch Operations**: Process multiple models efficiently ## 4. Run Evaluations on HF Jobs (Inference Providers) - **Inspect-AI Integration**: Run standard evaluations using the `inspect-ai` library - **UV Integration**: Seamlessly run Python scripts with ephemeral dependencies on HF infrastructure - **Zero-Config**: No Dockerfiles or Space management required - **Hardware Selection**: Configure CPU or GPU hardware for the evaluation job - **Secure Execution**: Handles API tokens safely via secrets passed through the CLI ## 5. Run Custom Model Evaluations with vLLM (NEW) ⚠️ **Important:** This approach is only possible on devices with `uv` installed and sufficient GPU memory. **Benefits:** No need to use `hf_jobs()` MCP tool, can run scripts directly in terminal **When to use:** User working in local device directly when GPU is available ### Before running the script - check the script path - check uv is installed - check gpu is available with `nvidia-smi` ### Running the script ```bash uv run scripts/train_sft_example.py ``` ### Features - **vLLM Backend**: High-performance GPU inference (5-10x faster than standard HF methods) - **lighteval Framework**: HuggingFace's evaluation library with Open LLM Leaderboard tasks - **inspect-ai Framework**: UK AI Safety Institute's evaluation library - **Standalone or Jobs**: Run locally or submit to HF Jobs infrastructure # Usage Instructions The skill includes Python scripts in `scripts/` to perform operations. ### Prerequisites - Preferred: use `uv run` (PEP 723 header auto-installs deps) - Optional manual fallback: `uv pip install huggingface-hub markdown-it-py python-dotenv pyyaml requests` - Set `HF_TOKEN` environment variable with Write-access token - For Artificial Analysis: Set `AA_API_KEY` environment variable - `.env` is loaded automatically if `python-dotenv` is installed ### Method 1: Extract from README (CLI workflow) Recommended flow (matches `--help`): ```bash # 1) Inspect tables to get table numbers and column hints uv run scripts/evaluation_manager.py inspect-tables --repo-id "username/model" # 2) Extract a specific table (prints YAML by default) uv run scripts/evaluation_manager.py extract-readme \ --repo-id "username/model" \ --table 1 \ [--model-column-index ] \ [--model-name-override ""] # use exact header text if you can't use the index # 3) Apply changes (push or PR) uv run scripts/evaluation_manager.py extract-readme \ --repo-id "username/model" \ --table 1 \ --apply # push directly # or uv run scripts/evaluation_manager.py extract-readme \ --repo-id "username/model" \ --table 1 \ --create-pr # open a PR ``` Validation checklist: - YAML is printed by default; compare against the README table before applying. - Prefer `--model-column-index`; if using `--model-name-override`, the column header text must be exact. - For transposed tables (models as rows), ensure only one row is extracted. ### Method 2: Import from Artificial Analysis Fetch benchmark scores from Artificial Analysis API and add them to a model card. **Basic Usage:** ```bash AA_API_KEY="your-api-key" uv run scripts/evaluation_manager.py import-aa \ --creator-slug "anthropic" \ --model-name "claude-sonnet-4" \ --repo-id "username/model-name" ``` **With Environment File:** ```bash # Create .env file echo "AA_API_KEY=your-api-key" >> .env echo "HF_TOKEN=your-hf-token" >> .env # Run import uv run scripts/evaluation_manager.py import-aa \ --creator-slug "anthropic" \ --model-name "claude-sonnet-4" \ --repo-id "username/model-name" ``` **Create Pull Request:** ```bash uv run scripts/evaluation_manager.py import-aa \ --creator-slug "anthropic" \ --model-name "claude-sonnet-4" \ --repo-id "username/model-name" \ --create-pr ``` ### Method 3: Run Evaluation Job Submit an evaluation job on Hugging Face infrastructure using the `hf jobs uv run` CLI. **Direct CLI Usage:** ```bash HF_TOKEN=$HF_TOKEN \ hf jobs uv run hf-evaluation/scripts/inspect_eval_uv.py \ --flavor cpu-basic \ --secret HF_TOKEN=$HF_TOKEN \ -- --model "meta-llama/Llama-2-7b-hf" \ --task "mmlu" ``` **GPU Example (A10G):** ```bash HF_TOKEN=$HF_TOKEN \ hf jobs uv run hf-evaluation/scripts/inspect_eval_uv.py \ --flavor a10g-small \ --secret HF_TOKEN=$HF_TOKEN \ -- --model "meta-llama/Llama-2-7b-hf" \ --task "gsm8k" ``` **Python Helper (optional):** ```bash uv run scripts/run_eval_job.py \ --model "meta-llama/Llama-2-7b-hf" \ --task "mmlu" \ --hardware "t4-small" ``` ### Method 4: Run Custom Model Evaluation with vLLM Evaluate custom HuggingFace models directly on GPU using vLLM or accelerate backends. These scripts are **separate from inference provider scripts** and run models locally on the job's hardware. #### When to Use vLLM Evaluation (vs Inference Providers) | Feature | vLLM Scripts | Inference Provider Scripts | |---------|-------------|---------------------------| | Model access | Any HF model | Models with API endpoints | | Hardware | Your GPU (or HF Jobs GPU) | Provider's infrastructure | | Cost | HF Jobs compute cost | API usage fees | | Speed | vLLM optimized | Depends on provider | | Offline | Yes (after download) | No | #### Option A: lighteval with vLLM Backend lighteval is HuggingFace's evaluation library, supporting Open LLM Leaderboard tasks. **Standalone (local GPU):** ```bash # Run MMLU 5-shot with vLLM uv run scripts/lighteval_vllm_uv.py \ --model meta-llama/Llama-3.2-1B \ --tasks "leaderboard|mmlu|5" # Run multiple tasks uv run scripts/lighteval_vllm_uv.py \ --model meta-llama/Llama-3.2-1B \ --tasks "leaderboard|mmlu|5,leaderboard|gsm8k|5" # Use accelerate backend instead of vLLM uv run scripts/lighteval_vllm_uv.py \ --model meta-llama/Llama-3.2-1B \ --tasks "leaderboard|mmlu|5" \ --backend accelerate # Chat/instruction-tuned models uv run scripts/lighteval_vllm_uv.py \ --model meta-llama/Llama-3.2-1B-Instruct \ --tasks "leaderboard|mmlu|5" \ --use-chat-template ``` **Via HF Jobs:** ```bash hf jobs uv run scripts/lighteval_vllm_uv.py \ --flavor a10g-small \ --secrets HF_TOKEN=$HF_TOKEN \ -- --model meta-llama/Llama-3.2-1B \ --tasks "leaderboard|mmlu|5" ``` **lighteval Task Format:** Tasks use the format `suite|task|num_fewshot`: - `leaderboard|mmlu|5` - MMLU with 5-shot - `leaderboard|gsm8k|5` - GSM8K with 5-shot - `lighteval|hellaswag|0` - HellaSwag zero-shot - `leaderboard|arc_challenge|25` - ARC-Challenge with 25-shot **Finding Available Tasks:** The complete list of available lighteval tasks can be found at: https://github.com/huggingface/lighteval/blob/main/examples/tasks/all_tasks.txt This file contains all supported tasks in the format `suite|task|num_fewshot|0` (the trailing `0` is a version flag and can be ignored). Common suites include: - `leaderboard` - Open LLM Leaderboard tasks (MMLU, GSM8K, ARC, HellaSwag, etc.) - `lighteval` - Additional lighteval tasks - `bigbench` - BigBench tasks - `original` - Original benchmark tasks To use a task from the list, extract the `suite|task|num_fewshot` portion (without the trailing `0`) and pass it to the `--tasks` parameter. For example: - From file: `leaderboard|mmlu|0` → Use: `leaderboard|mmlu|0` (or change to `5` for 5-shot) - From file: `bigbench|abstract_narrative_understanding|0` → Use: `bigbench|abstract_narrative_understanding|0` - From file: `lighteval|wmt14:hi-en|0` → Use: `lighteval|wmt14:hi-en|0` Multiple tasks can be specified as comma-separated values: `--tasks "leaderboard|mmlu|5,leaderboard|gsm8k|5"` #### Option B: inspect-ai with vLLM Backend inspect-ai is the UK AI Safety Institute's evaluation framework. **Standalone (local GPU):** ```bash # Run MMLU with vLLM uv run scripts/inspect_vllm_uv.py \ --model meta-llama/Llama-3.2-1B \ --task mmlu # Use HuggingFace Transformers backend uv run scripts/inspect_vllm_uv.py \ --model meta-llama/Llama-3.2-1B \ --task mmlu \ --backend hf # Multi-GPU with tensor parallelism uv run scripts/inspect_vllm_uv.py \ --model meta-llama/Llama-3.2-70B \ --task mmlu \ --tensor-parallel-size 4 ``` **Via HF Jobs:** ```bash hf jobs uv run scripts/inspect_vllm_uv.py \ --flavor a10g-small \ --secrets HF_TOKEN=$HF_TOKEN \ -- --model meta-llama/Llama-3.2-1B \ --task mmlu ``` **Available inspect-ai Tasks:** - `mmlu` - Massive Multitask Language Understanding - `gsm8k` - Grade School Math - `hellaswag` - Common sense reasoning - `arc_challenge` - AI2 Reasoning Challenge - `truthfulqa` - TruthfulQA benchmark - `winogrande` - Winograd Schema Challenge - `humaneval` - Code generation #### Option C: Python Helper Script The helper script auto-selects hardware and simplifies job submission: ```bash # Auto-detect hardware based on model size uv run scripts/run_vllm_eval_job.py \ --model meta-llama/Llama-3.2-1B \ --task "leaderboard|mmlu|5" \ --framework lighteval # Explicit hardware selection uv run scripts/run_vllm_eval_job.py \ --model meta-llama/Llama-3.2-70B \ --task mmlu \ --framework inspect \ --hardware a100-large \ --tensor-parallel-size 4 # Use HF Transformers backend uv run scripts/run_vllm_eval_job.py \ --model microsoft/phi-2 \ --task mmlu \ --framework inspect \ --backend hf ``` **Hardware Recommendations:** | Model Size | Recommended Hardware | |------------|---------------------| | < 3B params | `t4-small` | | 3B - 13B | `a10g-small` | | 13B - 34B | `a10g-large` | | 34B+ | `a100-large` | ### Commands Reference **Top-level help and version:** ```bash uv run scripts/evaluation_manager.py --help uv run scripts/evaluation_manager.py --version ``` **Inspect Tables (start here):** ```bash uv run scripts/evaluation_manager.py inspect-tables --repo-id "username/model-name" ``` **Extract from README:** ```bash uv run scripts/evaluation_manager.py extract-readme \ --repo-id "username/model-name" \ --table N \ [--model-column-index N] \ [--model-name-override "Exact Column Header or Model Name"] \ [--task-type "text-generation"] \ [--dataset-name "Custom Benchmarks"] \ [--apply | --create-pr] ``` **Import from Artificial Analysis:** ```bash AA_API_KEY=... uv run scripts/evaluation_manager.py import-aa \ --creator-slug "creator-name" \ --model-name "model-slug" \ --repo-id "username/model-name" \ [--create-pr] ``` **View / Validate:** ```bash uv run scripts/evaluation_manager.py show --repo-id "username/model-name" uv run scripts/evaluation_manager.py validate --repo-id "username/model-name" ``` **Check Open PRs (ALWAYS run before --create-pr):** ```bash uv run scripts/evaluation_manager.py get-prs --repo-id "username/model-name" ``` Lists all open pull requests for the model repository. Shows PR number, title, author, date, and URL. **Run Evaluation Job (Inference Providers):** ```bash hf jobs uv run scripts/inspect_eval_uv.py \ --flavor "cpu-basic|t4-small|..." \ --secret HF_TOKEN=$HF_TOKEN \ -- --model "model-id" \ --task "task-name" ``` or use the Python helper: ```bash uv run scripts/run_eval_job.py \ --model "model-id" \ --task "task-name" \ --hardware "cpu-basic|t4-small|..." ``` **Run vLLM Evaluation (Custom Models):** ```bash # lighteval with vLLM hf jobs uv run scripts/lighteval_vllm_uv.py \ --flavor "a10g-small" \ --secrets HF_TOKEN=$HF_TOKEN \ -- --model "model-id" \ --tasks "leaderboard|mmlu|5" # inspect-ai with vLLM hf jobs uv run scripts/inspect_vllm_uv.py \ --flavor "a10g-small" \ --secrets HF_TOKEN=$HF_TOKEN \ -- --model "model-id" \ --task "mmlu" # Helper script (auto hardware selection) uv run scripts/run_vllm_eval_job.py \ --model "model-id" \ --task "leaderboard|mmlu|5" \ --framework lighteval ``` ### Model-Index Format The generated model-index follows this structure: ```yaml model-index: - name: Model Name results: - task: type: text-generation dataset: name: Benchmark Dataset type: benchmark_type metrics: - name: MMLU type: mmlu value: 85.2 - name: HumanEval type: humaneval value: 72.5 source: name: Source Name url: https://source-url.com ``` WARNING: Do not use markdown formatting in the model name. Use the exact name from the table. Only use urls in the source.url field. ### Error Handling - **Table Not Found**: Script will report if no evaluation tables are detected - **Invalid Format**: Clear error messages for malformed tables - **API Errors**: Retry logic for transient Artificial Analysis API failures - **Token Issues**: Validation before attempting updates - **Merge Conflicts**: Preserves existing model-index entries when adding new ones - **Space Creation**: Handles naming conflicts and hardware request failures gracefully ### Best Practices 1. **Check for existing PRs first**: Run `get-prs` before creating any new PR to avoid duplicates 2. **Always start with `inspect-tables`**: See table structure and get the correct extraction command 3. **Use `--help` for guidance**: Run `inspect-tables --help` to see the complete workflow 4. **Preview first**: Default behavior prints YAML; review it before using `--apply` or `--create-pr` 5. **Verify extracted values**: Compare YAML output against the README table manually 6. **Use `--table N` for multi-table READMEs**: Required when multiple evaluation tables exist 7. **Use `--model-name-override` for comparison tables**: Copy the exact column header from `inspect-tables` output 8. **Create PRs for Others**: Use `--create-pr` when updating models you don't own 9. **One model per repo**: Only add the main model's results to model-index 10. **No markdown in YAML names**: The model name field in YAML should be plain text ### Model Name Matching When extracting evaluation tables with multiple models (either as columns or rows), the script uses **exact normalized token matching**: - Removes markdown formatting (bold `**`, links `[]()` ) - Normalizes names (lowercase, replace `-` and `_` with spaces) - Compares token sets: `"OLMo-3-32B"` → `{"olmo", "3", "32b"}` matches `"**Olmo 3 32B**"` or `"[Olmo-3-32B](...)` - Only extracts if tokens match exactly (handles different word orders and separators) - Fails if no exact match found (rather than guessing from similar names) **For column-based tables** (benchmarks as rows, models as columns): - Finds the column header matching the model name - Extracts scores from that column only **For transposed tables** (models as rows, benchmarks as columns): - Finds the row in the first column matching the model name - Extracts all benchmark scores from that row only This ensures only the correct model's scores are extracted, never unrelated models or training checkpoints. ### Common Patterns **Update Your Own Model:** ```bash # Extract from README and push directly uv run scripts/evaluation_manager.py extract-readme \ --repo-id "your-username/your-model" \ --task-type "text-generation" ``` **Update Someone Else's Model (Full Workflow):** ```bash # Step 1: ALWAYS check for existing PRs first uv run scripts/evaluation_manager.py get-prs \ --repo-id "other-username/their-model" # Step 2: If NO open PRs exist, proceed with creating one uv run scripts/evaluation_manager.py extract-readme \ --repo-id "other-username/their-model" \ --create-pr # If open PRs DO exist: # - Warn the user about existing PRs # - Show them the PR URLs # - Do NOT create a new PR unless user explicitly confirms ``` **Import Fresh Benchmarks:** ```bash # Step 1: Check for existing PRs uv run scripts/evaluation_manager.py get-prs \ --repo-id "anthropic/claude-sonnet-4" # Step 2: If no PRs, import from Artificial Analysis AA_API_KEY=... uv run scripts/evaluation_manager.py import-aa \ --creator-slug "anthropic" \ --model-name "claude-sonnet-4" \ --repo-id "anthropic/claude-sonnet-4" \ --create-pr ``` ### Troubleshooting **Issue**: "No evaluation tables found in README" - **Solution**: Check if README contains markdown tables with numeric scores **Issue**: "Could not find model 'X' in transposed table" - **Solution**: The script will display available models. Use `--model-name-override` with the exact name from the list - **Example**: `--model-name-override "**Olmo 3-32B**"` **Issue**: "AA_API_KEY not set" - **Solution**: Set environment variable or add to .env file **Issue**: "Token does not have write access" - **Solution**: Ensure HF_TOKEN has write permissions for the repository **Issue**: "Model not found in Artificial Analysis" - **Solution**: Verify creator-slug and model-name match API values **Issue**: "Payment required for hardware" - **Solution**: Add a payment method to your Hugging Face account to use non-CPU hardware **Issue**: "vLLM out of memory" or CUDA OOM - **Solution**: Use a larger hardware flavor, reduce `--gpu-memory-utilization`, or use `--tensor-parallel-size` for multi-GPU **Issue**: "Model architecture not supported by vLLM" - **Solution**: Use `--backend hf` (inspect-ai) or `--backend accelerate` (lighteval) for HuggingFace Transformers **Issue**: "Trust remote code required" - **Solution**: Add `--trust-remote-code` flag for models with custom code (e.g., Phi-2, Qwen) **Issue**: "Chat template not found" - **Solution**: Only use `--use-chat-template` for instruction-tuned models that include a chat template ### Integration Examples **Python Script Integration:** ```python import subprocess import os def update_model_evaluations(repo_id, readme_content): """Update model card with evaluations from README.""" result = subprocess.run([ "python", "scripts/evaluation_manager.py", "extract-readme", "--repo-id", repo_id, "--create-pr" ], capture_output=True, text=True) if result.returncode == 0: print(f"Successfully updated {repo_id}") else: print(f"Error: {result.stderr}") ``` ================================================ FILE: skills/hugging-face-evaluation/examples/USAGE_EXAMPLES.md ================================================ # Usage Examples This document provides practical examples for both methods of adding evaluations to HuggingFace model cards. ## Table of Contents 1. [Setup](#setup) 2. [Method 1: Extract from README](#method-1-extract-from-readme) 3. [Method 2: Import from Artificial Analysis](#method-2-import-from-artificial-analysis) 4. [Standalone vs Integrated](#standalone-vs-integrated) 5. [Common Workflows](#common-workflows) ## Setup ### Initial Configuration ```bash # Navigate to skill directory cd hf_evaluation_skill # Configure environment variables cp examples/.env.example .env # Edit .env with your tokens ``` Your `.env` file should contain: ```env HF_TOKEN=hf_your_write_token_here AA_API_KEY=aa_your_api_key_here # Optional for AA imports ``` ### Verify Installation ```bash uv run scripts/test_extraction.py ``` ## Method 1: Extract from README Extract evaluation tables from your model's existing README. ### Basic Extraction ```bash # Preview what will be extracted (dry run) uv run scripts/evaluation_manager.py extract-readme \ --repo-id "meta-llama/Llama-3.3-70B-Instruct" \ --dry-run ``` ### Apply Extraction to Your Model ```bash # Extract and update model card directly uv run scripts/evaluation_manager.py extract-readme \ --repo-id "your-username/your-model-7b" ``` ### Custom Task and Dataset Names ```bash uv run scripts/evaluation_manager.py extract-readme \ --repo-id "your-username/your-model-7b" \ --task-type "text-generation" \ --dataset-name "Standard Benchmarks" \ --dataset-type "llm_benchmarks" ``` ### Create Pull Request (for models you don't own) ```bash uv run scripts/evaluation_manager.py extract-readme \ --repo-id "organization/community-model" \ --create-pr ``` ### Example README Format Your model README should contain tables like: ```markdown ## Evaluation Results | Benchmark | Score | |---------------|-------| | MMLU | 85.2 | | HumanEval | 72.5 | | GSM8K | 91.3 | | HellaSwag | 88.9 | ``` ## Method 2: Import from Artificial Analysis Fetch benchmark scores directly from Artificial Analysis API. ### Integrated Approach (Recommended) ```bash # Import scores for Claude Sonnet 4.5 uv run scripts/evaluation_manager.py import-aa \ --creator-slug "anthropic" \ --model-name "claude-sonnet-4" \ --repo-id "your-username/claude-mirror" ``` ### With Pull Request ```bash # Create PR instead of direct commit uv run scripts/evaluation_manager.py import-aa \ --creator-slug "openai" \ --model-name "gpt-4" \ --repo-id "your-username/gpt-4-mirror" \ --create-pr ``` ### Standalone Script For simple, one-off imports, use the standalone script: ```bash # Navigate to examples directory cd examples # Run standalone script AA_API_KEY="your-key" HF_TOKEN="your-token" \ uv run artificial_analysis_to_hub.py \ --creator-slug "anthropic" \ --model-name "claude-sonnet-4" \ --repo-id "your-username/your-repo" ``` ### Finding Creator Slug and Model Name 1. Visit [Artificial Analysis](https://artificialanalysis.ai/) 2. Navigate to the model you want to import 3. The URL format is: `https://artificialanalysis.ai/models/{creator-slug}/{model-name}` 4. Or check their [API documentation](https://artificialanalysis.ai/api) Common examples: - Anthropic: `--creator-slug "anthropic" --model-name "claude-sonnet-4"` - OpenAI: `--creator-slug "openai" --model-name "gpt-4-turbo"` - Meta: `--creator-slug "meta" --model-name "llama-3-70b"` ## Standalone vs Integrated ### Standalone Script Features - ✓ Simple, single-purpose - ✓ Can run via `uv run` from URL - ✓ Minimal dependencies - ✗ No README extraction - ✗ No validation - ✗ No dry-run mode **Use when:** You only need AA imports and want a simple script. ### Integrated Script Features - ✓ Both README extraction AND AA import - ✓ Validation and show commands - ✓ Dry-run preview mode - ✓ Better error handling - ✓ Merge with existing evaluations - ✓ More flexible options **Use when:** You want full evaluation management capabilities. ## Common Workflows ### Workflow 1: New Model with README Tables You've just created a model with evaluation tables in the README. ```bash # Step 1: Preview extraction uv run scripts/evaluation_manager.py extract-readme \ --repo-id "your-username/new-model-7b" \ --dry-run # Step 2: Apply if it looks good uv run scripts/evaluation_manager.py extract-readme \ --repo-id "your-username/new-model-7b" # Step 3: Validate uv run scripts/evaluation_manager.py validate \ --repo-id "your-username/new-model-7b" # Step 4: View results uv run scripts/evaluation_manager.py show \ --repo-id "your-username/new-model-7b" ``` ### Workflow 2: Model Benchmarked on AA Your model appears on Artificial Analysis with fresh benchmarks. ```bash # Import scores and create PR for review uv run scripts/evaluation_manager.py import-aa \ --creator-slug "your-org" \ --model-name "your-model" \ --repo-id "your-org/your-model-hf" \ --create-pr ``` ### Workflow 3: Combine Both Methods You have README tables AND AA scores. ```bash # Step 1: Extract from README uv run scripts/evaluation_manager.py extract-readme \ --repo-id "your-username/hybrid-model" # Step 2: Import from AA (will merge with existing) uv run scripts/evaluation_manager.py import-aa \ --creator-slug "your-org" \ --model-name "hybrid-model" \ --repo-id "your-username/hybrid-model" # Step 3: View combined results uv run scripts/evaluation_manager.py show \ --repo-id "your-username/hybrid-model" ``` ### Workflow 4: Contributing to Community Models Help improve community models by adding missing evaluations. ```bash # Find a model with evaluations in README but no model-index # Example: community/awesome-7b # Create PR with extracted evaluations uv run scripts/evaluation_manager.py extract-readme \ --repo-id "community/awesome-7b" \ --create-pr # GitHub will notify the repository owner # They can review and merge your PR ``` ### Workflow 5: Batch Processing Update multiple models at once. ```bash # Create a list of repos cat > models.txt << EOF your-org/model-1-7b your-org/model-2-13b your-org/model-3-70b EOF # Process each while read repo_id; do echo "Processing $repo_id..." uv run scripts/evaluation_manager.py extract-readme \ --repo-id "$repo_id" done < models.txt ``` ### Workflow 6: Automated Updates (CI/CD) Set up automatic evaluation updates using GitHub Actions. ```yaml # .github/workflows/update-evals.yml name: Update Evaluations Weekly on: schedule: - cron: '0 0 * * 0' # Every Sunday workflow_dispatch: # Manual trigger jobs: update: runs-on: ubuntu-latest steps: - uses: actions/checkout@v4 - name: Set up uv uses: astral-sh/setup-uv@v5 - name: Set up Python uses: actions/setup-python@v5 with: python-version: '3.13' - name: Update from Artificial Analysis env: AA_API_KEY: ${{ secrets.AA_API_KEY }} HF_TOKEN: ${{ secrets.HF_TOKEN }} run: | uv run scripts/evaluation_manager.py import-aa \ --creator-slug "${{ vars.AA_CREATOR_SLUG }}" \ --model-name "${{ vars.AA_MODEL_NAME }}" \ --repo-id "${{ github.repository }}" \ --create-pr ``` ## Verification and Validation ### Check Current Evaluations ```bash uv run scripts/evaluation_manager.py show \ --repo-id "your-username/your-model" ``` ### Validate Format ```bash uv run scripts/evaluation_manager.py validate \ --repo-id "your-username/your-model" ``` ### View in HuggingFace UI After updating, visit: ``` https://huggingface.co/your-username/your-model ``` The evaluation widget should display your scores automatically. ## Troubleshooting Examples ### Problem: No tables found ```bash # Check what tables exist in your README uv run scripts/evaluation_manager.py extract-readme \ --repo-id "your-username/your-model" \ --dry-run # If no output, ensure your README has markdown tables with numeric scores ``` ### Problem: AA model not found ```bash # Verify the creator and model slugs # Check the AA website URL or API directly curl -H "x-api-key: $AA_API_KEY" \ https://artificialanalysis.ai/api/v2/data/llms/models | jq ``` ### Problem: Token permission error ```bash # Verify your token has write access # Generate a new token at: https://huggingface.co/settings/tokens # Ensure "Write" scope is enabled ``` ## Tips and Best Practices 1. **Always dry-run first**: Use `--dry-run` to preview changes 2. **Use PRs for others' repos**: Always use `--create-pr` for repositories you don't own 3. **Validate after updates**: Run `validate` to ensure proper formatting 4. **Keep evaluations current**: Set up automated updates for AA scores 5. **Document sources**: The tool automatically adds source attribution 6. **Check the UI**: Always verify the evaluation widget displays correctly ## Getting Help ```bash # General help uv run scripts/evaluation_manager.py --help # Command-specific help uv run scripts/evaluation_manager.py extract-readme --help uv run scripts/evaluation_manager.py import-aa --help ``` For issues or questions, consult: - `../SKILL.md` - Complete documentation - `../README.md` - Troubleshooting guide - `../QUICKSTART.md` - Quick start guide ================================================ FILE: skills/hugging-face-evaluation/examples/artificial_analysis_to_hub.py ================================================ # /// script # requires-python = ">=3.13" # dependencies = [ # "huggingface-hub>=1.1.4", # "python-dotenv>=1.2.1", # "pyyaml>=6.0.3", # "requests>=2.32.5", # ] # /// """ Add Artificial Analysis evaluations to a Hugging Face model card. NOTE: This is a standalone reference script. For integrated functionality with additional features (README extraction, validation, etc.), use: ../scripts/evaluation_manager.py import-aa [options] STANDALONE USAGE: AA_API_KEY="" HF_TOKEN="" \ uv run artificial_analysis_to_hub.py \ --creator-slug \ --model-name \ --repo-id INTEGRATED USAGE (Recommended): uv run ../scripts/evaluation_manager.py import-aa \ --creator-slug \ --model-name \ --repo-id \ [--create-pr] """ import argparse import os import requests import dotenv from huggingface_hub import ModelCard dotenv.load_dotenv() API_KEY = os.getenv("AA_API_KEY") HF_TOKEN = os.getenv("HF_TOKEN") URL = "https://artificialanalysis.ai/api/v2/data/llms/models" HEADERS = {"x-api-key": API_KEY} if not API_KEY: raise ValueError("AA_API_KEY is not set") if not HF_TOKEN: raise ValueError("HF_TOKEN is not set") def get_model_evaluations_data(creator_slug, model_name): response = requests.get(URL, headers=HEADERS) response_data = response.json()["data"] for model in response_data: if ( model["model_creator"]["slug"] == creator_slug and model["slug"] == model_name ): return model raise ValueError(f"Model {model_name} not found") def aa_evaluations_to_model_index( model, dataset_name="Artificial Analysis Benchmarks", dataset_type="artificial_analysis", task_type="evaluation", ): if not model: raise ValueError("Model data is required") model_name = model.get("name", model.get("slug", "unknown-model")) evaluations = model.get("evaluations", {}) metrics = [] for key, value in evaluations.items(): metrics.append( { "name": key.replace("_", " ").title(), "type": key, "value": value, } ) model_index = [ { "name": model_name, "results": [ { "task": {"type": task_type}, "dataset": {"name": dataset_name, "type": dataset_type}, "metrics": metrics, "source": { "name": "Artificial Analysis API", "url": "https://artificialanalysis.ai", }, } ], } ] return model_index def main(): parser = argparse.ArgumentParser() parser.add_argument("--creator-slug", type=str, required=True) parser.add_argument("--model-name", type=str, required=True) parser.add_argument("--repo-id", type=str, required=True) args = parser.parse_args() aa_evaluations_data = get_model_evaluations_data( creator_slug=args.creator_slug, model_name=args.model_name ) model_index = aa_evaluations_to_model_index(model=aa_evaluations_data) card = ModelCard.load(args.repo_id) card.data["model-index"] = model_index commit_message = ( f"Add Artificial Analysis evaluations for {args.model_name}" ) commit_description = ( f"This commit adds the Artificial Analysis evaluations for the {args.model_name} model to this repository. " "To see the scores, visit the [Artificial Analysis](https://artificialanalysis.ai) website." ) card.push_to_hub( args.repo_id, token=HF_TOKEN, commit_message=commit_message, commit_description=commit_description, create_pr=True, ) if __name__ == "__main__": main() ================================================ FILE: skills/hugging-face-evaluation/examples/example_readme_tables.md ================================================ # Example Evaluation Table Formats This file shows various formats of evaluation tables that can be extracted from model README files. ## Format 1: Benchmarks as Rows (Most Common) ```markdown | Benchmark | Score | |-----------|-------| | MMLU | 85.2 | | HumanEval | 72.5 | | GSM8K | 91.3 | | HellaSwag | 88.9 | ``` ## Format 2: Multiple Metric Columns ```markdown | Benchmark | Accuracy | F1 Score | |-----------|----------|----------| | MMLU | 85.2 | 0.84 | | GSM8K | 91.3 | 0.91 | | DROP | 78.5 | 0.77 | ``` ## Format 3: Benchmarks as Columns ```markdown | MMLU | HumanEval | GSM8K | HellaSwag | |------|-----------|-------|-----------| | 85.2 | 72.5 | 91.3 | 88.9 | ``` ## Format 4: Percentage Values ```markdown | Benchmark | Score | |---------------|----------| | MMLU | 85.2% | | HumanEval | 72.5% | | GSM8K | 91.3% | | TruthfulQA | 68.7% | ``` ## Format 5: Mixed Format with Categories ```markdown ### Reasoning | Benchmark | Score | |-----------|-------| | MMLU | 85.2 | | BBH | 82.4 | | GPQA | 71.3 | ### Coding | Benchmark | Score | |-----------|-------| | HumanEval | 72.5 | | MBPP | 78.9 | ### Math | Benchmark | Score | |-----------|-------| | GSM8K | 91.3 | | MATH | 65.8 | ``` ## Format 6: With Additional Columns ```markdown | Benchmark | Score | Rank | Notes | |-----------|-------|------|--------------------| | MMLU | 85.2 | #5 | 5-shot | | HumanEval | 72.5 | #8 | pass@1 | | GSM8K | 91.3 | #3 | 8-shot, maj@1 | ``` ## How the Extractor Works The script will: 1. Find all markdown tables in the README 2. Identify which tables contain evaluation results 3. Parse the table structure (rows vs columns) 4. Extract numeric values as scores 5. Convert to model-index YAML format ## Tips for README Authors To ensure your evaluation tables are properly extracted: 1. **Use clear headers**: Include "Benchmark", "Score", or similar terms 2. **Keep it simple**: Stick to benchmark name + score columns 3. **Use standard formats**: Follow markdown table syntax 4. **Include numeric values**: Ensure scores are parseable numbers 5. **Be consistent**: Use the same format across multiple tables ## Example Complete README Section ```markdown # Model Card for MyModel-7B ## Evaluation Results Our model was evaluated on several standard benchmarks: | Benchmark | Score | |---------------|-------| | MMLU | 85.2 | | HumanEval | 72.5 | | GSM8K | 91.3 | | HellaSwag | 88.9 | | ARC-Challenge | 81.7 | | TruthfulQA | 68.7 | ### Detailed Results For more detailed results and methodology, see our [paper](link). ``` ## Running the Extractor ```bash # Extract from this example uv run scripts/evaluation_manager.py extract-readme \ --repo-id "your-username/your-model" \ --dry-run # Apply to your model card uv run scripts/evaluation_manager.py extract-readme \ --repo-id "your-username/your-model" \ --task-type "text-generation" ``` ================================================ FILE: skills/hugging-face-evaluation/examples/metric_mapping.json ================================================ { "MMLU": { "type": "mmlu", "name": "Massive Multitask Language Understanding" }, "HumanEval": { "type": "humaneval", "name": "Code Generation (HumanEval)" }, "GSM8K": { "type": "gsm8k", "name": "Grade School Math" }, "HellaSwag": { "type": "hellaswag", "name": "HellaSwag Common Sense" }, "ARC-C": { "type": "arc_challenge", "name": "ARC Challenge" }, "ARC-E": { "type": "arc_easy", "name": "ARC Easy" }, "Winogrande": { "type": "winogrande", "name": "Winogrande" }, "TruthfulQA": { "type": "truthfulqa", "name": "TruthfulQA" }, "GPQA": { "type": "gpqa", "name": "Graduate-Level Google-Proof Q&A" }, "DROP": { "type": "drop", "name": "Discrete Reasoning Over Paragraphs" }, "BBH": { "type": "bbh", "name": "Big Bench Hard" }, "MATH": { "type": "math", "name": "MATH Dataset" } } ================================================ FILE: skills/hugging-face-evaluation/scripts/evaluation_manager.py ================================================ # /// script # requires-python = ">=3.13" # dependencies = [ # "huggingface-hub>=1.1.4", # "markdown-it-py>=3.0.0", # "python-dotenv>=1.2.1", # "pyyaml>=6.0.3", # "requests>=2.32.5", # ] # /// """ Manage evaluation results in Hugging Face model cards. This script provides two methods: 1. Extract evaluation tables from model README files 2. Import evaluation scores from Artificial Analysis API Both methods update the model-index metadata in model cards. """ import argparse import os import re from textwrap import dedent from typing import Any, Dict, List, Optional, Tuple def load_env() -> None: """Load .env if python-dotenv is available; keep help usable without it.""" try: import dotenv # type: ignore except ModuleNotFoundError: return dotenv.load_dotenv() def require_markdown_it(): try: from markdown_it import MarkdownIt # type: ignore except ModuleNotFoundError as exc: raise ModuleNotFoundError( "markdown-it-py is required for table parsing. " "Run with `uv run ...` or install with `uv pip install markdown-it-py`." ) from exc return MarkdownIt def require_model_card(): try: from huggingface_hub import ModelCard # type: ignore except ModuleNotFoundError as exc: raise ModuleNotFoundError( "huggingface-hub is required for model card operations. " "Run with `uv run ...` or install with `uv pip install huggingface-hub`." ) from exc return ModelCard def require_requests(): try: import requests # type: ignore except ModuleNotFoundError as exc: raise ModuleNotFoundError( "requests is required for Artificial Analysis import. " "Run with `uv run ...` or install with `uv pip install requests`." ) from exc return requests def require_yaml(): try: import yaml # type: ignore except ModuleNotFoundError as exc: raise ModuleNotFoundError( "PyYAML is required for YAML output. " "Run with `uv run ...` or install with `uv pip install pyyaml`." ) from exc return yaml # ============================================================================ # Method 1: Extract Evaluations from README # ============================================================================ def extract_tables_from_markdown(markdown_content: str) -> List[str]: """Extract all markdown tables from content.""" # Pattern to match markdown tables table_pattern = r"(\|[^\n]+\|(?:\r?\n\|[^\n]+\|)+)" tables = re.findall(table_pattern, markdown_content) return tables def parse_markdown_table(table_str: str) -> Tuple[List[str], List[List[str]]]: """ Parse a markdown table string into headers and rows. Returns: Tuple of (headers, data_rows) """ lines = [line.strip() for line in table_str.strip().split("\n")] # Remove separator line (the one with dashes) lines = [line for line in lines if not re.match(r"^\|[\s\-:]+\|$", line)] if len(lines) < 2: return [], [] # Parse header header = [cell.strip() for cell in lines[0].split("|")[1:-1]] # Parse data rows data_rows = [] for line in lines[1:]: cells = [cell.strip() for cell in line.split("|")[1:-1]] if cells: data_rows.append(cells) return header, data_rows def is_evaluation_table(header: List[str], rows: List[List[str]]) -> bool: """Determine if a table contains evaluation results.""" if not header or not rows: return False # Check if first column looks like benchmark names benchmark_keywords = [ "benchmark", "task", "dataset", "eval", "test", "metric", "mmlu", "humaneval", "gsm", "hellaswag", "arc", "winogrande", "truthfulqa", "boolq", "piqa", "siqa" ] first_col = header[0].lower() has_benchmark_header = any(keyword in first_col for keyword in benchmark_keywords) # Check if there are numeric values in the table has_numeric_values = False for row in rows: for cell in row: try: float(cell.replace("%", "").replace(",", "")) has_numeric_values = True break except ValueError: continue if has_numeric_values: break return has_benchmark_header or has_numeric_values def normalize_model_name(name: str) -> tuple[set[str], str]: """ Normalize a model name for matching. Args: name: Model name to normalize Returns: Tuple of (token_set, normalized_string) """ # Remove markdown formatting cleaned = re.sub(r'\[([^\]]+)\]\([^\)]+\)', r'\1', name) # Remove markdown links cleaned = re.sub(r'\*\*([^\*]+)\*\*', r'\1', cleaned) # Remove bold cleaned = cleaned.strip() # Normalize and tokenize normalized = cleaned.lower().replace("-", " ").replace("_", " ") tokens = set(normalized.split()) return tokens, normalized def find_main_model_column(header: List[str], model_name: str) -> Optional[int]: """ Identify the column index that corresponds to the main model. Only returns a column if there's an exact normalized match with the model name. This prevents extracting scores from training checkpoints or similar models. Args: header: Table column headers model_name: Model name from repo_id (e.g., "OLMo-3-32B-Think") Returns: Column index of the main model, or None if no exact match found """ if not header or not model_name: return None # Normalize model name and extract tokens model_tokens, _ = normalize_model_name(model_name) # Find exact matches only for i, col_name in enumerate(header): if not col_name: continue # Skip first column (benchmark names) if i == 0: continue col_tokens, _ = normalize_model_name(col_name) # Check for exact token match if model_tokens == col_tokens: return i # No exact match found return None def find_main_model_row( rows: List[List[str]], model_name: str ) -> tuple[Optional[int], List[str]]: """ Identify the row index that corresponds to the main model in a transposed table. In transposed tables, each row represents a different model, with the first column containing the model name. Args: rows: Table data rows model_name: Model name from repo_id (e.g., "OLMo-3-32B") Returns: Tuple of (row_index, available_models) - row_index: Index of the main model, or None if no exact match found - available_models: List of all model names found in the table """ if not rows or not model_name: return None, [] model_tokens, _ = normalize_model_name(model_name) available_models = [] for i, row in enumerate(rows): if not row or not row[0]: continue row_name = row[0].strip() # Skip separator/header rows if not row_name or row_name.startswith('---'): continue row_tokens, _ = normalize_model_name(row_name) # Collect all non-empty model names if row_tokens: available_models.append(row_name) # Check for exact token match if model_tokens == row_tokens: return i, available_models return None, available_models def is_transposed_table(header: List[str], rows: List[List[str]]) -> bool: """ Determine if a table is transposed (models as rows, benchmarks as columns). A table is considered transposed if: - The first column contains model-like names (not benchmark names) - Most other columns contain numeric values - Header row contains benchmark-like names Args: header: Table column headers rows: Table data rows Returns: True if table appears to be transposed, False otherwise """ if not header or not rows or len(header) < 3: return False # Check if first column header suggests model names first_col = header[0].lower() model_indicators = ["model", "system", "llm", "name"] has_model_header = any(indicator in first_col for indicator in model_indicators) # Check if remaining headers look like benchmarks benchmark_keywords = [ "mmlu", "humaneval", "gsm", "hellaswag", "arc", "winogrande", "eval", "score", "benchmark", "test", "math", "code", "mbpp", "truthfulqa", "boolq", "piqa", "siqa", "drop", "squad" ] benchmark_header_count = 0 for col_name in header[1:]: col_lower = col_name.lower() if any(keyword in col_lower for keyword in benchmark_keywords): benchmark_header_count += 1 has_benchmark_headers = benchmark_header_count >= 2 # Check if data rows have numeric values in most columns (except first) numeric_count = 0 total_cells = 0 for row in rows[:5]: # Check first 5 rows for cell in row[1:]: # Skip first column total_cells += 1 try: float(cell.replace("%", "").replace(",", "").strip()) numeric_count += 1 except (ValueError, AttributeError): continue has_numeric_data = total_cells > 0 and (numeric_count / total_cells) > 0.5 return (has_model_header or has_benchmark_headers) and has_numeric_data def extract_metrics_from_table( header: List[str], rows: List[List[str]], table_format: str = "auto", model_name: Optional[str] = None, model_column_index: Optional[int] = None ) -> List[Dict[str, Any]]: """ Extract metrics from parsed table data. Args: header: Table column headers rows: Table data rows table_format: "rows" (benchmarks as rows), "columns" (benchmarks as columns), "transposed" (models as rows, benchmarks as columns), or "auto" model_name: Optional model name to identify the correct column/row Returns: List of metric dictionaries with name, type, and value """ metrics = [] if table_format == "auto": # First check if it's a transposed table (models as rows) if is_transposed_table(header, rows): table_format = "transposed" else: # Check if first column header is empty/generic (indicates benchmarks in rows) first_header = header[0].lower().strip() if header else "" is_first_col_benchmarks = not first_header or first_header in ["", "benchmark", "task", "dataset", "metric", "eval"] if is_first_col_benchmarks: table_format = "rows" else: # Heuristic: if first row has mostly numeric values, benchmarks are columns try: numeric_count = sum( 1 for cell in rows[0] if cell and re.match(r"^\d+\.?\d*%?$", cell.replace(",", "").strip()) ) table_format = "columns" if numeric_count > len(rows[0]) / 2 else "rows" except (IndexError, ValueError): table_format = "rows" if table_format == "rows": # Benchmarks are in rows, scores in columns # Try to identify the main model column if model_name is provided target_column = model_column_index if target_column is None and model_name: target_column = find_main_model_column(header, model_name) for row in rows: if not row: continue benchmark_name = row[0].strip() if not benchmark_name: continue # If we identified a specific column, use it; otherwise use first numeric value if target_column is not None and target_column < len(row): try: value_str = row[target_column].replace("%", "").replace(",", "").strip() if value_str: value = float(value_str) metrics.append({ "name": benchmark_name, "type": benchmark_name.lower().replace(" ", "_"), "value": value }) except (ValueError, IndexError): pass else: # Extract numeric values from remaining columns (original behavior) for i, cell in enumerate(row[1:], start=1): try: # Remove common suffixes and convert to float value_str = cell.replace("%", "").replace(",", "").strip() if not value_str: continue value = float(value_str) # Determine metric name metric_name = benchmark_name if len(header) > i and header[i].lower() not in ["score", "value", "result"]: metric_name = f"{benchmark_name} ({header[i]})" metrics.append({ "name": metric_name, "type": benchmark_name.lower().replace(" ", "_"), "value": value }) break # Only take first numeric value per row except (ValueError, IndexError): continue elif table_format == "transposed": # Models are in rows (first column), benchmarks are in columns (header) # Find the row that matches the target model if not model_name: print("Warning: model_name required for transposed table format") return metrics target_row_idx, available_models = find_main_model_row(rows, model_name) if target_row_idx is None: print(f"\n⚠ Could not find model '{model_name}' in transposed table") if available_models: print("\nAvailable models in table:") for i, model in enumerate(available_models, 1): print(f" {i}. {model}") print("\nPlease select the correct model name from the list above.") print("You can specify it using the --model-name-override flag:") print(f' --model-name-override "{available_models[0]}"') return metrics target_row = rows[target_row_idx] # Extract metrics from each column (skip first column which is model name) for i in range(1, len(header)): benchmark_name = header[i].strip() if not benchmark_name or i >= len(target_row): continue try: value_str = target_row[i].replace("%", "").replace(",", "").strip() if not value_str: continue value = float(value_str) metrics.append({ "name": benchmark_name, "type": benchmark_name.lower().replace(" ", "_").replace("-", "_"), "value": value }) except (ValueError, AttributeError): continue else: # table_format == "columns" # Benchmarks are in columns if not rows: return metrics # Use first data row for values data_row = rows[0] for i, benchmark_name in enumerate(header): if not benchmark_name or i >= len(data_row): continue try: value_str = data_row[i].replace("%", "").replace(",", "").strip() if not value_str: continue value = float(value_str) metrics.append({ "name": benchmark_name, "type": benchmark_name.lower().replace(" ", "_"), "value": value }) except ValueError: continue return metrics def extract_evaluations_from_readme( repo_id: str, task_type: str = "text-generation", dataset_name: str = "Benchmarks", dataset_type: str = "benchmark", model_name_override: Optional[str] = None, table_index: Optional[int] = None, model_column_index: Optional[int] = None ) -> Optional[List[Dict[str, Any]]]: """ Extract evaluation results from a model's README. Args: repo_id: Hugging Face model repository ID task_type: Task type for model-index (e.g., "text-generation") dataset_name: Name for the benchmark dataset dataset_type: Type identifier for the dataset model_name_override: Override model name for matching (column header for comparison tables) table_index: 1-indexed table number from inspect-tables output Returns: Model-index formatted results or None if no evaluations found """ try: load_env() ModelCard = require_model_card() hf_token = os.getenv("HF_TOKEN") card = ModelCard.load(repo_id, token=hf_token) readme_content = card.content if not readme_content: print(f"No README content found for {repo_id}") return None # Extract model name from repo_id or use override if model_name_override: model_name = model_name_override print(f"Using model name override: '{model_name}'") else: model_name = repo_id.split("/")[-1] if "/" in repo_id else repo_id # Use markdown-it parser for accurate table extraction all_tables = extract_tables_with_parser(readme_content) if not all_tables: print(f"No tables found in README for {repo_id}") return None # If table_index specified, use that specific table if table_index is not None: if table_index < 1 or table_index > len(all_tables): print(f"Invalid table index {table_index}. Found {len(all_tables)} tables.") print("Run inspect-tables to see available tables.") return None tables_to_process = [all_tables[table_index - 1]] else: # Filter to evaluation tables only eval_tables = [] for table in all_tables: header = table.get("headers", []) rows = table.get("rows", []) if is_evaluation_table(header, rows): eval_tables.append(table) if len(eval_tables) > 1: print(f"\n⚠ Found {len(eval_tables)} evaluation tables.") print("Run inspect-tables first, then use --table to select one:") print(f' uv run scripts/evaluation_manager.py inspect-tables --repo-id "{repo_id}"') return None elif len(eval_tables) == 0: print(f"No evaluation tables found in README for {repo_id}") return None tables_to_process = eval_tables # Extract metrics from selected table(s) all_metrics = [] for table in tables_to_process: header = table.get("headers", []) rows = table.get("rows", []) metrics = extract_metrics_from_table( header, rows, model_name=model_name, model_column_index=model_column_index ) all_metrics.extend(metrics) if not all_metrics: print(f"No metrics extracted from table") return None # Build model-index structure display_name = repo_id.split("/")[-1] if "/" in repo_id else repo_id results = [{ "task": {"type": task_type}, "dataset": { "name": dataset_name, "type": dataset_type }, "metrics": all_metrics, "source": { "name": "Model README", "url": f"https://huggingface.co/{repo_id}" } }] return results except Exception as e: print(f"Error extracting evaluations from README: {e}") return None # ============================================================================ # Table Inspection (using markdown-it-py for accurate parsing) # ============================================================================ def extract_tables_with_parser(markdown_content: str) -> List[Dict[str, Any]]: """ Extract tables from markdown using markdown-it-py parser. Uses GFM (GitHub Flavored Markdown) which includes table support. """ MarkdownIt = require_markdown_it() # Disable linkify to avoid optional dependency errors; not needed for table parsing. md = MarkdownIt("gfm-like", {"linkify": False}) tokens = md.parse(markdown_content) tables = [] i = 0 while i < len(tokens): token = tokens[i] if token.type == "table_open": table_data = {"headers": [], "rows": []} current_row = [] in_header = False i += 1 while i < len(tokens) and tokens[i].type != "table_close": t = tokens[i] if t.type == "thead_open": in_header = True elif t.type == "thead_close": in_header = False elif t.type == "tr_open": current_row = [] elif t.type == "tr_close": if in_header: table_data["headers"] = current_row else: table_data["rows"].append(current_row) current_row = [] elif t.type == "inline": current_row.append(t.content.strip()) i += 1 if table_data["headers"] or table_data["rows"]: tables.append(table_data) i += 1 return tables def detect_table_format(table: Dict[str, Any], repo_id: str) -> Dict[str, Any]: """Analyze a table to detect its format and identify model columns.""" headers = table.get("headers", []) rows = table.get("rows", []) if not headers or not rows: return {"format": "unknown", "columns": headers, "model_columns": [], "row_count": 0, "sample_rows": []} first_header = headers[0].lower() if headers else "" is_first_col_benchmarks = not first_header or first_header in ["", "benchmark", "task", "dataset", "metric", "eval"] # Check for numeric columns numeric_columns = [] for col_idx in range(1, len(headers)): numeric_count = 0 for row in rows[:5]: if col_idx < len(row): try: val = re.sub(r'\s*\([^)]*\)', '', row[col_idx]) float(val.replace("%", "").replace(",", "").strip()) numeric_count += 1 except (ValueError, AttributeError): pass if numeric_count > len(rows[:5]) / 2: numeric_columns.append(col_idx) # Determine format if is_first_col_benchmarks and len(numeric_columns) > 1: format_type = "comparison" elif is_first_col_benchmarks and len(numeric_columns) == 1: format_type = "simple" elif len(numeric_columns) > len(headers) / 2: format_type = "transposed" else: format_type = "unknown" # Find model columns model_columns = [] model_name = repo_id.split("/")[-1] if "/" in repo_id else repo_id model_tokens, _ = normalize_model_name(model_name) for idx, header in enumerate(headers): if idx == 0 and is_first_col_benchmarks: continue if header: header_tokens, _ = normalize_model_name(header) is_match = model_tokens == header_tokens is_partial = model_tokens.issubset(header_tokens) or header_tokens.issubset(model_tokens) model_columns.append({ "index": idx, "header": header, "is_exact_match": is_match, "is_partial_match": is_partial and not is_match }) return { "format": format_type, "columns": headers, "model_columns": model_columns, "row_count": len(rows), "sample_rows": [row[0] for row in rows[:5] if row] } def inspect_tables(repo_id: str) -> None: """Inspect and display all evaluation tables in a model's README.""" try: load_env() ModelCard = require_model_card() hf_token = os.getenv("HF_TOKEN") card = ModelCard.load(repo_id, token=hf_token) readme_content = card.content if not readme_content: print(f"No README content found for {repo_id}") return tables = extract_tables_with_parser(readme_content) if not tables: print(f"No tables found in README for {repo_id}") return print(f"\n{'='*70}") print(f"Tables found in README for: {repo_id}") print(f"{'='*70}") eval_table_count = 0 for table in tables: analysis = detect_table_format(table, repo_id) if analysis["format"] == "unknown" and not analysis.get("sample_rows"): continue eval_table_count += 1 print(f"\n## Table {eval_table_count}") print(f" Format: {analysis['format']}") print(f" Rows: {analysis['row_count']}") print(f"\n Columns ({len(analysis['columns'])}):") for col_info in analysis.get("model_columns", []): idx = col_info["index"] header = col_info["header"] if col_info["is_exact_match"]: print(f" [{idx}] {header} ✓ EXACT MATCH") elif col_info["is_partial_match"]: print(f" [{idx}] {header} ~ partial match") else: print(f" [{idx}] {header}") if analysis.get("sample_rows"): print(f"\n Sample rows (first column):") for row_val in analysis["sample_rows"][:5]: print(f" - {row_val}") if eval_table_count == 0: print("\nNo evaluation tables detected.") else: print("\nSuggested next step:") print(f' uv run scripts/evaluation_manager.py extract-readme --repo-id "{repo_id}" --table [--model-column-index ]') print(f"\n{'='*70}\n") except Exception as e: print(f"Error inspecting tables: {e}") # ============================================================================ # Pull Request Management # ============================================================================ def get_open_prs(repo_id: str) -> List[Dict[str, Any]]: """ Fetch open pull requests for a Hugging Face model repository. Args: repo_id: Hugging Face model repository ID (e.g., "allenai/Olmo-3-32B-Think") Returns: List of open PR dictionaries with num, title, author, and createdAt """ requests = require_requests() url = f"https://huggingface.co/api/models/{repo_id}/discussions" try: response = requests.get(url, timeout=30, allow_redirects=True) response.raise_for_status() data = response.json() discussions = data.get("discussions", []) open_prs = [ { "num": d["num"], "title": d["title"], "author": d["author"]["name"], "createdAt": d.get("createdAt", "unknown"), } for d in discussions if d.get("status") == "open" and d.get("isPullRequest") ] return open_prs except requests.RequestException as e: print(f"Error fetching PRs from Hugging Face: {e}") return [] def list_open_prs(repo_id: str) -> None: """Display open pull requests for a model repository.""" prs = get_open_prs(repo_id) print(f"\n{'='*70}") print(f"Open Pull Requests for: {repo_id}") print(f"{'='*70}") if not prs: print("\nNo open pull requests found.") else: print(f"\nFound {len(prs)} open PR(s):\n") for pr in prs: print(f" PR #{pr['num']} - {pr['title']}") print(f" Author: {pr['author']}") print(f" Created: {pr['createdAt']}") print(f" URL: https://huggingface.co/{repo_id}/discussions/{pr['num']}") print() print(f"{'='*70}\n") # ============================================================================ # Method 2: Import from Artificial Analysis # ============================================================================ def get_aa_model_data(creator_slug: str, model_name: str) -> Optional[Dict[str, Any]]: """ Fetch model evaluation data from Artificial Analysis API. Args: creator_slug: Creator identifier (e.g., "anthropic", "openai") model_name: Model slug/identifier Returns: Model data dictionary or None if not found """ load_env() AA_API_KEY = os.getenv("AA_API_KEY") if not AA_API_KEY: raise ValueError("AA_API_KEY environment variable is not set") url = "https://artificialanalysis.ai/api/v2/data/llms/models" headers = {"x-api-key": AA_API_KEY} requests = require_requests() try: response = requests.get(url, headers=headers, timeout=30) response.raise_for_status() data = response.json().get("data", []) for model in data: creator = model.get("model_creator", {}) if creator.get("slug") == creator_slug and model.get("slug") == model_name: return model print(f"Model {creator_slug}/{model_name} not found in Artificial Analysis") return None except requests.RequestException as e: print(f"Error fetching data from Artificial Analysis: {e}") return None def aa_data_to_model_index( model_data: Dict[str, Any], dataset_name: str = "Artificial Analysis Benchmarks", dataset_type: str = "artificial_analysis", task_type: str = "evaluation" ) -> List[Dict[str, Any]]: """ Convert Artificial Analysis model data to model-index format. Args: model_data: Raw model data from AA API dataset_name: Dataset name for model-index dataset_type: Dataset type identifier task_type: Task type for model-index Returns: Model-index formatted results """ model_name = model_data.get("name", model_data.get("slug", "unknown-model")) evaluations = model_data.get("evaluations", {}) if not evaluations: print(f"No evaluations found for model {model_name}") return [] metrics = [] for key, value in evaluations.items(): if value is not None: metrics.append({ "name": key.replace("_", " ").title(), "type": key, "value": value }) results = [{ "task": {"type": task_type}, "dataset": { "name": dataset_name, "type": dataset_type }, "metrics": metrics, "source": { "name": "Artificial Analysis API", "url": "https://artificialanalysis.ai" } }] return results def import_aa_evaluations( creator_slug: str, model_name: str, repo_id: str ) -> Optional[List[Dict[str, Any]]]: """ Import evaluation results from Artificial Analysis for a model. Args: creator_slug: Creator identifier in AA model_name: Model identifier in AA repo_id: Hugging Face repository ID to update Returns: Model-index formatted results or None if import fails """ model_data = get_aa_model_data(creator_slug, model_name) if not model_data: return None results = aa_data_to_model_index(model_data) return results # ============================================================================ # Model Card Update Functions # ============================================================================ def update_model_card_with_evaluations( repo_id: str, results: List[Dict[str, Any]], create_pr: bool = False, commit_message: Optional[str] = None ) -> bool: """ Update a model card with evaluation results. Args: repo_id: Hugging Face repository ID results: Model-index formatted results create_pr: Whether to create a PR instead of direct push commit_message: Custom commit message Returns: True if successful, False otherwise """ try: load_env() ModelCard = require_model_card() hf_token = os.getenv("HF_TOKEN") if not hf_token: raise ValueError("HF_TOKEN environment variable is not set") # Load existing card card = ModelCard.load(repo_id, token=hf_token) # Get model name model_name = repo_id.split("/")[-1] if "/" in repo_id else repo_id # Create or update model-index model_index = [{ "name": model_name, "results": results }] # Merge with existing model-index if present if "model-index" in card.data: existing = card.data["model-index"] if isinstance(existing, list) and existing: # Keep existing name if present if "name" in existing[0]: model_index[0]["name"] = existing[0]["name"] # Merge results existing_results = existing[0].get("results", []) model_index[0]["results"].extend(existing_results) card.data["model-index"] = model_index # Prepare commit message if not commit_message: commit_message = f"Add evaluation results to {model_name}" commit_description = ( "This commit adds structured evaluation results to the model card. " "The results are formatted using the model-index specification and " "will be displayed in the model card's evaluation widget." ) # Push update card.push_to_hub( repo_id, token=hf_token, commit_message=commit_message, commit_description=commit_description, create_pr=create_pr ) action = "Pull request created" if create_pr else "Model card updated" print(f"✓ {action} successfully for {repo_id}") return True except Exception as e: print(f"Error updating model card: {e}") return False def show_evaluations(repo_id: str) -> None: """Display current evaluations in a model card.""" try: load_env() ModelCard = require_model_card() hf_token = os.getenv("HF_TOKEN") card = ModelCard.load(repo_id, token=hf_token) if "model-index" not in card.data: print(f"No model-index found in {repo_id}") return model_index = card.data["model-index"] print(f"\nEvaluations for {repo_id}:") print("=" * 60) for model_entry in model_index: model_name = model_entry.get("name", "Unknown") print(f"\nModel: {model_name}") results = model_entry.get("results", []) for i, result in enumerate(results, 1): print(f"\n Result Set {i}:") task = result.get("task", {}) print(f" Task: {task.get('type', 'unknown')}") dataset = result.get("dataset", {}) print(f" Dataset: {dataset.get('name', 'unknown')}") metrics = result.get("metrics", []) print(f" Metrics ({len(metrics)}):") for metric in metrics: name = metric.get("name", "Unknown") value = metric.get("value", "N/A") print(f" - {name}: {value}") source = result.get("source", {}) if source: print(f" Source: {source.get('name', 'Unknown')}") print("\n" + "=" * 60) except Exception as e: print(f"Error showing evaluations: {e}") def validate_model_index(repo_id: str) -> bool: """Validate model-index format in a model card.""" try: load_env() ModelCard = require_model_card() hf_token = os.getenv("HF_TOKEN") card = ModelCard.load(repo_id, token=hf_token) if "model-index" not in card.data: print(f"✗ No model-index found in {repo_id}") return False model_index = card.data["model-index"] if not isinstance(model_index, list): print("✗ model-index must be a list") return False for i, entry in enumerate(model_index): if "name" not in entry: print(f"✗ Entry {i} missing 'name' field") return False if "results" not in entry: print(f"✗ Entry {i} missing 'results' field") return False for j, result in enumerate(entry["results"]): if "task" not in result: print(f"✗ Result {j} in entry {i} missing 'task' field") return False if "dataset" not in result: print(f"✗ Result {j} in entry {i} missing 'dataset' field") return False if "metrics" not in result: print(f"✗ Result {j} in entry {i} missing 'metrics' field") return False print(f"✓ Model-index format is valid for {repo_id}") return True except Exception as e: print(f"Error validating model-index: {e}") return False # ============================================================================ # CLI Interface # ============================================================================ def main(): parser = argparse.ArgumentParser( description=( "Manage evaluation results in Hugging Face model cards.\n\n" "Use standard Python or `uv run scripts/evaluation_manager.py ...` " "to auto-resolve dependencies from the PEP 723 header." ), formatter_class=argparse.RawTextHelpFormatter, epilog=dedent( """\ Typical workflows: - Inspect tables first: uv run scripts/evaluation_manager.py inspect-tables --repo-id - Extract from README (prints YAML by default): uv run scripts/evaluation_manager.py extract-readme --repo-id --table N - Apply changes: uv run scripts/evaluation_manager.py extract-readme --repo-id --table N --apply - Import from Artificial Analysis: AA_API_KEY=... uv run scripts/evaluation_manager.py import-aa --creator-slug org --model-name slug --repo-id Tips: - YAML is printed by default; use --apply or --create-pr to write changes. - Set HF_TOKEN (and AA_API_KEY for import-aa); .env is loaded automatically if python-dotenv is installed. - When multiple tables exist, run inspect-tables then select with --table N. - To apply changes (push or PR), rerun extract-readme with --apply or --create-pr. """ ), ) parser.add_argument("--version", action="version", version="evaluation_manager 1.2.0") subparsers = parser.add_subparsers(dest="command", help="Command to execute") # Extract from README command extract_parser = subparsers.add_parser( "extract-readme", help="Extract evaluation tables from model README", formatter_class=argparse.RawTextHelpFormatter, description="Parse README tables into model-index YAML. Default behavior prints YAML; use --apply/--create-pr to write changes.", epilog=dedent( """\ Examples: uv run scripts/evaluation_manager.py extract-readme --repo-id username/model uv run scripts/evaluation_manager.py extract-readme --repo-id username/model --table 2 --model-column-index 3 uv run scripts/evaluation_manager.py extract-readme --repo-id username/model --table 2 --model-name-override \"**Model 7B**\" # exact header text uv run scripts/evaluation_manager.py extract-readme --repo-id username/model --table 2 --create-pr Apply changes: - Default: prints YAML to stdout (no writes). - Add --apply to push directly, or --create-pr to open a PR. Model selection: - Preferred: --model-column-index
- If using --model-name-override, copy the column header text exactly. """ ), ) extract_parser.add_argument("--repo-id", type=str, required=True, help="HF repository ID") extract_parser.add_argument("--table", type=int, help="Table number (1-indexed, from inspect-tables output)") extract_parser.add_argument("--model-column-index", type=int, help="Preferred: column index from inspect-tables output (exact selection)") extract_parser.add_argument("--model-name-override", type=str, help="Exact column header/model name for comparison/transpose tables (when index is not used)") extract_parser.add_argument("--task-type", type=str, default="text-generation", help="Sets model-index task.type (e.g., text-generation, summarization)") extract_parser.add_argument("--dataset-name", type=str, default="Benchmarks", help="Dataset name") extract_parser.add_argument("--dataset-type", type=str, default="benchmark", help="Dataset type") extract_parser.add_argument("--create-pr", action="store_true", help="Create PR instead of direct push") extract_parser.add_argument("--apply", action="store_true", help="Apply changes (default is to print YAML only)") extract_parser.add_argument("--dry-run", action="store_true", help="Preview YAML without updating (default)") # Import from AA command aa_parser = subparsers.add_parser( "import-aa", help="Import evaluation scores from Artificial Analysis", formatter_class=argparse.RawTextHelpFormatter, description="Fetch scores from Artificial Analysis API and write them into model-index.", epilog=dedent( """\ Examples: AA_API_KEY=... uv run scripts/evaluation_manager.py import-aa --creator-slug anthropic --model-name claude-sonnet-4 --repo-id username/model uv run scripts/evaluation_manager.py import-aa --creator-slug openai --model-name gpt-4o --repo-id username/model --create-pr Requires: AA_API_KEY in env (or .env if python-dotenv installed). """ ), ) aa_parser.add_argument("--creator-slug", type=str, required=True, help="AA creator slug") aa_parser.add_argument("--model-name", type=str, required=True, help="AA model name") aa_parser.add_argument("--repo-id", type=str, required=True, help="HF repository ID") aa_parser.add_argument("--create-pr", action="store_true", help="Create PR instead of direct push") # Show evaluations command show_parser = subparsers.add_parser( "show", help="Display current evaluations in model card", formatter_class=argparse.RawTextHelpFormatter, description="Print model-index content from the model card (requires HF_TOKEN for private repos).", ) show_parser.add_argument("--repo-id", type=str, required=True, help="HF repository ID") # Validate command validate_parser = subparsers.add_parser( "validate", help="Validate model-index format", formatter_class=argparse.RawTextHelpFormatter, description="Schema sanity check for model-index section of the card.", ) validate_parser.add_argument("--repo-id", type=str, required=True, help="HF repository ID") # Inspect tables command inspect_parser = subparsers.add_parser( "inspect-tables", help="Inspect tables in README → outputs suggested extract-readme command", formatter_class=argparse.RawDescriptionHelpFormatter, epilog=""" Workflow: 1. inspect-tables → see table structure, columns, and table numbers 2. extract-readme → run with --table N (from step 1); YAML prints by default 3. apply changes → rerun extract-readme with --apply or --create-pr Reminder: - Preferred: use --model-column-index . If needed, use --model-name-override with the exact column header text. """ ) inspect_parser.add_argument("--repo-id", type=str, required=True, help="HF repository ID") # Get PRs command prs_parser = subparsers.add_parser( "get-prs", help="List open pull requests for a model repository", formatter_class=argparse.RawTextHelpFormatter, description="Check for existing open PRs before creating new ones to avoid duplicates.", epilog=dedent( """\ Examples: uv run scripts/evaluation_manager.py get-prs --repo-id "allenai/Olmo-3-32B-Think" IMPORTANT: Always run this before using --create-pr to avoid duplicate PRs. """ ), ) prs_parser.add_argument("--repo-id", type=str, required=True, help="HF repository ID") args = parser.parse_args() if not args.command: parser.print_help() return try: # Execute command if args.command == "extract-readme": results = extract_evaluations_from_readme( repo_id=args.repo_id, task_type=args.task_type, dataset_name=args.dataset_name, dataset_type=args.dataset_type, model_name_override=args.model_name_override, table_index=args.table, model_column_index=args.model_column_index ) if not results: print("No evaluations extracted") return apply_changes = args.apply or args.create_pr # Default behavior: print YAML (dry-run) yaml = require_yaml() print("\nExtracted evaluations (YAML):") print( yaml.dump( {"model-index": [{"name": args.repo_id.split('/')[-1], "results": results}]}, sort_keys=False ) ) if apply_changes: if args.model_name_override and args.model_column_index is not None: print("Note: --model-column-index takes precedence over --model-name-override.") update_model_card_with_evaluations( repo_id=args.repo_id, results=results, create_pr=args.create_pr, commit_message="Extract evaluation results from README" ) elif args.command == "import-aa": results = import_aa_evaluations( creator_slug=args.creator_slug, model_name=args.model_name, repo_id=args.repo_id ) if not results: print("No evaluations imported") return update_model_card_with_evaluations( repo_id=args.repo_id, results=results, create_pr=args.create_pr, commit_message=f"Add Artificial Analysis evaluations for {args.model_name}" ) elif args.command == "show": show_evaluations(args.repo_id) elif args.command == "validate": validate_model_index(args.repo_id) elif args.command == "inspect-tables": inspect_tables(args.repo_id) elif args.command == "get-prs": list_open_prs(args.repo_id) except ModuleNotFoundError as exc: # Surface dependency hints cleanly when user only needs help output print(exc) except Exception as exc: print(f"Error: {exc}") if __name__ == "__main__": main() ================================================ FILE: skills/hugging-face-evaluation/scripts/inspect_eval_uv.py ================================================ # /// script # requires-python = ">=3.10" # dependencies = [ # "inspect-ai>=0.3.0", # "inspect-evals", # "openai", # ] # /// """ Entry point script for running inspect-ai evaluations via `hf jobs uv run`. """ from __future__ import annotations import argparse import os import subprocess import sys from pathlib import Path from typing import Optional def _inspect_evals_tasks_root() -> Optional[Path]: """Return the installed inspect_evals package path if available.""" try: import inspect_evals return Path(inspect_evals.__file__).parent except Exception: return None def _normalize_task(task: str) -> str: """Allow lighteval-style `suite|task|shots` strings by keeping the task name.""" if "|" in task: parts = task.split("|") if len(parts) >= 2 and parts[1]: return parts[1] return task def main() -> None: parser = argparse.ArgumentParser(description="Inspect-ai job runner") parser.add_argument("--model", required=True, help="Model ID on Hugging Face Hub") parser.add_argument("--task", required=True, help="inspect-ai task to execute") parser.add_argument("--limit", type=int, default=None, help="Limit number of samples to evaluate") parser.add_argument( "--tasks-root", default=None, help="Optional path to inspect task files. Defaults to the installed inspect_evals package.", ) parser.add_argument( "--sandbox", default="local", help="Sandbox backend to use (default: local for HF jobs without Docker).", ) args = parser.parse_args() # Ensure downstream libraries can read the token passed as a secret hf_token = os.getenv("HF_TOKEN") if hf_token: os.environ.setdefault("HUGGING_FACE_HUB_TOKEN", hf_token) os.environ.setdefault("HF_HUB_TOKEN", hf_token) task = _normalize_task(args.task) tasks_root = Path(args.tasks_root) if args.tasks_root else _inspect_evals_tasks_root() if tasks_root and not tasks_root.exists(): tasks_root = None cmd = [ "inspect", "eval", task, "--model", f"hf-inference-providers/{args.model}", "--log-level", "info", # Reduce batch size to avoid OOM errors (default is 32) "--max-connections", "1", # Set a small positive temperature (HF doesn't allow temperature=0) "--temperature", "0.001", ] if args.sandbox: cmd.extend(["--sandbox", args.sandbox]) if args.limit: cmd.extend(["--limit", str(args.limit)]) try: subprocess.run(cmd, check=True, cwd=tasks_root) print("Evaluation complete.") except subprocess.CalledProcessError as exc: location = f" (cwd={tasks_root})" if tasks_root else "" print(f"Evaluation failed with exit code {exc.returncode}{location}", file=sys.stderr) raise if __name__ == "__main__": main() ================================================ FILE: skills/hugging-face-evaluation/scripts/inspect_vllm_uv.py ================================================ # /// script # requires-python = ">=3.10" # dependencies = [ # "inspect-ai>=0.3.0", # "inspect-evals", # "vllm>=0.4.0", # "torch>=2.0.0", # "transformers>=4.40.0", # ] # /// """ Entry point script for running inspect-ai evaluations with vLLM or HuggingFace Transformers backend. This script runs evaluations on custom HuggingFace models using local GPU inference, separate from inference provider scripts (which use external APIs). Usage (standalone): python inspect_vllm_uv.py --model "meta-llama/Llama-3.2-1B" --task "mmlu" Usage (via HF Jobs): hf jobs uv run inspect_vllm_uv.py \\ --flavor a10g-small \\ --secret HF_TOKEN=$HF_TOKEN \\ -- --model "meta-llama/Llama-3.2-1B" --task "mmlu" Model backends: - vllm: Fast inference with vLLM (recommended for large models) - hf: HuggingFace Transformers backend (broader model compatibility) """ from __future__ import annotations import argparse import os import subprocess import sys from typing import Optional def setup_environment() -> None: """Configure environment variables for HuggingFace authentication.""" hf_token = os.getenv("HF_TOKEN") if hf_token: os.environ.setdefault("HUGGING_FACE_HUB_TOKEN", hf_token) os.environ.setdefault("HF_HUB_TOKEN", hf_token) def run_inspect_vllm( model_id: str, task: str, limit: Optional[int] = None, max_connections: int = 4, temperature: float = 0.0, tensor_parallel_size: int = 1, gpu_memory_utilization: float = 0.8, dtype: str = "auto", trust_remote_code: bool = False, log_level: str = "info", ) -> None: """ Run inspect-ai evaluation with vLLM backend. Args: model_id: HuggingFace model ID task: inspect-ai task to execute (e.g., "mmlu", "gsm8k") limit: Limit number of samples to evaluate max_connections: Maximum concurrent connections temperature: Sampling temperature tensor_parallel_size: Number of GPUs for tensor parallelism gpu_memory_utilization: GPU memory fraction dtype: Data type (auto, float16, bfloat16) trust_remote_code: Allow remote code execution log_level: Logging level """ setup_environment() model_spec = f"vllm/{model_id}" cmd = [ "inspect", "eval", task, "--model", model_spec, "--log-level", log_level, "--max-connections", str(max_connections), ] # vLLM supports temperature=0 unlike HF inference providers cmd.extend(["--temperature", str(temperature)]) # Older inspect-ai CLI versions do not support --model-args; rely on defaults # and let vLLM choose sensible settings for small models. if tensor_parallel_size != 1: cmd.extend(["--tensor-parallel-size", str(tensor_parallel_size)]) if gpu_memory_utilization != 0.8: cmd.extend(["--gpu-memory-utilization", str(gpu_memory_utilization)]) if dtype != "auto": cmd.extend(["--dtype", dtype]) if trust_remote_code: cmd.append("--trust-remote-code") if limit: cmd.extend(["--limit", str(limit)]) print(f"Running: {' '.join(cmd)}") try: subprocess.run(cmd, check=True) print("Evaluation complete.") except subprocess.CalledProcessError as exc: print(f"Evaluation failed with exit code {exc.returncode}", file=sys.stderr) sys.exit(exc.returncode) def run_inspect_hf( model_id: str, task: str, limit: Optional[int] = None, max_connections: int = 1, temperature: float = 0.001, device: str = "auto", dtype: str = "auto", trust_remote_code: bool = False, log_level: str = "info", ) -> None: """ Run inspect-ai evaluation with HuggingFace Transformers backend. Use this when vLLM doesn't support the model architecture. Args: model_id: HuggingFace model ID task: inspect-ai task to execute limit: Limit number of samples max_connections: Maximum concurrent connections (keep low for memory) temperature: Sampling temperature device: Device to use (auto, cuda, cpu) dtype: Data type trust_remote_code: Allow remote code execution log_level: Logging level """ setup_environment() model_spec = f"hf/{model_id}" cmd = [ "inspect", "eval", task, "--model", model_spec, "--log-level", log_level, "--max-connections", str(max_connections), "--temperature", str(temperature), ] if device != "auto": cmd.extend(["--device", device]) if dtype != "auto": cmd.extend(["--dtype", dtype]) if trust_remote_code: cmd.append("--trust-remote-code") if limit: cmd.extend(["--limit", str(limit)]) print(f"Running: {' '.join(cmd)}") try: subprocess.run(cmd, check=True) print("Evaluation complete.") except subprocess.CalledProcessError as exc: print(f"Evaluation failed with exit code {exc.returncode}", file=sys.stderr) sys.exit(exc.returncode) def main() -> None: parser = argparse.ArgumentParser( description="Run inspect-ai evaluations with vLLM or HuggingFace Transformers on custom models", formatter_class=argparse.RawDescriptionHelpFormatter, epilog=""" Examples: # Run MMLU with vLLM backend python inspect_vllm_uv.py --model meta-llama/Llama-3.2-1B --task mmlu # Run with HuggingFace Transformers backend python inspect_vllm_uv.py --model meta-llama/Llama-3.2-1B --task mmlu --backend hf # Run with limited samples for testing python inspect_vllm_uv.py --model meta-llama/Llama-3.2-1B --task mmlu --limit 10 # Run on multiple GPUs with tensor parallelism python inspect_vllm_uv.py --model meta-llama/Llama-3.2-70B --task mmlu --tensor-parallel-size 4 Available tasks (from inspect-evals): - mmlu: Massive Multitask Language Understanding - gsm8k: Grade School Math - hellaswag: Common sense reasoning - arc_challenge: AI2 Reasoning Challenge - truthfulqa: TruthfulQA benchmark - winogrande: Winograd Schema Challenge - humaneval: Code generation (HumanEval) Via HF Jobs: hf jobs uv run inspect_vllm_uv.py \\ --flavor a10g-small \\ --secret HF_TOKEN=$HF_TOKEN \\ -- --model meta-llama/Llama-3.2-1B --task mmlu """, ) parser.add_argument( "--model", required=True, help="HuggingFace model ID (e.g., meta-llama/Llama-3.2-1B)", ) parser.add_argument( "--task", required=True, help="inspect-ai task to execute (e.g., mmlu, gsm8k)", ) parser.add_argument( "--backend", choices=["vllm", "hf"], default="vllm", help="Model backend (default: vllm)", ) parser.add_argument( "--limit", type=int, default=None, help="Limit number of samples to evaluate", ) parser.add_argument( "--max-connections", type=int, default=None, help="Maximum concurrent connections (default: 4 for vllm, 1 for hf)", ) parser.add_argument( "--temperature", type=float, default=None, help="Sampling temperature (default: 0.0 for vllm, 0.001 for hf)", ) parser.add_argument( "--tensor-parallel-size", type=int, default=1, help="Number of GPUs for tensor parallelism (vLLM only, default: 1)", ) parser.add_argument( "--gpu-memory-utilization", type=float, default=0.8, help="GPU memory fraction to use (vLLM only, default: 0.8)", ) parser.add_argument( "--dtype", default="auto", choices=["auto", "float16", "bfloat16", "float32"], help="Data type for model weights (default: auto)", ) parser.add_argument( "--device", default="auto", help="Device for HF backend (auto, cuda, cpu)", ) parser.add_argument( "--trust-remote-code", action="store_true", help="Allow executing remote code from model repository", ) parser.add_argument( "--log-level", default="info", choices=["debug", "info", "warning", "error"], help="Logging level (default: info)", ) args = parser.parse_args() if args.backend == "vllm": run_inspect_vllm( model_id=args.model, task=args.task, limit=args.limit, max_connections=args.max_connections or 4, temperature=args.temperature if args.temperature is not None else 0.0, tensor_parallel_size=args.tensor_parallel_size, gpu_memory_utilization=args.gpu_memory_utilization, dtype=args.dtype, trust_remote_code=args.trust_remote_code, log_level=args.log_level, ) else: run_inspect_hf( model_id=args.model, task=args.task, limit=args.limit, max_connections=args.max_connections or 1, temperature=args.temperature if args.temperature is not None else 0.001, device=args.device, dtype=args.dtype, trust_remote_code=args.trust_remote_code, log_level=args.log_level, ) if __name__ == "__main__": main() ================================================ FILE: skills/hugging-face-evaluation/scripts/lighteval_vllm_uv.py ================================================ # /// script # requires-python = ">=3.10" # dependencies = [ # "lighteval[accelerate,vllm]>=0.6.0", # "torch>=2.0.0", # "transformers>=4.40.0", # "accelerate>=0.30.0", # "vllm>=0.4.0", # ] # /// """ Entry point script for running lighteval evaluations with vLLM backend via `hf jobs uv run`. This script runs evaluations using vLLM for efficient GPU inference on custom HuggingFace models. It is separate from inference provider scripts and evaluates models directly on the hardware. Usage (standalone): python lighteval_vllm_uv.py --model "meta-llama/Llama-3.2-1B" --tasks "leaderboard|mmlu|5" Usage (via HF Jobs): hf jobs uv run lighteval_vllm_uv.py \\ --flavor a10g-small \\ --secret HF_TOKEN=$HF_TOKEN \\ -- --model "meta-llama/Llama-3.2-1B" --tasks "leaderboard|mmlu|5" """ from __future__ import annotations import argparse import os import subprocess import sys from typing import Optional def setup_environment() -> None: """Configure environment variables for HuggingFace authentication.""" hf_token = os.getenv("HF_TOKEN") if hf_token: os.environ.setdefault("HUGGING_FACE_HUB_TOKEN", hf_token) os.environ.setdefault("HF_HUB_TOKEN", hf_token) def run_lighteval_vllm( model_id: str, tasks: str, output_dir: Optional[str] = None, max_samples: Optional[int] = None, batch_size: int = 1, tensor_parallel_size: int = 1, gpu_memory_utilization: float = 0.8, dtype: str = "auto", trust_remote_code: bool = False, use_chat_template: bool = False, system_prompt: Optional[str] = None, ) -> None: """ Run lighteval with vLLM backend for efficient GPU inference. Args: model_id: HuggingFace model ID (e.g., "meta-llama/Llama-3.2-1B") tasks: Task specification (e.g., "leaderboard|mmlu|5" or "lighteval|hellaswag|0") output_dir: Directory for evaluation results max_samples: Limit number of samples per task batch_size: Batch size for evaluation tensor_parallel_size: Number of GPUs for tensor parallelism gpu_memory_utilization: GPU memory fraction to use (0.0-1.0) dtype: Data type for model weights (auto, float16, bfloat16) trust_remote_code: Allow executing remote code from model repo use_chat_template: Apply chat template for conversational models system_prompt: System prompt for chat models """ setup_environment() # Build lighteval vllm command cmd = [ "lighteval", "vllm", model_id, tasks, "--batch-size", str(batch_size), "--tensor-parallel-size", str(tensor_parallel_size), "--gpu-memory-utilization", str(gpu_memory_utilization), "--dtype", dtype, ] if output_dir: cmd.extend(["--output-dir", output_dir]) if max_samples: cmd.extend(["--max-samples", str(max_samples)]) if trust_remote_code: cmd.append("--trust-remote-code") if use_chat_template: cmd.append("--use-chat-template") if system_prompt: cmd.extend(["--system-prompt", system_prompt]) print(f"Running: {' '.join(cmd)}") try: subprocess.run(cmd, check=True) print("Evaluation complete.") except subprocess.CalledProcessError as exc: print(f"Evaluation failed with exit code {exc.returncode}", file=sys.stderr) sys.exit(exc.returncode) def run_lighteval_accelerate( model_id: str, tasks: str, output_dir: Optional[str] = None, max_samples: Optional[int] = None, batch_size: int = 1, dtype: str = "bfloat16", trust_remote_code: bool = False, use_chat_template: bool = False, system_prompt: Optional[str] = None, ) -> None: """ Run lighteval with accelerate backend for multi-GPU distributed inference. Use this backend when vLLM is not available or for models not supported by vLLM. Args: model_id: HuggingFace model ID tasks: Task specification output_dir: Directory for evaluation results max_samples: Limit number of samples per task batch_size: Batch size for evaluation dtype: Data type for model weights trust_remote_code: Allow executing remote code use_chat_template: Apply chat template system_prompt: System prompt for chat models """ setup_environment() # Build lighteval accelerate command cmd = [ "lighteval", "accelerate", model_id, tasks, "--batch-size", str(batch_size), "--dtype", dtype, ] if output_dir: cmd.extend(["--output-dir", output_dir]) if max_samples: cmd.extend(["--max-samples", str(max_samples)]) if trust_remote_code: cmd.append("--trust-remote-code") if use_chat_template: cmd.append("--use-chat-template") if system_prompt: cmd.extend(["--system-prompt", system_prompt]) print(f"Running: {' '.join(cmd)}") try: subprocess.run(cmd, check=True) print("Evaluation complete.") except subprocess.CalledProcessError as exc: print(f"Evaluation failed with exit code {exc.returncode}", file=sys.stderr) sys.exit(exc.returncode) def main() -> None: parser = argparse.ArgumentParser( description="Run lighteval evaluations with vLLM or accelerate backend on custom HuggingFace models", formatter_class=argparse.RawDescriptionHelpFormatter, epilog=""" Examples: # Run MMLU evaluation with vLLM python lighteval_vllm_uv.py --model meta-llama/Llama-3.2-1B --tasks "leaderboard|mmlu|5" # Run with accelerate backend instead of vLLM python lighteval_vllm_uv.py --model meta-llama/Llama-3.2-1B --tasks "leaderboard|mmlu|5" --backend accelerate # Run with chat template for instruction-tuned models python lighteval_vllm_uv.py --model meta-llama/Llama-3.2-1B-Instruct --tasks "leaderboard|mmlu|5" --use-chat-template # Run with limited samples for testing python lighteval_vllm_uv.py --model meta-llama/Llama-3.2-1B --tasks "leaderboard|mmlu|5" --max-samples 10 Task format: Tasks use the format: "suite|task|num_fewshot" - leaderboard|mmlu|5 (MMLU with 5-shot) - lighteval|hellaswag|0 (HellaSwag zero-shot) - leaderboard|gsm8k|5 (GSM8K with 5-shot) - Multiple tasks: "leaderboard|mmlu|5,leaderboard|gsm8k|5" """, ) parser.add_argument( "--model", required=True, help="HuggingFace model ID (e.g., meta-llama/Llama-3.2-1B)", ) parser.add_argument( "--tasks", required=True, help="Task specification (e.g., 'leaderboard|mmlu|5')", ) parser.add_argument( "--backend", choices=["vllm", "accelerate"], default="vllm", help="Inference backend to use (default: vllm)", ) parser.add_argument( "--output-dir", default=None, help="Directory for evaluation results", ) parser.add_argument( "--max-samples", type=int, default=None, help="Limit number of samples per task (useful for testing)", ) parser.add_argument( "--batch-size", type=int, default=1, help="Batch size for evaluation (default: 1)", ) parser.add_argument( "--tensor-parallel-size", type=int, default=1, help="Number of GPUs for tensor parallelism (vLLM only, default: 1)", ) parser.add_argument( "--gpu-memory-utilization", type=float, default=0.8, help="GPU memory fraction to use (vLLM only, default: 0.8)", ) parser.add_argument( "--dtype", default="auto", choices=["auto", "float16", "bfloat16", "float32"], help="Data type for model weights (default: auto)", ) parser.add_argument( "--trust-remote-code", action="store_true", help="Allow executing remote code from model repository", ) parser.add_argument( "--use-chat-template", action="store_true", help="Apply chat template for instruction-tuned/chat models", ) parser.add_argument( "--system-prompt", default=None, help="System prompt for chat models", ) args = parser.parse_args() if args.backend == "vllm": run_lighteval_vllm( model_id=args.model, tasks=args.tasks, output_dir=args.output_dir, max_samples=args.max_samples, batch_size=args.batch_size, tensor_parallel_size=args.tensor_parallel_size, gpu_memory_utilization=args.gpu_memory_utilization, dtype=args.dtype, trust_remote_code=args.trust_remote_code, use_chat_template=args.use_chat_template, system_prompt=args.system_prompt, ) else: run_lighteval_accelerate( model_id=args.model, tasks=args.tasks, output_dir=args.output_dir, max_samples=args.max_samples, batch_size=args.batch_size, dtype=args.dtype if args.dtype != "auto" else "bfloat16", trust_remote_code=args.trust_remote_code, use_chat_template=args.use_chat_template, system_prompt=args.system_prompt, ) if __name__ == "__main__": main() ================================================ FILE: skills/hugging-face-evaluation/scripts/run_eval_job.py ================================================ # /// script # requires-python = ">=3.10" # dependencies = [ # "huggingface-hub>=0.26.0", # "python-dotenv>=1.2.1", # ] # /// """ Submit evaluation jobs using the `hf jobs uv run` CLI. This wrapper constructs the appropriate command to execute the local `inspect_eval_uv.py` script on Hugging Face Jobs with the requested hardware. """ import argparse import os import subprocess import sys from pathlib import Path from typing import Optional from huggingface_hub import get_token from dotenv import load_dotenv load_dotenv() SCRIPT_PATH = Path(__file__).with_name("inspect_eval_uv.py").resolve() def create_eval_job( model_id: str, task: str, hardware: str = "cpu-basic", hf_token: Optional[str] = None, limit: Optional[int] = None, ) -> None: """ Submit an evaluation job using the Hugging Face Jobs CLI. """ token = hf_token or os.getenv("HF_TOKEN") or get_token() if not token: raise ValueError("HF_TOKEN is required. Set it in environment or pass as argument.") if not SCRIPT_PATH.exists(): raise FileNotFoundError(f"Script not found at {SCRIPT_PATH}") print(f"Preparing evaluation job for {model_id} on task {task} (hardware: {hardware})") cmd = [ "hf", "jobs", "uv", "run", str(SCRIPT_PATH), "--flavor", hardware, "--secrets", f"HF_TOKEN={token}", "--", "--model", model_id, "--task", task, ] if limit: cmd.extend(["--limit", str(limit)]) print("Executing:", " ".join(cmd)) try: subprocess.run(cmd, check=True) except subprocess.CalledProcessError as exc: print("hf jobs command failed", file=sys.stderr) raise def main() -> None: parser = argparse.ArgumentParser(description="Run inspect-ai evaluations on Hugging Face Jobs") parser.add_argument("--model", required=True, help="Model ID (e.g. Qwen/Qwen3-0.6B)") parser.add_argument("--task", required=True, help="Inspect task (e.g. mmlu, gsm8k)") parser.add_argument("--hardware", default="cpu-basic", help="Hardware flavor (e.g. t4-small, a10g-small)") parser.add_argument("--limit", type=int, default=None, help="Limit number of samples to evaluate") args = parser.parse_args() create_eval_job( model_id=args.model, task=args.task, hardware=args.hardware, limit=args.limit, ) if __name__ == "__main__": main() ================================================ FILE: skills/hugging-face-evaluation/scripts/run_vllm_eval_job.py ================================================ # /// script # requires-python = ">=3.10" # dependencies = [ # "huggingface-hub>=0.26.0", # "python-dotenv>=1.2.1", # ] # /// """ Submit vLLM-based evaluation jobs using the `hf jobs uv run` CLI. This wrapper constructs the appropriate command to execute vLLM evaluation scripts (lighteval or inspect-ai) on Hugging Face Jobs with GPU hardware. Unlike run_eval_job.py (which uses inference providers/APIs), this script runs models directly on the job's GPU using vLLM or HuggingFace Transformers. Usage: python run_vllm_eval_job.py \\ --model meta-llama/Llama-3.2-1B \\ --task mmlu \\ --framework lighteval \\ --hardware a10g-small """ from __future__ import annotations import argparse import os import subprocess import sys from pathlib import Path from typing import Optional from huggingface_hub import get_token from dotenv import load_dotenv load_dotenv() # Script paths for different evaluation frameworks SCRIPT_DIR = Path(__file__).parent.resolve() LIGHTEVAL_SCRIPT = SCRIPT_DIR / "lighteval_vllm_uv.py" INSPECT_SCRIPT = SCRIPT_DIR / "inspect_vllm_uv.py" # Hardware flavor recommendations for different model sizes HARDWARE_RECOMMENDATIONS = { "small": "t4-small", # < 3B parameters "medium": "a10g-small", # 3B - 13B parameters "large": "a10g-large", # 13B - 34B parameters "xlarge": "a100-large", # 34B+ parameters } def estimate_hardware(model_id: str) -> str: """ Estimate appropriate hardware based on model ID naming conventions. Returns a hardware flavor recommendation. """ model_lower = model_id.lower() # Check for explicit size indicators in model name if any(x in model_lower for x in ["70b", "72b", "65b"]): return "a100-large" elif any(x in model_lower for x in ["34b", "33b", "32b", "30b"]): return "a10g-large" elif any(x in model_lower for x in ["13b", "14b", "7b", "8b"]): return "a10g-small" elif any(x in model_lower for x in ["3b", "2b", "1b", "0.5b", "small", "mini"]): return "t4-small" # Default to medium hardware return "a10g-small" def create_lighteval_job( model_id: str, tasks: str, hardware: str, hf_token: Optional[str] = None, max_samples: Optional[int] = None, backend: str = "vllm", batch_size: int = 1, tensor_parallel_size: int = 1, trust_remote_code: bool = False, use_chat_template: bool = False, ) -> None: """ Submit a lighteval evaluation job on HuggingFace Jobs. """ token = hf_token or os.getenv("HF_TOKEN") or get_token() if not token: raise ValueError("HF_TOKEN is required. Set it in environment or pass as argument.") if not LIGHTEVAL_SCRIPT.exists(): raise FileNotFoundError(f"Script not found at {LIGHTEVAL_SCRIPT}") print(f"Preparing lighteval job for {model_id}") print(f" Tasks: {tasks}") print(f" Backend: {backend}") print(f" Hardware: {hardware}") cmd = [ "hf", "jobs", "uv", "run", str(LIGHTEVAL_SCRIPT), "--flavor", hardware, "--secrets", f"HF_TOKEN={token}", "--", "--model", model_id, "--tasks", tasks, "--backend", backend, "--batch-size", str(batch_size), "--tensor-parallel-size", str(tensor_parallel_size), ] if max_samples: cmd.extend(["--max-samples", str(max_samples)]) if trust_remote_code: cmd.append("--trust-remote-code") if use_chat_template: cmd.append("--use-chat-template") print(f"\nExecuting: {' '.join(cmd)}") try: subprocess.run(cmd, check=True) except subprocess.CalledProcessError as exc: print("hf jobs command failed", file=sys.stderr) raise def create_inspect_job( model_id: str, task: str, hardware: str, hf_token: Optional[str] = None, limit: Optional[int] = None, backend: str = "vllm", tensor_parallel_size: int = 1, trust_remote_code: bool = False, ) -> None: """ Submit an inspect-ai evaluation job on HuggingFace Jobs. """ token = hf_token or os.getenv("HF_TOKEN") or get_token() if not token: raise ValueError("HF_TOKEN is required. Set it in environment or pass as argument.") if not INSPECT_SCRIPT.exists(): raise FileNotFoundError(f"Script not found at {INSPECT_SCRIPT}") print(f"Preparing inspect-ai job for {model_id}") print(f" Task: {task}") print(f" Backend: {backend}") print(f" Hardware: {hardware}") cmd = [ "hf", "jobs", "uv", "run", str(INSPECT_SCRIPT), "--flavor", hardware, "--secrets", f"HF_TOKEN={token}", "--", "--model", model_id, "--task", task, "--backend", backend, "--tensor-parallel-size", str(tensor_parallel_size), ] if limit: cmd.extend(["--limit", str(limit)]) if trust_remote_code: cmd.append("--trust-remote-code") print(f"\nExecuting: {' '.join(cmd)}") try: subprocess.run(cmd, check=True) except subprocess.CalledProcessError as exc: print("hf jobs command failed", file=sys.stderr) raise def main() -> None: parser = argparse.ArgumentParser( description="Submit vLLM-based evaluation jobs to HuggingFace Jobs", formatter_class=argparse.RawDescriptionHelpFormatter, epilog=""" Examples: # Run lighteval with vLLM on A10G GPU python run_vllm_eval_job.py \\ --model meta-llama/Llama-3.2-1B \\ --task "leaderboard|mmlu|5" \\ --framework lighteval \\ --hardware a10g-small # Run inspect-ai on larger model with multi-GPU python run_vllm_eval_job.py \\ --model meta-llama/Llama-3.2-70B \\ --task mmlu \\ --framework inspect \\ --hardware a100-large \\ --tensor-parallel-size 4 # Auto-detect hardware based on model size python run_vllm_eval_job.py \\ --model meta-llama/Llama-3.2-1B \\ --task mmlu \\ --framework inspect # Run with HF Transformers backend (instead of vLLM) python run_vllm_eval_job.py \\ --model microsoft/phi-2 \\ --task mmlu \\ --framework inspect \\ --backend hf Hardware flavors: - t4-small: T4 GPU, good for models < 3B - a10g-small: A10G GPU, good for models 3B-13B - a10g-large: A10G GPU, good for models 13B-34B - a100-large: A100 GPU, good for models 34B+ Frameworks: - lighteval: HuggingFace's lighteval library - inspect: UK AI Safety's inspect-ai library Task formats: - lighteval: "suite|task|num_fewshot" (e.g., "leaderboard|mmlu|5") - inspect: task name (e.g., "mmlu", "gsm8k") """, ) parser.add_argument( "--model", required=True, help="HuggingFace model ID (e.g., meta-llama/Llama-3.2-1B)", ) parser.add_argument( "--task", required=True, help="Evaluation task (format depends on framework)", ) parser.add_argument( "--framework", choices=["lighteval", "inspect"], default="lighteval", help="Evaluation framework to use (default: lighteval)", ) parser.add_argument( "--hardware", default=None, help="Hardware flavor (auto-detected if not specified)", ) parser.add_argument( "--backend", choices=["vllm", "hf", "accelerate"], default="vllm", help="Model backend (default: vllm)", ) parser.add_argument( "--limit", "--max-samples", type=int, default=None, dest="limit", help="Limit number of samples to evaluate", ) parser.add_argument( "--batch-size", type=int, default=1, help="Batch size for evaluation (lighteval only)", ) parser.add_argument( "--tensor-parallel-size", type=int, default=1, help="Number of GPUs for tensor parallelism", ) parser.add_argument( "--trust-remote-code", action="store_true", help="Allow executing remote code from model repository", ) parser.add_argument( "--use-chat-template", action="store_true", help="Apply chat template (lighteval only)", ) args = parser.parse_args() # Auto-detect hardware if not specified hardware = args.hardware or estimate_hardware(args.model) print(f"Using hardware: {hardware}") # Map backend names between frameworks backend = args.backend if args.framework == "lighteval" and backend == "hf": backend = "accelerate" # lighteval uses "accelerate" for HF backend if args.framework == "lighteval": create_lighteval_job( model_id=args.model, tasks=args.task, hardware=hardware, max_samples=args.limit, backend=backend, batch_size=args.batch_size, tensor_parallel_size=args.tensor_parallel_size, trust_remote_code=args.trust_remote_code, use_chat_template=args.use_chat_template, ) else: create_inspect_job( model_id=args.model, task=args.task, hardware=hardware, limit=args.limit, backend=backend if backend != "accelerate" else "hf", tensor_parallel_size=args.tensor_parallel_size, trust_remote_code=args.trust_remote_code, ) if __name__ == "__main__": main() ================================================ FILE: skills/hugging-face-evaluation/scripts/test_extraction.py ================================================ #!/usr/bin/env python3 # /// script # requires-python = ">=3.10" # dependencies = [ # "pyyaml", # ] # /// """ Test script for evaluation extraction functionality. This script demonstrates the table extraction capabilities without requiring HF tokens or making actual API calls. Note: This script imports from evaluation_manager.py (same directory). Run from the scripts/ directory: cd scripts && uv run test_extraction.py """ import yaml from evaluation_manager import ( extract_tables_from_markdown, parse_markdown_table, is_evaluation_table, extract_metrics_from_table ) # Sample README content with various table formats SAMPLE_README = """ # My Awesome Model ## Evaluation Results Here are the benchmark results: | Benchmark | Score | |-----------|-------| | MMLU | 85.2 | | HumanEval | 72.5 | | GSM8K | 91.3 | ### Detailed Breakdown | Category | MMLU | GSM8K | HumanEval | |---------------|-------|-------|-----------| | Performance | 85.2 | 91.3 | 72.5 | ## Other Information This is not an evaluation table: | Feature | Value | |---------|-------| | Size | 7B | | Type | Chat | ## More Results | Benchmark | Accuracy | F1 Score | |---------------|----------|----------| | HellaSwag | 88.9 | 0.87 | | TruthfulQA | 68.7 | 0.65 | """ def test_table_extraction(): """Test markdown table extraction.""" print("=" * 60) print("TEST 1: Table Extraction") print("=" * 60) tables = extract_tables_from_markdown(SAMPLE_README) print(f"Found {len(tables)} tables in the sample README\n") for i, table in enumerate(tables, 1): print(f"Table {i}:") print(table[:100] + "..." if len(table) > 100 else table) print() return tables def test_table_parsing(tables): """Test table parsing.""" print("\n" + "=" * 60) print("TEST 2: Table Parsing") print("=" * 60) parsed_tables = [] for i, table in enumerate(tables, 1): print(f"\nParsing Table {i}:") header, rows = parse_markdown_table(table) print(f" Header: {header}") print(f" Rows: {len(rows)}") for j, row in enumerate(rows[:3], 1): # Show first 3 rows print(f" Row {j}: {row}") if len(rows) > 3: print(f" ... and {len(rows) - 3} more rows") parsed_tables.append((header, rows)) return parsed_tables def test_evaluation_detection(parsed_tables): """Test evaluation table detection.""" print("\n" + "=" * 60) print("TEST 3: Evaluation Table Detection") print("=" * 60) eval_tables = [] for i, (header, rows) in enumerate(parsed_tables, 1): is_eval = is_evaluation_table(header, rows) status = "✓ IS" if is_eval else "✗ NOT" print(f"\nTable {i}: {status} an evaluation table") print(f" Header: {header}") if is_eval: eval_tables.append((header, rows)) print(f"\nFound {len(eval_tables)} evaluation tables") return eval_tables def test_metric_extraction(eval_tables): """Test metric extraction.""" print("\n" + "=" * 60) print("TEST 4: Metric Extraction") print("=" * 60) all_metrics = [] for i, (header, rows) in enumerate(eval_tables, 1): print(f"\nExtracting metrics from table {i}:") metrics = extract_metrics_from_table(header, rows, table_format="auto") print(f" Extracted {len(metrics)} metrics:") for metric in metrics: print(f" - {metric['name']}: {metric['value']} (type: {metric['type']})") all_metrics.extend(metrics) return all_metrics def test_model_index_format(metrics): """Test model-index format generation.""" print("\n" + "=" * 60) print("TEST 5: Model-Index Format") print("=" * 60) model_index = { "model-index": [ { "name": "test-model", "results": [ { "task": {"type": "text-generation"}, "dataset": { "name": "Benchmarks", "type": "benchmark" }, "metrics": metrics, "source": { "name": "Model README", "url": "https://huggingface.co/test/model" } } ] } ] } print("\nGenerated model-index structure:") print(yaml.dump(model_index, sort_keys=False, default_flow_style=False)) def main(): """Run all tests.""" print("\n" + "=" * 60) print("EVALUATION EXTRACTION TEST SUITE") print("=" * 60) print("\nThis test demonstrates the table extraction capabilities") print("without requiring API access or tokens.\n") # Run tests tables = test_table_extraction() parsed_tables = test_table_parsing(tables) eval_tables = test_evaluation_detection(parsed_tables) metrics = test_metric_extraction(eval_tables) test_model_index_format(metrics) # Summary print("\n" + "=" * 60) print("TEST SUMMARY") print("=" * 60) print(f"✓ Found {len(tables)} total tables") print(f"✓ Identified {len(eval_tables)} evaluation tables") print(f"✓ Extracted {len(metrics)} metrics") print("✓ Generated model-index format successfully") print("\n" + "=" * 60) print("All tests completed! The extraction logic is working correctly.") print("=" * 60 + "\n") if __name__ == "__main__": main() ================================================ FILE: skills/hugging-face-jobs/SKILL.md ================================================ --- name: hugging-face-jobs description: This skill should be used when users want to run any workload on Hugging Face Jobs infrastructure. Covers UV scripts, Docker-based jobs, hardware selection, cost estimation, authentication with tokens, secrets management, timeout configuration, and result persistence. Designed for general-purpose compute workloads including data processing, inference, experiments, batch jobs, and any Python-based tasks. Should be invoked for tasks involving cloud compute, GPU workloads, or when users mention running jobs on Hugging Face infrastructure without local setup. license: Complete terms in LICENSE.txt --- # Running Workloads on Hugging Face Jobs ## Overview Run any workload on fully managed Hugging Face infrastructure. No local setup required—jobs run on cloud CPUs, GPUs, or TPUs and can persist results to the Hugging Face Hub. **Common use cases:** - **Data Processing** - Transform, filter, or analyze large datasets - **Batch Inference** - Run inference on thousands of samples - **Experiments & Benchmarks** - Reproducible ML experiments - **Model Training** - Fine-tune models (see `model-trainer` skill for TRL-specific training) - **Synthetic Data Generation** - Generate datasets using LLMs - **Development & Testing** - Test code without local GPU setup - **Scheduled Jobs** - Automate recurring tasks **For model training specifically:** See the `model-trainer` skill for TRL-based training workflows. ## When to Use This Skill Use this skill when users want to: - Run Python workloads on cloud infrastructure - Execute jobs without local GPU/TPU setup - Process data at scale - Run batch inference or experiments - Schedule recurring tasks - Use GPUs/TPUs for any workload - Persist results to the Hugging Face Hub ## Key Directives When assisting with jobs: 1. **ALWAYS use `hf_jobs()` MCP tool** - Submit jobs using `hf_jobs("uv", {...})` or `hf_jobs("run", {...})`. The `script` parameter accepts Python code directly. Do NOT save to local files unless the user explicitly requests it. Pass the script content as a string to `hf_jobs()`. 2. **Always handle authentication** - Jobs that interact with the Hub require `HF_TOKEN` via secrets. See Token Usage section below. 3. **Provide job details after submission** - After submitting, provide job ID, monitoring URL, estimated time, and note that the user can request status checks later. 4. **Set appropriate timeouts** - Default 30min may be insufficient for long-running tasks. ## Prerequisites Checklist Before starting any job, verify: ### ✅ **Account & Authentication** - Hugging Face Account with [Pro](https://hf.co/pro), [Team](https://hf.co/enterprise), or [Enterprise](https://hf.co/enterprise) plan (Jobs require paid plan) - Authenticated login: Check with `hf_whoami()` - **HF_TOKEN for Hub Access** ⚠️ CRITICAL - Required for any Hub operations (push models/datasets, download private repos, etc.) - Token must have appropriate permissions (read for downloads, write for uploads) ### ✅ **Token Usage** (See Token Usage section for details) **When tokens are required:** - Pushing models/datasets to Hub - Accessing private repositories - Using Hub APIs in scripts - Any authenticated Hub operations **How to provide tokens:** ```python # hf_jobs MCP tool — $HF_TOKEN is auto-replaced with real token: {"secrets": {"HF_TOKEN": "$HF_TOKEN"}} # HfApi().run_uv_job() — MUST pass actual token: from huggingface_hub import get_token secrets={"HF_TOKEN": get_token()} ``` **⚠️ CRITICAL:** The `$HF_TOKEN` placeholder is ONLY auto-replaced by the `hf_jobs` MCP tool. When using `HfApi().run_uv_job()`, you MUST pass the real token via `get_token()`. Passing the literal string `"$HF_TOKEN"` results in a 9-character invalid token and 401 errors. ## Token Usage Guide ### Understanding Tokens **What are HF Tokens?** - Authentication credentials for Hugging Face Hub - Required for authenticated operations (push, private repos, API access) - Stored securely on your machine after `hf auth login` **Token Types:** - **Read Token** - Can download models/datasets, read private repos - **Write Token** - Can push models/datasets, create repos, modify content - **Organization Token** - Can act on behalf of an organization ### When Tokens Are Required **Always Required:** - Pushing models/datasets to Hub - Accessing private repositories - Creating new repositories - Modifying existing repositories - Using Hub APIs programmatically **Not Required:** - Downloading public models/datasets - Running jobs that don't interact with Hub - Reading public repository information ### How to Provide Tokens to Jobs #### Method 1: Automatic Token (Recommended) ```python hf_jobs("uv", { "script": "your_script.py", "secrets": {"HF_TOKEN": "$HF_TOKEN"} # ✅ Automatic replacement }) ``` **How it works:** - `$HF_TOKEN` is a placeholder that gets replaced with your actual token - Uses the token from your logged-in session (`hf auth login`) - Most secure and convenient method - Token is encrypted server-side when passed as a secret **Benefits:** - No token exposure in code - Uses your current login session - Automatically updated if you re-login - Works seamlessly with MCP tools #### Method 2: Explicit Token (Not Recommended) ```python hf_jobs("uv", { "script": "your_script.py", "secrets": {"HF_TOKEN": "hf_abc123..."} # ⚠️ Hardcoded token }) ``` **When to use:** - Only if automatic token doesn't work - Testing with a specific token - Organization tokens (use with caution) **Security concerns:** - Token visible in code/logs - Must manually update if token rotates - Risk of token exposure #### Method 3: Environment Variable (Less Secure) ```python hf_jobs("uv", { "script": "your_script.py", "env": {"HF_TOKEN": "hf_abc123..."} # ⚠️ Less secure than secrets }) ``` **Difference from secrets:** - `env` variables are visible in job logs - `secrets` are encrypted server-side - Always prefer `secrets` for tokens ### Using Tokens in Scripts **In your Python script, tokens are available as environment variables:** ```python # /// script # dependencies = ["huggingface-hub"] # /// import os from huggingface_hub import HfApi # Token is automatically available if passed via secrets token = os.environ.get("HF_TOKEN") # Use with Hub API api = HfApi(token=token) # Or let huggingface_hub auto-detect api = HfApi() # Automatically uses HF_TOKEN env var ``` **Best practices:** - Don't hardcode tokens in scripts - Use `os.environ.get("HF_TOKEN")` to access - Let `huggingface_hub` auto-detect when possible - Verify token exists before Hub operations ### Token Verification **Check if you're logged in:** ```python from huggingface_hub import whoami user_info = whoami() # Returns your username if authenticated ``` **Verify token in job:** ```python import os assert "HF_TOKEN" in os.environ, "HF_TOKEN not found!" token = os.environ["HF_TOKEN"] print(f"Token starts with: {token[:7]}...") # Should start with "hf_" ``` ### Common Token Issues **Error: 401 Unauthorized** - **Cause:** Token missing or invalid - **Fix:** Add `secrets={"HF_TOKEN": "$HF_TOKEN"}` to job config - **Verify:** Check `hf_whoami()` works locally **Error: 403 Forbidden** - **Cause:** Token lacks required permissions - **Fix:** Ensure token has write permissions for push operations - **Check:** Token type at https://huggingface.co/settings/tokens **Error: Token not found in environment** - **Cause:** `secrets` not passed or wrong key name - **Fix:** Use `secrets={"HF_TOKEN": "$HF_TOKEN"}` (not `env`) - **Verify:** Script checks `os.environ.get("HF_TOKEN")` **Error: Repository access denied** - **Cause:** Token doesn't have access to private repo - **Fix:** Use token from account with access - **Check:** Verify repo visibility and your permissions ### Token Security Best Practices 1. **Never commit tokens** - Use `$HF_TOKEN` placeholder or environment variables 2. **Use secrets, not env** - Secrets are encrypted server-side 3. **Rotate tokens regularly** - Generate new tokens periodically 4. **Use minimal permissions** - Create tokens with only needed permissions 5. **Don't share tokens** - Each user should use their own token 6. **Monitor token usage** - Check token activity in Hub settings ### Complete Token Example ```python # Example: Push results to Hub hf_jobs("uv", { "script": """ # /// script # dependencies = ["huggingface-hub", "datasets"] # /// import os from huggingface_hub import HfApi from datasets import Dataset # Verify token is available assert "HF_TOKEN" in os.environ, "HF_TOKEN required!" # Use token for Hub operations api = HfApi(token=os.environ["HF_TOKEN"]) # Create and push dataset data = {"text": ["Hello", "World"]} dataset = Dataset.from_dict(data) dataset.push_to_hub("username/my-dataset", token=os.environ["HF_TOKEN"]) print("✅ Dataset pushed successfully!") """, "flavor": "cpu-basic", "timeout": "30m", "secrets": {"HF_TOKEN": "$HF_TOKEN"} # ✅ Token provided securely }) ``` ## Quick Start: Two Approaches ### Approach 1: UV Scripts (Recommended) UV scripts use PEP 723 inline dependencies for clean, self-contained workloads. **MCP Tool:** ```python hf_jobs("uv", { "script": """ # /// script # dependencies = ["transformers", "torch"] # /// from transformers import pipeline import torch # Your workload here classifier = pipeline("sentiment-analysis") result = classifier("I love Hugging Face!") print(result) """, "flavor": "cpu-basic", "timeout": "30m" }) ``` **CLI Equivalent:** ```bash hf jobs uv run my_script.py --flavor cpu-basic --timeout 30m ``` **Python API:** ```python from huggingface_hub import run_uv_job run_uv_job("my_script.py", flavor="cpu-basic", timeout="30m") ``` **Benefits:** Direct MCP tool usage, clean code, dependencies declared inline, no file saving required **When to use:** Default choice for all workloads, custom logic, any scenario requiring `hf_jobs()` #### Custom Docker Images for UV Scripts By default, UV scripts use `ghcr.io/astral-sh/uv:python3.12-bookworm-slim`. For ML workloads with complex dependencies, use pre-built images: ```python hf_jobs("uv", { "script": "inference.py", "image": "vllm/vllm-openai:latest", # Pre-built image with vLLM "flavor": "a10g-large" }) ``` **CLI:** ```bash hf jobs uv run --image vllm/vllm-openai:latest --flavor a10g-large inference.py ``` **Benefits:** Faster startup, pre-installed dependencies, optimized for specific frameworks #### Python Version By default, UV scripts use Python 3.12. Specify a different version: ```python hf_jobs("uv", { "script": "my_script.py", "python": "3.11", # Use Python 3.11 "flavor": "cpu-basic" }) ``` **Python API:** ```python from huggingface_hub import run_uv_job run_uv_job("my_script.py", python="3.11") ``` #### Working with Scripts ⚠️ **Important:** There are *two* "script path" stories depending on how you run Jobs: - **Using the `hf_jobs()` MCP tool (recommended in this repo)**: the `script` value must be **inline code** (a string) or a **URL**. A local filesystem path (like `"./scripts/foo.py"`) won't exist inside the remote container. - **Using the `hf jobs uv run` CLI**: local file paths **do work** (the CLI uploads your script). **Common mistake with `hf_jobs()` MCP tool:** ```python # ❌ Will fail (remote container can't see your local path) hf_jobs("uv", {"script": "./scripts/foo.py"}) ``` **Correct patterns with `hf_jobs()` MCP tool:** ```python # ✅ Inline: read the local script file and pass its *contents* from pathlib import Path script = Path("hf-jobs/scripts/foo.py").read_text() hf_jobs("uv", {"script": script}) # ✅ URL: host the script somewhere reachable hf_jobs("uv", {"script": "https://huggingface.co/datasets/uv-scripts/.../raw/main/foo.py"}) # ✅ URL from GitHub hf_jobs("uv", {"script": "https://raw.githubusercontent.com/huggingface/trl/main/trl/scripts/sft.py"}) ``` **CLI equivalent (local paths supported):** ```bash hf jobs uv run ./scripts/foo.py -- --your --args ``` #### Adding Dependencies at Runtime Add extra dependencies beyond what's in the PEP 723 header: ```python hf_jobs("uv", { "script": "inference.py", "dependencies": ["transformers", "torch>=2.0"], # Extra deps "flavor": "a10g-small" }) ``` **Python API:** ```python from huggingface_hub import run_uv_job run_uv_job("inference.py", dependencies=["transformers", "torch>=2.0"]) ``` ### Approach 2: Docker-Based Jobs Run jobs with custom Docker images and commands. **MCP Tool:** ```python hf_jobs("run", { "image": "python:3.12", "command": ["python", "-c", "print('Hello from HF Jobs!')"], "flavor": "cpu-basic", "timeout": "30m" }) ``` **CLI Equivalent:** ```bash hf jobs run python:3.12 python -c "print('Hello from HF Jobs!')" ``` **Python API:** ```python from huggingface_hub import run_job run_job(image="python:3.12", command=["python", "-c", "print('Hello!')"], flavor="cpu-basic") ``` **Benefits:** Full Docker control, use pre-built images, run any command **When to use:** Need specific Docker images, non-Python workloads, complex environments **Example with GPU:** ```python hf_jobs("run", { "image": "pytorch/pytorch:2.6.0-cuda12.4-cudnn9-devel", "command": ["python", "-c", "import torch; print(torch.cuda.get_device_name())"], "flavor": "a10g-small", "timeout": "1h" }) ``` **Using Hugging Face Spaces as Images:** You can use Docker images from HF Spaces: ```python hf_jobs("run", { "image": "hf.co/spaces/lhoestq/duckdb", # Space as Docker image "command": ["duckdb", "-c", "SELECT 'Hello from DuckDB!'"], "flavor": "cpu-basic" }) ``` **CLI:** ```bash hf jobs run hf.co/spaces/lhoestq/duckdb duckdb -c "SELECT 'Hello!'" ``` ### Finding More UV Scripts on Hub The `uv-scripts` organization provides ready-to-use UV scripts stored as datasets on Hugging Face Hub: ```python # Discover available UV script collections dataset_search({"author": "uv-scripts", "sort": "downloads", "limit": 20}) # Explore a specific collection hub_repo_details(["uv-scripts/classification"], repo_type="dataset", include_readme=True) ``` **Popular collections:** OCR, classification, synthetic-data, vLLM, dataset-creation ## Hardware Selection > **Reference:** [HF Jobs Hardware Docs](https://huggingface.co/docs/hub/en/spaces-config-reference) (updated 07/2025) | Workload Type | Recommended Hardware | Use Case | |---------------|---------------------|----------| | Data processing, testing | `cpu-basic`, `cpu-upgrade` | Lightweight tasks | | Small models, demos | `t4-small` | <1B models, quick tests | | Medium models | `t4-medium`, `l4x1` | 1-7B models | | Large models, production | `a10g-small`, `a10g-large` | 7-13B models | | Very large models | `a100-large` | 13B+ models | | Batch inference | `a10g-large`, `a100-large` | High-throughput | | Multi-GPU workloads | `l4x4`, `a10g-largex2`, `a10g-largex4` | Parallel/large models | | TPU workloads | `v5e-1x1`, `v5e-2x2`, `v5e-2x4` | JAX/Flax, TPU-optimized | **All Available Flavors:** - **CPU:** `cpu-basic`, `cpu-upgrade` - **GPU:** `t4-small`, `t4-medium`, `l4x1`, `l4x4`, `a10g-small`, `a10g-large`, `a10g-largex2`, `a10g-largex4`, `a100-large` - **TPU:** `v5e-1x1`, `v5e-2x2`, `v5e-2x4` **Guidelines:** - Start with smaller hardware for testing - Scale up based on actual needs - Use multi-GPU for parallel workloads or large models - Use TPUs for JAX/Flax workloads - See `references/hardware_guide.md` for detailed specifications ## Critical: Saving Results **⚠️ EPHEMERAL ENVIRONMENT—MUST PERSIST RESULTS** The Jobs environment is temporary. All files are deleted when the job ends. If results aren't persisted, **ALL WORK IS LOST**. ### Persistence Options **1. Push to Hugging Face Hub (Recommended)** ```python # Push models model.push_to_hub("username/model-name", token=os.environ["HF_TOKEN"]) # Push datasets dataset.push_to_hub("username/dataset-name", token=os.environ["HF_TOKEN"]) # Push artifacts api.upload_file( path_or_fileobj="results.json", path_in_repo="results.json", repo_id="username/results", token=os.environ["HF_TOKEN"] ) ``` **2. Use External Storage** ```python # Upload to S3, GCS, etc. import boto3 s3 = boto3.client('s3') s3.upload_file('results.json', 'my-bucket', 'results.json') ``` **3. Send Results via API** ```python # POST results to your API import requests requests.post("https://your-api.com/results", json=results) ``` ### Required Configuration for Hub Push **In job submission:** ```python # hf_jobs MCP tool: {"secrets": {"HF_TOKEN": "$HF_TOKEN"}} # auto-replaced # HfApi().run_uv_job(): from huggingface_hub import get_token secrets={"HF_TOKEN": get_token()} # must pass real token ``` **In script:** ```python import os from huggingface_hub import HfApi # Token automatically available from secrets api = HfApi(token=os.environ.get("HF_TOKEN")) # Push your results api.upload_file(...) ``` ### Verification Checklist Before submitting: - [ ] Results persistence method chosen - [ ] Token in secrets if using Hub (MCP: `"$HF_TOKEN"`, Python API: `get_token()`) - [ ] Script handles missing token gracefully - [ ] Test persistence path works **See:** `references/hub_saving.md` for detailed Hub persistence guide ## Timeout Management **⚠️ DEFAULT: 30 MINUTES** Jobs automatically stop after the timeout. For long-running tasks like training, always set a custom timeout. ### Setting Timeouts **MCP Tool:** ```python { "timeout": "2h" # 2 hours } ``` **Supported formats:** - Integer/float: seconds (e.g., `300` = 5 minutes) - String with suffix: `"5m"` (minutes), `"2h"` (hours), `"1d"` (days) - Examples: `"90m"`, `"2h"`, `"1.5h"`, `300`, `"1d"` **Python API:** ```python from huggingface_hub import run_job, run_uv_job run_job(image="python:3.12", command=[...], timeout="2h") run_uv_job("script.py", timeout=7200) # 2 hours in seconds ``` ### Timeout Guidelines | Scenario | Recommended | Notes | |----------|-------------|-------| | Quick test | 10-30 min | Verify setup | | Data processing | 1-2 hours | Depends on data size | | Batch inference | 2-4 hours | Large batches | | Experiments | 4-8 hours | Multiple runs | | Long-running | 8-24 hours | Production workloads | **Always add 20-30% buffer** for setup, network delays, and cleanup. **On timeout:** Job killed immediately, all unsaved progress lost ## Cost Estimation **General guidelines:** ``` Total Cost = (Hours of runtime) × (Cost per hour) ``` **Example calculations:** **Quick test:** - Hardware: cpu-basic ($0.10/hour) - Time: 15 minutes (0.25 hours) - Cost: $0.03 **Data processing:** - Hardware: l4x1 ($2.50/hour) - Time: 2 hours - Cost: $5.00 **Batch inference:** - Hardware: a10g-large ($5/hour) - Time: 4 hours - Cost: $20.00 **Cost optimization tips:** 1. Start small - Test on cpu-basic or t4-small 2. Monitor runtime - Set appropriate timeouts 3. Use checkpoints - Resume if job fails 4. Optimize code - Reduce unnecessary compute 5. Choose right hardware - Don't over-provision ## Monitoring and Tracking ### Check Job Status **MCP Tool:** ```python # List all jobs hf_jobs("ps") # Inspect specific job hf_jobs("inspect", {"job_id": "your-job-id"}) # View logs hf_jobs("logs", {"job_id": "your-job-id"}) # Cancel a job hf_jobs("cancel", {"job_id": "your-job-id"}) ``` **Python API:** ```python from huggingface_hub import list_jobs, inspect_job, fetch_job_logs, cancel_job # List your jobs jobs = list_jobs() # List running jobs only running = [j for j in list_jobs() if j.status.stage == "RUNNING"] # Inspect specific job job_info = inspect_job(job_id="your-job-id") # View logs for log in fetch_job_logs(job_id="your-job-id"): print(log) # Cancel a job cancel_job(job_id="your-job-id") ``` **CLI:** ```bash hf jobs ps # List jobs hf jobs logs # View logs hf jobs cancel # Cancel job ``` **Remember:** Wait for user to request status checks. Avoid polling repeatedly. ### Job URLs After submission, jobs have monitoring URLs: ``` https://huggingface.co/jobs/username/job-id ``` View logs, status, and details in the browser. ### Wait for Multiple Jobs ```python import time from huggingface_hub import inspect_job, run_job # Run multiple jobs jobs = [run_job(image=img, command=cmd) for img, cmd in workloads] # Wait for all to complete for job in jobs: while inspect_job(job_id=job.id).status.stage not in ("COMPLETED", "ERROR"): time.sleep(10) ``` ## Scheduled Jobs Run jobs on a schedule using CRON expressions or predefined schedules. **MCP Tool:** ```python # Schedule a UV script that runs every hour hf_jobs("scheduled uv", { "script": "your_script.py", "schedule": "@hourly", "flavor": "cpu-basic" }) # Schedule with CRON syntax hf_jobs("scheduled uv", { "script": "your_script.py", "schedule": "0 9 * * 1", # 9 AM every Monday "flavor": "cpu-basic" }) # Schedule a Docker-based job hf_jobs("scheduled run", { "image": "python:3.12", "command": ["python", "-c", "print('Scheduled!')"], "schedule": "@daily", "flavor": "cpu-basic" }) ``` **Python API:** ```python from huggingface_hub import create_scheduled_job, create_scheduled_uv_job # Schedule a Docker job create_scheduled_job( image="python:3.12", command=["python", "-c", "print('Running on schedule!')"], schedule="@hourly" ) # Schedule a UV script create_scheduled_uv_job("my_script.py", schedule="@daily", flavor="cpu-basic") # Schedule with GPU create_scheduled_uv_job( "ml_inference.py", schedule="0 */6 * * *", # Every 6 hours flavor="a10g-small" ) ``` **Available schedules:** - `@annually`, `@yearly` - Once per year - `@monthly` - Once per month - `@weekly` - Once per week - `@daily` - Once per day - `@hourly` - Once per hour - CRON expression - Custom schedule (e.g., `"*/5 * * * *"` for every 5 minutes) **Manage scheduled jobs:** ```python # MCP Tool hf_jobs("scheduled ps") # List scheduled jobs hf_jobs("scheduled inspect", {"job_id": "..."}) # Inspect details hf_jobs("scheduled suspend", {"job_id": "..."}) # Pause hf_jobs("scheduled resume", {"job_id": "..."}) # Resume hf_jobs("scheduled delete", {"job_id": "..."}) # Delete ``` **Python API for management:** ```python from huggingface_hub import ( list_scheduled_jobs, inspect_scheduled_job, suspend_scheduled_job, resume_scheduled_job, delete_scheduled_job ) # List all scheduled jobs scheduled = list_scheduled_jobs() # Inspect a scheduled job info = inspect_scheduled_job(scheduled_job_id) # Suspend (pause) a scheduled job suspend_scheduled_job(scheduled_job_id) # Resume a scheduled job resume_scheduled_job(scheduled_job_id) # Delete a scheduled job delete_scheduled_job(scheduled_job_id) ``` ## Webhooks: Trigger Jobs on Events Trigger jobs automatically when changes happen in Hugging Face repositories. **Python API:** ```python from huggingface_hub import create_webhook # Create webhook that triggers a job when a repo changes webhook = create_webhook( job_id=job.id, watched=[ {"type": "user", "name": "your-username"}, {"type": "org", "name": "your-org-name"} ], domains=["repo", "discussion"], secret="your-secret" ) ``` **How it works:** 1. Webhook listens for changes in watched repositories 2. When triggered, the job runs with `WEBHOOK_PAYLOAD` environment variable 3. Your script can parse the payload to understand what changed **Use cases:** - Auto-process new datasets when uploaded - Trigger inference when models are updated - Run tests when code changes - Generate reports on repository activity **Access webhook payload in script:** ```python import os import json payload = json.loads(os.environ.get("WEBHOOK_PAYLOAD", "{}")) print(f"Event type: {payload.get('event', {}).get('action')}") ``` See [Webhooks Documentation](https://huggingface.co/docs/huggingface_hub/guides/webhooks) for more details. ## Common Workload Patterns This repository ships ready-to-run UV scripts in `hf-jobs/scripts/`. Prefer using them instead of inventing new templates. ### Pattern 1: Dataset → Model Responses (vLLM) — `scripts/generate-responses.py` **What it does:** loads a Hub dataset (chat `messages` or a `prompt` column), applies a model chat template, generates responses with vLLM, and **pushes** the output dataset + dataset card back to the Hub. **Requires:** GPU + **write** token (it pushes a dataset). ```python from pathlib import Path script = Path("hf-jobs/scripts/generate-responses.py").read_text() hf_jobs("uv", { "script": script, "script_args": [ "username/input-dataset", "username/output-dataset", "--messages-column", "messages", "--model-id", "Qwen/Qwen3-30B-A3B-Instruct-2507", "--temperature", "0.7", "--top-p", "0.8", "--max-tokens", "2048", ], "flavor": "a10g-large", "timeout": "4h", "secrets": {"HF_TOKEN": "$HF_TOKEN"}, }) ``` ### Pattern 2: CoT Self-Instruct Synthetic Data — `scripts/cot-self-instruct.py` **What it does:** generates synthetic prompts/answers via CoT Self-Instruct, optionally filters outputs (answer-consistency / RIP), then **pushes** the generated dataset + dataset card to the Hub. **Requires:** GPU + **write** token (it pushes a dataset). ```python from pathlib import Path script = Path("hf-jobs/scripts/cot-self-instruct.py").read_text() hf_jobs("uv", { "script": script, "script_args": [ "--seed-dataset", "davanstrien/s1k-reasoning", "--output-dataset", "username/synthetic-math", "--task-type", "reasoning", "--num-samples", "5000", "--filter-method", "answer-consistency", ], "flavor": "l4x4", "timeout": "8h", "secrets": {"HF_TOKEN": "$HF_TOKEN"}, }) ``` ### Pattern 3: Streaming Dataset Stats (Polars + HF Hub) — `scripts/finepdfs-stats.py` **What it does:** scans parquet directly from Hub (no 300GB download), computes temporal stats, and (optionally) uploads results to a Hub dataset repo. **Requires:** CPU is often enough; token needed **only** if you pass `--output-repo` (upload). ```python from pathlib import Path script = Path("hf-jobs/scripts/finepdfs-stats.py").read_text() hf_jobs("uv", { "script": script, "script_args": [ "--limit", "10000", "--show-plan", "--output-repo", "username/finepdfs-temporal-stats", ], "flavor": "cpu-upgrade", "timeout": "2h", "env": {"HF_XET_HIGH_PERFORMANCE": "1"}, "secrets": {"HF_TOKEN": "$HF_TOKEN"}, }) ``` ## Common Failure Modes ### Out of Memory (OOM) **Fix:** 1. Reduce batch size or data chunk size 2. Process data in smaller batches 3. Upgrade hardware: cpu → t4 → a10g → a100 ### Job Timeout **Fix:** 1. Check logs for actual runtime 2. Increase timeout with buffer: `"timeout": "3h"` 3. Optimize code for faster execution 4. Process data in chunks ### Hub Push Failures **Fix:** 1. Add token to secrets: MCP uses `"$HF_TOKEN"` (auto-replaced), Python API uses `get_token()` (must pass real token) 2. Verify token in script: `assert "HF_TOKEN" in os.environ` 3. Check token permissions 4. Verify repo exists or can be created ### Missing Dependencies **Fix:** Add to PEP 723 header: ```python # /// script # dependencies = ["package1", "package2>=1.0.0"] # /// ``` ### Authentication Errors **Fix:** 1. Check `hf_whoami()` works locally 2. Verify token in secrets — MCP: `"$HF_TOKEN"`, Python API: `get_token()` (NOT `"$HF_TOKEN"`) 3. Re-login: `hf auth login` 4. Check token has required permissions ## Troubleshooting **Common issues:** - Job times out → Increase timeout, optimize code - Results not saved → Check persistence method, verify HF_TOKEN - Out of Memory → Reduce batch size, upgrade hardware - Import errors → Add dependencies to PEP 723 header - Authentication errors → Check token, verify secrets parameter **See:** `references/troubleshooting.md` for complete troubleshooting guide ## Resources ### References (In This Skill) - `references/token_usage.md` - Complete token usage guide - `references/hardware_guide.md` - Hardware specs and selection - `references/hub_saving.md` - Hub persistence guide - `references/troubleshooting.md` - Common issues and solutions ### Scripts (In This Skill) - `scripts/generate-responses.py` - vLLM batch generation: dataset → responses → push to Hub - `scripts/cot-self-instruct.py` - CoT Self-Instruct synthetic data generation + filtering → push to Hub - `scripts/finepdfs-stats.py` - Polars streaming stats over `finepdfs-edu` parquet on Hub (optional push) ### External Links **Official Documentation:** - [HF Jobs Guide](https://huggingface.co/docs/huggingface_hub/guides/jobs) - Main documentation - [HF Jobs CLI Reference](https://huggingface.co/docs/huggingface_hub/guides/cli#hf-jobs) - Command line interface - [HF Jobs API Reference](https://huggingface.co/docs/huggingface_hub/package_reference/hf_api) - Python API details - [Hardware Flavors Reference](https://huggingface.co/docs/hub/en/spaces-config-reference) - Available hardware **Related Tools:** - [UV Scripts Guide](https://docs.astral.sh/uv/guides/scripts/) - PEP 723 inline dependencies - [UV Scripts Organization](https://huggingface.co/uv-scripts) - Community UV script collection - [HF Hub Authentication](https://huggingface.co/docs/huggingface_hub/quick-start#authentication) - Token setup - [Webhooks Documentation](https://huggingface.co/docs/huggingface_hub/guides/webhooks) - Event triggers ## Key Takeaways 1. **Submit scripts inline** - The `script` parameter accepts Python code directly; no file saving required unless user requests 2. **Jobs are asynchronous** - Don't wait/poll; let user check when ready 3. **Always set timeout** - Default 30 min may be insufficient; set appropriate timeout 4. **Always persist results** - Environment is ephemeral; without persistence, all work is lost 5. **Use tokens securely** - MCP: `secrets={"HF_TOKEN": "$HF_TOKEN"}`, Python API: `secrets={"HF_TOKEN": get_token()}` — `"$HF_TOKEN"` only works with MCP tool 6. **Choose appropriate hardware** - Start small, scale up based on needs (see hardware guide) 7. **Use UV scripts** - Default to `hf_jobs("uv", {...})` with inline scripts for Python workloads 8. **Handle authentication** - Verify tokens are available before Hub operations 9. **Monitor jobs** - Provide job URLs and status check commands 10. **Optimize costs** - Choose right hardware, set appropriate timeouts ## Quick Reference: MCP Tool vs CLI vs Python API | Operation | MCP Tool | CLI | Python API | |-----------|----------|-----|------------| | Run UV script | `hf_jobs("uv", {...})` | `hf jobs uv run script.py` | `run_uv_job("script.py")` | | Run Docker job | `hf_jobs("run", {...})` | `hf jobs run image cmd` | `run_job(image, command)` | | List jobs | `hf_jobs("ps")` | `hf jobs ps` | `list_jobs()` | | View logs | `hf_jobs("logs", {...})` | `hf jobs logs ` | `fetch_job_logs(job_id)` | | Cancel job | `hf_jobs("cancel", {...})` | `hf jobs cancel ` | `cancel_job(job_id)` | | Schedule UV | `hf_jobs("scheduled uv", {...})` | - | `create_scheduled_uv_job()` | | Schedule Docker | `hf_jobs("scheduled run", {...})` | - | `create_scheduled_job()` | ================================================ FILE: skills/hugging-face-jobs/index.html ================================================ hf-jobs - Run Workloads on Hugging Face Jobs

Agent Skill : hf-jobs

Run any workload on Hugging Face Jobs.

Use this skill when you want to run GPU/CPU workloads (batch inference, synthetic data generation, dataset stats, experiments) on Hugging Face Jobs, with correct token handling and result persistence back to the Hub.

Skill Name: hf-jobs

Main Documentation: hf-jobs/SKILL.md

Scripts Directory: hf-jobs/scripts/

References Directory: hf-jobs/references/

Overview

This skill focuses on running real workloads via Hugging Face Jobs. It includes ready-to-run UV scripts and guides for authentication (HF tokens), secrets vs env vars, timeouts, hardware selection, and pushing results to the Hub.

Core Documentation

  • SKILL.md
    hf-jobs/SKILL.md
    Complete skill documentation (how to submit jobs, tokens/secrets, timeouts, persistence, and how to use the bundled scripts)

References

  • token_usage.md
    hf-jobs/references/token_usage.md
    Token best practices: secrets vs env, permissions, common errors (401/403), and secure patterns
  • hub_saving.md
    hf-jobs/references/hub_saving.md
    How to persist results: push datasets/models/files to the Hub (ephemeral job filesystem)
  • hardware_guide.md
    hf-jobs/references/hardware_guide.md
    Flavor selection guidance for CPU/GPU/TPU workloads
  • troubleshooting.md
    hf-jobs/references/troubleshooting.md
    Common failure modes (timeouts, missing deps, OOM, auth) and fixes

Scripts

  • generate-responses.py
    hf-jobs/scripts/generate-responses.py
    vLLM batch generation: load prompts/messages from a dataset, generate responses, push dataset + card to Hub
  • cot-self-instruct.py
    hf-jobs/scripts/cot-self-instruct.py
    CoT Self-Instruct synthetic data generation (reasoning/instruction) + optional filtering, pushes dataset + card
  • finepdfs-stats.py
    hf-jobs/scripts/finepdfs-stats.py
    Polars streaming stats over Hub parquet (finepdfs-edu); optional upload of computed stats to a dataset repo
================================================ FILE: skills/hugging-face-jobs/references/hardware_guide.md ================================================ # Hardware Selection Guide Choosing the right hardware (flavor) is critical for cost-effective workloads. > **Reference:** [HF Jobs Hardware Documentation](https://huggingface.co/docs/hub/en/spaces-config-reference) (updated 07/2025) ## Available Hardware ### CPU Flavors | Flavor | Description | Use Case | |--------|-------------|----------| | `cpu-basic` | Basic CPU instance | Testing, lightweight scripts | | `cpu-upgrade` | Enhanced CPU instance | Data processing, parallel workloads | **Use cases:** Data processing, testing scripts, lightweight workloads **Not recommended for:** Model training, GPU-accelerated workloads ### GPU Flavors | Flavor | GPU | VRAM | Use Case | |--------|-----|------|----------| | `t4-small` | NVIDIA T4 | 16GB | <1B models, demos, quick tests | | `t4-medium` | NVIDIA T4 | 16GB | 1-3B models, development | | `l4x1` | NVIDIA L4 | 24GB | 3-7B models, efficient workloads | | `l4x4` | 4x NVIDIA L4 | 96GB | Multi-GPU, parallel workloads | | `a10g-small` | NVIDIA A10G | 24GB | 3-7B models, production | | `a10g-large` | NVIDIA A10G | 24GB | 7-13B models, batch inference | | `a10g-largex2` | 2x NVIDIA A10G | 48GB | Multi-GPU, large models | | `a10g-largex4` | 4x NVIDIA A10G | 96GB | Multi-GPU, very large models | | `a100-large` | NVIDIA A100 | 40GB | 13B+ models, fastest GPU option | ### TPU Flavors | Flavor | Configuration | Use Case | |--------|---------------|----------| | `v5e-1x1` | TPU v5e (1x1) | Small TPU workloads | | `v5e-2x2` | TPU v5e (2x2) | Medium TPU workloads | | `v5e-2x4` | TPU v5e (2x4) | Large TPU workloads | **TPU Use Cases:** - JAX/Flax model training - Large-scale inference - TPU-optimized workloads ## Selection Guidelines ### By Workload Type **Data Processing** - **Recommended:** `cpu-upgrade` or `l4x1` - **Use case:** Transform, filter, analyze datasets - **Batch size:** Depends on data size - **Time:** Varies by dataset size **Batch Inference** - **Recommended:** `a10g-large` or `a100-large` - **Use case:** Run inference on thousands of samples - **Batch size:** 8-32 depending on model - **Time:** Depends on number of samples **Experiments & Benchmarks** - **Recommended:** `a10g-small` or `a10g-large` - **Use case:** Reproducible ML experiments - **Batch size:** Varies - **Time:** Depends on experiment complexity **Model Training** (see `model-trainer` skill for details) - **Recommended:** See model-trainer skill - **Use case:** Fine-tuning models - **Batch size:** Depends on model size - **Time:** Hours to days **Synthetic Data Generation** - **Recommended:** `a10g-large` or `a100-large` - **Use case:** Generate datasets using LLMs - **Batch size:** Depends on generation method - **Time:** Hours for large datasets ### By Budget **Minimal Budget (<$5 total)** - Use `cpu-basic` or `t4-small` - Process small datasets - Quick tests and demos **Small Budget ($5-20)** - Use `t4-medium` or `a10g-small` - Process medium datasets - Run experiments **Medium Budget ($20-50)** - Use `a10g-small` or `a10g-large` - Process large datasets - Production workloads **Large Budget ($50-200)** - Use `a10g-large` or `a100-large` - Large-scale processing - Multiple experiments ### By Model Size (for inference/processing) **Tiny Models (<1B parameters)** - **Recommended:** `t4-small` - **Example:** Qwen2.5-0.5B, TinyLlama - **Batch size:** 8-16 **Small Models (1-3B parameters)** - **Recommended:** `t4-medium` or `a10g-small` - **Example:** Qwen2.5-1.5B, Phi-2 - **Batch size:** 4-8 **Medium Models (3-7B parameters)** - **Recommended:** `a10g-small` or `a10g-large` - **Example:** Qwen2.5-7B, Mistral-7B - **Batch size:** 2-4 **Large Models (7-13B parameters)** - **Recommended:** `a10g-large` or `a100-large` - **Example:** Llama-3-8B - **Batch size:** 1-2 **Very Large Models (13B+ parameters)** - **Recommended:** `a100-large` - **Example:** Llama-3-13B, Llama-3-70B - **Batch size:** 1 ## Memory Considerations ### Estimating Memory Requirements **For inference:** ``` Memory (GB) ≈ (Model params in billions) × 2-4 ``` **For training:** ``` Memory (GB) ≈ (Model params in billions) × 20 (full) or × 4 (LoRA) ``` **Examples:** - Qwen2.5-0.5B inference: ~1-2GB ✅ fits t4-small - Qwen2.5-7B inference: ~14-28GB ✅ fits a10g-large - Qwen2.5-7B training: ~140GB ❌ not feasible without LoRA ### Memory Optimization If hitting memory limits: 1. **Reduce batch size** ```python batch_size = 1 ``` 2. **Process in chunks** ```python for chunk in chunks: process(chunk) ``` 3. **Use smaller models** - Use quantized models - Use LoRA adapters 4. **Upgrade hardware** - cpu → t4 → a10g → a100 ## Cost Estimation ### Formula ``` Total Cost = (Hours of runtime) × (Cost per hour) ``` ### Example Calculations **Data processing:** - Hardware: cpu-upgrade ($0.50/hour) - Time: 1 hour - Cost: $0.50 **Batch inference:** - Hardware: a10g-large ($5/hour) - Time: 2 hours - Cost: $10.00 **Experiments:** - Hardware: a10g-small ($3.50/hour) - Time: 4 hours - Cost: $14.00 ### Cost Optimization Tips 1. **Start small:** Test on cpu-basic or t4-small 2. **Monitor runtime:** Set appropriate timeouts 3. **Optimize code:** Reduce unnecessary compute 4. **Choose right hardware:** Don't over-provision 5. **Use checkpoints:** Resume if job fails 6. **Monitor costs:** Check running jobs regularly ## Multi-GPU Workloads Multi-GPU flavors automatically distribute workloads: **Multi-GPU flavors:** - `l4x4` - 4x L4 GPUs (96GB total VRAM) - `a10g-largex2` - 2x A10G GPUs (48GB total VRAM) - `a10g-largex4` - 4x A10G GPUs (96GB total VRAM) **When to use:** - Large models (>13B parameters) - Need faster processing (linear speedup) - Large datasets (>100K samples) - Parallel workloads - Tensor parallelism for inference **MCP Tool Example:** ```python hf_jobs("uv", { "script": "process.py", "flavor": "a10g-largex2", # 2 GPUs "timeout": "4h", "secrets": {"HF_TOKEN": "$HF_TOKEN"} }) ``` **CLI Equivalent:** ```bash hf jobs uv run process.py --flavor a10g-largex2 --timeout 4h ``` ## Choosing Between Options ### CPU vs GPU **Choose CPU when:** - No GPU acceleration needed - Data processing only - Budget constrained - Simple workloads **Choose GPU when:** - Model inference/training - GPU-accelerated libraries - Need faster processing - Large models ### a10g vs a100 **Choose a10g when:** - Model <13B parameters - Budget conscious - Processing time not critical **Choose a100 when:** - Model 13B+ parameters - Need fastest processing - Memory requirements high - Budget allows ### Single vs Multi-GPU **Choose single GPU when:** - Model <7B parameters - Budget constrained - Simpler debugging **Choose multi-GPU when:** - Model >13B parameters - Need faster processing - Large batch sizes required - Cost-effective for large jobs ## Quick Reference ### All Available Flavors ```python # Official flavor list (updated 07/2025) FLAVORS = { # CPU "cpu-basic", # Testing, lightweight "cpu-upgrade", # Data processing # GPU - Single "t4-small", # 16GB - <1B models "t4-medium", # 16GB - 1-3B models "l4x1", # 24GB - 3-7B models "a10g-small", # 24GB - 3-7B production "a10g-large", # 24GB - 7-13B models "a100-large", # 40GB - 13B+ models # GPU - Multi "l4x4", # 4x L4 (96GB total) "a10g-largex2", # 2x A10G (48GB total) "a10g-largex4", # 4x A10G (96GB total) # TPU "v5e-1x1", # TPU v5e 1x1 "v5e-2x2", # TPU v5e 2x2 "v5e-2x4", # TPU v5e 2x4 } ``` ### Workload → Hardware Mapping ```python HARDWARE_MAP = { "data_processing": "cpu-upgrade", "batch_inference_small": "t4-small", "batch_inference_medium": "a10g-large", "batch_inference_large": "a100-large", "experiments": "a10g-small", "tpu_workloads": "v5e-1x1", "training": "see model-trainer skill" } ``` ### CLI Examples ```bash # CPU job hf jobs run python:3.12 python script.py # GPU job hf jobs run --flavor a10g-large pytorch/pytorch:2.6.0-cuda12.4-cudnn9-devel python script.py # TPU job hf jobs run --flavor v5e-1x1 your-tpu-image python script.py # UV script with GPU hf jobs uv run --flavor a10g-small my_script.py ``` ================================================ FILE: skills/hugging-face-jobs/references/hub_saving.md ================================================ # Saving Results to Hugging Face Hub **⚠️ CRITICAL:** Job environments are ephemeral. ALL results are lost when a job completes unless persisted to the Hub or external storage. ## Why Persistence is Required When running on Hugging Face Jobs: - Environment is temporary - All files deleted on job completion - No local disk persistence - Cannot access results after job ends **Without persistence, all work is permanently lost.** ## Persistence Options ### Option 1: Push to Hugging Face Hub (Recommended) **For models:** ```python from transformers import AutoModel model.push_to_hub("username/model-name", token=os.environ.get("HF_TOKEN")) ``` **For datasets:** ```python from datasets import Dataset dataset.push_to_hub("username/dataset-name", token=os.environ.get("HF_TOKEN")) ``` **For files/artifacts:** ```python from huggingface_hub import HfApi api = HfApi(token=os.environ.get("HF_TOKEN")) api.upload_file( path_or_fileobj="results.json", path_in_repo="results.json", repo_id="username/results", repo_type="dataset" ) ``` ### Option 2: External Storage **S3:** ```python import boto3 s3 = boto3.client('s3') s3.upload_file('results.json', 'my-bucket', 'results.json') ``` **Google Cloud Storage:** ```python from google.cloud import storage client = storage.Client() bucket = client.bucket('my-bucket') blob = bucket.blob('results.json') blob.upload_from_filename('results.json') ``` ### Option 3: API Endpoint ```python import requests requests.post("https://your-api.com/results", json=results) ``` ## Required Configuration for Hub Push ### Job Configuration **Always include HF_TOKEN:** ```python hf_jobs("uv", { "script": "your_script.py", "secrets": {"HF_TOKEN": "$HF_TOKEN"} # ✅ Required for Hub operations }) ``` ### Script Configuration **Verify token exists:** ```python import os assert "HF_TOKEN" in os.environ, "HF_TOKEN required for Hub operations!" ``` **Use token for Hub operations:** ```python from huggingface_hub import HfApi # Auto-detects HF_TOKEN from environment api = HfApi() # Or explicitly pass token api = HfApi(token=os.environ.get("HF_TOKEN")) ``` ## Complete Examples ### Example 1: Push Dataset ```python hf_jobs("uv", { "script": """ # /// script # dependencies = ["datasets", "huggingface-hub"] # /// import os from datasets import Dataset from huggingface_hub import HfApi # Verify token assert "HF_TOKEN" in os.environ, "HF_TOKEN required!" # Process data data = {"text": ["Sample 1", "Sample 2"]} dataset = Dataset.from_dict(data) # Push to Hub dataset.push_to_hub("username/my-dataset") print("✅ Dataset pushed!") """, "flavor": "cpu-basic", "timeout": "30m", "secrets": {"HF_TOKEN": "$HF_TOKEN"} }) ``` ### Example 2: Push Model ```python hf_jobs("uv", { "script": """ # /// script # dependencies = ["transformers"] # /// import os from transformers import AutoModel, AutoTokenizer # Verify token assert "HF_TOKEN" in os.environ, "HF_TOKEN required!" # Load and process model model = AutoModel.from_pretrained("base-model") tokenizer = AutoTokenizer.from_pretrained("base-model") # ... process model ... # Push to Hub model.push_to_hub("username/my-model") tokenizer.push_to_hub("username/my-model") print("✅ Model pushed!") """, "flavor": "a10g-large", "timeout": "2h", "secrets": {"HF_TOKEN": "$HF_TOKEN"} }) ``` ### Example 3: Push Artifacts ```python hf_jobs("uv", { "script": """ # /// script # dependencies = ["huggingface-hub", "pandas"] # /// import os import json import pandas as pd from huggingface_hub import HfApi # Verify token assert "HF_TOKEN" in os.environ, "HF_TOKEN required!" # Generate results results = {"accuracy": 0.95, "loss": 0.05} df = pd.DataFrame([results]) # Save files with open("results.json", "w") as f: json.dump(results, f) df.to_csv("results.csv", index=False) # Push to Hub api = HfApi() api.upload_file("results.json", "results.json", "username/results", repo_type="dataset") api.upload_file("results.csv", "results.csv", "username/results", repo_type="dataset") print("✅ Results pushed!") """, "flavor": "cpu-basic", "timeout": "30m", "secrets": {"HF_TOKEN": "$HF_TOKEN"} }) ``` ## Authentication Methods ### Method 1: Automatic Token (Recommended) ```python "secrets": {"HF_TOKEN": "$HF_TOKEN"} ``` Uses your logged-in Hugging Face token automatically. ### Method 2: Explicit Token ```python "secrets": {"HF_TOKEN": "hf_abc123..."} ``` Provide token explicitly (not recommended for security). ### Method 3: Environment Variable ```python "env": {"HF_TOKEN": "hf_abc123..."} ``` Pass as regular environment variable (less secure than secrets). **Always prefer Method 1** for security and convenience. ## Verification Checklist Before submitting any job that saves to Hub, verify: - [ ] `secrets={"HF_TOKEN": "$HF_TOKEN"}` in job config - [ ] Script checks for token: `assert "HF_TOKEN" in os.environ` - [ ] Hub push code included in script - [ ] Repository name doesn't conflict with existing repos - [ ] You have write access to the target namespace ## Repository Setup ### Automatic Creation If repository doesn't exist, it's created automatically when first pushing (if token has write permissions). ### Manual Creation Create repository before pushing: ```python from huggingface_hub import HfApi api = HfApi() api.create_repo( repo_id="username/repo-name", repo_type="model", # or "dataset" private=False, # or True for private repo ) ``` ### Repository Naming **Valid names:** - `username/my-model` - `username/model-name` - `organization/model-name` **Invalid names:** - `model-name` (missing username) - `username/model name` (spaces not allowed) - `username/MODEL` (uppercase discouraged) ## Troubleshooting ### Error: 401 Unauthorized **Cause:** HF_TOKEN not provided or invalid **Solutions:** 1. Verify `secrets={"HF_TOKEN": "$HF_TOKEN"}` in job config 2. Check you're logged in: `hf_whoami()` 3. Re-login: `hf auth login` ### Error: 403 Forbidden **Cause:** No write access to repository **Solutions:** 1. Check repository namespace matches your username 2. Verify you're a member of organization (if using org namespace) 3. Check token has write permissions ### Error: Repository not found **Cause:** Repository doesn't exist and auto-creation failed **Solutions:** 1. Manually create repository first 2. Check repository name format 3. Verify namespace exists ### Error: Push failed **Cause:** Network issues or Hub unavailable **Solutions:** 1. Check logs for specific error 2. Verify token is valid 3. Retry push operation ## Best Practices 1. **Always verify token exists** before Hub operations 2. **Use descriptive repo names** (e.g., `my-experiment-results` not `results`) 3. **Push incrementally** for large results (use checkpoints) 4. **Verify push success** in logs before job completes 5. **Use appropriate repo types** (model vs dataset) 6. **Add README** with result descriptions 7. **Tag repos** with relevant tags ## Monitoring Push Progress Check logs for push progress: **MCP Tool:** ```python hf_jobs("logs", {"job_id": "your-job-id"}) ``` **CLI:** ```bash hf jobs logs ``` **Python API:** ```python from huggingface_hub import fetch_job_logs for log in fetch_job_logs(job_id="your-job-id"): print(log) ``` **Look for:** ``` Pushing to username/repo-name... Upload file results.json: 100% ✅ Push successful ``` ## Key Takeaway **Without `secrets={"HF_TOKEN": "$HF_TOKEN"}` and persistence code, all results are permanently lost.** Always verify both are configured before submitting any job that produces results. ================================================ FILE: skills/hugging-face-jobs/references/token_usage.md ================================================ # Token Usage Guide for Hugging Face Jobs **⚠️ CRITICAL:** Proper token usage is essential for any job that interacts with the Hugging Face Hub. ## Overview Hugging Face tokens are authentication credentials that allow your jobs to interact with the Hub. They're required for: - Pushing models/datasets to Hub - Accessing private repositories - Creating new repositories - Using Hub APIs programmatically - Any authenticated Hub operations ## Token Types ### Read Token - **Permissions:** Download models/datasets, read private repos - **Use case:** Jobs that only need to download/read content - **Creation:** https://huggingface.co/settings/tokens ### Write Token - **Permissions:** Push models/datasets, create repos, modify content - **Use case:** Jobs that need to upload results (most common) - **Creation:** https://huggingface.co/settings/tokens - **⚠️ Required for:** Pushing models, datasets, or any uploads ### Organization Token - **Permissions:** Act on behalf of an organization - **Use case:** Jobs running under organization namespace - **Creation:** Organization settings → Tokens ## Providing Tokens to Jobs ### Method 1: `hf_jobs` MCP tool with `$HF_TOKEN` (Recommended) ⭐ ```python hf_jobs("uv", { "script": "your_script.py", "secrets": {"HF_TOKEN": "$HF_TOKEN"} # ✅ Automatic replacement }) ``` **How it works:** 1. `$HF_TOKEN` is a placeholder that gets replaced with your actual token 2. Uses the token from your logged-in session (`hf auth login`) 3. Token is encrypted server-side when passed as a secret 4. Most secure and convenient method **Benefits:** - ✅ No token exposure in code - ✅ Uses your current login session - ✅ Automatically updated if you re-login - ✅ Works seamlessly with MCP tools - ✅ Token encrypted server-side **Requirements:** - Must be logged in: `hf auth login` or `hf_whoami()` works - Token must have required permissions **⚠️ CRITICAL:** `$HF_TOKEN` auto-replacement is an `hf_jobs` MCP tool feature ONLY. It does NOT work with `HfApi().run_uv_job()` — see Method 1b below. ### Method 1b: `HfApi().run_uv_job()` with `get_token()` (Required for Python API) ```python from huggingface_hub import HfApi, get_token api = HfApi() api.run_uv_job( script="your_script.py", secrets={"HF_TOKEN": get_token()}, # ✅ Passes actual token value ) ``` **How it works:** 1. `get_token()` retrieves the token from your logged-in session 2. The actual token value is passed to the `secrets` parameter 3. Token is encrypted server-side **Why `"$HF_TOKEN"` fails with `HfApi().run_uv_job()`:** - The Python API passes the literal string `"$HF_TOKEN"` (9 characters) as the token - The Jobs server receives this invalid string instead of a real token - Result: `401 Unauthorized` errors when the script tries to authenticate - You MUST use `get_token()` from `huggingface_hub` to get the real token ### Method 2: Explicit Token (Not Recommended) ```python hf_jobs("uv", { "script": "your_script.py", "secrets": {"HF_TOKEN": "hf_abc123..."} # ⚠️ Hardcoded token }) ``` **When to use:** - Only if automatic token doesn't work - Testing with a specific token - Organization tokens (use with caution) **Security concerns:** - ❌ Token visible in code/logs - ❌ Must manually update if token rotates - ❌ Risk of token exposure - ❌ Not recommended for production ### Method 3: Environment Variable (Less Secure) ```python hf_jobs("uv", { "script": "your_script.py", "env": {"HF_TOKEN": "hf_abc123..."} # ⚠️ Less secure than secrets }) ``` **Difference from secrets:** - `env` variables are visible in job logs - `secrets` are encrypted server-side - Always prefer `secrets` for tokens **When to use:** - Only for non-sensitive configuration - Never use for tokens (use `secrets` instead) ## Using Tokens in Scripts ### Accessing Tokens Tokens passed via `secrets` are available as environment variables in your script: ```python import os # Get token from environment token = os.environ.get("HF_TOKEN") # Verify token exists if not token: raise ValueError("HF_TOKEN not found in environment!") ``` ### Using with Hugging Face Hub **Option 1: Explicit token parameter** ```python from huggingface_hub import HfApi api = HfApi(token=os.environ.get("HF_TOKEN")) api.upload_file(...) ``` **Option 2: Auto-detection (Recommended)** ```python from huggingface_hub import HfApi # Automatically uses HF_TOKEN env var api = HfApi() # ✅ Simpler, uses token from environment api.upload_file(...) ``` **Option 3: With transformers/datasets** ```python from transformers import AutoModel from datasets import load_dataset # Auto-detects HF_TOKEN from environment model = AutoModel.from_pretrained("username/model") dataset = load_dataset("username/dataset") # For push operations, token is auto-detected model.push_to_hub("username/new-model") dataset.push_to_hub("username/new-dataset") ``` ### Complete Example ```python # /// script # dependencies = ["huggingface-hub", "datasets"] # /// import os from huggingface_hub import HfApi from datasets import Dataset # Verify token is available assert "HF_TOKEN" in os.environ, "HF_TOKEN required for Hub operations!" # Use token for Hub operations api = HfApi() # Auto-detects HF_TOKEN # Create and push dataset data = {"text": ["Hello", "World"]} dataset = Dataset.from_dict(data) # Push to Hub (token auto-detected) dataset.push_to_hub("username/my-dataset") print("✅ Dataset pushed successfully!") ``` ## Token Verification ### Check Authentication Locally ```python from huggingface_hub import whoami try: user_info = whoami() print(f"✅ Logged in as: {user_info['name']}") except Exception as e: print(f"❌ Not authenticated: {e}") ``` ### Verify Token in Job ```python import os # Check token exists if "HF_TOKEN" not in os.environ: raise ValueError("HF_TOKEN not found in environment!") token = os.environ["HF_TOKEN"] # Verify token format (should start with "hf_") if not token.startswith("hf_"): raise ValueError(f"Invalid token format: {token[:10]}...") # Test token works from huggingface_hub import whoami try: user_info = whoami(token=token) print(f"✅ Token valid for user: {user_info['name']}") except Exception as e: raise ValueError(f"Token validation failed: {e}") ``` ## Common Token Issues ### Error: 401 Unauthorized **Symptoms:** ``` 401 Client Error: Unauthorized for url: https://huggingface.co/api/... ``` **Causes:** 1. Token missing from job 2. Token invalid or expired 3. Token not passed correctly **Solutions:** 1. Add `secrets={"HF_TOKEN": "$HF_TOKEN"}` to job config 2. Verify `hf_whoami()` works locally 3. Re-login: `hf auth login` 4. Check token hasn't expired **Verification:** ```python # In your script import os assert "HF_TOKEN" in os.environ, "HF_TOKEN missing!" ``` ### Error: 403 Forbidden **Symptoms:** ``` 403 Client Error: Forbidden for url: https://huggingface.co/api/... ``` **Causes:** 1. Token lacks required permissions (read-only token used for write) 2. No access to private repository 3. Organization permissions insufficient **Solutions:** 1. Ensure token has write permissions 2. Check token type at https://huggingface.co/settings/tokens 3. Verify access to target repository 4. Use organization token if needed **Check token permissions:** ```python from huggingface_hub import whoami user_info = whoami() print(f"User: {user_info['name']}") print(f"Type: {user_info.get('type', 'user')}") ``` ### Error: Token not found in environment **Symptoms:** ``` KeyError: 'HF_TOKEN' ValueError: HF_TOKEN not found ``` **Causes:** 1. `secrets` not passed in job config 2. Wrong key name (should be `HF_TOKEN`) 3. Using `env` instead of `secrets` **Solutions:** 1. Use `secrets={"HF_TOKEN": "$HF_TOKEN"}` (not `env`) 2. Verify key name is exactly `HF_TOKEN` 3. Check job config syntax **Correct configuration:** ```python # ✅ Correct hf_jobs("uv", { "script": "...", "secrets": {"HF_TOKEN": "$HF_TOKEN"} }) # ❌ Wrong - using env instead of secrets hf_jobs("uv", { "script": "...", "env": {"HF_TOKEN": "$HF_TOKEN"} # Less secure }) # ❌ Wrong - wrong key name hf_jobs("uv", { "script": "...", "secrets": {"TOKEN": "$HF_TOKEN"} # Wrong key }) ``` ### Error: Repository access denied **Symptoms:** ``` 403 Client Error: Forbidden Repository not found or access denied ``` **Causes:** 1. Token doesn't have access to private repo 2. Repository doesn't exist and can't be created 3. Wrong namespace **Solutions:** 1. Use token from account with access 2. Verify repo visibility (public vs private) 3. Check namespace matches token owner 4. Create repo first if needed **Check repository access:** ```python from huggingface_hub import HfApi api = HfApi() try: repo_info = api.repo_info("username/repo-name") print(f"✅ Access granted: {repo_info.id}") except Exception as e: print(f"❌ Access denied: {e}") ``` ## Token Security Best Practices ### 1. Never Commit Tokens **❌ Bad:** ```python # Never do this! token = "hf_abc123xyz..." api = HfApi(token=token) ``` **✅ Good:** ```python # Use environment variable token = os.environ.get("HF_TOKEN") api = HfApi(token=token) ``` ### 2. Use Secrets, Not Environment Variables **❌ Bad:** ```python hf_jobs("uv", { "script": "...", "env": {"HF_TOKEN": "$HF_TOKEN"} # Visible in logs }) ``` **✅ Good:** ```python hf_jobs("uv", { "script": "...", "secrets": {"HF_TOKEN": "$HF_TOKEN"} # Encrypted server-side }) ``` ### 3. Use Automatic Token Replacement **❌ Bad:** ```python hf_jobs("uv", { "script": "...", "secrets": {"HF_TOKEN": "hf_abc123..."} # Hardcoded }) ``` **✅ Good:** ```python hf_jobs("uv", { "script": "...", "secrets": {"HF_TOKEN": "$HF_TOKEN"} # Automatic }) ``` ### 4. Rotate Tokens Regularly - Generate new tokens periodically - Revoke old tokens - Update job configurations - Monitor token usage ### 5. Use Minimal Permissions - Create tokens with only needed permissions - Use read tokens when write isn't needed - Don't use admin tokens for regular jobs ### 6. Don't Share Tokens - Each user should use their own token - Don't commit tokens to repositories - Don't share tokens in logs or messages ### 7. Monitor Token Usage - Check token activity in Hub settings - Review job logs for token issues - Set up alerts for unauthorized access ## Token Workflow Examples ### Example 1: Push Model to Hub ```python hf_jobs("uv", { "script": """ # /// script # dependencies = ["transformers"] # /// import os from transformers import AutoModel, AutoTokenizer # Verify token assert "HF_TOKEN" in os.environ, "HF_TOKEN required!" # Load and process model model = AutoModel.from_pretrained("base-model") # ... process model ... # Push to Hub (token auto-detected) model.push_to_hub("username/my-model") print("✅ Model pushed!") """, "flavor": "a10g-large", "timeout": "2h", "secrets": {"HF_TOKEN": "$HF_TOKEN"} # ✅ Token provided }) ``` ### Example 2: Access Private Dataset ```python hf_jobs("uv", { "script": """ # /// script # dependencies = ["datasets"] # /// import os from datasets import load_dataset # Verify token assert "HF_TOKEN" in os.environ, "HF_TOKEN required!" # Load private dataset (token auto-detected) dataset = load_dataset("private-org/private-dataset") print(f"✅ Loaded {len(dataset)} examples") """, "flavor": "cpu-basic", "timeout": "30m", "secrets": {"HF_TOKEN": "$HF_TOKEN"} # ✅ Token provided }) ``` ### Example 3: Create and Push Dataset ```python hf_jobs("uv", { "script": """ # /// script # dependencies = ["datasets", "huggingface-hub"] # /// import os from datasets import Dataset from huggingface_hub import HfApi # Verify token assert "HF_TOKEN" in os.environ, "HF_TOKEN required!" # Create dataset data = {"text": ["Sample 1", "Sample 2"]} dataset = Dataset.from_dict(data) # Push to Hub api = HfApi() # Auto-detects HF_TOKEN dataset.push_to_hub("username/my-dataset") print("✅ Dataset pushed!") """, "flavor": "cpu-basic", "timeout": "30m", "secrets": {"HF_TOKEN": "$HF_TOKEN"} # ✅ Token provided }) ``` ## Quick Reference ### Token Checklist Before submitting a job that uses Hub: - [ ] Job includes `secrets={"HF_TOKEN": "$HF_TOKEN"}` - [ ] Script checks for token: `assert "HF_TOKEN" in os.environ` - [ ] Token has required permissions (read/write) - [ ] User is logged in: `hf_whoami()` works - [ ] Token not hardcoded in script - [ ] Using `secrets` not `env` for token ### Common Patterns **Pattern 1: Auto-detect token** ```python from huggingface_hub import HfApi api = HfApi() # Uses HF_TOKEN from environment ``` **Pattern 2: Explicit token** ```python import os from huggingface_hub import HfApi api = HfApi(token=os.environ.get("HF_TOKEN")) ``` **Pattern 3: Verify token** ```python import os assert "HF_TOKEN" in os.environ, "HF_TOKEN required!" ``` ## Key Takeaways 1. **Always use `secrets={"HF_TOKEN": "$HF_TOKEN"}`** for Hub operations 2. **Never hardcode tokens** in scripts or job configs 3. **Verify token exists** in script before Hub operations 4. **Use auto-detection** when possible (`HfApi()` without token parameter) 5. **Check permissions** - ensure token has required access 6. **Monitor token usage** - review activity regularly 7. **Rotate tokens** - generate new tokens periodically ================================================ FILE: skills/hugging-face-jobs/references/troubleshooting.md ================================================ # Troubleshooting Guide Common issues and solutions for Hugging Face Jobs. ## Authentication Issues ### Error: 401 Unauthorized **Symptoms:** ``` 401 Client Error: Unauthorized for url: https://huggingface.co/api/... ``` **Causes:** - Token missing from job - Token invalid or expired - Token not passed correctly **Solutions:** 1. Add token to secrets: `hf_jobs` MCP uses `"$HF_TOKEN"` (auto-replaced); `HfApi().run_uv_job()` MUST use `get_token()` from `huggingface_hub` (the literal string `"$HF_TOKEN"` will NOT work with the Python API) 2. Verify `hf_whoami()` works locally 3. Re-login: `hf auth login` 4. Check token hasn't expired **Verification:** ```python # In your script import os assert "HF_TOKEN" in os.environ, "HF_TOKEN missing!" ``` ### Error: 403 Forbidden **Symptoms:** ``` 403 Client Error: Forbidden for url: https://huggingface.co/api/... ``` **Causes:** - Token lacks required permissions - No access to private repository - Organization permissions insufficient **Solutions:** 1. Ensure token has write permissions 2. Check token type at https://huggingface.co/settings/tokens 3. Verify access to target repository 4. Use organization token if needed ### Error: Token not found in environment **Symptoms:** ``` KeyError: 'HF_TOKEN' ValueError: HF_TOKEN not found ``` **Causes:** - `secrets` not passed in job config - Wrong key name (should be `HF_TOKEN`) - Using `env` instead of `secrets` **Solutions:** 1. Use `secrets` (not `env`) — with `hf_jobs` MCP: `"$HF_TOKEN"`; with `HfApi().run_uv_job()`: `get_token()` 2. Verify key name is exactly `HF_TOKEN` 3. Check job config syntax ## Job Execution Issues ### Error: Job Timeout **Symptoms:** - Job stops unexpectedly - Status shows "TIMEOUT" - Partial results only **Causes:** - Default 30min timeout exceeded - Job takes longer than expected - No timeout specified **Solutions:** 1. Check logs for actual runtime 2. Increase timeout with buffer: `"timeout": "3h"` 3. Optimize code for faster execution 4. Process data in chunks 5. Add 20-30% buffer to estimated time **MCP Tool Example:** ```python hf_jobs("uv", { "script": "...", "timeout": "2h" # Set appropriate timeout }) ``` **Python API Example:** ```python from huggingface_hub import run_uv_job, inspect_job, fetch_job_logs job = run_uv_job("script.py", timeout="4h") # Check if job failed job_info = inspect_job(job_id=job.id) if job_info.status.stage == "ERROR": print(f"Job failed: {job_info.status.message}") # Check logs for details for log in fetch_job_logs(job_id=job.id): print(log) ``` ### Error: Out of Memory (OOM) **Symptoms:** ``` RuntimeError: CUDA out of memory MemoryError: Unable to allocate array ``` **Causes:** - Batch size too large - Model too large for hardware - Insufficient GPU memory **Solutions:** 1. Reduce batch size 2. Process data in smaller chunks 3. Upgrade hardware: cpu → t4 → a10g → a100 4. Use smaller models or quantization 5. Enable gradient checkpointing (for training) **Example:** ```python # Reduce batch size batch_size = 1 # Process in chunks for chunk in chunks: process(chunk) ``` ### Error: Missing Dependencies **Symptoms:** ``` ModuleNotFoundError: No module named 'package_name' ImportError: cannot import name 'X' ``` **Causes:** - Package not in dependencies - Wrong package name - Version mismatch **Solutions:** 1. Add to PEP 723 header: ```python # /// script # dependencies = ["package-name>=1.0.0"] # /// ``` 2. Check package name spelling 3. Specify version if needed 4. Check package availability ### Error: Script Not Found **Symptoms:** ``` FileNotFoundError: script.py not found ``` **Causes:** - Local file path used (not supported) - URL incorrect - Script not accessible **Solutions:** 1. Use inline script (recommended) 2. Use publicly accessible URL 3. Upload script to Hub first 4. Check URL is correct **Correct approaches:** ```python # ✅ Inline code hf_jobs("uv", {"script": "# /// script\n# dependencies = [...]\n# ///\n\n"}) # ✅ From URL hf_jobs("uv", {"script": "https://huggingface.co/user/repo/resolve/main/script.py"}) ``` ## Hub Push Issues ### Error: Push Failed **Symptoms:** ``` Error pushing to Hub Upload failed ``` **Causes:** - Network issues - Token missing or invalid - Repository access denied - File too large **Solutions:** 1. Check token: `assert "HF_TOKEN" in os.environ` 2. Verify repository exists or can be created 3. Check network connectivity in logs 4. Retry push operation 5. Split large files into chunks ### Error: Repository Not Found **Symptoms:** ``` 404 Client Error: Not Found Repository not found ``` **Causes:** - Repository doesn't exist - Wrong repository name - No access to private repo **Solutions:** 1. Create repository first: ```python from huggingface_hub import HfApi api = HfApi() api.create_repo("username/repo-name", repo_type="dataset") ``` 2. Check repository name format 3. Verify namespace exists 4. Check repository visibility ### Error: Results Not Saved **Symptoms:** - Job completes successfully - No results visible on Hub - Files not persisted **Causes:** - No persistence code in script - Push code not executed - Push failed silently **Solutions:** 1. Add persistence code to script 2. Verify push executes successfully 3. Check logs for push errors 4. Add error handling around push **Example:** ```python try: dataset.push_to_hub("username/dataset") print("✅ Push successful") except Exception as e: print(f"❌ Push failed: {e}") raise ``` ## Hardware Issues ### Error: GPU Not Available **Symptoms:** ``` CUDA not available No GPU found ``` **Causes:** - CPU flavor used instead of GPU - GPU not requested - CUDA not installed in image **Solutions:** 1. Use GPU flavor: `"flavor": "a10g-large"` 2. Check image has CUDA support 3. Verify GPU availability in logs ### Error: Slow Performance **Symptoms:** - Job takes longer than expected - Low GPU utilization - CPU bottleneck **Causes:** - Wrong hardware selected - Inefficient code - Data loading bottleneck **Solutions:** 1. Upgrade hardware 2. Optimize code 3. Use batch processing 4. Profile code to find bottlenecks ## General Issues ### Error: Job Status Unknown **Symptoms:** - Can't check job status - Status API returns error **Solutions:** 1. Use job URL: `https://huggingface.co/jobs/username/job-id` 2. Check logs: `hf_jobs("logs", {"job_id": "..."})` 3. Inspect job: `hf_jobs("inspect", {"job_id": "..."})` ### Error: Logs Not Available **Symptoms:** - No logs visible - Logs delayed **Causes:** - Job just started (logs delayed 30-60s) - Job failed before logging - Logs not yet generated **Solutions:** 1. Wait 30-60 seconds after job start 2. Check job status first 3. Use job URL for web interface ### Error: Cost Unexpectedly High **Symptoms:** - Job costs more than expected - Longer runtime than estimated **Causes:** - Job ran longer than timeout - Wrong hardware selected - Inefficient code **Solutions:** 1. Monitor job runtime 2. Set appropriate timeout 3. Optimize code 4. Choose right hardware 5. Check cost estimates before running ## Debugging Tips ### 1. Add Logging ```python import logging logging.basicConfig(level=logging.INFO) logger = logging.getLogger(__name__) logger.info("Starting processing...") logger.info(f"Processed {count} items") ``` ### 2. Verify Environment ```python import os print(f"Python version: {os.sys.version}") print(f"CUDA available: {torch.cuda.is_available()}") print(f"HF_TOKEN present: {'HF_TOKEN' in os.environ}") ``` ### 3. Test Locally First Run script locally before submitting to catch errors early: ```bash python script.py # Or with uv uv run script.py ``` ### 4. Check Job Logs **MCP Tool:** ```python # View logs hf_jobs("logs", {"job_id": "your-job-id"}) ``` **CLI:** ```bash hf jobs logs ``` **Python API:** ```python from huggingface_hub import fetch_job_logs for log in fetch_job_logs(job_id="your-job-id"): print(log) ``` **Or use job URL:** `https://huggingface.co/jobs/username/job-id` ### 5. Add Error Handling ```python try: # Your code process_data() except Exception as e: print(f"Error: {e}") import traceback traceback.print_exc() raise ``` ### 6. Check Job Status Programmatically ```python from huggingface_hub import inspect_job, fetch_job_logs job_info = inspect_job(job_id="your-job-id") print(f"Status: {job_info.status.stage}") print(f"Message: {job_info.status.message}") if job_info.status.stage == "ERROR": print("Job failed! Logs:") for log in fetch_job_logs(job_id="your-job-id"): print(log) ``` ## Quick Reference ### Common Error Codes | Code | Meaning | Solution | |------|---------|----------| | 401 | Unauthorized | Add token to secrets: MCP uses `"$HF_TOKEN"`, Python API uses `get_token()` | | 403 | Forbidden | Check token permissions | | 404 | Not Found | Verify repository exists | | 500 | Server Error | Retry or contact support | ### Checklist Before Submitting - [ ] Token configured: MCP uses `secrets={"HF_TOKEN": "$HF_TOKEN"}`, Python API uses `secrets={"HF_TOKEN": get_token()}` - [ ] Script checks for token: `assert "HF_TOKEN" in os.environ` - [ ] Timeout set appropriately - [ ] Hardware selected correctly - [ ] Dependencies listed in PEP 723 header - [ ] Persistence code included - [ ] Error handling added - [ ] Logging added for debugging ## Getting Help If issues persist: 1. **Check logs** - Most errors include detailed messages 2. **Review documentation** - See main SKILL.md 3. **Check Hub status** - https://status.huggingface.co 4. **Community forums** - https://discuss.huggingface.co 5. **GitHub issues** - For bugs in huggingface_hub ## Key Takeaways 1. **Always include token** - MCP: `secrets={"HF_TOKEN": "$HF_TOKEN"}`, Python API: `secrets={"HF_TOKEN": get_token()}` 2. **Set appropriate timeout** - Default 30min may be insufficient 3. **Verify persistence** - Results won't persist without code 4. **Check logs** - Most issues visible in job logs 5. **Test locally** - Catch errors before submitting 6. **Add error handling** - Better debugging information 7. **Monitor costs** - Set timeouts to avoid unexpected charges ================================================ FILE: skills/hugging-face-jobs/scripts/cot-self-instruct.py ================================================ # /// script # requires-python = ">=3.10" # dependencies = [ # "datasets", # "transformers", # "vllm>=0.6.5", # "huggingface-hub[hf_transfer]", # "torch", # "numpy", # "tqdm", # "scikit-learn", # ] # /// """ Generate high-quality synthetic data using Chain-of-Thought Self-Instruct methodology. This script implements the CoT-Self-Instruct approach from the paper "CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning tasks" (2025). It supports two modes: 1. Reasoning tasks: Generates both questions and answers with Chain-of-Thought 2. Instruction tasks: Generates diverse prompts for general instruction following Example usage: # Reasoning tasks with Answer-Consistency filtering uv run cot-self-instruct.py \\ --seed-dataset davanstrien/s1k-reasoning \\ --output-dataset username/synthetic-math \\ --task-type reasoning \\ --num-samples 5000 \\ --filter-method answer-consistency # Instruction tasks with RIP filtering uv run cot-self-instruct.py \\ --seed-dataset wildchat-filtered \\ --output-dataset username/synthetic-prompts \\ --task-type instruction \\ --filter-method rip \\ --reward-model Nexusflow/Athene-RM-8B # HF Jobs execution hf jobs uv run --flavor l4x4 \\ --image vllm/vllm-openai \\ -e HF_TOKEN=$(python3 -c "from huggingface_hub import get_token; print(get_token())") \\ https://huggingface.co/datasets/uv-scripts/synthetic-data/raw/main/cot-self-instruct.py \\ [args...] """ import argparse import json import logging import os import random import re import sys from collections import Counter from datetime import datetime from typing import Dict, List, Optional, Tuple, Union import numpy as np import torch from datasets import Dataset, load_dataset from huggingface_hub import DatasetCard, login from sklearn.cluster import KMeans from tqdm.auto import tqdm from transformers import AutoTokenizer from vllm import LLM, SamplingParams # Enable HF Transfer for faster downloads os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1" logging.basicConfig( level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s" ) logger = logging.getLogger(__name__) # Prompt templates from the paper REASONING_PROMPT_TEMPLATE = """You are a reasoning question generator assistant. Your goal is to create a novel, and challenging reasoning question. You are provided the following seed questions: Seed Question 1: {seed1} Seed Question 2: {seed2} Your task is to: 1. Write a brand-new, self-contained reasoning question that meets the following requirements: (a) The question draws inspiration from the seed question without copying it verbatim, remaining novel and of comparable difficulty. (b) The question's final answer should be a single, unambiguous scalar value (e.g., an integer, reduced fraction, exact radical), or another answer type that can be verified in one step (e.g., 'yes/no,' a choice from A to D). 2. Then reason step by step, solve the new question and format your output as follows: [New Question Begin]{{your_generated_question}}[New Question End] [Final Answer to New Question Begin]\\boxed{{your_final_answer}}[Final Answer to New Question End]""" INSTRUCTION_PROMPT_TEMPLATE = """You are a prompt generator assistant. Your goal is to create diverse and creative synthetic prompts. Please follow the steps below to create synthetic prompts. Step 1: Carefully read #Prompt 1# and #Prompt 2#. Identify and list all the common elements between these two prompts. If no common elements are found, list the main elements from each prompt. Step 2: Develop a comprehensive plan based on the #Common Elements List# or #Main Elements List# from Step 1. This plan will guide the generation of new synthetic prompts that are similar to the original prompts. Step 3: Execute the plan step by step and provide one #Synthetic Prompt#. Please reply strictly in the following format: - Step 1 #Common Elements List# or #Main Elements List#: - Step 2 #Plan#: - Step 3 #Synthetic Prompt#: #Prompt 1#: {prompt1} #Prompt 2#: {prompt2}""" def check_gpu_availability() -> int: """Check if CUDA is available and return the number of GPUs.""" if not torch.cuda.is_available(): logger.error("CUDA is not available. This script requires a GPU.") logger.error( "Please run on a machine with NVIDIA GPU or use HF Jobs with GPU flavor." ) sys.exit(1) num_gpus = torch.cuda.device_count() for i in range(num_gpus): gpu_name = torch.cuda.get_device_name(i) gpu_memory = torch.cuda.get_device_properties(i).total_memory / 1024**3 logger.info(f"GPU {i}: {gpu_name} with {gpu_memory:.1f} GB memory") return num_gpus def parse_thinking_output(text: str) -> str: """Remove thinking tokens from model output.""" # Remove ... blocks text = re.sub(r'.*?', '', text, flags=re.DOTALL) return text.strip() def extract_reasoning_output(text: str) -> Tuple[Optional[str], Optional[str]]: """Extract question and answer from reasoning task output.""" text = parse_thinking_output(text) # Extract question question_match = re.search(r'\[New Question Begin\](.*?)\[New Question End\]', text, re.DOTALL) if not question_match: return None, None question = question_match.group(1).strip() # Extract answer answer_match = re.search(r'\[Final Answer to New Question Begin\]\\?boxed\{(.*?)\}\[Final Answer to New Question End\]', text, re.DOTALL) if not answer_match: # Try without \boxed answer_match = re.search(r'\[Final Answer to New Question Begin\](.*?)\[Final Answer to New Question End\]', text, re.DOTALL) if not answer_match: return question, None answer = answer_match.group(1).strip() return question, answer def extract_instruction_output(text: str) -> Optional[str]: """Extract synthetic prompt from instruction task output.""" text = parse_thinking_output(text) # Look for the synthetic prompt after "Step 3 #Synthetic Prompt#:" match = re.search(r'Step 3 #Synthetic Prompt#:\s*(.+)', text, re.DOTALL) if match: return match.group(1).strip() return None def categorize_prompts(prompts: List[str], num_categories: int = 8) -> Dict[int, List[int]]: """Categorize prompts using clustering for instruction tasks.""" from transformers import AutoModel logger.info(f"Categorizing {len(prompts)} prompts into {num_categories} categories...") # Use a small model for embeddings tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2") model = AutoModel.from_pretrained("sentence-transformers/all-MiniLM-L6-v2") # Get embeddings embeddings = [] for prompt in tqdm(prompts, desc="Computing embeddings"): inputs = tokenizer(prompt, return_tensors="pt", truncation=True, max_length=512) with torch.no_grad(): outputs = model(**inputs) embedding = outputs.last_hidden_state.mean(dim=1).numpy() embeddings.append(embedding[0]) # Cluster kmeans = KMeans(n_clusters=num_categories, random_state=42) labels = kmeans.fit_predict(embeddings) # Group by category categories = {} for idx, label in enumerate(labels): if label not in categories: categories[label] = [] categories[label].append(idx) return categories def generate_synthetic_data( llm: LLM, seed_data: List[Dict], task_type: str, num_samples: int, categories: Optional[Dict[int, List[int]]] = None, ) -> List[Dict]: """Generate synthetic data using CoT-Self-Instruct.""" synthetic_data = [] # Set up progress bar pbar = tqdm(total=num_samples, desc="Generating synthetic data") while len(synthetic_data) < num_samples: # Sample seed data if task_type == "reasoning": # Random sampling for reasoning tasks seeds = random.sample(seed_data, min(2, len(seed_data))) prompt = REASONING_PROMPT_TEMPLATE.format( seed1=seeds[0].get("question", seeds[0].get("prompt", "")), seed2=seeds[1].get("question", seeds[1].get("prompt", "")) if len(seeds) > 1 else seeds[0].get("question", seeds[0].get("prompt", "")) ) else: # Category-aware sampling for instruction tasks if categories: # Pick a random category category = random.choice(list(categories.keys())) category_indices = categories[category] indices = random.sample(category_indices, min(2, len(category_indices))) seeds = [seed_data[i] for i in indices] else: seeds = random.sample(seed_data, min(2, len(seed_data))) prompt = INSTRUCTION_PROMPT_TEMPLATE.format( prompt1=seeds[0].get("prompt", seeds[0].get("question", "")), prompt2=seeds[1].get("prompt", seeds[1].get("question", "")) if len(seeds) > 1 else seeds[0].get("prompt", seeds[0].get("question", "")) ) # Generate sampling_params = SamplingParams( temperature=0.7 if task_type == "reasoning" else 0.8, top_p=0.95 if task_type == "reasoning" else 0.9, max_tokens=2048, ) outputs = llm.generate([prompt], sampling_params) output_text = outputs[0].outputs[0].text # Parse output if task_type == "reasoning": question, answer = extract_reasoning_output(output_text) if question and answer: synthetic_data.append({ "question": question, "answer": answer, "seed_indices": [seed_data.index(s) for s in seeds], }) pbar.update(1) else: synthetic_prompt = extract_instruction_output(output_text) if synthetic_prompt: synthetic_data.append({ "prompt": synthetic_prompt, "seed_indices": [seed_data.index(s) for s in seeds], }) pbar.update(1) pbar.close() return synthetic_data def answer_consistency_filter( llm: LLM, synthetic_data: List[Dict], k_responses: int = 16, threshold: float = 0.5, ) -> List[Dict]: """Filter reasoning tasks using Answer-Consistency.""" logger.info(f"Applying Answer-Consistency filter with K={k_responses}") filtered_data = [] for item in tqdm(synthetic_data, desc="Answer-Consistency filtering"): question = item["question"] original_answer = item["answer"] # Generate K responses prompts = [question] * k_responses sampling_params = SamplingParams( temperature=0.6, top_p=0.95, max_tokens=1024, ) outputs = llm.generate(prompts, sampling_params) # Extract answers answers = [] for output in outputs: text = output.outputs[0].text # Try to extract boxed answer match = re.search(r'\\boxed\{(.*?)\}', text) if match: answers.append(match.group(1).strip()) if not answers: continue # Get majority answer answer_counts = Counter(answers) if answer_counts: majority_answer, count = answer_counts.most_common(1)[0] # Check if majority answer matches original and meets threshold if (majority_answer == original_answer and count / len(answers) >= threshold): item["consistency_score"] = count / len(answers) filtered_data.append(item) logger.info(f"Answer-Consistency: kept {len(filtered_data)}/{len(synthetic_data)} examples") return filtered_data def rip_filter( llm: LLM, synthetic_data: List[Dict], reward_model_id: str, k_responses: int = 32, threshold: float = 0.5, ) -> List[Dict]: """Filter using Rejecting Instruction Preferences (RIP).""" logger.info(f"Applying RIP filter with K={k_responses} and reward model {reward_model_id}") # Note: In a full implementation, you would load and use the actual reward model # For this example, we'll use a placeholder scoring mechanism logger.warning("RIP filtering requires a reward model implementation - using placeholder") filtered_data = [] for item in tqdm(synthetic_data, desc="RIP filtering"): prompt = item.get("prompt", item.get("question", "")) # Generate K responses prompts = [prompt] * k_responses sampling_params = SamplingParams( temperature=1.0, top_p=1.0, max_tokens=1024, ) outputs = llm.generate(prompts, sampling_params) # In real implementation: score each response with reward model # For now, use length as a proxy (longer responses often score higher) scores = [len(output.outputs[0].text) for output in outputs] # Use minimum score as quality indicator min_score = min(scores) if scores else 0 normalized_score = min_score / 1000 # Normalize to 0-1 range if normalized_score >= threshold: item["rip_score"] = normalized_score filtered_data.append(item) logger.info(f"RIP filter: kept {len(filtered_data)}/{len(synthetic_data)} examples") return filtered_data def create_dataset_card( task_type: str, source_dataset: str, generation_model: str, filter_method: str, num_generated: int, num_filtered: int, generation_time: str, additional_info: Dict = None, ) -> str: """Create a comprehensive dataset card.""" filter_info = "" if filter_method == "answer-consistency": filter_info = """ ### Answer-Consistency Filtering This dataset was filtered using Answer-Consistency: - Generated K responses for each synthetic question - Kept only examples where majority answer matched the generated answer - Ensures high-quality, correctly solved problems""" elif filter_method == "rip": filter_info = """ ### RIP (Rejecting Instruction Preferences) Filtering This dataset was filtered using RIP: - Generated K responses for each synthetic prompt - Scored responses using a reward model - Kept only prompts with high minimum scores""" return f"""--- tags: - synthetic-data - cot-self-instruct - {task_type} - uv-script --- # CoT-Self-Instruct Synthetic Data This dataset contains synthetic {task_type} data generated using the Chain-of-Thought Self-Instruct methodology. ## Generation Details - **Source Dataset**: [{source_dataset}](https://huggingface.co/datasets/{source_dataset}) - **Generation Model**: [{generation_model}](https://huggingface.co/{generation_model}) - **Task Type**: {task_type} - **Filter Method**: {filter_method} - **Generated Examples**: {num_generated:,} - **After Filtering**: {num_filtered:,} ({(num_filtered/num_generated)*100:.1f}% acceptance rate) - **Generation Date**: {generation_time} {filter_info} ## Methodology Generated using CoT-Self-Instruct, which: 1. Uses Chain-of-Thought reasoning to analyze seed examples 2. Generates new synthetic examples of similar quality and complexity 3. Applies quality filtering to ensure high-quality outputs Based on the paper: "CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning tasks" (2025) ## Generation Script Generated using the CoT-Self-Instruct script from [uv-scripts/synthetic-data](https://huggingface.co/datasets/uv-scripts/synthetic-data). To reproduce: ```bash uv run https://huggingface.co/datasets/uv-scripts/synthetic-data/raw/main/cot-self-instruct.py \\ --seed-dataset {source_dataset} \\ --output-dataset \\ --task-type {task_type} \\ --generation-model {generation_model} \\ --filter-method {filter_method} ``` """ def main(): parser = argparse.ArgumentParser( description="Generate synthetic data using CoT-Self-Instruct", formatter_class=argparse.RawDescriptionHelpFormatter, epilog=__doc__, ) # Dataset arguments parser.add_argument( "--seed-dataset", type=str, required=True, help="HuggingFace dataset ID containing seed examples", ) parser.add_argument( "--output-dataset", type=str, required=True, help="HuggingFace dataset ID for output", ) # Task configuration parser.add_argument( "--task-type", type=str, choices=["reasoning", "instruction", "auto"], default="auto", help="Type of task (reasoning generates Q&A, instruction generates prompts)", ) parser.add_argument( "--task-column", type=str, default=None, help="Column name containing tasks (auto-detected if not specified)", ) # Model configuration parser.add_argument( "--generation-model", type=str, default="Qwen/Qwen3-30B-A3B-Thinking-2507", help="Model for synthetic data generation", ) parser.add_argument( "--filter-model", type=str, default=None, help="Model for filtering (defaults to generation model)", ) parser.add_argument( "--reward-model", type=str, default="Nexusflow/Athene-RM-8B", help="Reward model for RIP filtering", ) # Generation parameters parser.add_argument( "--num-samples", type=int, default=5000, help="Number of synthetic examples to generate", ) parser.add_argument( "--batch-size", type=int, default=1, help="Batch size for generation", ) # Filtering parameters parser.add_argument( "--filter-method", type=str, choices=["answer-consistency", "rip", "both", "none"], default="answer-consistency", help="Quality filtering method", ) parser.add_argument( "--k-responses", type=int, default=16, help="Number of responses for filtering", ) parser.add_argument( "--quality-threshold", type=float, default=0.5, help="Minimum quality threshold for filtering", ) # GPU configuration parser.add_argument( "--tensor-parallel-size", type=int, default=None, help="Number of GPUs for tensor parallelism (auto-detected if not set)", ) parser.add_argument( "--gpu-memory-utilization", type=float, default=0.9, help="GPU memory utilization", ) # Other arguments parser.add_argument( "--hf-token", type=str, default=None, help="HuggingFace API token", ) parser.add_argument( "--seed", type=int, default=42, help="Random seed", ) args = parser.parse_args() # Set random seeds random.seed(args.seed) np.random.seed(args.seed) torch.manual_seed(args.seed) # Check GPU num_gpus = check_gpu_availability() tensor_parallel_size = args.tensor_parallel_size or num_gpus # Authentication hf_token = args.hf_token or os.environ.get("HF_TOKEN") if hf_token: login(token=hf_token) # Load seed dataset logger.info(f"Loading seed dataset: {args.seed_dataset}") seed_dataset = load_dataset(args.seed_dataset, split="train") # Auto-detect task type and column if needed if args.task_type == "auto": columns = seed_dataset.column_names if "question" in columns and "answer" in columns: args.task_type = "reasoning" logger.info("Auto-detected task type: reasoning") else: args.task_type = "instruction" logger.info("Auto-detected task type: instruction") if not args.task_column: if args.task_type == "reasoning": args.task_column = "question" else: # Try to find prompt column for col in ["prompt", "instruction", "text", "input"]: if col in seed_dataset.column_names: args.task_column = col break logger.info(f"Using task column: {args.task_column}") # Convert to list of dicts seed_data = seed_dataset.to_list() # Categorize prompts for instruction tasks categories = None if args.task_type == "instruction" and len(seed_data) > 100: prompts = [item.get(args.task_column, "") for item in seed_data] categories = categorize_prompts(prompts) # Initialize generation model logger.info(f"Loading generation model: {args.generation_model}") generation_llm = LLM( model=args.generation_model, tensor_parallel_size=tensor_parallel_size, gpu_memory_utilization=args.gpu_memory_utilization, ) # Generate synthetic data start_time = datetime.now() synthetic_data = generate_synthetic_data( generation_llm, seed_data, args.task_type, args.num_samples, categories, ) # Apply filtering filter_llm = generation_llm if args.filter_model and args.filter_model != args.generation_model: logger.info(f"Loading filter model: {args.filter_model}") # Clean up generation model del generation_llm torch.cuda.empty_cache() filter_llm = LLM( model=args.filter_model, tensor_parallel_size=tensor_parallel_size, gpu_memory_utilization=args.gpu_memory_utilization, ) filtered_data = synthetic_data if args.filter_method != "none": if args.filter_method == "answer-consistency" and args.task_type == "reasoning": filtered_data = answer_consistency_filter( filter_llm, synthetic_data, args.k_responses, args.quality_threshold, ) elif args.filter_method == "rip": filtered_data = rip_filter( filter_llm, synthetic_data, args.reward_model, args.k_responses, args.quality_threshold, ) elif args.filter_method == "both": if args.task_type == "reasoning": filtered_data = answer_consistency_filter( filter_llm, synthetic_data, args.k_responses, args.quality_threshold, ) filtered_data = rip_filter( filter_llm, filtered_data, args.reward_model, args.k_responses, args.quality_threshold, ) # Create HuggingFace dataset logger.info(f"Creating dataset with {len(filtered_data)} examples") dataset = Dataset.from_list(filtered_data) # Create dataset card generation_time = start_time.strftime("%Y-%m-%d %H:%M:%S UTC") dataset_card = create_dataset_card( args.task_type, args.seed_dataset, args.generation_model, args.filter_method, len(synthetic_data), len(filtered_data), generation_time, ) # Push to hub logger.info(f"Pushing dataset to: {args.output_dataset}") # Create dataset card card = DatasetCard(dataset_card) dataset.push_to_hub(args.output_dataset) # Push card separately card.push_to_hub(args.output_dataset) logger.info("Done! Dataset available at: https://huggingface.co/datasets/" + args.output_dataset) # Print example HF Jobs command if running locally if len(sys.argv) > 1: print("\nTo run on HF Jobs:") print(f"""hf jobs uv run --flavor l4x4 \\ --image vllm/vllm-openai \\ -e HF_TOKEN=$(python3 -c "from huggingface_hub import get_token; print(get_token())") \\ https://huggingface.co/datasets/uv-scripts/synthetic-data/raw/main/cot-self-instruct.py \\ --seed-dataset {args.seed_dataset} \\ --output-dataset {args.output_dataset} \\ --task-type {args.task_type} \\ --generation-model {args.generation_model} \\ --filter-method {args.filter_method} \\ --num-samples {args.num_samples}""") if __name__ == "__main__": main() ================================================ FILE: skills/hugging-face-jobs/scripts/finepdfs-stats.py ================================================ # /// script # requires-python = ">=3.12" # dependencies = [ # "polars>=1.31.0", # "huggingface-hub", # "datasets", # "ascii-graph", # ] # /// """ Analyze educational quality trends across CommonCrawl dumps using Polars streaming. Answers: "Is the web getting more educational over time?" Demonstrates Polars HF Hub integration - process 50M+ docs without downloading 300GB+. Example usage: # Analyze English PDFs (default) uv run finepdfs-stats.py # Analyze all 70+ languages uv run finepdfs-stats.py --all-languages # Quick test uv run finepdfs-stats.py --limit 10000 --show-plan # Save results to HF Hub uv run finepdfs-stats.py --output-repo username/finepdfs-temporal-stats # Run on HF Jobs hf jobs uv run \\ -s HF_TOKEN \\ -e HF_XET_HIGH_PERFORMANCE=1 \\ https://huggingface.co/datasets/uv-scripts/dataset-stats/raw/main/finepdfs-stats.py \\ -- --output-repo username/stats """ import argparse import logging import os import sys import time from pathlib import Path import polars as pl from ascii_graph import Pyasciigraph from datasets import Dataset from huggingface_hub import HfApi, create_repo, list_repo_tree, login logging.basicConfig( level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s" ) logger = logging.getLogger(__name__) # Common language+script codes for finepdfs-edu COMMON_LANGUAGES = { "eng_Latn": "English (Latin script)", "fra_Latn": "French (Latin script)", "deu_Latn": "German (Latin script)", "spa_Latn": "Spanish (Latin script)", "por_Latn": "Portuguese (Latin script)", "ita_Latn": "Italian (Latin script)", "nld_Latn": "Dutch (Latin script)", "pol_Latn": "Polish (Latin script)", "rus_Cyrl": "Russian (Cyrillic script)", "zho_Hans": "Chinese (Simplified)", "zho_Hant": "Chinese (Traditional)", "jpn_Jpan": "Japanese", "kor_Hang": "Korean", "ara_Arab": "Arabic", "hin_Deva": "Hindi (Devanagari)", } def list_available_languages(dataset_id: str) -> list[str]: """List available language subsets in the dataset.""" try: tree = list_repo_tree(dataset_id, path_in_repo="data", repo_type="dataset") languages = [ item.path.replace("data/", "") for item in tree if item.path.startswith("data/") and "/" not in item.path.replace("data/", "") ] return sorted(languages) except Exception as e: logger.warning(f"Could not list languages: {e}") return list(COMMON_LANGUAGES.keys()) def compute_temporal_stats(df: pl.LazyFrame, output_path: Path) -> pl.DataFrame: """Single scan: compute stats grouped by dump for temporal analysis.""" query = df.group_by("dump").agg( pl.len().alias("doc_count"), pl.col("token_count").sum().alias("total_tokens"), pl.col("fw_edu_scores").list.mean().mean().alias("avg_edu_score"), (pl.col("fw_edu_scores").list.mean() >= 3).sum().alias("high_edu_count"), ) query.sink_parquet(output_path, engine="streaming") return pl.read_parquet(output_path) def compute_global_stats(temporal: pl.DataFrame) -> pl.DataFrame: """Compute global stats from temporal breakdown.""" total = temporal["doc_count"].sum() return pl.DataFrame( { "total_docs": [total], "total_tokens": [temporal["total_tokens"].sum()], "avg_edu_score": [ (temporal["avg_edu_score"] * temporal["doc_count"]).sum() / total ], "high_edu_rate": [temporal["high_edu_count"].sum() / total], "num_dumps": [len(temporal)], } ) def format_temporal_stats(temporal: pl.DataFrame) -> pl.DataFrame: """Format temporal stats with high_edu_rate, sorted chronologically.""" return ( temporal.with_columns( (pl.col("high_edu_count") / pl.col("doc_count")).alias("high_edu_rate") ) .select(["dump", "doc_count", "avg_edu_score", "high_edu_rate"]) .sort( "dump" ) # Chronological order (CC-MAIN-2017-xx comes before CC-MAIN-2024-xx) ) def create_ascii_charts(temporal_stats: pl.DataFrame) -> str: """Create ASCII bar charts showing temporal trends.""" # Extract year from dump name (CC-MAIN-2024-42 -> 2024) # Group by year and average the values for cleaner display yearly = ( temporal_stats.with_columns( pl.col("dump").str.extract(r"CC-MAIN-(\d{4})", 1).alias("year") ) .group_by("year") .agg( pl.col("doc_count").sum(), pl.col("avg_edu_score").mean(), pl.col("high_edu_rate").mean(), ) .sort("year") ) lines = [] # High edu rate chart (more dramatic differences) data_rate = [ (row["year"], row["high_edu_rate"] * 100) for row in yearly.iter_rows(named=True) ] graph = Pyasciigraph(line_length=60, float_format="{0:.1f}%") lines.extend(graph.graph("High Educational Content (edu >= 3)", data_rate)) lines.append("") # Avg edu score chart data_score = [ (row["year"], row["avg_edu_score"]) for row in yearly.iter_rows(named=True) ] graph2 = Pyasciigraph(line_length=60, float_format="{0:.2f}") lines.extend(graph2.graph("Average Educational Score", data_score)) return "\n".join(lines) def create_readme( args, global_stats: pl.DataFrame, temporal_stats: pl.DataFrame, scan_time: float, ascii_charts: str, ) -> str: """Create README content for the stats dataset.""" stats = global_stats.to_dicts()[0] total_docs = stats.get("total_docs", 0) docs_per_sec = total_docs / scan_time if scan_time > 0 else 0 # Get first and last year averages for trend (more representative than single dumps) yearly = ( temporal_stats.with_columns( pl.col("dump").str.extract(r"CC-MAIN-(\d{4})", 1).alias("year") ) .group_by("year") .agg( pl.col("doc_count").sum(), pl.col("avg_edu_score").mean(), pl.col("high_edu_rate").mean(), ) .sort("year") ) first_year = yearly.head(1).to_dicts()[0] last_year = yearly.tail(1).to_dicts()[0] scope = ( "all languages" if args.all_languages else COMMON_LANGUAGES.get(args.lang, args.lang) ) return f"""--- tags: - uv-script - statistics - polars - finepdfs-edu - temporal-analysis license: odc-by configs: - config_name: global_stats data_files: global_stats/train-*.parquet - config_name: temporal_stats data_files: temporal_stats/train-*.parquet default_viewer_config: temporal_stats --- # Is the Web Getting More Educational? Temporal analysis of educational quality in **{scope}** across {stats.get("num_dumps", 0)} CommonCrawl dumps. ## Trend ``` {ascii_charts} ``` ## Key Finding | Year | Avg Edu Score | High Edu Rate | |------|---------------|---------------| | {first_year["year"]} | {first_year["avg_edu_score"]:.2f} | {first_year["high_edu_rate"] * 100:.1f}% | | {last_year["year"]} | {last_year["avg_edu_score"]:.2f} | {last_year["high_edu_rate"] * 100:.1f}% | ## Performance - **{total_docs:,} documents** processed in **{scan_time:.0f} seconds** - **{docs_per_sec:,.0f} docs/sec** using Polars streaming - Single scan, no full dataset download required ## Summary | Metric | Value | |--------|-------| | Scope | {scope} | | Total Documents | {total_docs:,} | | Total Tokens | {stats.get("total_tokens", 0):,} | | Avg Edu Score | {stats.get("avg_edu_score", 0):.3f} | | High Edu Rate | {stats.get("high_edu_rate", 0) * 100:.1f}% | | CommonCrawl Dumps | {stats.get("num_dumps", 0)} | ## Files - `global_stats` - Overall summary - `temporal_stats` - Per-dump breakdown (sorted chronologically) ## Reproduce ```bash uv run https://huggingface.co/datasets/uv-scripts/dataset-stats/raw/main/finepdfs-stats.py \\ {"--all-languages" if args.all_languages else f"--lang {args.lang}"} --output-repo your-username/stats ``` ## Source - **Dataset**: [{args.source_dataset}](https://huggingface.co/datasets/{args.source_dataset}) - **Script**: [uv-scripts/dataset-stats](https://huggingface.co/datasets/uv-scripts/dataset-stats) """ def main(): parser = argparse.ArgumentParser( description="Analyze educational quality trends across CommonCrawl dumps", formatter_class=argparse.RawDescriptionHelpFormatter, epilog=__doc__, ) parser.add_argument( "--source-dataset", type=str, default="HuggingFaceFW/finepdfs-edu", help="Source dataset (default: HuggingFaceFW/finepdfs-edu)", ) parser.add_argument( "--lang", type=str, default="eng_Latn", help="Language+script code (default: eng_Latn)", ) parser.add_argument( "--all-languages", action="store_true", help="Analyze all languages (70+) instead of single language", ) parser.add_argument( "--show-plan", action="store_true", help="Show Polars query plan (demonstrates optimization)", ) parser.add_argument( "--list-languages", action="store_true", help="List available languages and exit", ) parser.add_argument( "--limit", type=int, help="Limit to first N rows (for testing)", ) parser.add_argument( "--output-repo", type=str, help="HuggingFace dataset repository to upload results", ) parser.add_argument( "--output-dir", type=str, default="./stats_output", help="Local directory for output files", ) parser.add_argument( "--hf-token", type=str, help="HuggingFace API token (or set HF_TOKEN env var)", ) parser.add_argument( "--private", action="store_true", help="Make the output dataset private", ) args = parser.parse_args() # Check for high-performance mode if os.environ.get("HF_XET_HIGH_PERFORMANCE"): logger.info("High-performance mode enabled (HF_XET_HIGH_PERFORMANCE=1)") # List languages mode if args.list_languages: print(f"Available language+script codes for {args.source_dataset}:\n") print("Common languages:") for code, name in COMMON_LANGUAGES.items(): print(f" {code:12} - {name}") print("\nFetching full list from HF Hub...") all_langs = list_available_languages(args.source_dataset) print(f"\nAll available ({len(all_langs)} total):") for lang in all_langs[:30]: # Show first 30 name = COMMON_LANGUAGES.get(lang, "") print(f" {lang:12} {name}") if len(all_langs) > 30: print(f" ... and {len(all_langs) - 30} more") sys.exit(0) # Build the parquet path if args.all_languages: source_path = f"hf://datasets/{args.source_dataset}/data/*/train/*.parquet" scope_desc = "all languages" else: source_path = ( f"hf://datasets/{args.source_dataset}/data/{args.lang}/train/*.parquet" ) scope_desc = f"{args.lang} ({COMMON_LANGUAGES.get(args.lang, 'unknown')})" logger.info(f"Scanning: {source_path}") logger.info(f"Scope: {scope_desc}") # Create lazy frame - this doesn't load any data yet! logger.info("Creating lazy query plan...") df = pl.scan_parquet(source_path) # Apply limit if specified if args.limit: logger.info(f"Limiting to first {args.limit:,} rows") df = df.head(args.limit) # Show query plan if requested if args.show_plan: # Build a sample query to show the plan sample_query = df.select( pl.len(), pl.col("token_count").sum(), pl.col("language").n_unique(), ) print("\nQuery Plan (showing Polars optimization):") print("=" * 60) print(sample_query.explain()) print("=" * 60) print("\nNote: Polars uses projection pushdown - only reads columns needed!") print("The 'text' column is never loaded, making this very fast.\n") # Create output directory output_dir = Path(args.output_dir) output_dir.mkdir(parents=True, exist_ok=True) # Single scan: compute temporal stats logger.info("Computing temporal stats (single scan)...") start = time.perf_counter() temporal_path = output_dir / "temporal_stats.parquet" temporal_raw = compute_temporal_stats(df, temporal_path) scan_time = time.perf_counter() - start logger.info(f"Scan complete in {scan_time:.2f}s - {len(temporal_raw)} dumps") # Compute stats global_stats = compute_global_stats(temporal_raw) temporal_stats = format_temporal_stats(temporal_raw) # Save global_stats.write_parquet(output_dir / "global_stats.parquet") temporal_stats.write_parquet(output_dir / "temporal_stats.parquet") # Print results total_docs = global_stats["total_docs"][0] docs_per_sec = total_docs / scan_time if scan_time > 0 else 0 print("\n" + "=" * 70) print("IS THE WEB GETTING MORE EDUCATIONAL?") print("=" * 70) print(f"\nScope: {scope_desc}") print(f"Dataset: {args.source_dataset}") print("\n" + "-" * 70) print("GLOBAL STATS") print("-" * 70) print(global_stats) print("\n" + "-" * 70) print(f"TEMPORAL TREND ({len(temporal_stats)} CommonCrawl dumps)") print("-" * 70) # Show first 5 and last 5 if len(temporal_stats) > 10: print("Earliest dumps:") print(temporal_stats.head(5)) print("\n...") print("\nLatest dumps:") print(temporal_stats.tail(5)) else: print(temporal_stats) # Create ASCII charts ascii_charts = create_ascii_charts(temporal_stats) print("\n" + "-" * 70) print("TREND VISUALIZATION") print("-" * 70) print(ascii_charts) print("\n" + "-" * 70) print("PERFORMANCE") print("-" * 70) print(f"Scan time: {scan_time:.2f}s") print(f"Documents: {total_docs:,}") print(f"Throughput: {docs_per_sec:,.0f} docs/sec") logger.info(f"Results saved to: {output_dir}") # Upload to HF Hub if requested if args.output_repo: hf_token = args.hf_token or os.environ.get("HF_TOKEN") if hf_token: login(token=hf_token) api = HfApi(token=hf_token) logger.info(f"Creating/updating dataset repository: {args.output_repo}") create_repo( args.output_repo, repo_type="dataset", private=args.private, token=hf_token, exist_ok=True, ) # Upload each as a dataset config configs = [ ("global_stats", global_stats), ("temporal_stats", temporal_stats), ] for config_name, stats_df in configs: logger.info(f"Uploading {config_name}...") ds = Dataset.from_polars(stats_df) ds.push_to_hub( args.output_repo, config_name=config_name, token=hf_token, private=args.private, ) time.sleep(1) # Avoid 409 conflicts # Upload README readme_content = create_readme( args, global_stats, temporal_stats, scan_time, ascii_charts ) api.upload_file( path_or_fileobj=readme_content.encode(), path_in_repo="README.md", repo_id=args.output_repo, repo_type="dataset", token=hf_token, ) dataset_url = f"https://huggingface.co/datasets/{args.output_repo}" logger.info(f"Dataset uploaded: {dataset_url}") print(f"\nResults uploaded to: {dataset_url}") if __name__ == "__main__": if len(sys.argv) == 1: print("Is the Web Getting More Educational?") print("=" * 40) print("\nAnalyze educational quality trends across CommonCrawl dumps") print("using Polars streaming - no download needed!\n") print("Example commands:\n") print("# Quick test:") print("uv run finepdfs-stats.py --limit 10000\n") print("# Analyze English PDFs:") print("uv run finepdfs-stats.py\n") print("# Analyze ALL 70+ languages:") print("uv run finepdfs-stats.py --all-languages\n") print("# Show query plan (see Polars optimization):") print("uv run finepdfs-stats.py --show-plan --limit 1000\n") print("# Save results to HF Hub:") print("uv run finepdfs-stats.py --output-repo username/temporal-stats\n") print("# Run on HF Jobs:") print("hf jobs uv run \\") print(" -s HF_TOKEN \\") print(" -e HF_XET_HIGH_PERFORMANCE=1 \\") print( " https://huggingface.co/datasets/uv-scripts/dataset-stats/raw/main/finepdfs-stats.py \\" ) print(" -- --output-repo username/stats") sys.exit(0) main() ================================================ FILE: skills/hugging-face-jobs/scripts/generate-responses.py ================================================ # /// script # requires-python = ">=3.10" # dependencies = [ # "datasets", # "flashinfer-python", # "huggingface-hub[hf_transfer]", # "hf-xet>= 1.1.7", # "torch", # "transformers", # "vllm>=0.8.5", # ] # # /// """ Generate responses for prompts in a dataset using vLLM for efficient GPU inference. This script loads a dataset from Hugging Face Hub containing chat-formatted messages, applies the model's chat template, generates responses using vLLM, and saves the results back to the Hub with a comprehensive dataset card. Example usage: # Local execution with auto GPU detection uv run generate-responses.py \\ username/input-dataset \\ username/output-dataset \\ --messages-column messages # With custom model and sampling parameters uv run generate-responses.py \\ username/input-dataset \\ username/output-dataset \\ --model-id meta-llama/Llama-3.1-8B-Instruct \\ --temperature 0.9 \\ --top-p 0.95 \\ --max-tokens 2048 # HF Jobs execution (see script output for full command) hf jobs uv run --flavor a100x4 ... """ import argparse import logging import os import sys from datetime import datetime from typing import Optional from datasets import load_dataset from huggingface_hub import DatasetCard, get_token, login from torch import cuda from tqdm.auto import tqdm from transformers import AutoTokenizer from vllm import LLM, SamplingParams # Enable HF Transfer for faster downloads os.environ["HF_HUB_ENABLE_HF_TRANSFER"] = "1" logging.basicConfig( level=logging.INFO, format="%(asctime)s - %(levelname)s - %(message)s" ) logger = logging.getLogger(__name__) def check_gpu_availability() -> int: """Check if CUDA is available and return the number of GPUs.""" if not cuda.is_available(): logger.error("CUDA is not available. This script requires a GPU.") logger.error( "Please run on a machine with NVIDIA GPU or use HF Jobs with GPU flavor." ) sys.exit(1) num_gpus = cuda.device_count() for i in range(num_gpus): gpu_name = cuda.get_device_name(i) gpu_memory = cuda.get_device_properties(i).total_memory / 1024**3 logger.info(f"GPU {i}: {gpu_name} with {gpu_memory:.1f} GB memory") return num_gpus def create_dataset_card( source_dataset: str, model_id: str, messages_column: str, prompt_column: Optional[str], sampling_params: SamplingParams, tensor_parallel_size: int, num_examples: int, generation_time: str, num_skipped: int = 0, max_model_len_used: Optional[int] = None, ) -> str: """Create a comprehensive dataset card documenting the generation process.""" filtering_section = "" if num_skipped > 0: skip_percentage = (num_skipped / num_examples) * 100 processed = num_examples - num_skipped filtering_section = f""" ### Filtering Statistics - **Total Examples**: {num_examples:,} - **Processed**: {processed:,} ({100 - skip_percentage:.1f}%) - **Skipped (too long)**: {num_skipped:,} ({skip_percentage:.1f}%) - **Max Model Length Used**: {max_model_len_used:,} tokens Note: Prompts exceeding the maximum model length were skipped and have empty responses.""" return f"""--- tags: - generated - vllm - uv-script --- # Generated Responses Dataset This dataset contains generated responses for prompts from [{source_dataset}](https://huggingface.co/datasets/{source_dataset}). ## Generation Details - **Source Dataset**: [{source_dataset}](https://huggingface.co/datasets/{source_dataset}) - **Input Column**: `{prompt_column if prompt_column else messages_column}` ({"plain text prompts" if prompt_column else "chat messages"}) - **Model**: [{model_id}](https://huggingface.co/{model_id}) - **Number of Examples**: {num_examples:,} - **Generation Date**: {generation_time}{filtering_section} ### Sampling Parameters - **Temperature**: {sampling_params.temperature} - **Top P**: {sampling_params.top_p} - **Top K**: {sampling_params.top_k} - **Min P**: {sampling_params.min_p} - **Max Tokens**: {sampling_params.max_tokens} - **Repetition Penalty**: {sampling_params.repetition_penalty} ### Hardware Configuration - **Tensor Parallel Size**: {tensor_parallel_size} - **GPU Configuration**: {tensor_parallel_size} GPU(s) ## Dataset Structure The dataset contains all columns from the source dataset plus: - `response`: The generated response from the model ## Generation Script Generated using the vLLM inference script from [uv-scripts/vllm](https://huggingface.co/datasets/uv-scripts/vllm). To reproduce this generation: ```bash uv run https://huggingface.co/datasets/uv-scripts/vllm/raw/main/generate-responses.py \\ {source_dataset} \\ \\ --model-id {model_id} \\ {"--prompt-column " + prompt_column if prompt_column else "--messages-column " + messages_column} \\ --temperature {sampling_params.temperature} \\ --top-p {sampling_params.top_p} \\ --top-k {sampling_params.top_k} \\ --max-tokens {sampling_params.max_tokens}{f" \\\\\\n --max-model-len {max_model_len_used}" if max_model_len_used else ""} ``` """ def main( src_dataset_hub_id: str, output_dataset_hub_id: str, model_id: str = "Qwen/Qwen3-30B-A3B-Instruct-2507", messages_column: str = "messages", prompt_column: Optional[str] = None, output_column: str = "response", temperature: float = 0.7, top_p: float = 0.8, top_k: int = 20, min_p: float = 0.0, max_tokens: int = 16384, repetition_penalty: float = 1.0, gpu_memory_utilization: float = 0.90, max_model_len: Optional[int] = None, tensor_parallel_size: Optional[int] = None, skip_long_prompts: bool = True, max_samples: Optional[int] = None, hf_token: Optional[str] = None, ): """ Main generation pipeline. Args: src_dataset_hub_id: Input dataset on Hugging Face Hub output_dataset_hub_id: Where to save results on Hugging Face Hub model_id: Hugging Face model ID for generation messages_column: Column name containing chat messages prompt_column: Column name containing plain text prompts (alternative to messages_column) output_column: Column name for generated responses temperature: Sampling temperature top_p: Top-p sampling parameter top_k: Top-k sampling parameter min_p: Minimum probability threshold max_tokens: Maximum tokens to generate repetition_penalty: Repetition penalty parameter gpu_memory_utilization: GPU memory utilization factor max_model_len: Maximum model context length (None uses model default) tensor_parallel_size: Number of GPUs to use (auto-detect if None) skip_long_prompts: Skip prompts exceeding max_model_len instead of failing max_samples: Maximum number of samples to process (None for all) hf_token: Hugging Face authentication token """ generation_start_time = datetime.now().isoformat() # GPU check and configuration num_gpus = check_gpu_availability() if tensor_parallel_size is None: tensor_parallel_size = num_gpus logger.info( f"Auto-detected {num_gpus} GPU(s), using tensor_parallel_size={tensor_parallel_size}" ) else: logger.info(f"Using specified tensor_parallel_size={tensor_parallel_size}") if tensor_parallel_size > num_gpus: logger.warning( f"Requested {tensor_parallel_size} GPUs but only {num_gpus} available" ) # Authentication - try multiple methods HF_TOKEN = hf_token or os.environ.get("HF_TOKEN") or get_token() if not HF_TOKEN: logger.error("No HuggingFace token found. Please provide token via:") logger.error(" 1. --hf-token argument") logger.error(" 2. HF_TOKEN environment variable") logger.error(" 3. Run 'hf auth login' or use login() in Python") sys.exit(1) logger.info("HuggingFace token found, authenticating...") login(token=HF_TOKEN) # Initialize vLLM logger.info(f"Loading model: {model_id}") vllm_kwargs = { "model": model_id, "tensor_parallel_size": tensor_parallel_size, "gpu_memory_utilization": gpu_memory_utilization, } if max_model_len is not None: vllm_kwargs["max_model_len"] = max_model_len logger.info(f"Using max_model_len={max_model_len}") llm = LLM(**vllm_kwargs) # Load tokenizer for chat template logger.info("Loading tokenizer...") tokenizer = AutoTokenizer.from_pretrained(model_id) # Create sampling parameters sampling_params = SamplingParams( temperature=temperature, top_p=top_p, top_k=top_k, min_p=min_p, max_tokens=max_tokens, repetition_penalty=repetition_penalty, ) # Load dataset logger.info(f"Loading dataset: {src_dataset_hub_id}") dataset = load_dataset(src_dataset_hub_id, split="train") # Apply max_samples if specified if max_samples is not None and max_samples < len(dataset): logger.info(f"Limiting dataset to {max_samples} samples") dataset = dataset.select(range(max_samples)) total_examples = len(dataset) logger.info(f"Dataset loaded with {total_examples:,} examples") # Determine which column to use and validate if prompt_column: # Use prompt column mode if prompt_column not in dataset.column_names: logger.error( f"Column '{prompt_column}' not found. Available columns: {dataset.column_names}" ) sys.exit(1) logger.info(f"Using prompt column mode with column: '{prompt_column}'") use_messages = False else: # Use messages column mode if messages_column not in dataset.column_names: logger.error( f"Column '{messages_column}' not found. Available columns: {dataset.column_names}" ) sys.exit(1) logger.info(f"Using messages column mode with column: '{messages_column}'") use_messages = True # Get effective max length for filtering if max_model_len is not None: effective_max_len = max_model_len else: # Get model's default max length effective_max_len = llm.llm_engine.model_config.max_model_len logger.info(f"Using effective max model length: {effective_max_len}") # Process messages and apply chat template logger.info("Preparing prompts...") all_prompts = [] valid_prompts = [] valid_indices = [] skipped_info = [] for i, example in enumerate(tqdm(dataset, desc="Processing prompts")): if use_messages: # Messages mode: use existing chat messages messages = example[messages_column] # Apply chat template prompt = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True ) else: # Prompt mode: convert plain text to messages format user_prompt = example[prompt_column] messages = [{"role": "user", "content": user_prompt}] # Apply chat template prompt = tokenizer.apply_chat_template( messages, tokenize=False, add_generation_prompt=True ) all_prompts.append(prompt) # Count tokens if filtering is enabled if skip_long_prompts: tokens = tokenizer.encode(prompt) if len(tokens) <= effective_max_len: valid_prompts.append(prompt) valid_indices.append(i) else: skipped_info.append((i, len(tokens))) else: valid_prompts.append(prompt) valid_indices.append(i) # Log filtering results if skip_long_prompts and skipped_info: logger.warning( f"Skipped {len(skipped_info)} prompts that exceed max_model_len ({effective_max_len} tokens)" ) logger.info("Skipped prompt details (first 10):") for idx, (prompt_idx, token_count) in enumerate(skipped_info[:10]): logger.info( f" - Example {prompt_idx}: {token_count} tokens (exceeds by {token_count - effective_max_len})" ) if len(skipped_info) > 10: logger.info(f" ... and {len(skipped_info) - 10} more") skip_percentage = (len(skipped_info) / total_examples) * 100 if skip_percentage > 10: logger.warning(f"WARNING: {skip_percentage:.1f}% of prompts were skipped!") if not valid_prompts: logger.error("No valid prompts to process after filtering!") sys.exit(1) # Generate responses - vLLM handles batching internally logger.info(f"Starting generation for {len(valid_prompts):,} valid prompts...") logger.info("vLLM will handle batching and scheduling automatically") outputs = llm.generate(valid_prompts, sampling_params) # Extract generated text and create full response list logger.info("Extracting generated responses...") responses = [""] * total_examples # Initialize with empty strings for idx, output in enumerate(outputs): original_idx = valid_indices[idx] response = output.outputs[0].text.strip() responses[original_idx] = response # Add responses to dataset logger.info("Adding responses to dataset...") dataset = dataset.add_column(output_column, responses) # Create dataset card logger.info("Creating dataset card...") card_content = create_dataset_card( source_dataset=src_dataset_hub_id, model_id=model_id, messages_column=messages_column, prompt_column=prompt_column, sampling_params=sampling_params, tensor_parallel_size=tensor_parallel_size, num_examples=total_examples, generation_time=generation_start_time, num_skipped=len(skipped_info) if skip_long_prompts else 0, max_model_len_used=effective_max_len if skip_long_prompts else None, ) # Push dataset to hub logger.info(f"Pushing dataset to: {output_dataset_hub_id}") dataset.push_to_hub(output_dataset_hub_id, token=HF_TOKEN) # Push dataset card card = DatasetCard(card_content) card.push_to_hub(output_dataset_hub_id, token=HF_TOKEN) logger.info("✅ Generation complete!") logger.info( f"Dataset available at: https://huggingface.co/datasets/{output_dataset_hub_id}" ) if __name__ == "__main__": if len(sys.argv) > 1: parser = argparse.ArgumentParser( description="Generate responses for dataset prompts using vLLM", formatter_class=argparse.RawDescriptionHelpFormatter, epilog=""" Examples: # Basic usage with default Qwen model uv run generate-responses.py input-dataset output-dataset # With custom model and parameters uv run generate-responses.py input-dataset output-dataset \\ --model-id meta-llama/Llama-3.1-8B-Instruct \\ --temperature 0.9 \\ --max-tokens 2048 # Force specific GPU configuration uv run generate-responses.py input-dataset output-dataset \\ --tensor-parallel-size 2 \\ --gpu-memory-utilization 0.95 # Using environment variable for token HF_TOKEN=hf_xxx uv run generate-responses.py input-dataset output-dataset """, ) parser.add_argument( "src_dataset_hub_id", help="Input dataset on Hugging Face Hub (e.g., username/dataset-name)", ) parser.add_argument( "output_dataset_hub_id", help="Output dataset name on Hugging Face Hub" ) parser.add_argument( "--model-id", type=str, default="Qwen/Qwen3-30B-A3B-Instruct-2507", help="Model to use for generation (default: Qwen3-30B-A3B-Instruct-2507)", ) parser.add_argument( "--messages-column", type=str, default="messages", help="Column containing chat messages (default: messages)", ) parser.add_argument( "--prompt-column", type=str, help="Column containing plain text prompts (alternative to --messages-column)", ) parser.add_argument( "--output-column", type=str, default="response", help="Column name for generated responses (default: response)", ) parser.add_argument( "--max-samples", type=int, help="Maximum number of samples to process (default: all)", ) parser.add_argument( "--temperature", type=float, default=0.7, help="Sampling temperature (default: 0.7)", ) parser.add_argument( "--top-p", type=float, default=0.8, help="Top-p sampling parameter (default: 0.8)", ) parser.add_argument( "--top-k", type=int, default=20, help="Top-k sampling parameter (default: 20)", ) parser.add_argument( "--min-p", type=float, default=0.0, help="Minimum probability threshold (default: 0.0)", ) parser.add_argument( "--max-tokens", type=int, default=16384, help="Maximum tokens to generate (default: 16384)", ) parser.add_argument( "--repetition-penalty", type=float, default=1.0, help="Repetition penalty (default: 1.0)", ) parser.add_argument( "--gpu-memory-utilization", type=float, default=0.90, help="GPU memory utilization factor (default: 0.90)", ) parser.add_argument( "--max-model-len", type=int, help="Maximum model context length (default: model's default)", ) parser.add_argument( "--tensor-parallel-size", type=int, help="Number of GPUs to use (default: auto-detect)", ) parser.add_argument( "--hf-token", type=str, help="Hugging Face token (can also use HF_TOKEN env var)", ) parser.add_argument( "--skip-long-prompts", action="store_true", default=True, help="Skip prompts that exceed max_model_len instead of failing (default: True)", ) parser.add_argument( "--no-skip-long-prompts", dest="skip_long_prompts", action="store_false", help="Fail on prompts that exceed max_model_len", ) args = parser.parse_args() main( src_dataset_hub_id=args.src_dataset_hub_id, output_dataset_hub_id=args.output_dataset_hub_id, model_id=args.model_id, messages_column=args.messages_column, prompt_column=args.prompt_column, output_column=args.output_column, temperature=args.temperature, top_p=args.top_p, top_k=args.top_k, min_p=args.min_p, max_tokens=args.max_tokens, repetition_penalty=args.repetition_penalty, gpu_memory_utilization=args.gpu_memory_utilization, max_model_len=args.max_model_len, tensor_parallel_size=args.tensor_parallel_size, skip_long_prompts=args.skip_long_prompts, max_samples=args.max_samples, hf_token=args.hf_token, ) else: # Show HF Jobs example when run without arguments print(""" vLLM Response Generation Script ============================== This script requires arguments. For usage information: uv run generate-responses.py --help Example HF Jobs command with multi-GPU: # If you're logged in with hf auth, token will be auto-detected hf jobs uv run \\ --flavor l4x4 \\ https://huggingface.co/datasets/uv-scripts/vllm/raw/main/generate-responses.py \\ username/input-dataset \\ username/output-dataset \\ --messages-column messages \\ --model-id Qwen/Qwen3-30B-A3B-Instruct-2507 \\ --temperature 0.7 \\ --max-tokens 16384 """) ================================================ FILE: skills/hugging-face-model-trainer/SKILL.md ================================================ --- name: hugging-face-model-trainer description: This skill should be used when users want to train or fine-tune language models using TRL (Transformer Reinforcement Learning) on Hugging Face Jobs infrastructure. Covers SFT, DPO, GRPO and reward modeling training methods, plus GGUF conversion for local deployment. Includes guidance on the TRL Jobs package, UV scripts with PEP 723 format, dataset preparation and validation, hardware selection, cost estimation, Trackio monitoring, Hub authentication, and model persistence. Should be invoked for tasks involving cloud GPU training, GGUF conversion, or when users mention training on Hugging Face Jobs without local GPU setup. license: Complete terms in LICENSE.txt --- # TRL Training on Hugging Face Jobs ## Overview Train language models using TRL (Transformer Reinforcement Learning) on fully managed Hugging Face infrastructure. No local GPU setup required—models train on cloud GPUs and results are automatically saved to the Hugging Face Hub. **TRL provides multiple training methods:** - **SFT** (Supervised Fine-Tuning) - Standard instruction tuning - **DPO** (Direct Preference Optimization) - Alignment from preference data - **GRPO** (Group Relative Policy Optimization) - Online RL training - **Reward Modeling** - Train reward models for RLHF **For detailed TRL method documentation:** ```python hf_doc_search("your query", product="trl") hf_doc_fetch("https://huggingface.co/docs/trl/sft_trainer") # SFT hf_doc_fetch("https://huggingface.co/docs/trl/dpo_trainer") # DPO # etc. ``` **See also:** `references/training_methods.md` for method overviews and selection guidance ## When to Use This Skill Use this skill when users want to: - Fine-tune language models on cloud GPUs without local infrastructure - Train with TRL methods (SFT, DPO, GRPO, etc.) - Run training jobs on Hugging Face Jobs infrastructure - Convert trained models to GGUF for local deployment (Ollama, LM Studio, llama.cpp) - Ensure trained models are permanently saved to the Hub - Use modern workflows with optimized defaults ### When to Use Unsloth Use **Unsloth** (`references/unsloth.md`) instead of standard TRL when: - **Limited GPU memory** - Unsloth uses ~60% less VRAM - **Speed matters** - Unsloth is ~2x faster - Training **large models (>13B)** - memory efficiency is critical - Training **Vision-Language Models (VLMs)** - Unsloth has `FastVisionModel` support See `references/unsloth.md` for complete Unsloth documentation and `scripts/unsloth_sft_example.py` for a production-ready training script. ## Key Directives When assisting with training jobs: 1. **ALWAYS use `hf_jobs()` MCP tool** - Submit jobs using `hf_jobs("uv", {...})`, NOT bash `trl-jobs` commands. The `script` parameter accepts Python code directly. Do NOT save to local files unless the user explicitly requests it. Pass the script content as a string to `hf_jobs()`. If user asks to "train a model", "fine-tune", or similar requests, you MUST create the training script AND submit the job immediately using `hf_jobs()`. 2. **Always include Trackio** - Every training script should include Trackio for real-time monitoring. Use example scripts in `scripts/` as templates. 3. **Provide job details after submission** - After submitting, provide job ID, monitoring URL, estimated time, and note that the user can request status checks later. 4. **Use example scripts as templates** - Reference `scripts/train_sft_example.py`, `scripts/train_dpo_example.py`, etc. as starting points. ## Local Script Execution Repository scripts use PEP 723 inline dependencies. Run them with `uv run`: ```bash uv run scripts/estimate_cost.py --help uv run scripts/dataset_inspector.py --help ``` ## Prerequisites Checklist Before starting any training job, verify: ### ✅ **Account & Authentication** - Hugging Face Account with [Pro](https://hf.co/pro), [Team](https://hf.co/enterprise), or [Enterprise](https://hf.co/enterprise) plan (Jobs require paid plan) - Authenticated login: Check with `hf_whoami()` - **HF_TOKEN for Hub Push** ⚠️ CRITICAL - Training environment is ephemeral, must push to Hub or ALL training results are lost - Token must have write permissions - **MUST pass `secrets={"HF_TOKEN": "$HF_TOKEN"}` in job config** to make token available (the `$HF_TOKEN` syntax references your actual token value) ### ✅ **Dataset Requirements** - Dataset must exist on Hub or be loadable via `datasets.load_dataset()` - Format must match training method (SFT: "messages"/text/prompt-completion; DPO: chosen/rejected; GRPO: prompt-only) - **ALWAYS validate unknown datasets** before GPU training to prevent format failures (see Dataset Validation section below) - Size appropriate for hardware (Demo: 50-100 examples on t4-small; Production: 1K-10K+ on a10g-large/a100-large) ### ⚠️ **Critical Settings** - **Timeout must exceed expected training time** - Default 30min is TOO SHORT for most training. Minimum recommended: 1-2 hours. Job fails and loses all progress if timeout is exceeded. - **Hub push must be enabled** - Config: `push_to_hub=True`, `hub_model_id="username/model-name"`; Job: `secrets={"HF_TOKEN": "$HF_TOKEN"}` ## Asynchronous Job Guidelines **⚠️ IMPORTANT: Training jobs run asynchronously and can take hours** ### Action Required **When user requests training:** 1. **Create the training script** with Trackio included (use `scripts/train_sft_example.py` as template) 2. **Submit immediately** using `hf_jobs()` MCP tool with script content inline - don't save to file unless user requests 3. **Report submission** with job ID, monitoring URL, and estimated time 4. **Wait for user** to request status checks - don't poll automatically ### Ground Rules - **Jobs run in background** - Submission returns immediately; training continues independently - **Initial logs delayed** - Can take 30-60 seconds for logs to appear - **User checks status** - Wait for user to request status updates - **Avoid polling** - Check logs only on user request; provide monitoring links instead ### After Submission **Provide to user:** - ✅ Job ID and monitoring URL - ✅ Expected completion time - ✅ Trackio dashboard URL - ✅ Note that user can request status checks later **Example Response:** ``` ✅ Job submitted successfully! Job ID: abc123xyz Monitor: https://huggingface.co/jobs/username/abc123xyz Expected time: ~2 hours Estimated cost: ~$10 The job is running in the background. Ask me to check status/logs when ready! ``` ## Quick Start: Three Approaches **💡 Tip for Demos:** For quick demos on smaller GPUs (t4-small), omit `eval_dataset` and `eval_strategy` to save ~40% memory. You'll still see training loss and learning progress. ### Sequence Length Configuration **TRL config classes use `max_length` (not `max_seq_length`)** to control tokenized sequence length: ```python # ✅ CORRECT - If you need to set sequence length SFTConfig(max_length=512) # Truncate sequences to 512 tokens DPOConfig(max_length=2048) # Longer context (2048 tokens) # ❌ WRONG - This parameter doesn't exist SFTConfig(max_seq_length=512) # TypeError! ``` **Default behavior:** `max_length=1024` (truncates from right). This works well for most training. **When to override:** - **Longer context**: Set higher (e.g., `max_length=2048`) - **Memory constraints**: Set lower (e.g., `max_length=512`) - **Vision models**: Set `max_length=None` (prevents cutting image tokens) **Usually you don't need to set this parameter at all** - the examples below use the sensible default. ### Approach 1: UV Scripts (Recommended—Default Choice) UV scripts use PEP 723 inline dependencies for clean, self-contained training. **This is the primary approach for Claude Code.** ```python hf_jobs("uv", { "script": """ # /// script # dependencies = ["trl>=0.12.0", "peft>=0.7.0", "trackio"] # /// from datasets import load_dataset from peft import LoraConfig from trl import SFTTrainer, SFTConfig import trackio dataset = load_dataset("trl-lib/Capybara", split="train") # Create train/eval split for monitoring dataset_split = dataset.train_test_split(test_size=0.1, seed=42) trainer = SFTTrainer( model="Qwen/Qwen2.5-0.5B", train_dataset=dataset_split["train"], eval_dataset=dataset_split["test"], peft_config=LoraConfig(r=16, lora_alpha=32), args=SFTConfig( output_dir="my-model", push_to_hub=True, hub_model_id="username/my-model", num_train_epochs=3, eval_strategy="steps", eval_steps=50, report_to="trackio", project="meaningful_prject_name", # project name for the training name (trackio) run_name="meaningful_run_name", # descriptive name for the specific training run (trackio) ) ) trainer.train() trainer.push_to_hub() """, "flavor": "a10g-large", "timeout": "2h", "secrets": {"HF_TOKEN": "$HF_TOKEN"} }) ``` **Benefits:** Direct MCP tool usage, clean code, dependencies declared inline (PEP 723), no file saving required, full control **When to use:** Default choice for all training tasks in Claude Code, custom training logic, any scenario requiring `hf_jobs()` #### Working with Scripts ⚠️ **Important:** The `script` parameter accepts either inline code (as shown above) OR a URL. **Local file paths do NOT work.** **Why local paths don't work:** Jobs run in isolated Docker containers without access to your local filesystem. Scripts must be: - Inline code (recommended for custom training) - Publicly accessible URLs - Private repo URLs (with HF_TOKEN) **Common mistakes:** ```python # ❌ These will all fail hf_jobs("uv", {"script": "train.py"}) hf_jobs("uv", {"script": "./scripts/train.py"}) hf_jobs("uv", {"script": "/path/to/train.py"}) ``` **Correct approaches:** ```python # ✅ Inline code (recommended) hf_jobs("uv", {"script": "# /// script\n# dependencies = [...]\n# ///\n\n"}) # ✅ From Hugging Face Hub hf_jobs("uv", {"script": "https://huggingface.co/user/repo/resolve/main/train.py"}) # ✅ From GitHub hf_jobs("uv", {"script": "https://raw.githubusercontent.com/user/repo/main/train.py"}) # ✅ From Gist hf_jobs("uv", {"script": "https://gist.githubusercontent.com/user/id/raw/train.py"}) ``` **To use local scripts:** Upload to HF Hub first: ```bash hf repos create my-training-scripts --type model hf upload my-training-scripts ./train.py train.py # Use: https://huggingface.co/USERNAME/my-training-scripts/resolve/main/train.py ``` ### Approach 2: TRL Maintained Scripts (Official Examples) TRL provides battle-tested scripts for all methods. Can be run from URLs: ```python hf_jobs("uv", { "script": "https://github.com/huggingface/trl/blob/main/trl/scripts/sft.py", "script_args": [ "--model_name_or_path", "Qwen/Qwen2.5-0.5B", "--dataset_name", "trl-lib/Capybara", "--output_dir", "my-model", "--push_to_hub", "--hub_model_id", "username/my-model" ], "flavor": "a10g-large", "timeout": "2h", "secrets": {"HF_TOKEN": "$HF_TOKEN"} }) ``` **Benefits:** No code to write, maintained by TRL team, production-tested **When to use:** Standard TRL training, quick experiments, don't need custom code **Available:** Scripts are available from https://github.com/huggingface/trl/tree/main/examples/scripts ### Finding More UV Scripts on Hub The `uv-scripts` organization provides ready-to-use UV scripts stored as datasets on Hugging Face Hub: ```python # Discover available UV script collections dataset_search({"author": "uv-scripts", "sort": "downloads", "limit": 20}) # Explore a specific collection hub_repo_details(["uv-scripts/classification"], repo_type="dataset", include_readme=True) ``` **Popular collections:** ocr, classification, synthetic-data, vllm, dataset-creation ### Approach 3: HF Jobs CLI (Direct Terminal Commands) When the `hf_jobs()` MCP tool is unavailable, use the `hf jobs` CLI directly. **⚠️ CRITICAL: CLI Syntax Rules** ```bash # ✅ CORRECT syntax - flags BEFORE script URL hf jobs uv run --flavor a10g-large --timeout 2h --secrets HF_TOKEN "https://example.com/train.py" # ❌ WRONG - "run uv" instead of "uv run" hf jobs run uv "https://example.com/train.py" --flavor a10g-large # ❌ WRONG - flags AFTER script URL (will be ignored!) hf jobs uv run "https://example.com/train.py" --flavor a10g-large # ❌ WRONG - "--secret" instead of "--secrets" (plural) hf jobs uv run --secret HF_TOKEN "https://example.com/train.py" ``` **Key syntax rules:** 1. Command order is `hf jobs uv run` (NOT `hf jobs run uv`) 2. All flags (`--flavor`, `--timeout`, `--secrets`) must come BEFORE the script URL 3. Use `--secrets` (plural), not `--secret` 4. Script URL must be the last positional argument **Complete CLI example:** ```bash hf jobs uv run \ --flavor a10g-large \ --timeout 2h \ --secrets HF_TOKEN \ "https://huggingface.co/user/repo/resolve/main/train.py" ``` **Check job status via CLI:** ```bash hf jobs ps # List all jobs hf jobs logs # View logs hf jobs inspect # Job details hf jobs cancel # Cancel a job ``` ### Approach 4: TRL Jobs Package (Simplified Training) The `trl-jobs` package provides optimized defaults and one-liner training. ```bash uvx trl-jobs sft \ --model_name Qwen/Qwen2.5-0.5B \ --dataset_name trl-lib/Capybara ``` **Benefits:** Pre-configured settings, automatic Trackio integration, automatic Hub push, one-line commands **When to use:** User working in terminal directly (not Claude Code context), quick local experimentation **Repository:** https://github.com/huggingface/trl-jobs ⚠️ **In Claude Code context, prefer using `hf_jobs()` MCP tool (Approach 1) when available.** ## Hardware Selection | Model Size | Recommended Hardware | Cost (approx/hr) | Use Case | |------------|---------------------|------------------|----------| | <1B params | `t4-small` | ~$0.75 | Demos, quick tests only without eval steps | | 1-3B params | `t4-medium`, `l4x1` | ~$1.50-2.50 | Development | | 3-7B params | `a10g-small`, `a10g-large` | ~$3.50-5.00 | Production training | | 7-13B params | `a10g-large`, `a100-large` | ~$5-10 | Large models (use LoRA) | | 13B+ params | `a100-large`, `a10g-largex2` | ~$10-20 | Very large (use LoRA) | **GPU Flavors:** cpu-basic/upgrade/performance/xl, t4-small/medium, l4x1/x4, a10g-small/large/largex2/largex4, a100-large, h100/h100x8 **Guidelines:** - Use **LoRA/PEFT** for models >7B to reduce memory - Multi-GPU automatically handled by TRL/Accelerate - Start with smaller hardware for testing **See:** `references/hardware_guide.md` for detailed specifications ## Critical: Saving Results to Hub **⚠️ EPHEMERAL ENVIRONMENT—MUST PUSH TO HUB** The Jobs environment is temporary. All files are deleted when the job ends. If the model isn't pushed to Hub, **ALL TRAINING IS LOST**. ### Required Configuration **In training script/config:** ```python SFTConfig( push_to_hub=True, hub_model_id="username/model-name", # MUST specify hub_strategy="every_save", # Optional: push checkpoints ) ``` **In job submission:** ```python { "secrets": {"HF_TOKEN": "$HF_TOKEN"} # Enables authentication } ``` ### Verification Checklist Before submitting: - [ ] `push_to_hub=True` set in config - [ ] `hub_model_id` includes username/repo-name - [ ] `secrets` parameter includes HF_TOKEN - [ ] User has write access to target repo **See:** `references/hub_saving.md` for detailed troubleshooting ## Timeout Management **⚠️ DEFAULT: 30 MINUTES—TOO SHORT FOR TRAINING** ### Setting Timeouts ```python { "timeout": "2h" # 2 hours (formats: "90m", "2h", "1.5h", or seconds as integer) } ``` ### Timeout Guidelines | Scenario | Recommended | Notes | |----------|-------------|-------| | Quick demo (50-100 examples) | 10-30 min | Verify setup | | Development training | 1-2 hours | Small datasets | | Production (3-7B model) | 4-6 hours | Full datasets | | Large model with LoRA | 3-6 hours | Depends on dataset | **Always add 20-30% buffer** for model/dataset loading, checkpoint saving, Hub push operations, and network delays. **On timeout:** Job killed immediately, all unsaved progress lost, must restart from beginning ## Cost Estimation **Offer to estimate cost when planning jobs with known parameters.** Use `scripts/estimate_cost.py`: ```bash uv run scripts/estimate_cost.py \ --model meta-llama/Llama-2-7b-hf \ --dataset trl-lib/Capybara \ --hardware a10g-large \ --dataset-size 16000 \ --epochs 3 ``` Output includes estimated time, cost, recommended timeout (with buffer), and optimization suggestions. **When to offer:** User planning a job, asks about cost/time, choosing hardware, job will run >1 hour or cost >$5 ## Example Training Scripts **Production-ready templates with all best practices:** Load these scripts for correctly: - **`scripts/train_sft_example.py`** - Complete SFT training with Trackio, LoRA, checkpoints - **`scripts/train_dpo_example.py`** - DPO training for preference learning - **`scripts/train_grpo_example.py`** - GRPO training for online RL These scripts demonstrate proper Hub saving, Trackio integration, checkpoint management, and optimized parameters. Pass their content inline to `hf_jobs()` or use as templates for custom scripts. ## Monitoring and Tracking **Trackio** provides real-time metrics visualization. See `references/trackio_guide.md` for complete setup guide. **Key points:** - Add `trackio` to dependencies - Configure trainer with `report_to="trackio" and run_name="meaningful_name"` ### Trackio Configuration Defaults **Use sensible defaults unless user specifies otherwise.** When generating training scripts with Trackio: **Default Configuration:** - **Space ID**: `{username}/trackio` (use "trackio" as default space name) - **Run naming**: Unless otherwise specified, name the run in a way the user will recognize (e.g., descriptive of the task, model, or purpose) - **Config**: Keep minimal - only include hyperparameters and model/dataset info - **Project Name**: Use a Project Name to associate runs with a particular Project **User overrides:** If user requests specific trackio configuration (custom space, run naming, grouping, or additional config), apply their preferences instead of defaults. This is useful for managing multiple jobs with the same configuration or keeping training scripts portable. See `references/trackio_guide.md` for complete documentation including grouping runs for experiments. ### Check Job Status ```python # List all jobs hf_jobs("ps") # Inspect specific job hf_jobs("inspect", {"job_id": "your-job-id"}) # View logs hf_jobs("logs", {"job_id": "your-job-id"}) ``` **Remember:** Wait for user to request status checks. Avoid polling repeatedly. ## Dataset Validation **Validate dataset format BEFORE launching GPU training to prevent the #1 cause of training failures: format mismatches.** ### Why Validate - 50%+ of training failures are due to dataset format issues - DPO especially strict: requires exact column names (`prompt`, `chosen`, `rejected`) - Failed GPU jobs waste $1-10 and 30-60 minutes - Validation on CPU costs ~$0.01 and takes <1 minute ### When to Validate **ALWAYS validate for:** - Unknown or custom datasets - DPO training (CRITICAL - 90% of datasets need mapping) - Any dataset not explicitly TRL-compatible **Skip validation for known TRL datasets:** - `trl-lib/ultrachat_200k`, `trl-lib/Capybara`, `HuggingFaceH4/ultrachat_200k`, etc. ### Usage ```python hf_jobs("uv", { "script": "https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py", "script_args": ["--dataset", "username/dataset-name", "--split", "train"] }) ``` The script is fast, and will usually complete synchronously. ### Reading Results The output shows compatibility for each training method: - **`✓ READY`** - Dataset is compatible, use directly - **`✗ NEEDS MAPPING`** - Compatible but needs preprocessing (mapping code provided) - **`✗ INCOMPATIBLE`** - Cannot be used for this method When mapping is needed, the output includes a **"MAPPING CODE"** section with copy-paste ready Python code. ### Example Workflow ```python # 1. Inspect dataset (costs ~$0.01, <1 min on CPU) hf_jobs("uv", { "script": "https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py", "script_args": ["--dataset", "argilla/distilabel-math-preference-dpo", "--split", "train"] }) # 2. Check output markers: # ✓ READY → proceed with training # ✗ NEEDS MAPPING → apply mapping code below # ✗ INCOMPATIBLE → choose different method/dataset # 3. If mapping needed, apply before training: def format_for_dpo(example): return { 'prompt': example['instruction'], 'chosen': example['chosen_response'], 'rejected': example['rejected_response'], } dataset = dataset.map(format_for_dpo, remove_columns=dataset.column_names) # 4. Launch training job with confidence ``` ### Common Scenario: DPO Format Mismatch Most DPO datasets use non-standard column names. Example: ``` Dataset has: instruction, chosen_response, rejected_response DPO expects: prompt, chosen, rejected ``` The validator detects this and provides exact mapping code to fix it. ## Converting Models to GGUF After training, convert models to **GGUF format** for use with llama.cpp, Ollama, LM Studio, and other local inference tools. **What is GGUF:** - Optimized for CPU/GPU inference with llama.cpp - Supports quantization (4-bit, 5-bit, 8-bit) to reduce model size - Compatible with Ollama, LM Studio, Jan, GPT4All, llama.cpp - Typically 2-8GB for 7B models (vs 14GB unquantized) **When to convert:** - Running models locally with Ollama or LM Studio - Reducing model size with quantization - Deploying to edge devices - Sharing models for local-first use **See:** `references/gguf_conversion.md` for complete conversion guide, including production-ready conversion script, quantization options, hardware requirements, usage examples, and troubleshooting. **Quick conversion:** ```python hf_jobs("uv", { "script": "", "flavor": "a10g-large", "timeout": "45m", "secrets": {"HF_TOKEN": "$HF_TOKEN"}, "env": { "ADAPTER_MODEL": "username/my-finetuned-model", "BASE_MODEL": "Qwen/Qwen2.5-0.5B", "OUTPUT_REPO": "username/my-model-gguf" } }) ``` ## Common Training Patterns See `references/training_patterns.md` for detailed examples including: - Quick demo (5-10 minutes) - Production with checkpoints - Multi-GPU training - DPO training (preference learning) - GRPO training (online RL) ## Common Failure Modes ### Out of Memory (OOM) **Fix (try in order):** 1. Reduce batch size: `per_device_train_batch_size=1`, increase `gradient_accumulation_steps=8`. Effective batch size is `per_device_train_batch_size` x `gradient_accumulation_steps`. For best performance keep effective batch size close to 128. 2. Enable: `gradient_checkpointing=True` 3. Upgrade hardware: t4-small → l4x1, a10g-small → a10g-large etc. ### Dataset Misformatted **Fix:** 1. Validate first with dataset inspector: ```bash uv run https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py \ --dataset name --split train ``` 2. Check output for compatibility markers (✓ READY, ✗ NEEDS MAPPING, ✗ INCOMPATIBLE) 3. Apply mapping code from inspector output if needed ### Job Timeout **Fix:** 1. Check logs for actual runtime: `hf_jobs("logs", {"job_id": "..."})` 2. Increase timeout with buffer: `"timeout": "3h"` (add 30% to estimated time) 3. Or reduce training: lower `num_train_epochs`, use smaller dataset, enable `max_steps` 4. Save checkpoints: `save_strategy="steps"`, `save_steps=500`, `hub_strategy="every_save"` **Note:** Default 30min is insufficient for real training. Minimum 1-2 hours. ### Hub Push Failures **Fix:** 1. Add to job: `secrets={"HF_TOKEN": "$HF_TOKEN"}` 2. Add to config: `push_to_hub=True`, `hub_model_id="username/model-name"` 3. Verify auth: `mcp__huggingface__hf_whoami()` 4. Check token has write permissions and repo exists (or set `hub_private_repo=True`) ### Missing Dependencies **Fix:** Add to PEP 723 header: ```python # /// script # dependencies = ["trl>=0.12.0", "peft>=0.7.0", "trackio", "missing-package"] # /// ``` ## Troubleshooting **Common issues:** - Job times out → Increase timeout, reduce epochs/dataset, use smaller model/LoRA - Model not saved to Hub → Check push_to_hub=True, hub_model_id, secrets=HF_TOKEN - Out of Memory (OOM) → Reduce batch size, increase gradient accumulation, enable LoRA, use larger GPU - Dataset format error → Validate with dataset inspector (see Dataset Validation section) - Import/module errors → Add PEP 723 header with dependencies, verify format - Authentication errors → Check `mcp__huggingface__hf_whoami()`, token permissions, secrets parameter **See:** `references/troubleshooting.md` for complete troubleshooting guide ## Resources ### References (In This Skill) - `references/training_methods.md` - Overview of SFT, DPO, GRPO, KTO, PPO, Reward Modeling - `references/training_patterns.md` - Common training patterns and examples - `references/unsloth.md` - Unsloth for fast VLM training (~2x speed, 60% less VRAM) - `references/gguf_conversion.md` - Complete GGUF conversion guide - `references/trackio_guide.md` - Trackio monitoring setup - `references/hardware_guide.md` - Hardware specs and selection - `references/hub_saving.md` - Hub authentication troubleshooting - `references/troubleshooting.md` - Common issues and solutions - `references/local_training_macos.md` - Local training on macOS ### Scripts (In This Skill) - `scripts/train_sft_example.py` - Production SFT template - `scripts/train_dpo_example.py` - Production DPO template - `scripts/train_grpo_example.py` - Production GRPO template - `scripts/unsloth_sft_example.py` - Unsloth text LLM training template (faster, less VRAM) - `scripts/estimate_cost.py` - Estimate time and cost (offer when appropriate) - `scripts/convert_to_gguf.py` - Complete GGUF conversion script ### External Scripts - [Dataset Inspector](https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py) - Validate dataset format before training (use via `uv run` or `hf_jobs`) ### External Links - [TRL Documentation](https://huggingface.co/docs/trl) - [TRL Jobs Training Guide](https://huggingface.co/docs/trl/en/jobs_training) - [TRL Jobs Package](https://github.com/huggingface/trl-jobs) - [HF Jobs Documentation](https://huggingface.co/docs/huggingface_hub/guides/jobs) - [TRL Example Scripts](https://github.com/huggingface/trl/tree/main/examples/scripts) - [UV Scripts Guide](https://docs.astral.sh/uv/guides/scripts/) - [UV Scripts Organization](https://huggingface.co/uv-scripts) ## Key Takeaways 1. **Submit scripts inline** - The `script` parameter accepts Python code directly; no file saving required unless user requests 2. **Jobs are asynchronous** - Don't wait/poll; let user check when ready 3. **Always set timeout** - Default 30 min is insufficient; minimum 1-2 hours recommended 4. **Always enable Hub push** - Environment is ephemeral; without push, all results lost 5. **Include Trackio** - Use example scripts as templates for real-time monitoring 6. **Offer cost estimation** - When parameters are known, use `scripts/estimate_cost.py` 7. **Use UV scripts (Approach 1)** - Default to `hf_jobs("uv", {...})` with inline scripts; TRL maintained scripts for standard training; avoid bash `trl-jobs` commands in Claude Code 8. **Use hf_doc_fetch/hf_doc_search** for latest TRL documentation 9. **Validate dataset format** before training with dataset inspector (see Dataset Validation section) 10. **Choose appropriate hardware** for model size; use LoRA for models >7B ================================================ FILE: skills/hugging-face-model-trainer/references/gguf_conversion.md ================================================ # GGUF Conversion Guide After training models with TRL on Hugging Face Jobs, convert them to **GGUF format** for use with llama.cpp, Ollama, LM Studio, and other local inference tools. **This guide provides production-ready, tested code based on successful conversions.** All critical dependencies and build steps are included. ## What is GGUF? **GGUF** (GPT-Generated Unified Format): - Optimized format for CPU/GPU inference with llama.cpp - Supports quantization (4-bit, 5-bit, 8-bit) to reduce model size - Compatible with: Ollama, LM Studio, Jan, GPT4All, llama.cpp - Typically 2-8GB for 7B models (vs 14GB unquantized) ## When to Convert to GGUF **Convert when:** - Running models locally with Ollama or LM Studio - Using CPU-optimized inference - Reducing model size with quantization - Deploying to edge devices - Sharing models for local-first use ## Critical Success Factors Based on production testing, these are **essential** for reliable conversion: ### 1. ✅ Install Build Tools FIRST **Before cloning llama.cpp**, install build dependencies: ```python subprocess.run(["apt-get", "update", "-qq"], check=True, capture_output=True) subprocess.run(["apt-get", "install", "-y", "-qq", "build-essential", "cmake"], check=True, capture_output=True) ``` **Why:** The quantization tool requires gcc and cmake. Installing after cloning doesn't help. ### 2. ✅ Use CMake (Not Make) **Build the quantize tool with CMake:** ```python # Create build directory os.makedirs("/tmp/llama.cpp/build", exist_ok=True) # Configure subprocess.run([ "cmake", "-B", "/tmp/llama.cpp/build", "-S", "/tmp/llama.cpp", "-DGGML_CUDA=OFF" # Faster build, CUDA not needed for quantization ], check=True, capture_output=True, text=True) # Build subprocess.run([ "cmake", "--build", "/tmp/llama.cpp/build", "--target", "llama-quantize", "-j", "4" ], check=True, capture_output=True, text=True) # Binary path quantize_bin = "/tmp/llama.cpp/build/bin/llama-quantize" ``` **Why:** CMake is more reliable than `make` and produces consistent binary paths. ### 3. ✅ Include All Dependencies **PEP 723 header must include:** ```python # /// script # dependencies = [ # "transformers>=4.36.0", # "peft>=0.7.0", # "torch>=2.0.0", # "accelerate>=0.24.0", # "huggingface_hub>=0.20.0", # "sentencepiece>=0.1.99", # Required for tokenizer # "protobuf>=3.20.0", # Required for tokenizer # "numpy", # "gguf", # ] # /// ``` **Why:** `sentencepiece` and `protobuf` are critical for tokenizer conversion. Missing them causes silent failures. ### 4. ✅ Verify Names Before Use **Always verify repos exist:** ```python # Before submitting job, verify: hub_repo_details([ADAPTER_MODEL], repo_type="model") hub_repo_details([BASE_MODEL], repo_type="model") ``` **Why:** Non-existent dataset/model names cause job failures that could be caught in seconds. ## Complete Conversion Script See `scripts/convert_to_gguf.py` for the complete, production-ready script. **Key features:** - ✅ All dependencies in PEP 723 header - ✅ Build tools installed automatically - ✅ CMake build process (reliable) - ✅ Comprehensive error handling - ✅ Environment variable configuration - ✅ Automatic README generation ## Quick Conversion Job ```python # Before submitting: VERIFY MODELS EXIST hub_repo_details(["username/my-finetuned-model"], repo_type="model") hub_repo_details(["Qwen/Qwen2.5-0.5B"], repo_type="model") # Submit conversion job hf_jobs("uv", { "script": open("trl/scripts/convert_to_gguf.py").read(), # Or inline the script "flavor": "a10g-large", "timeout": "45m", "secrets": {"HF_TOKEN": "$HF_TOKEN"}, "env": { "ADAPTER_MODEL": "username/my-finetuned-model", "BASE_MODEL": "Qwen/Qwen2.5-0.5B", "OUTPUT_REPO": "username/my-model-gguf", "HF_USERNAME": "username" # Optional, for README } }) ``` ## Conversion Process The script performs these steps: 1. **Load and Merge** - Load base model and LoRA adapter, merge them 2. **Install Build Tools** - Install gcc, cmake (CRITICAL: before cloning llama.cpp) 3. **Setup llama.cpp** - Clone repo, install Python dependencies 4. **Convert to GGUF** - Create FP16 GGUF using llama.cpp converter 5. **Build Quantize Tool** - Use CMake to build `llama-quantize` 6. **Quantize** - Create Q4_K_M, Q5_K_M, Q8_0 versions 7. **Upload** - Upload all versions + README to Hub ## Quantization Options Common quantization formats (from smallest to largest): | Format | Size | Quality | Use Case | |--------|------|---------|----------| | **Q4_K_M** | ~300MB | Good | **Recommended** - best balance of size/quality | | **Q5_K_M** | ~350MB | Better | Higher quality, slightly larger | | **Q8_0** | ~500MB | Very High | Near-original quality | | **F16** | ~1GB | Original | Full precision, largest file | **Recommendation:** Create Q4_K_M, Q5_K_M, and Q8_0 versions to give users options. ## Hardware Requirements **For conversion:** - Small models (<1B): CPU-basic works, but slow - Medium models (1-7B): a10g-large recommended - Large models (7B+): a10g-large or a100-large **Time estimates:** - 0.5B model: ~15-25 minutes on A10G - 3B model: ~30-45 minutes on A10G - 7B model: ~45-60 minutes on A10G ## Using GGUF Models **GGUF models work on both CPU and GPU.** They're optimized for CPU inference but can also leverage GPU acceleration when available. ### With Ollama (auto-detects GPU) ```bash # Download GGUF hf download username/my-model-gguf model-q4_k_m.gguf # Create Modelfile echo "FROM ./model-q4_k_m.gguf" > Modelfile # Create and run (uses GPU automatically if available) ollama create my-model -f Modelfile ollama run my-model ``` ### With llama.cpp ```bash # CPU only ./llama-cli -m model-q4_k_m.gguf -p "Your prompt" # With GPU acceleration (offload 32 layers to GPU) ./llama-cli -m model-q4_k_m.gguf -ngl 32 -p "Your prompt" ``` ### With LM Studio 1. Download the `.gguf` file 2. Import into LM Studio 3. Start chatting ## Best Practices ### ✅ DO: 1. **Verify repos exist** before submitting jobs (use `hub_repo_details`) 2. **Install build tools FIRST** before cloning llama.cpp 3. **Use CMake** for building quantize tool (not make) 4. **Include all dependencies** in PEP 723 header (especially sentencepiece, protobuf) 5. **Create multiple quantizations** - Give users choice 6. **Test on known models** before production use 7. **Use A10G GPU** for faster conversion ### ❌ DON'T: 1. **Assume repos exist** - Always verify with hub tools 2. **Use make** instead of CMake - Less reliable 3. **Remove dependencies** to "simplify" - They're all needed 4. **Skip build tools** - Quantization will fail silently 5. **Use default paths** - CMake puts binaries in build/bin/ ## Common Issues ### Out of memory during merge **Fix:** - Use larger GPU (a10g-large or a100-large) - Ensure `device_map="auto"` for automatic placement - Use `dtype=torch.float16` or `torch.bfloat16` ### Conversion fails with architecture error **Fix:** - Ensure llama.cpp supports the model architecture - Check for standard architecture (Qwen, Llama, Mistral, etc.) - Update llama.cpp to latest: `git clone --depth 1 https://github.com/ggerganov/llama.cpp.git` - Check llama.cpp documentation for model support ### Quantization fails **Fix:** - Verify build tools installed: `apt-get install build-essential cmake` - Use CMake (not make) to build quantize tool - Check binary path: `/tmp/llama.cpp/build/bin/llama-quantize` - Verify FP16 GGUF exists before quantizing ### Missing sentencepiece error **Fix:** - Add to PEP 723 header: `"sentencepiece>=0.1.99", "protobuf>=3.20.0"` - Don't remove dependencies to "simplify" - all are required ### Upload fails or times out **Fix:** - Large models (>2GB) need longer timeout: `"timeout": "1h"` - Upload quantized versions separately if needed - Check network/Hub status ## Lessons Learned These are from production testing and real failures: ### 1. Always Verify Before Use **Lesson:** Don't assume repos/datasets exist. Check first. ```python # BEFORE submitting job hub_repo_details(["trl-lib/argilla-dpo-mix-7k"], repo_type="dataset") # Would catch error ``` **Prevented failures:** Non-existent dataset names, typos in model names ### 2. Prioritize Reliability Over Performance **Lesson:** Default to what's most likely to succeed. - Use CMake (not make) - more reliable - Disable CUDA in build - faster, not needed - Include all dependencies - don't "simplify" **Prevented failures:** Build failures, missing binaries ### 3. Create Atomic, Self-Contained Scripts **Lesson:** Don't remove dependencies or steps. Scripts should work as a unit. - All dependencies in PEP 723 header - All build steps included - Clear error messages **Prevented failures:** Missing tokenizer libraries, build tool failures ## References **In this skill:** - `scripts/convert_to_gguf.py` - Complete, production-ready script **External:** - [llama.cpp Repository](https://github.com/ggerganov/llama.cpp) - [GGUF Specification](https://github.com/ggerganov/ggml/blob/master/docs/gguf.md) - [Ollama Documentation](https://ollama.ai) - [LM Studio](https://lmstudio.ai) ## Summary **Critical checklist for GGUF conversion:** - [ ] Verify adapter and base models exist on Hub - [ ] Use production script from `scripts/convert_to_gguf.py` - [ ] All dependencies in PEP 723 header (including sentencepiece, protobuf) - [ ] Build tools installed before cloning llama.cpp - [ ] CMake used for building quantize tool (not make) - [ ] Correct binary path: `/tmp/llama.cpp/build/bin/llama-quantize` - [ ] A10G GPU selected for reasonable conversion time - [ ] Timeout set to 45m minimum - [ ] HF_TOKEN in secrets for Hub upload **The script in `scripts/convert_to_gguf.py` incorporates all these lessons and has been tested successfully in production.** ================================================ FILE: skills/hugging-face-model-trainer/references/hardware_guide.md ================================================ # Hardware Selection Guide Choosing the right hardware (flavor) is critical for cost-effective training. ## Available Hardware ### CPU - `cpu-basic` - Basic CPU, testing only - `cpu-upgrade` - Enhanced CPU **Use cases:** Dataset validation, preprocessing, testing scripts **Not recommended for training:** Too slow for any meaningful training ### GPU Options | Flavor | GPU | Memory | Use Case | Cost/hour | |--------|-----|--------|----------|-----------| | `t4-small` | NVIDIA T4 | 16GB | <1B models, demos | ~$0.50-1 | | `t4-medium` | NVIDIA T4 | 16GB | 1-3B models, development | ~$1-2 | | `l4x1` | NVIDIA L4 | 24GB | 3-7B models, efficient training | ~$2-3 | | `l4x4` | 4x NVIDIA L4 | 96GB | Multi-GPU training | ~$8-12 | | `a10g-small` | NVIDIA A10G | 24GB | 3-7B models, production | ~$3-4 | | `a10g-large` | NVIDIA A10G | 24GB | 7-13B models | ~$4-6 | | `a10g-largex2` | 2x NVIDIA A10G | 48GB | Multi-GPU, large models | ~$8-12 | | `a10g-largex4` | 4x NVIDIA A10G | 96GB | Multi-GPU, very large models | ~$16-24 | | `a100-large` | NVIDIA A100 | 40GB | 13B+ models, fast training | ~$8-12 | ### TPU Options | Flavor | Type | Use Case | |--------|------|----------| | `v5e-1x1` | TPU v5e | Small TPU workloads | | `v5e-2x2` | 4x TPU v5e | Medium TPU workloads | | `v5e-2x4` | 8x TPU v5e | Large TPU workloads | **Note:** TPUs require TPU-optimized code. Most TRL training uses GPUs. ## Selection Guidelines ### By Model Size **Tiny Models (<1B parameters)** - **Recommended:** `t4-small` - **Example:** Qwen2.5-0.5B, TinyLlama - **Batch size:** 4-8 - **Training time:** 1-2 hours for 1K examples **Small Models (1-3B parameters)** - **Recommended:** `t4-medium` or `a10g-small` - **Example:** Qwen2.5-1.5B, Phi-2 - **Batch size:** 2-4 - **Training time:** 2-4 hours for 10K examples **Medium Models (3-7B parameters)** - **Recommended:** `a10g-small` or `a10g-large` - **Example:** Qwen2.5-7B, Mistral-7B - **Batch size:** 1-2 (or LoRA with 4-8) - **Training time:** 4-8 hours for 10K examples **Large Models (7-13B parameters)** - **Recommended:** `a10g-large` or `a100-large` - **Example:** Llama-3-8B, Mixtral-8x7B (with LoRA) - **Batch size:** 1 (full fine-tuning) or 2-4 (LoRA) - **Training time:** 6-12 hours for 10K examples - **Note:** Always use LoRA/PEFT **Very Large Models (13B+ parameters)** - **Recommended:** `a100-large` with LoRA - **Example:** Llama-3-13B, Llama-3-70B (LoRA only) - **Batch size:** 1-2 with LoRA - **Training time:** 8-24 hours for 10K examples - **Note:** Full fine-tuning not feasible, use LoRA/PEFT ### By Budget **Minimal Budget (<$5 total)** - Use `t4-small` - Train on subset of data (100-500 examples) - Limit to 1-2 epochs - Use small model (<1B) **Small Budget ($5-20)** - Use `t4-medium` or `a10g-small` - Train on 1K-5K examples - 2-3 epochs - Model up to 3B parameters **Medium Budget ($20-50)** - Use `a10g-small` or `a10g-large` - Train on 5K-20K examples - 3-5 epochs - Model up to 7B parameters **Large Budget ($50-200)** - Use `a10g-large` or `a100-large` - Full dataset training - Multiple epochs - Model up to 13B parameters with LoRA ### By Training Type **Quick Demo/Experiment** - `t4-small` - 50-100 examples - 5-10 steps - ~10-15 minutes **Development/Iteration** - `t4-medium` or `a10g-small` - 1K examples - 1 epoch - ~30-60 minutes **Production Training** - `a10g-large` or `a100-large` - Full dataset - 3-5 epochs - 4-12 hours **Research/Experimentation** - `a100-large` - Multiple runs - Various hyperparameters - Budget for 20-50 hours ## Memory Considerations ### Estimating Memory Requirements **Full fine-tuning:** ``` Memory (GB) ≈ (Model params in billions) × 20 ``` **LoRA fine-tuning:** ``` Memory (GB) ≈ (Model params in billions) × 4 ``` **Examples:** - Qwen2.5-0.5B full: ~10GB ✅ fits t4-small - Qwen2.5-1.5B full: ~30GB ❌ exceeds most GPUs - Qwen2.5-1.5B LoRA: ~6GB ✅ fits t4-small - Qwen2.5-7B full: ~140GB ❌ not feasible - Qwen2.5-7B LoRA: ~28GB ✅ fits a10g-large ### Memory Optimization If hitting memory limits: 1. **Use LoRA/PEFT** ```python peft_config=LoraConfig(r=16, lora_alpha=32) ``` 2. **Reduce batch size** ```python per_device_train_batch_size=1 ``` 3. **Increase gradient accumulation** ```python gradient_accumulation_steps=8 # Effective batch size = 1×8 ``` 4. **Enable gradient checkpointing** ```python gradient_checkpointing=True ``` 5. **Use mixed precision** ```python bf16=True # or fp16=True ``` 6. **Upgrade to larger GPU** - t4 → a10g → a100 ## Cost Estimation ### Formula ``` Total Cost = (Hours of training) × (Cost per hour) ``` ### Example Calculations **Quick demo:** - Hardware: t4-small ($0.75/hour) - Time: 15 minutes (0.25 hours) - Cost: $0.19 **Development training:** - Hardware: a10g-small ($3.50/hour) - Time: 2 hours - Cost: $7.00 **Production training:** - Hardware: a10g-large ($5/hour) - Time: 6 hours - Cost: $30.00 **Large model with LoRA:** - Hardware: a100-large ($10/hour) - Time: 8 hours - Cost: $80.00 ### Cost Optimization Tips 1. **Start small:** Test on t4-small with subset 2. **Use LoRA:** 4-5x cheaper than full fine-tuning 3. **Optimize hyperparameters:** Fewer epochs if possible 4. **Set appropriate timeout:** Don't waste compute on stalled jobs 5. **Use checkpointing:** Resume if job fails 6. **Monitor costs:** Check running jobs regularly ## Multi-GPU Training TRL automatically handles multi-GPU training with Accelerate when using multi-GPU flavors. **Multi-GPU flavors:** - `l4x4` - 4x L4 GPUs - `a10g-largex2` - 2x A10G GPUs - `a10g-largex4` - 4x A10G GPUs **When to use:** - Models >13B parameters - Need faster training (linear speedup) - Large datasets (>50K examples) **Example:** ```python hf_jobs("uv", { "script": "train.py", "flavor": "a10g-largex2", # 2 GPUs "timeout": "4h", "secrets": {"HF_TOKEN": "$HF_TOKEN"} }) ``` No code changes needed—TRL/Accelerate handles distribution automatically. ## Choosing Between Options ### a10g vs a100 **Choose a10g when:** - Model <13B parameters - Budget conscious - Training time not critical **Choose a100 when:** - Model 13B+ parameters - Need fastest training - Memory requirements high - Budget allows ### Single vs Multi-GPU **Choose single GPU when:** - Model <7B parameters - Budget constrained - Simpler debugging **Choose multi-GPU when:** - Model >13B parameters - Need faster training - Large batch sizes required - Cost-effective for large jobs ## Quick Reference ```python # Model size → Hardware selection HARDWARE_MAP = { "<1B": "t4-small", "1-3B": "a10g-small", "3-7B": "a10g-large", "7-13B": "a10g-large (LoRA) or a100-large", ">13B": "a100-large (LoRA required)" } ``` ================================================ FILE: skills/hugging-face-model-trainer/references/hub_saving.md ================================================ # Saving Training Results to Hugging Face Hub **⚠️ CRITICAL:** Training environments are ephemeral. ALL results are lost when a job completes unless pushed to the Hub. ## Why Hub Push is Required When running on Hugging Face Jobs: - Environment is temporary - All files deleted on job completion - No local disk persistence - Cannot access results after job ends **Without Hub push, training is completely wasted.** ## Required Configuration ### 1. Training Configuration In your SFTConfig or trainer config: ```python SFTConfig( push_to_hub=True, # Enable Hub push hub_model_id="username/model-name", # Target repository ) ``` ### 2. Job Configuration When submitting the job: ```python hf_jobs("uv", { "script": "train.py", "secrets": {"HF_TOKEN": "$HF_TOKEN"} # Provide authentication }) ``` **The `$HF_TOKEN` placeholder is automatically replaced with your Hugging Face token.** ## Complete Example ```python # train.py # /// script # dependencies = ["trl"] # /// from trl import SFTTrainer, SFTConfig from datasets import load_dataset dataset = load_dataset("trl-lib/Capybara", split="train") # Configure with Hub push config = SFTConfig( output_dir="my-model", num_train_epochs=3, # ✅ CRITICAL: Hub push configuration push_to_hub=True, hub_model_id="myusername/my-trained-model", # Optional: Push strategy push_to_hub_model_id="myusername/my-trained-model", push_to_hub_organization=None, push_to_hub_token=None, # Uses environment token ) trainer = SFTTrainer( model="Qwen/Qwen2.5-0.5B", train_dataset=dataset, args=config, ) trainer.train() # ✅ Push final model trainer.push_to_hub() print("✅ Model saved to: https://huggingface.co/myusername/my-trained-model") ``` **Submit with authentication:** ```python hf_jobs("uv", { "script": "train.py", "flavor": "a10g-large", "timeout": "2h", "secrets": {"HF_TOKEN": "$HF_TOKEN"} # ✅ Required! }) ``` ## What Gets Saved When `push_to_hub=True`: 1. **Model weights** - Final trained parameters 2. **Tokenizer** - Associated tokenizer 3. **Configuration** - Model config (config.json) 4. **Training arguments** - Hyperparameters used 5. **Model card** - Auto-generated documentation 6. **Checkpoints** - If `save_strategy="steps"` enabled ## Checkpoint Saving Save intermediate checkpoints during training: ```python SFTConfig( output_dir="my-model", push_to_hub=True, hub_model_id="username/my-model", # Checkpoint configuration save_strategy="steps", save_steps=100, # Save every 100 steps save_total_limit=3, # Keep only last 3 checkpoints ) ``` **Benefits:** - Resume training if job fails - Compare checkpoint performance - Use intermediate models **Checkpoints are pushed to:** `username/my-model` (same repo) ## Authentication Methods ### Method 1: Automatic Token (Recommended) ```python "secrets": {"HF_TOKEN": "$HF_TOKEN"} ``` Uses your logged-in Hugging Face token automatically. ### Method 2: Explicit Token ```python "secrets": {"HF_TOKEN": "hf_abc123..."} ``` Provide token explicitly (not recommended for security). ### Method 3: Environment Variable ```python "env": {"HF_TOKEN": "hf_abc123..."} ``` Pass as regular environment variable (less secure than secrets). **Always prefer Method 1** for security and convenience. ## Verification Checklist Before submitting any training job, verify: - [ ] `push_to_hub=True` in training config - [ ] `hub_model_id` is specified (format: `username/model-name`) - [ ] `secrets={"HF_TOKEN": "$HF_TOKEN"}` in job config - [ ] Repository name doesn't conflict with existing repos - [ ] You have write access to the target namespace ## Repository Setup ### Automatic Creation If repository doesn't exist, it's created automatically when first pushing. ### Manual Creation Create repository before training: ```python from huggingface_hub import HfApi api = HfApi() api.create_repo( repo_id="username/model-name", repo_type="model", private=False, # or True for private repo ) ``` ### Repository Naming **Valid names:** - `username/my-model` - `username/model-name` - `organization/model-name` **Invalid names:** - `model-name` (missing username) - `username/model name` (spaces not allowed) - `username/MODEL` (uppercase discouraged) ## Troubleshooting ### Error: 401 Unauthorized **Cause:** HF_TOKEN not provided or invalid **Solutions:** 1. Verify `secrets={"HF_TOKEN": "$HF_TOKEN"}` in job config 2. Check you're logged in: `hf auth whoami` 3. Re-login: `hf auth login` ### Error: 403 Forbidden **Cause:** No write access to repository **Solutions:** 1. Check repository namespace matches your username 2. Verify you're a member of organization (if using org namespace) 3. Check repository isn't private (if accessing org repo) ### Error: Repository not found **Cause:** Repository doesn't exist and auto-creation failed **Solutions:** 1. Manually create repository first 2. Check repository name format 3. Verify namespace exists ### Error: Push failed during training **Cause:** Network issues or Hub unavailable **Solutions:** 1. Training continues but final push fails 2. Checkpoints may be saved 3. Re-run push manually after job completes ### Issue: Model saved but not visible **Possible causes:** 1. Repository is private—check https://huggingface.co/username 2. Wrong namespace—verify `hub_model_id` matches login 3. Push still in progress—wait a few minutes ## Manual Push After Training If training completes but push fails, push manually: ```python from transformers import AutoModel, AutoTokenizer # Load from local checkpoint model = AutoModel.from_pretrained("./output_dir") tokenizer = AutoTokenizer.from_pretrained("./output_dir") # Push to Hub model.push_to_hub("username/model-name", token="hf_abc123...") tokenizer.push_to_hub("username/model-name", token="hf_abc123...") ``` **Note:** Only possible if job hasn't completed (files still exist). ## Best Practices 1. **Always enable `push_to_hub=True`** 2. **Use checkpoint saving** for long training runs 3. **Verify Hub push** in logs before job completes 4. **Set appropriate `save_total_limit`** to avoid excessive checkpoints 5. **Use descriptive repo names** (e.g., `qwen-capybara-sft` not `model1`) 6. **Add model card** with training details 7. **Tag models** with relevant tags (e.g., `text-generation`, `fine-tuned`) ## Monitoring Push Progress Check logs for push progress: ```python hf_jobs("logs", {"job_id": "your-job-id"}) ``` **Look for:** ``` Pushing model to username/model-name... Upload file pytorch_model.bin: 100% ✅ Model pushed successfully ``` ## Example: Full Production Setup ```python # production_train.py # /// script # dependencies = ["trl>=0.12.0", "peft>=0.7.0"] # /// from datasets import load_dataset from peft import LoraConfig from trl import SFTTrainer, SFTConfig import os # Verify token is available assert "HF_TOKEN" in os.environ, "HF_TOKEN not found in environment!" # Load dataset dataset = load_dataset("trl-lib/Capybara", split="train") print(f"✅ Dataset loaded: {len(dataset)} examples") # Configure with comprehensive Hub settings config = SFTConfig( output_dir="qwen-capybara-sft", # Hub configuration push_to_hub=True, hub_model_id="myusername/qwen-capybara-sft", hub_strategy="checkpoint", # Push checkpoints # Checkpoint configuration save_strategy="steps", save_steps=100, save_total_limit=3, # Training settings num_train_epochs=3, per_device_train_batch_size=4, # Logging logging_steps=10, logging_first_step=True, ) # Train with LoRA trainer = SFTTrainer( model="Qwen/Qwen2.5-0.5B", train_dataset=dataset, args=config, peft_config=LoraConfig(r=16, lora_alpha=32), ) print("🚀 Starting training...") trainer.train() print("💾 Pushing final model to Hub...") trainer.push_to_hub() print("✅ Training complete!") print(f"Model available at: https://huggingface.co/myusername/qwen-capybara-sft") ``` **Submit:** ```python hf_jobs("uv", { "script": "production_train.py", "flavor": "a10g-large", "timeout": "6h", "secrets": {"HF_TOKEN": "$HF_TOKEN"} }) ``` ## Key Takeaway **Without `push_to_hub=True` and `secrets={"HF_TOKEN": "$HF_TOKEN"}`, all training results are permanently lost.** Always verify both are configured before submitting any training job. ================================================ FILE: skills/hugging-face-model-trainer/references/local_training_macos.md ================================================ # Local Training on macOS (Apple Silicon) Run small LoRA fine-tuning jobs locally on Mac for smoke tests and quick iteration before submitting to HF Jobs. ## When to Use Local Mac vs HF Jobs | Local Mac | HF Jobs / Cloud GPU | |-----------|-------------------| | Model ≤3B, text-only | Model 7B+ | | LoRA/PEFT only | QLoRA 4-bit (CUDA/bitsandbytes) | | Short context (≤1024) | Long context / full fine-tuning | | Smoke tests, dataset validation | Production runs, VLMs | **Typical workflow:** local smoke test → HF Jobs with same config → export/quantize ([gguf_conversion.md](gguf_conversion.md)) ## Recommended Defaults | Setting | Value | Notes | |---------|-------|-------| | Model size | 0.5B–1.5B first run | Scale up after verifying | | Max seq length | 512–1024 | Lower = less memory | | Batch size | 1 | Scale via gradient accumulation | | Gradient accumulation | 8–16 | Effective batch = 8–16 | | LoRA rank (r) | 8–16 | alpha = 2×r | | Dtype | float32 | fp16 causes NaN on MPS; bf16 only on M1 Pro+ and M2/M3/M4 | ### Memory by hardware | Unified RAM | Max Model Size | |-------------|---------------| | 16 GB | ~0.5B–1.5B | | 32 GB | ~1.5B–3B | | 64 GB | ~3B (short context) | ## Setup ```bash xcode-select --install python3 -m venv .venv && source .venv/bin/activate pip install -U "torch>=2.2" "transformers>=4.40" "trl>=0.12" "peft>=0.10" \ datasets accelerate safetensors huggingface_hub ``` Verify MPS: ```bash python -c "import torch; print(torch.__version__, '| MPS:', torch.backends.mps.is_available())" ``` Optional — configure Accelerate for local Mac (no distributed, no mixed precision, MPS device): ```bash accelerate config ``` ## Training Script
train_lora_sft.py ```python import os from dataclasses import dataclass from typing import Optional import torch from datasets import load_dataset from transformers import AutoModelForCausalLM, AutoTokenizer, set_seed from peft import LoraConfig from trl import SFTTrainer, SFTConfig set_seed(42) @dataclass class Cfg: model_id: str = os.environ.get("MODEL_ID", "Qwen/Qwen2.5-0.5B-Instruct") dataset_id: str = os.environ.get("DATASET_ID", "HuggingFaceH4/ultrachat_200k") dataset_split: str = os.environ.get("DATASET_SPLIT", "train_sft[:500]") data_files: Optional[str] = os.environ.get("DATA_FILES", None) text_field: str = os.environ.get("TEXT_FIELD", "") messages_field: str = os.environ.get("MESSAGES_FIELD", "messages") out_dir: str = os.environ.get("OUT_DIR", "outputs/local-lora") max_seq_length: int = int(os.environ.get("MAX_SEQ_LENGTH", "512")) max_steps: int = int(os.environ.get("MAX_STEPS", "-1")) cfg = Cfg() device = "mps" if torch.backends.mps.is_available() else "cpu" tokenizer = AutoTokenizer.from_pretrained(cfg.model_id, use_fast=True) if tokenizer.pad_token is None: tokenizer.pad_token = tokenizer.eos_token tokenizer.padding_side = "right" model = AutoModelForCausalLM.from_pretrained(cfg.model_id, torch_dtype=torch.float32) model.to(device) model.config.use_cache = False if cfg.data_files: ds = load_dataset("json", data_files=cfg.data_files, split="train") else: ds = load_dataset(cfg.dataset_id, split=cfg.dataset_split) def format_example(ex): if cfg.text_field and isinstance(ex.get(cfg.text_field), str): ex["text"] = ex[cfg.text_field] return ex msgs = ex.get(cfg.messages_field) if isinstance(msgs, list): if hasattr(tokenizer, "apply_chat_template"): try: ex["text"] = tokenizer.apply_chat_template(msgs, tokenize=False, add_generation_prompt=False) return ex except Exception: pass ex["text"] = "\n".join([str(m) for m in msgs]) return ex ex["text"] = str(ex) return ex ds = ds.map(format_example) ds = ds.remove_columns([c for c in ds.column_names if c != "text"]) lora = LoraConfig(r=16, lora_alpha=32, lora_dropout=0.05, bias="none", task_type="CAUSAL_LM", target_modules=["q_proj", "k_proj", "v_proj", "o_proj"]) sft_kwargs = dict( output_dir=cfg.out_dir, per_device_train_batch_size=1, gradient_accumulation_steps=8, learning_rate=2e-4, logging_steps=10, save_steps=200, save_total_limit=2, gradient_checkpointing=True, report_to="none", fp16=False, bf16=False, max_seq_length=cfg.max_seq_length, dataset_text_field="text", ) if cfg.max_steps > 0: sft_kwargs["max_steps"] = cfg.max_steps else: sft_kwargs["num_train_epochs"] = 1 trainer = SFTTrainer(model=model, train_dataset=ds, peft_config=lora, args=SFTConfig(**sft_kwargs), processing_class=tokenizer) trainer.train() trainer.save_model(cfg.out_dir) print(f"✅ Saved to: {cfg.out_dir}") ```
### Run ```bash python train_lora_sft.py ``` **Env overrides:** ```bash MODEL_ID="Qwen/Qwen2.5-1.5B-Instruct" python train_lora_sft.py # different model MAX_STEPS=50 python train_lora_sft.py # quick 50-step test DATA_FILES="my_data.jsonl" python train_lora_sft.py # local JSONL file PYTORCH_ENABLE_MPS_FALLBACK=1 python train_lora_sft.py # MPS op fallback to CPU PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0 python train_lora_sft.py # disable MPS memory limit (use with caution) ``` **Local JSONL format** — chat messages or plain text: ```jsonl {"messages": [{"role": "user", "content": "Hello"}, {"role": "assistant", "content": "Hi!"}]} ``` ```jsonl {"text": "User: Hello\nAssistant: Hi!"} ``` For plain text: `DATA_FILES="file.jsonl" TEXT_FIELD="text" MESSAGES_FIELD="" python train_lora_sft.py` ### Verify Success - Loss decreases over steps - `outputs/local-lora/` contains `adapter_config.json` + `*.safetensors` ## Quick Evaluation
eval_generate.py ```python import os, torch from transformers import AutoTokenizer, AutoModelForCausalLM from peft import PeftModel BASE = os.environ.get("MODEL_ID", "Qwen/Qwen2.5-0.5B-Instruct") ADAPTER = os.environ.get("ADAPTER_DIR", "outputs/local-lora") device = "mps" if torch.backends.mps.is_available() else "cpu" tokenizer = AutoTokenizer.from_pretrained(BASE, use_fast=True) model = AutoModelForCausalLM.from_pretrained(BASE, torch_dtype=torch.float32) model.to(device) model = PeftModel.from_pretrained(model, ADAPTER) prompt = os.environ.get("PROMPT", "Explain gradient accumulation in 3 bullet points.") inputs = tokenizer(prompt, return_tensors="pt").to(model.device) with torch.no_grad(): out = model.generate(**inputs, max_new_tokens=120, do_sample=True, temperature=0.7, top_p=0.9) print(tokenizer.decode(out[0], skip_special_tokens=True)) ```
## Troubleshooting (macOS-Specific) For general training issues, see [troubleshooting.md](troubleshooting.md). | Problem | Fix | |---------|-----| | MPS unsupported op / crash | `PYTORCH_ENABLE_MPS_FALLBACK=1` | | OOM / system instability | Reduce `MAX_SEQ_LENGTH`, use smaller model, set `PYTORCH_MPS_HIGH_WATERMARK_RATIO=0.0` (caution) | | fp16 NaN / loss explosion | Keep `fp16=False` (default), lower learning rate | | LoRA "module not found" | Print `model.named_modules()` to find correct target names | | TRL TypeError on args | Check TRL version; script uses `SFTConfig` + `processing_class` (TRL ≥0.12) | | Intel Mac | No MPS — use HF Jobs instead | **Common LoRA target modules by architecture:** | Architecture | target_modules | |-------------|---------------| | Llama/Qwen/Mistral | `q_proj`, `k_proj`, `v_proj`, `o_proj` | | GPT-2/GPT-J | `c_attn`, `c_proj` | | BLOOM | `query_key_value`, `dense` | ## MLX Alternative [MLX](https://github.com/ml-explore/mlx) offers tighter Apple Silicon integration but has a smaller ecosystem and less mature training APIs. For this skill's workflow (local validation → HF Jobs), PyTorch + MPS is recommended for consistency. See [mlx-lm](https://github.com/ml-explore/mlx-lm) for MLX-based fine-tuning. ## See Also - [troubleshooting.md](troubleshooting.md) — General TRL troubleshooting - [hardware_guide.md](hardware_guide.md) — GPU selection for HF Jobs - [gguf_conversion.md](gguf_conversion.md) — Export for on-device inference - [training_methods.md](training_methods.md) — SFT, DPO, GRPO overview ================================================ FILE: skills/hugging-face-model-trainer/references/reliability_principles.md ================================================ # Reliability Principles for Training Jobs These principles are derived from real production failures and successful fixes. Following them prevents common failure modes and ensures reliable job execution. ## Principle 1: Always Verify Before Use **Rule:** Never assume repos, datasets, or resources exist. Verify with tools first. ### What It Prevents - **Non-existent datasets** - Jobs fail immediately when dataset doesn't exist - **Typos in names** - Simple mistakes like "argilla-dpo-mix-7k" vs "ultrafeedback_binarized" - **Incorrect paths** - Old or moved repos, renamed files - **Missing dependencies** - Undocumented requirements ### How to Apply **Before submitting ANY job:** ```python # Verify dataset exists dataset_search({"query": "dataset-name", "author": "author-name", "limit": 5}) hub_repo_details(["author/dataset-name"], repo_type="dataset") # Verify model exists hub_repo_details(["org/model-name"], repo_type="model") # Check script/file paths (for URL-based scripts) # Verify before using: https://github.com/user/repo/blob/main/script.py ``` **Examples that would have caught errors:** ```python # ❌ WRONG: Assumed dataset exists hf_jobs("uv", { "script": """...""", "env": {"DATASET": "trl-lib/argilla-dpo-mix-7k"} # Doesn't exist! }) # ✅ CORRECT: Verify first dataset_search({"query": "argilla dpo", "author": "trl-lib"}) # Would show: "trl-lib/ultrafeedback_binarized" is the correct name hub_repo_details(["trl-lib/ultrafeedback_binarized"], repo_type="dataset") # Confirms it exists before using ``` ### Implementation Checklist - [ ] Check dataset exists before training - [ ] Verify base model exists before fine-tuning - [ ] Confirm adapter model exists before GGUF conversion - [ ] Test script URLs are valid before submitting - [ ] Validate file paths in repositories - [ ] Check for recent updates/renames of resources **Time cost:** 5-10 seconds **Time saved:** Hours of failed job time + debugging --- ## Principle 2: Prioritize Reliability Over Performance **Rule:** Default to what is most likely to succeed, not what is theoretically fastest. ### What It Prevents - **Hardware incompatibilities** - Features that fail on certain GPUs - **Unstable optimizations** - Speed-ups that cause crashes - **Complex configurations** - More failure points - **Build system issues** - Unreliable compilation methods ### How to Apply **Choose reliability:** ```python # ❌ RISKY: Aggressive optimization that may fail SFTConfig( torch_compile=True, # Can fail on T4, A10G GPUs optim="adamw_bnb_8bit", # Requires specific setup fp16=False, # May cause training instability ... ) # ✅ SAFE: Proven defaults SFTConfig( # torch_compile=True, # Commented with note: "Enable on H100 for 20% speedup" optim="adamw_torch", # Standard, always works fp16=True, # Stable and fast ... ) ``` **For build processes:** ```python # ❌ UNRELIABLE: Uses make (platform-dependent) subprocess.run(["make", "-C", "/tmp/llama.cpp", "llama-quantize"], check=True) # ✅ RELIABLE: Uses CMake (consistent, documented) subprocess.run([ "cmake", "-B", "/tmp/llama.cpp/build", "-S", "/tmp/llama.cpp", "-DGGML_CUDA=OFF" # Disable CUDA for faster, more reliable build ], check=True) subprocess.run([ "cmake", "--build", "/tmp/llama.cpp/build", "--target", "llama-quantize", "-j", "4" ], check=True) ``` ### Real-World Example **The `torch.compile` failure:** - Added for "20% speedup" on H100 - **Failed fatally on T4-medium** with cryptic error - Misdiagnosed as dataset issue (cost hours) - **Fix:** Disable by default, add as optional comment **Result:** Reliability > 20% performance gain ### Implementation Checklist - [ ] Use proven, standard configurations by default - [ ] Comment out performance optimizations with hardware notes - [ ] Use stable build systems (CMake > make) - [ ] Test on target hardware before production - [ ] Document known incompatibilities - [ ] Provide "safe" and "fast" variants when needed **Performance loss:** 10-20% in best case **Reliability gain:** 95%+ success rate vs 60-70% --- ## Principle 3: Create Atomic, Self-Contained Scripts **Rule:** Scripts should work as complete, independent units. Don't remove parts to "simplify." ### What It Prevents - **Missing dependencies** - Removed "unnecessary" packages that are actually required - **Incomplete processes** - Skipped steps that seem redundant - **Environment assumptions** - Scripts that need pre-setup - **Partial failures** - Some parts work, others fail silently ### How to Apply **Complete dependency specifications:** ```python # ❌ INCOMPLETE: "Simplified" by removing dependencies # /// script # dependencies = [ # "transformers", # "peft", # "torch", # ] # /// # ✅ COMPLETE: All dependencies explicit # /// script # dependencies = [ # "transformers>=4.36.0", # "peft>=0.7.0", # "torch>=2.0.0", # "accelerate>=0.24.0", # "huggingface_hub>=0.20.0", # "sentencepiece>=0.1.99", # Required for tokenizers # "protobuf>=3.20.0", # Required for tokenizers # "numpy", # "gguf", # ] # /// ``` **Complete build processes:** ```python # ❌ INCOMPLETE: Assumes build tools exist subprocess.run(["git", "clone", "https://github.com/ggerganov/llama.cpp.git", "/tmp/llama.cpp"]) subprocess.run(["make", "-C", "/tmp/llama.cpp", "llama-quantize"]) # FAILS: no gcc/make # ✅ COMPLETE: Installs all requirements subprocess.run(["apt-get", "update", "-qq"], check=True) subprocess.run(["apt-get", "install", "-y", "-qq", "build-essential", "cmake"], check=True) subprocess.run(["git", "clone", "https://github.com/ggerganov/llama.cpp.git", "/tmp/llama.cpp"]) # ... then build ``` ### Real-World Example **The `sentencepiece` failure:** - Original script had it: worked fine - "Simplified" version removed it: "doesn't look necessary" - **GGUF conversion failed silently** - tokenizer couldn't convert - Hard to debug: no obvious error message - **Fix:** Restore all original dependencies **Result:** Don't remove dependencies without thorough testing ### Implementation Checklist - [ ] All dependencies in PEP 723 header with version pins - [ ] All system packages installed by script - [ ] No assumptions about pre-existing environment - [ ] No "optional" steps that are actually required - [ ] Test scripts in clean environment - [ ] Document why each dependency is needed **Complexity:** Slightly longer scripts **Reliability:** Scripts "just work" every time --- ## Principle 4: Provide Clear Error Context **Rule:** When things fail, make it obvious what went wrong and how to fix it. ### How to Apply **Wrap subprocess calls:** ```python # ❌ UNCLEAR: Silent failure subprocess.run([...], check=True, capture_output=True) # ✅ CLEAR: Shows what failed try: result = subprocess.run( [...], check=True, capture_output=True, text=True ) print(result.stdout) if result.stderr: print("Warnings:", result.stderr) except subprocess.CalledProcessError as e: print(f"❌ Command failed!") print("STDOUT:", e.stdout) print("STDERR:", e.stderr) raise ``` **Validate inputs:** ```python # ❌ UNCLEAR: Fails later with cryptic error model = load_model(MODEL_NAME) # ✅ CLEAR: Fails fast with clear message if not MODEL_NAME: raise ValueError("MODEL_NAME environment variable not set!") print(f"Loading model: {MODEL_NAME}") try: model = load_model(MODEL_NAME) print(f"✅ Model loaded successfully") except Exception as e: print(f"❌ Failed to load model: {MODEL_NAME}") print(f"Error: {e}") print("Hint: Check that model exists on Hub") raise ``` ### Implementation Checklist - [ ] Wrap external calls with try/except - [ ] Print stdout/stderr on failure - [ ] Validate environment variables early - [ ] Add progress indicators (✅, ❌, 🔄) - [ ] Include hints for common failures - [ ] Log configuration at start --- ## Principle 5: Test the Happy Path on Known-Good Inputs **Rule:** Before using new code in production, test with inputs you know work. ### How to Apply **Known-good test inputs:** ```python # For training TEST_DATASET = "trl-lib/Capybara" # Small, well-formatted, widely used TEST_MODEL = "Qwen/Qwen2.5-0.5B" # Small, fast, reliable # For GGUF conversion TEST_ADAPTER = "evalstate/qwen-capybara-medium" # Known working model TEST_BASE = "Qwen/Qwen2.5-0.5B" # Compatible base ``` **Testing workflow:** 1. Test with known-good inputs first 2. If that works, try production inputs 3. If production fails, you know it's the inputs (not code) 4. Isolate the difference ### Implementation Checklist - [ ] Maintain list of known-good test models/datasets - [ ] Test new scripts with test inputs first - [ ] Document what makes inputs "good" - [ ] Keep test jobs cheap (small models, short timeouts) - [ ] Only move to production after test succeeds **Time cost:** 5-10 minutes for test run **Debugging time saved:** Hours --- ## Summary: The Reliability Checklist Before submitting ANY job: ### Pre-Flight Checks - [ ] **Verified** all repos/datasets exist (hub_repo_details) - [ ] **Tested** with known-good inputs if new code - [ ] **Using** proven hardware/configuration - [ ] **Included** all dependencies in PEP 723 header - [ ] **Installed** system requirements (build tools, etc.) - [ ] **Set** appropriate timeout (not default 30m) - [ ] **Configured** Hub push with HF_TOKEN - [ ] **Added** clear error handling ### Script Quality - [ ] Self-contained (no external setup needed) - [ ] Complete dependencies listed - [ ] Build tools installed by script - [ ] Progress indicators included - [ ] Error messages are clear - [ ] Configuration logged at start ### Job Configuration - [ ] Timeout > expected runtime + 30% buffer - [ ] Hardware appropriate for model size - [ ] Secrets include HF_TOKEN - [ ] Environment variables set correctly - [ ] Cost estimated and acceptable **Following these principles transforms job success rate from ~60-70% to ~95%+** --- ## When Principles Conflict Sometimes reliability and performance conflict. Here's how to choose: | Scenario | Choose | Rationale | |----------|--------|-----------| | Demo/test | Reliability | Fast failure is worse than slow success | | Production (first run) | Reliability | Prove it works before optimizing | | Production (proven) | Performance | Safe to optimize after validation | | Time-critical | Reliability | Failures cause more delay than slow runs | | Cost-critical | Balanced | Test with small model, then optimize | **General rule:** Reliability first, optimize second. --- ## Further Reading - `troubleshooting.md` - Common issues and fixes - `training_patterns.md` - Proven training configurations - `gguf_conversion.md` - Production GGUF workflow ================================================ FILE: skills/hugging-face-model-trainer/references/trackio_guide.md ================================================ # Trackio Integration for TRL Training **Trackio** is an experiment tracking library that provides real-time metrics visualization for remote training on Hugging Face Jobs infrastructure. ⚠️ **IMPORTANT**: For Jobs training (remote cloud GPUs): - Training happens on ephemeral cloud runners (not your local machine) - Trackio syncs metrics to a Hugging Face Space for real-time monitoring - Without a Space, metrics are lost when the job completes - The Space dashboard persists your training metrics permanently ## Setting Up Trackio for Jobs **Step 1: Add trackio dependency** ```python # /// script # dependencies = [ # "trl>=0.12.0", # "trackio", # Required! # ] # /// ``` **Step 2: Create a Trackio Space (one-time setup)** **Option A: Let Trackio auto-create (Recommended)** Pass a `space_id` to `trackio.init()` and Trackio will automatically create the Space if it doesn't exist. **Option B: Create manually** - Create Space via Hub UI at https://huggingface.co/new-space - Select Gradio SDK - OR use command: `hf repos create my-trackio-dashboard --type space --space-sdk gradio` **Step 3: Initialize Trackio with space_id** ```python import trackio trackio.init( project="my-training", space_id="username/trackio", # CRITICAL for Jobs! Replace 'username' with your HF username config={ "model": "Qwen/Qwen2.5-0.5B", "dataset": "trl-lib/Capybara", "learning_rate": 2e-5, } ) ``` **Step 4: Configure TRL to use Trackio** ```python SFTConfig( report_to="trackio", # ... other config ) ``` **Step 5: Finish tracking** ```python trainer.train() trackio.finish() # Ensures final metrics are synced ``` ## What Trackio Tracks Trackio automatically logs: - ✅ Training loss - ✅ Learning rate - ✅ GPU utilization - ✅ Memory usage - ✅ Training throughput - ✅ Custom metrics ## How It Works with Jobs 1. **Training runs** → Metrics logged to local SQLite DB 2. **Every 5 minutes** → Trackio syncs DB to HF Dataset (Parquet) 3. **Space dashboard** → Reads from Dataset, displays metrics in real-time 4. **Job completes** → Final sync ensures all metrics persisted ## Default Configuration Pattern **Use sensible defaults for trackio configuration unless user requests otherwise.** ### Recommended Defaults ```python import trackio trackio.init( project="qwen-capybara-sft", name="baseline-run", # Descriptive name user will recognize space_id="username/trackio", # Default space: {username}/trackio config={ # Keep config minimal - hyperparameters and model/dataset info only "model": "Qwen/Qwen2.5-0.5B", "dataset": "trl-lib/Capybara", "learning_rate": 2e-5, "num_epochs": 3, } ) ``` **Key principles:** - **Space ID**: Use `{username}/trackio` with "trackio" as default space name - **Run naming**: Unless otherwise specified, name the run in a way the user will recognize - **Config**: Keep minimal - don't automatically capture job metadata unless requested - **Grouping**: Optional - only use if user requests organizing related experiments ## Grouping Runs (Optional) The `group` parameter helps organize related runs together in the dashboard sidebar. This is useful when user is running multiple experiments with different configurations but wants to compare them together: ```python # Example: Group runs by experiment type trackio.init(project="my-project", run_name="baseline-run-1", group="baseline") trackio.init(project="my-project", run_name="augmented-run-1", group="augmented") trackio.init(project="my-project", run_name="tuned-run-1", group="tuned") ``` Runs with the same group name can be grouped together in the sidebar, making it easier to compare related experiments. You can group by any configuration parameter: ```python # Hyperparameter sweep - group by learning rate trackio.init(project="hyperparam-sweep", run_name="lr-0.001-run", group="lr_0.001") trackio.init(project="hyperparam-sweep", run_name="lr-0.01-run", group="lr_0.01") ``` ## Environment Variables for Jobs You can configure trackio using environment variables instead of passing parameters to `trackio.init()`. This is useful for managing configuration across multiple jobs. **`HF_TOKEN`** Required for creating Spaces and writing to datasets (passed via `secrets`): ```python hf_jobs("uv", { "script": "...", "secrets": { "HF_TOKEN": "$HF_TOKEN" # Enables Space creation and Hub push } }) ``` ### Example with Environment Variables ```python hf_jobs("uv", { "script": """ # Training script - trackio config from environment import trackio from datetime import datetime # Auto-generate run name timestamp = datetime.now().strftime("%Y-%m-%d_%H-%M") run_name = f"sft_qwen25_{timestamp}" # Project and space_id can come from environment variables trackio.init(run_name=run_name, group="SFT") # ... training code ... trackio.finish() """, "flavor": "a10g-large", "timeout": "2h", "secrets": {"HF_TOKEN": "$HF_TOKEN"} }) ``` **When to use environment variables:** - Managing multiple jobs with same configuration - Keeping training scripts portable across projects - Separating configuration from code **When to use direct parameters:** - Single job with specific configuration - When clarity in code is preferred - When each job has different project/space ## Viewing the Dashboard After starting training: 1. Navigate to the Space: `https://huggingface.co/spaces/username/trackio` 2. The Gradio dashboard shows all tracked experiments 3. Filter by project, compare runs, view charts with smoothing ## Recommendation - **Trackio**: Best for real-time monitoring during long training runs - **Weights & Biases**: Best for team collaboration, requires account ================================================ FILE: skills/hugging-face-model-trainer/references/training_methods.md ================================================ # TRL Training Methods Overview TRL (Transformer Reinforcement Learning) provides multiple training methods for fine-tuning and aligning language models. This reference provides a brief overview of each method. ## Supervised Fine-Tuning (SFT) **What it is:** Standard instruction tuning with supervised learning on demonstration data. **When to use:** - Initial fine-tuning of base models on task-specific data - Teaching new capabilities or domains - Most common starting point for fine-tuning **Dataset format:** Conversational format with "messages" field, OR text field, OR prompt/completion pairs **Example:** ```python from trl import SFTTrainer, SFTConfig trainer = SFTTrainer( model="Qwen/Qwen2.5-0.5B", train_dataset=dataset, args=SFTConfig( output_dir="my-model", push_to_hub=True, hub_model_id="username/my-model", eval_strategy="no", # Disable eval for simple example # max_length=1024 is the default - only set if you need different length ) ) trainer.train() ``` **Note:** For production training with evaluation monitoring, see `scripts/train_sft_example.py` **Documentation:** `hf_doc_fetch("https://huggingface.co/docs/trl/sft_trainer")` ## Direct Preference Optimization (DPO) **What it is:** Alignment method that trains directly on preference pairs (chosen vs rejected responses) without requiring a reward model. **When to use:** - Aligning models to human preferences - Improving response quality after SFT - Have paired preference data (chosen/rejected responses) **Dataset format:** Preference pairs with "chosen" and "rejected" fields **Example:** ```python from trl import DPOTrainer, DPOConfig trainer = DPOTrainer( model="Qwen/Qwen2.5-0.5B-Instruct", # Use instruct model train_dataset=dataset, args=DPOConfig( output_dir="dpo-model", beta=0.1, # KL penalty coefficient eval_strategy="no", # Disable eval for simple example # max_length=1024 is the default - only set if you need different length ) ) trainer.train() ``` **Note:** For production training with evaluation monitoring, see `scripts/train_dpo_example.py` **Documentation:** `hf_doc_fetch("https://huggingface.co/docs/trl/dpo_trainer")` ## Group Relative Policy Optimization (GRPO) **What it is:** Online RL method that optimizes relative to group performance, useful for tasks with verifiable rewards. **When to use:** - Tasks with automatic reward signals (code execution, math verification) - Online learning scenarios - When DPO offline data is insufficient **Dataset format:** Prompt-only format (model generates responses, reward computed online) **Example:** ```python # Use TRL maintained script hf_jobs("uv", { "script": "https://raw.githubusercontent.com/huggingface/trl/main/examples/scripts/grpo.py", "script_args": [ "--model_name_or_path", "Qwen/Qwen2.5-0.5B-Instruct", "--dataset_name", "trl-lib/math_shepherd", "--output_dir", "grpo-model" ], "flavor": "a10g-large", "timeout": "4h", "secrets": {"HF_TOKEN": "$HF_TOKEN"} }) ``` **Documentation:** `hf_doc_fetch("https://huggingface.co/docs/trl/grpo_trainer")` ## Reward Modeling **What it is:** Train a reward model to score responses, used as a component in RLHF pipelines. **When to use:** - Building RLHF pipeline - Need automatic quality scoring - Creating reward signals for PPO training **Dataset format:** Preference pairs with "chosen" and "rejected" responses **Documentation:** `hf_doc_fetch("https://huggingface.co/docs/trl/reward_trainer")` ## Method Selection Guide | Method | Complexity | Data Required | Use Case | |--------|-----------|---------------|----------| | **SFT** | Low | Demonstrations | Initial fine-tuning | | **DPO** | Medium | Paired preferences | Post-SFT alignment | | **GRPO** | Medium | Prompts + reward fn | Online RL with automatic rewards | | **Reward** | Medium | Paired preferences | Building RLHF pipeline | ## Recommended Pipeline **For most use cases:** 1. **Start with SFT** - Fine-tune base model on task data 2. **Follow with DPO** - Align to preferences using paired data 3. **Optional: GGUF conversion** - Deploy for local inference **For advanced RL scenarios:** 1. **Start with SFT** - Fine-tune base model 2. **Train reward model** - On preference data ## Dataset Format Reference For complete dataset format specifications, use: ```python hf_doc_fetch("https://huggingface.co/docs/trl/dataset_formats") ``` Or validate your dataset: ```bash uv run https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py \ --dataset your/dataset --split train ``` ## See Also - `references/training_patterns.md` - Common training patterns and examples - `scripts/train_sft_example.py` - Complete SFT template - `scripts/train_dpo_example.py` - Complete DPO template - [Dataset Inspector](https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py) - Dataset format validation tool ================================================ FILE: skills/hugging-face-model-trainer/references/training_patterns.md ================================================ # Common Training Patterns This guide provides common training patterns and use cases for TRL on Hugging Face Jobs. ## Multi-GPU Training Automatic distributed training across multiple GPUs. TRL/Accelerate handles distribution automatically: ```python hf_jobs("uv", { "script": """ # Your training script here (same as single GPU) # No changes needed - Accelerate detects multiple GPUs """, "flavor": "a10g-largex2", # 2x A10G GPUs "timeout": "4h", "secrets": {"HF_TOKEN": "$HF_TOKEN"} }) ``` **Tips for multi-GPU:** - No code changes needed - Use `per_device_train_batch_size` (per GPU, not total) - Effective batch size = `per_device_train_batch_size` × `num_gpus` × `gradient_accumulation_steps` - Monitor GPU utilization to ensure both GPUs are being used ## DPO Training (Preference Learning) Train with preference data for alignment: ```python hf_jobs("uv", { "script": """ # /// script # dependencies = ["trl>=0.12.0", "trackio"] # /// from datasets import load_dataset from trl import DPOTrainer, DPOConfig import trackio dataset = load_dataset("trl-lib/ultrafeedback_binarized", split="train") # Create train/eval split dataset_split = dataset.train_test_split(test_size=0.1, seed=42) config = DPOConfig( output_dir="dpo-model", push_to_hub=True, hub_model_id="username/dpo-model", num_train_epochs=1, beta=0.1, # KL penalty coefficient eval_strategy="steps", eval_steps=50, report_to="trackio", run_name="baseline_run", # use a meaningful run name # max_length=1024, # Default - only set if you need different sequence length ) trainer = DPOTrainer( model="Qwen/Qwen2.5-0.5B-Instruct", # Use instruct model as base train_dataset=dataset_split["train"], eval_dataset=dataset_split["test"], # IMPORTANT: Provide eval_dataset when eval_strategy is enabled args=config, ) trainer.train() trainer.push_to_hub() trackio.finish() """, "flavor": "a10g-large", "timeout": "3h", "secrets": {"HF_TOKEN": "$HF_TOKEN"} }) ``` **For DPO documentation:** Use `hf_doc_fetch("https://huggingface.co/docs/trl/dpo_trainer")` ## GRPO Training (Online RL) Group Relative Policy Optimization for online reinforcement learning: ```python hf_jobs("uv", { "script": "https://raw.githubusercontent.com/huggingface/trl/main/examples/scripts/grpo.py", "script_args": [ "--model_name_or_path", "Qwen/Qwen2.5-0.5B-Instruct", "--dataset_name", "trl-lib/math_shepherd", "--output_dir", "grpo-model", "--push_to_hub", "--hub_model_id", "username/grpo-model" ], "flavor": "a10g-large", "timeout": "4h", "secrets": {"HF_TOKEN": "$HF_TOKEN"} }) ``` **For GRPO documentation:** Use `hf_doc_fetch("https://huggingface.co/docs/trl/grpo_trainer")` ## Trackio Configuration **Use sensible defaults for trackio setup.** See `references/trackio_guide.md` for complete documentation including grouping runs for experiments. ### Basic Pattern ```python import trackio trackio.init( project="my-training", run_name="baseline-run", # Descriptive name user will recognize space_id="username/trackio", # Default space: {username}/trackio config={ # Keep config minimal - hyperparameters and model/dataset info only "model": "Qwen/Qwen2.5-0.5B", "dataset": "trl-lib/Capybara", "learning_rate": 2e-5, } ) # Your training code... trackio.finish() ``` ### Grouping for Experiments (Optional) When user wants to compare related runs, use the `group` parameter: ```python # Hyperparameter sweep trackio.init(project="hyperparam-sweep", run_name="lr-0.001", group="lr_0.001") trackio.init(project="hyperparam-sweep", run_name="lr-0.01", group="lr_0.01") ``` ## Pattern Selection Guide | Use Case | Pattern | Hardware | Time | |----------|---------|----------|------| | SFT training | `scripts/train_sft_example.py` | a10g-large | 2-6 hours | | Large dataset (>10K) | Multi-GPU | a10g-largex2 | 4-12 hours | | Preference learning | DPO Training | a10g-large | 2-4 hours | | Online RL | GRPO Training | a10g-large | 3-6 hours | ## Critical: Evaluation Dataset Requirements **⚠️ IMPORTANT**: If you set `eval_strategy="steps"` or `eval_strategy="epoch"`, you **MUST** provide an `eval_dataset` to the trainer, or the training will hang. ### ✅ CORRECT - With eval dataset: ```python dataset_split = dataset.train_test_split(test_size=0.1, seed=42) trainer = SFTTrainer( model="Qwen/Qwen2.5-0.5B", train_dataset=dataset_split["train"], eval_dataset=dataset_split["test"], # ← MUST provide when eval_strategy is enabled args=SFTConfig(eval_strategy="steps", ...), ) ``` ### ❌ WRONG - Will hang: ```python trainer = SFTTrainer( model="Qwen/Qwen2.5-0.5B", train_dataset=dataset, # NO eval_dataset but eval_strategy="steps" ← WILL HANG args=SFTConfig(eval_strategy="steps", ...), ) ``` ### Option: Disable evaluation if no eval dataset ```python config = SFTConfig( eval_strategy="no", # ← Explicitly disable evaluation # ... other config ) trainer = SFTTrainer( model="Qwen/Qwen2.5-0.5B", train_dataset=dataset, # No eval_dataset needed args=config, ) ``` ## Best Practices 1. **Use train/eval splits** - Create evaluation split for monitoring progress 2. **Enable Trackio** - Monitor progress in real-time 3. **Add 20-30% buffer to timeout** - Account for loading/saving overhead 4. **Test with TRL official scripts first** - Use maintained examples before custom code 5. **Always provide eval_dataset** - When using eval_strategy, or set to "no" 6. **Use multi-GPU for large models** - 7B+ models benefit significantly ## See Also - `scripts/train_sft_example.py` - Complete SFT template with Trackio and eval split - `scripts/train_dpo_example.py` - Complete DPO template - `scripts/train_grpo_example.py` - Complete GRPO template - `references/hardware_guide.md` - Detailed hardware specifications - `references/training_methods.md` - Overview of all TRL training methods - `references/troubleshooting.md` - Common issues and solutions ================================================ FILE: skills/hugging-face-model-trainer/references/troubleshooting.md ================================================ # Troubleshooting TRL Training Jobs Common issues and solutions when training with TRL on Hugging Face Jobs. ## Training Hangs at "Starting training..." Step **Problem:** Job starts but hangs at the training step - never progresses, never times out, just sits there. **Root Cause:** Using `eval_strategy="steps"` or `eval_strategy="epoch"` without providing an `eval_dataset` to the trainer. **Solution:** **Option A: Provide eval_dataset (recommended)** ```python # Create train/eval split dataset_split = dataset.train_test_split(test_size=0.1, seed=42) trainer = SFTTrainer( model="Qwen/Qwen2.5-0.5B", train_dataset=dataset_split["train"], eval_dataset=dataset_split["test"], # ← MUST provide when eval_strategy is enabled args=SFTConfig( eval_strategy="steps", eval_steps=50, ... ), ) ``` **Option B: Disable evaluation** ```python trainer = SFTTrainer( model="Qwen/Qwen2.5-0.5B", train_dataset=dataset, # No eval_dataset args=SFTConfig( eval_strategy="no", # ← Explicitly disable ... ), ) ``` **Prevention:** - Always create train/eval split for better monitoring - Use `dataset.train_test_split(test_size=0.1, seed=42)` - Check example scripts: `scripts/train_sft_example.py` includes proper eval setup ## Job Times Out **Problem:** Job terminates before training completes, all progress lost. **Solutions:** - Increase timeout parameter (e.g., `"timeout": "4h"`) - Reduce `num_train_epochs` or use smaller dataset slice - Use smaller model or enable LoRA/PEFT to speed up training - Add 20-30% buffer to estimated time for loading/saving overhead **Prevention:** - Always start with a quick demo run to estimate timing - Use `scripts/estimate_cost.py` to get time estimates - Monitor first runs closely via Trackio or logs ## Model Not Saved to Hub **Problem:** Training completes but model doesn't appear on Hub - all work lost. **Check:** - [ ] `push_to_hub=True` in training config - [ ] `hub_model_id` specified with username (e.g., `"username/model-name"`) - [ ] `secrets={"HF_TOKEN": "$HF_TOKEN"}` in job submission - [ ] User has write access to target repo - [ ] Token has write permissions (check at https://huggingface.co/settings/tokens) - [ ] Training script calls `trainer.push_to_hub()` at the end **See:** `references/hub_saving.md` for detailed Hub authentication troubleshooting ## Out of Memory (OOM) **Problem:** Job fails with CUDA out of memory error. **Solutions (in order of preference):** 1. **Reduce batch size:** Lower `per_device_train_batch_size` (try 4 → 2 → 1) 2. **Increase gradient accumulation:** Raise `gradient_accumulation_steps` to maintain effective batch size 3. **Disable evaluation:** Remove `eval_dataset` and `eval_strategy` (saves ~40% memory, good for demos) 4. **Enable LoRA/PEFT:** Use `peft_config=LoraConfig(r=8, lora_alpha=16)` to train adapters only (smaller rank = less memory) 5. **Use larger GPU:** Switch from `t4-small` → `l4x1` → `a10g-large` → `a100-large` 6. **Enable gradient checkpointing:** Set `gradient_checkpointing=True` in config (slower but saves memory) 7. **Use smaller model:** Try a smaller variant (e.g., 0.5B instead of 3B) **Memory guidelines:** - T4 (16GB): <1B models with LoRA - A10G (24GB): 1-3B models with LoRA, <1B full fine-tune - A100 (40GB/80GB): 7B+ models with LoRA, 3B full fine-tune ## Parameter Naming Issues **Problem:** `TypeError: SFTConfig.__init__() got an unexpected keyword argument 'max_seq_length'` **Cause:** TRL config classes use `max_length`, not `max_seq_length`. **Solution:** ```python # ✅ CORRECT - TRL uses max_length SFTConfig(max_length=512) DPOConfig(max_length=512) # ❌ WRONG - This will fail SFTConfig(max_seq_length=512) ``` **Note:** Most TRL configs don't require explicit max_length - the default (1024) works well. Only set if you need a specific value. ## Dataset Format Error **Problem:** Training fails with dataset format errors or missing fields. **Solutions:** 1. **Check format documentation:** ```python hf_doc_fetch("https://huggingface.co/docs/trl/dataset_formats") ``` 2. **Validate dataset before training:** ```bash uv run https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py \ --dataset --split train ``` Or via hf_jobs: ```python hf_jobs("uv", { "script": "https://huggingface.co/datasets/mcp-tools/skills/raw/main/dataset_inspector.py", "script_args": ["--dataset", "dataset-name", "--split", "train"] }) ``` 3. **Verify field names:** - **SFT:** Needs "messages" field (conversational), OR "text" field, OR "prompt"/"completion" - **DPO:** Needs "chosen" and "rejected" fields - **GRPO:** Needs prompt-only format 4. **Check dataset split:** - Ensure split exists (e.g., `split="train"`) - Preview dataset: `load_dataset("name", split="train[:5]")` ## Import/Module Errors **Problem:** Job fails with "ModuleNotFoundError" or import errors. **Solutions:** 1. **Add PEP 723 header with dependencies:** ```python # /// script # dependencies = [ # "trl>=0.12.0", # "peft>=0.7.0", # "transformers>=4.36.0", # ] # /// ``` 2. **Verify exact format:** - Must have `# ///` delimiters (with space after `#`) - Dependencies must be valid PyPI package names - Check spelling and version constraints 3. **Test locally first:** ```bash uv run train.py # Tests if dependencies are correct ``` ## Authentication Errors **Problem:** Job fails with authentication or permission errors when pushing to Hub. **Solutions:** 1. **Verify authentication:** ```python mcp__huggingface__hf_whoami() # Check who's authenticated ``` 2. **Check token permissions:** - Go to https://huggingface.co/settings/tokens - Ensure token has "write" permission - Token must not be "read-only" 3. **Verify token in job:** ```python "secrets": {"HF_TOKEN": "$HF_TOKEN"} # Must be in job config ``` 4. **Check repo permissions:** - User must have write access to target repo - If org repo, user must be member with write access - Repo must exist or user must have permission to create ## Job Stuck or Not Starting **Problem:** Job shows "pending" or "starting" for extended period. **Solutions:** - Check Jobs dashboard for status: https://huggingface.co/jobs - Verify hardware availability (some GPU types may have queues) - Try different hardware flavor if one is heavily utilized - Check for account billing issues (Jobs requires paid plan) **Typical startup times:** - CPU jobs: 10-30 seconds - GPU jobs: 30-90 seconds - If >3 minutes: likely queued or stuck ## Training Loss Not Decreasing **Problem:** Training runs but loss stays flat or doesn't improve. **Solutions:** 1. **Check learning rate:** May be too low (try 2e-5 to 5e-5) or too high (try 1e-6) 2. **Verify dataset quality:** Inspect examples to ensure they're reasonable 3. **Check model size:** Very small models may not have capacity for task 4. **Increase training steps:** May need more epochs or larger dataset 5. **Verify dataset format:** Wrong format may cause degraded training ## Logs Not Appearing **Problem:** Cannot see training logs or progress. **Solutions:** 1. **Wait 30-60 seconds:** Initial logs can be delayed 2. **Check logs via MCP tool:** ```python hf_jobs("logs", {"job_id": "your-job-id"}) ``` 3. **Use Trackio for real-time monitoring:** See `references/trackio_guide.md` 4. **Verify job is actually running:** ```python hf_jobs("inspect", {"job_id": "your-job-id"}) ``` ## Checkpoint/Resume Issues **Problem:** Cannot resume from checkpoint or checkpoint not saved. **Solutions:** 1. **Enable checkpoint saving:** ```python SFTConfig( save_strategy="steps", save_steps=100, hub_strategy="every_save", # Push each checkpoint ) ``` 2. **Verify checkpoints pushed to Hub:** Check model repo for checkpoint folders 3. **Resume from checkpoint:** ```python trainer = SFTTrainer( model="username/model-name", # Can be checkpoint path resume_from_checkpoint="username/model-name/checkpoint-1000", ) ``` ## Getting Help If issues persist: 1. **Check TRL documentation:** ```python hf_doc_search("your issue", product="trl") ``` 2. **Check Jobs documentation:** ```python hf_doc_fetch("https://huggingface.co/docs/huggingface_hub/guides/jobs") ``` 3. **Review related guides:** - `references/hub_saving.md` - Hub authentication issues - `references/hardware_guide.md` - Hardware selection and specs - `references/training_patterns.md` - Eval dataset requirements - SKILL.md "Working with Scripts" section - Script format and URL issues 4. **Ask in HF forums:** https://discuss.huggingface.co/ ================================================ FILE: skills/hugging-face-model-trainer/references/unsloth.md ================================================ # Unsloth: Fast Fine-Tuning with Memory Optimization **Unsloth** is a fine-tuning library that provides ~2x faster training and ~60% less VRAM usage for LLM training. It's particularly useful when working with limited GPU memory or when speed is critical. - **GitHub**: [unslothai/unsloth](https://github.com/unslothai/unsloth) - **Docs**: [unsloth.ai/docs](https://unsloth.ai/docs) ## When to Use Unsloth Use Unsloth if instructed to do so, or one of the following use cases applies: | Use Case | Recommendation | |----------|----------------| | Standard text LLM fine-tuning | TRL is sufficient, but Unsloth is faster | | Limited GPU memory | **Use Unsloth** - 60% less VRAM | | Need maximum speed | **Use Unsloth** - 2x faster | | Large models (>13B) | **Use Unsloth** - memory efficiency critical | ## Supported Models Unsloth supports many popular models including: - **Text LLMs**: Llama 3/3.1/3.2/3.3, Qwen 2.5/3, Mistral, Phi-4, Gemma 2/3, LFM2/2.5 - **Vision LLMs**: Qwen3-VL, Gemma 3, Llama 3.2 Vision, Pixtral Use Unsloth's pre-optimized model variants when available: ```python # Unsloth-optimized models load faster and use less memory model_id = "unsloth/LFM2.5-1.2B-Instruct" # 4-bit quantized model_id = "unsloth/gemma-3-4b-pt" # Vision model model_id = "unsloth/Qwen3-VL-8B-Instruct" # Vision model ``` ## Installation ```python # /// script # dependencies = [ # "unsloth", # "trl", # "datasets", # "trackio", # ] # /// ``` ## Basic Usage: Text LLM ```python from unsloth import FastLanguageModel from trl import SFTTrainer, SFTConfig from datasets import load_dataset # Load model with Unsloth optimizations model, tokenizer = FastLanguageModel.from_pretrained( model_name="LiquidAI/LFM2.5-1.2B-Instruct", max_seq_length=4096, ) # Add LoRA adapters model = FastLanguageModel.get_peft_model( model, r=16, lora_alpha=16, target_modules=["q_proj", "k_proj", "v_proj", "out_proj", "in_proj", "w1", "w2", "w3"], lora_dropout=0, bias="none", use_gradient_checkpointing="unsloth", random_state=3407, ) # Load dataset dataset = load_dataset("trl-lib/Capybara", split="train") # Train with TRL trainer = SFTTrainer( model=model, tokenizer=tokenizer, train_dataset=dataset, args=SFTConfig( output_dir="./output", per_device_train_batch_size=2, gradient_accumulation_steps=4, max_steps=500, learning_rate=2e-4, report_to="trackio", ), ) trainer.train() ``` ## LFM2.5 Specific Settings For LFM2.5 inference, use these recommended generation parameters: **Instruct models:** ```python temperature = 0.1 top_k = 50 top_p = 0.1 repetition_penalty = 1.05 ``` **Thinking models:** ```python temperature = 0.05 top_k = 50 repetition_penalty = 1.05 ``` ## Vision-Language Models (VLMs) Unsloth provides specialized support for VLMs with `FastVisionModel`: ```python from unsloth import FastVisionModel, get_chat_template from unsloth.trainer import UnslothVisionDataCollator from trl import SFTTrainer, SFTConfig from datasets import load_dataset # Load VLM with Unsloth model, processor = FastVisionModel.from_pretrained( "unsloth/gemma-3-4b-pt", # or "unsloth/Qwen3-VL-8B-Instruct" load_in_4bit=True, use_gradient_checkpointing="unsloth", ) # Add LoRA for all modalities model = FastVisionModel.get_peft_model( model, finetune_vision_layers=True, # Train vision encoder finetune_language_layers=True, # Train language model finetune_attention_modules=True, # Train attention finetune_mlp_modules=True, # Train MLPs r=16, lora_alpha=32, target_modules="all-linear", ) # Apply chat template (required for base models) processor = get_chat_template(processor, "gemma-3") # Load VLM dataset (with images and messages) dataset = load_dataset("your-vlm-dataset", split="train", streaming=True) # Enable training mode FastVisionModel.for_training(model) # Train with VLM-specific collator trainer = SFTTrainer( model=model, train_dataset=dataset, processing_class=processor.tokenizer, data_collator=UnslothVisionDataCollator(model, processor), args=SFTConfig( output_dir="./vlm-output", per_device_train_batch_size=2, gradient_accumulation_steps=4, max_steps=500, learning_rate=2e-4, # VLM-specific settings remove_unused_columns=False, dataset_text_field="", dataset_kwargs={"skip_prepare_dataset": True}, report_to="trackio", ), ) trainer.train() ``` ## Key Differences from Standard TRL | Aspect | Standard TRL | Unsloth | |--------|--------------|---------| | Model loading | `AutoModelForCausalLM.from_pretrained()` | `FastLanguageModel.from_pretrained()` | | LoRA setup | `PeftModel` / `LoraConfig` | `FastLanguageModel.get_peft_model()` | | VLM loading | Limited support | `FastVisionModel.from_pretrained()` | | VLM collator | Manual | `UnslothVisionDataCollator` | | Memory usage | Standard | ~60% less | | Training speed | Standard | ~2x faster | ## VLM Dataset Format VLM datasets should have: - `images`: List of PIL images or image paths - `messages`: Conversation format with image references ```python { "images": [, ...], "messages": [ {"role": "user", "content": [ {"type": "image"}, {"type": "text", "text": "Describe this image"} ]}, {"role": "assistant", "content": "This image shows..."} ] } ``` ## Streaming Datasets For large VLM datasets, use streaming to avoid disk space issues: ```python dataset = load_dataset( "your-vlm-dataset", split="train", streaming=True, # Stream from Hub ) # Must use max_steps with streaming (no epoch-based training) SFTConfig(max_steps=500, ...) ``` ## Saving Models ### Save LoRA Adapter ```python model.save_pretrained("./adapter") processor.save_pretrained("./adapter") # Push to Hub model.push_to_hub("username/my-vlm-adapter") processor.push_to_hub("username/my-vlm-adapter") ``` ### Merge and Save Full Model ```python # Merge LoRA weights into base model model = model.merge_and_unload() # Save merged model model.save_pretrained("./merged") tokenizer.save_pretrained("./merged") ``` ### Convert to GGUF Unsloth models can be converted to GGUF for llama.cpp/Ollama: ```python # Save in 16-bit for GGUF conversion model.save_pretrained_gguf("./gguf", tokenizer, quantization_method="f16") # Or directly quantize model.save_pretrained_gguf("./gguf", tokenizer, quantization_method="q4_k_m") ``` ## Qwen3-VL Specific Settings For Qwen3-VL models, use these recommended settings: **Instruct models:** ```python temperature = 0.7 top_p = 0.8 presence_penalty = 1.5 ``` **Thinking models:** ```python temperature = 1.0 top_p = 0.95 presence_penalty = 0.0 ``` ## Hardware Requirements | Model | Min VRAM (Unsloth 4-bit) | Recommended GPU | |-------|--------------------------|-----------------| | 2B-4B | 8GB | T4, L4 | | 7B-8B | 16GB | A10G, L4x4 | | 13B | 24GB | A10G-large | | 30B+ | 48GB+ | A100 | ## Example: Full VLM Training Script See `scripts/unsloth_sft_example.py` for a complete production-ready example that includes: - Unsloth VLM setup - Streaming dataset support - Trackio monitoring - Hub push - CLI arguments Run locally: ```bash uv run scripts/unsloth_sft_example.py \ --dataset trl-lib/Capybara \ --max-steps 500 \ --output-repo username/my-model ``` Run on HF Jobs: ```python hf_jobs("uv", { "script": " ``` ## Core Concepts ### 1. Pipeline API The pipeline API is the easiest way to use models. It groups together preprocessing, model inference, and postprocessing: ```javascript import { pipeline } from '@huggingface/transformers'; // Create a pipeline for a specific task const pipe = await pipeline('sentiment-analysis'); // Use the pipeline const result = await pipe('I love transformers!'); // Output: [{ label: 'POSITIVE', score: 0.999817686 }] // IMPORTANT: Always dispose when done to free memory await classifier.dispose(); ``` **⚠️ Memory Management:** All pipelines must be disposed with `pipe.dispose()` when finished to prevent memory leaks. See examples in [Code Examples](./references/EXAMPLES.md) for cleanup patterns across different environments. ### 2. Model Selection You can specify a custom model as the second argument: ```javascript const pipe = await pipeline( 'sentiment-analysis', 'Xenova/bert-base-multilingual-uncased-sentiment' ); ``` **Finding Models:** Browse available Transformers.js models on Hugging Face Hub: - **All models**: https://huggingface.co/models?library=transformers.js&sort=trending - **By task**: Add `pipeline_tag` parameter - Text generation: https://huggingface.co/models?pipeline_tag=text-generation&library=transformers.js&sort=trending - Image classification: https://huggingface.co/models?pipeline_tag=image-classification&library=transformers.js&sort=trending - Speech recognition: https://huggingface.co/models?pipeline_tag=automatic-speech-recognition&library=transformers.js&sort=trending **Tip:** Filter by task type, sort by trending/downloads, and check model cards for performance metrics and usage examples. ### 3. Device Selection Choose where to run the model: ```javascript // Run on CPU (default for WASM) const pipe = await pipeline('sentiment-analysis', 'model-id'); // Run on GPU (WebGPU - experimental) const pipe = await pipeline('sentiment-analysis', 'model-id', { device: 'webgpu', }); ``` ### 4. Quantization Options Control model precision vs. performance: ```javascript // Use quantized model (faster, smaller) const pipe = await pipeline('sentiment-analysis', 'model-id', { dtype: 'q4', // Options: 'fp32', 'fp16', 'q8', 'q4' }); ``` ## Supported Tasks **Note:** All examples below show basic usage. ### Natural Language Processing #### Text Classification ```javascript const classifier = await pipeline('text-classification'); const result = await classifier('This movie was amazing!'); ``` #### Named Entity Recognition (NER) ```javascript const ner = await pipeline('token-classification'); const entities = await ner('My name is John and I live in New York.'); ``` #### Question Answering ```javascript const qa = await pipeline('question-answering'); const answer = await qa({ question: 'What is the capital of France?', context: 'Paris is the capital and largest city of France.' }); ``` #### Text Generation ```javascript const generator = await pipeline('text-generation', 'onnx-community/gemma-3-270m-it-ONNX'); const text = await generator('Once upon a time', { max_new_tokens: 100, temperature: 0.7 }); ``` **For streaming and chat:** See **[Text Generation Guide](./references/TEXT_GENERATION.md)** for: - Streaming token-by-token output with `TextStreamer` - Chat/conversation format with system/user/assistant roles - Generation parameters (temperature, top_k, top_p) - Browser and Node.js examples - React components and API endpoints #### Translation ```javascript const translator = await pipeline('translation', 'Xenova/nllb-200-distilled-600M'); const output = await translator('Hello, how are you?', { src_lang: 'eng_Latn', tgt_lang: 'fra_Latn' }); ``` #### Summarization ```javascript const summarizer = await pipeline('summarization'); const summary = await summarizer(longText, { max_length: 100, min_length: 30 }); ``` #### Zero-Shot Classification ```javascript const classifier = await pipeline('zero-shot-classification'); const result = await classifier('This is a story about sports.', ['politics', 'sports', 'technology']); ``` ### Computer Vision #### Image Classification ```javascript const classifier = await pipeline('image-classification'); const result = await classifier('https://example.com/image.jpg'); // Or with local file const result = await classifier(imageUrl); ``` #### Object Detection ```javascript const detector = await pipeline('object-detection'); const objects = await detector('https://example.com/image.jpg'); // Returns: [{ label: 'person', score: 0.95, box: { xmin, ymin, xmax, ymax } }, ...] ``` #### Image Segmentation ```javascript const segmenter = await pipeline('image-segmentation'); const segments = await segmenter('https://example.com/image.jpg'); ``` #### Depth Estimation ```javascript const depthEstimator = await pipeline('depth-estimation'); const depth = await depthEstimator('https://example.com/image.jpg'); ``` #### Zero-Shot Image Classification ```javascript const classifier = await pipeline('zero-shot-image-classification'); const result = await classifier('image.jpg', ['cat', 'dog', 'bird']); ``` ### Audio Processing #### Automatic Speech Recognition ```javascript const transcriber = await pipeline('automatic-speech-recognition'); const result = await transcriber('audio.wav'); // Returns: { text: 'transcribed text here' } ``` #### Audio Classification ```javascript const classifier = await pipeline('audio-classification'); const result = await classifier('audio.wav'); ``` #### Text-to-Speech ```javascript const synthesizer = await pipeline('text-to-speech', 'Xenova/speecht5_tts'); const audio = await synthesizer('Hello, this is a test.', { speaker_embeddings: speakerEmbeddings }); ``` ### Multimodal #### Image-to-Text (Image Captioning) ```javascript const captioner = await pipeline('image-to-text'); const caption = await captioner('image.jpg'); ``` #### Document Question Answering ```javascript const docQA = await pipeline('document-question-answering'); const answer = await docQA('document-image.jpg', 'What is the total amount?'); ``` #### Zero-Shot Object Detection ```javascript const detector = await pipeline('zero-shot-object-detection'); const objects = await detector('image.jpg', ['person', 'car', 'tree']); ``` ### Feature Extraction (Embeddings) ```javascript const extractor = await pipeline('feature-extraction'); const embeddings = await extractor('This is a sentence to embed.'); // Returns: tensor of shape [1, sequence_length, hidden_size] // For sentence embeddings (mean pooling) const extractor = await pipeline('feature-extraction', 'onnx-community/all-MiniLM-L6-v2-ONNX'); const embeddings = await extractor('Text to embed', { pooling: 'mean', normalize: true }); ``` ## Finding and Choosing Models ### Browsing the Hugging Face Hub Discover compatible Transformers.js models on Hugging Face Hub: **Base URL (all models):** ``` https://huggingface.co/models?library=transformers.js&sort=trending ``` **Filter by task** using the `pipeline_tag` parameter: | Task | URL | |------|-----| | **Text Generation** | https://huggingface.co/models?pipeline_tag=text-generation&library=transformers.js&sort=trending | | **Text Classification** | https://huggingface.co/models?pipeline_tag=text-classification&library=transformers.js&sort=trending | | **Translation** | https://huggingface.co/models?pipeline_tag=translation&library=transformers.js&sort=trending | | **Summarization** | https://huggingface.co/models?pipeline_tag=summarization&library=transformers.js&sort=trending | | **Question Answering** | https://huggingface.co/models?pipeline_tag=question-answering&library=transformers.js&sort=trending | | **Image Classification** | https://huggingface.co/models?pipeline_tag=image-classification&library=transformers.js&sort=trending | | **Object Detection** | https://huggingface.co/models?pipeline_tag=object-detection&library=transformers.js&sort=trending | | **Image Segmentation** | https://huggingface.co/models?pipeline_tag=image-segmentation&library=transformers.js&sort=trending | | **Speech Recognition** | https://huggingface.co/models?pipeline_tag=automatic-speech-recognition&library=transformers.js&sort=trending | | **Audio Classification** | https://huggingface.co/models?pipeline_tag=audio-classification&library=transformers.js&sort=trending | | **Image-to-Text** | https://huggingface.co/models?pipeline_tag=image-to-text&library=transformers.js&sort=trending | | **Feature Extraction** | https://huggingface.co/models?pipeline_tag=feature-extraction&library=transformers.js&sort=trending | | **Zero-Shot Classification** | https://huggingface.co/models?pipeline_tag=zero-shot-classification&library=transformers.js&sort=trending | **Sort options:** - `&sort=trending` - Most popular recently - `&sort=downloads` - Most downloaded overall - `&sort=likes` - Most liked by community - `&sort=modified` - Recently updated ### Choosing the Right Model Consider these factors when selecting a model: **1. Model Size** - **Small (< 100MB)**: Fast, suitable for browsers, limited accuracy - **Medium (100MB - 500MB)**: Balanced performance, good for most use cases - **Large (> 500MB)**: High accuracy, slower, better for Node.js or powerful devices **2. Quantization** Models are often available in different quantization levels: - `fp32` - Full precision (largest, most accurate) - `fp16` - Half precision (smaller, still accurate) - `q8` - 8-bit quantized (much smaller, slight accuracy loss) - `q4` - 4-bit quantized (smallest, noticeable accuracy loss) **3. Task Compatibility** Check the model card for: - Supported tasks (some models support multiple tasks) - Input/output formats - Language support (multilingual vs. English-only) - License restrictions **4. Performance Metrics** Model cards typically show: - Accuracy scores - Benchmark results - Inference speed - Memory requirements ### Example: Finding a Text Generation Model ```javascript // 1. Visit: https://huggingface.co/models?pipeline_tag=text-generation&library=transformers.js&sort=trending // 2. Browse and select a model (e.g., onnx-community/gemma-3-270m-it-ONNX) // 3. Check model card for: // - Model size: ~270M parameters // - Quantization: q4 available // - Language: English // - Use case: Instruction-following chat // 4. Use the model: import { pipeline } from '@huggingface/transformers'; const generator = await pipeline( 'text-generation', 'onnx-community/gemma-3-270m-it-ONNX', { dtype: 'q4' } // Use quantized version for faster inference ); const output = await generator('Explain quantum computing in simple terms.', { max_new_tokens: 100 }); await generator.dispose(); ``` ### Tips for Model Selection 1. **Start Small**: Test with a smaller model first, then upgrade if needed 2. **Check ONNX Support**: Ensure the model has ONNX files (look for `onnx` folder in model repo) 3. **Read Model Cards**: Model cards contain usage examples, limitations, and benchmarks 4. **Test Locally**: Benchmark inference speed and memory usage in your environment 5. **Community Models**: Look for models by `Xenova` (Transformers.js maintainer) or `onnx-community` 6. **Version Pin**: Use specific git commits in production for stability: ```javascript const pipe = await pipeline('task', 'model-id', { revision: 'abc123' }); ``` ## Advanced Configuration ### Environment Configuration (`env`) The `env` object provides comprehensive control over Transformers.js execution, caching, and model loading. **Quick Overview:** ```javascript import { env } from '@huggingface/transformers'; // View version console.log(env.version); // e.g., '3.8.1' // Common settings env.allowRemoteModels = true; // Load from Hugging Face Hub env.allowLocalModels = false; // Load from file system env.localModelPath = '/models/'; // Local model directory env.useFSCache = true; // Cache models on disk (Node.js) env.useBrowserCache = true; // Cache models in browser env.cacheDir = './.cache'; // Cache directory location ``` **Configuration Patterns:** ```javascript // Development: Fast iteration with remote models env.allowRemoteModels = true; env.useFSCache = true; // Production: Local models only env.allowRemoteModels = false; env.allowLocalModels = true; env.localModelPath = '/app/models/'; // Custom CDN env.remoteHost = 'https://cdn.example.com/models'; // Disable caching (testing) env.useFSCache = false; env.useBrowserCache = false; ``` For complete documentation on all configuration options, caching strategies, cache management, pre-downloading models, and more, see: **→ [Configuration Reference](./references/CONFIGURATION.md)** ### Working with Tensors ```javascript import { AutoTokenizer, AutoModel } from '@huggingface/transformers'; // Load tokenizer and model separately for more control const tokenizer = await AutoTokenizer.from_pretrained('bert-base-uncased'); const model = await AutoModel.from_pretrained('bert-base-uncased'); // Tokenize input const inputs = await tokenizer('Hello world!'); // Run model const outputs = await model(inputs); ``` ### Batch Processing ```javascript const classifier = await pipeline('sentiment-analysis'); // Process multiple texts const results = await classifier([ 'I love this!', 'This is terrible.', 'It was okay.' ]); ``` ## Browser-Specific Considerations ### WebGPU Usage WebGPU provides GPU acceleration in browsers: ```javascript const pipe = await pipeline('text-generation', 'onnx-community/gemma-3-270m-it-ONNX', { device: 'webgpu', dtype: 'fp32' }); ``` **Note**: WebGPU is experimental. Check browser compatibility and file issues if problems occur. ### WASM Performance Default browser execution uses WASM: ```javascript // Optimized for browsers with quantization const pipe = await pipeline('sentiment-analysis', 'model-id', { dtype: 'q8' // or 'q4' for even smaller size }); ``` ### Progress Tracking & Loading Indicators Models can be large (ranging from a few MB to several GB) and consist of multiple files. Track download progress by passing a callback to the `pipeline()` function: ```javascript import { pipeline } from '@huggingface/transformers'; // Track progress for each file const fileProgress = {}; function onProgress(info) { console.log(`${info.status}: ${info.file}`); if (info.status === 'progress') { fileProgress[info.file] = info.progress; console.log(`${info.file}: ${info.progress.toFixed(1)}%`); } if (info.status === 'done') { console.log(`✓ ${info.file} complete`); } } // Pass callback to pipeline const classifier = await pipeline('sentiment-analysis', null, { progress_callback: onProgress }); ``` **Progress Info Properties:** ```typescript interface ProgressInfo { status: 'initiate' | 'download' | 'progress' | 'done' | 'ready'; name: string; // Model id or path file: string; // File being processed progress?: number; // Percentage (0-100, only for 'progress' status) loaded?: number; // Bytes downloaded (only for 'progress' status) total?: number; // Total bytes (only for 'progress' status) } ``` For complete examples including browser UIs, React components, CLI progress bars, and retry logic, see: **→ [Pipeline Options - Progress Callback](./references/PIPELINE_OPTIONS.md#progress-callback)** ## Error Handling ```javascript try { const pipe = await pipeline('sentiment-analysis', 'model-id'); const result = await pipe('text to analyze'); } catch (error) { if (error.message.includes('fetch')) { console.error('Model download failed. Check internet connection.'); } else if (error.message.includes('ONNX')) { console.error('Model execution failed. Check model compatibility.'); } else { console.error('Unknown error:', error); } } ``` ## Performance Tips 1. **Reuse Pipelines**: Create pipeline once, reuse for multiple inferences 2. **Use Quantization**: Start with `q8` or `q4` for faster inference 3. **Batch Processing**: Process multiple inputs together when possible 4. **Cache Models**: Models are cached automatically (see **[Caching Reference](./references/CACHE.md)** for details on browser Cache API, Node.js filesystem cache, and custom implementations) 5. **WebGPU for Large Models**: Use WebGPU for models that benefit from GPU acceleration 6. **Prune Context**: For text generation, limit `max_new_tokens` to avoid memory issues 7. **Clean Up Resources**: Call `pipe.dispose()` when done to free memory ## Memory Management **IMPORTANT:** Always call `pipe.dispose()` when finished to prevent memory leaks. ```javascript const pipe = await pipeline('sentiment-analysis'); const result = await pipe('Great product!'); await pipe.dispose(); // ✓ Free memory (100MB - several GB per model) ``` **When to dispose:** - Application shutdown or component unmount - Before loading a different model - After batch processing in long-running apps Models consume significant memory and hold GPU/CPU resources. Disposal is critical for browser memory limits and server stability. For detailed patterns (React cleanup, servers, browser), see **[Code Examples](./references/EXAMPLES.md)** ## Troubleshooting ### Model Not Found - Verify model exists on Hugging Face Hub - Check model name spelling - Ensure model has ONNX files (look for `onnx` folder in model repo) ### Memory Issues - Use smaller models or quantized versions (`dtype: 'q4'`) - Reduce batch size - Limit sequence length with `max_length` ### WebGPU Errors - Check browser compatibility (Chrome 113+, Edge 113+) - Try `dtype: 'fp16'` if `fp32` fails - Fall back to WASM if WebGPU unavailable ## Reference Documentation ### This Skill - **[Pipeline Options](./references/PIPELINE_OPTIONS.md)** - Configure `pipeline()` with `progress_callback`, `device`, `dtype`, etc. - **[Configuration Reference](./references/CONFIGURATION.md)** - Global `env` configuration for caching and model loading - **[Caching Reference](./references/CACHE.md)** - Browser Cache API, Node.js filesystem cache, and custom cache implementations - **[Text Generation Guide](./references/TEXT_GENERATION.md)** - Streaming, chat format, and generation parameters - **[Model Architectures](./references/MODEL_ARCHITECTURES.md)** - Supported models and selection tips - **[Code Examples](./references/EXAMPLES.md)** - Real-world implementations for different runtimes ### Official Transformers.js - Official docs: https://huggingface.co/docs/transformers.js - API reference: https://huggingface.co/docs/transformers.js/api/pipelines - Model hub: https://huggingface.co/models?library=transformers.js - GitHub: https://github.com/huggingface/transformers.js - Examples: https://github.com/huggingface/transformers.js/tree/main/examples ## Best Practices 1. **Always Dispose Pipelines**: Call `pipe.dispose()` when done - critical for preventing memory leaks 2. **Start with Pipelines**: Use the pipeline API unless you need fine-grained control 3. **Test Locally First**: Test models with small inputs before deploying 4. **Monitor Model Sizes**: Be aware of model download sizes for web applications 5. **Handle Loading States**: Show progress indicators for better UX 6. **Version Pin**: Pin specific model versions for production stability 7. **Error Boundaries**: Always wrap pipeline calls in try-catch blocks 8. **Progressive Enhancement**: Provide fallbacks for unsupported browsers 9. **Reuse Models**: Load once, use many times - don't recreate pipelines unnecessarily 10. **Graceful Shutdown**: Dispose models on SIGTERM/SIGINT in servers ## Quick Reference: Task IDs | Task | Task ID | |------|---------| | Text classification | `text-classification` or `sentiment-analysis` | | Token classification | `token-classification` or `ner` | | Question answering | `question-answering` | | Fill mask | `fill-mask` | | Summarization | `summarization` | | Translation | `translation` | | Text generation | `text-generation` | | Text-to-text generation | `text2text-generation` | | Zero-shot classification | `zero-shot-classification` | | Image classification | `image-classification` | | Image segmentation | `image-segmentation` | | Object detection | `object-detection` | | Depth estimation | `depth-estimation` | | Image-to-image | `image-to-image` | | Zero-shot image classification | `zero-shot-image-classification` | | Zero-shot object detection | `zero-shot-object-detection` | | Automatic speech recognition | `automatic-speech-recognition` | | Audio classification | `audio-classification` | | Text-to-speech | `text-to-speech` or `text-to-audio` | | Image-to-text | `image-to-text` | | Document question answering | `document-question-answering` | | Feature extraction | `feature-extraction` | | Sentence similarity | `sentence-similarity` | --- This skill enables you to integrate state-of-the-art machine learning capabilities directly into JavaScript applications without requiring separate ML servers or Python environments. ================================================ FILE: skills/transformers.js/references/CACHE.md ================================================ # Caching Reference Complete guide to caching strategies for Transformers.js models across different environments. ## Table of Contents 1. [Overview](#overview) 2. [Browser Caching](#browser-caching) 3. [Node.js Caching](#nodejs-caching) 4. [Custom Cache Implementation](#custom-cache-implementation) 5. [Cache Configuration](#cache-configuration) ## Overview Transformers.js models can be large (from a few MB to several GB), so caching is critical for performance. The caching strategy differs based on the environment: - **Browser**: Uses the Cache API (browser cache storage) - **Node.js**: Uses filesystem cache in `~/.cache/huggingface/` - **Custom**: Implement your own cache (database, cloud storage, etc.) ### Default Behavior ```javascript import { pipeline } from '@huggingface/transformers'; // First load: downloads model const pipe = await pipeline('sentiment-analysis'); // Subsequent loads: uses cached model const pipe2 = await pipeline('sentiment-analysis'); // Fast! ``` Caching is **automatic** and enabled by default. Models are cached after the first download. ## Browser Caching ### Using the Cache API In browser environments, Transformers.js uses the [Cache API](https://developer.mozilla.org/en-US/docs/Web/API/Cache) to store models: ```javascript import { env, pipeline } from '@huggingface/transformers'; // Browser cache is enabled by default console.log(env.useBrowserCache); // true // Load model (cached automatically) const classifier = await pipeline('sentiment-analysis'); ``` **How it works:** 1. Model files are downloaded from Hugging Face Hub 2. Files are stored in the browser's Cache Storage 3. Subsequent loads retrieve from cache (no network request) 4. Cache persists across page reloads and browser sessions ### Cache Location Browser caches are stored in: - **Chrome/Edge**: `Cache Storage` in DevTools → Application tab → Cache storage - **Firefox**: `about:cache` → Storage - **Safari**: Web Inspector → Storage tab ### Disable Browser Cache ```javascript import { env } from '@huggingface/transformers'; // Disable browser caching (not recommended) env.useBrowserCache = false; // Models will be re-downloaded on every page load ``` **Use case:** Testing, development, or debugging cache issues. ### Browser Storage Limits Browsers impose storage quotas: - **Chrome**: ~60% of available disk space (but can evict data) - **Firefox**: ~50% of available disk space - **Safari**: ~1GB per origin (prompt for more) **Tip:** Monitor storage usage with the [Storage API](https://developer.mozilla.org/en-US/docs/Web/API/Storage_API): ```javascript if ('storage' in navigator && 'estimate' in navigator.storage) { const estimate = await navigator.storage.estimate(); const percentUsed = (estimate.usage / estimate.quota) * 100; console.log(`Storage: ${percentUsed.toFixed(2)}% used`); console.log(`Available: ${((estimate.quota - estimate.usage) / 1024 / 1024).toFixed(2)} MB`); } ``` ## Node.js Caching ### Filesystem Cache In Node.js, models are cached to the filesystem: ```javascript import { env, pipeline } from '@huggingface/transformers'; // Default cache directory (Node.js) console.log(env.cacheDir); // './.cache' (relative to current directory) // Filesystem cache is enabled by default console.log(env.useFSCache); // true // Load model (cached to disk) const classifier = await pipeline('sentiment-analysis'); ``` ### Default Cache Location **Default behavior:** - Cache directory: `./.cache` (relative to where Node.js process runs) - Full default path: `~/.cache/huggingface/` when using Hugging Face tools **Note:** The statement "Models are cached automatically in `~/.cache/huggingface/`" from performance tips is specific to Hugging Face's Python tooling convention. In Transformers.js for Node.js, the default is `./.cache` unless configured otherwise. ### Custom Cache Directory ```javascript import { env, pipeline } from '@huggingface/transformers'; // Set custom cache directory env.cacheDir = '/var/cache/transformers'; // Or use environment variable (Node.js convention) env.cacheDir = process.env.HF_HOME || '~/.cache/huggingface'; // Now load model const classifier = await pipeline('sentiment-analysis'); // Cached to: /var/cache/transformers/models--Xenova--distilbert-base-uncased-finetuned-sst-2-english/ ``` **Pattern:** `models--{organization}--{model-name}/` ### Disable Filesystem Cache ```javascript import { env } from '@huggingface/transformers'; // Disable filesystem caching (not recommended) env.useFSCache = false; // Models will be re-downloaded on every load ``` **Use case:** Testing, CI/CD environments, or containers with ephemeral storage. ## Custom Cache Implementation Implement your own cache for specialized storage backends. ### Custom Cache Interface ```typescript interface CacheInterface { /** * Check if a URL is cached */ match(url: string): Promise; /** * Store a URL and its response */ put(url: string, response: Response): Promise; } ``` ### Example: Cloud Storage Cache (S3) ```javascript import { env, pipeline } from '@huggingface/transformers'; import { S3Client, GetObjectCommand, PutObjectCommand } from '@aws-sdk/client-s3'; import { Readable } from 'stream'; class S3Cache { constructor(bucket, region = 'us-east-1') { this.bucket = bucket; this.s3 = new S3Client({ region }); } async match(url) { const key = this.urlToKey(url); try { const command = new GetObjectCommand({ Bucket: this.bucket, Key: key }); const response = await this.s3.send(command); // Convert stream to buffer const chunks = []; for await (const chunk of response.Body) { chunks.push(chunk); } const body = Buffer.concat(chunks); return new Response(body, { status: 200, headers: JSON.parse(response.Metadata.headers || '{}') }); } catch (error) { if (error.name === 'NoSuchKey') return undefined; throw error; } } async put(url, response) { const key = this.urlToKey(url); const clonedResponse = response.clone(); const body = Buffer.from(await clonedResponse.arrayBuffer()); const headers = JSON.stringify(Object.fromEntries(response.headers.entries())); const command = new PutObjectCommand({ Bucket: this.bucket, Key: key, Body: body, Metadata: { headers } }); await this.s3.send(command); } urlToKey(url) { // Convert URL to S3 key (remove protocol, replace slashes) return url.replace(/^https?:\/\//, '').replace(/\//g, '_'); } } // Configure S3 cache env.useCustomCache = true; env.customCache = new S3Cache('my-transformers-cache', 'us-east-1'); env.useFSCache = false; // Use S3 cache const classifier = await pipeline('sentiment-analysis'); ``` ## Cache Configuration ### Environment Variables Use environment variables to configure caching: ```javascript import { env } from '@huggingface/transformers'; // Configure cache directory from environment env.cacheDir = process.env.TRANSFORMERS_CACHE || './.cache'; // Disable caching in CI/CD if (process.env.CI === 'true') { env.useFSCache = false; env.useBrowserCache = false; } // Production: use pre-cached models if (process.env.NODE_ENV === 'production') { env.allowRemoteModels = false; env.allowLocalModels = true; env.localModelPath = process.env.MODEL_PATH || '/app/models'; } ``` ### Configuration Patterns #### Development: Enable All Caching ```javascript import { env } from '@huggingface/transformers'; env.allowRemoteModels = true; env.useFSCache = true; // Node.js env.useBrowserCache = true; // Browser env.cacheDir = './.cache'; ``` #### Production: Local Models Only ```javascript import { env } from '@huggingface/transformers'; env.allowRemoteModels = false; env.allowLocalModels = true; env.localModelPath = '/app/models'; env.useFSCache = true; ``` #### Testing: Disable Caching ```javascript import { env } from '@huggingface/transformers'; env.useFSCache = false; env.useBrowserCache = false; env.allowRemoteModels = true; // Download every time ``` #### Hybrid: Cache + Remote Fallback ```javascript import { env } from '@huggingface/transformers'; // Try local cache first, fall back to remote env.allowRemoteModels = true; env.allowLocalModels = true; env.useFSCache = true; env.localModelPath = './models'; ``` --- ## Summary Transformers.js provides flexible caching options: - **Browser**: Cache API (automatic, persistent) - **Node.js**: Filesystem cache (default `./.cache`, configurable) - **Custom**: Implement your own (database, cloud storage, etc.) **Key takeaways:** 1. Caching is enabled by default and automatic 2. Configure cache **before** loading models 3. Browser uses Cache API, Node.js uses filesystem 4. Custom caches enable advanced storage backends 5. Monitor cache size and implement cleanup strategies 6. Pre-download models for production deployments For more configuration options, see: - [Configuration Reference](./CONFIGURATION.md) - [Pipeline Options](./PIPELINE_OPTIONS.md) ================================================ FILE: skills/transformers.js/references/CONFIGURATION.md ================================================ # Environment Configuration Reference Complete guide to configuring Transformers.js behavior using the `env` object. ## Table of Contents 1. [Overview](#overview) 2. [Remote Model Configuration](#remote-model-configuration) 3. [Local Model Configuration](#local-model-configuration) 4. [Cache Configuration](#cache-configuration) 5. [WASM Configuration](#wasm-configuration) 6. [Common Configuration Patterns](#common-configuration-patterns) 7. [Environment Best Practices](#environment-best-practices) ## Overview The `env` object provides comprehensive control over Transformers.js execution, caching, and model loading: ```javascript import { env } from '@huggingface/transformers'; // View current version console.log(env.version); // e.g., '3.8.1' ``` ### Available Properties ```typescript interface TransformersEnvironment { // Version info version: string; // Backend configuration backends: { onnx: Partial; }; // Remote model settings allowRemoteModels: boolean; remoteHost: string; remotePathTemplate: string; // Local model settings allowLocalModels: boolean; localModelPath: string; useFS: boolean; // Cache settings useBrowserCache: boolean; useFSCache: boolean; cacheDir: string | null; useCustomCache: boolean; customCache: CacheInterface | null; useWasmCache: boolean; cacheKey: string; } ``` ## Remote Model Configuration Control how models are loaded from remote sources (default: Hugging Face Hub). ### Disable Remote Loading ```javascript import { env } from '@huggingface/transformers'; // Force local-only mode (no network requests) env.allowRemoteModels = false; ``` **Use case:** Offline applications, security requirements, or air-gapped environments. ### Custom Model Host ```javascript import { env } from '@huggingface/transformers'; // Use your own CDN or model server env.remoteHost = 'https://cdn.example.com/models'; // Customize the URL pattern // Default: '{model}/resolve/{revision}/{file}' env.remotePathTemplate = 'custom/{model}/{file}'; ``` **Use case:** Self-hosting models, using a CDN for faster downloads, or corporate proxies. ### Example: Private Model Server ```javascript import { env, pipeline } from '@huggingface/transformers'; // Configure custom model host env.remoteHost = 'https://models.mycompany.com'; env.remotePathTemplate = '{model}/{file}'; // Models will be loaded from: // https://models.mycompany.com/my-model/model.onnx const pipe = await pipeline('sentiment-analysis', 'my-model'); ``` ## Local Model Configuration Control loading models from the local file system. ### Enable Local Models ```javascript import { env } from '@huggingface/transformers'; // Enable local file system loading env.allowLocalModels = true; // Set the base path for local models env.localModelPath = '/path/to/models/'; ``` **Default values:** - Browser: `allowLocalModels = false`, `localModelPath = '/models/'` - Node.js: `allowLocalModels = true`, `localModelPath = '/models/'` ### File System Control ```javascript import { env } from '@huggingface/transformers'; // Disable file system usage entirely (Node.js only) env.useFS = false; ``` ### Example: Local Model Directory Structure ``` /app/models/ ├── onnx-community/ │ ├── Supertonic-TTS-ONNX/ │ │ ├── config.json │ │ ├── tokenizer.json │ │ ├── model.onnx │ │ └── ... │ └── yolo26l-pose-ONNX/ │ ├── config.json │ ├── preprocessor_config.json │ ├── model.onnx │ └── ... ``` ```javascript env.allowLocalModels = true; env.localModelPath = '/app/models/'; env.allowRemoteModels = false; // Offline mode const classifier = await pipeline('sentiment-analysis', 'Xenova/distilbert-base-uncased-finetuned-sst-2-english'); ``` ## Cache Configuration Transformers.js supports multiple caching strategies to improve performance and reduce network usage. ### Quick Configuration ```javascript import { env } from '@huggingface/transformers'; // Browser cache (Cache API) env.useBrowserCache = true; // default: true env.cacheKey = 'my-app-transformers-cache'; // default: 'transformers-cache' // Node.js filesystem cache env.useFSCache = true; // default: true env.cacheDir = './custom-cache-dir'; // default: './.cache' // Custom cache implementation env.useCustomCache = true; env.customCache = new CustomCache(); // Implement Cache API interface // WASM binary caching env.useWasmCache = true; // default: true ``` ### Disable Caching ```javascript import { env } from '@huggingface/transformers'; // Disable all caching (re-download on every load) env.useFSCache = false; env.useBrowserCache = false; env.useWasmCache = false; env.cacheDir = null; ``` For comprehensive caching documentation including: - Browser Cache API details and storage limits - Node.js filesystem cache structure and management - Custom cache implementations (Redis, database, S3) - Cache clearing and monitoring strategies - Best practices and troubleshooting See **[Caching Reference](./CACHE.md)** ## WASM Configuration Configure ONNX Runtime Web Assembly backend settings. ### Basic WASM Settings ```javascript import { env } from '@huggingface/transformers'; // Set custom WASM paths env.backends.onnx.wasm.wasmPaths = 'https://cdn.jsdelivr.net/npm/onnxruntime-web/dist/'; // Configure number of threads (Node.js only) env.backends.onnx.wasm.numThreads = 4; // Enable/disable SIMD (single instruction, multiple data) env.backends.onnx.wasm.simd = true; ``` ### Proxy Configuration ```javascript import { env } from '@huggingface/transformers'; // Configure proxy for WASM downloads env.backends.onnx.wasm.proxy = true; ``` ### Self-Hosted WASM Files ```javascript import { env } from '@huggingface/transformers'; // Host WASM files on your own server env.backends.onnx.wasm.wasmPaths = '/static/wasm/'; ``` **Required files:** - `ort-wasm.wasm` - Main WASM binary - `ort-wasm-simd.wasm` - SIMD-enabled WASM binary - `ort-wasm-threaded.wasm` - Multi-threaded WASM binary - `ort-wasm-simd-threaded.wasm` - SIMD + multi-threaded WASM binary ## Common Configuration Patterns ### Development Setup ```javascript import { env } from '@huggingface/transformers'; // Fast iteration with caching env.allowRemoteModels = true; env.useBrowserCache = true; // Browser env.useFSCache = true; // Node.js env.cacheDir = './.cache'; ``` ### Production (Local Models) ```javascript import { env } from '@huggingface/transformers'; // Secure, offline-capable setup env.allowRemoteModels = false; env.allowLocalModels = true; env.localModelPath = '/app/models/'; env.useFSCache = false; // Models already local ``` ### Offline-First Application ```javascript import { env } from '@huggingface/transformers'; // Try local first, fall back to remote env.allowLocalModels = true; env.localModelPath = './models/'; env.allowRemoteModels = true; env.useFSCache = true; env.cacheDir = './cache'; ``` ### Custom CDN ```javascript import { env } from '@huggingface/transformers'; // Use your own model hosting env.remoteHost = 'https://cdn.example.com/ml-models'; env.remotePathTemplate = '{model}/{file}'; env.useBrowserCache = true; ``` ### Memory-Constrained Environment ```javascript import { env } from '@huggingface/transformers'; // Minimize disk/memory usage env.useFSCache = false; env.useBrowserCache = false; env.useWasmCache = false; env.cacheDir = null; ``` ### Testing/CI Environment ```javascript import { env } from '@huggingface/transformers'; // Predictable, isolated testing env.allowRemoteModels = false; env.allowLocalModels = true; env.localModelPath = './test-fixtures/models/'; env.useFSCache = false; ``` ## Environment Best Practices ### 1. Configure Early Set `env` properties before loading any models: ```javascript import { env, pipeline } from '@huggingface/transformers'; // ✓ Good: Configure before loading env.allowRemoteModels = false; env.localModelPath = '/app/models/'; const pipe = await pipeline('sentiment-analysis'); // ✗ Bad: Configuring after loading may not take effect const pipe = await pipeline('sentiment-analysis'); env.allowRemoteModels = false; // Too late! ``` ### 2. Use Environment Variables ```javascript import { env } from '@huggingface/transformers'; // Configure based on environment env.allowRemoteModels = process.env.NODE_ENV === 'development'; env.cacheDir = process.env.MODEL_CACHE_DIR || './.cache'; env.localModelPath = process.env.LOCAL_MODELS_PATH || '/app/models/'; ``` ### 3. Handle Errors Gracefully ```javascript import { pipeline, env } from '@huggingface/transformers'; try { env.allowRemoteModels = false; const pipe = await pipeline('sentiment-analysis', 'my-model'); } catch (error) { if (error.message.includes('not found')) { console.error('Model not found locally. Enable remote models or download the model.'); } throw error; } ``` ### 4. Log Configuration ```javascript import { env } from '@huggingface/transformers'; console.log('Transformers.js Configuration:', { version: env.version, allowRemoteModels: env.allowRemoteModels, allowLocalModels: env.allowLocalModels, localModelPath: env.localModelPath, cacheDir: env.cacheDir, useFSCache: env.useFSCache, useBrowserCache: env.useBrowserCache }); ``` ## Related Documentation - **[Caching Reference](./CACHE.md)** - Comprehensive caching guide (browser, Node.js, custom implementations) - [Pipeline Options](./PIPELINE_OPTIONS.md) - Configure pipeline loading with `progress_callback`, `device`, `dtype`, etc. - [Model Architectures](./MODEL_ARCHITECTURES.md) - Supported models and architectures - [Examples](./EXAMPLES.md) - Code examples for different runtimes - [Main Skill Guide](../SKILL.md) - Getting started and common usage ================================================ FILE: skills/transformers.js/references/EXAMPLES.md ================================================ # Transformers.js Code Examples Working examples showing how to use Transformers.js across different runtimes and frameworks. All examples use the same task and model for consistency: - **Task**: `feature-extraction` - **Model**: `onnx-community/all-MiniLM-L6-v2-ONNX` ## Table of Contents 1. [Browser (Vanilla JS)](#browser-vanilla-js) 2. [Node.js](#nodejs) 3. [React](#react) 4. [Express API](#express-api) ## Browser (Vanilla JS) ### Basic Usage ```html Feature Extraction

Text Embedding Generator

``` ### With Progress Tracking ```html Feature Extraction with Progress

Text Embedding Generator

Loading model...

``` ## Node.js ### Basic Script ```javascript // embed.js import { pipeline } from '@huggingface/transformers'; async function generateEmbedding(text) { const extractor = await pipeline( 'feature-extraction', 'onnx-community/all-MiniLM-L6-v2-ONNX' ); const output = await extractor(text, { pooling: 'mean', normalize: true }); console.log('Text:', text); console.log('Embedding dimensions:', output.data.length); console.log('First 5 values:', Array.from(output.data).slice(0, 5)); await extractor.dispose(); } generateEmbedding('Hello, world!'); ``` ### Batch Processing ```javascript // batch-embed.js import { pipeline } from '@huggingface/transformers'; import fs from 'fs/promises'; async function embedDocuments(documents) { const extractor = await pipeline( 'feature-extraction', 'onnx-community/all-MiniLM-L6-v2-ONNX' ); console.log(`Processing ${documents.length} documents...`); const embeddings = []; for (let i = 0; i < documents.length; i++) { const output = await extractor(documents[i], { pooling: 'mean', normalize: true }); embeddings.push({ text: documents[i], embedding: Array.from(output.data) }); console.log(`Processed ${i + 1}/${documents.length}`); } await fs.writeFile( 'embeddings.json', JSON.stringify(embeddings, null, 2) ); console.log('Saved to embeddings.json'); await extractor.dispose(); } const documents = [ 'The cat sat on the mat', 'A dog played in the park', 'Machine learning is fascinating' ]; embedDocuments(documents); ``` ### CLI with Progress ```javascript // cli-embed.js import { pipeline } from '@huggingface/transformers'; async function main() { const text = process.argv[2] || 'Hello, world!'; console.log('Loading model...'); const fileProgress = {}; const extractor = await pipeline( 'feature-extraction', 'onnx-community/all-MiniLM-L6-v2-ONNX', { progress_callback: (info) => { if (info.status === 'progress') { fileProgress[info.file] = info.progress; // Show all files progress const progressLines = Object.entries(fileProgress) .map(([file, progress]) => ` ${file}: ${progress.toFixed(1)}%`) .join('\n'); process.stdout.write(`\r\x1b[K${progressLines}`); } if (info.status === 'done') { console.log(`\n✓ ${info.file} complete`); } if (info.status === 'ready') { console.log('\nModel ready!'); } } } ); console.log('Generating embedding...'); const output = await extractor(text, { pooling: 'mean', normalize: true }); console.log(`\nText: "${text}"`); console.log(`Dimensions: ${output.data.length}`); console.log(`First 5 values: ${Array.from(output.data).slice(0, 5).join(', ')}`); await extractor.dispose(); } main(); ``` ## React ### Basic Component ```jsx // EmbeddingGenerator.jsx import { useState, useRef, useEffect } from 'react'; import { pipeline } from '@huggingface/transformers'; export function EmbeddingGenerator() { const extractorRef = useRef(null); const [text, setText] = useState(''); const [embedding, setEmbedding] = useState(null); const [loading, setLoading] = useState(false); const generate = async () => { if (!text) return; setLoading(true); // Load model on first generate if (!extractorRef.current) { extractorRef.current = await pipeline( 'feature-extraction', 'onnx-community/all-MiniLM-L6-v2-ONNX' ); } const output = await extractorRef.current(text, { pooling: 'mean', normalize: true }); setEmbedding(Array.from(output.data)); setLoading(false); }; // Cleanup on unmount useEffect(() => { return () => { if (extractorRef.current) { extractorRef.current.dispose(); } }; }, []); return (

Text Embedding Generator

``` ### React ```jsx import { useState, useRef, useEffect } from 'react'; import { pipeline, TextStreamer } from '@huggingface/transformers'; function StreamingGenerator() { const generatorRef = useRef(null); const [output, setOutput] = useState(''); const [loading, setLoading] = useState(false); const handleGenerate = async (prompt) => { if (!prompt) return; setLoading(true); setOutput(''); // Load model on first generate if (!generatorRef.current) { generatorRef.current = await pipeline( 'text-generation', 'onnx-community/Qwen2.5-0.5B-Instruct', { dtype: 'q4' } ); } const streamer = new TextStreamer(generatorRef.current.tokenizer, { skip_prompt: true, skip_special_tokens: true, callback_function: (token) => { setOutput((prev) => prev + token); }, }); await generatorRef.current(prompt, { max_new_tokens: 200, temperature: 0.7, streamer, }); setLoading(false); }; // Cleanup on unmount useEffect(() => { return () => { if (generatorRef.current) { generatorRef.current.dispose(); } }; }, []); return (
{output}
); } ``` ## Chat Format Use structured messages for conversations. Works with both basic generation and streaming (just add `streamer` parameter). ### Single Turn ```javascript import { pipeline } from '@huggingface/transformers'; const generator = await pipeline( 'text-generation', 'onnx-community/Qwen2.5-0.5B-Instruct', { dtype: 'q4' } ); const messages = [ { role: 'system', content: 'You are a helpful assistant.' }, { role: 'user', content: 'How do I create an async function?' } ]; const result = await generator(messages, { max_new_tokens: 256, temperature: 0.7, }); console.log(result[0].generated_text); ``` ### Multi-turn Conversation ```javascript const conversation = [ { role: 'system', content: 'You are a helpful assistant.' }, { role: 'user', content: 'What is JavaScript?' }, { role: 'assistant', content: 'JavaScript is a programming language...' }, { role: 'user', content: 'Can you show an example?' } ]; const result = await generator(conversation, { max_new_tokens: 200, temperature: 0.7, }); // To add streaming, just pass a streamer: // streamer: new TextStreamer(generator.tokenizer, {...}) ``` ## Generation Parameters ### Common Parameters ```javascript await generator(prompt, { // Token limits max_new_tokens: 512, // Maximum tokens to generate min_new_tokens: 0, // Minimum tokens to generate // Sampling temperature: 0.7, // Randomness (0.0-2.0) top_k: 50, // Consider top K tokens top_p: 0.95, // Nucleus sampling do_sample: true, // Use random sampling (false = always pick most likely token) // Repetition control repetition_penalty: 1.0, // Penalty for repeating (1.0 = no penalty) no_repeat_ngram_size: 0, // Prevent repeating n-grams // Streaming streamer: streamer, // TextStreamer instance }); ``` ### Parameter Effects **Temperature:** - Low (0.1-0.5): More focused and deterministic - Medium (0.6-0.9): Balanced creativity and coherence - High (1.0-2.0): More creative and random ```javascript // Focused output await generator(prompt, { temperature: 0.3, max_new_tokens: 100 }); // Creative output await generator(prompt, { temperature: 1.2, max_new_tokens: 100 }); ``` **Sampling Methods:** ```javascript // Greedy (deterministic) await generator(prompt, { do_sample: false, max_new_tokens: 100 }); // Top-k sampling await generator(prompt, { top_k: 50, temperature: 0.7, max_new_tokens: 100 }); // Top-p (nucleus) sampling await generator(prompt, { top_p: 0.95, temperature: 0.7, max_new_tokens: 100 }); ``` ## Model Selection Browse available text generation models on Hugging Face Hub: **https://huggingface.co/models?pipeline_tag=text-generation&library=transformers.js&sort=trending** ### Selection Tips - **Small models (< 1B params)**: Fast, browser-friendly, use `dtype: 'q4'` - **Medium models (1-3B params)**: Balanced quality/speed, use `dtype: 'q4'` or `fp16` - **Large models (> 3B params)**: High quality, slower, best for Node.js with `dtype: 'fp16'` Check model cards for: - Parameter count and model size - Supported languages - Benchmark scores - License restrictions ## Best Practices 1. **Model Size**: Use quantized models (`q4`) for browsers, larger models (`fp16`) for servers 2. **Streaming**: Use streaming for better UX - shows progress and feels responsive 3. **Token Limits**: Set `max_new_tokens` to prevent runaway generation 4. **Temperature**: Tune based on use case (creative: 0.8-1.2, factual: 0.3-0.7) 5. **Memory**: Always call `dispose()` when done 6. **Caching**: Load model once, reuse for multiple requests ## Related Documentation - [Pipeline Options](./PIPELINE_OPTIONS.md) - Configure pipeline loading - [Configuration Reference](./CONFIGURATION.md) - Environment settings - [Code Examples](./EXAMPLES.md) - More examples for different runtimes - [Main Skill Guide](../SKILL.md) - Getting started guide