[ { "path": ".agents/skills/skill-creator/LICENSE.txt", "content": "\n Apache License\n Version 2.0, January 2004\n http://www.apache.org/licenses/\n\n TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION\n\n 1. Definitions.\n\n \"License\" shall mean the terms and conditions for use, reproduction,\n and distribution as defined by Sections 1 through 9 of this document.\n\n \"Licensor\" shall mean the copyright owner or entity authorized by\n the copyright owner that is granting the License.\n\n \"Legal Entity\" shall mean the union of the acting entity and all\n other entities that control, are controlled by, or are under common\n control with that entity. For the purposes of this definition,\n \"control\" means (i) the power, direct or indirect, to cause the\n direction or management of such entity, whether by contract or\n otherwise, or (ii) ownership of fifty percent (50%) or more of the\n outstanding shares, or (iii) beneficial ownership of such entity.\n\n \"You\" (or \"Your\") shall mean an individual or Legal Entity\n exercising permissions granted by this License.\n\n \"Source\" form shall mean the preferred form for making modifications,\n including but not limited to software source code, documentation\n source, and configuration files.\n\n \"Object\" form shall mean any form resulting from mechanical\n transformation or translation of a Source form, including but\n not limited to compiled object code, generated documentation,\n and conversions to other media types.\n\n \"Work\" shall mean the work of authorship, whether in Source or\n Object form, made available under the License, as indicated by a\n copyright notice that is included in or attached to the work\n (an example is provided in the Appendix below).\n\n \"Derivative Works\" shall mean any work, whether in Source or Object\n form, that is based on (or derived from) the Work and for which the\n editorial revisions, annotations, elaborations, or other modifications\n represent, as a whole, an original work of authorship. For the purposes\n of this License, Derivative Works shall not include works that remain\n separable from, or merely link (or bind by name) to the interfaces of,\n the Work and Derivative Works thereof.\n\n \"Contribution\" shall mean any work of authorship, including\n the original version of the Work and any modifications or additions\n to that Work or Derivative Works thereof, that is intentionally\n submitted to Licensor for inclusion in the Work by the copyright owner\n or by an individual or Legal Entity authorized to submit on behalf of\n the copyright owner. For the purposes of this definition, \"submitted\"\n means any form of electronic, verbal, or written communication sent\n to the Licensor or its representatives, including but not limited to\n communication on electronic mailing lists, source code control systems,\n and issue tracking systems that are managed by, or on behalf of, the\n Licensor for the purpose of discussing and improving the Work, but\n excluding communication that is conspicuously marked or otherwise\n designated in writing by the copyright owner as \"Not a Contribution.\"\n\n \"Contributor\" shall mean Licensor and any individual or Legal Entity\n on behalf of whom a Contribution has been received by Licensor and\n subsequently incorporated within the Work.\n\n 2. Grant of Copyright License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n copyright license to reproduce, prepare Derivative Works of,\n publicly display, publicly perform, sublicense, and distribute the\n Work and such Derivative Works in Source or Object form.\n\n 3. Grant of Patent License. Subject to the terms and conditions of\n this License, each Contributor hereby grants to You a perpetual,\n worldwide, non-exclusive, no-charge, royalty-free, irrevocable\n (except as stated in this section) patent license to make, have made,\n use, offer to sell, sell, import, and otherwise transfer the Work,\n where such license applies only to those patent claims licensable\n by such Contributor that are necessarily infringed by their\n Contribution(s) alone or by combination of their Contribution(s)\n with the Work to which such Contribution(s) was submitted. If You\n institute patent litigation against any entity (including a\n cross-claim or counterclaim in a lawsuit) alleging that the Work\n or a Contribution incorporated within the Work constitutes direct\n or contributory patent infringement, then any patent licenses\n granted to You under this License for that Work shall terminate\n as of the date such litigation is filed.\n\n 4. Redistribution. You may reproduce and distribute copies of the\n Work or Derivative Works thereof in any medium, with or without\n modifications, and in Source or Object form, provided that You\n meet the following conditions:\n\n (a) You must give any other recipients of the Work or\n Derivative Works a copy of this License; and\n\n (b) You must cause any modified files to carry prominent notices\n stating that You changed the files; and\n\n (c) You must retain, in the Source form of any Derivative Works\n that You distribute, all copyright, patent, trademark, and\n attribution notices from the Source form of the Work,\n excluding those notices that do not pertain to any part of\n the Derivative Works; and\n\n (d) If the Work includes a \"NOTICE\" text file as part of its\n distribution, then any Derivative Works that You distribute must\n include a readable copy of the attribution notices contained\n within such NOTICE file, excluding those notices that do not\n pertain to any part of the Derivative Works, in at least one\n of the following places: within a NOTICE text file distributed\n as part of the Derivative Works; within the Source form or\n documentation, if provided along with the Derivative Works; or,\n within a display generated by the Derivative Works, if and\n wherever such third-party notices normally appear. The contents\n of the NOTICE file are for informational purposes only and\n do not modify the License. You may add Your own attribution\n notices within Derivative Works that You distribute, alongside\n or as an addendum to the NOTICE text from the Work, provided\n that such additional attribution notices cannot be construed\n as modifying the License.\n\n You may add Your own copyright statement to Your modifications and\n may provide additional or different license terms and conditions\n for use, reproduction, or distribution of Your modifications, or\n for any such Derivative Works as a whole, provided Your use,\n reproduction, and distribution of the Work otherwise complies with\n the conditions stated in this License.\n\n 5. Submission of Contributions. Unless You explicitly state otherwise,\n any Contribution intentionally submitted for inclusion in the Work\n by You to the Licensor shall be under the terms and conditions of\n this License, without any additional terms or conditions.\n Notwithstanding the above, nothing herein shall supersede or modify\n the terms of any separate license agreement you may have executed\n with Licensor regarding such Contributions.\n\n 6. Trademarks. This License does not grant permission to use the trade\n names, trademarks, service marks, or product names of the Licensor,\n except as required for reasonable and customary use in describing the\n origin of the Work and reproducing the content of the NOTICE file.\n\n 7. Disclaimer of Warranty. Unless required by applicable law or\n agreed to in writing, Licensor provides the Work (and each\n Contributor provides its Contributions) on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or\n implied, including, without limitation, any warranties or conditions\n of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A\n PARTICULAR PURPOSE. You are solely responsible for determining the\n appropriateness of using or redistributing the Work and assume any\n risks associated with Your exercise of permissions under this License.\n\n 8. Limitation of Liability. In no event and under no legal theory,\n whether in tort (including negligence), contract, or otherwise,\n unless required by applicable law (such as deliberate and grossly\n negligent acts) or agreed to in writing, shall any Contributor be\n liable to You for damages, including any direct, indirect, special,\n incidental, or consequential damages of any character arising as a\n result of this License or out of the use or inability to use the\n Work (including but not limited to damages for loss of goodwill,\n work stoppage, computer failure or malfunction, or any and all\n other commercial damages or losses), even if such Contributor\n has been advised of the possibility of such damages.\n\n 9. Accepting Warranty or Additional Liability. While redistributing\n the Work or Derivative Works thereof, You may choose to offer,\n and charge a fee for, acceptance of support, warranty, indemnity,\n or other liability obligations and/or rights consistent with this\n License. However, in accepting such obligations, You may act only\n on Your own behalf and on Your sole responsibility, not on behalf\n of any other Contributor, and only if You agree to indemnify,\n defend, and hold each Contributor harmless for any liability\n incurred by, or claims asserted against, such Contributor by reason\n of your accepting any such warranty or additional liability.\n\n END OF TERMS AND CONDITIONS\n\n APPENDIX: How to apply the Apache License to your work.\n\n To apply the Apache License to your work, attach the following\n boilerplate notice, with the fields enclosed by brackets \"[]\"\n replaced with your own identifying information. (Don't include\n the brackets!) The text should be enclosed in the appropriate\n comment syntax for the file format. We also recommend that a\n file or class name and description of purpose be included on the\n same \"printed page\" as the copyright notice for easier\n identification within third-party archives.\n\n Copyright 2026 Anthropic, PBC.\n\n Licensed under the Apache License, Version 2.0 (the \"License\");\n you may not use this file except in compliance with the License.\n You may obtain a copy of the License at\n\n http://www.apache.org/licenses/LICENSE-2.0\n\n Unless required by applicable law or agreed to in writing, software\n distributed under the License is distributed on an \"AS IS\" BASIS,\n WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n See the License for the specific language governing permissions and\n limitations under the License." }, { "path": ".agents/skills/skill-creator/SKILL.md", "content": "---\nname: skill-creator\ndescription: Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.\n---\n\n# Skill Creator\n\nA skill for creating new skills and iteratively improving them.\n\nAt a high level, the process of creating a skill goes like this:\n\n- Decide what you want the skill to do and roughly how it should do it\n- Write a draft of the skill\n- Create a few test prompts and run claude-with-access-to-the-skill on them\n- Help the user evaluate the results both qualitatively and quantitatively\n - While the runs happen in the background, draft some quantitative evals if there aren't any (if there are some, you can either use as is or modify if you feel something needs to change about them). Then explain them to the user (or if they already existed, explain the ones that already exist)\n - Use the `eval-viewer/generate_review.py` script to show the user the results for them to look at, and also let them look at the quantitative metrics\n- Rewrite the skill based on feedback from the user's evaluation of the results (and also if there are any glaring flaws that become apparent from the quantitative benchmarks)\n- Repeat until you're satisfied\n- Expand the test set and try again at larger scale\n\nYour job when using this skill is to figure out where the user is in this process and then jump in and help them progress through these stages. So for instance, maybe they're like \"I want to make a skill for X\". You can help narrow down what they mean, write a draft, write the test cases, figure out how they want to evaluate, run all the prompts, and repeat.\n\nOn the other hand, maybe they already have a draft of the skill. In this case you can go straight to the eval/iterate part of the loop.\n\nOf course, you should always be flexible and if the user is like \"I don't need to run a bunch of evaluations, just vibe with me\", you can do that instead.\n\nThen after the skill is done (but again, the order is flexible), you can also run the skill description improver, which we have a whole separate script for, to optimize the triggering of the skill.\n\nCool? Cool.\n\n## Communicating with the user\n\nThe skill creator is liable to be used by people across a wide range of familiarity with coding jargon. If you haven't heard (and how could you, it's only very recently that it started), there's a trend now where the power of Claude is inspiring plumbers to open up their terminals, parents and grandparents to google \"how to install npm\". On the other hand, the bulk of users are probably fairly computer-literate.\n\nSo please pay attention to context cues to understand how to phrase your communication! In the default case, just to give you some idea:\n\n- \"evaluation\" and \"benchmark\" are borderline, but OK\n- for \"JSON\" and \"assertion\" you want to see serious cues from the user that they know what those things are before using them without explaining them\n\nIt's OK to briefly explain terms if you're in doubt, and feel free to clarify terms with a short definition if you're unsure if the user will get it.\n\n---\n\n## Creating a skill\n\n### Capture Intent\n\nStart by understanding the user's intent. The current conversation might already contain a workflow the user wants to capture (e.g., they say \"turn this into a skill\"). If so, extract answers from the conversation history first — the tools used, the sequence of steps, corrections the user made, input/output formats observed. The user may need to fill the gaps, and should confirm before proceeding to the next step.\n\n1. What should this skill enable Claude to do?\n2. When should this skill trigger? (what user phrases/contexts)\n3. What's the expected output format?\n4. Should we set up test cases to verify the skill works? Skills with objectively verifiable outputs (file transforms, data extraction, code generation, fixed workflow steps) benefit from test cases. Skills with subjective outputs (writing style, art) often don't need them. Suggest the appropriate default based on the skill type, but let the user decide.\n\n### Interview and Research\n\nProactively ask questions about edge cases, input/output formats, example files, success criteria, and dependencies. Wait to write test prompts until you've got this part ironed out.\n\nCheck available MCPs - if useful for research (searching docs, finding similar skills, looking up best practices), research in parallel via subagents if available, otherwise inline. Come prepared with context to reduce burden on the user.\n\n### Write the SKILL.md\n\nBased on the user interview, fill in these components:\n\n- **name**: Skill identifier\n- **description**: When to trigger, what it does. This is the primary triggering mechanism - include both what the skill does AND specific contexts for when to use it. All \"when to use\" info goes here, not in the body. Note: currently Claude has a tendency to \"undertrigger\" skills -- to not use them when they'd be useful. To combat this, please make the skill descriptions a little bit \"pushy\". So for instance, instead of \"How to build a simple fast dashboard to display internal Anthropic data.\", you might write \"How to build a simple fast dashboard to display internal Anthropic data. Make sure to use this skill whenever the user mentions dashboards, data visualization, internal metrics, or wants to display any kind of company data, even if they don't explicitly ask for a 'dashboard.'\"\n- **compatibility**: Required tools, dependencies (optional, rarely needed)\n- **the rest of the skill :)**\n\n### Skill Writing Guide\n\n#### Anatomy of a Skill\n\n```\nskill-name/\n├── SKILL.md (required)\n│ ├── YAML frontmatter (name, description required)\n│ └── Markdown instructions\n└── Bundled Resources (optional)\n ├── scripts/ - Executable code for deterministic/repetitive tasks\n ├── references/ - Docs loaded into context as needed\n └── assets/ - Files used in output (templates, icons, fonts)\n```\n\n#### Progressive Disclosure\n\nSkills use a three-level loading system:\n\n1. **Metadata** (name + description) - Always in context (~100 words)\n2. **SKILL.md body** - In context whenever skill triggers (<500 lines ideal)\n3. **Bundled resources** - As needed (unlimited, scripts can execute without loading)\n\nThese word counts are approximate and you can feel free to go longer if needed.\n\n**Key patterns:**\n\n- Keep SKILL.md under 500 lines; if you're approaching this limit, add an additional layer of hierarchy along with clear pointers about where the model using the skill should go next to follow up.\n- Reference files clearly from SKILL.md with guidance on when to read them\n- For large reference files (>300 lines), include a table of contents\n\n**Domain organization**: When a skill supports multiple domains/frameworks, organize by variant:\n\n```\ncloud-deploy/\n├── SKILL.md (workflow + selection)\n└── references/\n ├── aws.md\n ├── gcp.md\n └── azure.md\n```\n\nClaude reads only the relevant reference file.\n\n#### Principle of Lack of Surprise\n\nThis goes without saying, but skills must not contain malware, exploit code, or any content that could compromise system security. A skill's contents should not surprise the user in their intent if described. Don't go along with requests to create misleading skills or skills designed to facilitate unauthorized access, data exfiltration, or other malicious activities. Things like a \"roleplay as an XYZ\" are OK though.\n\n#### Writing Patterns\n\nPrefer using the imperative form in instructions.\n\n**Defining output formats** - You can do it like this:\n\n```markdown\n## Report structure\n\nALWAYS use this exact template:\n\n# [Title]\n\n## Executive summary\n\n## Key findings\n\n## Recommendations\n```\n\n**Examples pattern** - It's useful to include examples. You can format them like this (but if \"Input\" and \"Output\" are in the examples you might want to deviate a little):\n\n```markdown\n## Commit message format\n\n**Example 1:**\nInput: Added user authentication with JWT tokens\nOutput: feat(auth): implement JWT-based authentication\n```\n\n### Writing Style\n\nTry to explain to the model why things are important in lieu of heavy-handed musty MUSTs. Use theory of mind and try to make the skill general and not super-narrow to specific examples. Start by writing a draft and then look at it with fresh eyes and improve it.\n\n### Test Cases\n\nAfter writing the skill draft, come up with 2-3 realistic test prompts — the kind of thing a real user would actually say. Share them with the user: [you don't have to use this exact language] \"Here are a few test cases I'd like to try. Do these look right, or do you want to add more?\" Then run them.\n\nSave test cases to `evals/evals.json`. Don't write assertions yet — just the prompts. You'll draft assertions in the next step while the runs are in progress.\n\n```json\n{\n \"skill_name\": \"example-skill\",\n \"evals\": [\n {\n \"id\": 1,\n \"prompt\": \"User's task prompt\",\n \"expected_output\": \"Description of expected result\",\n \"files\": []\n }\n ]\n}\n```\n\nSee `references/schemas.md` for the full schema (including the `assertions` field, which you'll add later).\n\n## Running and evaluating test cases\n\nThis section is one continuous sequence — don't stop partway through. Do NOT use `/skill-test` or any other testing skill.\n\nPut results in `-workspace/` as a sibling to the skill directory. Within the workspace, organize results by iteration (`iteration-1/`, `iteration-2/`, etc.) and within that, each test case gets a directory (`eval-0/`, `eval-1/`, etc.). Don't create all of this upfront — just create directories as you go.\n\n### Step 1: Spawn all runs (with-skill AND baseline) in the same turn\n\nFor each test case, spawn two subagents in the same turn — one with the skill, one without. This is important: don't spawn the with-skill runs first and then come back for baselines later. Launch everything at once so it all finishes around the same time.\n\n**With-skill run:**\n\n```\nExecute this task:\n- Skill path: \n- Task: \n- Input files: \n- Save outputs to: /iteration-/eval-/with_skill/outputs/\n- Outputs to save: \n```\n\n**Baseline run** (same prompt, but the baseline depends on context):\n\n- **Creating a new skill**: no skill at all. Same prompt, no skill path, save to `without_skill/outputs/`.\n- **Improving an existing skill**: the old version. Before editing, snapshot the skill (`cp -r /skill-snapshot/`), then point the baseline subagent at the snapshot. Save to `old_skill/outputs/`.\n\nWrite an `eval_metadata.json` for each test case (assertions can be empty for now). Give each eval a descriptive name based on what it's testing — not just \"eval-0\". Use this name for the directory too. If this iteration uses new or modified eval prompts, create these files for each new eval directory — don't assume they carry over from previous iterations.\n\n```json\n{\n \"eval_id\": 0,\n \"eval_name\": \"descriptive-name-here\",\n \"prompt\": \"The user's task prompt\",\n \"assertions\": []\n}\n```\n\n### Step 2: While runs are in progress, draft assertions\n\nDon't just wait for the runs to finish — you can use this time productively. Draft quantitative assertions for each test case and explain them to the user. If assertions already exist in `evals/evals.json`, review them and explain what they check.\n\nGood assertions are objectively verifiable and have descriptive names — they should read clearly in the benchmark viewer so someone glancing at the results immediately understands what each one checks. Subjective skills (writing style, design quality) are better evaluated qualitatively — don't force assertions onto things that need human judgment.\n\nUpdate the `eval_metadata.json` files and `evals/evals.json` with the assertions once drafted. Also explain to the user what they'll see in the viewer — both the qualitative outputs and the quantitative benchmark.\n\n### Step 3: As runs complete, capture timing data\n\nWhen each subagent task completes, you receive a notification containing `total_tokens` and `duration_ms`. Save this data immediately to `timing.json` in the run directory:\n\n```json\n{\n \"total_tokens\": 84852,\n \"duration_ms\": 23332,\n \"total_duration_seconds\": 23.3\n}\n```\n\nThis is the only opportunity to capture this data — it comes through the task notification and isn't persisted elsewhere. Process each notification as it arrives rather than trying to batch them.\n\n### Step 4: Grade, aggregate, and launch the viewer\n\nOnce all runs are done:\n\n1. **Grade each run** — spawn a grader subagent (or grade inline) that reads `agents/grader.md` and evaluates each assertion against the outputs. Save results to `grading.json` in each run directory. The grading.json expectations array must use the fields `text`, `passed`, and `evidence` (not `name`/`met`/`details` or other variants) — the viewer depends on these exact field names. For assertions that can be checked programmatically, write and run a script rather than eyeballing it — scripts are faster, more reliable, and can be reused across iterations.\n\n2. **Aggregate into benchmark** — run the aggregation script from the skill-creator directory:\n\n ```bash\n python -m scripts.aggregate_benchmark /iteration-N --skill-name \n ```\n\n This produces `benchmark.json` and `benchmark.md` with pass_rate, time, and tokens for each configuration, with mean ± stddev and the delta. If generating benchmark.json manually, see `references/schemas.md` for the exact schema the viewer expects.\n Put each with_skill version before its baseline counterpart.\n\n3. **Do an analyst pass** — read the benchmark data and surface patterns the aggregate stats might hide. See `agents/analyzer.md` (the \"Analyzing Benchmark Results\" section) for what to look for — things like assertions that always pass regardless of skill (non-discriminating), high-variance evals (possibly flaky), and time/token tradeoffs.\n\n4. **Launch the viewer** with both qualitative outputs and quantitative data:\n\n ```bash\n nohup python /eval-viewer/generate_review.py \\\n /iteration-N \\\n --skill-name \"my-skill\" \\\n --benchmark /iteration-N/benchmark.json \\\n > /dev/null 2>&1 &\n VIEWER_PID=$!\n ```\n\n For iteration 2+, also pass `--previous-workspace /iteration-`.\n\n **Cowork / headless environments:** If `webbrowser.open()` is not available or the environment has no display, use `--static ` to write a standalone HTML file instead of starting a server. Feedback will be downloaded as a `feedback.json` file when the user clicks \"Submit All Reviews\". After download, copy `feedback.json` into the workspace directory for the next iteration to pick up.\n\nNote: please use generate_review.py to create the viewer; there's no need to write custom HTML.\n\n5. **Tell the user** something like: \"I've opened the results in your browser. There are two tabs — 'Outputs' lets you click through each test case and leave feedback, 'Benchmark' shows the quantitative comparison. When you're done, come back here and let me know.\"\n\n### What the user sees in the viewer\n\nThe \"Outputs\" tab shows one test case at a time:\n\n- **Prompt**: the task that was given\n- **Output**: the files the skill produced, rendered inline where possible\n- **Previous Output** (iteration 2+): collapsed section showing last iteration's output\n- **Formal Grades** (if grading was run): collapsed section showing assertion pass/fail\n- **Feedback**: a textbox that auto-saves as they type\n- **Previous Feedback** (iteration 2+): their comments from last time, shown below the textbox\n\nThe \"Benchmark\" tab shows the stats summary: pass rates, timing, and token usage for each configuration, with per-eval breakdowns and analyst observations.\n\nNavigation is via prev/next buttons or arrow keys. When done, they click \"Submit All Reviews\" which saves all feedback to `feedback.json`.\n\n### Step 5: Read the feedback\n\nWhen the user tells you they're done, read `feedback.json`:\n\n```json\n{\n \"reviews\": [\n {\n \"run_id\": \"eval-0-with_skill\",\n \"feedback\": \"the chart is missing axis labels\",\n \"timestamp\": \"...\"\n },\n { \"run_id\": \"eval-1-with_skill\", \"feedback\": \"\", \"timestamp\": \"...\" },\n { \"run_id\": \"eval-2-with_skill\", \"feedback\": \"perfect, love this\", \"timestamp\": \"...\" }\n ],\n \"status\": \"complete\"\n}\n```\n\nEmpty feedback means the user thought it was fine. Focus your improvements on the test cases where the user had specific complaints.\n\nKill the viewer server when you're done with it:\n\n```bash\nkill $VIEWER_PID 2>/dev/null\n```\n\n---\n\n## Improving the skill\n\nThis is the heart of the loop. You've run the test cases, the user has reviewed the results, and now you need to make the skill better based on their feedback.\n\n### How to think about improvements\n\n1. **Generalize from the feedback.** The big picture thing that's happening here is that we're trying to create skills that can be used a million times (maybe literally, maybe even more who knows) across many different prompts. Here you and the user are iterating on only a few examples over and over again because it helps move faster. The user knows these examples in and out and it's quick for them to assess new outputs. But if the skill you and the user are codeveloping works only for those examples, it's useless. Rather than put in fiddly overfitty changes, or oppressively constrictive MUSTs, if there's some stubborn issue, you might try branching out and using different metaphors, or recommending different patterns of working. It's relatively cheap to try and maybe you'll land on something great.\n\n2. **Keep the prompt lean.** Remove things that aren't pulling their weight. Make sure to read the transcripts, not just the final outputs — if it looks like the skill is making the model waste a bunch of time doing things that are unproductive, you can try getting rid of the parts of the skill that are making it do that and seeing what happens.\n\n3. **Explain the why.** Try hard to explain the **why** behind everything you're asking the model to do. Today's LLMs are _smart_. They have good theory of mind and when given a good harness can go beyond rote instructions and really make things happen. Even if the feedback from the user is terse or frustrated, try to actually understand the task and why the user is writing what they wrote, and what they actually wrote, and then transmit this understanding into the instructions. If you find yourself writing ALWAYS or NEVER in all caps, or using super rigid structures, that's a yellow flag — if possible, reframe and explain the reasoning so that the model understands why the thing you're asking for is important. That's a more humane, powerful, and effective approach.\n\n4. **Look for repeated work across test cases.** Read the transcripts from the test runs and notice if the subagents all independently wrote similar helper scripts or took the same multi-step approach to something. If all 3 test cases resulted in the subagent writing a `create_docx.py` or a `build_chart.py`, that's a strong signal the skill should bundle that script. Write it once, put it in `scripts/`, and tell the skill to use it. This saves every future invocation from reinventing the wheel.\n\nThis task is pretty important (we are trying to create billions a year in economic value here!) and your thinking time is not the blocker; take your time and really mull things over. I'd suggest writing a draft revision and then looking at it anew and making improvements. Really do your best to get into the head of the user and understand what they want and need.\n\n### The iteration loop\n\nAfter improving the skill:\n\n1. Apply your improvements to the skill\n2. Rerun all test cases into a new `iteration-/` directory, including baseline runs. If you're creating a new skill, the baseline is always `without_skill` (no skill) — that stays the same across iterations. If you're improving an existing skill, use your judgment on what makes sense as the baseline: the original version the user came in with, or the previous iteration.\n3. Launch the reviewer with `--previous-workspace` pointing at the previous iteration\n4. Wait for the user to review and tell you they're done\n5. Read the new feedback, improve again, repeat\n\nKeep going until:\n\n- The user says they're happy\n- The feedback is all empty (everything looks good)\n- You're not making meaningful progress\n\n---\n\n## Advanced: Blind comparison\n\nFor situations where you want a more rigorous comparison between two versions of a skill (e.g., the user asks \"is the new version actually better?\"), there's a blind comparison system. Read `agents/comparator.md` and `agents/analyzer.md` for the details. The basic idea is: give two outputs to an independent agent without telling it which is which, and let it judge quality. Then analyze why the winner won.\n\nThis is optional, requires subagents, and most users won't need it. The human review loop is usually sufficient.\n\n---\n\n## Description Optimization\n\nThe description field in SKILL.md frontmatter is the primary mechanism that determines whether Claude invokes a skill. After creating or improving a skill, offer to optimize the description for better triggering accuracy.\n\n### Step 1: Generate trigger eval queries\n\nCreate 20 eval queries — a mix of should-trigger and should-not-trigger. Save as JSON:\n\n```json\n[\n { \"query\": \"the user prompt\", \"should_trigger\": true },\n { \"query\": \"another prompt\", \"should_trigger\": false }\n]\n```\n\nThe queries must be realistic and something a Claude Code or Claude.ai user would actually type. Not abstract requests, but requests that are concrete and specific and have a good amount of detail. For instance, file paths, personal context about the user's job or situation, column names and values, company names, URLs. A little bit of backstory. Some might be in lowercase or contain abbreviations or typos or casual speech. Use a mix of different lengths, and focus on edge cases rather than making them clear-cut (the user will get a chance to sign off on them).\n\nBad: `\"Format this data\"`, `\"Extract text from PDF\"`, `\"Create a chart\"`\n\nGood: `\"ok so my boss just sent me this xlsx file (its in my downloads, called something like 'Q4 sales final FINAL v2.xlsx') and she wants me to add a column that shows the profit margin as a percentage. The revenue is in column C and costs are in column D i think\"`\n\nFor the **should-trigger** queries (8-10), think about coverage. You want different phrasings of the same intent — some formal, some casual. Include cases where the user doesn't explicitly name the skill or file type but clearly needs it. Throw in some uncommon use cases and cases where this skill competes with another but should win.\n\nFor the **should-not-trigger** queries (8-10), the most valuable ones are the near-misses — queries that share keywords or concepts with the skill but actually need something different. Think adjacent domains, ambiguous phrasing where a naive keyword match would trigger but shouldn't, and cases where the query touches on something the skill does but in a context where another tool is more appropriate.\n\nThe key thing to avoid: don't make should-not-trigger queries obviously irrelevant. \"Write a fibonacci function\" as a negative test for a PDF skill is too easy — it doesn't test anything. The negative cases should be genuinely tricky.\n\n### Step 2: Review with user\n\nPresent the eval set to the user for review using the HTML template:\n\n1. Read the template from `assets/eval_review.html`\n2. Replace the placeholders:\n - `__EVAL_DATA_PLACEHOLDER__` → the JSON array of eval items (no quotes around it — it's a JS variable assignment)\n - `__SKILL_NAME_PLACEHOLDER__` → the skill's name\n - `__SKILL_DESCRIPTION_PLACEHOLDER__` → the skill's current description\n3. Write to a temp file (e.g., `/tmp/eval_review_.html`) and open it: `open /tmp/eval_review_.html`\n4. The user can edit queries, toggle should-trigger, add/remove entries, then click \"Export Eval Set\"\n5. The file downloads to `~/Downloads/eval_set.json` — check the Downloads folder for the most recent version in case there are multiple (e.g., `eval_set (1).json`)\n\nThis step matters — bad eval queries lead to bad descriptions.\n\n### Step 3: Run the optimization loop\n\nTell the user: \"This will take some time — I'll run the optimization loop in the background and check on it periodically.\"\n\nSave the eval set to the workspace, then run in the background:\n\n```bash\npython -m scripts.run_loop \\\n --eval-set \\\n --skill-path \\\n --model \\\n --max-iterations 5 \\\n --verbose\n```\n\nUse the model ID from your system prompt (the one powering the current session) so the triggering test matches what the user actually experiences.\n\nWhile it runs, periodically tail the output to give the user updates on which iteration it's on and what the scores look like.\n\nThis handles the full optimization loop automatically. It splits the eval set into 60% train and 40% held-out test, evaluates the current description (running each query 3 times to get a reliable trigger rate), then calls Claude to propose improvements based on what failed. It re-evaluates each new description on both train and test, iterating up to 5 times. When it's done, it opens an HTML report in the browser showing the results per iteration and returns JSON with `best_description` — selected by test score rather than train score to avoid overfitting.\n\n### How skill triggering works\n\nUnderstanding the triggering mechanism helps design better eval queries. Skills appear in Claude's `available_skills` list with their name + description, and Claude decides whether to consult a skill based on that description. The important thing to know is that Claude only consults skills for tasks it can't easily handle on its own — simple, one-step queries like \"read this PDF\" may not trigger a skill even if the description matches perfectly, because Claude can handle them directly with basic tools. Complex, multi-step, or specialized queries reliably trigger skills when the description matches.\n\nThis means your eval queries should be substantive enough that Claude would actually benefit from consulting a skill. Simple queries like \"read file X\" are poor test cases — they won't trigger skills regardless of description quality.\n\n### Step 4: Apply the result\n\nTake `best_description` from the JSON output and update the skill's SKILL.md frontmatter. Show the user before/after and report the scores.\n\n---\n\n### Package and Present (only if `present_files` tool is available)\n\nCheck whether you have access to the `present_files` tool. If you don't, skip this step. If you do, package the skill and present the .skill file to the user:\n\n```bash\npython -m scripts.package_skill \n```\n\nAfter packaging, direct the user to the resulting `.skill` file path so they can install it.\n\n---\n\n## Claude.ai-specific instructions\n\nIn Claude.ai, the core workflow is the same (draft → test → review → improve → repeat), but because Claude.ai doesn't have subagents, some mechanics change. Here's what to adapt:\n\n**Running test cases**: No subagents means no parallel execution. For each test case, read the skill's SKILL.md, then follow its instructions to accomplish the test prompt yourself. Do them one at a time. This is less rigorous than independent subagents (you wrote the skill and you're also running it, so you have full context), but it's a useful sanity check — and the human review step compensates. Skip the baseline runs — just use the skill to complete the task as requested.\n\n**Reviewing results**: If you can't open a browser (e.g., Claude.ai's VM has no display, or you're on a remote server), skip the browser reviewer entirely. Instead, present results directly in the conversation. For each test case, show the prompt and the output. If the output is a file the user needs to see (like a .docx or .xlsx), save it to the filesystem and tell them where it is so they can download and inspect it. Ask for feedback inline: \"How does this look? Anything you'd change?\"\n\n**Benchmarking**: Skip the quantitative benchmarking — it relies on baseline comparisons which aren't meaningful without subagents. Focus on qualitative feedback from the user.\n\n**The iteration loop**: Same as before — improve the skill, rerun the test cases, ask for feedback — just without the browser reviewer in the middle. You can still organize results into iteration directories on the filesystem if you have one.\n\n**Description optimization**: This section requires the `claude` CLI tool (specifically `claude -p`) which is only available in Claude Code. Skip it if you're on Claude.ai.\n\n**Blind comparison**: Requires subagents. Skip it.\n\n**Packaging**: The `package_skill.py` script works anywhere with Python and a filesystem. On Claude.ai, you can run it and the user can download the resulting `.skill` file.\n\n**Updating an existing skill**: The user might be asking you to update an existing skill, not create a new one. In this case:\n\n- **Preserve the original name.** Note the skill's directory name and `name` frontmatter field -- use them unchanged. E.g., if the installed skill is `research-helper`, output `research-helper.skill` (not `research-helper-v2`).\n- **Copy to a writeable location before editing.** The installed skill path may be read-only. Copy to `/tmp/skill-name/`, edit there, and package from the copy.\n- **If packaging manually, stage in `/tmp/` first**, then copy to the output directory -- direct writes may fail due to permissions.\n\n---\n\n## Cowork-Specific Instructions\n\nIf you're in Cowork, the main things to know are:\n\n- You have subagents, so the main workflow (spawn test cases in parallel, run baselines, grade, etc.) all works. (However, if you run into severe problems with timeouts, it's OK to run the test prompts in series rather than parallel.)\n- You don't have a browser or display, so when generating the eval viewer, use `--static ` to write a standalone HTML file instead of starting a server. Then proffer a link that the user can click to open the HTML in their browser.\n- For whatever reason, the Cowork setup seems to disincline Claude from generating the eval viewer after running the tests, so just to reiterate: whether you're in Cowork or in Claude Code, after running tests, you should always generate the eval viewer for the human to look at examples before revising the skill yourself and trying to make corrections, using `generate_review.py` (not writing your own boutique html code). Sorry in advance but I'm gonna go all caps here: GENERATE THE EVAL VIEWER _BEFORE_ evaluating inputs yourself. You want to get them in front of the human ASAP!\n- Feedback works differently: since there's no running server, the viewer's \"Submit All Reviews\" button will download `feedback.json` as a file. You can then read it from there (you may have to request access first).\n- Packaging works — `package_skill.py` just needs Python and a filesystem.\n- Description optimization (`run_loop.py` / `run_eval.py`) should work in Cowork just fine since it uses `claude -p` via subprocess, not a browser, but please save it until you've fully finished making the skill and the user agrees it's in good shape.\n- **Updating an existing skill**: The user might be asking you to update an existing skill, not create a new one. Follow the update guidance in the claude.ai section above.\n\n---\n\n## Reference files\n\nThe agents/ directory contains instructions for specialized subagents. Read them when you need to spawn the relevant subagent.\n\n- `agents/grader.md` — How to evaluate assertions against outputs\n- `agents/comparator.md` — How to do blind A/B comparison between two outputs\n- `agents/analyzer.md` — How to analyze why one version beat another\n\nThe references/ directory has additional documentation:\n\n- `references/schemas.md` — JSON structures for evals.json, grading.json, etc.\n\n---\n\nRepeating one more time the core loop here for emphasis:\n\n- Figure out what the skill is about\n- Draft or edit the skill\n- Run claude-with-access-to-the-skill on test prompts\n- With the user, evaluate the outputs:\n - Create benchmark.json and run `eval-viewer/generate_review.py` to help the user review them\n - Run quantitative evals\n- Repeat until you and the user are satisfied\n- Package the final skill and return it to the user.\n\nPlease add steps to your TodoList, if you have such a thing, to make sure you don't forget. If you're in Cowork, please specifically put \"Create evals JSON and run `eval-viewer/generate_review.py` so human can review test cases\" in your TodoList to make sure it happens.\n\nGood luck!\n" }, { "path": ".agents/skills/skill-creator/agents/analyzer.md", "content": "# Post-hoc Analyzer Agent\n\nAnalyze blind comparison results to understand WHY the winner won and generate improvement suggestions.\n\n## Role\n\nAfter the blind comparator determines a winner, the Post-hoc Analyzer \"unblids\" the results by examining the skills and transcripts. The goal is to extract actionable insights: what made the winner better, and how can the loser be improved?\n\n## Inputs\n\nYou receive these parameters in your prompt:\n\n- **winner**: \"A\" or \"B\" (from blind comparison)\n- **winner_skill_path**: Path to the skill that produced the winning output\n- **winner_transcript_path**: Path to the execution transcript for the winner\n- **loser_skill_path**: Path to the skill that produced the losing output\n- **loser_transcript_path**: Path to the execution transcript for the loser\n- **comparison_result_path**: Path to the blind comparator's output JSON\n- **output_path**: Where to save the analysis results\n\n## Process\n\n### Step 1: Read Comparison Result\n\n1. Read the blind comparator's output at comparison_result_path\n2. Note the winning side (A or B), the reasoning, and any scores\n3. Understand what the comparator valued in the winning output\n\n### Step 2: Read Both Skills\n\n1. Read the winner skill's SKILL.md and key referenced files\n2. Read the loser skill's SKILL.md and key referenced files\n3. Identify structural differences:\n - Instructions clarity and specificity\n - Script/tool usage patterns\n - Example coverage\n - Edge case handling\n\n### Step 3: Read Both Transcripts\n\n1. Read the winner's transcript\n2. Read the loser's transcript\n3. Compare execution patterns:\n - How closely did each follow their skill's instructions?\n - What tools were used differently?\n - Where did the loser diverge from optimal behavior?\n - Did either encounter errors or make recovery attempts?\n\n### Step 4: Analyze Instruction Following\n\nFor each transcript, evaluate:\n\n- Did the agent follow the skill's explicit instructions?\n- Did the agent use the skill's provided tools/scripts?\n- Were there missed opportunities to leverage skill content?\n- Did the agent add unnecessary steps not in the skill?\n\nScore instruction following 1-10 and note specific issues.\n\n### Step 5: Identify Winner Strengths\n\nDetermine what made the winner better:\n\n- Clearer instructions that led to better behavior?\n- Better scripts/tools that produced better output?\n- More comprehensive examples that guided edge cases?\n- Better error handling guidance?\n\nBe specific. Quote from skills/transcripts where relevant.\n\n### Step 6: Identify Loser Weaknesses\n\nDetermine what held the loser back:\n\n- Ambiguous instructions that led to suboptimal choices?\n- Missing tools/scripts that forced workarounds?\n- Gaps in edge case coverage?\n- Poor error handling that caused failures?\n\n### Step 7: Generate Improvement Suggestions\n\nBased on the analysis, produce actionable suggestions for improving the loser skill:\n\n- Specific instruction changes to make\n- Tools/scripts to add or modify\n- Examples to include\n- Edge cases to address\n\nPrioritize by impact. Focus on changes that would have changed the outcome.\n\n### Step 8: Write Analysis Results\n\nSave structured analysis to `{output_path}`.\n\n## Output Format\n\nWrite a JSON file with this structure:\n\n```json\n{\n \"comparison_summary\": {\n \"winner\": \"A\",\n \"winner_skill\": \"path/to/winner/skill\",\n \"loser_skill\": \"path/to/loser/skill\",\n \"comparator_reasoning\": \"Brief summary of why comparator chose winner\"\n },\n \"winner_strengths\": [\n \"Clear step-by-step instructions for handling multi-page documents\",\n \"Included validation script that caught formatting errors\",\n \"Explicit guidance on fallback behavior when OCR fails\"\n ],\n \"loser_weaknesses\": [\n \"Vague instruction 'process the document appropriately' led to inconsistent behavior\",\n \"No script for validation, agent had to improvise and made errors\",\n \"No guidance on OCR failure, agent gave up instead of trying alternatives\"\n ],\n \"instruction_following\": {\n \"winner\": {\n \"score\": 9,\n \"issues\": [\"Minor: skipped optional logging step\"]\n },\n \"loser\": {\n \"score\": 6,\n \"issues\": [\n \"Did not use the skill's formatting template\",\n \"Invented own approach instead of following step 3\",\n \"Missed the 'always validate output' instruction\"\n ]\n }\n },\n \"improvement_suggestions\": [\n {\n \"priority\": \"high\",\n \"category\": \"instructions\",\n \"suggestion\": \"Replace 'process the document appropriately' with explicit steps: 1) Extract text, 2) Identify sections, 3) Format per template\",\n \"expected_impact\": \"Would eliminate ambiguity that caused inconsistent behavior\"\n },\n {\n \"priority\": \"high\",\n \"category\": \"tools\",\n \"suggestion\": \"Add validate_output.py script similar to winner skill's validation approach\",\n \"expected_impact\": \"Would catch formatting errors before final output\"\n },\n {\n \"priority\": \"medium\",\n \"category\": \"error_handling\",\n \"suggestion\": \"Add fallback instructions: 'If OCR fails, try: 1) different resolution, 2) image preprocessing, 3) manual extraction'\",\n \"expected_impact\": \"Would prevent early failure on difficult documents\"\n }\n ],\n \"transcript_insights\": {\n \"winner_execution_pattern\": \"Read skill -> Followed 5-step process -> Used validation script -> Fixed 2 issues -> Produced output\",\n \"loser_execution_pattern\": \"Read skill -> Unclear on approach -> Tried 3 different methods -> No validation -> Output had errors\"\n }\n}\n```\n\n## Guidelines\n\n- **Be specific**: Quote from skills and transcripts, don't just say \"instructions were unclear\"\n- **Be actionable**: Suggestions should be concrete changes, not vague advice\n- **Focus on skill improvements**: The goal is to improve the losing skill, not critique the agent\n- **Prioritize by impact**: Which changes would most likely have changed the outcome?\n- **Consider causation**: Did the skill weakness actually cause the worse output, or is it incidental?\n- **Stay objective**: Analyze what happened, don't editorialize\n- **Think about generalization**: Would this improvement help on other evals too?\n\n## Categories for Suggestions\n\nUse these categories to organize improvement suggestions:\n\n| Category | Description |\n| ---------------- | ---------------------------------------------- |\n| `instructions` | Changes to the skill's prose instructions |\n| `tools` | Scripts, templates, or utilities to add/modify |\n| `examples` | Example inputs/outputs to include |\n| `error_handling` | Guidance for handling failures |\n| `structure` | Reorganization of skill content |\n| `references` | External docs or resources to add |\n\n## Priority Levels\n\n- **high**: Would likely change the outcome of this comparison\n- **medium**: Would improve quality but may not change win/loss\n- **low**: Nice to have, marginal improvement\n\n---\n\n# Analyzing Benchmark Results\n\nWhen analyzing benchmark results, the analyzer's purpose is to **surface patterns and anomalies** across multiple runs, not suggest skill improvements.\n\n## Role\n\nReview all benchmark run results and generate freeform notes that help the user understand skill performance. Focus on patterns that wouldn't be visible from aggregate metrics alone.\n\n## Inputs\n\nYou receive these parameters in your prompt:\n\n- **benchmark_data_path**: Path to the in-progress benchmark.json with all run results\n- **skill_path**: Path to the skill being benchmarked\n- **output_path**: Where to save the notes (as JSON array of strings)\n\n## Process\n\n### Step 1: Read Benchmark Data\n\n1. Read the benchmark.json containing all run results\n2. Note the configurations tested (with_skill, without_skill)\n3. Understand the run_summary aggregates already calculated\n\n### Step 2: Analyze Per-Assertion Patterns\n\nFor each expectation across all runs:\n\n- Does it **always pass** in both configurations? (may not differentiate skill value)\n- Does it **always fail** in both configurations? (may be broken or beyond capability)\n- Does it **always pass with skill but fail without**? (skill clearly adds value here)\n- Does it **always fail with skill but pass without**? (skill may be hurting)\n- Is it **highly variable**? (flaky expectation or non-deterministic behavior)\n\n### Step 3: Analyze Cross-Eval Patterns\n\nLook for patterns across evals:\n\n- Are certain eval types consistently harder/easier?\n- Do some evals show high variance while others are stable?\n- Are there surprising results that contradict expectations?\n\n### Step 4: Analyze Metrics Patterns\n\nLook at time_seconds, tokens, tool_calls:\n\n- Does the skill significantly increase execution time?\n- Is there high variance in resource usage?\n- Are there outlier runs that skew the aggregates?\n\n### Step 5: Generate Notes\n\nWrite freeform observations as a list of strings. Each note should:\n\n- State a specific observation\n- Be grounded in the data (not speculation)\n- Help the user understand something the aggregate metrics don't show\n\nExamples:\n\n- \"Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value\"\n- \"Eval 3 shows high variance (50% ± 40%) - run 2 had an unusual failure that may be flaky\"\n- \"Without-skill runs consistently fail on table extraction expectations (0% pass rate)\"\n- \"Skill adds 13s average execution time but improves pass rate by 50%\"\n- \"Token usage is 80% higher with skill, primarily due to script output parsing\"\n- \"All 3 without-skill runs for eval 1 produced empty output\"\n\n### Step 6: Write Notes\n\nSave notes to `{output_path}` as a JSON array of strings:\n\n```json\n[\n \"Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value\",\n \"Eval 3 shows high variance (50% ± 40%) - run 2 had an unusual failure\",\n \"Without-skill runs consistently fail on table extraction expectations\",\n \"Skill adds 13s average execution time but improves pass rate by 50%\"\n]\n```\n\n## Guidelines\n\n**DO:**\n\n- Report what you observe in the data\n- Be specific about which evals, expectations, or runs you're referring to\n- Note patterns that aggregate metrics would hide\n- Provide context that helps interpret the numbers\n\n**DO NOT:**\n\n- Suggest improvements to the skill (that's for the improvement step, not benchmarking)\n- Make subjective quality judgments (\"the output was good/bad\")\n- Speculate about causes without evidence\n- Repeat information already in the run_summary aggregates\n" }, { "path": ".agents/skills/skill-creator/agents/comparator.md", "content": "# Blind Comparator Agent\n\nCompare two outputs WITHOUT knowing which skill produced them.\n\n## Role\n\nThe Blind Comparator judges which output better accomplishes the eval task. You receive two outputs labeled A and B, but you do NOT know which skill produced which. This prevents bias toward a particular skill or approach.\n\nYour judgment is based purely on output quality and task completion.\n\n## Inputs\n\nYou receive these parameters in your prompt:\n\n- **output_a_path**: Path to the first output file or directory\n- **output_b_path**: Path to the second output file or directory\n- **eval_prompt**: The original task/prompt that was executed\n- **expectations**: List of expectations to check (optional - may be empty)\n\n## Process\n\n### Step 1: Read Both Outputs\n\n1. Examine output A (file or directory)\n2. Examine output B (file or directory)\n3. Note the type, structure, and content of each\n4. If outputs are directories, examine all relevant files inside\n\n### Step 2: Understand the Task\n\n1. Read the eval_prompt carefully\n2. Identify what the task requires:\n - What should be produced?\n - What qualities matter (accuracy, completeness, format)?\n - What would distinguish a good output from a poor one?\n\n### Step 3: Generate Evaluation Rubric\n\nBased on the task, generate a rubric with two dimensions:\n\n**Content Rubric** (what the output contains):\n| Criterion | 1 (Poor) | 3 (Acceptable) | 5 (Excellent) |\n|-----------|----------|----------------|---------------|\n| Correctness | Major errors | Minor errors | Fully correct |\n| Completeness | Missing key elements | Mostly complete | All elements present |\n| Accuracy | Significant inaccuracies | Minor inaccuracies | Accurate throughout |\n\n**Structure Rubric** (how the output is organized):\n| Criterion | 1 (Poor) | 3 (Acceptable) | 5 (Excellent) |\n|-----------|----------|----------------|---------------|\n| Organization | Disorganized | Reasonably organized | Clear, logical structure |\n| Formatting | Inconsistent/broken | Mostly consistent | Professional, polished |\n| Usability | Difficult to use | Usable with effort | Easy to use |\n\nAdapt criteria to the specific task. For example:\n\n- PDF form → \"Field alignment\", \"Text readability\", \"Data placement\"\n- Document → \"Section structure\", \"Heading hierarchy\", \"Paragraph flow\"\n- Data output → \"Schema correctness\", \"Data types\", \"Completeness\"\n\n### Step 4: Evaluate Each Output Against the Rubric\n\nFor each output (A and B):\n\n1. **Score each criterion** on the rubric (1-5 scale)\n2. **Calculate dimension totals**: Content score, Structure score\n3. **Calculate overall score**: Average of dimension scores, scaled to 1-10\n\n### Step 5: Check Assertions (if provided)\n\nIf expectations are provided:\n\n1. Check each expectation against output A\n2. Check each expectation against output B\n3. Count pass rates for each output\n4. Use expectation scores as secondary evidence (not the primary decision factor)\n\n### Step 6: Determine the Winner\n\nCompare A and B based on (in priority order):\n\n1. **Primary**: Overall rubric score (content + structure)\n2. **Secondary**: Assertion pass rates (if applicable)\n3. **Tiebreaker**: If truly equal, declare a TIE\n\nBe decisive - ties should be rare. One output is usually better, even if marginally.\n\n### Step 7: Write Comparison Results\n\nSave results to a JSON file at the path specified (or `comparison.json` if not specified).\n\n## Output Format\n\nWrite a JSON file with this structure:\n\n```json\n{\n \"winner\": \"A\",\n \"reasoning\": \"Output A provides a complete solution with proper formatting and all required fields. Output B is missing the date field and has formatting inconsistencies.\",\n \"rubric\": {\n \"A\": {\n \"content\": {\n \"correctness\": 5,\n \"completeness\": 5,\n \"accuracy\": 4\n },\n \"structure\": {\n \"organization\": 4,\n \"formatting\": 5,\n \"usability\": 4\n },\n \"content_score\": 4.7,\n \"structure_score\": 4.3,\n \"overall_score\": 9.0\n },\n \"B\": {\n \"content\": {\n \"correctness\": 3,\n \"completeness\": 2,\n \"accuracy\": 3\n },\n \"structure\": {\n \"organization\": 3,\n \"formatting\": 2,\n \"usability\": 3\n },\n \"content_score\": 2.7,\n \"structure_score\": 2.7,\n \"overall_score\": 5.4\n }\n },\n \"output_quality\": {\n \"A\": {\n \"score\": 9,\n \"strengths\": [\"Complete solution\", \"Well-formatted\", \"All fields present\"],\n \"weaknesses\": [\"Minor style inconsistency in header\"]\n },\n \"B\": {\n \"score\": 5,\n \"strengths\": [\"Readable output\", \"Correct basic structure\"],\n \"weaknesses\": [\"Missing date field\", \"Formatting inconsistencies\", \"Partial data extraction\"]\n }\n },\n \"expectation_results\": {\n \"A\": {\n \"passed\": 4,\n \"total\": 5,\n \"pass_rate\": 0.8,\n \"details\": [\n { \"text\": \"Output includes name\", \"passed\": true },\n { \"text\": \"Output includes date\", \"passed\": true },\n { \"text\": \"Format is PDF\", \"passed\": true },\n { \"text\": \"Contains signature\", \"passed\": false },\n { \"text\": \"Readable text\", \"passed\": true }\n ]\n },\n \"B\": {\n \"passed\": 3,\n \"total\": 5,\n \"pass_rate\": 0.6,\n \"details\": [\n { \"text\": \"Output includes name\", \"passed\": true },\n { \"text\": \"Output includes date\", \"passed\": false },\n { \"text\": \"Format is PDF\", \"passed\": true },\n { \"text\": \"Contains signature\", \"passed\": false },\n { \"text\": \"Readable text\", \"passed\": true }\n ]\n }\n }\n}\n```\n\nIf no expectations were provided, omit the `expectation_results` field entirely.\n\n## Field Descriptions\n\n- **winner**: \"A\", \"B\", or \"TIE\"\n- **reasoning**: Clear explanation of why the winner was chosen (or why it's a tie)\n- **rubric**: Structured rubric evaluation for each output\n - **content**: Scores for content criteria (correctness, completeness, accuracy)\n - **structure**: Scores for structure criteria (organization, formatting, usability)\n - **content_score**: Average of content criteria (1-5)\n - **structure_score**: Average of structure criteria (1-5)\n - **overall_score**: Combined score scaled to 1-10\n- **output_quality**: Summary quality assessment\n - **score**: 1-10 rating (should match rubric overall_score)\n - **strengths**: List of positive aspects\n - **weaknesses**: List of issues or shortcomings\n- **expectation_results**: (Only if expectations provided)\n - **passed**: Number of expectations that passed\n - **total**: Total number of expectations\n - **pass_rate**: Fraction passed (0.0 to 1.0)\n - **details**: Individual expectation results\n\n## Guidelines\n\n- **Stay blind**: DO NOT try to infer which skill produced which output. Judge purely on output quality.\n- **Be specific**: Cite specific examples when explaining strengths and weaknesses.\n- **Be decisive**: Choose a winner unless outputs are genuinely equivalent.\n- **Output quality first**: Assertion scores are secondary to overall task completion.\n- **Be objective**: Don't favor outputs based on style preferences; focus on correctness and completeness.\n- **Explain your reasoning**: The reasoning field should make it clear why you chose the winner.\n- **Handle edge cases**: If both outputs fail, pick the one that fails less badly. If both are excellent, pick the one that's marginally better.\n" }, { "path": ".agents/skills/skill-creator/agents/grader.md", "content": "# Grader Agent\n\nEvaluate expectations against an execution transcript and outputs.\n\n## Role\n\nThe Grader reviews a transcript and output files, then determines whether each expectation passes or fails. Provide clear evidence for each judgment.\n\nYou have two jobs: grade the outputs, and critique the evals themselves. A passing grade on a weak assertion is worse than useless — it creates false confidence. When you notice an assertion that's trivially satisfied, or an important outcome that no assertion checks, say so.\n\n## Inputs\n\nYou receive these parameters in your prompt:\n\n- **expectations**: List of expectations to evaluate (strings)\n- **transcript_path**: Path to the execution transcript (markdown file)\n- **outputs_dir**: Directory containing output files from execution\n\n## Process\n\n### Step 1: Read the Transcript\n\n1. Read the transcript file completely\n2. Note the eval prompt, execution steps, and final result\n3. Identify any issues or errors documented\n\n### Step 2: Examine Output Files\n\n1. List files in outputs_dir\n2. Read/examine each file relevant to the expectations. If outputs aren't plain text, use the inspection tools provided in your prompt — don't rely solely on what the transcript says the executor produced.\n3. Note contents, structure, and quality\n\n### Step 3: Evaluate Each Assertion\n\nFor each expectation:\n\n1. **Search for evidence** in the transcript and outputs\n2. **Determine verdict**:\n - **PASS**: Clear evidence the expectation is true AND the evidence reflects genuine task completion, not just surface-level compliance\n - **FAIL**: No evidence, or evidence contradicts the expectation, or the evidence is superficial (e.g., correct filename but empty/wrong content)\n3. **Cite the evidence**: Quote the specific text or describe what you found\n\n### Step 4: Extract and Verify Claims\n\nBeyond the predefined expectations, extract implicit claims from the outputs and verify them:\n\n1. **Extract claims** from the transcript and outputs:\n - Factual statements (\"The form has 12 fields\")\n - Process claims (\"Used pypdf to fill the form\")\n - Quality claims (\"All fields were filled correctly\")\n\n2. **Verify each claim**:\n - **Factual claims**: Can be checked against the outputs or external sources\n - **Process claims**: Can be verified from the transcript\n - **Quality claims**: Evaluate whether the claim is justified\n\n3. **Flag unverifiable claims**: Note claims that cannot be verified with available information\n\nThis catches issues that predefined expectations might miss.\n\n### Step 5: Read User Notes\n\nIf `{outputs_dir}/user_notes.md` exists:\n\n1. Read it and note any uncertainties or issues flagged by the executor\n2. Include relevant concerns in the grading output\n3. These may reveal problems even when expectations pass\n\n### Step 6: Critique the Evals\n\nAfter grading, consider whether the evals themselves could be improved. Only surface suggestions when there's a clear gap.\n\nGood suggestions test meaningful outcomes — assertions that are hard to satisfy without actually doing the work correctly. Think about what makes an assertion _discriminating_: it passes when the skill genuinely succeeds and fails when it doesn't.\n\nSuggestions worth raising:\n\n- An assertion that passed but would also pass for a clearly wrong output (e.g., checking filename existence but not file content)\n- An important outcome you observed — good or bad — that no assertion covers at all\n- An assertion that can't actually be verified from the available outputs\n\nKeep the bar high. The goal is to flag things the eval author would say \"good catch\" about, not to nitpick every assertion.\n\n### Step 7: Write Grading Results\n\nSave results to `{outputs_dir}/../grading.json` (sibling to outputs_dir).\n\n## Grading Criteria\n\n**PASS when**:\n\n- The transcript or outputs clearly demonstrate the expectation is true\n- Specific evidence can be cited\n- The evidence reflects genuine substance, not just surface compliance (e.g., a file exists AND contains correct content, not just the right filename)\n\n**FAIL when**:\n\n- No evidence found for the expectation\n- Evidence contradicts the expectation\n- The expectation cannot be verified from available information\n- The evidence is superficial — the assertion is technically satisfied but the underlying task outcome is wrong or incomplete\n- The output appears to meet the assertion by coincidence rather than by actually doing the work\n\n**When uncertain**: The burden of proof to pass is on the expectation.\n\n### Step 8: Read Executor Metrics and Timing\n\n1. If `{outputs_dir}/metrics.json` exists, read it and include in grading output\n2. If `{outputs_dir}/../timing.json` exists, read it and include timing data\n\n## Output Format\n\nWrite a JSON file with this structure:\n\n```json\n{\n \"expectations\": [\n {\n \"text\": \"The output includes the name 'John Smith'\",\n \"passed\": true,\n \"evidence\": \"Found in transcript Step 3: 'Extracted names: John Smith, Sarah Johnson'\"\n },\n {\n \"text\": \"The spreadsheet has a SUM formula in cell B10\",\n \"passed\": false,\n \"evidence\": \"No spreadsheet was created. The output was a text file.\"\n },\n {\n \"text\": \"The assistant used the skill's OCR script\",\n \"passed\": true,\n \"evidence\": \"Transcript Step 2 shows: 'Tool: Bash - python ocr_script.py image.png'\"\n }\n ],\n \"summary\": {\n \"passed\": 2,\n \"failed\": 1,\n \"total\": 3,\n \"pass_rate\": 0.67\n },\n \"execution_metrics\": {\n \"tool_calls\": {\n \"Read\": 5,\n \"Write\": 2,\n \"Bash\": 8\n },\n \"total_tool_calls\": 15,\n \"total_steps\": 6,\n \"errors_encountered\": 0,\n \"output_chars\": 12450,\n \"transcript_chars\": 3200\n },\n \"timing\": {\n \"executor_duration_seconds\": 165.0,\n \"grader_duration_seconds\": 26.0,\n \"total_duration_seconds\": 191.0\n },\n \"claims\": [\n {\n \"claim\": \"The form has 12 fillable fields\",\n \"type\": \"factual\",\n \"verified\": true,\n \"evidence\": \"Counted 12 fields in field_info.json\"\n },\n {\n \"claim\": \"All required fields were populated\",\n \"type\": \"quality\",\n \"verified\": false,\n \"evidence\": \"Reference section was left blank despite data being available\"\n }\n ],\n \"user_notes_summary\": {\n \"uncertainties\": [\"Used 2023 data, may be stale\"],\n \"needs_review\": [],\n \"workarounds\": [\"Fell back to text overlay for non-fillable fields\"]\n },\n \"eval_feedback\": {\n \"suggestions\": [\n {\n \"assertion\": \"The output includes the name 'John Smith'\",\n \"reason\": \"A hallucinated document that mentions the name would also pass — consider checking it appears as the primary contact with matching phone and email from the input\"\n },\n {\n \"reason\": \"No assertion checks whether the extracted phone numbers match the input — I observed incorrect numbers in the output that went uncaught\"\n }\n ],\n \"overall\": \"Assertions check presence but not correctness. Consider adding content verification.\"\n }\n}\n```\n\n## Field Descriptions\n\n- **expectations**: Array of graded expectations\n - **text**: The original expectation text\n - **passed**: Boolean - true if expectation passes\n - **evidence**: Specific quote or description supporting the verdict\n- **summary**: Aggregate statistics\n - **passed**: Count of passed expectations\n - **failed**: Count of failed expectations\n - **total**: Total expectations evaluated\n - **pass_rate**: Fraction passed (0.0 to 1.0)\n- **execution_metrics**: Copied from executor's metrics.json (if available)\n - **output_chars**: Total character count of output files (proxy for tokens)\n - **transcript_chars**: Character count of transcript\n- **timing**: Wall clock timing from timing.json (if available)\n - **executor_duration_seconds**: Time spent in executor subagent\n - **total_duration_seconds**: Total elapsed time for the run\n- **claims**: Extracted and verified claims from the output\n - **claim**: The statement being verified\n - **type**: \"factual\", \"process\", or \"quality\"\n - **verified**: Boolean - whether the claim holds\n - **evidence**: Supporting or contradicting evidence\n- **user_notes_summary**: Issues flagged by the executor\n - **uncertainties**: Things the executor wasn't sure about\n - **needs_review**: Items requiring human attention\n - **workarounds**: Places where the skill didn't work as expected\n- **eval_feedback**: Improvement suggestions for the evals (only when warranted)\n - **suggestions**: List of concrete suggestions, each with a `reason` and optionally an `assertion` it relates to\n - **overall**: Brief assessment — can be \"No suggestions, evals look solid\" if nothing to flag\n\n## Guidelines\n\n- **Be objective**: Base verdicts on evidence, not assumptions\n- **Be specific**: Quote the exact text that supports your verdict\n- **Be thorough**: Check both transcript and output files\n- **Be consistent**: Apply the same standard to each expectation\n- **Explain failures**: Make it clear why evidence was insufficient\n- **No partial credit**: Each expectation is pass or fail, not partial\n" }, { "path": ".agents/skills/skill-creator/assets/eval_review.html", "content": "\n\n \n \n \n Eval Set Review - __SKILL_NAME_PLACEHOLDER__\n \n \n \n \n \n \n

Eval Set Review: __SKILL_NAME_PLACEHOLDER__

\n

\n Current description: __SKILL_DESCRIPTION_PLACEHOLDER__\n

\n\n
\n \n \n
\n\n \n \n \n \n \n \n \n \n \n
QueryShould TriggerActions
\n\n

\n\n \n \n\n" }, { "path": ".agents/skills/skill-creator/eval-viewer/generate_review.py", "content": "#!/usr/bin/env python3\n\"\"\"Generate and serve a review page for eval results.\n\nReads the workspace directory, discovers runs (directories with outputs/),\nembeds all output data into a self-contained HTML page, and serves it via\na tiny HTTP server. Feedback auto-saves to feedback.json in the workspace.\n\nUsage:\n python generate_review.py [--port PORT] [--skill-name NAME]\n python generate_review.py --previous-feedback /path/to/old/feedback.json\n\nNo dependencies beyond the Python stdlib are required.\n\"\"\"\n\nimport argparse\nimport base64\nimport json\nimport mimetypes\nimport os\nimport re\nimport signal\nimport subprocess\nimport sys\nimport time\nimport webbrowser\nfrom functools import partial\nfrom http.server import HTTPServer, BaseHTTPRequestHandler\nfrom pathlib import Path\n\n# Files to exclude from output listings\nMETADATA_FILES = {\"transcript.md\", \"user_notes.md\", \"metrics.json\"}\n\n# Extensions we render as inline text\nTEXT_EXTENSIONS = {\n \".txt\", \".md\", \".json\", \".csv\", \".py\", \".js\", \".ts\", \".tsx\", \".jsx\",\n \".yaml\", \".yml\", \".xml\", \".html\", \".css\", \".sh\", \".rb\", \".go\", \".rs\",\n \".java\", \".c\", \".cpp\", \".h\", \".hpp\", \".sql\", \".r\", \".toml\",\n}\n\n# Extensions we render as inline images\nIMAGE_EXTENSIONS = {\".png\", \".jpg\", \".jpeg\", \".gif\", \".svg\", \".webp\"}\n\n# MIME type overrides for common types\nMIME_OVERRIDES = {\n \".svg\": \"image/svg+xml\",\n \".xlsx\": \"application/vnd.openxmlformats-officedocument.spreadsheetml.sheet\",\n \".docx\": \"application/vnd.openxmlformats-officedocument.wordprocessingml.document\",\n \".pptx\": \"application/vnd.openxmlformats-officedocument.presentationml.presentation\",\n}\n\n\ndef get_mime_type(path: Path) -> str:\n ext = path.suffix.lower()\n if ext in MIME_OVERRIDES:\n return MIME_OVERRIDES[ext]\n mime, _ = mimetypes.guess_type(str(path))\n return mime or \"application/octet-stream\"\n\n\ndef find_runs(workspace: Path) -> list[dict]:\n \"\"\"Recursively find directories that contain an outputs/ subdirectory.\"\"\"\n runs: list[dict] = []\n _find_runs_recursive(workspace, workspace, runs)\n runs.sort(key=lambda r: (r.get(\"eval_id\", float(\"inf\")), r[\"id\"]))\n return runs\n\n\ndef _find_runs_recursive(root: Path, current: Path, runs: list[dict]) -> None:\n if not current.is_dir():\n return\n\n outputs_dir = current / \"outputs\"\n if outputs_dir.is_dir():\n run = build_run(root, current)\n if run:\n runs.append(run)\n return\n\n skip = {\"node_modules\", \".git\", \"__pycache__\", \"skill\", \"inputs\"}\n for child in sorted(current.iterdir()):\n if child.is_dir() and child.name not in skip:\n _find_runs_recursive(root, child, runs)\n\n\ndef build_run(root: Path, run_dir: Path) -> dict | None:\n \"\"\"Build a run dict with prompt, outputs, and grading data.\"\"\"\n prompt = \"\"\n eval_id = None\n\n # Try eval_metadata.json\n for candidate in [run_dir / \"eval_metadata.json\", run_dir.parent / \"eval_metadata.json\"]:\n if candidate.exists():\n try:\n metadata = json.loads(candidate.read_text())\n prompt = metadata.get(\"prompt\", \"\")\n eval_id = metadata.get(\"eval_id\")\n except (json.JSONDecodeError, OSError):\n pass\n if prompt:\n break\n\n # Fall back to transcript.md\n if not prompt:\n for candidate in [run_dir / \"transcript.md\", run_dir / \"outputs\" / \"transcript.md\"]:\n if candidate.exists():\n try:\n text = candidate.read_text()\n match = re.search(r\"## Eval Prompt\\n\\n([\\s\\S]*?)(?=\\n##|$)\", text)\n if match:\n prompt = match.group(1).strip()\n except OSError:\n pass\n if prompt:\n break\n\n if not prompt:\n prompt = \"(No prompt found)\"\n\n run_id = str(run_dir.relative_to(root)).replace(\"/\", \"-\").replace(\"\\\\\", \"-\")\n\n # Collect output files\n outputs_dir = run_dir / \"outputs\"\n output_files: list[dict] = []\n if outputs_dir.is_dir():\n for f in sorted(outputs_dir.iterdir()):\n if f.is_file() and f.name not in METADATA_FILES:\n output_files.append(embed_file(f))\n\n # Load grading if present\n grading = None\n for candidate in [run_dir / \"grading.json\", run_dir.parent / \"grading.json\"]:\n if candidate.exists():\n try:\n grading = json.loads(candidate.read_text())\n except (json.JSONDecodeError, OSError):\n pass\n if grading:\n break\n\n return {\n \"id\": run_id,\n \"prompt\": prompt,\n \"eval_id\": eval_id,\n \"outputs\": output_files,\n \"grading\": grading,\n }\n\n\ndef embed_file(path: Path) -> dict:\n \"\"\"Read a file and return an embedded representation.\"\"\"\n ext = path.suffix.lower()\n mime = get_mime_type(path)\n\n if ext in TEXT_EXTENSIONS:\n try:\n content = path.read_text(errors=\"replace\")\n except OSError:\n content = \"(Error reading file)\"\n return {\n \"name\": path.name,\n \"type\": \"text\",\n \"content\": content,\n }\n elif ext in IMAGE_EXTENSIONS:\n try:\n raw = path.read_bytes()\n b64 = base64.b64encode(raw).decode(\"ascii\")\n except OSError:\n return {\"name\": path.name, \"type\": \"error\", \"content\": \"(Error reading file)\"}\n return {\n \"name\": path.name,\n \"type\": \"image\",\n \"mime\": mime,\n \"data_uri\": f\"data:{mime};base64,{b64}\",\n }\n elif ext == \".pdf\":\n try:\n raw = path.read_bytes()\n b64 = base64.b64encode(raw).decode(\"ascii\")\n except OSError:\n return {\"name\": path.name, \"type\": \"error\", \"content\": \"(Error reading file)\"}\n return {\n \"name\": path.name,\n \"type\": \"pdf\",\n \"data_uri\": f\"data:{mime};base64,{b64}\",\n }\n elif ext == \".xlsx\":\n try:\n raw = path.read_bytes()\n b64 = base64.b64encode(raw).decode(\"ascii\")\n except OSError:\n return {\"name\": path.name, \"type\": \"error\", \"content\": \"(Error reading file)\"}\n return {\n \"name\": path.name,\n \"type\": \"xlsx\",\n \"data_b64\": b64,\n }\n else:\n # Binary / unknown — base64 download link\n try:\n raw = path.read_bytes()\n b64 = base64.b64encode(raw).decode(\"ascii\")\n except OSError:\n return {\"name\": path.name, \"type\": \"error\", \"content\": \"(Error reading file)\"}\n return {\n \"name\": path.name,\n \"type\": \"binary\",\n \"mime\": mime,\n \"data_uri\": f\"data:{mime};base64,{b64}\",\n }\n\n\ndef load_previous_iteration(workspace: Path) -> dict[str, dict]:\n \"\"\"Load previous iteration's feedback and outputs.\n\n Returns a map of run_id -> {\"feedback\": str, \"outputs\": list[dict]}.\n \"\"\"\n result: dict[str, dict] = {}\n\n # Load feedback\n feedback_map: dict[str, str] = {}\n feedback_path = workspace / \"feedback.json\"\n if feedback_path.exists():\n try:\n data = json.loads(feedback_path.read_text())\n feedback_map = {\n r[\"run_id\"]: r[\"feedback\"]\n for r in data.get(\"reviews\", [])\n if r.get(\"feedback\", \"\").strip()\n }\n except (json.JSONDecodeError, OSError, KeyError):\n pass\n\n # Load runs (to get outputs)\n prev_runs = find_runs(workspace)\n for run in prev_runs:\n result[run[\"id\"]] = {\n \"feedback\": feedback_map.get(run[\"id\"], \"\"),\n \"outputs\": run.get(\"outputs\", []),\n }\n\n # Also add feedback for run_ids that had feedback but no matching run\n for run_id, fb in feedback_map.items():\n if run_id not in result:\n result[run_id] = {\"feedback\": fb, \"outputs\": []}\n\n return result\n\n\ndef generate_html(\n runs: list[dict],\n skill_name: str,\n previous: dict[str, dict] | None = None,\n benchmark: dict | None = None,\n) -> str:\n \"\"\"Generate the complete standalone HTML page with embedded data.\"\"\"\n template_path = Path(__file__).parent / \"viewer.html\"\n template = template_path.read_text()\n\n # Build previous_feedback and previous_outputs maps for the template\n previous_feedback: dict[str, str] = {}\n previous_outputs: dict[str, list[dict]] = {}\n if previous:\n for run_id, data in previous.items():\n if data.get(\"feedback\"):\n previous_feedback[run_id] = data[\"feedback\"]\n if data.get(\"outputs\"):\n previous_outputs[run_id] = data[\"outputs\"]\n\n embedded = {\n \"skill_name\": skill_name,\n \"runs\": runs,\n \"previous_feedback\": previous_feedback,\n \"previous_outputs\": previous_outputs,\n }\n if benchmark:\n embedded[\"benchmark\"] = benchmark\n\n data_json = json.dumps(embedded)\n\n return template.replace(\"/*__EMBEDDED_DATA__*/\", f\"const EMBEDDED_DATA = {data_json};\")\n\n\n# ---------------------------------------------------------------------------\n# HTTP server (stdlib only, zero dependencies)\n# ---------------------------------------------------------------------------\n\ndef _kill_port(port: int) -> None:\n \"\"\"Kill any process listening on the given port.\"\"\"\n try:\n result = subprocess.run(\n [\"lsof\", \"-ti\", f\":{port}\"],\n capture_output=True, text=True, timeout=5,\n )\n for pid_str in result.stdout.strip().split(\"\\n\"):\n if pid_str.strip():\n try:\n os.kill(int(pid_str.strip()), signal.SIGTERM)\n except (ProcessLookupError, ValueError):\n pass\n if result.stdout.strip():\n time.sleep(0.5)\n except subprocess.TimeoutExpired:\n pass\n except FileNotFoundError:\n print(\"Note: lsof not found, cannot check if port is in use\", file=sys.stderr)\n\nclass ReviewHandler(BaseHTTPRequestHandler):\n \"\"\"Serves the review HTML and handles feedback saves.\n\n Regenerates the HTML on each page load so that refreshing the browser\n picks up new eval outputs without restarting the server.\n \"\"\"\n\n def __init__(\n self,\n workspace: Path,\n skill_name: str,\n feedback_path: Path,\n previous: dict[str, dict],\n benchmark_path: Path | None,\n *args,\n **kwargs,\n ):\n self.workspace = workspace\n self.skill_name = skill_name\n self.feedback_path = feedback_path\n self.previous = previous\n self.benchmark_path = benchmark_path\n super().__init__(*args, **kwargs)\n\n def do_GET(self) -> None:\n if self.path == \"/\" or self.path == \"/index.html\":\n # Regenerate HTML on each request (re-scans workspace for new outputs)\n runs = find_runs(self.workspace)\n benchmark = None\n if self.benchmark_path and self.benchmark_path.exists():\n try:\n benchmark = json.loads(self.benchmark_path.read_text())\n except (json.JSONDecodeError, OSError):\n pass\n html = generate_html(runs, self.skill_name, self.previous, benchmark)\n content = html.encode(\"utf-8\")\n self.send_response(200)\n self.send_header(\"Content-Type\", \"text/html; charset=utf-8\")\n self.send_header(\"Content-Length\", str(len(content)))\n self.end_headers()\n self.wfile.write(content)\n elif self.path == \"/api/feedback\":\n data = b\"{}\"\n if self.feedback_path.exists():\n data = self.feedback_path.read_bytes()\n self.send_response(200)\n self.send_header(\"Content-Type\", \"application/json\")\n self.send_header(\"Content-Length\", str(len(data)))\n self.end_headers()\n self.wfile.write(data)\n else:\n self.send_error(404)\n\n def do_POST(self) -> None:\n if self.path == \"/api/feedback\":\n length = int(self.headers.get(\"Content-Length\", 0))\n body = self.rfile.read(length)\n try:\n data = json.loads(body)\n if not isinstance(data, dict) or \"reviews\" not in data:\n raise ValueError(\"Expected JSON object with 'reviews' key\")\n self.feedback_path.write_text(json.dumps(data, indent=2) + \"\\n\")\n resp = b'{\"ok\":true}'\n self.send_response(200)\n except (json.JSONDecodeError, OSError, ValueError) as e:\n resp = json.dumps({\"error\": str(e)}).encode()\n self.send_response(500)\n self.send_header(\"Content-Type\", \"application/json\")\n self.send_header(\"Content-Length\", str(len(resp)))\n self.end_headers()\n self.wfile.write(resp)\n else:\n self.send_error(404)\n\n def log_message(self, format: str, *args: object) -> None:\n # Suppress request logging to keep terminal clean\n pass\n\n\ndef main() -> None:\n parser = argparse.ArgumentParser(description=\"Generate and serve eval review\")\n parser.add_argument(\"workspace\", type=Path, help=\"Path to workspace directory\")\n parser.add_argument(\"--port\", \"-p\", type=int, default=3117, help=\"Server port (default: 3117)\")\n parser.add_argument(\"--skill-name\", \"-n\", type=str, default=None, help=\"Skill name for header\")\n parser.add_argument(\n \"--previous-workspace\", type=Path, default=None,\n help=\"Path to previous iteration's workspace (shows old outputs and feedback as context)\",\n )\n parser.add_argument(\n \"--benchmark\", type=Path, default=None,\n help=\"Path to benchmark.json to show in the Benchmark tab\",\n )\n parser.add_argument(\n \"--static\", \"-s\", type=Path, default=None,\n help=\"Write standalone HTML to this path instead of starting a server\",\n )\n args = parser.parse_args()\n\n workspace = args.workspace.resolve()\n if not workspace.is_dir():\n print(f\"Error: {workspace} is not a directory\", file=sys.stderr)\n sys.exit(1)\n\n runs = find_runs(workspace)\n if not runs:\n print(f\"No runs found in {workspace}\", file=sys.stderr)\n sys.exit(1)\n\n skill_name = args.skill_name or workspace.name.replace(\"-workspace\", \"\")\n feedback_path = workspace / \"feedback.json\"\n\n previous: dict[str, dict] = {}\n if args.previous_workspace:\n previous = load_previous_iteration(args.previous_workspace.resolve())\n\n benchmark_path = args.benchmark.resolve() if args.benchmark else None\n benchmark = None\n if benchmark_path and benchmark_path.exists():\n try:\n benchmark = json.loads(benchmark_path.read_text())\n except (json.JSONDecodeError, OSError):\n pass\n\n if args.static:\n html = generate_html(runs, skill_name, previous, benchmark)\n args.static.parent.mkdir(parents=True, exist_ok=True)\n args.static.write_text(html)\n print(f\"\\n Static viewer written to: {args.static}\\n\")\n sys.exit(0)\n\n # Kill any existing process on the target port\n port = args.port\n _kill_port(port)\n handler = partial(ReviewHandler, workspace, skill_name, feedback_path, previous, benchmark_path)\n try:\n server = HTTPServer((\"127.0.0.1\", port), handler)\n except OSError:\n # Port still in use after kill attempt — find a free one\n server = HTTPServer((\"127.0.0.1\", 0), handler)\n port = server.server_address[1]\n\n url = f\"http://localhost:{port}\"\n print(f\"\\n Eval Viewer\")\n print(f\" ─────────────────────────────────\")\n print(f\" URL: {url}\")\n print(f\" Workspace: {workspace}\")\n print(f\" Feedback: {feedback_path}\")\n if previous:\n print(f\" Previous: {args.previous_workspace} ({len(previous)} runs)\")\n if benchmark_path:\n print(f\" Benchmark: {benchmark_path}\")\n print(f\"\\n Press Ctrl+C to stop.\\n\")\n\n webbrowser.open(url)\n\n try:\n server.serve_forever()\n except KeyboardInterrupt:\n print(\"\\nStopped.\")\n server.server_close()\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": ".agents/skills/skill-creator/eval-viewer/viewer.html", "content": "\n\n \n \n \n Eval Review\n \n \n \n \n \n \n \n
\n
\n
\n

Eval Review:

\n
\n Review each output and leave feedback below. Navigate with arrow keys or buttons. When\n done, copy feedback and paste into Claude Code.\n
\n
\n
\n
\n\n \n
\n \n \n
\n\n \n
\n
\n \n
\n
\n Prompt \n
\n
\n
\n
\n
\n\n \n
\n
Output
\n
\n
No output files found
\n
\n
\n\n \n
\n
\n
\n ▶\n Previous Output\n
\n
\n
\n
\n\n \n
\n
\n
\n ▶\n Formal Grades\n
\n
\n
\n
\n\n \n
\n
Your Feedback
\n
\n \n
\n
\n
Previous feedback
\n
\n
\n
\n
\n
\n\n
\n \n \n \n
\n
\n \n\n \n
\n
\n
\n No benchmark data available. Run a benchmark to see quantitative results here.\n
\n
\n
\n
\n\n \n
\n
\n

Review Complete

\n

\n Your feedback has been saved. Go back to your Claude Code session and tell Claude you're\n done reviewing.\n

\n
\n \n
\n
\n
\n\n \n
\n\n \n \n\n" }, { "path": ".agents/skills/skill-creator/references/schemas.md", "content": "# JSON Schemas\n\nThis document defines the JSON schemas used by skill-creator.\n\n---\n\n## evals.json\n\nDefines the evals for a skill. Located at `evals/evals.json` within the skill directory.\n\n```json\n{\n \"skill_name\": \"example-skill\",\n \"evals\": [\n {\n \"id\": 1,\n \"prompt\": \"User's example prompt\",\n \"expected_output\": \"Description of expected result\",\n \"files\": [\"evals/files/sample1.pdf\"],\n \"expectations\": [\"The output includes X\", \"The skill used script Y\"]\n }\n ]\n}\n```\n\n**Fields:**\n\n- `skill_name`: Name matching the skill's frontmatter\n- `evals[].id`: Unique integer identifier\n- `evals[].prompt`: The task to execute\n- `evals[].expected_output`: Human-readable description of success\n- `evals[].files`: Optional list of input file paths (relative to skill root)\n- `evals[].expectations`: List of verifiable statements\n\n---\n\n## history.json\n\nTracks version progression in Improve mode. Located at workspace root.\n\n```json\n{\n \"started_at\": \"2026-01-15T10:30:00Z\",\n \"skill_name\": \"pdf\",\n \"current_best\": \"v2\",\n \"iterations\": [\n {\n \"version\": \"v0\",\n \"parent\": null,\n \"expectation_pass_rate\": 0.65,\n \"grading_result\": \"baseline\",\n \"is_current_best\": false\n },\n {\n \"version\": \"v1\",\n \"parent\": \"v0\",\n \"expectation_pass_rate\": 0.75,\n \"grading_result\": \"won\",\n \"is_current_best\": false\n },\n {\n \"version\": \"v2\",\n \"parent\": \"v1\",\n \"expectation_pass_rate\": 0.85,\n \"grading_result\": \"won\",\n \"is_current_best\": true\n }\n ]\n}\n```\n\n**Fields:**\n\n- `started_at`: ISO timestamp of when improvement started\n- `skill_name`: Name of the skill being improved\n- `current_best`: Version identifier of the best performer\n- `iterations[].version`: Version identifier (v0, v1, ...)\n- `iterations[].parent`: Parent version this was derived from\n- `iterations[].expectation_pass_rate`: Pass rate from grading\n- `iterations[].grading_result`: \"baseline\", \"won\", \"lost\", or \"tie\"\n- `iterations[].is_current_best`: Whether this is the current best version\n\n---\n\n## grading.json\n\nOutput from the grader agent. Located at `/grading.json`.\n\n```json\n{\n \"expectations\": [\n {\n \"text\": \"The output includes the name 'John Smith'\",\n \"passed\": true,\n \"evidence\": \"Found in transcript Step 3: 'Extracted names: John Smith, Sarah Johnson'\"\n },\n {\n \"text\": \"The spreadsheet has a SUM formula in cell B10\",\n \"passed\": false,\n \"evidence\": \"No spreadsheet was created. The output was a text file.\"\n }\n ],\n \"summary\": {\n \"passed\": 2,\n \"failed\": 1,\n \"total\": 3,\n \"pass_rate\": 0.67\n },\n \"execution_metrics\": {\n \"tool_calls\": {\n \"Read\": 5,\n \"Write\": 2,\n \"Bash\": 8\n },\n \"total_tool_calls\": 15,\n \"total_steps\": 6,\n \"errors_encountered\": 0,\n \"output_chars\": 12450,\n \"transcript_chars\": 3200\n },\n \"timing\": {\n \"executor_duration_seconds\": 165.0,\n \"grader_duration_seconds\": 26.0,\n \"total_duration_seconds\": 191.0\n },\n \"claims\": [\n {\n \"claim\": \"The form has 12 fillable fields\",\n \"type\": \"factual\",\n \"verified\": true,\n \"evidence\": \"Counted 12 fields in field_info.json\"\n }\n ],\n \"user_notes_summary\": {\n \"uncertainties\": [\"Used 2023 data, may be stale\"],\n \"needs_review\": [],\n \"workarounds\": [\"Fell back to text overlay for non-fillable fields\"]\n },\n \"eval_feedback\": {\n \"suggestions\": [\n {\n \"assertion\": \"The output includes the name 'John Smith'\",\n \"reason\": \"A hallucinated document that mentions the name would also pass\"\n }\n ],\n \"overall\": \"Assertions check presence but not correctness.\"\n }\n}\n```\n\n**Fields:**\n\n- `expectations[]`: Graded expectations with evidence\n- `summary`: Aggregate pass/fail counts\n- `execution_metrics`: Tool usage and output size (from executor's metrics.json)\n- `timing`: Wall clock timing (from timing.json)\n- `claims`: Extracted and verified claims from the output\n- `user_notes_summary`: Issues flagged by the executor\n- `eval_feedback`: (optional) Improvement suggestions for the evals, only present when the grader identifies issues worth raising\n\n---\n\n## metrics.json\n\nOutput from the executor agent. Located at `/outputs/metrics.json`.\n\n```json\n{\n \"tool_calls\": {\n \"Read\": 5,\n \"Write\": 2,\n \"Bash\": 8,\n \"Edit\": 1,\n \"Glob\": 2,\n \"Grep\": 0\n },\n \"total_tool_calls\": 18,\n \"total_steps\": 6,\n \"files_created\": [\"filled_form.pdf\", \"field_values.json\"],\n \"errors_encountered\": 0,\n \"output_chars\": 12450,\n \"transcript_chars\": 3200\n}\n```\n\n**Fields:**\n\n- `tool_calls`: Count per tool type\n- `total_tool_calls`: Sum of all tool calls\n- `total_steps`: Number of major execution steps\n- `files_created`: List of output files created\n- `errors_encountered`: Number of errors during execution\n- `output_chars`: Total character count of output files\n- `transcript_chars`: Character count of transcript\n\n---\n\n## timing.json\n\nWall clock timing for a run. Located at `/timing.json`.\n\n**How to capture:** When a subagent task completes, the task notification includes `total_tokens` and `duration_ms`. Save these immediately — they are not persisted anywhere else and cannot be recovered after the fact.\n\n```json\n{\n \"total_tokens\": 84852,\n \"duration_ms\": 23332,\n \"total_duration_seconds\": 23.3,\n \"executor_start\": \"2026-01-15T10:30:00Z\",\n \"executor_end\": \"2026-01-15T10:32:45Z\",\n \"executor_duration_seconds\": 165.0,\n \"grader_start\": \"2026-01-15T10:32:46Z\",\n \"grader_end\": \"2026-01-15T10:33:12Z\",\n \"grader_duration_seconds\": 26.0\n}\n```\n\n---\n\n## benchmark.json\n\nOutput from Benchmark mode. Located at `benchmarks//benchmark.json`.\n\n```json\n{\n \"metadata\": {\n \"skill_name\": \"pdf\",\n \"skill_path\": \"/path/to/pdf\",\n \"executor_model\": \"claude-sonnet-4-20250514\",\n \"analyzer_model\": \"most-capable-model\",\n \"timestamp\": \"2026-01-15T10:30:00Z\",\n \"evals_run\": [1, 2, 3],\n \"runs_per_configuration\": 3\n },\n\n \"runs\": [\n {\n \"eval_id\": 1,\n \"eval_name\": \"Ocean\",\n \"configuration\": \"with_skill\",\n \"run_number\": 1,\n \"result\": {\n \"pass_rate\": 0.85,\n \"passed\": 6,\n \"failed\": 1,\n \"total\": 7,\n \"time_seconds\": 42.5,\n \"tokens\": 3800,\n \"tool_calls\": 18,\n \"errors\": 0\n },\n \"expectations\": [{ \"text\": \"...\", \"passed\": true, \"evidence\": \"...\" }],\n \"notes\": [\"Used 2023 data, may be stale\", \"Fell back to text overlay for non-fillable fields\"]\n }\n ],\n\n \"run_summary\": {\n \"with_skill\": {\n \"pass_rate\": { \"mean\": 0.85, \"stddev\": 0.05, \"min\": 0.8, \"max\": 0.9 },\n \"time_seconds\": { \"mean\": 45.0, \"stddev\": 12.0, \"min\": 32.0, \"max\": 58.0 },\n \"tokens\": { \"mean\": 3800, \"stddev\": 400, \"min\": 3200, \"max\": 4100 }\n },\n \"without_skill\": {\n \"pass_rate\": { \"mean\": 0.35, \"stddev\": 0.08, \"min\": 0.28, \"max\": 0.45 },\n \"time_seconds\": { \"mean\": 32.0, \"stddev\": 8.0, \"min\": 24.0, \"max\": 42.0 },\n \"tokens\": { \"mean\": 2100, \"stddev\": 300, \"min\": 1800, \"max\": 2500 }\n },\n \"delta\": {\n \"pass_rate\": \"+0.50\",\n \"time_seconds\": \"+13.0\",\n \"tokens\": \"+1700\"\n }\n },\n\n \"notes\": [\n \"Assertion 'Output is a PDF file' passes 100% in both configurations - may not differentiate skill value\",\n \"Eval 3 shows high variance (50% ± 40%) - may be flaky or model-dependent\",\n \"Without-skill runs consistently fail on table extraction expectations\",\n \"Skill adds 13s average execution time but improves pass rate by 50%\"\n ]\n}\n```\n\n**Fields:**\n\n- `metadata`: Information about the benchmark run\n - `skill_name`: Name of the skill\n - `timestamp`: When the benchmark was run\n - `evals_run`: List of eval names or IDs\n - `runs_per_configuration`: Number of runs per config (e.g. 3)\n- `runs[]`: Individual run results\n - `eval_id`: Numeric eval identifier\n - `eval_name`: Human-readable eval name (used as section header in the viewer)\n - `configuration`: Must be `\"with_skill\"` or `\"without_skill\"` (the viewer uses this exact string for grouping and color coding)\n - `run_number`: Integer run number (1, 2, 3...)\n - `result`: Nested object with `pass_rate`, `passed`, `total`, `time_seconds`, `tokens`, `errors`\n- `run_summary`: Statistical aggregates per configuration\n - `with_skill` / `without_skill`: Each contains `pass_rate`, `time_seconds`, `tokens` objects with `mean` and `stddev` fields\n - `delta`: Difference strings like `\"+0.50\"`, `\"+13.0\"`, `\"+1700\"`\n- `notes`: Freeform observations from the analyzer\n\n**Important:** The viewer reads these field names exactly. Using `config` instead of `configuration`, or putting `pass_rate` at the top level of a run instead of nested under `result`, will cause the viewer to show empty/zero values. Always reference this schema when generating benchmark.json manually.\n\n---\n\n## comparison.json\n\nOutput from blind comparator. Located at `/comparison-N.json`.\n\n```json\n{\n \"winner\": \"A\",\n \"reasoning\": \"Output A provides a complete solution with proper formatting and all required fields. Output B is missing the date field and has formatting inconsistencies.\",\n \"rubric\": {\n \"A\": {\n \"content\": {\n \"correctness\": 5,\n \"completeness\": 5,\n \"accuracy\": 4\n },\n \"structure\": {\n \"organization\": 4,\n \"formatting\": 5,\n \"usability\": 4\n },\n \"content_score\": 4.7,\n \"structure_score\": 4.3,\n \"overall_score\": 9.0\n },\n \"B\": {\n \"content\": {\n \"correctness\": 3,\n \"completeness\": 2,\n \"accuracy\": 3\n },\n \"structure\": {\n \"organization\": 3,\n \"formatting\": 2,\n \"usability\": 3\n },\n \"content_score\": 2.7,\n \"structure_score\": 2.7,\n \"overall_score\": 5.4\n }\n },\n \"output_quality\": {\n \"A\": {\n \"score\": 9,\n \"strengths\": [\"Complete solution\", \"Well-formatted\", \"All fields present\"],\n \"weaknesses\": [\"Minor style inconsistency in header\"]\n },\n \"B\": {\n \"score\": 5,\n \"strengths\": [\"Readable output\", \"Correct basic structure\"],\n \"weaknesses\": [\"Missing date field\", \"Formatting inconsistencies\", \"Partial data extraction\"]\n }\n },\n \"expectation_results\": {\n \"A\": {\n \"passed\": 4,\n \"total\": 5,\n \"pass_rate\": 0.8,\n \"details\": [{ \"text\": \"Output includes name\", \"passed\": true }]\n },\n \"B\": {\n \"passed\": 3,\n \"total\": 5,\n \"pass_rate\": 0.6,\n \"details\": [{ \"text\": \"Output includes name\", \"passed\": true }]\n }\n }\n}\n```\n\n---\n\n## analysis.json\n\nOutput from post-hoc analyzer. Located at `/analysis.json`.\n\n```json\n{\n \"comparison_summary\": {\n \"winner\": \"A\",\n \"winner_skill\": \"path/to/winner/skill\",\n \"loser_skill\": \"path/to/loser/skill\",\n \"comparator_reasoning\": \"Brief summary of why comparator chose winner\"\n },\n \"winner_strengths\": [\n \"Clear step-by-step instructions for handling multi-page documents\",\n \"Included validation script that caught formatting errors\"\n ],\n \"loser_weaknesses\": [\n \"Vague instruction 'process the document appropriately' led to inconsistent behavior\",\n \"No script for validation, agent had to improvise\"\n ],\n \"instruction_following\": {\n \"winner\": {\n \"score\": 9,\n \"issues\": [\"Minor: skipped optional logging step\"]\n },\n \"loser\": {\n \"score\": 6,\n \"issues\": [\n \"Did not use the skill's formatting template\",\n \"Invented own approach instead of following step 3\"\n ]\n }\n },\n \"improvement_suggestions\": [\n {\n \"priority\": \"high\",\n \"category\": \"instructions\",\n \"suggestion\": \"Replace 'process the document appropriately' with explicit steps\",\n \"expected_impact\": \"Would eliminate ambiguity that caused inconsistent behavior\"\n }\n ],\n \"transcript_insights\": {\n \"winner_execution_pattern\": \"Read skill -> Followed 5-step process -> Used validation script\",\n \"loser_execution_pattern\": \"Read skill -> Unclear on approach -> Tried 3 different methods\"\n }\n}\n```\n" }, { "path": ".agents/skills/skill-creator/scripts/__init__.py", "content": "" }, { "path": ".agents/skills/skill-creator/scripts/aggregate_benchmark.py", "content": "#!/usr/bin/env python3\n\"\"\"\nAggregate individual run results into benchmark summary statistics.\n\nReads grading.json files from run directories and produces:\n- run_summary with mean, stddev, min, max for each metric\n- delta between with_skill and without_skill configurations\n\nUsage:\n python aggregate_benchmark.py \n\nExample:\n python aggregate_benchmark.py benchmarks/2026-01-15T10-30-00/\n\nThe script supports two directory layouts:\n\n Workspace layout (from skill-creator iterations):\n /\n └── eval-N/\n ├── with_skill/\n │ ├── run-1/grading.json\n │ └── run-2/grading.json\n └── without_skill/\n ├── run-1/grading.json\n └── run-2/grading.json\n\n Legacy layout (with runs/ subdirectory):\n /\n └── runs/\n └── eval-N/\n ├── with_skill/\n │ └── run-1/grading.json\n └── without_skill/\n └── run-1/grading.json\n\"\"\"\n\nimport argparse\nimport json\nimport math\nimport sys\nfrom datetime import datetime, timezone\nfrom pathlib import Path\n\n\ndef calculate_stats(values: list[float]) -> dict:\n \"\"\"Calculate mean, stddev, min, max for a list of values.\"\"\"\n if not values:\n return {\"mean\": 0.0, \"stddev\": 0.0, \"min\": 0.0, \"max\": 0.0}\n\n n = len(values)\n mean = sum(values) / n\n\n if n > 1:\n variance = sum((x - mean) ** 2 for x in values) / (n - 1)\n stddev = math.sqrt(variance)\n else:\n stddev = 0.0\n\n return {\n \"mean\": round(mean, 4),\n \"stddev\": round(stddev, 4),\n \"min\": round(min(values), 4),\n \"max\": round(max(values), 4)\n }\n\n\ndef load_run_results(benchmark_dir: Path) -> dict:\n \"\"\"\n Load all run results from a benchmark directory.\n\n Returns dict keyed by config name (e.g. \"with_skill\"/\"without_skill\",\n or \"new_skill\"/\"old_skill\"), each containing a list of run results.\n \"\"\"\n # Support both layouts: eval dirs directly under benchmark_dir, or under runs/\n runs_dir = benchmark_dir / \"runs\"\n if runs_dir.exists():\n search_dir = runs_dir\n elif list(benchmark_dir.glob(\"eval-*\")):\n search_dir = benchmark_dir\n else:\n print(f\"No eval directories found in {benchmark_dir} or {benchmark_dir / 'runs'}\")\n return {}\n\n results: dict[str, list] = {}\n\n for eval_idx, eval_dir in enumerate(sorted(search_dir.glob(\"eval-*\"))):\n metadata_path = eval_dir / \"eval_metadata.json\"\n if metadata_path.exists():\n try:\n with open(metadata_path) as mf:\n eval_id = json.load(mf).get(\"eval_id\", eval_idx)\n except (json.JSONDecodeError, OSError):\n eval_id = eval_idx\n else:\n try:\n eval_id = int(eval_dir.name.split(\"-\")[1])\n except ValueError:\n eval_id = eval_idx\n\n # Discover config directories dynamically rather than hardcoding names\n for config_dir in sorted(eval_dir.iterdir()):\n if not config_dir.is_dir():\n continue\n # Skip non-config directories (inputs, outputs, etc.)\n if not list(config_dir.glob(\"run-*\")):\n continue\n config = config_dir.name\n if config not in results:\n results[config] = []\n\n for run_dir in sorted(config_dir.glob(\"run-*\")):\n run_number = int(run_dir.name.split(\"-\")[1])\n grading_file = run_dir / \"grading.json\"\n\n if not grading_file.exists():\n print(f\"Warning: grading.json not found in {run_dir}\")\n continue\n\n try:\n with open(grading_file) as f:\n grading = json.load(f)\n except json.JSONDecodeError as e:\n print(f\"Warning: Invalid JSON in {grading_file}: {e}\")\n continue\n\n # Extract metrics\n result = {\n \"eval_id\": eval_id,\n \"run_number\": run_number,\n \"pass_rate\": grading.get(\"summary\", {}).get(\"pass_rate\", 0.0),\n \"passed\": grading.get(\"summary\", {}).get(\"passed\", 0),\n \"failed\": grading.get(\"summary\", {}).get(\"failed\", 0),\n \"total\": grading.get(\"summary\", {}).get(\"total\", 0),\n }\n\n # Extract timing — check grading.json first, then sibling timing.json\n timing = grading.get(\"timing\", {})\n result[\"time_seconds\"] = timing.get(\"total_duration_seconds\", 0.0)\n timing_file = run_dir / \"timing.json\"\n if result[\"time_seconds\"] == 0.0 and timing_file.exists():\n try:\n with open(timing_file) as tf:\n timing_data = json.load(tf)\n result[\"time_seconds\"] = timing_data.get(\"total_duration_seconds\", 0.0)\n result[\"tokens\"] = timing_data.get(\"total_tokens\", 0)\n except json.JSONDecodeError:\n pass\n\n # Extract metrics if available\n metrics = grading.get(\"execution_metrics\", {})\n result[\"tool_calls\"] = metrics.get(\"total_tool_calls\", 0)\n if not result.get(\"tokens\"):\n result[\"tokens\"] = metrics.get(\"output_chars\", 0)\n result[\"errors\"] = metrics.get(\"errors_encountered\", 0)\n\n # Extract expectations — viewer requires fields: text, passed, evidence\n raw_expectations = grading.get(\"expectations\", [])\n for exp in raw_expectations:\n if \"text\" not in exp or \"passed\" not in exp:\n print(f\"Warning: expectation in {grading_file} missing required fields (text, passed, evidence): {exp}\")\n result[\"expectations\"] = raw_expectations\n\n # Extract notes from user_notes_summary\n notes_summary = grading.get(\"user_notes_summary\", {})\n notes = []\n notes.extend(notes_summary.get(\"uncertainties\", []))\n notes.extend(notes_summary.get(\"needs_review\", []))\n notes.extend(notes_summary.get(\"workarounds\", []))\n result[\"notes\"] = notes\n\n results[config].append(result)\n\n return results\n\n\ndef aggregate_results(results: dict) -> dict:\n \"\"\"\n Aggregate run results into summary statistics.\n\n Returns run_summary with stats for each configuration and delta.\n \"\"\"\n run_summary = {}\n configs = list(results.keys())\n\n for config in configs:\n runs = results.get(config, [])\n\n if not runs:\n run_summary[config] = {\n \"pass_rate\": {\"mean\": 0.0, \"stddev\": 0.0, \"min\": 0.0, \"max\": 0.0},\n \"time_seconds\": {\"mean\": 0.0, \"stddev\": 0.0, \"min\": 0.0, \"max\": 0.0},\n \"tokens\": {\"mean\": 0, \"stddev\": 0, \"min\": 0, \"max\": 0}\n }\n continue\n\n pass_rates = [r[\"pass_rate\"] for r in runs]\n times = [r[\"time_seconds\"] for r in runs]\n tokens = [r.get(\"tokens\", 0) for r in runs]\n\n run_summary[config] = {\n \"pass_rate\": calculate_stats(pass_rates),\n \"time_seconds\": calculate_stats(times),\n \"tokens\": calculate_stats(tokens)\n }\n\n # Calculate delta between the first two configs (if two exist)\n if len(configs) >= 2:\n primary = run_summary.get(configs[0], {})\n baseline = run_summary.get(configs[1], {})\n else:\n primary = run_summary.get(configs[0], {}) if configs else {}\n baseline = {}\n\n delta_pass_rate = primary.get(\"pass_rate\", {}).get(\"mean\", 0) - baseline.get(\"pass_rate\", {}).get(\"mean\", 0)\n delta_time = primary.get(\"time_seconds\", {}).get(\"mean\", 0) - baseline.get(\"time_seconds\", {}).get(\"mean\", 0)\n delta_tokens = primary.get(\"tokens\", {}).get(\"mean\", 0) - baseline.get(\"tokens\", {}).get(\"mean\", 0)\n\n run_summary[\"delta\"] = {\n \"pass_rate\": f\"{delta_pass_rate:+.2f}\",\n \"time_seconds\": f\"{delta_time:+.1f}\",\n \"tokens\": f\"{delta_tokens:+.0f}\"\n }\n\n return run_summary\n\n\ndef generate_benchmark(benchmark_dir: Path, skill_name: str = \"\", skill_path: str = \"\") -> dict:\n \"\"\"\n Generate complete benchmark.json from run results.\n \"\"\"\n results = load_run_results(benchmark_dir)\n run_summary = aggregate_results(results)\n\n # Build runs array for benchmark.json\n runs = []\n for config in results:\n for result in results[config]:\n runs.append({\n \"eval_id\": result[\"eval_id\"],\n \"configuration\": config,\n \"run_number\": result[\"run_number\"],\n \"result\": {\n \"pass_rate\": result[\"pass_rate\"],\n \"passed\": result[\"passed\"],\n \"failed\": result[\"failed\"],\n \"total\": result[\"total\"],\n \"time_seconds\": result[\"time_seconds\"],\n \"tokens\": result.get(\"tokens\", 0),\n \"tool_calls\": result.get(\"tool_calls\", 0),\n \"errors\": result.get(\"errors\", 0)\n },\n \"expectations\": result[\"expectations\"],\n \"notes\": result[\"notes\"]\n })\n\n # Determine eval IDs from results\n eval_ids = sorted(set(\n r[\"eval_id\"]\n for config in results.values()\n for r in config\n ))\n\n benchmark = {\n \"metadata\": {\n \"skill_name\": skill_name or \"\",\n \"skill_path\": skill_path or \"\",\n \"executor_model\": \"\",\n \"analyzer_model\": \"\",\n \"timestamp\": datetime.now(timezone.utc).strftime(\"%Y-%m-%dT%H:%M:%SZ\"),\n \"evals_run\": eval_ids,\n \"runs_per_configuration\": 3\n },\n \"runs\": runs,\n \"run_summary\": run_summary,\n \"notes\": [] # To be filled by analyzer\n }\n\n return benchmark\n\n\ndef generate_markdown(benchmark: dict) -> str:\n \"\"\"Generate human-readable benchmark.md from benchmark data.\"\"\"\n metadata = benchmark[\"metadata\"]\n run_summary = benchmark[\"run_summary\"]\n\n # Determine config names (excluding \"delta\")\n configs = [k for k in run_summary if k != \"delta\"]\n config_a = configs[0] if len(configs) >= 1 else \"config_a\"\n config_b = configs[1] if len(configs) >= 2 else \"config_b\"\n label_a = config_a.replace(\"_\", \" \").title()\n label_b = config_b.replace(\"_\", \" \").title()\n\n lines = [\n f\"# Skill Benchmark: {metadata['skill_name']}\",\n \"\",\n f\"**Model**: {metadata['executor_model']}\",\n f\"**Date**: {metadata['timestamp']}\",\n f\"**Evals**: {', '.join(map(str, metadata['evals_run']))} ({metadata['runs_per_configuration']} runs each per configuration)\",\n \"\",\n \"## Summary\",\n \"\",\n f\"| Metric | {label_a} | {label_b} | Delta |\",\n \"|--------|------------|---------------|-------|\",\n ]\n\n a_summary = run_summary.get(config_a, {})\n b_summary = run_summary.get(config_b, {})\n delta = run_summary.get(\"delta\", {})\n\n # Format pass rate\n a_pr = a_summary.get(\"pass_rate\", {})\n b_pr = b_summary.get(\"pass_rate\", {})\n lines.append(f\"| Pass Rate | {a_pr.get('mean', 0)*100:.0f}% ± {a_pr.get('stddev', 0)*100:.0f}% | {b_pr.get('mean', 0)*100:.0f}% ± {b_pr.get('stddev', 0)*100:.0f}% | {delta.get('pass_rate', '—')} |\")\n\n # Format time\n a_time = a_summary.get(\"time_seconds\", {})\n b_time = b_summary.get(\"time_seconds\", {})\n lines.append(f\"| Time | {a_time.get('mean', 0):.1f}s ± {a_time.get('stddev', 0):.1f}s | {b_time.get('mean', 0):.1f}s ± {b_time.get('stddev', 0):.1f}s | {delta.get('time_seconds', '—')}s |\")\n\n # Format tokens\n a_tokens = a_summary.get(\"tokens\", {})\n b_tokens = b_summary.get(\"tokens\", {})\n lines.append(f\"| Tokens | {a_tokens.get('mean', 0):.0f} ± {a_tokens.get('stddev', 0):.0f} | {b_tokens.get('mean', 0):.0f} ± {b_tokens.get('stddev', 0):.0f} | {delta.get('tokens', '—')} |\")\n\n # Notes section\n if benchmark.get(\"notes\"):\n lines.extend([\n \"\",\n \"## Notes\",\n \"\"\n ])\n for note in benchmark[\"notes\"]:\n lines.append(f\"- {note}\")\n\n return \"\\n\".join(lines)\n\n\ndef main():\n parser = argparse.ArgumentParser(\n description=\"Aggregate benchmark run results into summary statistics\"\n )\n parser.add_argument(\n \"benchmark_dir\",\n type=Path,\n help=\"Path to the benchmark directory\"\n )\n parser.add_argument(\n \"--skill-name\",\n default=\"\",\n help=\"Name of the skill being benchmarked\"\n )\n parser.add_argument(\n \"--skill-path\",\n default=\"\",\n help=\"Path to the skill being benchmarked\"\n )\n parser.add_argument(\n \"--output\", \"-o\",\n type=Path,\n help=\"Output path for benchmark.json (default: /benchmark.json)\"\n )\n\n args = parser.parse_args()\n\n if not args.benchmark_dir.exists():\n print(f\"Directory not found: {args.benchmark_dir}\")\n sys.exit(1)\n\n # Generate benchmark\n benchmark = generate_benchmark(args.benchmark_dir, args.skill_name, args.skill_path)\n\n # Determine output paths\n output_json = args.output or (args.benchmark_dir / \"benchmark.json\")\n output_md = output_json.with_suffix(\".md\")\n\n # Write benchmark.json\n with open(output_json, \"w\") as f:\n json.dump(benchmark, f, indent=2)\n print(f\"Generated: {output_json}\")\n\n # Write benchmark.md\n markdown = generate_markdown(benchmark)\n with open(output_md, \"w\") as f:\n f.write(markdown)\n print(f\"Generated: {output_md}\")\n\n # Print summary\n run_summary = benchmark[\"run_summary\"]\n configs = [k for k in run_summary if k != \"delta\"]\n delta = run_summary.get(\"delta\", {})\n\n print(f\"\\nSummary:\")\n for config in configs:\n pr = run_summary[config][\"pass_rate\"][\"mean\"]\n label = config.replace(\"_\", \" \").title()\n print(f\" {label}: {pr*100:.1f}% pass rate\")\n print(f\" Delta: {delta.get('pass_rate', '—')}\")\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": ".agents/skills/skill-creator/scripts/generate_report.py", "content": "#!/usr/bin/env python3\n\"\"\"Generate an HTML report from run_loop.py output.\n\nTakes the JSON output from run_loop.py and generates a visual HTML report\nshowing each description attempt with check/x for each test case.\nDistinguishes between train and test queries.\n\"\"\"\n\nimport argparse\nimport html\nimport json\nimport sys\nfrom pathlib import Path\n\n\ndef generate_html(data: dict, auto_refresh: bool = False, skill_name: str = \"\") -> str:\n \"\"\"Generate HTML report from loop output data. If auto_refresh is True, adds a meta refresh tag.\"\"\"\n history = data.get(\"history\", [])\n holdout = data.get(\"holdout\", 0)\n title_prefix = html.escape(skill_name + \" \\u2014 \") if skill_name else \"\"\n\n # Get all unique queries from train and test sets, with should_trigger info\n train_queries: list[dict] = []\n test_queries: list[dict] = []\n if history:\n for r in history[0].get(\"train_results\", history[0].get(\"results\", [])):\n train_queries.append({\"query\": r[\"query\"], \"should_trigger\": r.get(\"should_trigger\", True)})\n if history[0].get(\"test_results\"):\n for r in history[0].get(\"test_results\", []):\n test_queries.append({\"query\": r[\"query\"], \"should_trigger\": r.get(\"should_trigger\", True)})\n\n refresh_tag = ' \\n' if auto_refresh else \"\"\n\n html_parts = [\"\"\"\n\n\n \n\"\"\" + refresh_tag + \"\"\" \"\"\" + title_prefix + \"\"\"Skill Description Optimization\n \n \n \n \n\n\n

\"\"\" + title_prefix + \"\"\"Skill Description Optimization

\n
\n Optimizing your skill's description. This page updates automatically as Claude tests different versions of your skill's description. Each row is an iteration — a new description attempt. The columns show test queries: green checkmarks mean the skill triggered correctly (or correctly didn't trigger), red crosses mean it got it wrong. The \"Train\" score shows performance on queries used to improve the description; the \"Test\" score shows performance on held-out queries the optimizer hasn't seen. When it's done, Claude will apply the best-performing description to your skill.\n
\n\"\"\"]\n\n # Summary section\n best_test_score = data.get('best_test_score')\n best_train_score = data.get('best_train_score')\n html_parts.append(f\"\"\"\n
\n

Original: {html.escape(data.get('original_description', 'N/A'))}

\n

Best: {html.escape(data.get('best_description', 'N/A'))}

\n

Best Score: {data.get('best_score', 'N/A')} {'(test)' if best_test_score else '(train)'}

\n

Iterations: {data.get('iterations_run', 0)} | Train: {data.get('train_size', '?')} | Test: {data.get('test_size', '?')}

\n
\n\"\"\")\n\n # Legend\n html_parts.append(\"\"\"\n
\n Query columns:\n Should trigger\n Should NOT trigger\n Train\n Test\n
\n\"\"\")\n\n # Table header\n html_parts.append(\"\"\"\n
\n \n \n \n \n \n \n \n\"\"\")\n\n # Add column headers for train queries\n for qinfo in train_queries:\n polarity = \"positive-col\" if qinfo[\"should_trigger\"] else \"negative-col\"\n html_parts.append(f' \\n')\n\n # Add column headers for test queries (different color)\n for qinfo in test_queries:\n polarity = \"positive-col\" if qinfo[\"should_trigger\"] else \"negative-col\"\n html_parts.append(f' \\n')\n\n html_parts.append(\"\"\" \n \n \n\"\"\")\n\n # Find best iteration for highlighting\n if test_queries:\n best_iter = max(history, key=lambda h: h.get(\"test_passed\") or 0).get(\"iteration\")\n else:\n best_iter = max(history, key=lambda h: h.get(\"train_passed\", h.get(\"passed\", 0))).get(\"iteration\")\n\n # Add rows for each iteration\n for h in history:\n iteration = h.get(\"iteration\", \"?\")\n train_passed = h.get(\"train_passed\", h.get(\"passed\", 0))\n train_total = h.get(\"train_total\", h.get(\"total\", 0))\n test_passed = h.get(\"test_passed\")\n test_total = h.get(\"test_total\")\n description = h.get(\"description\", \"\")\n train_results = h.get(\"train_results\", h.get(\"results\", []))\n test_results = h.get(\"test_results\", [])\n\n # Create lookups for results by query\n train_by_query = {r[\"query\"]: r for r in train_results}\n test_by_query = {r[\"query\"]: r for r in test_results} if test_results else {}\n\n # Compute aggregate correct/total runs across all retries\n def aggregate_runs(results: list[dict]) -> tuple[int, int]:\n correct = 0\n total = 0\n for r in results:\n runs = r.get(\"runs\", 0)\n triggers = r.get(\"triggers\", 0)\n total += runs\n if r.get(\"should_trigger\", True):\n correct += triggers\n else:\n correct += runs - triggers\n return correct, total\n\n train_correct, train_runs = aggregate_runs(train_results)\n test_correct, test_runs = aggregate_runs(test_results)\n\n # Determine score classes\n def score_class(correct: int, total: int) -> str:\n if total > 0:\n ratio = correct / total\n if ratio >= 0.8:\n return \"score-good\"\n elif ratio >= 0.5:\n return \"score-ok\"\n return \"score-bad\"\n\n train_class = score_class(train_correct, train_runs)\n test_class = score_class(test_correct, test_runs)\n\n row_class = \"best-row\" if iteration == best_iter else \"\"\n\n html_parts.append(f\"\"\" \n \n \n \n \n\"\"\")\n\n # Add result for each train query\n for qinfo in train_queries:\n r = train_by_query.get(qinfo[\"query\"], {})\n did_pass = r.get(\"pass\", False)\n triggers = r.get(\"triggers\", 0)\n runs = r.get(\"runs\", 0)\n\n icon = \"✓\" if did_pass else \"✗\"\n css_class = \"pass\" if did_pass else \"fail\"\n\n html_parts.append(f' \\n')\n\n # Add result for each test query (with different background)\n for qinfo in test_queries:\n r = test_by_query.get(qinfo[\"query\"], {})\n did_pass = r.get(\"pass\", False)\n triggers = r.get(\"triggers\", 0)\n runs = r.get(\"runs\", 0)\n\n icon = \"✓\" if did_pass else \"✗\"\n css_class = \"pass\" if did_pass else \"fail\"\n\n html_parts.append(f' \\n')\n\n html_parts.append(\" \\n\")\n\n html_parts.append(\"\"\" \n
IterTrainTestDescription{html.escape(qinfo[\"query\"])}{html.escape(qinfo[\"query\"])}
{iteration}{train_correct}/{train_runs}{test_correct}/{test_runs}{html.escape(description)}{icon}{triggers}/{runs}{icon}{triggers}/{runs}
\n
\n\"\"\")\n\n html_parts.append(\"\"\"\n\n\n\"\"\")\n\n return \"\".join(html_parts)\n\n\ndef main():\n parser = argparse.ArgumentParser(description=\"Generate HTML report from run_loop output\")\n parser.add_argument(\"input\", help=\"Path to JSON output from run_loop.py (or - for stdin)\")\n parser.add_argument(\"-o\", \"--output\", default=None, help=\"Output HTML file (default: stdout)\")\n parser.add_argument(\"--skill-name\", default=\"\", help=\"Skill name to include in the report title\")\n args = parser.parse_args()\n\n if args.input == \"-\":\n data = json.load(sys.stdin)\n else:\n data = json.loads(Path(args.input).read_text())\n\n html_output = generate_html(data, skill_name=args.skill_name)\n\n if args.output:\n Path(args.output).write_text(html_output)\n print(f\"Report written to {args.output}\", file=sys.stderr)\n else:\n print(html_output)\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": ".agents/skills/skill-creator/scripts/improve_description.py", "content": "#!/usr/bin/env python3\n\"\"\"Improve a skill description based on eval results.\n\nTakes eval results (from run_eval.py) and generates an improved description\nby calling `claude -p` as a subprocess (same auth pattern as run_eval.py —\nuses the session's Claude Code auth, no separate ANTHROPIC_API_KEY needed).\n\"\"\"\n\nimport argparse\nimport json\nimport os\nimport re\nimport subprocess\nimport sys\nfrom pathlib import Path\n\nfrom scripts.utils import parse_skill_md\n\n\ndef _call_claude(prompt: str, model: str | None, timeout: int = 300) -> str:\n \"\"\"Run `claude -p` with the prompt on stdin and return the text response.\n\n Prompt goes over stdin (not argv) because it embeds the full SKILL.md\n body and can easily exceed comfortable argv length.\n \"\"\"\n cmd = [\"claude\", \"-p\", \"--output-format\", \"text\"]\n if model:\n cmd.extend([\"--model\", model])\n\n # Remove CLAUDECODE env var to allow nesting claude -p inside a\n # Claude Code session. The guard is for interactive terminal conflicts;\n # programmatic subprocess usage is safe. Same pattern as run_eval.py.\n env = {k: v for k, v in os.environ.items() if k != \"CLAUDECODE\"}\n\n result = subprocess.run(\n cmd,\n input=prompt,\n capture_output=True,\n text=True,\n env=env,\n timeout=timeout,\n )\n if result.returncode != 0:\n raise RuntimeError(\n f\"claude -p exited {result.returncode}\\nstderr: {result.stderr}\"\n )\n return result.stdout\n\n\ndef improve_description(\n skill_name: str,\n skill_content: str,\n current_description: str,\n eval_results: dict,\n history: list[dict],\n model: str,\n test_results: dict | None = None,\n log_dir: Path | None = None,\n iteration: int | None = None,\n) -> str:\n \"\"\"Call Claude to improve the description based on eval results.\"\"\"\n failed_triggers = [\n r for r in eval_results[\"results\"]\n if r[\"should_trigger\"] and not r[\"pass\"]\n ]\n false_triggers = [\n r for r in eval_results[\"results\"]\n if not r[\"should_trigger\"] and not r[\"pass\"]\n ]\n\n # Build scores summary\n train_score = f\"{eval_results['summary']['passed']}/{eval_results['summary']['total']}\"\n if test_results:\n test_score = f\"{test_results['summary']['passed']}/{test_results['summary']['total']}\"\n scores_summary = f\"Train: {train_score}, Test: {test_score}\"\n else:\n scores_summary = f\"Train: {train_score}\"\n\n prompt = f\"\"\"You are optimizing a skill description for a Claude Code skill called \"{skill_name}\". A \"skill\" is sort of like a prompt, but with progressive disclosure -- there's a title and description that Claude sees when deciding whether to use the skill, and then if it does use the skill, it reads the .md file which has lots more details and potentially links to other resources in the skill folder like helper files and scripts and additional documentation or examples.\n\nThe description appears in Claude's \"available_skills\" list. When a user sends a query, Claude decides whether to invoke the skill based solely on the title and on this description. Your goal is to write a description that triggers for relevant queries, and doesn't trigger for irrelevant ones.\n\nHere's the current description:\n\n\"{current_description}\"\n\n\nCurrent scores ({scores_summary}):\n\n\"\"\"\n if failed_triggers:\n prompt += \"FAILED TO TRIGGER (should have triggered but didn't):\\n\"\n for r in failed_triggers:\n prompt += f' - \"{r[\"query\"]}\" (triggered {r[\"triggers\"]}/{r[\"runs\"]} times)\\n'\n prompt += \"\\n\"\n\n if false_triggers:\n prompt += \"FALSE TRIGGERS (triggered but shouldn't have):\\n\"\n for r in false_triggers:\n prompt += f' - \"{r[\"query\"]}\" (triggered {r[\"triggers\"]}/{r[\"runs\"]} times)\\n'\n prompt += \"\\n\"\n\n if history:\n prompt += \"PREVIOUS ATTEMPTS (do NOT repeat these — try something structurally different):\\n\\n\"\n for h in history:\n train_s = f\"{h.get('train_passed', h.get('passed', 0))}/{h.get('train_total', h.get('total', 0))}\"\n test_s = f\"{h.get('test_passed', '?')}/{h.get('test_total', '?')}\" if h.get('test_passed') is not None else None\n score_str = f\"train={train_s}\" + (f\", test={test_s}\" if test_s else \"\")\n prompt += f'\\n'\n prompt += f'Description: \"{h[\"description\"]}\"\\n'\n if \"results\" in h:\n prompt += \"Train results:\\n\"\n for r in h[\"results\"]:\n status = \"PASS\" if r[\"pass\"] else \"FAIL\"\n prompt += f' [{status}] \"{r[\"query\"][:80]}\" (triggered {r[\"triggers\"]}/{r[\"runs\"]})\\n'\n if h.get(\"note\"):\n prompt += f'Note: {h[\"note\"]}\\n'\n prompt += \"\\n\\n\"\n\n prompt += f\"\"\"\n\nSkill content (for context on what the skill does):\n\n{skill_content}\n\n\nBased on the failures, write a new and improved description that is more likely to trigger correctly. When I say \"based on the failures\", it's a bit of a tricky line to walk because we don't want to overfit to the specific cases you're seeing. So what I DON'T want you to do is produce an ever-expanding list of specific queries that this skill should or shouldn't trigger for. Instead, try to generalize from the failures to broader categories of user intent and situations where this skill would be useful or not useful. The reason for this is twofold:\n\n1. Avoid overfitting\n2. The list might get loooong and it's injected into ALL queries and there might be a lot of skills, so we don't want to blow too much space on any given description.\n\nConcretely, your description should not be more than about 100-200 words, even if that comes at the cost of accuracy. There is a hard limit of 1024 characters — descriptions over that will be truncated, so stay comfortably under it.\n\nHere are some tips that we've found to work well in writing these descriptions:\n- The skill should be phrased in the imperative -- \"Use this skill for\" rather than \"this skill does\"\n- The skill description should focus on the user's intent, what they are trying to achieve, vs. the implementation details of how the skill works.\n- The description competes with other skills for Claude's attention — make it distinctive and immediately recognizable.\n- If you're getting lots of failures after repeated attempts, change things up. Try different sentence structures or wordings.\n\nI'd encourage you to be creative and mix up the style in different iterations since you'll have multiple opportunities to try different approaches and we'll just grab the highest-scoring one at the end. \n\nPlease respond with only the new description text in tags, nothing else.\"\"\"\n\n text = _call_claude(prompt, model)\n\n match = re.search(r\"(.*?)\", text, re.DOTALL)\n description = match.group(1).strip().strip('\"') if match else text.strip().strip('\"')\n\n transcript: dict = {\n \"iteration\": iteration,\n \"prompt\": prompt,\n \"response\": text,\n \"parsed_description\": description,\n \"char_count\": len(description),\n \"over_limit\": len(description) > 1024,\n }\n\n # Safety net: the prompt already states the 1024-char hard limit, but if\n # the model blew past it anyway, make one fresh single-turn call that\n # quotes the too-long version and asks for a shorter rewrite. (The old\n # SDK path did this as a true multi-turn; `claude -p` is one-shot, so we\n # inline the prior output into the new prompt instead.)\n if len(description) > 1024:\n shorten_prompt = (\n f\"{prompt}\\n\\n\"\n f\"---\\n\\n\"\n f\"A previous attempt produced this description, which at \"\n f\"{len(description)} characters is over the 1024-character hard limit:\\n\\n\"\n f'\"{description}\"\\n\\n'\n f\"Rewrite it to be under 1024 characters while keeping the most \"\n f\"important trigger words and intent coverage. Respond with only \"\n f\"the new description in tags.\"\n )\n shorten_text = _call_claude(shorten_prompt, model)\n match = re.search(r\"(.*?)\", shorten_text, re.DOTALL)\n shortened = match.group(1).strip().strip('\"') if match else shorten_text.strip().strip('\"')\n\n transcript[\"rewrite_prompt\"] = shorten_prompt\n transcript[\"rewrite_response\"] = shorten_text\n transcript[\"rewrite_description\"] = shortened\n transcript[\"rewrite_char_count\"] = len(shortened)\n description = shortened\n\n transcript[\"final_description\"] = description\n\n if log_dir:\n log_dir.mkdir(parents=True, exist_ok=True)\n log_file = log_dir / f\"improve_iter_{iteration or 'unknown'}.json\"\n log_file.write_text(json.dumps(transcript, indent=2))\n\n return description\n\n\ndef main():\n parser = argparse.ArgumentParser(description=\"Improve a skill description based on eval results\")\n parser.add_argument(\"--eval-results\", required=True, help=\"Path to eval results JSON (from run_eval.py)\")\n parser.add_argument(\"--skill-path\", required=True, help=\"Path to skill directory\")\n parser.add_argument(\"--history\", default=None, help=\"Path to history JSON (previous attempts)\")\n parser.add_argument(\"--model\", required=True, help=\"Model for improvement\")\n parser.add_argument(\"--verbose\", action=\"store_true\", help=\"Print thinking to stderr\")\n args = parser.parse_args()\n\n skill_path = Path(args.skill_path)\n if not (skill_path / \"SKILL.md\").exists():\n print(f\"Error: No SKILL.md found at {skill_path}\", file=sys.stderr)\n sys.exit(1)\n\n eval_results = json.loads(Path(args.eval_results).read_text())\n history = []\n if args.history:\n history = json.loads(Path(args.history).read_text())\n\n name, _, content = parse_skill_md(skill_path)\n current_description = eval_results[\"description\"]\n\n if args.verbose:\n print(f\"Current: {current_description}\", file=sys.stderr)\n print(f\"Score: {eval_results['summary']['passed']}/{eval_results['summary']['total']}\", file=sys.stderr)\n\n new_description = improve_description(\n skill_name=name,\n skill_content=content,\n current_description=current_description,\n eval_results=eval_results,\n history=history,\n model=args.model,\n )\n\n if args.verbose:\n print(f\"Improved: {new_description}\", file=sys.stderr)\n\n # Output as JSON with both the new description and updated history\n output = {\n \"description\": new_description,\n \"history\": history + [{\n \"description\": current_description,\n \"passed\": eval_results[\"summary\"][\"passed\"],\n \"failed\": eval_results[\"summary\"][\"failed\"],\n \"total\": eval_results[\"summary\"][\"total\"],\n \"results\": eval_results[\"results\"],\n }],\n }\n print(json.dumps(output, indent=2))\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": ".agents/skills/skill-creator/scripts/package_skill.py", "content": "#!/usr/bin/env python3\n\"\"\"\nSkill Packager - Creates a distributable .skill file of a skill folder\n\nUsage:\n python utils/package_skill.py [output-directory]\n\nExample:\n python utils/package_skill.py skills/public/my-skill\n python utils/package_skill.py skills/public/my-skill ./dist\n\"\"\"\n\nimport fnmatch\nimport sys\nimport zipfile\nfrom pathlib import Path\nfrom scripts.quick_validate import validate_skill\n\n# Patterns to exclude when packaging skills.\nEXCLUDE_DIRS = {\"__pycache__\", \"node_modules\"}\nEXCLUDE_GLOBS = {\"*.pyc\"}\nEXCLUDE_FILES = {\".DS_Store\"}\n# Directories excluded only at the skill root (not when nested deeper).\nROOT_EXCLUDE_DIRS = {\"evals\"}\n\n\ndef should_exclude(rel_path: Path) -> bool:\n \"\"\"Check if a path should be excluded from packaging.\"\"\"\n parts = rel_path.parts\n if any(part in EXCLUDE_DIRS for part in parts):\n return True\n # rel_path is relative to skill_path.parent, so parts[0] is the skill\n # folder name and parts[1] (if present) is the first subdir.\n if len(parts) > 1 and parts[1] in ROOT_EXCLUDE_DIRS:\n return True\n name = rel_path.name\n if name in EXCLUDE_FILES:\n return True\n return any(fnmatch.fnmatch(name, pat) for pat in EXCLUDE_GLOBS)\n\n\ndef package_skill(skill_path, output_dir=None):\n \"\"\"\n Package a skill folder into a .skill file.\n\n Args:\n skill_path: Path to the skill folder\n output_dir: Optional output directory for the .skill file (defaults to current directory)\n\n Returns:\n Path to the created .skill file, or None if error\n \"\"\"\n skill_path = Path(skill_path).resolve()\n\n # Validate skill folder exists\n if not skill_path.exists():\n print(f\"❌ Error: Skill folder not found: {skill_path}\")\n return None\n\n if not skill_path.is_dir():\n print(f\"❌ Error: Path is not a directory: {skill_path}\")\n return None\n\n # Validate SKILL.md exists\n skill_md = skill_path / \"SKILL.md\"\n if not skill_md.exists():\n print(f\"❌ Error: SKILL.md not found in {skill_path}\")\n return None\n\n # Run validation before packaging\n print(\"🔍 Validating skill...\")\n valid, message = validate_skill(skill_path)\n if not valid:\n print(f\"❌ Validation failed: {message}\")\n print(\" Please fix the validation errors before packaging.\")\n return None\n print(f\"✅ {message}\\n\")\n\n # Determine output location\n skill_name = skill_path.name\n if output_dir:\n output_path = Path(output_dir).resolve()\n output_path.mkdir(parents=True, exist_ok=True)\n else:\n output_path = Path.cwd()\n\n skill_filename = output_path / f\"{skill_name}.skill\"\n\n # Create the .skill file (zip format)\n try:\n with zipfile.ZipFile(skill_filename, 'w', zipfile.ZIP_DEFLATED) as zipf:\n # Walk through the skill directory, excluding build artifacts\n for file_path in skill_path.rglob('*'):\n if not file_path.is_file():\n continue\n arcname = file_path.relative_to(skill_path.parent)\n if should_exclude(arcname):\n print(f\" Skipped: {arcname}\")\n continue\n zipf.write(file_path, arcname)\n print(f\" Added: {arcname}\")\n\n print(f\"\\n✅ Successfully packaged skill to: {skill_filename}\")\n return skill_filename\n\n except Exception as e:\n print(f\"❌ Error creating .skill file: {e}\")\n return None\n\n\ndef main():\n if len(sys.argv) < 2:\n print(\"Usage: python utils/package_skill.py [output-directory]\")\n print(\"\\nExample:\")\n print(\" python utils/package_skill.py skills/public/my-skill\")\n print(\" python utils/package_skill.py skills/public/my-skill ./dist\")\n sys.exit(1)\n\n skill_path = sys.argv[1]\n output_dir = sys.argv[2] if len(sys.argv) > 2 else None\n\n print(f\"📦 Packaging skill: {skill_path}\")\n if output_dir:\n print(f\" Output directory: {output_dir}\")\n print()\n\n result = package_skill(skill_path, output_dir)\n\n if result:\n sys.exit(0)\n else:\n sys.exit(1)\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": ".agents/skills/skill-creator/scripts/quick_validate.py", "content": "#!/usr/bin/env python3\n\"\"\"\nQuick validation script for skills - minimal version\n\"\"\"\n\nimport sys\nimport os\nimport re\nimport yaml\nfrom pathlib import Path\n\ndef validate_skill(skill_path):\n \"\"\"Basic validation of a skill\"\"\"\n skill_path = Path(skill_path)\n\n # Check SKILL.md exists\n skill_md = skill_path / 'SKILL.md'\n if not skill_md.exists():\n return False, \"SKILL.md not found\"\n\n # Read and validate frontmatter\n content = skill_md.read_text()\n if not content.startswith('---'):\n return False, \"No YAML frontmatter found\"\n\n # Extract frontmatter\n match = re.match(r'^---\\n(.*?)\\n---', content, re.DOTALL)\n if not match:\n return False, \"Invalid frontmatter format\"\n\n frontmatter_text = match.group(1)\n\n # Parse YAML frontmatter\n try:\n frontmatter = yaml.safe_load(frontmatter_text)\n if not isinstance(frontmatter, dict):\n return False, \"Frontmatter must be a YAML dictionary\"\n except yaml.YAMLError as e:\n return False, f\"Invalid YAML in frontmatter: {e}\"\n\n # Define allowed properties\n ALLOWED_PROPERTIES = {'name', 'description', 'license', 'allowed-tools', 'metadata', 'compatibility'}\n\n # Check for unexpected properties (excluding nested keys under metadata)\n unexpected_keys = set(frontmatter.keys()) - ALLOWED_PROPERTIES\n if unexpected_keys:\n return False, (\n f\"Unexpected key(s) in SKILL.md frontmatter: {', '.join(sorted(unexpected_keys))}. \"\n f\"Allowed properties are: {', '.join(sorted(ALLOWED_PROPERTIES))}\"\n )\n\n # Check required fields\n if 'name' not in frontmatter:\n return False, \"Missing 'name' in frontmatter\"\n if 'description' not in frontmatter:\n return False, \"Missing 'description' in frontmatter\"\n\n # Extract name for validation\n name = frontmatter.get('name', '')\n if not isinstance(name, str):\n return False, f\"Name must be a string, got {type(name).__name__}\"\n name = name.strip()\n if name:\n # Check naming convention (kebab-case: lowercase with hyphens)\n if not re.match(r'^[a-z0-9-]+$', name):\n return False, f\"Name '{name}' should be kebab-case (lowercase letters, digits, and hyphens only)\"\n if name.startswith('-') or name.endswith('-') or '--' in name:\n return False, f\"Name '{name}' cannot start/end with hyphen or contain consecutive hyphens\"\n # Check name length (max 64 characters per spec)\n if len(name) > 64:\n return False, f\"Name is too long ({len(name)} characters). Maximum is 64 characters.\"\n\n # Extract and validate description\n description = frontmatter.get('description', '')\n if not isinstance(description, str):\n return False, f\"Description must be a string, got {type(description).__name__}\"\n description = description.strip()\n if description:\n # Check for angle brackets\n if '<' in description or '>' in description:\n return False, \"Description cannot contain angle brackets (< or >)\"\n # Check description length (max 1024 characters per spec)\n if len(description) > 1024:\n return False, f\"Description is too long ({len(description)} characters). Maximum is 1024 characters.\"\n\n # Validate compatibility field if present (optional)\n compatibility = frontmatter.get('compatibility', '')\n if compatibility:\n if not isinstance(compatibility, str):\n return False, f\"Compatibility must be a string, got {type(compatibility).__name__}\"\n if len(compatibility) > 500:\n return False, f\"Compatibility is too long ({len(compatibility)} characters). Maximum is 500 characters.\"\n\n return True, \"Skill is valid!\"\n\nif __name__ == \"__main__\":\n if len(sys.argv) != 2:\n print(\"Usage: python quick_validate.py \")\n sys.exit(1)\n \n valid, message = validate_skill(sys.argv[1])\n print(message)\n sys.exit(0 if valid else 1)" }, { "path": ".agents/skills/skill-creator/scripts/run_eval.py", "content": "#!/usr/bin/env python3\n\"\"\"Run trigger evaluation for a skill description.\n\nTests whether a skill's description causes Claude to trigger (read the skill)\nfor a set of queries. Outputs results as JSON.\n\"\"\"\n\nimport argparse\nimport json\nimport os\nimport select\nimport subprocess\nimport sys\nimport time\nimport uuid\nfrom concurrent.futures import ProcessPoolExecutor, as_completed\nfrom pathlib import Path\n\nfrom scripts.utils import parse_skill_md\n\n\ndef find_project_root() -> Path:\n \"\"\"Find the project root by walking up from cwd looking for .claude/.\n\n Mimics how Claude Code discovers its project root, so the command file\n we create ends up where claude -p will look for it.\n \"\"\"\n current = Path.cwd()\n for parent in [current, *current.parents]:\n if (parent / \".claude\").is_dir():\n return parent\n return current\n\n\ndef run_single_query(\n query: str,\n skill_name: str,\n skill_description: str,\n timeout: int,\n project_root: str,\n model: str | None = None,\n) -> bool:\n \"\"\"Run a single query and return whether the skill was triggered.\n\n Creates a command file in .claude/commands/ so it appears in Claude's\n available_skills list, then runs `claude -p` with the raw query.\n Uses --include-partial-messages to detect triggering early from\n stream events (content_block_start) rather than waiting for the\n full assistant message, which only arrives after tool execution.\n \"\"\"\n unique_id = uuid.uuid4().hex[:8]\n clean_name = f\"{skill_name}-skill-{unique_id}\"\n project_commands_dir = Path(project_root) / \".claude\" / \"commands\"\n command_file = project_commands_dir / f\"{clean_name}.md\"\n\n try:\n project_commands_dir.mkdir(parents=True, exist_ok=True)\n # Use YAML block scalar to avoid breaking on quotes in description\n indented_desc = \"\\n \".join(skill_description.split(\"\\n\"))\n command_content = (\n f\"---\\n\"\n f\"description: |\\n\"\n f\" {indented_desc}\\n\"\n f\"---\\n\\n\"\n f\"# {skill_name}\\n\\n\"\n f\"This skill handles: {skill_description}\\n\"\n )\n command_file.write_text(command_content)\n\n cmd = [\n \"claude\",\n \"-p\", query,\n \"--output-format\", \"stream-json\",\n \"--verbose\",\n \"--include-partial-messages\",\n ]\n if model:\n cmd.extend([\"--model\", model])\n\n # Remove CLAUDECODE env var to allow nesting claude -p inside a\n # Claude Code session. The guard is for interactive terminal conflicts;\n # programmatic subprocess usage is safe.\n env = {k: v for k, v in os.environ.items() if k != \"CLAUDECODE\"}\n\n process = subprocess.Popen(\n cmd,\n stdout=subprocess.PIPE,\n stderr=subprocess.DEVNULL,\n cwd=project_root,\n env=env,\n )\n\n triggered = False\n start_time = time.time()\n buffer = \"\"\n # Track state for stream event detection\n pending_tool_name = None\n accumulated_json = \"\"\n\n try:\n while time.time() - start_time < timeout:\n if process.poll() is not None:\n remaining = process.stdout.read()\n if remaining:\n buffer += remaining.decode(\"utf-8\", errors=\"replace\")\n break\n\n ready, _, _ = select.select([process.stdout], [], [], 1.0)\n if not ready:\n continue\n\n chunk = os.read(process.stdout.fileno(), 8192)\n if not chunk:\n break\n buffer += chunk.decode(\"utf-8\", errors=\"replace\")\n\n while \"\\n\" in buffer:\n line, buffer = buffer.split(\"\\n\", 1)\n line = line.strip()\n if not line:\n continue\n\n try:\n event = json.loads(line)\n except json.JSONDecodeError:\n continue\n\n # Early detection via stream events\n if event.get(\"type\") == \"stream_event\":\n se = event.get(\"event\", {})\n se_type = se.get(\"type\", \"\")\n\n if se_type == \"content_block_start\":\n cb = se.get(\"content_block\", {})\n if cb.get(\"type\") == \"tool_use\":\n tool_name = cb.get(\"name\", \"\")\n if tool_name in (\"Skill\", \"Read\"):\n pending_tool_name = tool_name\n accumulated_json = \"\"\n else:\n return False\n\n elif se_type == \"content_block_delta\" and pending_tool_name:\n delta = se.get(\"delta\", {})\n if delta.get(\"type\") == \"input_json_delta\":\n accumulated_json += delta.get(\"partial_json\", \"\")\n if clean_name in accumulated_json:\n return True\n\n elif se_type in (\"content_block_stop\", \"message_stop\"):\n if pending_tool_name:\n return clean_name in accumulated_json\n if se_type == \"message_stop\":\n return False\n\n # Fallback: full assistant message\n elif event.get(\"type\") == \"assistant\":\n message = event.get(\"message\", {})\n for content_item in message.get(\"content\", []):\n if content_item.get(\"type\") != \"tool_use\":\n continue\n tool_name = content_item.get(\"name\", \"\")\n tool_input = content_item.get(\"input\", {})\n if tool_name == \"Skill\" and clean_name in tool_input.get(\"skill\", \"\"):\n triggered = True\n elif tool_name == \"Read\" and clean_name in tool_input.get(\"file_path\", \"\"):\n triggered = True\n return triggered\n\n elif event.get(\"type\") == \"result\":\n return triggered\n finally:\n # Clean up process on any exit path (return, exception, timeout)\n if process.poll() is None:\n process.kill()\n process.wait()\n\n return triggered\n finally:\n if command_file.exists():\n command_file.unlink()\n\n\ndef run_eval(\n eval_set: list[dict],\n skill_name: str,\n description: str,\n num_workers: int,\n timeout: int,\n project_root: Path,\n runs_per_query: int = 1,\n trigger_threshold: float = 0.5,\n model: str | None = None,\n) -> dict:\n \"\"\"Run the full eval set and return results.\"\"\"\n results = []\n\n with ProcessPoolExecutor(max_workers=num_workers) as executor:\n future_to_info = {}\n for item in eval_set:\n for run_idx in range(runs_per_query):\n future = executor.submit(\n run_single_query,\n item[\"query\"],\n skill_name,\n description,\n timeout,\n str(project_root),\n model,\n )\n future_to_info[future] = (item, run_idx)\n\n query_triggers: dict[str, list[bool]] = {}\n query_items: dict[str, dict] = {}\n for future in as_completed(future_to_info):\n item, _ = future_to_info[future]\n query = item[\"query\"]\n query_items[query] = item\n if query not in query_triggers:\n query_triggers[query] = []\n try:\n query_triggers[query].append(future.result())\n except Exception as e:\n print(f\"Warning: query failed: {e}\", file=sys.stderr)\n query_triggers[query].append(False)\n\n for query, triggers in query_triggers.items():\n item = query_items[query]\n trigger_rate = sum(triggers) / len(triggers)\n should_trigger = item[\"should_trigger\"]\n if should_trigger:\n did_pass = trigger_rate >= trigger_threshold\n else:\n did_pass = trigger_rate < trigger_threshold\n results.append({\n \"query\": query,\n \"should_trigger\": should_trigger,\n \"trigger_rate\": trigger_rate,\n \"triggers\": sum(triggers),\n \"runs\": len(triggers),\n \"pass\": did_pass,\n })\n\n passed = sum(1 for r in results if r[\"pass\"])\n total = len(results)\n\n return {\n \"skill_name\": skill_name,\n \"description\": description,\n \"results\": results,\n \"summary\": {\n \"total\": total,\n \"passed\": passed,\n \"failed\": total - passed,\n },\n }\n\n\ndef main():\n parser = argparse.ArgumentParser(description=\"Run trigger evaluation for a skill description\")\n parser.add_argument(\"--eval-set\", required=True, help=\"Path to eval set JSON file\")\n parser.add_argument(\"--skill-path\", required=True, help=\"Path to skill directory\")\n parser.add_argument(\"--description\", default=None, help=\"Override description to test\")\n parser.add_argument(\"--num-workers\", type=int, default=10, help=\"Number of parallel workers\")\n parser.add_argument(\"--timeout\", type=int, default=30, help=\"Timeout per query in seconds\")\n parser.add_argument(\"--runs-per-query\", type=int, default=3, help=\"Number of runs per query\")\n parser.add_argument(\"--trigger-threshold\", type=float, default=0.5, help=\"Trigger rate threshold\")\n parser.add_argument(\"--model\", default=None, help=\"Model to use for claude -p (default: user's configured model)\")\n parser.add_argument(\"--verbose\", action=\"store_true\", help=\"Print progress to stderr\")\n args = parser.parse_args()\n\n eval_set = json.loads(Path(args.eval_set).read_text())\n skill_path = Path(args.skill_path)\n\n if not (skill_path / \"SKILL.md\").exists():\n print(f\"Error: No SKILL.md found at {skill_path}\", file=sys.stderr)\n sys.exit(1)\n\n name, original_description, content = parse_skill_md(skill_path)\n description = args.description or original_description\n project_root = find_project_root()\n\n if args.verbose:\n print(f\"Evaluating: {description}\", file=sys.stderr)\n\n output = run_eval(\n eval_set=eval_set,\n skill_name=name,\n description=description,\n num_workers=args.num_workers,\n timeout=args.timeout,\n project_root=project_root,\n runs_per_query=args.runs_per_query,\n trigger_threshold=args.trigger_threshold,\n model=args.model,\n )\n\n if args.verbose:\n summary = output[\"summary\"]\n print(f\"Results: {summary['passed']}/{summary['total']} passed\", file=sys.stderr)\n for r in output[\"results\"]:\n status = \"PASS\" if r[\"pass\"] else \"FAIL\"\n rate_str = f\"{r['triggers']}/{r['runs']}\"\n print(f\" [{status}] rate={rate_str} expected={r['should_trigger']}: {r['query'][:70]}\", file=sys.stderr)\n\n print(json.dumps(output, indent=2))\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": ".agents/skills/skill-creator/scripts/run_loop.py", "content": "#!/usr/bin/env python3\n\"\"\"Run the eval + improve loop until all pass or max iterations reached.\n\nCombines run_eval.py and improve_description.py in a loop, tracking history\nand returning the best description found. Supports train/test split to prevent\noverfitting.\n\"\"\"\n\nimport argparse\nimport json\nimport random\nimport sys\nimport tempfile\nimport time\nimport webbrowser\nfrom pathlib import Path\n\nfrom scripts.generate_report import generate_html\nfrom scripts.improve_description import improve_description\nfrom scripts.run_eval import find_project_root, run_eval\nfrom scripts.utils import parse_skill_md\n\n\ndef split_eval_set(eval_set: list[dict], holdout: float, seed: int = 42) -> tuple[list[dict], list[dict]]:\n \"\"\"Split eval set into train and test sets, stratified by should_trigger.\"\"\"\n random.seed(seed)\n\n # Separate by should_trigger\n trigger = [e for e in eval_set if e[\"should_trigger\"]]\n no_trigger = [e for e in eval_set if not e[\"should_trigger\"]]\n\n # Shuffle each group\n random.shuffle(trigger)\n random.shuffle(no_trigger)\n\n # Calculate split points\n n_trigger_test = max(1, int(len(trigger) * holdout))\n n_no_trigger_test = max(1, int(len(no_trigger) * holdout))\n\n # Split\n test_set = trigger[:n_trigger_test] + no_trigger[:n_no_trigger_test]\n train_set = trigger[n_trigger_test:] + no_trigger[n_no_trigger_test:]\n\n return train_set, test_set\n\n\ndef run_loop(\n eval_set: list[dict],\n skill_path: Path,\n description_override: str | None,\n num_workers: int,\n timeout: int,\n max_iterations: int,\n runs_per_query: int,\n trigger_threshold: float,\n holdout: float,\n model: str,\n verbose: bool,\n live_report_path: Path | None = None,\n log_dir: Path | None = None,\n) -> dict:\n \"\"\"Run the eval + improvement loop.\"\"\"\n project_root = find_project_root()\n name, original_description, content = parse_skill_md(skill_path)\n current_description = description_override or original_description\n\n # Split into train/test if holdout > 0\n if holdout > 0:\n train_set, test_set = split_eval_set(eval_set, holdout)\n if verbose:\n print(f\"Split: {len(train_set)} train, {len(test_set)} test (holdout={holdout})\", file=sys.stderr)\n else:\n train_set = eval_set\n test_set = []\n\n history = []\n exit_reason = \"unknown\"\n\n for iteration in range(1, max_iterations + 1):\n if verbose:\n print(f\"\\n{'='*60}\", file=sys.stderr)\n print(f\"Iteration {iteration}/{max_iterations}\", file=sys.stderr)\n print(f\"Description: {current_description}\", file=sys.stderr)\n print(f\"{'='*60}\", file=sys.stderr)\n\n # Evaluate train + test together in one batch for parallelism\n all_queries = train_set + test_set\n t0 = time.time()\n all_results = run_eval(\n eval_set=all_queries,\n skill_name=name,\n description=current_description,\n num_workers=num_workers,\n timeout=timeout,\n project_root=project_root,\n runs_per_query=runs_per_query,\n trigger_threshold=trigger_threshold,\n model=model,\n )\n eval_elapsed = time.time() - t0\n\n # Split results back into train/test by matching queries\n train_queries_set = {q[\"query\"] for q in train_set}\n train_result_list = [r for r in all_results[\"results\"] if r[\"query\"] in train_queries_set]\n test_result_list = [r for r in all_results[\"results\"] if r[\"query\"] not in train_queries_set]\n\n train_passed = sum(1 for r in train_result_list if r[\"pass\"])\n train_total = len(train_result_list)\n train_summary = {\"passed\": train_passed, \"failed\": train_total - train_passed, \"total\": train_total}\n train_results = {\"results\": train_result_list, \"summary\": train_summary}\n\n if test_set:\n test_passed = sum(1 for r in test_result_list if r[\"pass\"])\n test_total = len(test_result_list)\n test_summary = {\"passed\": test_passed, \"failed\": test_total - test_passed, \"total\": test_total}\n test_results = {\"results\": test_result_list, \"summary\": test_summary}\n else:\n test_results = None\n test_summary = None\n\n history.append({\n \"iteration\": iteration,\n \"description\": current_description,\n \"train_passed\": train_summary[\"passed\"],\n \"train_failed\": train_summary[\"failed\"],\n \"train_total\": train_summary[\"total\"],\n \"train_results\": train_results[\"results\"],\n \"test_passed\": test_summary[\"passed\"] if test_summary else None,\n \"test_failed\": test_summary[\"failed\"] if test_summary else None,\n \"test_total\": test_summary[\"total\"] if test_summary else None,\n \"test_results\": test_results[\"results\"] if test_results else None,\n # For backward compat with report generator\n \"passed\": train_summary[\"passed\"],\n \"failed\": train_summary[\"failed\"],\n \"total\": train_summary[\"total\"],\n \"results\": train_results[\"results\"],\n })\n\n # Write live report if path provided\n if live_report_path:\n partial_output = {\n \"original_description\": original_description,\n \"best_description\": current_description,\n \"best_score\": \"in progress\",\n \"iterations_run\": len(history),\n \"holdout\": holdout,\n \"train_size\": len(train_set),\n \"test_size\": len(test_set),\n \"history\": history,\n }\n live_report_path.write_text(generate_html(partial_output, auto_refresh=True, skill_name=name))\n\n if verbose:\n def print_eval_stats(label, results, elapsed):\n pos = [r for r in results if r[\"should_trigger\"]]\n neg = [r for r in results if not r[\"should_trigger\"]]\n tp = sum(r[\"triggers\"] for r in pos)\n pos_runs = sum(r[\"runs\"] for r in pos)\n fn = pos_runs - tp\n fp = sum(r[\"triggers\"] for r in neg)\n neg_runs = sum(r[\"runs\"] for r in neg)\n tn = neg_runs - fp\n total = tp + tn + fp + fn\n precision = tp / (tp + fp) if (tp + fp) > 0 else 1.0\n recall = tp / (tp + fn) if (tp + fn) > 0 else 1.0\n accuracy = (tp + tn) / total if total > 0 else 0.0\n print(f\"{label}: {tp+tn}/{total} correct, precision={precision:.0%} recall={recall:.0%} accuracy={accuracy:.0%} ({elapsed:.1f}s)\", file=sys.stderr)\n for r in results:\n status = \"PASS\" if r[\"pass\"] else \"FAIL\"\n rate_str = f\"{r['triggers']}/{r['runs']}\"\n print(f\" [{status}] rate={rate_str} expected={r['should_trigger']}: {r['query'][:60]}\", file=sys.stderr)\n\n print_eval_stats(\"Train\", train_results[\"results\"], eval_elapsed)\n if test_summary:\n print_eval_stats(\"Test \", test_results[\"results\"], 0)\n\n if train_summary[\"failed\"] == 0:\n exit_reason = f\"all_passed (iteration {iteration})\"\n if verbose:\n print(f\"\\nAll train queries passed on iteration {iteration}!\", file=sys.stderr)\n break\n\n if iteration == max_iterations:\n exit_reason = f\"max_iterations ({max_iterations})\"\n if verbose:\n print(f\"\\nMax iterations reached ({max_iterations}).\", file=sys.stderr)\n break\n\n # Improve the description based on train results\n if verbose:\n print(f\"\\nImproving description...\", file=sys.stderr)\n\n t0 = time.time()\n # Strip test scores from history so improvement model can't see them\n blinded_history = [\n {k: v for k, v in h.items() if not k.startswith(\"test_\")}\n for h in history\n ]\n new_description = improve_description(\n skill_name=name,\n skill_content=content,\n current_description=current_description,\n eval_results=train_results,\n history=blinded_history,\n model=model,\n log_dir=log_dir,\n iteration=iteration,\n )\n improve_elapsed = time.time() - t0\n\n if verbose:\n print(f\"Proposed ({improve_elapsed:.1f}s): {new_description}\", file=sys.stderr)\n\n current_description = new_description\n\n # Find the best iteration by TEST score (or train if no test set)\n if test_set:\n best = max(history, key=lambda h: h[\"test_passed\"] or 0)\n best_score = f\"{best['test_passed']}/{best['test_total']}\"\n else:\n best = max(history, key=lambda h: h[\"train_passed\"])\n best_score = f\"{best['train_passed']}/{best['train_total']}\"\n\n if verbose:\n print(f\"\\nExit reason: {exit_reason}\", file=sys.stderr)\n print(f\"Best score: {best_score} (iteration {best['iteration']})\", file=sys.stderr)\n\n return {\n \"exit_reason\": exit_reason,\n \"original_description\": original_description,\n \"best_description\": best[\"description\"],\n \"best_score\": best_score,\n \"best_train_score\": f\"{best['train_passed']}/{best['train_total']}\",\n \"best_test_score\": f\"{best['test_passed']}/{best['test_total']}\" if test_set else None,\n \"final_description\": current_description,\n \"iterations_run\": len(history),\n \"holdout\": holdout,\n \"train_size\": len(train_set),\n \"test_size\": len(test_set),\n \"history\": history,\n }\n\n\ndef main():\n parser = argparse.ArgumentParser(description=\"Run eval + improve loop\")\n parser.add_argument(\"--eval-set\", required=True, help=\"Path to eval set JSON file\")\n parser.add_argument(\"--skill-path\", required=True, help=\"Path to skill directory\")\n parser.add_argument(\"--description\", default=None, help=\"Override starting description\")\n parser.add_argument(\"--num-workers\", type=int, default=10, help=\"Number of parallel workers\")\n parser.add_argument(\"--timeout\", type=int, default=30, help=\"Timeout per query in seconds\")\n parser.add_argument(\"--max-iterations\", type=int, default=5, help=\"Max improvement iterations\")\n parser.add_argument(\"--runs-per-query\", type=int, default=3, help=\"Number of runs per query\")\n parser.add_argument(\"--trigger-threshold\", type=float, default=0.5, help=\"Trigger rate threshold\")\n parser.add_argument(\"--holdout\", type=float, default=0.4, help=\"Fraction of eval set to hold out for testing (0 to disable)\")\n parser.add_argument(\"--model\", required=True, help=\"Model for improvement\")\n parser.add_argument(\"--verbose\", action=\"store_true\", help=\"Print progress to stderr\")\n parser.add_argument(\"--report\", default=\"auto\", help=\"Generate HTML report at this path (default: 'auto' for temp file, 'none' to disable)\")\n parser.add_argument(\"--results-dir\", default=None, help=\"Save all outputs (results.json, report.html, log.txt) to a timestamped subdirectory here\")\n args = parser.parse_args()\n\n eval_set = json.loads(Path(args.eval_set).read_text())\n skill_path = Path(args.skill_path)\n\n if not (skill_path / \"SKILL.md\").exists():\n print(f\"Error: No SKILL.md found at {skill_path}\", file=sys.stderr)\n sys.exit(1)\n\n name, _, _ = parse_skill_md(skill_path)\n\n # Set up live report path\n if args.report != \"none\":\n if args.report == \"auto\":\n timestamp = time.strftime(\"%Y%m%d_%H%M%S\")\n live_report_path = Path(tempfile.gettempdir()) / f\"skill_description_report_{skill_path.name}_{timestamp}.html\"\n else:\n live_report_path = Path(args.report)\n # Open the report immediately so the user can watch\n live_report_path.write_text(\"

Starting optimization loop...

\")\n webbrowser.open(str(live_report_path))\n else:\n live_report_path = None\n\n # Determine output directory (create before run_loop so logs can be written)\n if args.results_dir:\n timestamp = time.strftime(\"%Y-%m-%d_%H%M%S\")\n results_dir = Path(args.results_dir) / timestamp\n results_dir.mkdir(parents=True, exist_ok=True)\n else:\n results_dir = None\n\n log_dir = results_dir / \"logs\" if results_dir else None\n\n output = run_loop(\n eval_set=eval_set,\n skill_path=skill_path,\n description_override=args.description,\n num_workers=args.num_workers,\n timeout=args.timeout,\n max_iterations=args.max_iterations,\n runs_per_query=args.runs_per_query,\n trigger_threshold=args.trigger_threshold,\n holdout=args.holdout,\n model=args.model,\n verbose=args.verbose,\n live_report_path=live_report_path,\n log_dir=log_dir,\n )\n\n # Save JSON output\n json_output = json.dumps(output, indent=2)\n print(json_output)\n if results_dir:\n (results_dir / \"results.json\").write_text(json_output)\n\n # Write final HTML report (without auto-refresh)\n if live_report_path:\n live_report_path.write_text(generate_html(output, auto_refresh=False, skill_name=name))\n print(f\"\\nReport: {live_report_path}\", file=sys.stderr)\n\n if results_dir and live_report_path:\n (results_dir / \"report.html\").write_text(generate_html(output, auto_refresh=False, skill_name=name))\n\n if results_dir:\n print(f\"Results saved to: {results_dir}\", file=sys.stderr)\n\n\nif __name__ == \"__main__\":\n main()\n" }, { "path": ".agents/skills/skill-creator/scripts/utils.py", "content": "\"\"\"Shared utilities for skill-creator scripts.\"\"\"\n\nfrom pathlib import Path\n\n\n\ndef parse_skill_md(skill_path: Path) -> tuple[str, str, str]:\n \"\"\"Parse a SKILL.md file, returning (name, description, full_content).\"\"\"\n content = (skill_path / \"SKILL.md\").read_text()\n lines = content.split(\"\\n\")\n\n if lines[0].strip() != \"---\":\n raise ValueError(\"SKILL.md missing frontmatter (no opening ---)\")\n\n end_idx = None\n for i, line in enumerate(lines[1:], start=1):\n if line.strip() == \"---\":\n end_idx = i\n break\n\n if end_idx is None:\n raise ValueError(\"SKILL.md missing frontmatter (no closing ---)\")\n\n name = \"\"\n description = \"\"\n frontmatter_lines = lines[1:end_idx]\n i = 0\n while i < len(frontmatter_lines):\n line = frontmatter_lines[i]\n if line.startswith(\"name:\"):\n name = line[len(\"name:\"):].strip().strip('\"').strip(\"'\")\n elif line.startswith(\"description:\"):\n value = line[len(\"description:\"):].strip()\n # Handle YAML multiline indicators (>, |, >-, |-)\n if value in (\">\", \"|\", \">-\", \"|-\"):\n continuation_lines: list[str] = []\n i += 1\n while i < len(frontmatter_lines) and (frontmatter_lines[i].startswith(\" \") or frontmatter_lines[i].startswith(\"\\t\")):\n continuation_lines.append(frontmatter_lines[i].strip())\n i += 1\n description = \" \".join(continuation_lines)\n continue\n else:\n description = value.strip('\"').strip(\"'\")\n i += 1\n\n return name, description, content\n" }, { "path": ".browserslistrc", "content": "firefox esr\nios>=11\nlast 1 safari version\nlast 2 chrome versions\nlast 2 edge versions\nlast 2 firefox versions\nlast 2 opera versions\n" }, { "path": ".claude/agents/design-system-researcher.md", "content": "---\nname: design-system-researcher\ndescription: Use this agent when you need to conduct in-depth research on a single open source design system or component library. This agent should be invoked when:\\n\\n\\nContext: User wants to understand how a specific design system implements its theming architecture.\\nuser: \"How does Material UI handle theming and dark mode\"\\nassistant: \"I'm going to use the Task tool to launch the design-system-researcher agent to conduct this research on Material UI.\"\\nThe user has explicitly requested research on Material UI's implementation, which is the primary use case for this agent.\\n\\n\\n\\nContext: User is exploring how a specific library implements a feature.\\nuser: \"Can you look into how Mantine offers style customizations for its progress bar component?\"\\nassistant: \"I'll use the design-system-researcher agent to investigate Mantine's Progress Bar component and create a detailed report on how Mantine built this component.\"\\nThis is a targeted research task about a specific design system's component that requires investigation and documentation.\\n\\n\\n\ntools: Read, Grep, Glob, Bash, BashOutput, Write, WebFetch, Task\npermissionMode: bypassPermissions\nmodel: opus\ncolor: cyan\n---\n\nYou are an expert design systems researcher and technical analyst with deep expertise in frontend architecture, React ecosystems, component library design patterns, and open source software. Your role is to conduct thorough, methodical research on a project, producing actionable insights and comprehensive documentation.\n\n## Your Research Mandate\n\nYou will receive two key inputs:\n\n1. **Target Project**: The name of the open source design system or component library to research (e.g., \"Material UI\", \"Ant Design\", \"Radix\", \"Base UI\", \"Mantine\")\n\n2. **Research Goal**: The specific aspect, pattern, or feature you need to investigate (e.g., \"theming architecture\", \"React component props\", \"component composition patterns\", \"styling solutions\")\n\n## Design Systems\n\nThese are the projects you may be tasked to research:\n\n### Ant Design\n\n- Name: Ant Design\n- Docs: https://ant.design/components/overview/\n- Repo: https://github.com/ant-design/ant-design.git\n- Branch: master\n- Component Paths:\n - components/\n\n### Material UI\n\n- Name: Material UI\n- Docs: https://mui.com/material-ui/llms.txt\n- Repo: https://github.com/mui/material-ui.git\n- Branch: master\n- Component Paths:\n - packages/mui-material/src/\n\n### Base UI\n\n- Name: Base UI\n- Docs: https://base-ui.com/llms.txt\n- Repo: https://github.com/mui/base-ui.git\n- Branch: master\n- Component Paths:\n - packages/react/src/\n\n### Radix Primitives\n\n- Name: Radix\n- Docs: https://www.radix-ui.com/primitives/docs/overview/introduction\n- Repo: https://github.com/radix-ui/primitives.git\n- Branch: main\n- Component Paths:\n - packages/react/\n\n### Mantine\n\n- Name: Mantine\n- Docs: https://ui.mantine.dev/#main\n- Repo: https://github.com/mantinedev/mantine.git\n- Branch: master\n- Component Paths:\n - packages/@mantine/core/src/components/\n - packages/@mantine/dates/src/components/\n - packages/@mantine/carousel/src/\n - packages/@mantine/charts/src/\n - packages/@mantine/modals/src/\n\n### React Aria\n\n- Name: React Aria\n- Docs: https://react-aria.adobe.com/\n- Repo: https://github.com/adobe/react-spectrum.git\n- Branch: main\n- Component Paths:\n - packages/react-aria-components/src/\n\n### Tamagui\n\n- Name: Tamagui\n- Docs: https://tamagui.dev/docs/intro/introduction\n- Repo: https://github.com/tamagui/tamagui.git\n- Branch: master\n- Component Paths:\n - code/ui/\n\n## Research Methodology\n\nFollow this systematic approach:\n\n1. **Task Validation**\n - Identify the project you need to research from the `Design Systems` section above.\n - If the requested project is not in the `Design Systems` section above, abandon the research task\n - If the specified project cannot be found or is ambiguous, abandon the research task\n - If the theme of the research goal cannot be found within the project source code, abandon the research task\n\n2. **Environment Preparation**\n - Use the `git.repo-manager` skill (`.claude/skills/git.repo-manager/SKILL.md`) to ensure the project's repository is cloned and up to date in `temp/repo-cache/`.\n - Only manage the single repository you are researching.\n\n3. **Deep Technical Analysis**\n - Examine actual source code implementations, not just documentation\n - Use the `Component Paths` list from the `Design Systems` section above to focus your search on relevant files to the project\n - Identify key patterns, abstractions, and architectural decisions\n - Analyze how the project solves specific problems\n - Note any trade-offs, limitations, or known issues\n - Look for TypeScript types, interfaces, and API contracts\n - Understand the dependency footprint and external libraries used\n\n4. **Comparative Context**\n - When relevant, briefly note how this approach differs from common alternatives\n - Identify unique innovations or distinctive characteristics\n - Consider adoption complexity and developer experience implications\n\n5. **Practical Insights**\n - Extract concrete code examples that illustrate key concepts\n - Document actual APIs, prop interfaces, and usage patterns\n - Note configuration options and customization points\n - Identify best practices recommended by the maintainers\n\n## Research Report Structure\n\nCreate a comprehensive markdown report with the following structure:\n\n````markdown\n# [Project Name]: [Research Goal]\n\n## Executive Summary\n\n[2-3 sentences capturing the core findings and key takeaways]\n\n## Overview\n\n[Brief context about the project and the specific aspect being researched]\n\n## Key Findings\n\n### [Finding Category 1]\n\n[Detailed analysis with code examples where relevant]\n\n### [Finding Category 2]\n\n[Detailed analysis with code examples where relevant]\n\n[Continue with additional categories as needed]\n\n## Technical Implementation Details\n\n[Deep dive into how things work under the hood]\n\n## Code Examples\n\n```[language]\n[Concrete, runnable examples demonstrating key concepts]\n```\n\n## Strengths\n\n- [Specific advantage with explanation]\n- [Another strength]\n\n## Considerations & Trade-offs\n\n- [Potential limitation or complexity]\n- [Another consideration]\n\n## Relevance to the Coinbase Design System\n\n[How these findings might apply to or inform the Coinbase Design System]\n\n## References\n\n- [Link to source code]\n- [Link to documentation]\n- [Link to relevant discussions]\n````\n\n## File Management\n\n1. Reports should be organized in subdirectories within `.claude/research/` based on the research goal\n2. Convert the research goal into a kebab-case directory name (e.g., \"theming architecture\" → `theming-architecture`)\n3. Create your report in the research goal subdirectory: `.claude/research/[research-goal]/`\n4. Use a descriptive filename format: `[project]-[date].md`\n - Example: `.claude/research/theming-architecture/material-ui-2024-01-15.md`\n5. Ensure the directory exists before writing (create if needed)\n6. After completing your research and writing the report, explicitly communicate the full file path to the parent agent\n\n## Quality Standards\n\n- **Accuracy**: Verify all technical claims by examining actual source code\n- **Depth**: Go beyond surface-level documentation to understand implementation details\n- **Clarity**: Use precise technical language while remaining accessible\n- **Actionability**: Focus on insights that can inform design decisions\n- **Evidence**: Support claims with code examples, links, or quotes from official sources\n- **Brevity**: Be comprehensive but concise - every section should provide value\n\n## Communication Protocol\n\nWhen you complete your research:\n\n1. Confirm the report has been written successfully\n2. State the full path to the report file\n" }, { "path": ".claude/settings.json", "content": "{\n \"permissions\": {\n \"allow\": [\n \"WebFetch\",\n \"WebSearch\",\n \"Write\",\n \"Skill(cds-components)\",\n \"mcp__linear-server__get_issue\",\n \"mcp__linear-server__get_user\",\n \"mcp__linear-server__list_teams\",\n \"mcp__linear-server__list_issues\",\n \"mcp__linear-server__list_cycles\",\n \"mcp__figma-dev-mode-mcp-server__get_design_context\",\n \"mcp__figma-dev-mode-mcp-server__get_screenshot\",\n \"mcp__figma-dev-mode-mcp-server__get_metadata\",\n \"mcp__figma-dev-mode-mcp-server__get_variable_defs\",\n \"Bash(ls:*)\",\n \"Bash(find:*)\",\n \"Bash(yarn nx run-many:*)\",\n \"Bash(yarn nx run web:typecheck)\",\n \"Bash(yarn nx run mobile:typecheck)\",\n \"Bash(yarn nx run common:typecheck)\",\n \"Bash(yarn nx run web:test:*)\",\n \"Bash(yarn nx run mobile:test:*)\",\n \"Bash(yarn nx run common:test:*)\",\n \"Bash(yarn nx format:write)\",\n \"Bash(yarn nx format:check)\"\n ],\n \"deny\": [],\n \"ask\": []\n }\n}\n" }, { "path": ".claude/skills/components.best-practices/SKILL.md", "content": "---\nname: components.best-practices\ndescription: Use this skill whenever working on CDS React components in any package.\nuser-invocable: false\n---\n\n# React Component Development Rules\n\n## Component Development Workflow\n\n1. Research similar reference components and given requirements/description\n2. Optionally, ask clarifying questions about the component's requirements & behavior\n3. Implement the component with unit tests & stories on web first before proceeding to mobile if both platforms were requested.\n4. Never write figma code connect files unless explicitly instructed to do so.\n5. Follow remaining general coding standards and guidelines you've been given.\n\n## Reference Components\n\nThese high quality components demonstrate proper use of patterns/conventions:\n\n- **Select** (alpha/): generics, controlled/uncontrolled, compound architecture\n- **Stepper**: props-based defaults, metadata generics, compound components\n- **Carousel** (web): compound components, imperative handle, context + hook\n- **RollingNumber**: animation config extraction, measurement patterns\n- **SlideButton** (mobile): gesture handling, spring animations, accessibility actions\n\n## Organization\n\n### File Structure\n\nEvery main CDS component should live within its own folder:\n\n```\nComponentName/\n├── ComponentName.tsx # Main component file\n├── SubComponent.tsx # Supporting component (if needed)\n├── index.ts # Re-exports for public API\n├── __stories__/ # Storybook stories\n│ └── ComponentName.stories.tsx\n├── __tests__/ # Unit tests\n│ └── ComponentName.test.tsx\n├── __figma__/ # Figma Code Connect files\n│ └── ComponentName.figma.tsx\n```\n\n### Component Categories\n\nOrganize components into category folders:\n\n- `buttons` - Button, IconButton, SlideButton\n- `controls` - TextInput, Select, Checkbox, Radio, Switch\n- `cards` - Card, DataCard, ContentCard\n- `overlays` - Modal, Toast, Alert, Drawer\n- `layout` - Box, Stack, Divider\n- `typography` - Text, Heading\n- `icons` - Icon\n- `navigation` - Tabs, Breadcrumb\n\n## Component Conventions\n\n- **Memoize**: Always memoize components with React's memo HOC\n- **refs**: All components should accept a ref via React's forwardRef pattern\n- **Props documentation**: Every prop that does not have a falsy default must have JSDoc comments with `@default` tags\n- **Type exports**: Export both a `*BaseProps` and `*Props` type (e.g., `ButtonBaseProps`, `ButtonProps`)\n- **Style overrides**: All components MUST support a way to override styles (varries by web/mobile platform)\n- **testID**: Support `testID` prop on root element for every component\n- **Use design tokens**: Reference packages/common/src/core/theme.ts:57-331 as the definitive source for available token names\n- **Padding over margin**: Use padding in combination with flex gap to achieve spacing instead of margin.\n\n## Design Token System\n\n### Token Categories\n\nDesign tokens are defined in `packages/common/src/core/theme.ts`:\n\n- **Color**: fg, fgMuted, fgInverse, fgPrimary, bgPrimary, bgSecondary, bgNegative, bgPositive, etc.\n- **Space**: 0, 0.25, 0.5, 0.75, 1, 1.5, 2, 3, 4, 5, 6, 7, 8, 9, 10 (8px base unit)\n- **IconSize**: xs (12px), s (16px), m (24px), l (32px)\n- **AvatarSize**: s, m, l, xl, xxl, xxxl\n- **BorderWidth**: 0, 100, 200, 300, 400, 500\n- **BorderRadius**: 0, 100, 200, 300, 400, 500, 600, 700, 800, 900, 1000\n- **Font**: display1-3, title1-4, headline, body, label1-2, caption, legal\n- **Shadow**: elevation1, elevation2\n\n### Semantic Color System\n\nColors use a spectrum system with hue + step notation:\n\n- **Hues**: blue, green, orange, yellow, gray, indigo, pink, purple, red, teal, chartreuse\n- **Steps**: 0, 5, 10, 15, 20, 30, 40, 50, 60, 70, 80, 90, 100\n- **Example**: blue60 = Coinbase brand blue (#0052FF)\n\nSemantic tokens map to spectrum colors and adapt to light/dark mode:\n\n- `fgPrimary`: blue60 (light) / blue70 (dark)\n- `bgPrimary`: blue60 (light) / blue70 (dark)\n- `bgNegative`: red60 (both modes)\n- `bgPositive`: green60 (both modes)\n\n### Space Scale\n\n```typescript\nspace: {\n '0': 0, // 0px\n '0.25': 2, // 2px\n '0.5': 4, // 4px\n '0.75': 6, // 6px\n '1': 8, // 8px - base unit\n '1.5': 12, // 12px\n '2': 16, // 16px\n '3': 24, // 24px\n '4': 32, // 32px\n '5': 40, // 40px\n // ... up to 10 (80px)\n}\n```\n\n## Component Patterns\n\n### Compound Components\n\n- Break components down into discrete subcomponents (i.e. \"slots\")\n- Use this pattern for complex components with clear, distinct parts\n- Accept optional subcomponent props with sensible defaults using `*Component`/`Default*` naming:\n ```ts\n NavigationComponent = DefaultCarouselNavigation,\n PaginationComponent = DefaultCarouselPagination,\n ```\n- The names of classNames/styles keys must line up with the name of the subcomponents (e.g. `classNames.pagination`, `styles.pagination`).\n- Examples: Stepper, Carousel, Select (alpha)\n\n**Benefits:**\n\n- Complete customization without forking\n- Sensible defaults for common use case\n- Exported subcomponents for consumers to customize/wrap themselves\n\n### Context + Hook Pattern\n\n- Pair contexts with `use*Context()` hooks that throw descriptive errors on misuse:\n ```ts\n export const useCarouselContext = () => {\n const context = useContext(CarouselContext);\n if (!context) throw new Error('useCarouselContext must be used within Carousel');\n return context;\n };\n ```\n\n### Controlled/Uncontrolled Components\n\n- Support both patterns for input components; validate and throw if consumer mixes them (e.g., provides `value` but not `onChange`)\n- Use internal state with prop override: `const open = openProp ?? openInternal;`\n\n### Generics for Type Safety\n\n- Use generics for components with dynamic value types:\n ```ts\n type SelectComponent = (\n props: SelectProps,\n ) => React.ReactElement;\n ```\n- Examples: Select (alpha), Stepper\n\n### BaseProps & Props\n\n- Component modules encapsulate two prop Types: `*BaseProps` (platform-agnostic) and `*Props` (extends BaseProps with platform and component specific properties like `className`, `classNames`, `styles`, etc.)\n- Reuse other components' Types via utilities: `Pick` being preferred then secondarily `Omit`/`Exclude`\n- Compose prop types using Typescript intersections (`&`) in this order: (1) full types (2) Picks (3) Omits (4) other type literal(s):\n ```ts\n type MyComponentProps = BoxBaseProps &\n Pick &\n Omit & {\n propA: string;\n propB: number;\n };\n ```\n- When accepting components as props, define the contract types (`*Props`, `*Component`) in the main component file. These child component contracts do not use the `*BaseProps` pattern—only the main component needs BaseProps/Props separation. Default implementations can extend the contract with additional props in their own file:\n\n ```ts\n // In MyComponent.tsx - defines the contract\n type ChildProps = { id: string; label: ReactNode };\n type ChildComponent = React.FC;\n\n // In DefaultChild.tsx - extends for default implementation\n type DefaultChildProps = SharedProps & Omit & ChildProps;\n ```\n" }, { "path": ".claude/skills/components.styles/README.md", "content": "# components.styles agent skill\n\nThis skill may be invoked by the user following the examples below.\n\n**Usage:** `/components.styles [additional context]`\n\nExamples:\n\n- `/components.styles SlideButton`\n- `/components.styles Button add static classnames for sub elements`\n- `/components.styles Select add styles documentation`\n- `/components.styles Avatar mobile only`\n" }, { "path": ".claude/skills/components.styles/SKILL.md", "content": "---\nname: components.styles\ndescription: Guidelines writing styles API (styles, classNames, and static classNames) for a CDS component. Use this skill when adding customization options to a React component via `styles` or `classNames` props or when needing to update the docsite with component styles documentation.\nargument-hint: [additional context] (e.g., \"Button\", \"LineChart add real-time examples\")\n---\n\nGoal: Add styles API (styles, classNames, and static classNames) to a CDS component and/or update the component documentation with styles documentation.\n\nIf no component name is provided, ask the user which component they want to add styles to.\n\n## Step 1: Locate the Component\n\nFind the component source file:\n\n```bash\npackages/web/src/[source-category]/[ComponentName].tsx # for web\npackages/mobile/src/[source-category]/[ComponentName].tsx # for mobile\npackages/web-visualization/src/[source-category]/[ComponentName].tsx # for web visualization\npackages/mobile-visualization/src/[source-category]/[ComponentName].tsx # for mobile visualization\n```\n\n## Step 2: Evaluate Component Structure\n\n> **⚠️ IMPORTANT: Adding styles/classNames props is a commitment to the component's internal structure.**\n>\n> Before adding styles API, carefully review the component's JSX structure:\n>\n> - **Flag if the component could be simplified** (e.g., unnecessary wrappers, redundant containers)\n> - **Do NOT add styles to elements that may be refactored** - this creates breaking changes\n> - **Ask the user** if you notice the component structure could be improved before committing to it\n>\n> Once published, changing or removing selectors is a **breaking change** for consumers.\n\n## Step 3: Identify Styleable Elements\n\nReview the component's JSX to identify elements that should be targetable via styles/classNames:\n\n- **Root element**: The outermost container element\n- **Named sections**: Elements with semantic meaning (e.g., `start`, `content`, `end`, `header`, `footer`)\n- **Sub-components**: Internal elements that users might want to customize\n- **Conditional elements**: Elements that render based on props\n\n## Approved Selector Names\n\n> **IMPORTANT:** Before adding a new selector name not in this list, **get explicit confirmation from the user**.\n> When a new selector is approved, add it to this list.\n\n### Approved Selectors (alphabetical)\n\n| Selector | Description |\n| --------------------- | ----------------------------------------------------- |\n| `accessory` | Accessory element (e.g., chevron, icon at end) |\n| `activeIndicator` | Active indicator element (e.g., in tabs) |\n| `bottomContent` | Bottom section content |\n| `carousel` | Main carousel track element |\n| `carouselContainer` | Outer carousel container |\n| `childrenContainer` | Container wrapping children |\n| `content` | Main content area |\n| `contentContainer` | Container wrapping content |\n| `description` | Description text element |\n| `day` | Date cell in a calendar grid |\n| `end` | End slot content (e.g., actions, icons) |\n| `fill` | Fill/progress indicator within a track |\n| `header` | Header section |\n| `helperText` | Helper/assistive text below content |\n| `icon` | Icon element |\n| `intermediary` | Middle/intermediary element between sections |\n| `label` | Label text element |\n| `labels` | Container for multiple labels |\n| `logo` | Logo element |\n| `mainContent` | Primary content area |\n| `media` | Media element (image, avatar, icon) |\n| `navigation` | Navigation controls (e.g., prev/next buttons) |\n| `pagination` | Pagination indicators |\n| `pressable` | Pressable/interactive wrapper |\n| `progress` | Progress indicator element |\n| `progressBar` | ProgressBar sub-component within a composed component |\n| `root` | Root/outermost container element |\n| `start` | Start slot content (e.g., back button) |\n| `step` | Individual step element (in steppers) |\n| `substepContainer` | Container for nested sub-steps |\n| `subtitle` | Subtitle text element |\n| `tab` | Tab element (in tabs) |\n| `tabs` | Tabs container element |\n| `thumb` | Draggable thumb element (in sliders) |\n| `title` | Title text element |\n| `titleStack` | Stack containing title/subtitle/description |\n| `titleStackContainer` | Container wrapping titleStack |\n| `topContent` | Top section content |\n| `track` | Track/rail element (in progress bars, sliders) |\n| `trigger` | Trigger element that opens a dropdown/popover |\n\n## JSDoc Convention for Selector Descriptions\n\nSelector JSDoc comments describe **what the element is**, not what the prop does:\n\n- Sentence case, no trailing period\n- Concise noun phrase describing the element itself\n- Single-line format: `/** Description */`\n- For conditional elements, append context after a comma: `/** Header element, only rendered on phone viewport */`\n\n**Examples:**\n\n```tsx\n/** Root element */\n/** Title text element */\n/** Navigation controls element */\n/** Header element, only rendered on phone viewport in horizontal direction */\n```\n\n## Step 4: Add Styles API (Web Components)\n\nFor web components, add three things:\n\n### 4.1 Static Class Names\n\nAdd a static classNames object with JSDoc comments. Place this before the component's type definitions:\n\n```tsx\n/**\n * Static class names for [ComponentName] component parts.\n * Use these selectors to target specific elements with CSS.\n */\nexport const [componentName]ClassNames = {\n /** Root element */\n root: 'cds-[ComponentName]',\n /** [Concise element description] */\n [selectorName]: 'cds-[ComponentName]-[selectorName]',\n // ... more selectors as needed\n} as const;\n```\n\n**Naming conventions:**\n\n- Use `cds-` prefix for all class names\n- Use PascalCase for component name: `cds-NavigationBar`\n- Use camelCase for sub-elements: `cds-NavigationBar-contentWrapper`, `cds-Foo-titleStack`\n- Keep names descriptive but concise\n\n**Example:**\n\n```tsx\nexport const fooClassNames = {\n root: 'cds-Foo',\n contentWrapper: 'cds-Foo-contentWrapper',\n titleStack: 'cds-Foo-titleStack',\n helperText: 'cds-Foo-helperText',\n} as const;\n```\n\n### 4.2 Update Component Props Type\n\nImport and use the `StylesAndClassNames` utility type:\n\n```tsx\nimport type { StylesAndClassNames } from '../types';\n\nexport type [ComponentName]BaseProps = BoxBaseProps & {\n // ... other props (without styles/classNames)\n};\n\nexport type [ComponentName]Props = [ComponentName]BaseProps & StylesAndClassNames & Omit, 'children'>;\n```\n\nThis automatically generates the `styles` and `classNames` props based on your static classNames object.\n\n### 4.3 Apply in Component Implementation\n\nApply the static classNames, dynamic classNames, and styles in the component:\n\n```tsx\nimport { cx } from '../cx';\n\n// In the component:\n\n \n {children}\n \n\n```\n\n### 4.4 Add Tests for Static Class Names\n\nAdd tests to verify that static class names are applied correctly to the component. This ensures the class names remain stable for consumers who depend on them for CSS targeting.\n\n**Test pattern:**\n\n```tsx\nimport { [componentName]ClassNames } from '../[ComponentName]';\n\ndescribe('[ComponentName] static classNames', () => {\n it('applies static class names to component elements', () => {\n render(\n <[ComponentName]WithTheme\n start={
Start
} // Include props that render conditional elements\n >\n
Children
\n ,\n );\n\n // Test root element\n const root = screen.getByRole('[role]'); // or use testID/other selector\n expect(root).toHaveClass([componentName]ClassNames.root);\n\n // Test sub-elements using querySelector with the static class name\n expect(root.querySelector(`.${[componentName]ClassNames.start}`)).toBeInTheDocument();\n expect(root.querySelector(`.${[componentName]ClassNames.content}`)).toBeInTheDocument();\n });\n});\n```\n\n**Key testing principles:**\n\n- Import the static classNames object from the component\n- Use `toHaveClass()` for elements accessible via roles/queries\n- Use `querySelector()` with the static class name for internal elements\n- Test all selectors, including those on conditionally rendered elements (pass appropriate props)\n\n**Example from NavigationBar:**\n\n```tsx\nimport { navigationBarClassNames } from '../NavigationBar';\n\ndescribe('NavigationBar static classNames', () => {\n it('applies static class names to component elements', () => {\n render(\n Start}>\n
Children
\n ,\n );\n\n const nav = screen.getByRole('navigation');\n expect(nav).toHaveClass(navigationBarClassNames.root);\n expect(nav.querySelector(`.${navigationBarClassNames.start}`)).toBeInTheDocument();\n expect(nav.querySelector(`.${navigationBarClassNames.content}`)).toBeInTheDocument();\n });\n});\n```\n\n## Step 5: Add Styles API (Mobile Components)\n\nFor mobile components, the pattern is simpler (no static classNames):\n\n### 5.1 Add styles prop type\n\n```tsx\nexport type [ComponentName]Props = {\n // ... other props\n /** Custom styles for individual elements of the [ComponentName] component */\n styles?: {\n /** Root container element */\n root?: StyleProp;\n /** [Concise element description] */\n [selectorName]?: StyleProp;\n // ... more selectors as needed\n };\n};\n```\n\n### 5.2 Apply in Component Implementation\n\n```tsx\n\n {children}\n\n```\n\n## Step 6: Add JSDoc Notes for Special Cases\n\nIf any selectors have special rendering conditions, append the note after the element description with a comma:\n\n```tsx\nstyles?: {\n /** Header element, only rendered on phone viewport in horizontal direction */\n header?: React.CSSProperties;\n};\n```\n\nCommon cases to document:\n\n- Viewport-specific rendering (phone/tablet/desktop)\n- Direction-specific rendering (horizontal/vertical)\n- Conditional rendering based on props\n- Elements that only render with certain data (e.g., subSteps)\n\n## Reference: StylesAndClassNames Utility\n\nThe `StylesAndClassNames` utility type (from `packages/web/src/types.ts`) automatically generates:\n\n```tsx\n// Given:\nconst fooClassNames = {\n root: 'cds-Foo',\n contentWrapper: 'cds-Foo-contentWrapper',\n} as const;\n\n// StylesAndClassNames generates:\n{\n styles?: {\n root?: React.CSSProperties;\n contentWrapper?: React.CSSProperties;\n };\n classNames?: {\n root?: string;\n contentWrapper?: string;\n };\n}\n```\n\n## Reference: NavigationBar Example\n\nSee `packages/web/src/navigation/NavigationBar.tsx` for a complete example of the styles API pattern:\n\n- Lines 16-28: Static classNames with JSDoc\n- Line 80: Using `StylesAndClassNames` type on regular Props (not BaseProps)\n- Lines 117, 140, 149: Applying classNames with `cx()`\n- Lines 125, 142, 152: Applying styles\n\nSee `packages/web/src/navigation/__tests__/NavigationBar.test.tsx` for static classNames test example:\n\n- `NavigationBar static classNames` describe block: Tests all static class names are applied\n\n## Step 7: Update Documentation\n\nAfter adding the styles API to the component, update the documentation:\n\n1. **Run the docgen** to regenerate styles data:\n\n ```bash\n yarn nx run docs:docgen\n ```\n\n2. **Create or update the styles documentation** use the `components.write-docs` SKILL for general knowledge on how to write component documentation:\n - Create `_webStyles.mdx` with ComponentStylesTable and StylesExplorer\n - Create `_mobileStyles.mdx` with ComponentStylesTable (if mobile)\n - Update `index.mdx` to import and render the styles tables\n\n## Final Checklist\n\nBefore completing, verify:\n\n- [ ] Reviewed component structure for potential simplifications (flagged to user if found)\n- [ ] Selector names are from the approved list (or got user confirmation for new ones)\n- [ ] Each selector has a JSDoc comment following the convention (sentence case, no trailing period, concise noun phrase)\n- [ ] Class names follow `cds-ComponentName-selectorName` convention (camelCase)\n- [ ] Using `StylesAndClassNames` utility type on regular Props (not BaseProps) (web) or manual styles type (mobile)\n- [ ] Static classNames applied with `cx()` in component JSX (web only)\n- [ ] Dynamic classNames and styles props applied correctly\n- [ ] Special rendering conditions documented in JSDoc\n- [ ] Tests added for static classNames (web only) - see Step 4.4\n- [ ] Ran `yarn nx run docs:docgen` to regenerate styles data\n- [ ] Documentation updated to include new component styles information\n- [ ] Updated this file's \"Approved Selector Names\" table if new selectors were added\n" }, { "path": ".claude/skills/components.write-docs/README.md", "content": "# components.write-docs agent skill\n\nThis skill may be invoked by the user following the examples below.\n\n**Usage:** `/component-docs [additional context]`\n\nExamples:\n\n- `/component-docs Button`\n- `/component-docs LineChart add examples for real-time data updates`\n- `/component-docs Avatar needs accessibility improvements`\n" }, { "path": ".claude/skills/components.write-docs/SKILL.md", "content": "---\nname: components.write-docs\ndescription: Guidelines for creating or updating documentation for a CDS component on the docsite (apps/docs/). Use this skill after creating or making updates to a CDS React component to write high quality documentaiton in the CDS docsite.\nargument-hint: [additional context] (e.g., \"Button\", \"LineChart add real-time examples\")\nmodel: claude-sonnet-4-6\n---\n\nGoal: Create or update documentation for a CDS component on the docsite (apps/docs/).\n\nIf no component name is provided, ask the user which component they want to document.\n\n## Step 1: Check for Existing Documentation\n\nFirst, check if documentation already exists for this component:\n\n```bash\napps/docs/docs/components/*/[ComponentName]/\n```\n\n- **If docs exist**: Review the existing documentation and identify what needs to be added, updated, or improved. Consider the user's additional context if provided.\n- **If docs don't exist**: Follow the full workflow below to create new documentation.\n\nFor updates, focus on the specific areas that need improvement rather than rewriting everything.\n\n### Reference Components\n\nWhen creating or updating docs, reference these well-documented components to understand the documentation style and patterns:\n\n- **LineChart** (`apps/docs/docs/components/charts/LineChart/`) - Comprehensive example with many composed examples\n- **Button** (`apps/docs/docs/components/buttons/Button/`) - Good basic component documentation\n- **IconButton** (`apps/docs/docs/components/buttons/IconButton/`) - Simple component with clear examples\n- **Sidebar** (`apps/docs/docs/components/navigation/Sidebar/`) - Complex component with multiple sub-components\n\nReview these before writing to ensure consistency in style, structure, and depth.\n\n### Reference Files\n\nWhen writing examples, reference these files for valid values:\n\n- **Icon names** (`packages/icons/src/IconName.ts`) - All valid icon names (e.g., `'checkmark'`, `'close'`, `'warning'`)\n- **Design tokens** - Use the `components.best-practices` SKILL for knowledge on valid CDS design token values (Color, Space, BorderRadius, Font, etc.)\n\n## Step 2: Research Phase (for new docs or major updates)\n\nBefore writing documentation, research how other popular component libraries document the same (or similar) component. Use web search to find documentation for the component in:\n\n- **Material UI** (mui.com)\n- **Radix UI** (radix-ui.com)\n- **Mantine** (mantine.dev)\n- **Ant Design** (ant.design)\n- **Base UI** (base-ui.com)\n\nLook for:\n\n- What examples they provide and how they're organized\n- Common use cases they demonstrate\n- Edge cases or patterns worth highlighting\n- Accessibility guidance they include\n- How they explain complex features\n\nUse these insights to inform your documentation structure and examples.\n\n## Step 3: Check Component Availability\n\nVerify where the component exists:\n\n```bash\npackages/web/src/[source-category]/[ComponentName].tsx # for web\npackages/mobile/src/[source-category]/[ComponentName].tsx # for mobile\n```\n\nAlso check visualization packages if applicable:\n\n- `packages/web-visualization/src/...`\n- `packages/mobile-visualization/src/...`\n\nAlso check for Storybook stories (`packages/*/src/**/__stories__/[ComponentName].stories.tsx`). If one exists, add the `storybook` field to webMetadata.json.\n\n### Check for Styles\n\nCheck if the component supports the `styles` and/or `classNames` props by looking at its type definitions. Components with these props should have a styles tab in the documentation. Look for:\n\n- `styles?: { ... }` prop with named style selectors\n- `classNames?: { ... }` prop with named class selectors\n\nIf the component has these props, the docgen will generate styles data that can be used for the styles doc.\n\n## Step 4: Required Setup Steps (for new docs only)\n\nBefore creating the component documentation, complete these setup steps:\n\n### 4.1 Add to ReactLiveScope\n\nIn `apps/docs/src/components/page/ReactLiveScope.ts`, add the component imports and add them to the scope:\n\n```ts\n// Add imports\nimport { ComponentName } from '@coinbase/cds-web';\n\n// Add to scope object\nconst ReactLiveScope = {\n // ... existing scope\n ComponentName,\n};\n```\n\nThere is a chance that the component has already been imported.\n\n### 4.2 Update sidebars.ts\n\nIn `apps/docs/sidebars.ts`, add the component to its category section:\n\n```ts\nmodule.exports = {\n docs: [\n // ... other sections\n {\n type: 'category',\n label: 'Category', // e.g., 'Buttons', 'Layout', etc.\n items: [\n // ... other components\n 'components/category/ComponentName/index',\n ],\n },\n ],\n};\n```\n\n### 4.3 Update docgen.config.js\n\nIn `apps/docs/docgen.config.js`, add the component paths to generate props data:\n\n```js\nmodule.exports = {\n web: {\n // ... other configs\n category: {\n // e.g., 'buttons', 'layout', etc.\n ComponentName: {\n source: 'packages/web/src/category/ComponentName.tsx',\n },\n },\n },\n // If component has a mobile version\n mobile: {\n // ... other configs\n category: {\n ComponentName: {\n source: 'packages/mobile/src/category/ComponentName.tsx',\n },\n },\n },\n};\n```\n\n## Step 5: Create Directory Structure (for new docs only)\n\nCreate the documentation directory and files based on component availability:\n\n```bash\napps/docs/docs/components/[docs-category]/[ComponentName]/\n├── index.mdx # Required for all components\n├── webMetadata.json # If web version exists\n├── _webExamples.mdx # If web version exists\n├── _webPropsTable.mdx # If web version exists\n├── _webStyles.mdx # If web version has styles/classNames API\n├── mobileMetadata.json # If mobile version exists\n├── _mobileExamples.mdx # If mobile version exists\n├── _mobilePropsTable.mdx # If mobile version exists\n└── _mobileStyles.mdx # If mobile version has styles/classNames API\n```\n\n## File Templates\n\n### Metadata Files\n\n#### webMetadata.json\n\n```json\n{\n \"import\": \"import { ComponentName } from '@coinbase/cds-web/[source-category]/[ComponentName]'\",\n \"source\": \"https://github.com/coinbase/cds/blob/master/packages/web/src/[source-category]/[ComponentName].tsx\",\n \"description\": \"[Component description]\",\n \"figma\": \"[figma link]\",\n \"storybook\": \"[storybook link]\",\n \"relatedComponents\": [\n { \"label\": \"[componentName]\", \"url\": \"/components/[category]/[componentName]\" }\n ],\n \"dependencies\": [{ \"name\": \"[peer-dependency-name]\", \"version\": \"[version-range]\" }]\n}\n```\n\n**Notes:**\n\n- `description` should be the full component description - what the component is and when to use it (e.g., \"A non-intrusive notification component that temporarily displays brief messages at the bottom of the screen.\")\n- `figma` and `storybook` fields are optional - only add if provided (check for story files in `packages/web/src/**/__stories__/`)\n- `dependencies` is optional - only include if the component imports from external packages that are peer dependencies. To determine:\n 1. Check the component's source file for imports from external packages (e.g., `framer-motion`)\n 2. Cross-reference those imports with `peerDependencies` in `packages/web/package.json`\n 3. Use the exact version range from `peerDependencies` in the package.json file\n- `relatedComponents` should link to components commonly used together\n\n#### mobileMetadata.json\n\n```json\n{\n \"import\": \"import { ComponentName } from '@coinbase/cds-mobile/[source-category]/[ComponentName]'\",\n \"source\": \"https://github.com/coinbase/cds/blob/master/packages/mobile/src/[source-category]/[ComponentName].tsx\",\n \"description\": \"[Component description]\",\n \"figma\": \"[figma link]\",\n \"relatedComponents\": [\n { \"label\": \"[componentName]\", \"url\": \"/components/[category]/[componentName]\" }\n ],\n \"dependencies\": [{ \"name\": \"[peer-dependency-name]\", \"version\": \"[version-range]\" }]\n}\n```\n\n**Notes:**\n\n- `figma` is optional - only add if provided\n- `dependencies` is optional - only include if the component imports from external packages that are peer dependencies. To determine:\n 1. Check the component's source file for imports from external packages (e.g., `@shopify/react-native-skia`, `react-native-reanimated`, `react-native-gesture-handler`)\n 2. Cross-reference those imports with `peerDependencies` in `packages/mobile/package.json`\n 3. Use the exact version range from `peerDependencies` in the package.json file\n\n### Props Tables\n\n#### \\_webPropsTable.mdx\n\n```mdx\nimport ComponentPropsTable from '@site/src/components/page/ComponentPropsTable';\nimport webPropsData from ':docgen/web/[source-category]/[ComponentName]/data';\nimport { sharedParentTypes } from ':docgen/_types/sharedParentTypes';\nimport { sharedTypeAliases } from ':docgen/_types/sharedTypeAliases';\n\n\n```\n\n#### \\_mobilePropsTable.mdx\n\n```mdx\nimport ComponentPropsTable from '@site/src/components/page/ComponentPropsTable';\nimport mobilePropsData from ':docgen/mobile/[source-category]/[ComponentName]/data';\nimport { sharedParentTypes } from ':docgen/_types/sharedParentTypes';\nimport { sharedTypeAliases } from ':docgen/_types/sharedTypeAliases';\n\n\n```\n\n### Styles Doc\n\nStyles doc showcases the `styles` and `classNames` API for components that support custom styling of internal elements. Only create these files if the component has a styles/classNames API.\n\n#### \\_webStyles.mdx (with Explorer)\n\nFor web components, always include both the selectors table AND the interactive StylesExplorer. The StylesExplorer lets users hover or click on selectors to highlight the corresponding elements in a live example:\n\n```mdx\nimport { ComponentStylesTable } from '@site/src/components/page/ComponentStylesTable';\nimport { StylesExplorer } from '@site/src/components/page/StylesExplorer';\nimport { [ComponentName] } from '@coinbase/cds-web/[source-category]/[ComponentName]';\n\nimport webStylesData from ':docgen/web/[source-category]/[ComponentName]/styles-data';\n\n## Explorer\n\n\n {(classNames) => (\n <[ComponentName] {...exampleProps} classNames={classNames} />\n )}\n\n\n## Selectors\n\n\n```\n\nIf the component requires state management, bundle everything into a single exported example component.\n\n```mdx\nimport { useState } from 'react';\nimport { ComponentStylesTable } from '@site/src/components/page/ComponentStylesTable';\nimport { StylesExplorer } from '@site/src/components/page/StylesExplorer';\nimport { Select } from '@coinbase/cds-web/alpha/select';\n\nimport webStylesData from ':docgen/web/alpha/select/Select/styles-data';\n\nexport const SelectExample = ({ classNames }) => {\n const [value, setValue] = useState('1');\n const options = [\n { value: '1', label: 'Option 1' },\n { value: '2', label: 'Option 2' },\n { value: '3', label: 'Option 3' },\n ];\n return (\n \n );\n};\n\n## Explorer\n\n\n {(classNames) => }\n\n\n## Selectors\n\n\n```\n\n**Notes:**\n\n- For components with multiple variants (e.g., horizontal/vertical), add multiple explorer sections with h3 headings\n- The `StylesExplorer` passes `classNames` to highlight selected elements\n\n**Creating Comprehensive Explorer Examples:**\n\nThe explorer example should be designed to showcase **all available selectors**. Review the component's `classNames` type to identify all selectors, then configure your example to render the elements that use each selector:\n\n- **Check conditional rendering**: Some selectors only appear with certain props (e.g., Stepper's `header` only renders in horizontal mode)\n- **Include nested structures**: If the component has nested elements like `subSteps`, include them to showcase selectors like `substepContainer`\n- **Add optional content**: Include props like `title` if the component has a `title` selector\n- **Use realistic data**: Provide enough items/steps to demonstrate the full component structure\n\nExample for a component with conditional selectors:\n\n```mdx\n{/* Steps with subSteps to showcase all selectors including substepContainer */}\nexport const steps = [\n{ id: '1', label: 'Step 1' },\n{\nid: '2',\nlabel: 'Step 2',\nsubSteps: [\n{ id: '2a', label: 'Sub-step A' },\n{ id: '2b', label: 'Sub-step B' },\n],\n},\n{ id: '3', label: 'Step 3' },\n];\n```\n\n#### \\_mobileStyles.mdx (Selectors Only)\n\nFor mobile components, only include the selectors table (no interactive explorer):\n\n```mdx\nimport { ComponentStylesTable } from '@site/src/components/page/ComponentStylesTable';\n\nimport mobileStylesData from ':docgen/mobile/[source-category]/[ComponentName]/styles-data';\n\n## Selectors\n\n\n```\n\n### Main Documentation (index.mdx)\n\n#### For Web-Only Components\n\n```mdx\n---\nid: [component-id]\ntitle: [ComponentName]\nplatform_switcher_options: { web: true, mobile: false }\nhide_title: true\n---\n\nimport { VStack } from '@coinbase/cds-web/layout';\nimport { ComponentHeader } from '@site/src/components/page/ComponentHeader';\nimport { ComponentTabsContainer } from '@site/src/components/page/ComponentTabsContainer';\n\nimport webPropsToc from ':docgen/web/[source-category]/[ComponentName]/toc-props';\nimport WebPropsTable from './_webPropsTable.mdx';\n// If component has styles API, add this import:\nimport WebStyles, { toc as webStylesToc } from './_webStyles.mdx';\nimport WebExamples, { toc as webExamplesToc } from './_webExamples.mdx';\nimport webMetadata from './webMetadata.json';\n\n\n \n }\n webExamplesToc={webExamplesToc}\n webPropsTable={}\n webPropsToc={webPropsToc}\n // If component has styles API, add these props:\n webStylesTable={}\n webStylesToc={webStylesToc}\n />\n\n```\n\n#### For Mobile-Only Components\n\n```mdx\n---\nid: [component-id]\ntitle: [ComponentName]\nplatform_switcher_options: { web: false, mobile: true }\nhide_title: true\n---\n\nimport { VStack } from '@coinbase/cds-web/layout';\nimport { ComponentHeader } from '@site/src/components/page/ComponentHeader';\nimport { ComponentTabsContainer } from '@site/src/components/page/ComponentTabsContainer';\n\nimport mobilePropsToc from ':docgen/mobile/[source-category]/[ComponentName]/toc-props';\nimport MobilePropsTable from './_mobilePropsTable.mdx';\n// If component has styles API, add this import:\nimport MobileStyles, { toc as mobileStylesToc } from './_mobileStyles.mdx';\nimport MobileExamples, { toc as mobileExamplesToc } from './_mobileExamples.mdx';\nimport mobileMetadata from './mobileMetadata.json';\n\n\n \n }\n mobileExamplesToc={mobileExamplesToc}\n mobilePropsTable={}\n mobilePropsToc={mobilePropsToc}\n // If component has styles API, add these props:\n mobileStylesTable={}\n mobileStylesToc={mobileStylesToc}\n />\n\n```\n\n#### For Cross-Platform Components\n\n```mdx\n---\nid: [component-id]\ntitle: [ComponentName]\nplatform_switcher_options: { web: true, mobile: true }\nhide_title: true\n---\n\nimport { VStack } from '@coinbase/cds-web/layout';\nimport { ComponentHeader } from '@site/src/components/page/ComponentHeader';\nimport { ComponentTabsContainer } from '@site/src/components/page/ComponentTabsContainer';\n\nimport webPropsToc from ':docgen/web/[source-category]/[ComponentName]/toc-props';\nimport mobilePropsToc from ':docgen/mobile/[source-category]/[ComponentName]/toc-props';\nimport WebPropsTable from './_webPropsTable.mdx';\nimport MobilePropsTable from './_mobilePropsTable.mdx';\n// If component has styles API, add these imports:\nimport WebStyles, { toc as webStylesToc } from './_webStyles.mdx';\nimport MobileStyles, { toc as mobileStylesToc } from './_mobileStyles.mdx';\nimport WebExamples, { toc as webExamplesToc } from './_webExamples.mdx';\nimport MobileExamples, { toc as mobileExamplesToc } from './_mobileExamples.mdx';\nimport webMetadata from './webMetadata.json';\nimport mobileMetadata from './mobileMetadata.json';\n\n\n \n }\n mobileExamplesToc={mobileExamplesToc}\n mobilePropsTable={}\n mobilePropsToc={mobilePropsToc}\n // If component has styles API, add these props:\n mobileStylesTable={}\n mobileStylesToc={mobileStylesToc}\n webExamples={}\n webExamplesToc={webExamplesToc}\n webPropsTable={}\n webPropsToc={webPropsToc}\n // If component has styles API, add these props:\n webStylesTable={}\n webStylesToc={webStylesToc}\n />\n\n```\n\n### Examples\n\n#### Example Structure Guidelines\n\nExamples should follow this recommended structure:\n\n1. **Brief intro** - A short functional note (NOT the full description - that goes in metadata). Mention what the component uses/wraps or key dependencies.\n2. **Basics** - Simplest usage explaining how to use the core API\n3. **Feature sections** - Group related functionality by topic (e.g., Styling with Color/Sizing subsections)\n4. **Accessibility** - How to make the component accessible\n\n**Important:** Do NOT repeat the full component description from metadata in the examples. The examples should focus on _how_ to use the component, not _what_ it is.\n\n#### \\_webExamples.mdx (Live Examples)\n\nWeb examples use `jsx live` blocks which render interactively in the browser. For short, incomplete code snippets that are meant to illustrate a concept rather than be runnable, you may use plain `jsx` blocks instead.\n\n````mdx\n[ComponentName] uses [dependency/wrapper] to [brief functional note]. [Any key setup requirement in one sentence].\n\n## Basics\n\n[Explain how to use the component's core API - e.g., \"Call `toast.show()` with a message string to display a toast.\"]\n\n```jsx live\n<[ComponentName]\n requiredProp=\"value\"\n/>\n```\n\n## [Feature Category]\n\n[Brief explanation of this feature category]\n\n### [Specific Feature]\n\n```jsx live\n<[ComponentName]\n featureProp=\"value\"\n/>\n```\n\n## Styling\n\n### Color\n\n[Show color customization options]\n\n```jsx live\n<[ComponentName] color=\"fgPrimary\" />\n```\n\n### Sizing\n\n[Show sizing options]\n\n```jsx live\n<[ComponentName] size=\"large\" />\n```\n\n## Accessibility\n\nUse `accessibilityLabel` to provide context for screen readers. When [specific scenario], also consider [accessibility guidance].\n\n```jsx live\n<[ComponentName]\n accessibilityLabel=\"Descriptive label for screen readers\"\n requiredProp=\"value\"\n/>\n```\n````\n\n#### \\_mobileExamples.mdx (Static Examples)\n\nMobile examples use static `jsx` blocks only. **Do not use `jsx live`** - React Native cannot run in the browser.\n\n````mdx\n[ComponentName] uses [dependency/wrapper] to [brief functional note]. [Any mobile-specific behavior in one sentence, e.g., \"On mobile, toasts can be swiped away.\"]\n\n## Basics\n\n[Explain how to use the component's core API - e.g., \"Call `toast.show()` with a message string to display a toast.\"]\n\n```jsx\n<[ComponentName]\n requiredProp=\"value\"\n/>\n```\n\n## [Feature Category]\n\n[Brief explanation of this feature category]\n\n### [Specific Feature]\n\n```jsx\n<[ComponentName]\n featureProp=\"value\"\n/>\n```\n\n## Styling\n\n### Color\n\n[Show color customization options]\n\n```jsx\n<[ComponentName] color=\"fgPrimary\" />\n```\n\n### Sizing\n\n[Show sizing options]\n\n```jsx\n<[ComponentName] size=\"large\" />\n```\n\n## Accessibility\n\nUse `accessibilityLabel` to provide context for screen readers.\n\n```jsx\n<[ComponentName]\n accessibilityLabel=\"Descriptive label for screen readers\"\n requiredProp=\"value\"\n/>\n```\n````\n\n## Best Practices for Examples\n\n### Code Quality\n\n- **Use named functions** for complex examples that need state or effects\n- **Memoize with `useMemo`** for expensive computations or computed styles\n- **Memoize with `useCallback`** for event handlers passed as props\n- **Include accessibility labels** in interactive examples\n- **Format values for display** using `Intl.NumberFormat`, `Intl.DateTimeFormat`, etc.\n- **Ensure live examples are responsive** - The doc site can be viewed on mobile viewports, so examples should render well at narrow widths (e.g., add `flexWrap=\"wrap\"` to HStacks with multiple items)\n\n### Documentation Quality\n\n- **Start with introductory prose** explaining what the component does before any code\n- **Progress from simple to complex** - basic examples first, advanced examples last\n- **Cross-reference related components** using markdown links: `[ComponentName](/components/category/ComponentName)`\n- **Explain the \"why\"** not just the \"how\" - help users understand when to use each feature\n- **Show edge cases** like empty states, loading states, error states, missing data\n\n### Common Sections to Consider\n\nDepending on the component, consider including these sections:\n\n- **Setup** - Prerequisites or providers needed (especially for mobile)\n- **Basics** - Core API usage\n- **Feature-specific sections** - Group related functionality (e.g., if the component works with other components, show that integration)\n- **Styling** - Color, Sizing, and other customization options as subsections\n- **Accessibility** - Screen reader support, keyboard navigation, etc.\n- **Composed Examples** - Used for complex components where showing real-world patterns that combine multiple features into new component compositions adds significant value.\n\n## Final Checklist\n\nBefore completing, verify:\n\n- [ ] Researched similar components in other libraries for inspiration\n- [ ] Verified component existence in web/mobile\n- [ ] Created only necessary platform-specific files\n- [ ] Set correct `platform_switcher_options`\n- [ ] Metadata files have correct package imports\n- [ ] Added `dependencies` field if component has peer dependencies\n- [ ] Props tables import from correct package with correct variable names\n- [ ] Styles files created if component has styles/classNames API\n- [ ] StylesExplorer is used in web styles (if present) and includes working example with appropriate props\n- [ ] Examples start with introductory prose\n- [ ] Examples include accessibility guidance\n- [ ] Examples progress from basic to advanced\n- [ ] Web examples use `jsx live` (or `jsx` for short snippets); mobile examples use `jsx` only (no `live`)\n- [ ] ComponentTabsContainer includes only existing platform props\n- [ ] All imports use correct source categories\n- [ ] Component description is clear and helpful\n- [ ] Added storybook/figma links if story files exist or links are provided\n- [ ] Design token values are valid (followed `components.best-practices` SKILL)\n\n## Additional Notes\n\n1. Source category might differ from docs category\n2. Add storybook and figma links to metadata if provided\n3. Ensure all examples work and have proper code snippets\n4. Include accessibility section with specific examples\n5. Test all examples and props tables render correctly\n6. For visualization components, use paths like `web-visualization` or `mobile-visualization` instead of `web` or `mobile`\n" }, { "path": ".claude/skills/deprecate-cds-api/SKILL.md", "content": "---\nname: deprecate-cds-api\ndescription: |\n Deprecates a CDS component, hook, or other exported symbol with consistent JSDoc, version tags,\n and docsite metadata across every public export path (web, mobile, common, visualization), not only\n the original package. Use whenever the user asks to deprecate a CDS component or API, mark something\n as deprecated, add @deprecated / @deprecationExpectedRemoval, or update deprecation warnings in\n apps/docs metadata under components or hooks (webMetadata.json / mobileMetadata.json / metadata.json).\n Also use when replacing a component or hook and sunsetting the old one. Always finish by running\n `yarn nx run :lint` on modified packages so `internal/deprecated-jsdoc-has-removal-version` passes.\nallowed-tools: Read, Grep, Glob, StrReplace, Bash(yarn nx run:*)\nargument-hint: ' — replacement — [@deprecationExpectedRemoval major e.g. v10] — [optional notes]'\n---\n\n# Deprecate CDS public API\n\nAutomate the standard CDS deprecation workflow for symbols exported from `packages/web`, `packages/mobile`, `packages/common`, `packages/web-visualization`, or `packages/mobile-visualization`.\n\n## Inputs to confirm first\n\n1. **What is being deprecated?** Component name, hook, prop, or other exported symbol.\n2. **What should consumers use instead?** The replacement must be named in JSDoc and in docs `warning` text.\n3. **Which major should `@deprecationExpectedRemoval` use?** (e.g. `v9`, `v10`.) **Ask the user to confirm** if they have not already stated it. If they want a default, **suggest** the next major from the relevant `package.json` (see Step 2) and confirm they accept it before editing.\n\n---\n\n## Step 0 — Discover every public export (all packages)\n\n**Deprecate the symbol everywhere it is publicly reachable**, not only where it is first implemented.\n\n1. For each CDS package (`web`, `mobile`, `common`, `web-visualization`, `mobile-visualization`), trace the symbol from that package’s `package.json` **`exports`** map → barrel / `index` files → the module that declares or re-exports the symbol.\n2. **`Grep`** for the symbol name under `packages//src` (e.g. `export { Foo`, `export * from`, `Foo as`) to catch re-exports and alternate entry paths.\n3. **Every** package that publicly exports the symbol must end up with deprecation coverage: primary implementation **and** any re-export site where your tooling or consumers would not see JSDoc from the source file (add JSDoc on the re-export line or duplicate the tags as needed so imports from `@coinbase/cds-web`, `@coinbase/cds-mobile`, `@coinbase/cds-common`, etc. all surface the deprecation).\n\nDo **not** skip a package because the symbol is “originally” defined elsewhere—if consumers can import it from that package, it must be deprecated there too.\n\n---\n\n## Step 1 — JSDoc on the deprecated symbol\n\nAdd or extend JSDoc immediately above the deprecated export (component, function, type alias, const, interface field, etc.).\n\nUse the **standard JSDoc tag `@deprecated`** (not `@deprecate`).\n\n**Required shape:**\n\n```ts\n/**\n * …existing description if any…\n *\n * @deprecated . This will be removed in a future major release.\n * @deprecationExpectedRemoval v\n */\n```\n\nRules:\n\n- The `@deprecated` line must end with exactly: `This will be removed in a future major release.` (same sentence as the rest of the deprecation message, as in existing CDS examples).\n- `@deprecationExpectedRemoval` must match `v` + version (e.g. `v9` or `v9.0.0`; full semver is allowed by ESLint).\n\nThe repo’s ESLint rule **`internal/deprecated-jsdoc-has-removal-version`** (`libs/eslint-plugin-internal`) enforces the prose ending and the presence of `@deprecationExpectedRemoval`; **lint must pass** after edits (see **Step 6**).\n\n---\n\n## Step 2 — Removal version for `@deprecationExpectedRemoval`\n\nThe tag must satisfy `@deprecationExpectedRemoval v…` as enforced by ESLint (e.g. `v10` or `v10.0.0`).\n\n1. **Confirm with the user** which major **`N`** to use, unless they already specified it in **Inputs** (e.g. “remove in v10” → use `v10`).\n2. **Default suggestion** when the user wants a recommendation: read the **`version`** field from the relevant `package.json` and set **`N = current major + 1`**.\n - **`packages/web`**, **`packages/mobile`**, and **`packages/common`** always share the same semver — read **`version`** from any one of them (e.g. `8.60.0` → suggest **`v9`**).\n - Symbols owned only by **`packages/web-visualization`** or **`packages/mobile-visualization`**: read **that** package’s `package.json` (those versions are independent from web/mobile/common).\n3. After agreeing on **`N`**, use **`@deprecationExpectedRemoval v`** everywhere for this deprecation (same **Step 3**).\n\nDo **not** assume the default without checking—either the user names **`N`**, or they accept the suggested next-major after you show the current **`version`**.\n\n---\n\n## Step 3 — Consistency across export surfaces\n\n- Use the **same** `@deprecated` guidance and **`@deprecationExpectedRemoval v`** value everywhere the symbol is exported (adjust wording only if a platform’s API genuinely differs).\n- **Components** and **hooks** that exist as separate web and mobile implementations: deprecate **both** when both packages export the symbol.\n- **Shared** symbols in `packages/common` that are **also** re-exported from web or mobile: follow Step 0 — ensure deprecation is visible on **every** public import path (common barrels **and** web/mobile re-exports if applicable).\n- **Visualization** packages: same rules whenever those packages export the symbol.\n\n---\n\n## Step 4 — Docsite metadata (`apps/docs/docs`)\n\nOnly when the symbol has **existing** docs under the docs app. **Do not** add new doc folders unless the docs workflow already expects them.\n\n### Components (`apps/docs/docs/components/`)\n\n1. Locate the folder, e.g. `apps/docs/docs/components///`.\n2. If present, edit **`webMetadata.json`** and/or **`mobileMetadata.json`** (some components have both; some only one).\n3. Add or update the top-level **`warning`** string:\n\n```json\n\"warning\": \"This component is deprecated. Please use {replacement} instead.\"\n```\n\nExamples:\n\n- `\"Please use Tabs instead.\"`\n- `\"Please use MediaCard instead.\"`\n\nMatch the tone of existing deprecations when the replacement is not a single component name (e.g. “Use indeterminate ProgressCircle for loading indicators instead.”) — still keep the opening: **This component is deprecated.**\n\n### Hooks (`apps/docs/docs/hooks/`)\n\n1. Locate the hook folder, e.g. `apps/docs/docs/hooks//`.\n2. Hooks may use **`webMetadata.json`** and **`mobileMetadata.json`**, or a single shared **`metadata.json`** — update whichever file(s) exist for that hook.\n3. Add or update the top-level **`warning`** string using **hook** wording:\n\n```json\n\"warning\": \"This hook is deprecated. Please use {replacement} instead.\"\n```\n\nUse the same `{replacement}` phrasing as in JSDoc. If the replacement is not a single hook name, adapt the sentence but keep the opening: **This hook is deprecated.**\n\n---\n\n## Step 5 — Verification checklist\n\n- [ ] Every **public export path** across packages that expose the symbol has been found (Step 0) and carries deprecation (implementation and re-exports as needed).\n- [ ] `@deprecated` includes replacement guidance and the exact closing sentence about future major removal.\n- [ ] **`@deprecationExpectedRemoval v`** matches the **confirmed** removal major (Step 2), not an unverified default.\n- [ ] **Web + mobile** implementations and metadata (when applicable) are updated; nothing skipped because the symbol was “only” defined in common or another package.\n- [ ] `warning` in metadata matches the replacement story: **this component is deprecated** for component docs, **this hook is deprecated** for hook docs (`apps/docs/docs/hooks/`).\n- [ ] **`yarn nx run :lint`** has been run for every touched project (**Step 6**) and passes.\n\n---\n\n## Step 6 — Run ESLint (required)\n\nAfter all edits, **run the `lint` target** on **every Nx project** that contains changed source files so **`internal/deprecated-jsdoc-has-removal-version`** (and the rest of the package lint config) passes.\n\nUse the workspace convention:\n\n```bash\nyarn nx run :lint\n```\n\nExamples: `web`, `mobile`, `common`, `web-visualization`, `mobile-visualization` — run **each** project you touched. Fix any reported issues before finishing (most often: missing `@deprecationExpectedRemoval`, or `@deprecated` text not ending with the standard sentence).\n\n---\n\n## Reference examples in-repo\n\n- JSDoc: search for `@deprecationExpectedRemoval` under `packages/web/src` (e.g. `TabNavigation.tsx`, `Spinner.tsx`).\n- ESLint: `libs/eslint-plugin-internal` — rule **`deprecated-jsdoc-has-removal-version`** (exposed as **`internal/deprecated-jsdoc-has-removal-version`** in the root `eslint.config.mjs`).\n- Metadata: search for `\"warning\": \"This component is deprecated` under `apps/docs/docs/components`; hook docs live under `apps/docs/docs/hooks/` (use **This hook is deprecated** for hook `warning` text).\n" }, { "path": ".claude/skills/dev.cds-mobile/SKILL.md", "content": "---\nname: dev.cds-mobile\ndescription: USE THIS when asked to work on a new or existing (MOBILE) CDS React component in packages/mobile\n---\n\n\n\n# CDS Mobile Package Guidelines\n\nMobile-specific patterns for `@coinbase/cds-mobile`.\n\n## Component Config Adoption (Mobile)\n\nUse this guidance when adding `ComponentConfigProvider` defaults for the specific component you are editing.\n\n### Required implementation pattern\n\n1. Register the component in `packages/mobile/src/core/componentConfig.ts` using its `*BaseProps`:\n\n```ts\nimport type { MyComponentBaseProps } from '../category/MyComponent';\n\nexport type ComponentConfig = {\n MyComponent?: ConfigResolver;\n};\n```\n\n1. Adopt `useComponentConfig` in the component and destructure from merged props:\n\n```tsx\nimport { useComponentConfig } from '../hooks/useComponentConfig';\n\nexport const MyComponent = memo((_props: MyComponentProps) => {\n const mergedProps = useComponentConfig('MyComponent', _props);\n const { style, ...props } = mergedProps;\n\n return ;\n});\n```\n\n### Rules to preserve behavior\n\n- Provider config supplies defaults only; local props must continue to win.\n- Use `_props` as the input variable and `mergedProps` as the configured output.\n- Type resolver entries with `*BaseProps` (not full `*Props`).\n- Keep scope to prop-level theming defaults; do not alter component behavior or control flow.\n- When practical during the same change, prefer arrow-function component declarations.\n\n## Styling with StyleSheet\n\nUse `StyleSheet.create` for static styles and `useTheme()` for dynamic values:\n\n```tsx\nimport { StyleSheet, type StyleProp, type ViewStyle } from 'react-native';\n\nconst styles = StyleSheet.create({\n container: { position: 'relative', width: '100%' },\n});\n\n// Dynamic styles via theme hook\nconst theme = useTheme();\nconst dynamicStyle = {\n backgroundColor: theme.color.bgPrimary,\n padding: theme.space[2],\n};\n\n;\n```\n\n### Inline styles\n\n- Mobile components should all expose a `style` and `styles` object props for overriding default styles.\n- As styling is a concern of that specific component, the `style` and `styles` props should never be on the `*BaseProps` type.\n- `styles` can be used for granular overrides on child elements within the component.\n- **Always** merge styles into a react-native style array with `useMemo` in the correct order (default styles => `style` prop => `styles[ELEMENT_NAME]` prop).\n\n**Example:**\n\n```tsx\ntype ComponentProps = ComponentBaseProps & {\n style?: StyleProp;\n styles?: {\n root?: StyleProp;\n label?: StyleProp;\n };\n};\n\nconst theme = useTheme();\nconst containerStyles = useMemo(\n () => [\n { backgroundColor: theme.color.bgPrimary }, // default styles\n style, // from props\n styles.root, // from props\n ],\n [theme.color.bgPrimary, animatedStyles, style]\n);\n\n// Apply to component\n\n```\n\n## Animation\n\n### React Native Reanimated\n\n```tsx\nimport Animated, { useAnimatedStyle, useSharedValue, withTiming } from 'react-native-reanimated';\n\nconst opacity = useSharedValue(0);\nconst animatedStyle = useAnimatedStyle(() => ({\n opacity: opacity.value,\n transform: [{ translateY: withTiming(opacity.value * -8) }],\n}));\n\n;\n```\n\nWe DO NOT use React-Spring anymore for animations on mobile.\n\n## Gesture Handling\n\nUse `react-native-gesture-handler`:\n\n```tsx\nimport { Gesture, GestureDetector } from 'react-native-gesture-handler';\n\nconst panGesture = useMemo(\n () =>\n Gesture.Pan()\n .onStart(() => {\n /* ... */\n })\n .onUpdate(({ translationX }) => {\n /* ... */\n })\n .onEnd(({ translationX, velocityX }) => {\n /* ... */\n })\n .withTestId(testID)\n .runOnJS(true),\n [dependencies],\n);\n\n\n {/* ... */}\n;\n```\n\n## Layout Measurement\n\nUse `onLayout` callback instead of ResizeObserver:\n\n```tsx\nconst [size, onLayout] = useLayout();\n\n\n// Or inline\n setHeight(e.nativeEvent.layout.height)} />\n```\n\n## Accessibility\n\n- Use appropriate accessibilityLabel, accessibilityHint, and accessibilityRole, accessibilityState props\n- Support screen readers (VoiceOver and TalkBack)\n- Ensure touch targets meet minimum size requirements (44x44 points)\n\n**Example:** Use React Native accessibility props:\n\n```tsx\n\n \n\n```\n\n### Screen Reader Content\n\n```tsx\n// Hide visual content from screen readers\n\n {/* Animated/visual content */}\n\n\n// Provide accessible alternative\n\n {accessibleLabel}\n\n```\n\n## Reference Components\n\n- **SlideButton**: gesture handling, spring animations, accessibility actions\n- **RollingNumber**: Reanimated, measurement patterns, screen reader content\n- **Select** (alpha/): controlled/uncontrolled, Drawer integration\n- **Stepper**: direction-based defaults, shared logic from cds-common\n- **Tour**: animations, complexity\n- **DatePicker**: complexity\n" }, { "path": ".claude/skills/dev.cds-web/SKILL.md", "content": "---\nname: dev.cds-web\ndescription: USE THIS when asked to work on a new or existing (WEB) CDS React component in packages/web\n---\n\n\n\n# CDS Web Package Guidelines\n\n## Component Config Adoption (Web)\n\nUse this guidance when adding `ComponentConfigProvider` defaults for the specific component you are editing.\n\n### Required implementation pattern\n\n1. Register the component in `packages/web/src/core/componentConfig.ts` using its `*BaseProps`:\n\n```ts\nimport type { MyComponentBaseProps } from '../category/MyComponent';\n\nexport type ComponentConfig = {\n MyComponent?: ConfigResolver;\n};\n```\n\n1. Adopt `useComponentConfig` in the component and destructure from merged props:\n\n```tsx\nimport { useComponentConfig } from '../hooks/useComponentConfig';\n\nexport const MyComponent = memo((_props: MyComponentProps) => {\n const mergedProps = useComponentConfig('MyComponent', _props);\n const { className, style, ...props } = mergedProps;\n\n return ;\n});\n```\n\n### Rules to preserve behavior\n\n- Provider config supplies defaults only; local props must continue to win.\n- Use `_props` as the input variable and `mergedProps` as the configured output.\n- Type resolver entries with `*BaseProps` (not polymorphic/full `*Props`).\n- Keep scope to prop-level theming defaults; do not alter component behavior or control flow.\n- When practical during the same change, prefer arrow-function component declarations.\n\n## CSS with Linaria\n\nUse Linaria for zero-runtime CSS. **Always use CDS theme CSS variables** for colors, spacing, typography, and other design tokens.\nReference packages/web/src/core/theme.ts:53-119 for the CSS variable naming pattern.\n\n```tsx\nimport { css, cx } from '@linaria/core';\n\nconst containerCss = css`\n /* Spacing tokens */\n padding: var(--space-2);\n gap: var(--space-1);\n\n /* Color tokens */\n background: var(--color-bgPrimary);\n color: var(--color-fgPrimary);\n border: 1px solid var(--color-line);\n\n /* Border radius tokens */\n border-radius: var(--borderRadius-400);\n\n /* Typography tokens */\n font-size: var(--fontSize-body);\n\n &:hover {\n background: var(--color-bgPrimaryHover);\n }\n`;\n\n// Merge classNames with cx utility **in CORRECT ORDER**\n;\n```\n\n**IMPORTANT:** Using CSS variables ensures components respond correctly to theme changes (light/dark mode, brand themes).\n\n### CSS Variable Naming\n\nDesign tokens from `packages/common/src/core/theme.ts` map to CSS variables:\n\n- Colors: `--color-{tokenName}` (e.g., `--color-bgPrimary`, `--color-fgMuted`)\n- Spacing: `--space-{scale}` (e.g., `--space-2` = 16px, `--space-3` = 24px)\n- Typography: `--fontSize-{font}`, `--fontWeight-{font}`, `--lineHeight-{font}`, `--fontFamily-{font}`\n- Border: `--borderRadius-{size}`, `--borderWidth-{size}`\n- Sizing: `--iconSize-{size}`, `--avatarSize-{size}`, `--controlSize-{name}`\n- Shadows: `--shadow-{level}`\n\n### Responsive Breakpoints\n\nReference `packages/web/src/styles/media.ts` for breakpoint values:\n\n- **phone**: 0-767px\n- **tablet**: 768-1279px\n- **desktop**: 1280px+\n\nUse the `Box` component's responsive prop API for responsive values:\n\n```tsx\n\n```\n\n### Granular classNames\n\n- Components can expose a `classNames` object prop for granular overrides on child elements within the component.\n- Since the keys of the `classNames` object are specific to the component they should never be on the `*BaseProps` type\n\n### data-attributes\n\n- Use data attributes for state-based styling: `data-active`, `data-disabled`, `data-variant`, `data-filled`\n\n## Inline styles\n\n- Web components should all expose a `style` and `styles` object props for overriding inline styles.\n- As styling is a concern of that specific component, the `style` and `styles` props should never be on the `*BaseProps` type.\n- Styles should be merged into a single object with a `useMemo` hook and applied in the correct order (default styles => `style` prop => `styles[ELEMENT_NAME]` prop).\n\n**Example:**\n\n```tsx\ntype ComponentProps = ComponentBaseProps & {\n style?: React.CSSProperties;\n styles?: { root?: React.CSSProperties; label?: React.CSSProperties };\n};\n```\n\n## Animation\n\nUse Framer Motion for complex animations:\n\n```tsx\nimport { m as motion, AnimatePresence } from 'framer-motion';\n\n\n {visible && (\n \n )}\n;\n```\n\nFor simple transitions, prefer CSS transitions in Linaria.\n\n## Accessibility\n\nUse ARIA attributes:\n\n```tsx\n
\n
\n```\n\n- Implement keyboard navigation (Arrow keys, Home, End) for interactive components.\n- Provide descriptive labels for all interactive elements\n- Associate form inputs with helper text using aria-describedby\n- Use semantic HTML elements whenever possible\n- Follow WCAG 2.1 AA standards for color contrast\n- Support screen readers by providing descriptive labels and instructions\n\n## Web-Specific Props\n\n- `className?: string` - CSS class always applied to root element\n- `style?: React.CSSProperties` - inline styles always applied to root element\n- Polymorphic `as` prop for element type (where applicable e.g. see Box)\n\n## Reference Components\n\n- **Carousel**: compound components, imperative handle, context pattern\n- **Select** (alpha/): generics, controlled/uncontrolled\n- **RollingNumber**: animation config, measurement patterns\n" }, { "path": ".claude/skills/eslint-plugin-custom-rule/SKILL.md", "content": "---\nname: eslint-plugin-custom-rule\ndescription: USE THIS when asked to create a new eslint plugin rule for the eslint-plugin-cds package\n---\n\n# Writing new rules\n\n## Process\n\nWhen asked to create a new custom rule for the package, follow these steps in order:\n\n1. Run the scaffold-new-rule.mjs script found at packages/eslint-plugin-cds/scripts/scaffold-new-rule.mjs. The script should be used every time a new rule is created. This script will ensure that the same format is used for all custom rules created.\n2. Ask the user what the custom rule should do. If the user doesn't provide enough context in the prompt, ask for clarification or follow up. Do not make assumptions about how a custom rule is meant to behave or what guideline it's supposed to enforce. After gathering the needed context, ask the user which package(s) the rule should be applied to.\n3. Once you have all the needed context, ask the user to provide at least one valid and one invalid example demonstrating when the custom rule is being followed or not. This will help you to write tests for the custom rule in a future step. Similar to step 2, if the provided examples do not give you enough context to understand the difference between a valid versus invalid example, ask the user for clarification. Do not proceed until the user has provided at least one valid and one invalid example and you have enough needed context to understand a valid and invalid example.\n4. Based on the user's provided context, fill in the rule file generated under the src/rules directory. The main section you should fill out is the create() function key that's in the rule object. You can refer to packages/eslint-plugin-cds/src/rules/custom-rule.ts for more documentation on how custom rules are formatted and information to enter. Custom rules should use the AST visitor methods to determine when rules have been violated and should report warnings in the linter. Custom rules should also verify the packages being imported before performing AST parsing. If a file doesn't import a package specified in step 2, do not report an error in the custom rule. Most custom rules already do this. As an example, you can refer to the ImportDeclaration() function in the no-v7-imports rule found here: packages/eslint-plugin-cds/src/rules/no-v7-imports.ts. The rule only reports an error when one of the CDS_PACKAGES packages is used,\n5. Using the context from step 3, write in valid and invalid test cases for the test file generated from the scaffold-new-rule.mjs script. The valid and invalid examples provided in step 3 should be included. Based off those examples and the rule criteria provided in step 2, you can add additional valid or invalid examples.\n\n## Directory placement\n\nCustom rules should always be placed in the packages/eslint-plugin-cds/src/rules directory. Tests for each custom rule should be placed in the packages/eslint-plugin-cds/tests directory.\n" }, { "path": ".claude/skills/feature-planner/SKILL.md", "content": "---\nname: feature-planner\ndescription: A guided workflow for non-technical contributors (designers, PMs) to describe a feature idea and produce a detailed technical implementation plan for an engineer. Invoke manually with /feature-planner.\ndisable-model-invocation: true\n---\n\n# Feature Planner\n\nYou are helping a non-technical contributor — most likely a designer — turn a feature idea into a detailed implementation plan that an engineer can pick up and build from.\n\nYour job has two distinct phases:\n\n1. **Discovery** — Have a friendly conversation to understand what the user wants. Speak in plain language. Never use technical jargon, code snippets, or implementation details. You are trying to understand the _what_ and _why_, not the _how_.\n\n2. **Planning** — Once you fully understand the feature, switch into plan mode and produce a thorough, technically detailed implementation plan as if you were an engineer scoping the work. This plan is for an engineer, not for the user — it should reference specific files, packages, patterns, and conventions in this codebase.\n\nThe user will not see visual output from their session (no rendered UI, no browser preview), so avoid describing how things will \"look\" unless they bring it up. Focus on what things will _do_.\n\n---\n\n## Phase 1: Discovery\n\nStart by introducing yourself and what this workflow does. Keep it warm and brief — something like:\n\n> \"Hey! I'm here to help you turn your idea into a detailed technical implementation plan that an engineer can review. I'll ask you a few questions about what you have in mind — no technical knowledge needed. Let's start: **what would you like to build?**\"\n\n### The interview loop\n\nAfter the user describes their idea, ask follow-up questions to fill in the gaps. Ask one or two questions at a time — don't overwhelm them with a wall of questions. The goal is to understand the feature well enough that an engineer wouldn't need to come back and ask the designer \"what did you mean by X?\"\n\nHere are the kinds of things you want to understand (not a checklist to run through verbatim — use your judgment about what's relevant):\n\n- **What problem does this solve?** Who benefits and how?\n- **What should it do?** Walk through the expected behavior from a user's perspective.\n- **Where does it live?** Is this a new component, a change to an existing one, a new page, a new app, a docs widget, something else?\n- **Are there states or modes?** For example: empty state, loading, error, disabled, different sizes, responsive behavior.\n- **How does a user interact with it?** Clicks, hovers, keyboard, touch, drag — whatever is relevant.\n- **Are there any existing components or patterns it should build on?** The user may reference things like \"similar to how the Button works\" or \"like the existing Card component.\"\n- **What platforms?** Web only, mobile only, or both?\n\n### Ask for visual references\n\nAt a natural point in the conversation (usually after you understand the basic idea), ask if they have any visual references to share. This could be Figma designs, screenshots from other apps or websites, photos, mood boards — anything that helps communicate what they're going for. Something like:\n\n> \"Do you have any visuals to share? Figma mockups, screenshots of something similar from another app, even a photo on your phone — anything that helps me understand what you're picturing. No worries if not, we can work from the description.\"\n\nIf they share a Figma link, use the Figma MCP tools (`get_design_context`, `get_screenshot`) to pull in the design context. If they share screenshots or images, use the `Read` tool to view them. These will be valuable input for the technical plan.\n\nNote any design details you observe but do NOT make assumptions about specific visual implementation choices (colors, spacing values, exact layout) — those decisions will happen when the engineer and designer iterate together in a future session. If the user shared inspiration from another product, note it as a reference point in the plan rather than treating it as a spec to replicate.\n\n### Confirm understanding\n\nOnce you feel you have a solid picture, summarize the feature back to the user in plain language. Frame it as what the feature will **do**, not how it will be built. For example:\n\n> \"Here's what I'm hearing:\n>\n> - **Feature:** A color matcher widget for the docs site\n> - **What it does:** A user uploads or pastes an image, picks a spot on it, and the tool finds the closest CDS design token color to that pixel\n> - **Where it lives:** New widget in the docs app\n> - **Behavior:** Shows the matched token name, the hex value, and a visual swatch side by side\n>\n> Does that capture it? Anything you'd add or change?\"\n\nIf the user wants to adjust or add things, loop back. If they confirm, move to Phase 2.\n\n---\n\n## Phase 2: Technical Planning\n\nNow you're writing for an engineer. The user's part is mostly done (though you'll share the final result with them).\n\n### Research the codebase\n\nBefore entering plan mode, do your homework:\n\n- Read the project's `AGENTS.md` to understand the architecture, packages, and conventions.\n- Explore the relevant parts of the codebase — look at existing components similar to what's being requested, understand the patterns in use, find the right package and directory for the new work.\n- If the feature involves an existing component, read that component's source code.\n- If it's a new app or docs widget, look at how existing apps/widgets are structured.\n\n### Enter plan mode\n\nUse `EnterPlanMode` to draft the implementation plan. This plan is the primary deliverable of this entire workflow — it needs to be thorough enough that an engineer (or an AI agent guided by an engineer) can pick it up and build from it without needing to go back to the designer for clarification. Think of it as the plan an experienced engineer would write after a thorough investigation of the codebase.\n\nThe plan should include:\n\n- **Summary** — One paragraph describing the feature in plain language (this is what non-technical stakeholders will read in the Linear issue).\n- **Requirements** — The confirmed list of behaviors and constraints from the discovery phase. Include any Figma links or design context gathered.\n- **Technical approach** — Which packages are affected, what new files need to be created, what existing files need to be modified, and what patterns to follow. Reference specific paths and existing code. This section should be rich with detail:\n - **Proposed types/interfaces** — Sketch out the key TypeScript types the feature needs. For a new component, this means the props type. For a new app, this means the core data model. Show actual type definitions, not just descriptions.\n - **Code sketches** — For complex logic (algorithms, shared hooks, state management), include pseudocode or skeleton implementations that show the approach. These give the engineer a concrete starting point rather than forcing them to invent the architecture from scratch.\n - **Existing patterns to follow** — Reference specific existing files the engineer should model their work after (e.g., \"follow the pattern in `Switch.tsx` for controlled/uncontrolled behavior\"). Note which existing utilities, hooks, or base components to reuse.\n- **Suggested implementation steps** — A numbered list of steps an engineer would follow, in order. Each step should be concrete and actionable.\n- **Test strategy** — Key test cases the engineer should cover, organized by category (unit tests, accessibility, integration). Reference existing test files as patterns to follow.\n- **Open questions for the engineer** — Anything that requires a technical decision the designer can't make (e.g., \"should this use framer-motion or CSS transitions?\"). Keep this section focused — only include genuine technical trade-offs, not things you could reasonably decide yourself.\n- **What's NOT in scope** — Explicitly call out visual design details (exact colors, spacing, typography) as something to be finalized in a follow-up session between the engineer and designer. This avoids premature decisions. Note: this section is about deferring _visual_ decisions — it should NOT be used to skip technical depth. The plan should be as technically detailed as possible while keeping visual design open.\n\nUse `ExitPlanMode` when the plan is ready for review.\n\n### Create a Linear issue\n\nAfter the plan is confirmed, create a Linear issue in the CDS team backlog:\n\n- **Team:** CDS\n- **Title:** A short, descriptive title for the feature\n- **Description:** The full implementation plan from above, formatted in markdown\n- **Status:** Backlog\n\nUse the Linear MCP tools to create the issue. If Linear MCP is not available, write the plan to a markdown file instead and let the user know they'll need to manually create the issue.\n\nShare the Linear issue link (or file path) with the user and let them know the next step is for an engineer to review the plan.\n\n### Wrap up\n\nEnd with something like:\n\n> \"All set! I've created a ticket with the full plan: [link]. An engineer will review it and reach out if they have any questions. When they're ready to start building, you two can work together on the visual details. Nice work getting this documented!\"\n\n---\n\n## Important guidelines\n\n- **Never talk down to the user.** They're experts in their domain (design) — they just aren't engineers. Treat them as collaborators, not students.\n- **Never use code or technical terms in Phase 1.** No file paths, no component APIs, no \"props\" or \"hooks\" or \"CSS variables.\" If you need to reference something technical, translate it: \"the design tokens your team already uses\" instead of \"CSS variables from the theme system.\"\n- **Don't make visual assumptions.** You don't know what the final design will look like. The plan should describe behavior and functionality, not prescribe specific visual choices. When you reference design, stick to what the user explicitly told you or what you observed in their Figma mockups.\n- **Be concise in conversation.** Designers are busy. Don't write essays in the discovery phase. Short questions, short summaries, keep things moving.\n- **The plan is the deliverable.** Everything in Phase 1 is in service of producing a great plan in Phase 2. Keep that end goal in mind.\n" }, { "path": ".claude/skills/feature-planner/evals/evals.json", "content": "{\n \"skill_name\": \"feature-planner\",\n \"evals\": [\n {\n \"id\": 1,\n \"prompt\": \"The user has invoked /feature-planner. Simulate a conversation where the user is a designer who wants to build a color matcher widget for the docs site.\",\n \"expected_output\": \"Warm greeting, non-technical interview, summary for confirmation, detailed technical plan referencing apps/docs/ and theme tokens, visual design noted as out-of-scope.\",\n \"assertions\": [\n {\n \"id\": \"no-jargon-in-conversation\",\n \"text\": \"The conversation/interview portion (Phase 1) does not contain technical jargon like 'props', 'hooks', 'useState', 'CSS variables', 'component API', code snippets, or file paths. Terms should be translated to plain language.\",\n \"type\": \"content\"\n },\n {\n \"id\": \"summary-before-plan\",\n \"text\": \"A clear plain-language summary of the feature is presented for the user to confirm BEFORE any technical implementation planning begins.\",\n \"type\": \"structure\"\n },\n {\n \"id\": \"plan-references-codebase\",\n \"text\": \"The technical implementation plan references specific real files, directories, or packages from the CDS codebase (not generic advice).\",\n \"type\": \"content\"\n },\n {\n \"id\": \"visual-assumptions-avoided\",\n \"text\": \"The plan explicitly notes that specific visual design details (colors, spacing, exact layout) are out of scope or to be finalized in a follow-up session with the engineer.\",\n \"type\": \"content\"\n },\n {\n \"id\": \"references-docs-app\",\n \"text\": \"The plan specifically references apps/docs/ as the location for the widget and references the theme/token source files (e.g. packages/common/src/core/theme.ts or similar) as the source of color token data.\",\n \"type\": \"content\"\n }\n ]\n },\n {\n \"id\": 2,\n \"prompt\": \"The user has invoked /feature-planner. Simulate a conversation where the user is a designer who wants a new Rating component for both web and mobile.\",\n \"expected_output\": \"Non-technical interview, handles 'I'll share it later' screenshot gracefully, summary without visual prescriptions, plan covering both web and mobile packages.\",\n \"assertions\": [\n {\n \"id\": \"no-jargon-in-conversation\",\n \"text\": \"The conversation/interview portion (Phase 1) does not contain technical jargon like 'props', 'hooks', 'useState', 'CSS variables', 'component API', code snippets, or file paths. Terms should be translated to plain language.\",\n \"type\": \"content\"\n },\n {\n \"id\": \"summary-before-plan\",\n \"text\": \"A clear plain-language summary of the feature is presented for the user to confirm BEFORE any technical implementation planning begins.\",\n \"type\": \"structure\"\n },\n {\n \"id\": \"plan-references-codebase\",\n \"text\": \"The technical implementation plan references specific real files, directories, or packages from the CDS codebase (not generic advice).\",\n \"type\": \"content\"\n },\n {\n \"id\": \"visual-assumptions-avoided\",\n \"text\": \"The plan explicitly notes that specific visual design details (colors, spacing, exact layout) are out of scope or to be finalized in a follow-up session with the engineer.\",\n \"type\": \"content\"\n },\n {\n \"id\": \"covers-both-platforms\",\n \"text\": \"The implementation plan addresses both packages/web/ and packages/mobile/ with platform-specific considerations for each.\",\n \"type\": \"content\"\n }\n ]\n },\n {\n \"id\": 3,\n \"prompt\": \"The user has invoked /feature-planner. Simulate a conversation where the user is a designer proposing a new interactive playground app.\",\n \"expected_output\": \"Not intimidated by scope, asks smart questions, handles Figma link, notes visual details as out-of-scope, plans a new app in apps/, flags technical open questions.\",\n \"assertions\": [\n {\n \"id\": \"no-jargon-in-conversation\",\n \"text\": \"The conversation/interview portion (Phase 1) does not contain technical jargon like 'props', 'hooks', 'useState', 'CSS variables', 'component API', code snippets, or file paths. Terms should be translated to plain language.\",\n \"type\": \"content\"\n },\n {\n \"id\": \"summary-before-plan\",\n \"text\": \"A clear plain-language summary of the feature is presented for the user to confirm BEFORE any technical implementation planning begins.\",\n \"type\": \"structure\"\n },\n {\n \"id\": \"plan-references-codebase\",\n \"text\": \"The technical implementation plan references specific real files, directories, or packages from the CDS codebase (not generic advice).\",\n \"type\": \"content\"\n },\n {\n \"id\": \"visual-assumptions-avoided\",\n \"text\": \"The plan explicitly notes that specific visual design details (colors, spacing, exact layout) are out of scope or to be finalized in a follow-up session with the engineer.\",\n \"type\": \"content\"\n },\n {\n \"id\": \"plans-new-app\",\n \"text\": \"The plan describes creating a new application in the apps/ directory, with steps for scaffolding, build configuration, and integration with the monorepo tooling.\",\n \"type\": \"content\"\n }\n ]\n }\n ]\n}\n" }, { "path": ".claude/skills/figma.audit-connect/SKILL.md", "content": "---\nname: figma.audit-connect\ndescription: Examines a Figma Code Connect mapping file and provides a report on the mapping's accuracy and completeness\nmodel: claude-sonnet-4-6\nargument-hint: [Component name or path to component's code connect mapping file]\ndisable-model-invocation: true\n---\n\n## Task: Audit Figma Code Connect Mapping\n\nAudit the specificed Figma Code Connect mapping file.\n\nALWAYS refresh your memory of the React Code Connect documentation here: https://developers.figma.com/docs/code-connect/react/\n\n### Inputs\n\nYou will be provided with a name or path to a Figma Code Connect mapping file.\nCode Connect files (`.figma.tsx`) are colocated with their corresponding components in this repo, typically within the component's local `__figma__` directory.\n\nSearch for the mapping file and end your task if you cannot find it.\n\nWithin the current mapping file:\n\n- Study all the property mappings defined in `props: { ... }` of the code connect mapping file.\n- Study the figma variants covered (indicated by use of `variant: { ... }`)\n - variants should be defined as separate `figma.connect` calls with the same component.\n\n### Steps\n\n1. **Retrieve Figma component data**\n - ALWAYS call Figma MCP: `get_metadata` to understand the actual component structure:\n - What are the actual property names? (they often have spaces: \"show start\")\n - What are property values vs separate properties?\n - Is this a component or a component set with variants?\n - Then call Figma MCP: `get_design_context` with the code connect disabled option enabled to get even more metadata\n\n2. **Identify Property Types Correctly**\n Before analyzing mappings, study the Figma metadata you found:\n - **ALWAYS** reference the guidelines for writing code connect mappings in the `figma.connect-best-practices` SKILL\n - **Component Properties**: Boolean toggles, dropdowns/enums in the properties panel\n - **Property Values**: Options within enum properties (e.g., \"disabled\" is a value of \"state\")\n - **Text Layers**: Named text layers that need `figma.textContent()`\n - **Instance Layers**: Named instances that need `figma.instance()` or `figma.children()`\n - **Nested Properties**: Properties exposed from child layers (marked with ↳ in Figma)\n\n **For every `figma.textContent()` usage in the mapping file**, call `get_metadata` on the referenced layer's node ID and confirm the metadata reports it as a `` node. If it is an ``, ``, ``, or any other non-text type, `figma.textContent()` will fail at runtime. Use a hardcoded placeholder string instead.\n\n **For every `figma.children()` usage**, check whether all variants of the component have a child layer with that exact name. If different variants use different layer names for the same logical prop, the mapping needs to be split into variant-specific `figma.connect()` calls.\n\n3. **Read the React component source**\n - Find and read the component's TypeScript source file, including any of its sub-components' source files\n - Study the React props for the component(s)\n\n4. **Analyze Property Coverage**\n Create a mapping analysis table, where each row is a property from the Figma `get_metadata` structure:\n\n | Figma Property | Related React Prop(s) | Mapped? | Mapping Method | Notes |\n | -------------- | --------------------- | ------- | -------------- | ----- |\n\n For each Figma property, indicate:\n - ✅ Fully mapped\n - ⚠️ Partially mapped (explain gap)\n - ❌ Not mapped (explain why it should/shouldn't be)\n\n5. **Generate Report**\n Provide a summary with:\n - **Coverage**: X/Y properties mapped\n - **Missing Mappings**: List any unmapped Figma properties that should be mapped\n - **Missing Variants**: List any component variants that are not covered by the current state of the mapping file.\n - **Incorrect Mappings**: List any mappings whose type doesn't match the actual property type from the Figma metadata\n - **Unnecessary Mappings**: Any mappings that don't correspond to Figma properties\n - **Recommended Changes**: Prioritized list of improvements with code snippets. Before suggesting any specific code changes, ensure you have read the latest React Code Connect documentation, linked above.\n" }, { "path": ".claude/skills/figma.connect-best-practices/SKILL.md", "content": "---\nname: figma.connect-best-practices\ndescription: Guidelines for writing Figma Code Connect property mappings. Use this skill when working on Figma Code Connect files, which typically end in .figma.tsx.\n---\n\n# Guidelines for writing Figma Code Connect mappings\n\n## Property Mapping Guidelines\n\n- figma.enum() - For dropdowns/variants\n\n```tsx\nvariant: figma.enum('variant', {\n 'Figma Display Name': 'codeValue',\n 'Primary': 'primary',\n 'Secondary': 'secondary',\n}),\n```\n\n- figma.boolean() - For boolean properties\n\n```tsx\ndisabled: figma.boolean('disabled'),\nloading: figma.boolean('loading'),\n```\n\n- figma.boolean() for conditional properties\\\n\nIn some cases, you only want to render a certain prop if it matches some value in Figma.\nYou can do this either by passing a partial mapping object, or setting the value to undefined.\n\n```tsx\n// Don't render the prop if 'Has label' in figma is `false`\nfigma.boolean('has label', {\n true: figma.string('label'),\n false: undefined,\n});\n```\n\n- figma.string() - For text properties (component properties with text values)\n\n```tsx\nlabel: figma.string('label'),\n```\n\n- figma.textContent() - For text layer content (when text is stored in a layer, not a property)\n\nA common pattern in Figma design systems is to override text content directly on instances rather than using component properties. Use `figma.textContent()` to extract the actual text from a named text layer.\n\n```tsx\n// Extract text from a layer named 'Title'\ntitle: figma.textContent('Title'),\n```\n\n**Key difference**: Use `figma.string()` when text is controlled by a Figma component property. Use `figma.textContent()` when text lives as content in a text layer that designers override directly.\n\n**CRITICAL: `figma.textContent()` only works on actual TEXT layers — never on component instances.**\n\nBefore using `figma.textContent('LayerName')`, always call `get_metadata` on that layer's node ID to verify its type. In the metadata response, the node must appear as a `` element. If it appears as ``, ``, ``, or any other non-text type, `figma.textContent()` will fail at runtime in Figma with a \"Layer not found\" error even though the layer name is correct.\n\nWhen the target layer is a component instance (e.g. a reusable text component), use a hardcoded placeholder string instead:\n\n```tsx\n// ❌ Wrong: 'string.label' is a component instance, not a text layer\nlabel: figma.boolean('show label', {\n true: figma.textContent('string.label'),\n false: undefined,\n}),\n\n// ✅ Correct: use a placeholder string when the layer is an instance\nlabel: figma.boolean('show label', {\n true: 'Your label here.',\n false: undefined,\n}),\n```\n\n- figma.instance() - For instance-swap properties (component slots)\n\nUse\nfigma.instance() returns the JSX from another figma.connect() call that you can use in the example.\nThis is useful for components that accept a node of another React component as a prop.\n\nIn the example below, Button accepts an instance of Icon as the icon prop.\nWe would need to have another call to figma.connect() for the `Icon` component somewhere in our code connect setup.\n\n```tsx\nfigma.connect(Button, 'https://...', {\n props: {\n icon: figma.instance('Icon'),\n },\n example: ({ icon }) => {\n return ;\n },\n});\n```\n\n- figma.children() - For child instances by layer name\n\nUse this property mapping when your React component accepts children. `figma.children` maps a Figma layer name to the `children` prop.\n\n```tsx\n// Maps child instances that aren't bound to an instance-swap prop\nicon: figma.children('IconLayer'),\n```\n\n- figma.nestedProps() - For accessing properties from child component layers\n\n```tsx\n// Access properties from a nested instance layer named 'Avatar'\navatar: figma.nestedProps('Avatar', {\n size: figma.enum('size', { ... }),\n src: figma.string('src'),\n}),\n// In example: use avatar.size, avatar.src\n```\n\n## Understanding Nested Properties (Important)\n\nIn Figma's properties panel, you may see properties with the `↳` symbol (e.g., `↳ subtitle`). This indicates the property is **exposed from a child layer**, not defined directly on the parent component.\n\n**Why this matters:** The Code Connect validation run during `figma connect publish` has limited coverage. It only validates these prop kinds:\n\n- `figma.boolean()`, `figma.enum()`, `figma.string()` - validates the property name exists\n- `figma.children()` - validates the layer name exists\n\nThese prop kinds are **NOT validated** at all:\n\n- `figma.nestedProps()` - layer name and inner property mappings are not checked\n- `figma.instance()` - layer/instance name is not checked\n- `figma.textContent()` - layer name is not checked\n\nAdditionally, validation does **not recurse** into boolean `true`/`false` branch values.\n\nThis can result in technically incorrect mappings being published to Figma withoug being caught during validation.\n\n**Incorrect approach** (will pass validation but fail at runtime):\n\n```tsx\n// ❌ Wrong: 'subtitle' should be a nested property, not a direct component property\nsubtitle: figma.boolean('show subtitle', {\n true: figma.string('subtitle'),\n false: undefined,\n}),\n```\n\n**Correct approach using figma.nestedProps():**\n\n```tsx\n// ✅ Correct: Use nestedProps to access properties from the child layer\nsubtitle: figma.boolean('show subtitle', {\n true: figma.nestedProps('subtitle', {\n text: figma.string('subtitle'),\n }),\n false: { text: undefined },\n}),\n// In example: use subtitle.text\n```\n\n**Tip:** When in doubt about whether a property is direct or nested, check if it has the `↳` symbol in Figma's properties panel. If it does, you likely need `figma.nestedProps()` or `figma.textContent()`.\n\n## Multi-Variant Support\n\nFor components with multiple variants in Figma, create separate figma.connect() calls:\n\n```tsx\n// Default variant\nfigma.connect(ComponentName, 'figma-url', {\n /* props */\n});\n\n// Specific variant\nfigma.connect(ComponentName, 'figma-url', {\n variant: { 'show suffix': true },\n props: {\n /* variant-specific props */\n },\n example: (props) => ,\n});\n```\n\n**Use variant-specific connects when child layer names differ across variants.**\n\n`figma.children('LayerName')` only matches layers with that exact name. If different variants use different layer names for the same logical prop (e.g. `Button` for the single-action variant and `ButtonGroup` for the multi-action variant), a single `figma.children()` call will silently produce nothing for the non-matching variants. Split into separate connects with `variant: { ... }` filters:\n\n```tsx\n// ❌ Wrong: 'ButtonGroup' doesn't exist in the single-action variant\nfigma.connect(Footer, url, {\n props: { action: figma.children('ButtonGroup') },\n example: ({ action }) =>