[
  {
    "path": ".github/workflows/publish.yml",
    "content": "name: Publish Package\n\non:\n  workflow_dispatch:\n    inputs:\n      version:\n        description: \"Version to publish (must match package.json)\"\n        required: true\n        type: string\n\n  release:\n    types: [published]\n\npermissions:\n  id-token: write\n  contents: read\n\njobs:\n  publish:\n    runs-on: ubuntu-latest\n    steps:\n      - uses: actions/checkout@v4\n\n      - uses: actions/setup-node@v4\n        with:\n          node-version: '22'\n          registry-url: 'https://registry.npmjs.org'\n\n      - run: npm ci\n\n      - name: Verify version\n        run: |\n          PKG_VERSION=$(node -p \"require('./package.json').version\")\n          if [ \"$PKG_VERSION\" != \"${{ inputs.version }}\" ]; then\n            echo \"ERROR: package.json version ($PKG_VERSION) != requested version (${{ inputs.version }})\"\n            exit 1\n          fi\n          echo \"Publishing pi-search-hub@$PKG_VERSION\"\n\n      - run: npm publish"
  },
  {
    "path": ".gitignore",
    "content": "node_modules/\ndist/\n*.log\n.DS_Store\n\n# API keys — NEVER commit these\n.pi/\nsearch.json\nsearch.json.local\n*.secret.*\n\n# Dev/research docs (local only — never push)\nfindings.md\nresearcher*.md\nscout*.md\nhandoffs/\ndocs/\nbenchmark/\n\n# Benchmark reports\nbenchmark/benchmark-current-report.md\n\n# Subagent session artifacts\ncontext.md\nresearch.md\n\n# Other extensions (not search-related)\nextensions/lint-markdown.ts\nextensions/ralph-loop.ts\n\n# Ralph loop state (local development)\n.ralph/\n\n# Personal/social media drafts\ngithub-meta.md\nreddit-post.md\n"
  },
  {
    "path": ".npmignore",
    "content": "# Session artifacts\nhandoffs/\nscout*.md\nresearcher*.md\ncontext.md\nresearch.md\n\n# Personal/social media drafts\ngithub-meta.md\nreddit-post.md\n\n# Git\n.git/\n.gitignore\n\n# All markdown except README\n*.md\n!README.md\n\n# All TypeScript except the search extension and backends\nextensions/*.ts\n!extensions/search-hub.ts\n\n# All non-search files\n.pi/\nbenchmark/\n\n# Ralph loop state\n.ralph/\n\n# Docs artifacts\ndocs/\n\n# Tests\nbackends/*.test.ts\n\n# Misc\n*.log\n.DS_Store"
  },
  {
    "path": "README.md",
    "content": "# pi-search-hub\n\nUnified web search + content extraction extension for [pi](https://pi.dev) with **12 backend providers** (all working). One `web_search` tool, one `web_read` tool, auto-fallback, RRF-ranked combine mode, and credential resolution via env/shell/literal.\n\n## Installation\n\n```bash\npi install npm:pi-search-hub\n```\n\n> **Note for DuckDuckGo backend:** Requires the `ddgs` Python package. Install with:\n>\n> - Linux/macOS: `pip3 install ddgs`\n> - Windows: `pip install ddgs`\n\n## Usage\n\n### Web Search\n\nAfter installing, just ask naturally:\n\n```text\nSearch for recent AI agent frameworks.\n```\n\n```text\nWhat's the latest news on Llama 4?\n```\n\nOr use the tools directly — the agent picks the best configured backend automatically:\n\n- `web_search` — search the web with auto-fallback or parallel combine mode\n- `web_read` — fetch any URL as clean markdown\n\n### Combine Mode\n\nSet `combine=true` to query **ALL enabled backends in parallel** with Reciprocal Rank Fusion (RRF) ranking:\n\n```text\nSearch for \"Rust vs Go performance benchmarks\" with combine=true to get results from all backends\n```\n\n**Combine mode benefits:**\n\n- Broader coverage across multiple search indexes\n- Results ranked by RRF — position-based scoring across all backends\n- Each result shows which backend found it\n- URL deduplication with content-aware merge (prefers richest result)\n- Useful for comprehensive research or when you want diverse sources\n\n**Tradeoff:** Uses more API quota per query (all backends are called), but you get more comprehensive results.\n\n### Read Web Pages\n\nFetch any URL as clean markdown — great for extracting article content, docs, or reference pages.\n**Note: `web_read` uses [Jina Reader](https://r.jina.ai/) to fetch and convert URLs to markdown.**\n\n```text\nRead https://docs.example.com/api-reference\n```\n\nThe `web_read` tool supports:\n\n- **objective** — CSS selector to target specific content (e.g. \"div.article-body\")\n- **keywords** — relevant terms to highlight on long pages\n- **mode** — `rush` for speed (return innerText) or `smart` (markdown extraction)\n- **fresh** — bypass cache when freshness matters\n\n## Supported Backends\n\n| #   | Backend               | Free Tier                     | API Key? | How to get key                                                    |\n| --- | --------------------- | ----------------------------- | :------: | ----------------------------------------------------------------- |\n| 1   | **DuckDuckGo**        | Unlimited (rate-limited)      |  **No**  | `pip install ddgs` (Linux/macOS: `pip3`)                          |\n| 2   | **Jina AI**           | Search: key req. web_read: free (no key) |   Yes   | [jina.ai](https://jina.ai)   |\n| 3   | **Marginalia Search** | Unlimited (rate-limited)      | **No**†  | [marginalia.nu](https://www.marginalia.nu/marginalia-search/api/) |\n| 4   | **Tavily**            | 1,000 calls/month             |   Yes    | [tavily.com](https://tavily.com)                                  |\n| 5   | **Serper** (Google)   | 2,500 free queries (one-time) |   Yes    | [serper.dev](https://serper.dev)                                  |\n| 6   | **Brave**             | 2,000 queries/month           |   Yes    | [brave.com/search/api](https://brave.com/search/api)              |\n| 7   | **Firecrawl**         | 500 free credits              |   Yes    | [firecrawl.dev](https://www.firecrawl.dev)                        |\n| 8   | **Exa**               | 1,000 free queries/month      |   Yes    | [exa.ai](https://dashboard.exa.ai/api-keys)                       |\n| 9   | **LangSearch**        | Genuinely free, no CC         |   Yes    | [langsearch.com](https://langsearch.com)                          |\n| 10  | **WebSearchAPI.ai**   | 2,000 free credits            |   Yes    | [websearchapi.ai](https://www.websearchapi.ai)                    |\n| 11  | **Perplexity Sonar**  | Paid (usage-based)            |   Yes    | [perplexity.ai](https://docs.perplexity.ai)                       |\n| 12  | **SearXNG**           | Self-hosted, unlimited        |  **No**  | [docs.searxng.org](https://docs.searxng.org)                      |\n\n> † Marginalia Search uses `public` as a shared API key — no registration required, but subject to a shared rate limit.\n>\n> **Jina AI:** Search (`s.jina.ai`) requires a free API key from [jina.ai](https://jina.ai). Content extraction via `web_read` uses Jina Reader (`r.jina.ai`) which is **free and needs no API key**.\n>\n> **Perplexity Sonar** supports multiple model variants. Set `model` in your Perplexity backend config to choose: `sonar` (default, fast), `sonar-pro` (higher quality), `sonar-deep-research` (multi-step reasoning), or `sonar-reasoning` (DeepSeek R1-based).\n>\n> **SearXNG** is a self-hosted metasearch engine. Run your own instance (or use a public one), no API key required. Configure the instance URL in `.pi/search.json`.\n>\n> **Firecrawl** uses `api.firecrawl.dev/v2/search` with a `data.web[]` response shape. The v1 endpoint is deprecated.\n>\n> **Exa** (March 2026) includes content for the first 10 results per request at no extra cost. Content extraction is enabled by default.\n\n## Configuration\n\nConfigure backends globally (all projects) or per-project:\n\n**Global:** `~/.pi/agent/extensions/search.json`\n**Project:** `.pi/search.json` (project takes precedence)\n\n```json\n{\n  \"defaultBackend\": \"auto\",\n  \"backends\": {\n    \"duckduckgo\": { \"enabled\": true },\n    \"jina\": { \"enabled\": true, \"apiKey\": \"JINA_API_KEY\" },\n    \"marginalia\": { \"enabled\": true },\n    \"serper\": { \"enabled\": true, \"apiKey\": \"SERPER_API_KEY\" },\n    \"tavily\": { \"enabled\": true, \"apiKey\": \"TAVILY_API_KEY\" },\n    \"brave\": { \"enabled\": true, \"apiKey\": \"BRAVE_API_KEY\" },\n    \"exa\": { \"enabled\": true, \"apiKey\": \"EXA_API_KEY\" },\n    \"firecrawl\": { \"enabled\": true, \"apiKey\": \"FIRECRAWL_API_KEY\" },\n    \"langsearch\": { \"enabled\": true, \"apiKey\": \"LANGSEARCH_API_KEY\" },\n    \"websearchapi\": { \"enabled\": true, \"apiKey\": \"WEBSEARCHAPI_API_KEY\" },\n    \"perplexity\": {\n      \"enabled\": true,\n      \"apiKey\": \"PERPLEXITY_API_KEY\",\n      \"model\": \"sonar\"\n    },\n    \"searxng\": { \"enabled\": true, \"instanceUrl\": \"http://localhost:8888\" }\n  }\n}\n```\n\n### Credential Resolution\n\nThe `apiKey` field supports four formats (following pi-web-providers convention):\n\n| `apiKey` value            | Resolved from                           | Example                            |\n| ------------------------- | --------------------------------------- | ---------------------------------- |\n| `\"SERPER_API_KEY\"`        | `process.env.SERPER_API_KEY`            | ALL_CAPS → env var                 |\n| `\"!pass show api/serper\"` | stdout of shell command (cached)        | `!` prefix → exec                  |\n| `\"sk-abc123...\"`          | Used as-is                              | Literal key (backwards compatible) |\n| _(unset)_                 | `SEARCH_<BACKEND>_API_KEY` env fallback | Auto-enables backend               |\n\n**Env var references:** Any ALL_CAPS string is treated as an environment variable name (not a literal). If the referenced env var is unset, a warning is printed (your literal key is not silently discarded).\n\n**Shell commands:** Commands prefixed with `!` are executed via `execSync` with a 5s timeout. Results are cached and invalidated when config is reloaded (editing the config file clears the cache).\n\n**Convenience env vars:** Backends are auto-enabled when these env vars are set (even with no config entry):\n\n```bash\nexport SEARCH_SERPER_API_KEY=\"sk-...\"\nexport SEARCH_TAVILY_API_KEY=\"sk-...\"\nexport SEARCH_EXA_API_KEY=\"sk-...\"\n# ...\n```\n\n```json\n{\n  \"backends\": {\n    \"serper\": { \"enabled\": true, \"apiKey\": \"SERPER_API_KEY\" }\n  }\n}\n```\n\n**To rotate a shell-command key:** Update the secret in your password manager, then trigger a config reload (edit the config file, or wait 10s for automatic refresh).\n\nOr use the interactive setup:\n\n```\n/search-setup\n```\n\n## Commands\n\n| Command          | Description                                                       |\n| ---------------- | ----------------------------------------------------------------- |\n| `/search-setup`  | Interactive prompt to configure API keys and instance URLs        |\n| `/search-status` | Show which backends are active, which have keys, and their status |\n\n> **Tip:** After running `/search-setup` or editing your config, run `/reload` to activate changes without restarting pi.\n\n## How auto mode works\n\n### Fallback Mode (default, `combine=false`)\n\n1. Tries each enabled backend in order from your config\n2. If a backend fails (rate limit, auth error, etc.), moves to the next one\n3. Jina AI search requires a free API key from [jina.ai](https://jina.ai) (get one at jina.ai/reader). DuckDuckGo requires no API key. Both serve as safety nets\n4. Returns results from the first backend that succeeds\n5. If all backends fail, reports the collected errors\n\n### Combine Mode (`combine=true`)\n\n1. Queries **ALL** enabled backends in parallel\n2. Each backend receives `numResults / numBackends` as a target\n3. Results are merged using **Reciprocal Rank Fusion** (RRF) — position-based scoring that works across incompatible ranking systems\n4. Each result shows its source backend (e.g., `*Source: Tavily*`)\n5. URL dedup prefers the result with the richest content (content > snippet)\n6. Backend statistics are displayed (which succeeded, result counts, errors)\n\n### RRF Scoring\n\nRRF assigns each result a score of `Σ(1 / (60 + rank_i))` across all backends that returned it. Results are ranked by score, then by number of backends that found them. This means a result ranked #1 by one backend and #5 by another beats a result ranked #4 by two backends.\n\n## Security\n\n- API keys are stored in local config files only (`~/.pi/agent/extensions/search.json` or `.pi/search.json`), never sent to any third party besides the chosen backend\n- **Env vars and shell commands** are supported for credential resolution — the config file is trusted (you own it), but never commit plain API keys to version control\n- DuckDuckGo queries use spawned Python subprocess (abortable via signal)\n- All HTTP backends have a 30-second timeout; shell commands for credentials have a 5-second timeout\n- Error messages are sanitized — API response bodies are truncated and key-like patterns are redacted\n- The `.pi/` directory is in `.gitignore` — **never commit API keys to version control**\n\n## Testing\n\n```bash\n# Run unit tests for backend parsers\nnpx vitest run backends/parsers.test.ts\n\n# Quick test Jina AI (with your free API key)\ncurl -s -H \"Authorization: Bearer $JINA_API_KEY\" \"https://s.jina.ai/?q=test&format=json\" | jq .\n\n# Quick test via curl with your configured key\ncurl -X POST \"https://api.exa.ai/search\" \\\n  -H \"Content-Type: application/json\" \\\n  -H \"x-api-key: $KEY\" \\\n  -d '{\"query\": \"test\", \"numResults\": 3, \"contents\": {\"text\": true}}'\n\n# Quick test Perplexity Sonar (use \"sonar-pro\" or \"sonar-deep-research\" for model)\ncurl -X POST \"https://api.perplexity.ai/chat/completions\" \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer $KEY\" \\\n  -d '{\"model\": \"sonar\", \"messages\": [{\"role\": \"user\", \"content\": \"test\"}], \"search_context_size\": \"low\"}'\n\n# Quick test Firecrawl (v2 endpoint — code still uses v1)\ncurl -X POST \"https://api.firecrawl.dev/v2/search\" \\\n  -H \"Content-Type: application/json\" \\\n  -H \"Authorization: Bearer $KEY\" \\\n  -d '{\"query\": \"test\", \"limit\": 3}'\n\n# Quick test SearXNG (replace URL with your instance)\ncurl \"http://localhost:8888/search?q=test&format=json&count=3\"\n```\n\n## Adding a new backend\n\nBackends are registered via the `BACKEND_DEFS` registry in `extensions/search-hub.ts`. Define a `search` function and add one entry to the registry:\n\n```typescript\nconst BACKEND_DEFS: Record<string, BackendRunner> = {\n  // ... existing entries\n  mybackend: {\n    needsKey: true,\n    needsKeyFromConfig: false,\n    needsInstanceUrl: false,\n    label: \"My Backend\",\n    setupLabel: \"My Backend (free tier description)\",\n    search: async (query, numResults, { key, signal }) => {\n      const result = await searchMyBackend(query, numResults, key!, signal);\n      return { results: result.results };\n    },\n  },\n};\n```\n\nThe registry handles dispatching, key resolution, formatting labels, and setup menu — no other edits needed.\n\n## License\n\nMIT\n\n---\n\n<p align=\"true\">Proudly created with <a href=\"https://pi.dev\">pi</a></p>\n"
  },
  {
    "path": "backends/parsers.test.ts",
    "content": "import { describe, it, expect } from \"vitest\";\nimport {\n\tparseMarginalia,\n\tparseWebSearchAPI,\n\tparseSerper,\n\tparseTavily,\n\tparseExa,\n\tparseBrave,\n\tparseLangSearch,\n\tparseFirecrawl,\n\tparsePerplexity,\n\tparseSearXNG,\n\tparseJina,\n} from \"./parsers.js\";\n\n// ---------------------------------------------------------------------------\n// Marginalia\n// ---------------------------------------------------------------------------\n\ndescribe(\"parseMarginalia\", () => {\n\tit(\"parses standard response\", () => {\n\t\tconst data = {\n\t\t\tresults: [\n\t\t\t\t{ title: \"Test 1\", url: \"https://example.com/1\", description: \"Desc 1\" },\n\t\t\t\t{ title: \"Test 2\", url: \"https://example.com/2\", description: \"Desc 2\" },\n\t\t\t],\n\t\t};\n\t\tconst results = parseMarginalia(data, 10);\n\t\texpect(results).toHaveLength(2);\n\t\texpect(results[0]).toEqual({ title: \"Test 1\", url: \"https://example.com/1\", snippet: \"Desc 1\" });\n\t});\n\n\tit(\"handles missing fields gracefully\", () => {\n\t\tconst data = { results: [{}] };\n\t\tconst results = parseMarginalia(data, 10);\n\t\texpect(results).toHaveLength(1);\n\t\texpect(results[0]).toEqual({ title: \"\", url: \"\", snippet: \"\" });\n\t});\n\n\tit(\"truncates long descriptions to 500 chars\", () => {\n\t\tconst data = { results: [{ description: \"x\".repeat(600) }] };\n\t\tconst results = parseMarginalia(data, 10);\n\t\texpect(results[0].snippet.length).toBe(500);\n\t});\n\n\tit(\"respects numResults limit\", () => {\n\t\tconst data = { results: Array.from({ length: 10 }, (_, i) => ({ title: `T${i}`, url: `https://e.com/${i}` })) };\n\t\tconst results = parseMarginalia(data, 3);\n\t\texpect(results).toHaveLength(3);\n\t});\n\n\tit(\"handles empty results\", () => {\n\t\tconst results = parseMarginalia({}, 10);\n\t\texpect(results).toHaveLength(0);\n\t});\n});\n\n// ---------------------------------------------------------------------------\n// WebSearchAPI\n// ---------------------------------------------------------------------------\n\ndescribe(\"parseWebSearchAPI\", () => {\n\tit(\"parses organic results\", () => {\n\t\tconst data = {\n\t\t\torganic: [\n\t\t\t\t{ title: \"Web 1\", url: \"https://web.com/1\", description: \"Web desc\" },\n\t\t\t],\n\t\t};\n\t\tconst results = parseWebSearchAPI(data, 10);\n\t\texpect(results).toHaveLength(1);\n\t\texpect(results[0]).toEqual({ title: \"Web 1\", url: \"https://web.com/1\", snippet: \"Web desc\" });\n\t});\n\n\tit(\"handles missing organic field\", () => {\n\t\tconst results = parseWebSearchAPI({}, 10);\n\t\texpect(results).toHaveLength(0);\n\t});\n\n\tit(\"handles organic as non-array\", () => {\n\t\tconst results = parseWebSearchAPI({ organic: \"not an array\" }, 10);\n\t\texpect(results).toHaveLength(0);\n\t});\n});\n\n// ---------------------------------------------------------------------------\n// Serper\n// ---------------------------------------------------------------------------\n\ndescribe(\"parseSerper\", () => {\n\tit(\"maps link to url\", () => {\n\t\tconst data = { organic: [{ title: \"S\", link: \"https://s.com\", snippet: \"snip\" }] };\n\t\tconst results = parseSerper(data, 10);\n\t\texpect(results[0]).toEqual({ title: \"S\", url: \"https://s.com\", snippet: \"snip\" });\n\t});\n});\n\n// ---------------------------------------------------------------------------\n// Tavily\n// ---------------------------------------------------------------------------\n\ndescribe(\"parseTavily\", () => {\n\tit(\"maps content to snippet and preserves content field\", () => {\n\t\tconst data = { results: [{ title: \"T\", url: \"https://t.com\", content: \"full content\" }] };\n\t\tconst results = parseTavily(data, 10);\n\t\texpect(results[0].snippet).toBe(\"full content\");\n\t\texpect(results[0].content).toBe(\"full content\");\n\t});\n});\n\n// ---------------------------------------------------------------------------\n// Exa\n// ---------------------------------------------------------------------------\n\ndescribe(\"parseExa\", () => {\n\tit(\"prefers text over highlight for snippet\", () => {\n\t\tconst data = { results: [{ title: \"E\", url: \"https://e.com\", text: \"text val\", highlight: \"high val\" }] };\n\t\tconst results = parseExa(data, 10);\n\t\texpect(results[0].snippet).toBe(\"text val\");\n\t});\n\n\tit(\"falls back to highlight when no text\", () => {\n\t\tconst data = { results: [{ title: \"E\", url: \"https://e.com\", highlight: \"high val\" }] };\n\t\tconst results = parseExa(data, 10);\n\t\texpect(results[0].snippet).toBe(\"high val\");\n\t});\n});\n\n// ---------------------------------------------------------------------------\n// Brave\n// ---------------------------------------------------------------------------\n\ndescribe(\"parseBrave\", () => {\n\tit(\"navigates web.results path\", () => {\n\t\tconst data = { web: { results: [{ title: \"B\", url: \"https://b.com\", description: \"desc\" }] } };\n\t\tconst results = parseBrave(data, 10);\n\t\texpect(results[0]).toEqual({ title: \"B\", url: \"https://b.com\", snippet: \"desc\" });\n\t});\n\n\tit(\"returns empty when web is missing\", () => {\n\t\texpect(parseBrave({}, 10)).toHaveLength(0);\n\t});\n});\n\n// ---------------------------------------------------------------------------\n// LangSearch\n// ---------------------------------------------------------------------------\n\ndescribe(\"parseLangSearch\", () => {\n\tit(\"navigates data.webPages.value path\", () => {\n\t\tconst data = { data: { webPages: { value: [{ name: \"LS\", url: \"https://ls.com\", snippet: \"sn\" }] } } };\n\t\tconst results = parseLangSearch(data, 10);\n\t\texpect(results[0].title).toBe(\"LS\");\n\t\texpect(results[0].snippet).toBe(\"sn\");\n\t});\n\n\tit(\"prefers name over title\", () => {\n\t\tconst data = { data: { webPages: { value: [{ name: \"Name\", title: \"Title\", url: \"https://ls.com\" }] } } };\n\t\tconst results = parseLangSearch(data, 10);\n\t\texpect(results[0].title).toBe(\"Name\");\n\t});\n});\n\n// ---------------------------------------------------------------------------\n// Firecrawl v2\n// ---------------------------------------------------------------------------\n\ndescribe(\"parseFirecrawl\", () => {\n\tit(\"parses v2 object response with web array\", () => {\n\t\tconst data = { data: { web: [{ title: \"FC\", url: \"https://fc.com\", description: \"d\" }] } };\n\t\tconst results = parseFirecrawl(data, 10);\n\t\texpect(results[0]).toEqual({ title: \"FC\", url: \"https://fc.com\", snippet: \"d\" });\n\t});\n\n\tit(\"parses v2 flat array response\", () => {\n\t\tconst data = { data: [{ title: \"FC\", url: \"https://fc.com\" }] };\n\t\tconst results = parseFirecrawl(data, 10);\n\t\texpect(results).toHaveLength(1);\n\t});\n\n\tit(\"falls back to v1 results field\", () => {\n\t\tconst data = { results: [{ title: \"FC1\", url: \"https://fc.com/1\" }] };\n\t\tconst results = parseFirecrawl(data, 10);\n\t\texpect(results).toHaveLength(1);\n\t});\n\n\tit(\"falls back to images when web is empty\", () => {\n\t\tconst data = { data: { web: [], images: [{ title: \"Img\", url: \"https://img.com\" }] } };\n\t\tconst results = parseFirecrawl(data, 10);\n\t\texpect(results[0].title).toBe(\"Img\");\n\t});\n});\n\n// ---------------------------------------------------------------------------\n// Perplexity\n// ---------------------------------------------------------------------------\n\ndescribe(\"parsePerplexity\", () => {\n\tit(\"builds answer result from content + citations\", () => {\n\t\tconst data = {\n\t\t\tcitations: [\"https://src1.com\", \"https://src2.com\"],\n\t\t\tchoices: [{ message: { content: \"The answer is 42\" } }],\n\t\t};\n\t\tconst results = parsePerplexity(data, \"what is the answer\", 10);\n\t\texpect(results[0].title).toBe(\"Answer: what is the answer\");\n\t\texpect(results[0].snippet).toBe(\"The answer is 42\");\n\t\texpect(results).toHaveLength(3); // answer + 2 citations\n\t});\n\n\tit(\"extracts hostname as title from citation URLs\", () => {\n\t\tconst data = { citations: [\"https://www.example.com/path/to/page\"] };\n\t\tconst results = parsePerplexity(data, \"test\", 10);\n\t\texpect(results[0].title).toBe(\"example.com/path/to/page\");\n\t});\n\n\tit(\"handles empty citations\", () => {\n\t\tconst data = { citations: [] };\n\t\tconst results = parsePerplexity(data, \"test\", 10);\n\t\texpect(results).toHaveLength(0);\n\t});\n});\n\n// ---------------------------------------------------------------------------\n// SearXNG\n// ---------------------------------------------------------------------------\n\ndescribe(\"parseSearXNG\", () => {\n\tit(\"prefers content over snippet\", () => {\n\t\tconst data = { results: [{ title: \"SX\", url: \"https://sx.com\", content: \"content\", snippet: \"snip\" }] };\n\t\tconst results = parseSearXNG(data, 10);\n\t\texpect(results[0].snippet).toBe(\"content\");\n\t});\n});\n\n// ---------------------------------------------------------------------------\n// Jina\n// ---------------------------------------------------------------------------\n\ndescribe(\"parseJina\", () => {\n\tit(\"parses data array with content\", () => {\n\t\tconst data = { data: [{ title: \"J\", url: \"https://j.com\", content: \"full article\" }] };\n\t\tconst results = parseJina(data, 10);\n\t\texpect(results[0].title).toBe(\"J\");\n\t\texpect(results[0].content).toBe(\"full article\");\n\t\texpect(results[0].snippet).toBe(\"full article\");\n\t});\n\n\tit(\"truncates content to 2000 chars\", () => {\n\t\tconst data = { data: [{ title: \"J\", url: \"https://j.com\", content: \"x\".repeat(3000) }] };\n\t\tconst results = parseJina(data, 10);\n\t\texpect(results[0].content.length).toBe(2000);\n\t});\n});\n"
  },
  {
    "path": "backends/parsers.ts",
    "content": "/**\n * Pure response parsers for search backends.\n * Each takes raw JSON data and returns normalized results.\n * No HTTP, no side effects — easy to unit test.\n */\n\nexport interface ParsedResult {\n\ttitle: string;\n\turl: string;\n\tsnippet: string;\n}\n\n// ---------------------------------------------------------------------------\n// Marginalia Search\n// Response: { results: [{ title, url, description }] }\n// ---------------------------------------------------------------------------\n\nexport function parseMarginalia(\n\tdata: Record<string, unknown>,\n\tnumResults: number,\n): ParsedResult[] {\n\tconst results = (data.results || []) as Array<Record<string, unknown>>;\n\treturn results.slice(0, numResults).map((r) => ({\n\t\ttitle: (r.title as string) || \"\",\n\t\turl: (r.url as string) || \"\",\n\t\tsnippet: ((r.description as string) || \"\").slice(0, 500),\n\t}));\n}\n\n// ---------------------------------------------------------------------------\n// WebSearchAPI.ai\n// Response: { organic: [{ title, url, description }] }\n// ---------------------------------------------------------------------------\n\nexport function parseWebSearchAPI(\n\tdata: Record<string, unknown>,\n\tnumResults: number,\n): ParsedResult[] {\n\tconst rawResults = data.organic;\n\tconst organic = Array.isArray(rawResults) ? rawResults : [];\n\treturn organic.slice(0, numResults).map((r) => ({\n\t\ttitle: (r.title as string) || \"\",\n\t\turl: (r.url as string) || \"\",\n\t\tsnippet: ((r.description as string) || \"\").slice(0, 500),\n\t}));\n}\n\n// ---------------------------------------------------------------------------\n// Serper.dev (Google)\n// Response: { organic: [{ title, link, snippet }] }\n// ---------------------------------------------------------------------------\n\nexport function parseSerper(\n\tdata: Record<string, unknown>,\n\tnumResults: number,\n): ParsedResult[] {\n\tconst rawResults = data.organic;\n\tconst results = Array.isArray(rawResults) ? rawResults : [];\n\treturn results.slice(0, numResults).map((r) => ({\n\t\ttitle: (r.title as string) || \"\",\n\t\turl: (r.link as string) || \"\",\n\t\tsnippet: (r.snippet as string) || \"\",\n\t}));\n}\n\n// ---------------------------------------------------------------------------\n// Tavily\n// Response: { results: [{ title, url, content }] }\n// ---------------------------------------------------------------------------\n\nexport interface TavilyParsedResult extends ParsedResult {\n\tcontent?: string;\n}\n\nexport function parseTavily(\n\tdata: Record<string, unknown>,\n\tnumResults: number,\n): TavilyParsedResult[] {\n\tconst rawResults = data.results;\n\tconst results = Array.isArray(rawResults) ? rawResults : [];\n\treturn results.slice(0, numResults).map((r) => ({\n\t\ttitle: (r.title as string) || \"\",\n\t\turl: (r.url as string) || \"\",\n\t\tsnippet: (r.content as string) || \"\",\n\t\tcontent: r.content as string,\n\t}));\n}\n\n// ---------------------------------------------------------------------------\n// Exa\n// Response: { results: [{ title, url, text, highlight }] }\n// ---------------------------------------------------------------------------\n\nexport function parseExa(\n\tdata: Record<string, unknown>,\n\tnumResults: number,\n): ParsedResult[] {\n\tconst rawResults = data.results;\n\tconst results = Array.isArray(rawResults) ? rawResults : [];\n\treturn results.slice(0, numResults).map((r) => ({\n\t\ttitle: (r.title as string) || \"\",\n\t\turl: (r.url as string) || \"\",\n\t\tsnippet: ((r.text as string) || (r.highlight as string) || \"\").slice(0, 500),\n\t}));\n}\n\n// ---------------------------------------------------------------------------\n// Brave Search\n// Response: { web: { results: [{ title, url, description }] } }\n// ---------------------------------------------------------------------------\n\nexport function parseBrave(\n\tdata: Record<string, unknown>,\n\tnumResults: number,\n): ParsedResult[] {\n\tconst web = data.web;\n\tif (!web || typeof web !== \"object\") {\n\t\treturn [];\n\t}\n\tconst rawResults = (web as Record<string, unknown>).results;\n\tconst results = Array.isArray(rawResults) ? rawResults : [];\n\treturn results.slice(0, numResults).map((r) => ({\n\t\ttitle: (r.title as string) || \"\",\n\t\turl: (r.url as string) || \"\",\n\t\tsnippet: ((r.description as string) || \"\").slice(0, 500),\n\t}));\n}\n\n// ---------------------------------------------------------------------------\n// LangSearch\n// Response: { data: { webPages: { value: [{ name, url, snippet, description }] } } }\n// ---------------------------------------------------------------------------\n\nexport function parseLangSearch(\n\tdata: Record<string, unknown>,\n\tnumResults: number,\n): ParsedResult[] {\n\tconst pages = (data.data as Record<string, unknown>)?.webPages as Record<string, unknown> | undefined;\n\tconst results = (pages?.value || data.results || data.data || []) as Array<Record<string, unknown>>;\n\treturn results.slice(0, numResults).map((r) => ({\n\t\ttitle: (r.name as string) || (r.title as string) || \"\",\n\t\turl: (r.url as string) || (r.link as string) || \"\",\n\t\tsnippet: ((r.snippet as string) || (r.description as string) || \"\").slice(0, 500),\n\t}));\n}\n\n// ---------------------------------------------------------------------------\n// Firecrawl v2\n// Response: { data: { web: [...] } or data: [...] or { results: [...] } (v1 fallback)\n// ---------------------------------------------------------------------------\n\nexport function parseFirecrawl(\n\tdata: Record<string, unknown>,\n\tnumResults: number,\n): ParsedResult[] {\n\tconst rawData = data.data;\n\tlet results: Array<Record<string, unknown>> = [];\n\tif (Array.isArray(rawData)) {\n\t\tresults = rawData;\n\t} else if (typeof rawData === \"object\" && rawData !== null) {\n\t\tconst obj = rawData as Record<string, unknown>;\n\t\tresults = Array.isArray(obj.web) ? obj.web : [];\n\t\tif (results.length === 0) {\n\t\t\tif (Array.isArray(obj.images)) results = obj.images as Array<Record<string, unknown>>;\n\t\t\telse if (Array.isArray(obj.news)) results = obj.news as Array<Record<string, unknown>>;\n\t\t}\n\t} else if (Array.isArray(data.results)) {\n\t\tresults = data.results;\n\t}\n\treturn results.slice(0, numResults).map((r) => ({\n\t\ttitle: (r.title as string) || \"\",\n\t\turl: (r.url as string) || \"\",\n\t\tsnippet: ((r.description as string) || (r.snippet as string) || \"\").slice(0, 500),\n\t}));\n}\n\n// ---------------------------------------------------------------------------\n// Perplexity Sonar\n// Response: { citations: string[], choices: [{ message: { content } }] }\n// ---------------------------------------------------------------------------\n\nexport function parsePerplexity(\n\tdata: Record<string, unknown>,\n\tquery: string,\n\tnumResults: number,\n): ParsedResult[] {\n\tconst citations = (data.citations as string[]) || [];\n\tconst message = (data.choices as Array<Record<string, unknown>>)?.[0]?.message as Record<string, unknown> | undefined;\n\tconst answerText = (message?.content as string) || \"\";\n\n\tconst results: ParsedResult[] = [];\n\n\tif (answerText) {\n\t\tresults.push({\n\t\t\ttitle: `Answer: ${query}`,\n\t\t\turl: citations[0] || \"\",\n\t\t\tsnippet: answerText.slice(0, 500),\n\t\t});\n\t}\n\n\tfor (const url of citations) {\n\t\ttry {\n\t\t\tconst u = new URL(url);\n\t\t\tconst title = u.hostname.replace(/^www\\./, \"\") + (u.pathname !== \"/\" ? u.pathname.slice(0, 60) : \"\");\n\t\t\tresults.push({ title: title || url, url, snippet: \"\" });\n\t\t} catch {\n\t\t\tresults.push({ title: url, url, snippet: \"\" });\n\t\t}\n\t}\n\n\treturn results.slice(0, numResults);\n}\n\n// ---------------------------------------------------------------------------\n// SearXNG\n// Response: { results: [{ title, url, content, snippet }] }\n// ---------------------------------------------------------------------------\n\nexport function parseSearXNG(\n\tdata: Record<string, unknown>,\n\tnumResults: number,\n): ParsedResult[] {\n\tconst rawResults = data.results as Array<Record<string, unknown>> | undefined;\n\tconst results = Array.isArray(rawResults) ? rawResults : [];\n\treturn results.slice(0, numResults).map((r) => ({\n\t\ttitle: (r.title as string) || \"\",\n\t\turl: (r.url as string) || \"\",\n\t\tsnippet: ((r.content as string) || (r.snippet as string) || \"\").slice(0, 500),\n\t}));\n}\n\n// ---------------------------------------------------------------------------\n// Jina AI (s.jina.ai)\n// Response: { data: [{ title, url, content, description }] }\n// ---------------------------------------------------------------------------\n\nexport interface JinaParsedResult extends ParsedResult {\n\tcontent: string;\n}\n\nexport function parseJina(\n\tdata: Record<string, unknown>,\n\tnumResults: number,\n): JinaParsedResult[] {\n\tconst rawData = data.data as Array<Record<string, unknown>> | undefined;\n\tconst results = Array.isArray(rawData) ? rawData : [];\n\treturn results.slice(0, numResults).map((r) => ({\n\t\ttitle: (r.title as string) || \"\",\n\t\turl: (r.url as string) || \"\",\n\t\tcontent: ((r.content as string) || (r.description as string) || \"\").slice(0, 2000),\n\t\tsnippet: ((r.content as string) || (r.description as string) || \"\").slice(0, 500),\n\t}));\n}\n"
  },
  {
    "path": "extensions/search-hub.ts",
    "content": "/**\n * Extension — Unified web search (12 backends) + content extraction (web_read)\n *\n * Backends (choose any, all disabled by default):\n *   duckduckgo    — ✅ Free, no key, via Python ddgs lib. Rate-limited.\n *   jina          — ✅ Free tier (API key optional for higher rate limits), full markdown via s.jina.ai\n *   marginalia    — ✅ Anti-SEO, \"public\" key optional. 354ms avg\n *   serper        — ✅ Google via serper.dev, 2500 free/mo. 667ms\n *   brave         — ✅ Brave Search, 2000 free/mo. 460ms\n *   tavily        — ✅ AI search, 1000 free/mo. 356ms BEST QUALITY\n *   exa           — ✅ AI-native, 10 QPS free tier. 137ms FASTEST\n *   firecrawl     — ✅ Search+crawl, 500 free credits. 644ms\n *   langsearch    — ✅ Free tier, no CC. 1816ms\n *   websearchapi  — ✅ Google-powered, 2000 free credits. 1323ms\n *   perplexity    — ✅ Unlimited free Sonar, citation-based answers\n *   searxng       — ✅ Self-hosted, 70+ aggregators. Needs instance URL\n *\n * Tools: web_search (auto-fallback + RRF combine mode), web_read (URL content)\n * Config: ~/.pi/agent/extensions/search.json + .pi/search.json (project wins)\n * Credentials: env var refs (ALL_CAPS), shell commands (!command), or literal keys\n *\n * Example .pi/search.json:\n *   {\n *     \"defaultBackend\": \"auto\",\n *     \"backends\": {\n *       \"duckduckgo\": { \"enabled\": true },\n *       \"marginalia\": { \"enabled\": true },\n *       \"serper\": { \"enabled\": true, \"apiKey\": \"...\" },\n *       \"tavily\": { \"enabled\": true, \"apiKey\": \"...\" },\n *       \"exa\": { \"enabled\": true, \"apiKey\": \"...\" },\n *       \"firecrawl\": { \"enabled\": true, \"apiKey\": \"...\" },\n *       \"langsearch\": { \"enabled\": true, \"apiKey\": \"...\" },\n *       \"websearchapi\": { \"enabled\": true, \"apiKey\": \"...\" },\n *       \"perplexity\": { \"enabled\": true, \"apiKey\": \"...\" },\n *       \"searxng\": { \"enabled\": true, \"instanceUrl\": \"http://localhost:8888\" }\n *     }\n *   }\n */\n\nimport { execSync, spawn } from \"node:child_process\";\nimport { existsSync, readFileSync, writeFileSync, mkdirSync } from \"node:fs\";\nimport { join } from \"node:path\";\nimport type { ExtensionAPI } from \"@earendil-works/pi-coding-agent\";\nimport { StringEnum } from \"@earendil-works/pi-ai\";\nimport { Type } from \"typebox\";\nimport {\n\tparseMarginalia, parseWebSearchAPI, parseSerper, parseTavily,\n\tparseExa, parseBrave, parseLangSearch, parseFirecrawl,\n\tparsePerplexity, parseSearXNG, parseJina,\n} from \"../backends/parsers.js\";\n\n// ---------------------------------------------------------------------------\n// Types & Config\n// ---------------------------------------------------------------------------\n\ninterface BackendConfig {\n\tenabled?: boolean;\n\tapiKey?: string;\n\t/** SearXNG-specific: base URL of the self-hosted instance (e.g. http://localhost:8888) */\n\tinstanceUrl?: string;\n\t/** Perplexity-specific: model variant (sonar, sonar-pro, sonar-deep-research, sonar-reasoning). Default: sonar */\n\tmodel?: string;\n}\n\ninterface SearchConfig {\n\tdefaultBackend?: string;\n\tbackends?: {\n\t\tduckduckgo?: BackendConfig;\n\t\tmarginalia?: BackendConfig;\n\n\t\tserper?: BackendConfig;\n\t\ttavily?: BackendConfig;\n\t\texa?: BackendConfig;\n\t\tbrave?: BackendConfig;\n\t\tlangsearch?: BackendConfig;\n\t\tfirecrawl?: BackendConfig;\n\t\twebsearchapi?: BackendConfig;\n\t\tperplexity?: BackendConfig;\n\t\tsearxng?: BackendConfig;\n\t};\n}\n\nfunction getAgentDir(): string {\n\treturn join(process.env.HOME || process.env.USERPROFILE || \"~\", \".pi\", \"agent\");\n}\n\nconst commandValueCache = new Map<string, { value?: string; errorMessage?: string }>();\nconst COMMAND_TIMEOUT_MS = 5_000;\n\n/**\n * Resolve a credential reference à la pi-web-providers:\n *   • \"!command\"   → execute shell command, return trimmed stdout (cached)\n *   • \"ALL_CAPS\"   → read process.env[ALL_CAPS]\n *   • otherwise     → return as literal string (actual key)\n */\nfunction resolveConfigValue(reference: string | undefined): string | undefined {\n\tif (!reference) return undefined;\n\n\t// !command — execute shell command, cache result\n\tif (reference.startsWith(\"!\")) {\n\t\tconst cached = commandValueCache.get(reference);\n\t\tif (cached) {\n\t\t\tif (cached.errorMessage) throw new Error(cached.errorMessage);\n\t\t\treturn cached.value;\n\t\t}\n\t\ttry {\n\t\t\tconst output = execSync(reference.slice(1), {\n\t\t\t\tencoding: \"utf-8\",\n\t\t\t\tstdio: [\"ignore\", \"pipe\", \"pipe\"],\n\t\t\t\ttimeout: COMMAND_TIMEOUT_MS,\n\t\t\t})\n\t\t\t\t.trim();\n\t\t\tconst value = output.length > 0 ? output : undefined;\n\t\t\tcommandValueCache.set(reference, { value });\n\t\t\treturn value;\n\t\t} catch (error) {\n\t\t\tconst errorMessage = (error as Error).message;\n\t\t\tcommandValueCache.set(reference, { errorMessage });\n\t\t\tthrow error;\n\t\t}\n\t}\n\n\t// ALL_CAPS → env var lookup\n\tconst envValue = process.env[reference];\n\tif (envValue !== undefined) return envValue;\n\tif (/^[A-Z][A-Z0-9_]*$/.test(reference)) {\n\t\t// Warn: value looks like an env var reference but the env var is unset.\n\t\t// If this was intended as a literal key, rename it or set the env var.\n\t\tconsole.warn(`[pi-search] Credential reference \"${reference}\" matches ALL_CAPS env-var pattern ` +\n\t\t\t`but process.env.${reference} is not set. If this is a literal key, ` +\n\t\t\t`use a different name to avoid confusion.`);\n\t\treturn undefined;\n\t}\n\n\t// Otherwise → literal string (actual key in config)\n\treturn reference;\n}\n\n/** Convenience env vars checked as fallback when config has no apiKey for a backend. */\nconst FALLBACK_ENV_MAP: Record<string, string> = {\n\tjina: \"SEARCH_JINA_API_KEY\",\n\tserper: \"SEARCH_SERPER_API_KEY\",\n\ttavily: \"SEARCH_TAVILY_API_KEY\",\n\texa: \"SEARCH_EXA_API_KEY\",\n\tbrave: \"SEARCH_BRAVE_API_KEY\",\n\tlangsearch: \"SEARCH_LANGSEARCH_API_KEY\",\n\tfirecrawl: \"SEARCH_FIRECRAWL_API_KEY\",\n\twebsearchapi: \"SEARCH_WEBSEARCHAPI_API_KEY\",\n\tperplexity: \"SEARCH_PERPLEXITY_API_KEY\",\n};\n\n/** Invalidate cached shell-command credentials so key rotation takes effect. */\nfunction clearCredentialCache(): void {\n\tcommandValueCache.clear();\n}\n\n/** Lazy resolution: config.apiKey → resolveConfigValue() → FALLBACK_ENV_MAP fallback. */\nfunction resolveBackendKey(backend: string): string | undefined {\n\tconst bc = config.backends?.[backend as keyof typeof config.backends];\n\tif (bc?.apiKey) {\n\t\tconst resolved = resolveConfigValue(bc.apiKey);\n\t\tif (resolved) return resolved;\n\t}\n\tconst fallbackEnv = FALLBACK_ENV_MAP[backend];\n\tif (fallbackEnv) {\n\t\tconst envValue = process.env[fallbackEnv];\n\t\tif (envValue && envValue.trim().length > 0) return envValue.trim();\n\t}\n\treturn undefined;\n}\n\n/** Describe where a backend's key comes from (for search-status display). */\nfunction getKeySource(backend: string): { configured: boolean; source: string } {\n\tconst bc = config.backends?.[backend as keyof typeof config.backends];\n\tif (!bc?.apiKey) {\n\t\tconst fallbackEnv = FALLBACK_ENV_MAP[backend];\n\t\tif (fallbackEnv && process.env[fallbackEnv]) {\n\t\t\treturn { configured: true, source: `env:${fallbackEnv}` };\n\t\t}\n\t\treturn { configured: false, source: \"\" };\n\t}\n\tconst ref = bc.apiKey;\n\tif (ref.startsWith(\"!\")) {\n\t\treturn { configured: true, source: `shell:${ref.slice(0, 40)}...` };\n\t}\n\tif (/^[A-Z][A-Z0-9_]*$/.test(ref)) {\n\t\tconst envValue = process.env[ref];\n\t\tif (envValue) return { configured: true, source: `env:${ref}` };\n\t\treturn { configured: false, source: `env:${ref} (unset)` };\n\t}\n\treturn { configured: true, source: \"literal\" };\n}\n\n\n\nfunction loadConfig(cwd: string): SearchConfig {\n\tconst globalPath = join(getAgentDir(), \"extensions\", \"search.json\");\n\tconst projectPath = join(cwd, \".pi\", \"search.json\");\n\n\tlet config: SearchConfig = { defaultBackend: \"duckduckgo\", backends: {} };\n\n\tif (existsSync(globalPath)) {\n\t\ttry {\n\t\t\tconfig = { ...config, ...JSON.parse(readFileSync(globalPath, \"utf-8\")) };\n\t\t} catch {\n\t\t\t// ignore\n\t\t}\n\t}\n\n\t// Save global backends before project config overwrites them\n\tconst preProjectBackends = { ...(config.backends ?? {}) };\n\n\tif (existsSync(projectPath)) {\n\t\ttry {\n\t\t\tconst project = JSON.parse(readFileSync(projectPath, \"utf-8\"));\n\t\t\tconfig = { ...config, ...project };\n\t\t\t// Guard: if project config set backends to null/undefined, restore global backends\n\t\t\tif (config.backends == null) {\n\t\t\t\tconfig.backends = preProjectBackends;\n\t\t\t}\n\t\t\tif (project.backends && typeof project.backends === \"object\") {\n\t\t\t\t// Deep merge: merge per-backend so global backends not re-listed in project config are preserved\n\t\t\t\tconst merged = { ...preProjectBackends, ...config.backends };\n\t\t\t\tfor (const [key, val] of Object.entries(project.backends)) {\n\t\t\t\t\tif (val && merged[key]) {\n\t\t\t\t\t\tmerged[key] = { ...merged[key], ...val };\n\t\t\t\t\t} else {\n\t\t\t\t\t\tmerged[key] = val;\n\t\t\t\t\t}\n\t\t\t\t}\n\t\t\t\tconfig.backends = merged;\n\t\t\t}\n\t\t} catch {\n\t\t\t// ignore\n\t\t}\n\t}\n\n\t// Auto-enable backends that have a convenience env var but no explicit config yet.\n\t// Only enables if the backend is not explicitly disabled (enabled !== false).\n\tfor (const [backend, envVar] of Object.entries(FALLBACK_ENV_MAP)) {\n\t\tconst envValue = process.env[envVar];\n\t\tif (envValue && envValue.trim().length > 0) {\n\t\t\tconst configBackends = config.backends ?? {};\n\t\t\tconst existing = configBackends[backend as keyof typeof configBackends];\n\t\t\tif (!existing || existing.enabled === undefined) {\n\t\t\t\tif (!config.backends) config.backends = {};\n\t\t\t\t(config.backends as Record<string, BackendConfig>)[backend] = {\n\t\t\t\t\t...existing,\n\t\t\t\t\tenabled: true,\n\t\t\t\t};\n\t\t\t}\n\t\t}\n\t}\n\n\treturn config;\n}\n\nconst MISSING_KEY_HELP =\n\t\"Set the API key via env var (SEARCH_<BACKEND>_API_KEY), \" +\n\t\"config reference (\\\"apiKey\\\": \\\"SOME_ENV_VAR\\\"), \" +\n\t\"shell command (\\\"apiKey\\\": \\\"!pass show api/backend\\\"), \" +\n\t\"or a literal key in ~/.pi/agent/extensions/search.json. \" +\n\t\"DuckDuckGo & Marginalia need no key.\";\n\nconst HTTP_TIMEOUT_MS = 30_000;\n\n/** Simple per-backend cooldown to avoid hammering rate-limited APIs. */\nconst COOLDOWN_MS = 2_000;\nconst backendCooldowns = new Map<string, number>();\n\nfunction waitForCooldown(backend: string): Promise<void> {\n\tconst until = backendCooldowns.get(backend);\n\tif (!until) return Promise.resolve();\n\tconst delay = until - Date.now();\n\tif (delay <= 0) return Promise.resolve();\n\treturn new Promise(r => setTimeout(r, delay));\n}\n\nfunction markCooldown(backend: string) {\n\tbackendCooldowns.set(backend, Date.now() + COOLDOWN_MS);\n}\n\n/** Combine an optional caller signal with a default timeout. */\nfunction timeoutSignal(signal?: AbortSignal): AbortSignal | undefined {\n\tif (!signal) return AbortSignal.timeout(HTTP_TIMEOUT_MS);\n\treturn AbortSignal.any([signal, AbortSignal.timeout(HTTP_TIMEOUT_MS)]);\n}\n\n/** Sanitize API error text — truncate and strip potential secrets. */\nfunction sanitizeError(status: number, text: string): string {\n\tconst safe = text\n\t\t// Redact \"Bearer <token>\" and \"Token <value>\" patterns\n\t\t.replace(/(bearer|token)\\s+[\\w.\\/-]{8,}/gi, \"$1 [redacted]\")\n\t\t// Redact key=value or \"key\": \"value\" pairs for known secret keys\n\t\t.replace(/(api[-_]?key|bearer|token|authorization|secret|password)[\"']?\\s*[:=]\\s*[\"']?[\\w.\\/-]{8,}/gi, \"[redacted]\")\n\t\t// Redact JSON key-value pairs where the value looks like a key\n\t\t.replace(/\"(?:api[-_]?key|apiKey|token|secret|password|bearer)\"\\s*:\\s*\"[^\"']{8,}\"/gi, '\"[redacted]\"')\n\t\t// Redact x-api-key / Authorization header values in raw text\n\t\t.replace(/(x-api-key|authorization)\\s*:\\s*[\\w.\\/-]{8,}/gi, \"$1: [redacted]\")\n\t\t.slice(0, 300);\n\treturn `API error (${status}): ${safe}`;\n}\n\n\n// ---------------------------------------------------------------------------\n// Backend: DuckDuckGo (free, no key needed)\n// ---------------------------------------------------------------------------\n\ninterface DuckDuckGoResult {\n\ttitle: string;\n\turl: string;\n\tsnippet: string;\n}\n\nasync function searchDuckDuckGo(\n\tquery: string,\n\tnumResults: number,\n\tsignal?: AbortSignal,\n): Promise<{ results: DuckDuckGoResult[] }> {\n\tif (signal?.aborted) throw new Error(\"DuckDuckGo search aborted\");\n\n\tconst pyScript = `\nimport json, sys\ntry:\n    from ddgs import DDGS\nexcept ImportError:\n    # ddgs may be installed as a uv tool — find it and add to sys.path\n    import subprocess, pathlib\n    try:\n        ddgs_bin = subprocess.check_output([\"which\", \"ddgs\"], text=True, stderr=subprocess.DEVNULL).strip()\n        if ddgs_bin:\n            # Walk up from the binary until we find site-packages — no hardcoded depth assumption\n            ddgs_path = pathlib.Path(ddgs_bin).resolve()\n            found = False\n            for parent in [ddgs_path, *ddgs_path.parents]:\n                for py_ver_dir in sorted((parent / \"lib\").iterdir(), reverse=True):\n                    sp = py_ver_dir / \"site-packages\"\n                    if sp.is_dir():\n                        sys.path.insert(0, str(sp))\n                        found = True\n                        break\n                if found:\n                    break\n            if not found:\n                sys.exit(1)\n    except Exception:\n        sys.exit(1)\n    from ddgs import DDGS\nresults = []\nwith DDGS() as ddgs:\n    for i, r in enumerate(ddgs.text(${JSON.stringify(query)}, max_results=${numResults})):\n        results.append({\"title\": r.get(\"title\",\"\"), \"url\": r.get(\"href\",\"\"), \"snippet\": r.get(\"body\",\"\")})\nprint(json.dumps({\"results\": results}))\n`;\n\n\treturn new Promise((resolve, reject) => {\n\t\tconst pythonCmd = process.platform === \"win32\" ? \"python\" : \"python3\";\n\t\tconst proc = spawn(pythonCmd, [\"-c\", pyScript], {\n\t\t\tstdio: [\"pipe\", \"pipe\", \"pipe\"],\n\t\t});\n\n\t\tlet stdout = \"\";\n\t\tlet stderr = \"\";\n\n\t\tproc.stdout.on(\"data\", (data: Buffer) => { stdout += data.toString(); });\n\t\tproc.stderr.on(\"data\", (data: Buffer) => { stderr += data.toString(); });\n\n\t\t// Timeout timer\n\t\tconst timeout = setTimeout(() => {\n\t\t\tproc.kill();\n\t\t\treject(new Error(\"DuckDuckGo search timed out\"));\n\t\t}, HTTP_TIMEOUT_MS);\n\n\t\t// Abort signal handler\n\t\tconst onAbort = () => {\n\t\t\tclearTimeout(timeout);\n\t\t\tproc.kill();\n\t\t\treject(new Error(\"DuckDuckGo search aborted\"));\n\t\t};\n\t\tif (signal) {\n\t\t\tif (signal.aborted) { clearTimeout(timeout); reject(new Error(\"DuckDuckGo search aborted\")); return; }\n\t\t\tsignal.addEventListener(\"abort\", onAbort, { once: true });\n\t\t}\n\n\t\tproc.on(\"close\", (code) => {\n\t\t\tclearTimeout(timeout);\n\t\t\tif (signal) signal.removeEventListener(\"abort\", onAbort);\n\t\t\tif (code === 0) {\n\t\t\t\ttry {\n\t\t\t\t\tresolve(JSON.parse(stdout.trim()));\n\t\t\t\t} catch {\n\t\t\t\t\treject(new Error(`DuckDuckGo search: invalid JSON output: ${stdout.slice(0, 200)}`));\n\t\t\t\t}\n\t\t\t} else {\n\t\t\t\tconst msg = stderr.trim().slice(0, 300);\n\t\t\t\treject(new Error(`DuckDuckGo search failed (exit ${code}): ${msg || \"unknown error\"}`));\n\t\t\t}\n\t\t});\n\n\t\tproc.on(\"error\", (err) => {\n\t\t\tclearTimeout(timeout);\n\t\t\tif (signal) signal.removeEventListener(\"abort\", onAbort);\n\t\t\treject(new Error(`DuckDuckGo search failed: ${err.message}`));\n\t\t});\n\t});\n}\n\n// ---------------------------------------------------------------------------\n// Backend: Marginalia Search (anti-SEO independent search, uses \"public\" key)\n// ---------------------------------------------------------------------------\n\nasync function searchMarginalia(\n\tquery: string,\n\tnumResults: number,\n\tapiKey: string | undefined,\n\tsignal?: AbortSignal,\n): Promise<{ results: Array<{ title: string; url: string; snippet: string }> }> {\n\tconst key = apiKey || \"public\";\n\tconst response = await fetch(\n\t\t`https://api.marginalia.nu/${encodeURIComponent(key)}/search/${encodeURIComponent(query)}?index=0&count=${Math.min(numResults, 50)}`,\n\t\t{\n\t\t\tsignal: timeoutSignal(signal),\n\t\t\theaders: { \"Accept\": \"application/json\" },\n\t\t},\n\t);\n\n\tif (!response.ok) {\n\t\tconst text = await response.text().catch(() => \"\");\n\t\tthrow new Error(`Marginalia ${sanitizeError(response.status, text)}`);\n\t}\n\n\tconst data = (await response.json()) as Record<string, unknown>;\n\n\treturn {\n\t\tresults: parseMarginalia(data, numResults),\n\t};\n}\n\n// ---------------------------------------------------------------------------\n// Backend: Serper.dev (Google search, needs API key)\n// ---------------------------------------------------------------------------\n\nasync function searchSerper(\n\tquery: string,\n\tnumResults: number,\n\tapiKey: string,\n\tsignal?: AbortSignal,\n): Promise<{ results: Array<{ title: string; url: string; snippet: string }> }> {\n\tconst body = { q: query, num: Math.min(numResults, 100) };\n\tconst response = await fetch(\"https://google.serper.dev/search\", {\n\t\tmethod: \"POST\",\n\t\theaders: {\n\t\t\t\"X-API-KEY\": apiKey,\n\t\t\t\"Content-Type\": \"application/json\",\n\t\t},\n\t\tbody: JSON.stringify(body),\n\t\tsignal: timeoutSignal(signal),\n\t});\n\tif (!response.ok) {\n\t\tconst text = await response.text().catch(() => \"\");\n\t\tthrow new Error(`Serper ${sanitizeError(response.status, text)}`);\n\t}\n\tconst data = (await response.json()) as Record<string, unknown>;\n\treturn {\n\t\tresults: parseSerper(data, numResults),\n\t};\n}\n\n// ---------------------------------------------------------------------------\n// Backend: Tavily (AI-agent search, needs API key)\n// ---------------------------------------------------------------------------\n\nasync function searchTavily(\n\tquery: string,\n\tnumResults: number,\n\tapiKey: string,\n\tsignal?: AbortSignal,\n): Promise<{ results: Array<{ title: string; url: string; snippet: string; content?: string }> }> {\n\tconst body = {\n\t\tquery,\n\t\tmax_results: Math.min(numResults, 20),\n\t\tinclude_answer: false,\n\t};\n\tconst response = await fetch(\"https://api.tavily.com/search\", {\n\t\tmethod: \"POST\",\n\t\theaders: {\n\t\t\t\"Content-Type\": \"application/json\",\n\t\t\tAuthorization: `Bearer ${apiKey}`,\n\t\t},\n\t\tbody: JSON.stringify(body),\n\t\tsignal: timeoutSignal(signal),\n\t});\n\tif (!response.ok) {\n\t\tconst text = await response.text().catch(() => \"\");\n\t\tthrow new Error(`Tavily ${sanitizeError(response.status, text)}`);\n\t}\n\tconst data = (await response.json()) as Record<string, unknown>;\n\treturn {\n\t\tresults: parseTavily(data, numResults),\n\t};\n}\n\n// ---------------------------------------------------------------------------\n// Backend: Exa (optional, needs API key)\n// ---------------------------------------------------------------------------\n\nasync function searchExa(\n\tquery: string,\n\tnumResults: number,\n\tapiKey: string,\n\tsignal?: AbortSignal,\n): Promise<{ results: Array<{ title: string; url: string; snippet?: string }> }> {\n\tconst body = {\n\t\tquery,\n\t\tnumResults: Math.min(numResults, 25),\n\t\tcontents: { text: true, highlights: true },\n\t};\n\tconst response = await fetch(\"https://api.exa.ai/search\", {\n\t\tmethod: \"POST\",\n\t\theaders: {\n\t\t\t\"Content-Type\": \"application/json\",\n\t\t\t\"x-api-key\": apiKey,\n\t\t},\n\t\tbody: JSON.stringify(body),\n\t\tsignal: timeoutSignal(signal),\n\t});\n\tif (!response.ok) {\n\t\tconst text = await response.text().catch(() => \"\");\n\t\tlet detail = text;\n\t\ttry {\n\t\t\tconst json = JSON.parse(text);\n\t\t\tdetail = json.error || json.message || text;\n\t\t} catch {\n\t\t\t// use raw\n\t\t}\n\t\tthrow new Error(`Exa ${sanitizeError(response.status, detail)}`);\n\t}\n\tconst data = (await response.json()) as Record<string, unknown>;\n\treturn {\n\t\tresults: parseExa(data, numResults),\n\t};\n}\n\n// ---------------------------------------------------------------------------\n// Backend: Brave Search (metered billing ~$5/mo credit, needs API key)\n// ---------------------------------------------------------------------------\n\nasync function searchBrave(\n\tquery: string,\n\tnumResults: number,\n\tapiKey: string,\n\tsignal?: AbortSignal,\n): Promise<{ results: Array<{ title: string; url: string; snippet?: string }> }> {\n\tconst params = new URLSearchParams({ q: query, count: String(Math.min(numResults, 20)) });\n\tconst response = await fetch(`https://api.search.brave.com/res/v1/web/search?${params}`, {\n\t\tmethod: \"GET\",\n\t\theaders: {\n\t\t\t\"Accept\": \"application/json\",\n\t\t\t\"Accept-Encoding\": \"gzip\",\n\t\t\t\"X-Subscription-Token\": apiKey,\n\t\t},\n\t\tsignal: timeoutSignal(signal),\n\t});\n\tif (!response.ok) {\n\t\tconst text = await response.text().catch(() => \"\");\n\t\tthrow new Error(`Brave ${sanitizeError(response.status, text)}`);\n\t}\n\tconst data = (await response.json()) as Record<string, unknown>;\n\treturn {\n\t\tresults: parseBrave(data, numResults),\n\t};\n}\n\n// ---------------------------------------------------------------------------\n// Backend: LangSearch (genuinely free tier, no credit card, needs API key)\n// Endpoint: POST /v1/web-search, auth: Authorization: Bearer\n// ---------------------------------------------------------------------------\n\nasync function searchLangSearch(\n\tquery: string,\n\tnumResults: number,\n\tapiKey: string,\n\tsignal?: AbortSignal,\n): Promise<{ results: Array<{ title: string; url: string; snippet?: string }> }> {\n\tconst body = { query, max_results: Math.min(numResults, 20) };\n\tconst response = await fetch(\"https://api.langsearch.com/v1/web-search\", {\n\t\tmethod: \"POST\",\n\t\theaders: {\n\t\t\t\"Content-Type\": \"application/json\",\n\t\t\t\"Authorization\": `Bearer ${apiKey}`,\n\t\t},\n\t\tbody: JSON.stringify(body),\n\t\tsignal: timeoutSignal(signal),\n\t});\n\tif (!response.ok) {\n\t\tconst text = await response.text().catch(() => \"\");\n\t\tthrow new Error(`LangSearch ${sanitizeError(response.status, text)}`);\n\t}\n\tconst data = (await response.json()) as Record<string, unknown>;\n\treturn {\n\t\tresults: parseLangSearch(data, numResults),\n\t};\n}\n\n// ---------------------------------------------------------------------------\n// Backend: Firecrawl (500 free credits, search+crawl+extract, needs API key)\n// ---------------------------------------------------------------------------\n\nasync function searchFirecrawl(\n\tquery: string,\n\tnumResults: number,\n\tapiKey: string,\n\tsignal?: AbortSignal,\n): Promise<{ results: Array<{ title: string; url: string; snippet?: string }> }> {\n\tconst body = { query, limit: Math.min(numResults, 20) };\n\tconst response = await fetch(\"https://api.firecrawl.dev/v2/search\", {\n\t\tmethod: \"POST\",\n\t\theaders: {\n\t\t\t\"Content-Type\": \"application/json\",\n\t\t\t\"Authorization\": `Bearer ${apiKey}`,\n\t\t},\n\t\tbody: JSON.stringify(body),\n\t\tsignal: timeoutSignal(signal),\n\t});\n\tif (!response.ok) {\n\t\tconst text = await response.text().catch(() => \"\");\n\t\tthrow new Error(`Firecrawl ${sanitizeError(response.status, text)}`);\n\t}\n\tconst data = (await response.json()) as Record<string, unknown>;\n\treturn {\n\t\tresults: parseFirecrawl(data, numResults),\n\t};\n}\n\n// ---------------------------------------------------------------------------\n// Backend: WebSearchAPI.ai (2000 free credits, needs API key)\n// Endpoint: POST /ai-search, auth: Authorization: Bearer\n// Params: maxResults, includeContent, country, language\n// ---------------------------------------------------------------------------\n\nasync function searchWebSearchAPI(\n\tquery: string,\n\tnumResults: number,\n\tapiKey: string,\n\tsignal?: AbortSignal,\n): Promise<{ results: Array<{ title: string; url: string; snippet?: string }> }> {\n\tconst body = {\n\t\tquery,\n\t\tmaxResults: Math.min(numResults, 20),\n\t\tincludeContent: false,\n\t\tcountry: \"us\",\n\t\tlanguage: \"en\",\n\t};\n\tconst response = await fetch(\"https://api.websearchapi.ai/ai-search\", {\n\t\tmethod: \"POST\",\n\t\theaders: {\n\t\t\t\"Content-Type\": \"application/json\",\n\t\t\t\"Authorization\": `Bearer ${apiKey}`,\n\t\t},\n\t\tbody: JSON.stringify(body),\n\t\tsignal: timeoutSignal(signal),\n\t});\n\tif (!response.ok) {\n\t\tconst text = await response.text().catch(() => \"\");\n\t\tthrow new Error(`WebSearchAPI ${sanitizeError(response.status, text)}`);\n\t}\n\tconst data = (await response.json()) as Record<string, unknown>;\n\treturn {\n\t\tresults: parseWebSearchAPI(data, numResults),\n\t};\n}\n// ---------------------------------------------------------------------------\n// Backend: Perplexity Sonar (free tier, unlimited queries, needs API key)\n// Endpoint: POST /chat/completions, auth: Authorization: Bearer\n// Uses sonar model (configurable), extracts citations from response as search results\n// ---------------------------------------------------------------------------\n\nasync function searchPerplexity(\n\tquery: string,\n\tnumResults: number,\n\tapiKey: string,\n\tsignal?: AbortSignal,\n\tmodel?: string,\n): Promise<{ results: Array<{ title: string; url: string; snippet?: string }> }> {\n\tconst body = {\n\t\tmodel: model || \"sonar\",\n\t\tmessages: [\n\t\t\t{\n\t\t\t\trole: \"user\",\n\t\t\t\tcontent: query,\n\t\t\t},\n\t\t],\n\t\tsearch_context_size: \"high\",\n\t};\n\n\tconst response = await fetch(\"https://api.perplexity.ai/chat/completions\", {\n\t\tmethod: \"POST\",\n\t\theaders: {\n\t\t\t\"Content-Type\": \"application/json\",\n\t\t\t\"Authorization\": `Bearer ${apiKey}`,\n\t\t},\n\t\tbody: JSON.stringify(body),\n\t\tsignal: timeoutSignal(signal),\n\t});\n\n\tif (!response.ok) {\n\t\tconst text = await response.text().catch(() => \"\");\n\t\tthrow new Error(`Perplexity ${sanitizeError(response.status, text)}`);\n\t}\n\n\tconst data = (await response.json()) as Record<string, unknown>;\n\n\treturn { results: parsePerplexity(data, query, numResults) };\n}\n\n// ---------------------------------------------------------------------------\n// Backend: SearXNG (self-hosted metasearch, aggregates 70+ providers)\n// Endpoint: GET /search?q=<query>&format=json, optional auth via API key header\n// Needs instance URL configured in search.json\n// ---------------------------------------------------------------------------\n\nasync function searchSearXNG(\n\tquery: string,\n\tnumResults: number,\n\tapiKey: string | undefined,\n\tinstanceUrl: string | undefined,\n\tsignal?: AbortSignal,\n): Promise<{ results: Array<{ title: string; url: string; snippet?: string }> }> {\n\tif (!instanceUrl) {\n\t\tthrow new Error(\"SearXNG instance URL not configured. Set searxng.instanceUrl in search.json (e.g. http://localhost:8888)\");\n\t}\n\n\tconst baseUrl = instanceUrl.replace(/\\/+$/, \"\");\n\tconst params = new URLSearchParams({\n\t\tq: query,\n\t\tformat: \"json\",\n\t\tcount: String(Math.min(numResults, 50)),\n\t});\n\n\tconst headers: Record<string, string> = {\n\t\t\"Accept\": \"application/json\",\n\t};\n\tif (apiKey) {\n\t\theaders[\"Authorization\"] = `Bearer ${apiKey}`;\n\t}\n\n\tconst response = await fetch(`${baseUrl}/search?${params}`, {\n\t\tmethod: \"GET\",\n\t\theaders,\n\t\tsignal: timeoutSignal(signal),\n\t});\n\n\tif (!response.ok) {\n\t\tconst text = await response.text().catch(() => \"\");\n\t\tthrow new Error(`SearXNG ${sanitizeError(response.status, text)}`);\n\t}\n\n\tconst data = (await response.json()) as Record<string, unknown>;\n\treturn {\n\t\tresults: parseSearXNG(data, numResults),\n\t};\n}\n\n// ---------------------------------------------------------------------------\n// Backend: Jina AI (s.jina.ai) — search results via s.jina.ai; requires API key from jina.ai (free). web_read uses r.jina.ai (Reader, no key needed).\n// Endpoint: GET https://s.jina.ai/?q=<query>, returns 5 results as markdown or JSON\n// ---------------------------------------------------------------------------\n\ninterface JinaResult {\n\ttitle: string;\n\turl: string;\n\tcontent: string;\n}\n\nasync function searchJina(\n\tquery: string,\n\tnumResults: number,\n\tapiKey?: string,\n\tsignal?: AbortSignal,\n): Promise<{ results: JinaResult[] }> {\n\tconst url = `https://s.jina.ai/?q=${encodeURIComponent(query)}&format=json`;\n\tconst headers: Record<string, string> = {\n\t\t\"Accept\": \"application/json\",\n\t};\n\tif (apiKey) {\n\t\theaders[\"Authorization\"] = `Bearer ${apiKey}`;\n\t}\n\tconst response = await fetch(url, {\n\t\tsignal: timeoutSignal(signal),\n\t\theaders,\n\t});\n\n\tif (!response.ok) {\n\t\tconst text = await response.text().catch(() => \"\");\n\t\tthrow new Error(`Jina AI ${sanitizeError(response.status, text)}`);\n\t}\n\n\tconst data = (await response.json()) as Record<string, unknown>;\n\treturn {\n\t\tresults: parseJina(data, numResults),\n\t};\n}\n\n// ---------------------------------------------------------------------------\n// Backend Registry\n// ---------------------------------------------------------------------------\n\ninterface BackendRunner {\n\tneedsKey: boolean;\n\tneedsKeyFromConfig: boolean;\n\toptionalKey: boolean;\n\tneedsInstanceUrl: boolean;\n\tlabel: string;\n\tsetupLabel: string | null;\n\tsearch: (query: string, numResults: number, deps: { key?: string; instanceUrl?: string; signal?: AbortSignal }) => Promise<{ results: Array<{ title: string; url: string; snippet?: string; content?: string }> }>;\n}\n\nconst BACKEND_DEFS: Record<string, BackendRunner> = {\n\tduckduckgo: {\n\t\tneedsKey: false,\n\t\tneedsKeyFromConfig: false,\n\t\toptionalKey: false,\n\t\tneedsInstanceUrl: false,\n\t\tlabel: \"DuckDuckGo\",\n\t\tsetupLabel: null,\n\t\tsearch: async (query, numResults, { signal }) => {\n\t\t\tconst ddg = await searchDuckDuckGo(query, numResults, signal);\n\t\t\treturn { results: ddg.results };\n\t\t},\n\t},\n\tjina: {\n\t\tneedsKey: false,\n\t\tneedsKeyFromConfig: false,\n\t\toptionalKey: true,\n\t\tneedsInstanceUrl: false,\n\t\tlabel: \"Jina AI\",\n\t\tsetupLabel: \"Jina AI (free tier, API key optional for higher rate limits)\",\n\t\tsearch: async (query, numResults, { key, signal }) => {\n\t\t\treturn await searchJina(query, numResults, key, signal);\n\t\t},\n\t},\n\tmarginalia: {\n\t\tneedsKey: false,\n\t\tneedsKeyFromConfig: true,\n\t\toptionalKey: false,\n\t\tneedsInstanceUrl: false,\n\t\tlabel: \"Marginalia\",\n\t\tsetupLabel: null,\n\t\tsearch: async (query, numResults, { key, signal }) => {\n\t\t\tconst marg = await searchMarginalia(query, numResults, key, signal);\n\t\t\treturn { results: marg.results };\n\t\t},\n\t},\n\tserper: {\n\t\tneedsKey: true,\n\t\tneedsKeyFromConfig: false,\n\t\toptionalKey: false,\n\t\tneedsInstanceUrl: false,\n\t\tlabel: \"Serper\",\n\t\tsetupLabel: \"Serper (Google — 2500 free queries, one-time)\",\n\t\tsearch: async (query, numResults, { key, signal }) => {\n\t\t\tconst serp = await searchSerper(query, numResults, key!, signal);\n\t\t\treturn { results: serp.results };\n\t\t},\n\t},\n\ttavily: {\n\t\tneedsKey: true,\n\t\tneedsKeyFromConfig: false,\n\t\toptionalKey: false,\n\t\tneedsInstanceUrl: false,\n\t\tlabel: \"Tavily\",\n\t\tsetupLabel: \"Tavily (AI agent search — 1000 free calls/month)\",\n\t\tsearch: async (query, numResults, { key, signal }) => {\n\t\t\tconst tav = await searchTavily(query, numResults, key!, signal);\n\t\t\treturn { results: tav.results };\n\t\t},\n\t},\n\texa: {\n\t\tneedsKey: true,\n\t\tneedsKeyFromConfig: false,\n\t\toptionalKey: false,\n\t\tneedsInstanceUrl: false,\n\t\tlabel: \"Exa\",\n\t\tsetupLabel: \"Exa (AI search — 1000 free queries/month)\",\n\t\tsearch: async (query, numResults, { key, signal }) => {\n\t\t\tconst exa = await searchExa(query, numResults, key!, signal);\n\t\t\treturn { results: exa.results };\n\t\t},\n\t},\n\tbrave: {\n\t\tneedsKey: true,\n\t\tneedsKeyFromConfig: false,\n\t\toptionalKey: false,\n\t\tneedsInstanceUrl: false,\n\t\tlabel: \"Brave\",\n\t\tsetupLabel: \"Brave Search (metered billing ~$5/mo credit)\",\n\t\tsearch: async (query, numResults, { key, signal }) => {\n\t\t\tconst br = await searchBrave(query, numResults, key!, signal);\n\t\t\treturn { results: br.results };\n\t\t},\n\t},\n\tlangsearch: {\n\t\tneedsKey: true,\n\t\tneedsKeyFromConfig: false,\n\t\toptionalKey: false,\n\t\tneedsInstanceUrl: false,\n\t\tlabel: \"LangSearch\",\n\t\tsetupLabel: \"LangSearch (genuinely free, no CC)\",\n\t\tsearch: async (query, numResults, { key, signal }) => {\n\t\t\tconst ls = await searchLangSearch(query, numResults, key!, signal);\n\t\t\treturn { results: ls.results };\n\t\t},\n\t},\n\tfirecrawl: {\n\t\tneedsKey: true,\n\t\tneedsKeyFromConfig: false,\n\t\toptionalKey: false,\n\t\tneedsInstanceUrl: false,\n\t\tlabel: \"Firecrawl\",\n\t\tsetupLabel: \"Firecrawl (500 free credits)\",\n\t\tsearch: async (query, numResults, { key, signal }) => {\n\t\t\tconst fc = await searchFirecrawl(query, numResults, key!, signal);\n\t\t\treturn { results: fc.results };\n\t\t},\n\t},\n\twebsearchapi: {\n\t\tneedsKey: true,\n\t\tneedsKeyFromConfig: false,\n\t\toptionalKey: false,\n\t\tneedsInstanceUrl: false,\n\t\tlabel: \"WebSearchAPI\",\n\t\tsetupLabel: \"WebSearchAPI.ai (2000 free credits)\",\n\t\tsearch: async (query, numResults, { key, signal }) => {\n\t\t\tconst ws = await searchWebSearchAPI(query, numResults, key!, signal);\n\t\t\treturn { results: ws.results };\n\t\t},\n\t},\n\tperplexity: {\n\t\tneedsKey: true,\n\t\tneedsKeyFromConfig: false,\n\t\toptionalKey: false,\n\t\tneedsInstanceUrl: false,\n\t\tlabel: \"Perplexity Sonar\",\n\t\tsetupLabel: \"Perplexity Sonar (paid, usage-based)\",\n\t\tsearch: async (query, numResults, { key, signal }) => {\n\t\t\tconst bc = (config.backends as Record<string, BackendConfig> | undefined)?.perplexity;\n\t\t\tconst model = (bc as Record<string, unknown>)?.model as string | undefined;\n\t\t\tconst pp = await searchPerplexity(query, numResults, key!, signal, model);\n\t\t\treturn { results: pp.results };\n\t\t},\n\t},\n\tsearxng: {\n\t\tneedsKey: false,\n\t\tneedsKeyFromConfig: false,\n\t\toptionalKey: false,\n\t\tneedsInstanceUrl: true,\n\t\tlabel: \"SearXNG\",\n\t\tsetupLabel: \"SearXNG (self-hosted, needs instance URL)\",\n\t\tsearch: async (query, numResults, { key, instanceUrl, signal }) => {\n\t\t\tconst sx = await searchSearXNG(query, numResults, key, instanceUrl, signal);\n\t\t\treturn { results: sx.results };\n\t\t},\n\t},\n};\n\n// ---------------------------------------------------------------------------\n// Reciprocal Rank Fusion\n// ---------------------------------------------------------------------------\n\n/**\n * RRF (Reciprocal Rank Fusion) — rank-based merge across backends.\n * Constant k=60 is standard from the original RRF paper.\n */\nconst RRF_K = 60;\n\nfunction reciprocalRankFusion(\n\tbackendResults: Array<{ backend: string; results: SearchResultWithBackend[] }>,\n\tnumResults: number,\n): SearchResultWithBackend[] {\n\t// Score each unique result by its rank positions across backends\n\tconst urlScores = new Map<string, { score: number; result: SearchResultWithBackend; seenBackends: Set<string> }>();\n\n\tfor (const { backend, results } of backendResults) {\n\t\tfor (let i = 0; i < results.length; i++) {\n\t\t\tconst r = results[i];\n\t\t\tconst normalizedUrl = r.url.replace(/\\/$/, \"\").toLowerCase(); // normalize trailing slash\n\n\t\t\tlet entry = urlScores.get(normalizedUrl);\n\t\t\tif (!entry) {\n\t\t\t\tentry = { score: 0, result: r, seenBackends: new Set() };\n\t\t\t\turlScores.set(normalizedUrl, entry);\n\t\t\t}\n\n\t\t\t// RRF: score += 1 / (k + rank)\n\t\t\tentry.score += 1 / (RRF_K + i);\n\t\t\tentry.seenBackends.add(backend);\n\n\t\t\t// Keep the result with the most complete data (prefer content over snippet)\n\t\t\tif (r.content && !entry.result.content) {\n\t\t\t\tentry.result = r;\n\t\t\t}\n\t\t}\n\t}\n\n\t// Sort by RRF score descending, then by number of backends that found it\n\tconst sorted = Array.from(urlScores.values())\n\t\t.sort((a, b) => {\n\t\t\tconst scoreDiff = b.score - a.score;\n\t\t\tif (scoreDiff !== 0) return scoreDiff;\n\t\t\treturn b.seenBackends.size - a.seenBackends.size;\n\t\t})\n\t\t.slice(0, numResults)\n\t\t.map(e => e.result);\n\n\treturn sorted;\n}\n\n// ---------------------------------------------------------------------------\n// Result formatting\n// ---------------------------------------------------------------------------\n\ninterface SearchResultWithBackend {\n\ttitle: string;\n\turl: string;\n\tsnippet?: string;\n\tcontent?: string;\n\tbackend?: string;\n}\n\nfunction formatResults(\n\tquery: string,\n\tbackend: string,\n\tresults: Array<{ title: string; url: string; snippet?: string; content?: string }>,\n): string {\n\t// Escape newlines and markdown heading chars in query to prevent injection\n\tconst safeQuery = query.replace(/[\\n\\r]/g, \" \").replace(/^#/gm, \"\\\\#\");\n\tconst lines: string[] = [\n\t\t`## Search Results: \"${safeQuery}\"`,\n\t\t`Backend: ${backend}  ·  Results: ${results.length}`,\n\t\t\"\",\n\t];\n\tfor (let i = 0; i < results.length; i++) {\n\t\tconst r = results[i];\n\t\tlines.push(`### ${i + 1}. ${r.title || \"Untitled\"}`);\n\t\tlines.push(`   URL: ${r.url}`);\n\t\tconst displayText = r.snippet || r.content || \"\";\n\t\tif (displayText) {\n\t\t\tconst text = displayText.slice(0, 500);\n\t\t\tlines.push(`   ${text}${displayText.length > 500 ? \"...\" : \"\"}`);\n\t\t}\n\t\tlines.push(\"\");\n\t}\n\treturn lines.join(\"\\n\");\n}\n\nfunction formatCombinedResults(\n\tquery: string,\n\tresults: SearchResultWithBackend[],\n\tbackendStats: Map<string, { success: boolean; count: number; error?: string }>,\n): string {\n\tconst safeQuery = query.replace(/[\\n\\r]/g, \" \").replace(/^#/gm, \"\\\\#\");\n\tconst lines: string[] = [\n\t\t`## Search Results: \"${safeQuery}\"`,\n\t\t`Mode: combined  ·  Results: ${results.length}`,\n\t\t\"\",\n\t];\n\n\t// Add backend stats (derived from registry)\n\tconst backendLabel = Object.fromEntries(\n\t\tObject.entries(BACKEND_DEFS).map(([k, v]) => [k, v.label])\n\t) as Record<string, string>;\n\n\tlines.push(\"**Backends queried:**\");\n\tfor (const [backend, stats] of backendStats.entries()) {\n\t\tconst label = backendLabel[backend] || backend;\n\t\tif (stats.success) {\n\t\t\tlines.push(`  - ${label}: ${stats.count} results`);\n\t\t} else {\n\t\t\tlines.push(`  - ${label}: failed (${stats.error || \"unknown error\"})`);\n\t\t}\n\t}\n\tlines.push(\"\");\n\n\t// Add results\n\tfor (let i = 0; i < results.length; i++) {\n\t\tconst r = results[i];\n\t\tlines.push(`### ${i + 1}. ${r.title || \"Untitled\"}`);\n\t\tif (r.backend) {\n\t\t\tlines.push(`   *Source: ${backendLabel[r.backend] || r.backend}*`);\n\t\t}\n\t\tlines.push(`   URL: ${r.url}`);\n\t\tconst displayText = r.snippet || r.content || \"\";\n\t\tif (displayText) {\n\t\t\tconst text = displayText.slice(0, 500);\n\t\t\tlines.push(`   ${text}${displayText.length > 500 ? \"...\" : \"\"}`);\n\t\t}\n\t\tlines.push(\"\");\n\t}\n\treturn lines.join(\"\\n\");\n}\n\n// ---------------------------------------------------------------------------\n// Extension\n// ---------------------------------------------------------------------------\n\n/** Module-level config accessible from helper functions like resolveBackendKey(). */\nlet config: SearchConfig = { defaultBackend: \"duckduckgo\", backends: {} };\n\nexport default function (pi: ExtensionAPI) {\n\tlet activeBackends: string[] = [];\n\tlet configCacheTime = 0;\n\tconst CONFIG_TTL_MS = 10_000; // re-read config at most every 10s\n\n\tfunction refreshConfig(cwd: string, force = false) {\n\t\tconst now = Date.now();\n\t\tif (!force && now - configCacheTime < CONFIG_TTL_MS) return;\n\n\t\tconfig = loadConfig(cwd);\n\t\tconfigCacheTime = now;\n\n\t\tactiveBackends = Object.entries(config.backends || {})\n\t\t\t.filter(([_, bc]) => bc?.enabled)\n\t\t\t.map(([name]) => name);\n\n\t\t// Always add duckduckgo if no backends explicitly enabled, since it needs no key\n\t\tif (activeBackends.length === 0) {\n\t\t\tactiveBackends.push(\"duckduckgo\");\n\t\t}\n\n\t\t// Honor defaultBackend: put it first in the auto-try order\n\t\tif (config.defaultBackend && activeBackends.includes(config.defaultBackend)) {\n\t\t\tactiveBackends = [\n\t\t\t\tconfig.defaultBackend,\n\t\t\t\t...activeBackends.filter(b => b !== config.defaultBackend),\n\t\t\t];\n\t\t} else {\n\t\t\tconfig.defaultBackend = activeBackends[0];\n\t\t}\n\n\t\t// Invalidate credential cache so shell-command keys refresh after config reload\n\t\tclearCredentialCache();\n\t}\n\n\t// -----------------------------------------------------------------------\n\t// Backend dispatcher\n\t// -----------------------------------------------------------------------\n\n\tasync function runBackend(\n\t\tbackend: string,\n\t\tquery: string,\n\t\tnumResults: number,\n\t\tsignal?: AbortSignal,\n\t): Promise<Array<{ title: string; url: string; snippet?: string; content?: string }>> {\n\t\tawait waitForCooldown(backend);\n\t\ttry {\n\t\t\tconst def = BACKEND_DEFS[backend];\n\t\t\tif (!def) throw new Error(`Unknown backend: ${backend}`);\n\n\t\t\tlet key: string | undefined;\n\t\t\tif (def.needsKeyFromConfig) {\n\t\t\t\tconst bc = (config.backends as Record<string, BackendConfig> | undefined)?.[backend];\n\t\t\t\tkey = bc?.apiKey;\n\t\t\t} else if (def.needsKey) {\n\t\t\t\tkey = resolveBackendKey(backend);\n\t\t\t\tif (!key) {\n\t\t\t\t\tconst label = def.label;\n\t\t\t\t\tthrow new Error(`${label} backend not configured. ${MISSING_KEY_HELP}`);\n\t\t\t\t}\n\t\t\t} else if (def.optionalKey) {\n\t\t\t\t// Optionally resolve key — don't throw if missing\n\t\t\t\tkey = resolveBackendKey(backend);\n\t\t\t}\n\n\t\t\tlet instanceUrl: string | undefined;\n\t\t\tif (def.needsInstanceUrl) {\n\t\t\t\tconst bc = (config.backends as Record<string, BackendConfig> | undefined)?.[backend];\n\t\t\t\tinstanceUrl = bc?.instanceUrl;\n\t\t\t\tif (!instanceUrl) {\n\t\t\t\t\tthrow new Error(`SearXNG instance URL not configured. Set searxng.instanceUrl in search.json`);\n\t\t\t\t}\n\t\t\t}\n\n\t\t\tconst result = await def.search(query, numResults, { key, instanceUrl, signal });\n\t\t\treturn result.results;\n\t\t} finally {\n\t\t\tmarkCooldown(backend);\n\t\t}\n\t}\n\n\t// -----------------------------------------------------------------------\n\t// Tool: web_search\n\t// -----------------------------------------------------------------------\n\n\tpi.registerTool({\n\t\tname: \"web_search\",\n\t\tlabel: \"Web Search\",\n\t\tdescription:\n\t\t\t\"Search the web using one of several backend search engines. \" +\n\t\t\t\"Supports DuckDuckGo (free, no key), \" +\n\t\t\t\"Marginalia Search (free, shared public key), Serper, Tavily, Exa, Brave, \" +\n\t\t\t\"LangSearch, Firecrawl, WebSearchAPI, Perplexity Sonar, and SearXNG (most need API keys). \" +\n\t\t\t\"The best available backend is used automatically. \" +\n\t\t\t\"Use combine=true to query all enabled backends in parallel for broader coverage. \" +\n\t\t\t\"Use for fact-finding, research, documentation lookups, and current events.\",\n\t\tpromptSnippet: \"Search the web (supports multiple search backends)\",\n\t\tpromptGuidelines: [\n\t\t\t\"Use web_search when you need up-to-date information, facts, or documentation from the web\",\n\t\t\t\"Auto mode tries enabled backends in order (DuckDuckGo is the free fallback)\",\n\t\t\t\"Set combine=true to query ALL backends in parallel and merge/deduplicate results\",\n\t\t\t\"Configure additional backends in .pi/search.json for better quality results\",\n\t\t],\n\t\tparameters: Type.Object({\n\t\t\tquery: Type.String({\n\t\t\t\tdescription: \"Search query (natural language works best)\",\n\t\t\t}),\n\t\t\tnumResults: Type.Optional(\n\t\t\t\tType.Number({\n\t\t\t\t\tdescription: \"Number of results (1-20, default 10)\",\n\t\t\t\t\tdefault: 10,\n\t\t\t\t}),\n\t\t\t),\n\t\t\tbackend: Type.Optional(\n\t\t\t\tStringEnum([\"duckduckgo\", \"jina\", \"marginalia\", \"serper\", \"tavily\", \"exa\",\n\t\t\t\t\t\"brave\", \"langsearch\", \"firecrawl\", \"websearchapi\", \"perplexity\", \"searxng\", \"auto\"] as const, {\n\t\t\t\t\tdescription:\n\t\t\t\t\t\t\"Backend to use. 'auto' picks the best configured backend (default)\",\n\t\t\t\t}),\n\t\t\t),\n\t\t\tcombine: Type.Optional(\n\t\t\t\tType.Boolean({\n\t\t\t\t\tdescription:\n\t\t\t\t\t\t\"When true, queries ALL enabled backends in parallel and merges/deduplicates results. \" +\n\t\t\t\t\t\t\"Default is false (fallback mode: uses first successful backend only). \" +\n\t\t\t\t\t\t\"Ignored when a specific backend is requested (backend != 'auto').\",\n\t\t\t\t\tdefault: false,\n\t\t\t\t}),\n\t\t\t),\n\t\t}),\n\t\tasync execute(_toolCallId, params, signal, _onUpdate, ctx) {\n\t\t\trefreshConfig(ctx.cwd);\n\t\t\tconst numResults = Math.max(1, Math.min(params.numResults ?? 10, 20));\n\t\t\tconst requestedBackend = params.backend || \"auto\";\n\t\t\tconst combine = params.combine ?? false;\n\n\t\t\tif (requestedBackend !== \"auto\") {\n\t\t\t\t// Specific backend requested — try it directly\n\t\t\t\tconst results = await runBackend(requestedBackend, params.query, numResults, signal);\n\t\t\t\treturn {\n\t\t\t\t\tcontent: [{ type: \"text\", text: formatResults(params.query, requestedBackend, results) }],\n\t\t\t\t\tdetails: { backend: requestedBackend, resultCount: results.length },\n\t\t\t\t};\n\t\t\t}\n\n\t\t\t// Auto mode\n\t\t\tif (combine) {\n\t\t\t\t// Combine mode: query all enabled backends in parallel\n\t\t\t\tconst resultsPerBackend = await Promise.all(\n\t\t\t\t\tactiveBackends.map(async (backend) => {\n\t\t\t\t\t\ttry {\n\t\t\t\t\t\t\tconst results = await runBackend(\n\t\t\t\t\t\t\t\tbackend,\n\t\t\t\t\t\t\t\tparams.query,\n\t\t\t\t\t\t\t\tMath.ceil(numResults / activeBackends.length),\n\t\t\t\t\t\t\t\tsignal,\n\t\t\t\t\t\t\t);\n\t\t\t\t\t\t\treturn {\n\t\t\t\t\t\t\t\tbackend,\n\t\t\t\t\t\t\t\tresults: results.map((r) => ({ ...r, backend })) as SearchResultWithBackend[],\n\t\t\t\t\t\t\t\tsuccess: true,\n\t\t\t\t\t\t\t};\n\t\t\t\t\t\t} catch (err) {\n\t\t\t\t\t\t\treturn {\n\t\t\t\t\t\t\t\tbackend,\n\t\t\t\t\t\t\t\tresults: [] as SearchResultWithBackend[],\n\t\t\t\t\t\t\t\tsuccess: false,\n\t\t\t\t\t\t\t\terror: (err as Error).message,\n\t\t\t\t\t\t\t};\n\t\t\t\t\t\t}\n\t\t\t\t\t}),\n\t\t\t\t);\n\n\t\t\t\t// Build backend stats map\n\t\t\t\tconst backendStats = new Map<\n\t\t\t\t\tstring,\n\t\t\t\t\t{ success: boolean; count: number; error?: string }\n\t\t\t\t>();\n\n\t\t\t\tfor (const { backend, results, success, error } of resultsPerBackend) {\n\t\t\t\t\tbackendStats.set(backend, {\n\t\t\t\t\t\tsuccess,\n\t\t\t\t\t\tcount: results.length,\n\t\t\t\t\t\terror,\n\t\t\t\t\t});\n\t\t\t\t}\n\n\t\t\t\t// Merge and re-rank using Reciprocal Rank Fusion\n\t\t\t\tconst successfulBackends = resultsPerBackend\n\t\t\t\t\t.filter(r => r.success && r.results.length > 0)\n\t\t\t\t\t.map(r => ({ backend: r.backend, results: r.results }));\n\n\t\t\t\tconst combined = successfulBackends.length > 0\n\t\t\t\t\t? reciprocalRankFusion(successfulBackends, numResults)\n\t\t\t\t\t: [];\n\n\t\t\t\treturn {\n\t\t\t\t\tcontent: [\n\t\t\t\t\t\t{\n\t\t\t\t\t\t\ttype: \"text\",\n\t\t\t\t\t\t\ttext: formatCombinedResults(params.query, combined, backendStats),\n\t\t\t\t\t\t},\n\t\t\t\t\t],\n\t\t\t\t\tdetails: {\n\t\t\t\t\t\tbackend: \"combined\",\n\t\t\t\t\t\tresultCount: combined.length,\n\t\t\t\t\t\tbackendStats: Object.fromEntries(backendStats),\n\t\t\t\t\t},\n\t\t\t\t};\n\t\t\t} else {\n\t\t\t\t// Fallback mode: try each enabled backend in order\n\t\t\t\tconst errors: string[] = [];\n\t\t\t\tfor (const backend of activeBackends) {\n\t\t\t\t\ttry {\n\t\t\t\t\t\tconst results = await runBackend(backend, params.query, numResults, signal);\n\t\t\t\t\t\treturn {\n\t\t\t\t\t\t\tcontent: [\n\t\t\t\t\t\t\t\t{\n\t\t\t\t\t\t\t\t\ttype: \"text\",\n\t\t\t\t\t\t\t\t\ttext: errors.length > 0\n\t\t\t\t\t\t\t\t\t\t? `${errors.join(\"; \")}\\n\\n${formatResults(params.query, backend, results)}`\n\t\t\t\t\t\t\t\t\t\t: formatResults(params.query, backend, results),\n\t\t\t\t\t\t\t\t},\n\t\t\t\t\t\t\t],\n\t\t\t\t\t\t\tdetails: {\n\t\t\t\t\t\t\t\tbackend: errors.length > 0 ? `${backend} (fallback)` : backend,\n\t\t\t\t\t\t\t\tresultCount: results.length,\n\t\t\t\t\t\t\t\terrors: errors.length > 0 ? errors : undefined,\n\t\t\t\t\t\t\t},\n\t\t\t\t\t\t};\n\t\t\t\t\t} catch (err) {\n\t\t\t\t\t\terrors.push(`${backend}: ${(err as Error).message}`);\n\t\t\t\t\t}\n\t\t\t\t}\n\n\t\t\t\tthrow new Error(`All backends failed: ${errors.join(\"; \")}`);\n\t\t\t}\n\t\t},\n\t});\n\n\t// -----------------------------------------------------------------------\n\t// Tool: web_read — Read/extract content from a URL\n\t// -----------------------------------------------------------------------\n\n\tpi.registerTool({\n\t\tname: \"web_read\",\n\t\tlabel: \"Read Web Page\",\n\t\tdescription:\n\t\t\t\"Fetch a URL as markdown. Use objective for a concrete question, keywords for long pages, \" +\n\t\t\t\"rush for speed, smart for better narrowing.\",\n\t\tpromptSnippet: \"Read content from a web page (supports markdown extraction)\",\n\t\tpromptGuidelines: [\n\t\t\t\"Use web_read when you need to read the content of a specific URL\",\n\t\t\t\"Set objective for a concrete question when only part of the page matters\",\n\t\t\t\"Add keywords for long pages when you know the relevant terms\",\n\t\t\t\"Choose rush for speed or smart for higher-quality narrowing\",\n\t\t],\n\t\tparameters: Type.Object({\n\t\t\turl: Type.String({\n\t\t\t\tdescription: \"HTTP(S) URL or bare domain to fetch\",\n\t\t\t}),\n\t\t\tfresh: Type.Optional(\n\t\t\t\tType.Boolean({\n\t\t\t\t\tdescription: \"Bypass cache when freshness matters\",\n\t\t\t\t}),\n\t\t\t),\n\t\t\tkeywords: Type.Optional(\n\t\t\t\tType.Array(Type.String(), {\n\t\t\t\t\tdescription: \"Keyword to focus extraction on relevant sections\",\n\t\t\t\t}),\n\t\t\t),\n\t\t\tmode: Type.Optional(\n\t\t\t\tStringEnum([\"rush\", \"smart\"] as const, {\n\t\t\t\t\tdescription: \"rush = faster mode, smart = better section selection on long/noisy pages\",\n\t\t\t\t}),\n\t\t\t),\n\t\t\tobjective: Type.Optional(\n\t\t\t\tType.String({\n\t\t\t\t\tdescription:\n\t\t\t\t\t\t\"CSS selector for targeted extraction. Use when only part of the page matters.\",\n\t\t\t\t}),\n\t\t\t),\n\t\t}),\n\t\tasync execute(_toolCallId, params, signal, _onUpdate, ctx) {\n\t\t\trefreshConfig(ctx.cwd);\n\n\t\t\tconst url = params.url.startsWith(\"https://\") || params.url.startsWith(\"http://\")\n\t\t\t\t? params.url\n\t\t\t\t: `https://${params.url}`;\n\n\t\t\t// Build Jina Reader URL\n\t\t\tconst readerUrl = new URL(\"https://r.jina.ai/\" + url);\n\n\t\t\tconst headers: Record<string, string> = {\n\t\t\t\t\"Accept\": \"text/plain\",\n\t\t\t};\n\n\t\t\t// Optional Jina API key for higher rate limits (fallback to no-auth)\n\t\t\tconst jinaKey = resolveBackendKey(\"jina\");\n\t\t\tif (jinaKey) {\n\t\t\t\theaders[\"Authorization\"] = `Bearer ${jinaKey}`;\n\t\t\t}\n\n\t\t\tif (params.fresh) {\n\t\t\t\theaders[\"x-no-cache\"] = \"true\";\n\t\t\t}\n\t\t\tif (params.keywords && params.keywords.length > 0) {\n\t\t\t\theaders[\"x-keywords\"] = params.keywords.join(\", \");\n\t\t\t}\n\t\t\tif (params.mode) {\n\t\t\t\theaders[\"x-respond-with\"] = params.mode === \"rush\" ? \"text\" : \"markdown\";\n\t\t\t}\n\t\t\tif (params.objective) {\n\t\t\t\theaders[\"x-target-selector\"] = params.objective;\n\t\t\t}\n\n\t\t\tconst response = await fetch(readerUrl.toString(), {\n\t\t\t\tsignal: timeoutSignal(signal),\n\t\t\t\theaders,\n\t\t\t});\n\n\t\t\tif (!response.ok) {\n\t\t\t\tconst text = await response.text().catch(() => \"\");\n\t\t\t\tthrow new Error(`Failed to read ${url}: ${sanitizeError(response.status, text)}`);\n\t\t\t}\n\n\t\t\tconst content = await response.text();\n\t\t\tconst truncated = content.length > 10000\n\t\t\t\t? content.slice(0, 10000) + `\\n\\n[... truncated, full length: ${content.length} chars]`\n\t\t\t\t: content;\n\n\t\t\treturn {\n\t\t\t\tcontent: [{ type: \"text\", text: truncated }],\n\t\t\t\tdetails: {\n\t\t\t\t\turl,\n\t\t\t\t\tlength: content.length,\n\t\t\t\t\ttruncated: content.length > 10000,\n\t\t\t\t},\n\t\t\t};\n\t\t},\n\t});\n\n\t// -----------------------------------------------------------------------\n\t// Commands\n\t// -----------------------------------------------------------------------\n\n\tpi.registerCommand(\"search-setup\", {\n\t\tdescription: \"Configure search backends interactively\",\n\t\thandler: async (_args, ctx) => {\n\t\t\tif (!ctx.hasUI) {\n\t\t\t\tctx.ui.notify(\"/search-setup requires interactive mode\", \"error\");\n\t\t\t\treturn;\n\t\t\t}\n\n\t\t\tconst backends = Object.values(BACKEND_DEFS)\n\t\t\t\t.filter(d => d.setupLabel !== null)\n\t\t\t\t.map(d => d.setupLabel!);\n\n\t\t\tconst backendKey: Record<string, string> = Object.fromEntries(\n\t\t\t\tObject.entries(BACKEND_DEFS)\n\t\t\t\t\t.filter(([_, d]) => d.setupLabel !== null)\n\t\t\t\t\t.map(([k, d]) => [d.setupLabel!, k])\n\t\t\t);\n\n\t\t\tconst option = await ctx.ui.select(\"Which backend do you want to configure?\", [\n\t\t\t\t...backends,\n\t\t\t\t\"✅ Done — save and exit\",\n\t\t\t]);\n\n\t\t\tif (!option || option.startsWith(\"✅ Done\")) {\n\t\t\t\tctx.ui.notify(\"Search setup complete.\", \"info\");\n\t\t\t\treturn;\n\t\t\t}\n\n\t\t\tconst backend = backendKey[option];\n\t\t\tconst label = option;\n\n\t\t\tconst key = await ctx.ui.input(`Enter your ${label} API key:`, {\n\t\t\t\tplaceholder: \"sk-...\",\n\t\t\t\tvalidate: (v: string) =>\n\t\t\t\t\tv.trim().length > 0 ? undefined : \"Key cannot be empty\",\n\t\t\t});\n\n\t\t\tif (!key) {\n\t\t\t\tctx.ui.notify(\"Setup cancelled.\", \"info\");\n\t\t\t\treturn;\n\t\t\t}\n\n\t\t\tconst configDir = join(getAgentDir(), \"extensions\");\n\t\t\tconst configPath = join(configDir, \"search.json\");\n\n\t\t\tmkdirSync(configDir, { recursive: true });\n\n\t\t\tlet existing: SearchConfig = {};\n\t\t\tif (existsSync(configPath)) {\n\t\t\t\ttry {\n\t\t\t\t\texisting = JSON.parse(readFileSync(configPath, \"utf-8\"));\n\t\t\t\t} catch {\n\t\t\t\t\t// ignore\n\t\t\t\t}\n\t\t\t}\n\n\t\t\t// SearXNG setup needs both instance URL and optional API key\n\t\t\tlet backendConfig: BackendConfig = { enabled: true };\n\t\t\tif (backend === \"searxng\") {\n\t\t\t\tconst url = await ctx.ui.input(\"Enter your SearXNG instance URL (e.g. http://localhost:8888):\", {\n\t\t\t\t\tplaceholder: \"http://localhost:8888\",\n\t\t\t\t\tvalidate: (v: string) =>\n\t\t\t\t\t\tv.trim().length > 0 ? undefined : \"URL cannot be empty\",\n\t\t\t\t});\n\t\t\t\tif (!url) {\n\t\t\t\t\tctx.ui.notify(\"Setup cancelled.\", \"info\");\n\t\t\t\t\treturn;\n\t\t\t\t}\n\t\t\t\tbackendConfig.instanceUrl = url.trim();\n\t\t\t\t// Optionally ask for API key (some instances require auth)\n\t\t\t\tconst optionalKey = await ctx.ui.input(\"Optional API key (leave empty if none):\", {\n\t\t\t\t\tplaceholder: \"sk-... (optional)\",\n\t\t\t\t});\n\t\t\t\tif (optionalKey && optionalKey.trim()) {\n\t\t\t\t\tbackendConfig.apiKey = optionalKey.trim();\n\t\t\t\t}\n\t\t\t} else {\n\t\t\t\tbackendConfig.apiKey = key?.trim() || \"\";\n\t\t\t}\n\n\t\t\tconst updated: SearchConfig = {\n\t\t\t\t...existing,\n\t\t\t\tbackends: {\n\t\t\t\t\t...existing.backends,\n\t\t\t\t\t[backend]: backendConfig,\n\t\t\t\t},\n\t\t\t};\n\n\t\t\twriteFileSync(configPath, JSON.stringify(updated, null, 2) + \"\\n\", { mode: 0o600 });\n\n\t\t\tctx.ui.notify(\n\t\t\t\t`${label} API key saved to ${configPath}. Run /reload to activate.`,\n\t\t\t\t\"success\",\n\t\t\t);\n\t\t},\n\t});\n\n\tpi.registerCommand(\"search-status\", {\n\t\tdescription: \"Show which search backends are configured and active\",\n\t\thandler: async (_args, ctx) => {\n\t\t\trefreshConfig(ctx.cwd);\n\n\t\t\tconst backendLabels: Record<string, string> = Object.fromEntries(\n\t\t\t\tObject.entries(BACKEND_DEFS).map(([k, v]) => [k, `${v.label}${k === \"duckduckgo\" ? \" (free, no key)\" : \"\"}`])\n\t\t\t);\n\n\t\t\t// Collect table rows first to compute aligned column widths\n\t\t\ttype Row = [string, string];\n\t\t\tconst rows: Row[] = [];\n\n\t\t\tfor (const [name, label] of Object.entries(backendLabels)) {\n\t\t\t\tconst { configured, source } = getKeySource(name);\n\t\t\t\tconst bc = config.backends?.[name as keyof typeof config.backends];\n\t\t\t\tif (name === \"duckduckgo\") {\n\t\t\t\t\trows.push([label, \"✓ enabled, key: — (free)\"]);\n\t\t\t\t} else if (name === \"marginalia\" && bc?.enabled) {\n\t\t\t\t\trows.push([label, \"✓ enabled, key: optional (public)\"]);\n\t\t\t\t} else if (name === \"searxng\" && bc?.enabled) {\n\t\t\t\t\tconst urlInfo = bc.instanceUrl ? `url: ${bc.instanceUrl}` : \"no URL set\";\n\t\t\t\t\trows.push([label, `✓ enabled, ${urlInfo}${configured ? `, key: ✓ (${source})` : \", key: —\"}`]);\n\t\t\t\t} else if (bc?.enabled) {\n\t\t\t\t\trows.push([label, `✓ enabled, key: ✓${source ? ` (${source})` : \"\"}`]);\n\t\t\t\t} else {\n\t\t\t\t\trows.push([label, `— disabled${configured ? `, key: ✓ (${source})` : \"\"}`]);\n\t\t\t\t}\n\t\t\t}\n\n\t\t\t// Compute column widths from headers + data\n\t\t\tconst col1Header = \"Backend\";\n\t\t\tconst col2Header = \"Status\";\n\t\t\tconst w1 = rows.reduce((max, [c]) => Math.max(max, c.length), col1Header.length);\n\t\t\tconst w2 = rows.reduce((max, [, s]) => Math.max(max, s.length), col2Header.length);\n\n\t\t\tconst pad = (s: string, w: number) => s + \" \".repeat(w - s.length);\n\n\t\t\tconst tableLines = [\n\t\t\t\t`| ${pad(col1Header, w1)} | ${pad(col2Header, w2)} |`,\n\t\t\t\t`| ${\"-\".repeat(w1)} | ${\"-\".repeat(w2)} |`,\n\t\t\t\t...rows.map(([c1, c2]) => `| ${pad(c1, w1)} | ${pad(c2, w2)} |`),\n\t\t\t];\n\n\t\t\tconst resolvedDefault = activeBackends[0] || \"none\";\n\t\t\tconst lines: string[] = [\n\t\t\t\t\"## Search Backend Status\",\n\t\t\t\t`Configured default: ${config.defaultBackend || \"none\"}`,\n\t\t\t\t`Resolved default: ${resolvedDefault}`,\n\t\t\t\t`Active: ${activeBackends.join(\", \") || \"none\"}`,\n\t\t\t\t\"\",\n\t\t\t\t...tableLines,\n\t\t\t];\n\n\t\t\tif (activeBackends.length === 1 && activeBackends[0] === \"duckduckgo\") {\n\t\t\t\tlines.push(\"\");\n\t\t\t\tlines.push(\"Only DuckDuckGo is active (no API key needed).\");\n\t\t\t\tlines.push(\"Add a search backend with /search-setup to get more results.\");\n\t\t\t}\n\n\t\t\tctx.ui.notify(lines.join(\"\\n\"), \"info\");\n\t\t},\n\t});\n\n\t// -----------------------------------------------------------------------\n\t// Session start\n\t// -----------------------------------------------------------------------\n\n\tpi.on(\"session_start\", async (_event, ctx) => {\n\t\tbackendCooldowns.clear();\n\t\trefreshConfig(ctx.cwd);\n\t\tconst status = activeBackends.join(\", \");\n\t\tctx.ui.setStatus(\"search\", `search: ${status}`);\n\t});\n}\n"
  },
  {
    "path": "package.json",
    "content": "{\n  \"name\": \"pi-search-hub\",\n  \"version\": \"1.4.4\",\n  \"description\": \"Unified web search + content extraction extension for pi with 12 backends (DuckDuckGo, Jina AI, Tavily, Brave, Exa, Serper, Firecrawl, Marginalia, LangSearch, WebSearchAPI, Perplexity Sonar, SearXNG). Auto-fallback, RRF combine mode, web_read tool, secure credential resolution.\",\n  \"keywords\": [\n    \"pi-package\",\n    \"pi\",\n    \"pi-coding-agent\",\n    \"search\",\n    \"web-search\",\n    \"web-read\",\n    \"content-extraction\",\n    \"search-hub\",\n    \"duckduckgo\",\n    \"jina\",\n    \"tavily\",\n    \"serper\",\n    \"exa\",\n    \"brave\",\n    \"firecrawl\",\n    \"langsearch\",\n    \"websearchapi\",\n    \"perplexity\",\n    \"searxng\",\n    \"rrf\",\n    \"ai-agent\"\n  ],\n  \"author\": \"\",\n  \"license\": \"MIT\",\n  \"type\": \"module\",\n  \"bugs\": {\n    \"url\": \"https://github.com/ronnieops/pi-search-hub/issues\"\n  },\n  \"repository\": {\n    \"type\": \"git\",\n    \"url\": \"git+https://github.com/ronnieops/pi-search-hub.git\"\n  },\n  \"homepage\": \"https://github.com/ronnieops/pi-search-hub#readme\",\n  \"pi\": {\n    \"extensions\": [\n      \"./extensions/search-hub.ts\"\n    ],\n    \"image\": \"https://pi.dev/assets/packages/pi-search-hub.png\"\n  },\n  \"peerDependencies\": {\n    \"@earendil-works/pi-ai\": \"*\",\n    \"@earendil-works/pi-coding-agent\": \"*\"\n  },\n  \"peerDependenciesMeta\": {\n    \"@earendil-works/pi-ai\": {\n      \"optional\": true\n    },\n    \"@earendil-works/pi-coding-agent\": {\n      \"optional\": true\n    }\n  },\n  \"dependencies\": {\n    \"typebox\": \"^1.1.24\"\n  },\n  \"devDependencies\": {\n    \"@types/node\": \"^25.6.0\",\n    \"typescript\": \"^6.0.3\",\n    \"vitest\": \"^4.1.7\"\n  }\n}\n"
  },
  {
    "path": "search.json.example",
    "content": "{\n  \"defaultBackend\": \"auto\",\n  \"backends\": {\n    \"duckduckgo\": { \"enabled\": true },\n    \"jina\": { \"enabled\": true },\n    \"marginalia\": { \"enabled\": true },\n    \"serper\": { \"enabled\": true, \"apiKey\": \"SERPER_API_KEY\" },\n    \"brave\": { \"enabled\": true, \"apiKey\": \"BRAVE_API_KEY\" },\n    \"tavily\": { \"enabled\": true, \"apiKey\": \"TAVILY_API_KEY\" },\n    \"exa\": { \"enabled\": true, \"apiKey\": \"EXA_API_KEY\" },\n    \"firecrawl\": { \"enabled\": true, \"apiKey\": \"FIRECRAWL_API_KEY\" },\n    \"langsearch\": { \"enabled\": true, \"apiKey\": \"LANGSEARCH_API_KEY\" },\n    \"websearchapi\": { \"enabled\": true, \"apiKey\": \"WEBSEARCHAPI_API_KEY\" },\n    \"perplexity\": { \"enabled\": true, \"apiKey\": \"PERPLEXITY_API_KEY\", \"model\": \"sonar\" },\n    \"searxng\": { \"enabled\": true, \"instanceUrl\": \"http://localhost:8888\" }\n  }\n}\n"
  }
]