[
{
"path": ".gitignore",
"content": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# OS files\n.DS_Store\n.DS_Store?\n._*\n.Spotlight-V100\n.Trashes\nehthumbs.db\nThumbs.db\nDesktop.ini\n\n# IDE files\n.vscode/\n.idea/\n.pytest_cache/\n\n# Distribution / packaging\ndist/\nbuild/\n*.egg-info/\n*.egg\n\n# Environment files\n.env\n\n# MCP files\nanytool/config/config_mcp.json\n\n# Logs\nlogs/\n\n# Embedding cache\n.anytool/\nembedding_cache/\ntool_quality/\n\n# MCP tool cache\nmcp_tool_cache.json\nmcp_tool_cache_sanitized.json\n\n# Config files\nanytool/config/config_dev.json\n\n# LLM keys\nanytool/llm/remote_client/\n\n# Local server temp files\nanytool/local_server/temp/\n\nexamples/"
},
{
"path": "COMMUNICATION.md",
"content": "We provide QR codes for joining the HKUDS discussion groups on **WeChat** and **Feishu**.\n\nYou can join by scanning the QR codes below:\n\n![\"WeChat](\"https://github.com/HKUDS/.github/blob/main/profile/QR.png\")
"
},
{
"path": "LICENSE",
"content": "MIT License\n\nCopyright (c) 2025 HKUDS\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof this software and associated documentation files (the \"Software\"), to deal\nin the Software without restriction, including without limitation the rights\nto use, copy, modify, merge, publish, distribute, sublicense, and/or sell\ncopies of the Software, and to permit persons to whom the Software is\nfurnished to do so, subject to the following conditions:\n\nThe above copyright notice and this permission notice shall be included in all\ncopies or substantial portions of the Software.\n\nTHE SOFTWARE IS PROVIDED \"AS IS\", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR\nIMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,\nFITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE\nAUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER\nLIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,\nOUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE\nSOFTWARE.\n"
},
{
"path": "README.md",
"content": "\n\n
\n ![\"AnyTool](\"assets/AnyTool_logo.png\")
\n\n\n## AnyTool: Universal Tool-Use Layer for AI Agents\n\n### โจ **One Line of Code to Supercharge any Agent with
Fast, Scalable and Powerful Tool Use** โจ\n\n[](https://github.com/HKUDS/AnyTool/)\n[](https://www.python.org/)\n[](https://opensource.org/licenses/MIT/)\n[](./COMMUNICATION.md) \n[](./COMMUNICATION.md)\n\n| โก **Fast - Lightning Tool Retrieval** | ๐ **Self-Evolving Tool Orchestration** | โก **Universal Tool Automation** |\n\n
\nmacOS Setup
\n\n```bash\n# Install macOS-specific dependencies\npip install pyobjc-core pyobjc-framework-cocoa pyobjc-framework-quartz atomacos\n```\n\n**Permissions Required**: macOS will automatically prompt for permissions when you first run the local server. Grant the following:\n- **Accessibility** (for GUI control)\n- **Screen Recording** (for screenshots and video capture)\n\n> If prompts don't appear, manually grant permissions in System Settings โ Privacy & Security.\n \n\n\nLinux Setup
\n\n```bash\n# Install Linux-specific dependencies\npip install python-xlib pyatspi numpy\n\n# Install system packages\nsudo apt install at-spi2-core python3-tk scrot\n```\n\n> [!NOTE]\n> **Optional dependencies:**\n> - Accessibility: `pyatspi` + `at-spi2-core`\n> - Window management: `wmctrl`\n> - Cursor in screenshots: `libx11-dev` + `libxfixes-dev`\n\n \n\n\nWindows Setup
\n\n```bash\n# Install Windows-specific dependencies\npip install pywinauto pywin32 PyGetWindow\n```\n \n\nAfter installing the platform-specific dependencies, start the local server:\n\n```bash\npython -m anytool.local_server.main\n```\n\n> [!NOTE]\n> See [`anytool/local_server/README.md`](anytool/local_server/README.md) for complete API documentation and advanced configuration.\n\n#### Mode Comparison\n\n| | Local Mode (`\"local\"`) | Server Mode (`\"server\"`) |\n|---|---|---|\n| **Setup** | Zero โ just run your agent | Start `local_server` first |\n| **Use case** | Same-machine development | Remote VMs, sandboxing, multi-machine |\n| **Shell execution** | `asyncio.subprocess` in-process | HTTP โ Flask โ `subprocess` |\n| **GUI execution** | Direct pyautogui / ScreenshotHelper | HTTP โ Flask โ pyautogui |\n| **Dependencies** | Only core AnyTool | Core + Flask + platform deps |\n| **Network** | None required | HTTP between agent โ server |\n\n### 3. Quick Integration\n\nAnyTool is a **plug-and-play Universal Tool-Use Layer** for any AI agent. The task passed to `execute()` can come from your agent's planning module, user input, or any workflow system.\n\n```python\nimport asyncio\nfrom anytool import AnyTool\nfrom anytool.tool_layer import AnyToolConfig\n\nasync def main():\n config = AnyToolConfig(\n enable_recording=True,\n recording_backends=[\"gui\", \"shell\", \"mcp\", \"web\"],\n enable_screenshot=True,\n enable_video=True,\n )\n \n async with AnyTool(config=config) as tool_layer:\n result = await tool_layer.execute(\n \"Research trending AI coding tools from GitHub and tech news, \"\n \"collect their features and user feedback, analyze adoption patterns, \"\n \"then create a comparison report with insights\"\n )\n print(result[\"response\"])\n\nasyncio.run(main())\n```\n\n> [!TIP]\n> **MCP Server Configuration**: For tasks requiring specific tools, add relevant MCP servers to `anytool/config/config_mcp.json`. Unsure which servers to add? Simply add all potentially useful ones, AnyTool's Smart Tool RAG will automatically select the appropriate tools for your task. See [MCP Configuration](#mcp-configuration) for details.\n\n---\n\n## Technical Innovation & Implementation\n\n### ๐งฉ Challenge 1: MCP Tool Context Overload\n\n**The Problem**. Current MCP agents suffer from a fundamental design flaw: they load ALL configured servers and tools at every execution step, creating an overwhelming action space, creates three critical issues:\n- โก **Slow Performance with Massive Context Loading**
\n Complete tool set from all pre-configured servers loaded simultaneously at every step, degrading execution speed\n \n- ๐ฏ **Poor Accuracy from Blind Tool Setup**
\n Users cannot preview tools before connecting, leading to over-setup \"just in case\" and confusing tool selection\n \n- ๐ธ **Resource Waste with No Memory**
\n Same tools reloaded at every execution step with no caching, causing redundant loading\n\n### โ
AnyTool's Solution: Tool Context Management Framework\n\n**Motivation**: \"Load Everything\" โ \"Retrieve What's Needed\"
\n**Improvement**: Faster tool selection, cleaner context, and efficient resource usage through smart retrieval and memory.\n\n#### **Technical Innovation**:
\n**๐ฏ Multi-Stage Tool Retrieval Pipeline**\n- **Progressive MCP Tool Filtering**: server selection โ tool name matching โ tool semantic search โ LLM ranking\n- **Reduces MCP Tool Search Space**: Each stage narrows down candidate tools for optimizing precision and speed\n\n**๐พ Long-Term Tool Memory**\n- **Save Once, Use Forever**: Pre-compute tool embeddings once and save them to disk for instant reuse\n- **Zero Waste Processing**: No more redundant processing - tools are ready to use immediately across all execution steps\n\n**๐ง Adaptive Tool Selection**\n- **Adaptive MCP Tool Ranking**: LLM-based tool selection refinement triggered only when MCP tool results are large or ambiguous\n- **Tool Selection Efficiency**: Balances MCP tool accuracy with computational efficiency\n\n**๐ On-Demand Resource Management**\n- **Lazy MCP Server Startup**: MCP server initialization triggered only when specific tools are needed\n- **Selective Tool Updates**: Incremental re-indexing of only changed MCP tools, not the entire tool set\n\n---\n\n### ๐จ Challenge 2: MCP Tool Quality Issues\n\n**The Problem**. Current MCP servers suffer from community contribution challenges that create three critical issues:\n- ๐ **Poor Tool Descriptions**
\n Misleading claims, non-existent advertised tools, and vague capability specifications lead to wrong tool selection.\n \n- ๐ **No Reliability Signals**
\n Cannot assess MCP tool quality before use, causing blind selection decisions.\n \n- โ ๏ธ **Security and Safety Gaps**
\n Unvetted community tools may execute dangerous operations without proper safeguards.\n\n### โ
**AnyTool Solution: Self-Contained Quality Management**\n\n**Motivation**: \"Blind Tool Trust\" โ \"Smart Quality Assessment\"
\n**Improvement**: Reliable tool selection, safe execution, and autonomous recovery through quality tracking and safety controls.\n\n#### **Technical Innovation:**
\n**๐ฏ Quality-Aware Tool Selection**\n- **Description Quality Check**: LLM-based evaluation of MCP tool description clarity and completeness.\n- **Performance-Based Ranking**: Track call/success rates for each MCP tool in persistent memory to prioritize reliable options.\n\n**๐พ Learning-Based Tool Memory**\n- **Track Tool Performance**: Remember which MCP tools work well and which fail over time.\n- **Smart Tool Prioritization**: Automatically rank tools based on past success rates and description quality.\n\n**๐ก๏ธ Safety-First Execution**\n- **Block Dangerous Operations**: Prevent arbitrary code execution and require user approval for sensitive MCP tool operations.\n- **Execution Safeguards**: Built-in safety controls for all MCP tool executions.\n\n**๐ Self-Healing Tool Management**\n- **Autonomous Tool Switching**: Switch failed MCP tools locally without restarting expensive planning loops.\n- **Local Failure Recovery**: Automatically switch to alternative MCP tools on failure without escalating to upper-level agents.\n \n---\n\n### ๐ Challenge 3: Limited MCP Capability Scope\n\n**The Problem**. Current MCP ecosystem focuses primarily on Web APIs and online services, creating significant automation gaps that prevent comprehensive task completion:\n\n- **๐ฅ๏ธ Missing System Operations**
\n No native support for file manipulation, process management, or command execution on local systems.\n\n- **๐ฑ๏ธ No Desktop Automation**
\n Cannot control GUI applications that lack APIs, limiting automation to web-only scenarios.\n\n- **๐ Incomplete Tool Coverage**
\n Limited server categories in community and incomplete tool sets within existing servers create workflow bottlenecks.\n\n### โ
AnyTool Solution: Universal Capability Extension
(MCP + System Commands + GUI Control โ Universal Task Completion)\n\n**Motivation**: \"Web-Only MCP\" โ \"Universal Task Completion\"
\n**Improvement**: Complete automation coverage through multi-backend architecture that seamlessly extends MCP capabilities beyond web APIs.\n\n**๐๏ธ Multi-Backend Architecture**\n- **MCP Backend**: Community servers for Web APIs and online services\n- **Shell Backend**: Bash/Python execution for system-level operations and file management\n- **GUI Backend**: Pixel-level automation for any visual application without API requirements\n- **Web Backend**: Deep web research and data extraction capabilities\n\n**๐ก Self-Evolving Capability Discovery**\n- **Intelligent Gap Detection**: Planning agent identifies when MCP tools are insufficient for task requirements\n- **Automatic Backend Selection**: Shell/GUI backends automatically fill capability gaps without manual intervention\n- **Dynamic Capability Expansion**: Previously impossible tasks become achievable through backend combination\n\n**๐ญ Unified Tool Orchestration**\n- **Uniform Tool Schema**: All backends expose identical interface for seamless agent tool selection\n- **Transparent Backend Switching**: Agents select optimal tools across backend types without knowing implementation details\n- **Intelligent Tool Routing**: Automatic routing to the most appropriate backend based on task requirements\n\n**๐ Seamless Integration Layer**\n- **Single Tool Interface**: Unified API that abstracts away backend complexity from AI agents.\n- **Cross-Backend Coordination**: Enable complex workflows that span multiple backend capabilities.\n- **Consistent Safety Controls**: Apply security and safety measures uniformly across all backend types.\n\n---\n\n## ๐ง Configuration Guide\n\n### Configuration Overview\n\nAnyTool uses a layered configuration system:\n\n- **`config_dev.json`** (highest priority): Local development overrides. Overrides all other configurations.\n- **`config_agents.json`**: Agent definitions and backend access control\n- **`config_mcp.json`**: MCP server registry\n- **`config_grounding.json`**: Backend-specific settings and Smart Tool RAG configuration\n- **`config_security.json`**: Security policies with runtime user confirmation for sensitive operations\n\n---\n\n### Agent Configuration\n\n**Path**: `anytool/config/config_agents.json`\n\n**Purpose**: Define agent roles, control backend access scope, and set execution limits to prevent infinite loops.\n\n**Example configuration**:\n\n```json\n{\n \"agents\": [\n {\n \"name\": \"GroundingAgent\",\n \"class_name\": \"GroundingAgent\",\n \"backend_scope\": [\"gui\", \"shell\", \"mcp\", \"system\", \"web\"],\n \"max_iterations\": 20\n }\n ]\n}\n```\n\n**Key Fields**:\n\n| Field | Description | Options/Example |\n|-------|-------------|-----------------|\n| `backend_scope` | Accessible backends | `[]` or any combination of `[\"gui\", \"shell\", \"mcp\", \"system\", \"web\"]` |\n| `max_iterations` | Maximum execution cycles | Any integer (e.g., `15`, `20`, `50`) or `null` (unlimited) |\n\n---\n\n### MCP Configuration\n\n**Path**: `anytool/config/config_mcp.json` (copy from `config_mcp.json.example`)\n\n**Purpose**: Register MCP servers with connection details. AnyTool automatically discovers tools from all registered servers and makes them available through Smart Tool RAG.\n\n**Example configuration**:\n\n```json\n{\n \"mcpServers\": {\n \"github\": {\n \"command\": \"npx\",\n \"args\": [\"-y\", \"@modelcontextprotocol/server-github\"],\n \"env\": {\n \"GITHUB_PERSONAL_ACCESS_TOKEN\": \"${GITHUB_TOKEN}\"\n }\n }\n }\n}\n```\n\n---\n\n\nRuntime Configuration (AnyToolConfig)
\n\n### Runtime Configuration (AnyToolConfig)\n\n**Complete example**:\n\n```python\nfrom anytool import AnyTool\nfrom anytool.tool_layer import AnyToolConfig\n\nconfig = AnyToolConfig(\n # LLM Configuration\n llm_model=\"anthropic/claude-sonnet-4-5\",\n llm_enable_thinking=False,\n llm_timeout=120.0,\n llm_max_retries=3,\n llm_rate_limit_delay=0.0,\n llm_kwargs={}, # Additional LiteLLM parameters\n \n # Separate models for specific tasks (None = use llm_model)\n tool_retrieval_model=None, # Model for tool retrieval LLM filter\n visual_analysis_model=None, # Model for visual analysis\n \n # Grounding Configuration\n grounding_config_path=None, # Path to custom config file\n grounding_max_iterations=20,\n grounding_system_prompt=None, # Custom system prompt\n \n # Backend Configuration\n backend_scope=[\"gui\", \"shell\", \"mcp\", \"web\", \"system\"],\n \n # Workspace Configuration\n workspace_dir=None, # Auto-create temp dir if None\n \n # Recording Configuration\n enable_recording=True,\n recording_backends=[\"gui\", \"shell\", \"mcp\"],\n recording_log_dir=\"./logs/recordings\",\n enable_screenshot=True,\n enable_video=True,\n enable_conversation_log=True, # Save LLM conversations to conversations.jsonl\n \n # Logging Configuration\n log_level=\"INFO\",\n log_to_file=False,\n log_file_path=None,\n)\n\nasync with AnyTool(config=config) as tool_layer:\n result = await tool_layer.execute(\"Your task here\")\n # Or with external task_id for benchmark integration:\n # result = await tool_layer.execute(\"Your task\", task_id=\"my-task-001\")\n```\n\n \n\n---\n\n\nOther Configuration Files
\n\n### Backend Configuration\n\n**Path**: `anytool/config/config_grounding.json`\n\n**Purpose**: Configure backend-specific behaviors, timeouts, Smart Tool RAG system for efficient tool selection, and Tool Quality Tracking for self-evolving tool intelligence.\n\n**Key Fields**:\n\n| Backend | Field | Description | Options/Default |\n|---------|-------|-------------|-----------------|\n| **shell** | `timeout` | Command timeout (seconds) | Any integer (default: `60`) |\n| | `conda_env` | Auto-activate conda environment | Environment name or `null` (default: `\"anytool\"`) |\n| | `working_dir` | Working directory for command execution | Any valid path (default: current directory) |\n| | `default_shell` | Shell to use | `\"/bin/bash\"`, `\"/bin/zsh\"`, etc. |\n| **gui** | `timeout` | Operation timeout (seconds) | Any integer (default: `90`) |\n| | `screenshot_on_error` | Capture screenshot on failure | `true` or `false` (default: `true`) |\n| | `driver_type` | GUI automation driver | `\"pyautogui\"` or other supported drivers |\n| **mcp** | `timeout` | Request timeout (seconds) | Any integer (default: `30`) |\n| | `sandbox` | Run in E2B sandbox | `true` or `false` (default: `false`) |\n| | `eager_sessions` | Pre-connect all servers at startup | `true` or `false` (default: `false`, lazy connection) |\n| **tool_search** | `search_mode` | Tool retrieval strategy | `\"semantic\"`, `\"hybrid\"` (semantic + LLM filter), or `\"llm\"` (default: `\"hybrid\"`) |\n| | `max_tools` | Maximum tools to return from search | Any integer (default: `40`) |\n| | `enable_llm_filter` | Enable LLM-based tool pre-filtering | `true` or `false` (default: `true`) |\n| | `llm_filter_threshold` | Enable LLM filter when tools exceed this count | Any integer (default: `50`) |\n| | `enable_cache_persistence` | Persist embedding cache to disk | `true` or `false` (default: `true`) |\n| **tool_quality** | `enabled` | Enable tool quality tracking | `true` or `false` (default: `true`) |\n| | `enable_persistence` | Persist quality data to disk | `true` or `false` (default: `true`) |\n| | `cache_dir` | Directory for quality cache | Path string (default: `.anytool/tool_quality` in project directory) |\n| | `auto_evaluate_descriptions` | Automatically evaluate tool descriptions using LLM | `true` or `false` (default: `true`) |\n| | `enable_quality_ranking` | Incorporate quality scores in tool ranking | `true` or `false` (default: `true`) |\n| | `evolve_interval` | Trigger self-evolution every N tool executions | Any integer 1-100 (default: `5`) |\n\n---\n\n### Security Configuration\n\n**Path**: `anytool/config/config_security.json`\n\n**Purpose**: Define security policies with command filtering and access control.\n\n**Key Fields**:\n\n| Section | Field | Description | Options |\n|---------|-------|-------------|---------|\n| **global** | `allow_shell_commands` | Enable shell command execution | `true` or `false` (default: `true`) |\n| | `allow_network_access` | Enable network operations | `true` or `false` (default: `true`) |\n| | `allow_file_access` | Enable file system operations | `true` or `false` (default: `true`) |\n| | `blocked_commands` | Platform-specific command blacklist | Object with `common`, `linux`, `darwin`, `windows` arrays |\n| | `sandbox_enabled` | Enable sandboxing for all operations | `true` or `false` (default: `false`) |\n| **backend** | `shell`, `mcp`, `gui`, `web` | Per-backend security overrides | Same fields as global, backend-specific |\n\n**Example blocked commands**: `rm -rf`, `shutdown`, `reboot`, `mkfs`, `dd`, `format`, `iptables`\n\n**Behavior**: \n- Blocked commands are **rejected automatically**\n- Sandbox mode isolates operations in secure environments (E2B sandbox for MCP)\n\n---\n\n### Developer Configuration\n\n**Path**: `anytool/config/config_dev.json` (copy from `config_dev.json.example`)\n\n**Loading Priority**: `config_grounding.json` โ `config_security.json` โ `config_dev.json` (dev.json overrides the former ones)\n\n \n\n---\n\n## ๐ Code Structure\n\n### ๐ Quick Overview\n\n> **Legend**: โก Core modules | ๐ง Supporting modules\n\n```\nAnyTool/\nโโโ anytool/\nโ โโโ __init__.py # Package exports\nโ โโโ __main__.py # CLI entry point (python -m anytool)\nโ โโโ tool_layer.py # AnyTool main class\nโ โ\nโ โโโ โก agents/ # Agent System\nโ โโโ โก grounding/ # Unified Backend System\nโ โ โโโ core/ # Core abstractions\nโ โ โโโ backends/ # Backend implementations\nโ โ โโโ shell/ # Shell command execution\nโ โ โโโ gui/ # Anthropic Computer Use\nโ โ โโโ mcp/ # Model Context Protocol\nโ โ โโโ web/ # Web search & browsing\nโ โ\nโ โโโ ๐ง prompts/ # Prompt Templates\nโ โโโ ๐ง llm/ # LLM Integration\nโ โโโ ๐ง config/ # Configuration System\nโ โโโ ๐ง local_server/ # GUI Backend Server\nโ โโโ ๐ง recording/ # Execution Recording\nโ โโโ ๐ง platform/ # Platform Integration\nโ โโโ ๐ง utils/ # Utilities\nโ\nโโโ .anytool/ # Runtime cache\nโ โโโ embedding_cache/ # Tool embeddings for Smart Tool RAG\nโ โโโ tool_quality/ # Persistent tool quality tracking data\nโ\nโโโ logs/ # Execution logs\nโ\nโโโ requirements.txt # Python dependencies\nโโโ pyproject.toml # Package configuration\nโโโ README.md\n```\n\n---\n\n### ๐ Detailed Module Structure\n\n\nโก agents/ - Agent System
\n\n```\nagents/\nโโโ __init__.py\nโโโ base.py # Base agent class with common functionality\nโโโ grounding_agent.py # Execution Agent (tool calling & iteration control)\n```\n\n**Key Responsibilities**: Task execution with intelligent tool selection and iteration control.\n\n \n\n\nโก grounding/ - Unified Backend System (Core Integration Layer)
\n\n**Key Responsibilities**: Unified tool abstraction, backend routing, session pooling, Smart Tool RAG, and Self-Evolving Quality Tracking*.\n\n#### Core Abstractions\n\n```\ngrounding/core/\nโโโ grounding_client.py # Unified interface across all backends\nโโโ provider.py # Abstract provider base class\nโโโ session.py # Session lifecycle management\nโโโ search_tools.py # Smart Tool RAG for semantic search\nโโโ exceptions.py # Custom exception definitions\nโโโ types.py # Shared type definitions\nโ\nโโโ tool/ # Tool abstraction layer\nโ โโโ base.py # Tool base class\nโ โโโ local_tool.py # Local tool implementation\nโ โโโ remote_tool.py # Remote tool implementation\nโ\nโโโ quality/ # Self-evolving tool quality tracking\nโ โโโ manager.py # Quality manager with adaptive ranking\nโ โโโ store.py # Persistent quality data storage\nโ โโโ types.py # Quality record data types\nโ\nโโโ security/ # Security & sandboxing ๐ง\nโ โโโ policies.py # Security policy enforcement\nโ โโโ sandbox.py # Sandbox abstraction\nโ โโโ e2b_sandbox.py # E2B sandbox integration\nโ\nโโโ system/ # System-level provider\nโ โโโ provider.py\nโ โโโ tool.py\nโ\nโโโ transport/ # Transport layer abstractions ๐ง\n โโโ connectors/\n โ โโโ base.py\n โ โโโ aiohttp_connector.py\n โโโ task_managers/\n โโโ base.py\n โโโ async_ctx.py\n โโโ aiohttp_connection_manager.py\n โโโ placeholder.py\n```\n\n#### Backend Implementations\n\n\nShell Backend - Command execution via local server
\n\n```\nbackends/shell/\nโโโ provider.py # Shell provider implementation\nโโโ session.py # Shell session management\nโโโ transport/\n โโโ connector.py # HTTP connector to local server\n```\n\n \n\n\nGUI Backend - Anthropic Computer Use integration
\n\n```\nbackends/gui/\nโโโ provider.py # GUI provider implementation\nโโโ session.py # GUI session management\nโโโ tool.py # GUI-specific tools\nโโโ anthropic_client.py # Anthropic API client wrapper\nโโโ anthropic_utils.py # Utility functions\nโโโ config.py # GUI configuration\nโโโ transport/\n โโโ connector.py # Computer Use API connector\n โโโ actions.py # Action execution logic\n```\n\n \n\n\nMCP Backend - Model Context Protocol servers
\n\n```\nbackends/mcp/\nโโโ provider.py # MCP provider implementation\nโโโ session.py # MCP session management\nโโโ client.py # MCP client\nโโโ config.py # MCP configuration loader\nโโโ installer.py # MCP server installer\nโโโ tool_converter.py # Convert MCP tools to unified format\nโโโ tool_cache.py # MCP tool cache for offline tool discovery\nโโโ transport/\n โโโ connectors/ # Multiple transport types\n โ โโโ base.py\n โ โโโ stdio.py # Standard I/O connector\n โ โโโ http.py # HTTP connector\n โ โโโ websocket.py # WebSocket connector\n โ โโโ sandbox.py # Sandboxed connector\n โ โโโ utils.py\n โโโ task_managers/ # Protocol-specific managers\n โโโ stdio.py\n โโโ sse.py\n โโโ streamable_http.py\n โโโ websocket.py\n```\n\n \n\n\nWeb Backend - Search and browsing
\n\n```\nbackends/web/\nโโโ provider.py # Web provider implementation\nโโโ session.py # Web session management\n```\n\n \n\n \n\n\n๐ง prompts/ - Prompt Templates
\n\n```\nprompts/\nโโโ __init__.py\nโโโ grounding_agent_prompts.py # Grounding agent system & tool selection prompts\n```\n\n \n\n\n๐ง llm/ - LLM Integration
\n\n```\nllm/\nโโโ __init__.py\nโโโ client.py # LiteLLM wrapper with retry logic\n```\n\n \n\n\n๐ง config/ - Configuration System
\n\n```\nconfig/\nโโโ __init__.py\nโโโ loader.py # Configuration file loader\nโโโ constants.py # System constants\nโโโ grounding.py # Grounding configuration dataclasses\nโโโ utils.py # Configuration utilities\nโ\nโโโ config_grounding.json # Backend-specific settings\nโโโ config_agents.json # Agent configurations\nโโโ config_mcp.json.example # MCP server definitions (copy to config_mcp.json)\nโโโ config_security.json # Security policies\nโโโ config_dev.json.example # Development config template\n```\n\n \n\n\n๐ง local_server/ - GUI Backend Server
\n\n```\nlocal_server/\nโโโ __init__.py\nโโโ main.py # Flask application entry point\nโโโ config.json # Server configuration\nโโโ feature_checker.py # Platform feature detection\nโโโ health_checker.py # Server health monitoring\nโโโ platform_adapters/ # OS-specific implementations\nโ โโโ macos_adapter.py # macOS automation (atomacos, pyobjc)\nโ โโโ linux_adapter.py # Linux automation (pyatspi, xlib)\nโ โโโ windows_adapter.py # Windows automation (pywinauto)\nโ โโโ pyxcursor.py # Custom cursor handling\nโโโ utils/\nโ โโโ accessibility.py # Accessibility tree utilities\nโ โโโ screenshot.py # Screenshot capture\nโโโ README.md\n```\n\n**Purpose**: Lightweight Flask service enabling computer control (GUI, Shell, Files, Screen capture).\n\n \n\n\n๐ง recording/ - Execution Recording
\n\n```\nrecording/\nโโโ __init__.py\nโโโ recorder.py # Main recording manager\nโโโ manager.py # Recording lifecycle management\nโโโ action_recorder.py # Action-level logging\nโโโ video.py # Video capture integration\nโโโ viewer.py # Trajectory viewer and analyzer\nโโโ utils.py # Recording utilities\n```\n\n**Purpose**: Execution audit with trajectory recording and video capture.\n\n \n\n\n๐ง platform/ - Platform Integration
\n\n```\nplatform/\nโโโ __init__.py\nโโโ config.py # Platform-specific configuration\nโโโ recording.py # Recording integration\nโโโ screenshot.py # Screenshot utilities\nโโโ system_info.py # System information gathering\n```\n\n \n\n\n๐ง utils/ - Shared Utilities
\n\n```\nutils/\nโโโ logging.py # Structured logging system\nโโโ ui.py # Terminal UI components\nโโโ display.py # Display formatting utilities\nโโโ cli_display.py # CLI-specific display\nโโโ ui_integration.py # UI integration helpers\nโโโ telemetry/ # Usage analytics (opt-in)\n โโโ __init__.py\n โโโ events.py\n โโโ telemetry.py\n โโโ utils.py\n```\n\n \n\n\n๐ logs/ - Execution Logs & Recordings
\n\n```\nlogs/\nโโโ / # Main application logs\nโ โโโ anytool_YYYY-MM-DD_HH-MM-SS.log # Timestamped log files\nโ\nโโโ recordings/ # Execution recordings\n โโโ task_/ # Individual recording session\n โโโ trajectory.json # Complete execution trajectory\n โโโ screenshots/ # Visual execution record (GUI backend)\n โ โโโ tool__.png\n โ โโโ tool__.png\n โ โโโ ... # Sequential screenshots\n โโโ workspace/ # Task workspace\n โ โโโ [generated files] # Files created during execution\n โโโ screen_recording.mp4 # Video recording (if enabled)\n```\n\n**Recording Control**: Enable via `AnyToolConfig(enable_recording=True)`, filter backends with `recording_backends=[\"gui\", \"shell\", ...]`\n\n \n\n---\n\n## ๐ Related Projects\n\nAnyTool builds upon excellent open-source projects, we sincerely thank their authors and contributors:\n\n- **[OSWorld](https://github.com/xlang-ai/OSWorld)**: Comprehensive benchmark for evaluating computer-use agents across diverse operating system tasks.\n- **[mcp-use](https://github.com/mcp-use/mcp-use)**: Platform that simplifies MCP agent development with client SDKs.\n\n---\n\n\n\n**๐ If this project helps you, please give us a Star!**\n\n**๐ค Empower AI Agent with intelligent tool orchestration!** \n\n
\n\n---\n\n\n โค๏ธ Thanks for visiting โจ AnyTool!
\n ![\"Views\"](\"https://visitor-badge.laobi.icu/badge?page_id=HKUDS.AnyTool&style=for-the-badge&color=00d4ff\")
\n
\n"
},
{
"path": "anytool/__init__.py",
"content": "from importlib import import_module as _imp\nfrom typing import Dict as _Dict, Any as _Any, TYPE_CHECKING as _TYPE_CHECKING\n\nif _TYPE_CHECKING:\n from anytool.tool_layer import AnyTool as AnyTool, AnyToolConfig as AnyToolConfig\n from anytool.agents import GroundingAgent as GroundingAgent\n from anytool.llm import LLMClient as LLMClient\n from anytool.recording import RecordingManager as RecordingManager\n\n__version__ = \"0.1.0\"\n\n__all__ = [\n # Version\n \"__version__\",\n \n # Main API\n \"AnyTool\",\n \"AnyToolConfig\",\n\n # Core Components\n \"GroundingAgent\",\n \"GroundingClient\",\n \"LLMClient\",\n \"BaseTool\",\n \"ToolResult\",\n \"BackendType\",\n\n # Recording System\n \"RecordingManager\",\n \"RecordingViewer\",\n]\n\n# Map attribute โ sub-module that provides it\n_attr_to_module: _Dict[str, str] = {\n # Main API\n \"AnyTool\": \"anytool.tool_layer\",\n \"AnyToolConfig\": \"anytool.tool_layer\",\n\n # Core Components\n \"GroundingAgent\": \"anytool.agents\",\n \"GroundingClient\": \"anytool.grounding.core.grounding_client\",\n \"LLMClient\": \"anytool.llm\",\n \"BaseTool\": \"anytool.grounding.core.tool.base\",\n \"ToolResult\": \"anytool.grounding.core.types\",\n \"BackendType\": \"anytool.grounding.core.types\",\n\n # Recording System\n \"RecordingManager\": \"anytool.recording\",\n \"RecordingViewer\": \"anytool.recording.viewer\",\n}\n\n\ndef __getattr__(name: str) -> _Any:\n \"\"\"Dynamically import sub-modules on first attribute access.\n\n This keeps the *initial* package import lightweight and avoids raising\n `ModuleNotFoundError` for optional / heavy dependencies until the\n corresponding functionality is explicitly used.\n \"\"\"\n if name not in _attr_to_module:\n raise AttributeError(f\"module 'anytool' has no attribute '{name}'\")\n\n module_name = _attr_to_module[name]\n module = _imp(module_name)\n value = getattr(module, name)\n globals()[name] = value \n return value\n\n\ndef __dir__():\n return sorted(list(globals().keys()) + list(_attr_to_module.keys()))"
},
{
"path": "anytool/__main__.py",
"content": "import asyncio\nimport argparse\nimport sys\nimport logging\nfrom typing import Optional\n\nfrom anytool.tool_layer import AnyTool, AnyToolConfig\nfrom anytool.utils.logging import Logger\nfrom anytool.utils.ui import create_ui, AnyToolUI\nfrom anytool.utils.ui_integration import UIIntegration\nfrom anytool.utils.cli_display import CLIDisplay\nfrom anytool.utils.display import colorize\n\nlogger = Logger.get_logger(__name__)\n\n\nclass UIManager:\n def __init__(self, ui: Optional[AnyToolUI], ui_integration: Optional[UIIntegration]):\n self.ui = ui\n self.ui_integration = ui_integration\n self._original_log_levels = {}\n \n async def start_live_display(self):\n if not self.ui or not self.ui_integration:\n return\n \n print()\n print(colorize(\" โฃ Starting real-time visualization...\", 'c'))\n print()\n await asyncio.sleep(1)\n \n self._suppress_logs()\n \n await self.ui.start_live_display()\n await self.ui_integration.start_monitoring(poll_interval=2.0)\n \n async def stop_live_display(self):\n if not self.ui or not self.ui_integration:\n return\n \n await self.ui_integration.stop_monitoring()\n await self.ui.stop_live_display()\n \n self._restore_logs()\n \n def print_summary(self, result: dict):\n if self.ui:\n self.ui.print_summary(result)\n else:\n CLIDisplay.print_result_summary(result)\n \n def _suppress_logs(self):\n log_names = [\"anytool\", \"anytool.grounding\", \"anytool.agents\"]\n for name in log_names:\n log = logging.getLogger(name)\n self._original_log_levels[name] = log.level\n log.setLevel(logging.CRITICAL)\n \n def _restore_logs(self):\n for name, level in self._original_log_levels.items():\n logging.getLogger(name).setLevel(level)\n self._original_log_levels.clear()\n\n\nasync def _execute_task(anytool: AnyTool, query: str, ui_manager: UIManager):\n await ui_manager.start_live_display()\n result = await anytool.execute(query)\n await ui_manager.stop_live_display()\n ui_manager.print_summary(result)\n return result\n\n\nasync def interactive_mode(anytool: AnyTool, ui_manager: UIManager):\n CLIDisplay.print_interactive_header()\n \n while True:\n try:\n prompt = colorize(\">>> \", 'c', bold=True)\n query = input(f\"\\n{prompt}\").strip()\n \n if not query:\n continue\n \n if query.lower() in ['exit', 'quit', 'q']:\n print(\"\\nExiting...\")\n break\n\n if query.lower() == 'status':\n _print_status(anytool)\n continue\n \n if query.lower() == 'help':\n CLIDisplay.print_help()\n continue\n\n CLIDisplay.print_task_header(query)\n await _execute_task(anytool, query, ui_manager)\n \n except KeyboardInterrupt:\n print(\"\\n\\nInterrupt signal detected, exiting...\")\n break\n except Exception as e:\n logger.error(f\"Error: {e}\", exc_info=True)\n print(f\"\\nError: {e}\")\n\n\nasync def single_query_mode(anytool: AnyTool, query: str, ui_manager: UIManager):\n CLIDisplay.print_task_header(query, title=\"โถ Single Query Execution\")\n await _execute_task(anytool, query, ui_manager)\n\n\ndef _print_status(anytool: AnyTool):\n \"\"\"Print system status\"\"\"\n from anytool.utils.display import Box, BoxStyle\n \n box = Box(width=70, style=BoxStyle.ROUNDED, color='bl')\n print()\n print(box.text_line(colorize(\"System Status\", 'bl', bold=True), \n align='center', indent=4, text_color=''))\n print(box.separator_line(indent=4))\n \n status_lines = [\n f\"Initialized: {colorize('Yes' if anytool.is_initialized() else 'No', 'g' if anytool.is_initialized() else 'rd')}\",\n f\"Running: {colorize('Yes' if anytool.is_running() else 'No', 'y' if anytool.is_running() else 'g')}\",\n f\"Model: {colorize(anytool.config.llm_model, 'c')}\",\n ]\n \n if anytool.is_initialized():\n backends = anytool.list_backends()\n status_lines.append(f\"Backends: {colorize(', '.join(backends), 'c')}\")\n \n sessions = anytool.list_sessions()\n status_lines.append(f\"Active Sessions: {colorize(str(len(sessions)), 'y')}\")\n \n for line in status_lines:\n print(box.text_line(f\" {line}\", indent=4, text_color=''))\n \n print(box.bottom_line(indent=4))\n print()\n\n\ndef _create_argument_parser() -> argparse.ArgumentParser:\n \"\"\"Create command-line argument parser\"\"\"\n parser = argparse.ArgumentParser(\n description='AnyTool - Universal Tool-Use Layer for AI Agents',\n formatter_class=argparse.RawDescriptionHelpFormatter,\n )\n w\n # Subcommands\n subparsers = parser.add_subparsers(dest='command', help='Available commands')\n \n # refresh-cache subcommand\n cache_parser = subparsers.add_parser(\n 'refresh-cache',\n help='Refresh MCP tool cache (starts all servers once)'\n )\n cache_parser.add_argument(\n '--config', '-c', type=str,\n help='MCP configuration file path'\n )\n \n # Basic arguments (for run mode)\n parser.add_argument('--config', '-c', type=str, help='Configuration file path (JSON format)')\n parser.add_argument('--query', '-q', type=str, help='Single query mode: execute query directly')\n \n # LLM arguments\n parser.add_argument('--model', '-m', type=str, help='LLM model name')\n \n # Logging arguments\n parser.add_argument('--log-level', type=str, choices=['DEBUG', 'INFO', 'WARNING', 'ERROR'], help='Log level')\n \n # Execution arguments\n parser.add_argument('--max-iterations', type=int, help='Maximum iteration count')\n parser.add_argument('--timeout', type=float, help='LLM API call timeout (seconds)')\n \n # UI arguments\n parser.add_argument('--interactive', '-i', action='store_true', help='Force interactive mode')\n parser.add_argument('--no-ui', action='store_true', help='Disable visualization UI')\n parser.add_argument('--ui-compact', action='store_true', help='Use compact UI layout')\n \n return parser\n\n\nasync def refresh_mcp_cache(config_path: Optional[str] = None):\n \"\"\"Refresh MCP tool cache by starting servers one by one and saving tool metadata.\"\"\"\n from anytool.grounding.backends.mcp import MCPProvider, get_tool_cache\n from anytool.grounding.core.types import SessionConfig, BackendType\n from anytool.config import load_config, get_config\n \n print(\"Refreshing MCP tool cache...\")\n print(\"Servers will be started one by one (start -> get tools -> close).\")\n print()\n \n # Load config\n if config_path:\n config = load_config(config_path)\n else:\n config = get_config()\n \n # Get MCP config\n mcp_config = getattr(config, 'mcp', None) or {}\n if hasattr(mcp_config, 'model_dump'):\n mcp_config = mcp_config.model_dump()\n \n # Skip dependency checks for refresh-cache (servers are pre-validated)\n mcp_config[\"check_dependencies\"] = False\n \n # Create provider\n provider = MCPProvider(config=mcp_config)\n await provider.initialize()\n \n servers = provider.list_servers()\n total = len(servers)\n print(f\"Found {total} MCP servers configured\")\n print()\n \n cache = get_tool_cache()\n cache.set_server_order(servers) # Preserve config order when saving\n total_tools = 0\n success_count = 0\n skipped_count = 0\n failed_servers = []\n \n # Load existing cache to skip already processed servers\n existing_cache = cache.get_all_tools()\n \n # Timeout for each server (in seconds)\n SERVER_TIMEOUT = 60\n \n # Process servers one by one\n for i, server_name in enumerate(servers, 1):\n # Skip if already cached (resume support)\n if server_name in existing_cache:\n cached_tools = existing_cache[server_name]\n total_tools += len(cached_tools)\n skipped_count += 1\n print(f\"[{i}/{total}] {server_name}... โญ cached ({len(cached_tools)} tools)\")\n continue\n \n print(f\"[{i}/{total}] {server_name}...\", end=\" \", flush=True)\n session_id = f\"mcp-{server_name}\"\n \n try:\n # Create session and get tools with timeout protection\n async with asyncio.timeout(SERVER_TIMEOUT):\n # Create session for this server\n cfg = SessionConfig(\n session_name=session_id,\n backend_type=BackendType.MCP,\n connection_params={\"server\": server_name},\n )\n session = await provider.create_session(cfg)\n \n # Get tools from this server\n tools = await session.list_tools()\n \n # Convert to metadata format\n tool_metadata = []\n for tool in tools:\n tool_metadata.append({\n \"name\": tool.schema.name,\n \"description\": tool.schema.description or \"\",\n \"parameters\": tool.schema.parameters or {},\n })\n \n # Save to cache (incremental)\n cache.save_server(server_name, tool_metadata)\n \n # Close session immediately to free resources\n await provider.close_session(session_id)\n \n total_tools += len(tools)\n success_count += 1\n print(f\"โ {len(tools)} tools\")\n \n except asyncio.TimeoutError:\n error_msg = f\"Timeout after {SERVER_TIMEOUT}s\"\n failed_servers.append((server_name, error_msg))\n print(f\"โ {error_msg}\")\n \n # Save failed server info to cache\n cache.save_failed_server(server_name, error_msg)\n \n # Try to close session if it was created\n try:\n await provider.close_session(session_id)\n except Exception:\n pass\n \n except Exception as e:\n error_msg = str(e)\n failed_servers.append((server_name, error_msg))\n print(f\"โ {error_msg[:50]}\")\n \n # Save failed server info to cache\n cache.save_failed_server(server_name, error_msg)\n \n # Try to close session if it was created\n try:\n await provider.close_session(session_id)\n except Exception:\n pass\n \n print()\n print(f\"{'='*50}\")\n print(f\"โ Collected {total_tools} tools from {success_count + skipped_count}/{total} servers\")\n if skipped_count > 0:\n print(f\" (skipped {skipped_count} cached, processed {success_count} new)\")\n print(f\"โ Cache saved to: {cache.cache_path}\")\n \n if failed_servers:\n print(f\"โ Failed servers ({len(failed_servers)}):\")\n for name, err in failed_servers[:10]:\n print(f\" - {name}: {err[:60]}\")\n if len(failed_servers) > 10:\n print(f\" ... and {len(failed_servers) - 10} more (see cache file for details)\")\n \n print()\n print(\"Done! Future list_tools() calls will use cache (no server startup).\")\n\n\ndef _load_config(args) -> AnyToolConfig:\n \"\"\"Load configuration\"\"\"\n cli_overrides = {}\n if args.model:\n cli_overrides['llm_model'] = args.model\n if args.max_iterations is not None:\n cli_overrides['grounding_max_iterations'] = args.max_iterations\n if args.timeout is not None:\n cli_overrides['llm_timeout'] = args.timeout\n if args.log_level:\n cli_overrides['log_level'] = args.log_level\n \n try:\n # Load from config file if provided\n if args.config:\n import json\n with open(args.config, 'r', encoding='utf-8') as f:\n config_dict = json.load(f)\n \n # Apply CLI overrides\n config_dict.update(cli_overrides)\n config = AnyToolConfig(**config_dict)\n \n print(f\"โ Loaded from config file: {args.config}\")\n else:\n # Use default config + CLI overrides\n config = AnyToolConfig(**cli_overrides)\n print(\"โ Using default configuration\")\n \n if cli_overrides:\n print(f\"โ CLI overrides: {', '.join(cli_overrides.keys())}\")\n \n if args.log_level:\n Logger.set_level(args.log_level)\n \n return config\n \n except Exception as e:\n logger.error(f\"Failed to load configuration: {e}\")\n sys.exit(1)\n\n\ndef _setup_ui(args) -> tuple[Optional[AnyToolUI], Optional[UIIntegration]]:\n if args.no_ui:\n CLIDisplay.print_banner()\n return None, None\n \n ui = create_ui(enable_live=True, compact=args.ui_compact)\n ui.print_banner()\n ui_integration = UIIntegration(ui)\n return ui, ui_integration\n\n\nasync def _initialize_anytool(config: AnyToolConfig, args) -> AnyTool:\n anytool = AnyTool(config)\n \n init_steps = [(\"Initializing AnyTool...\", \"loading\")]\n CLIDisplay.print_initialization_progress(init_steps, show_header=False)\n \n if not args.config:\n original_log_level = Logger.get_logger(\"anytool\").level\n for log_name in [\"anytool\", \"anytool.grounding\", \"anytool.agents\"]:\n Logger.get_logger(log_name).setLevel(logging.WARNING)\n \n await anytool.initialize()\n \n # Restore log level\n if not args.config:\n for log_name in [\"anytool\", \"anytool.grounding\", \"anytool.agents\"]:\n Logger.get_logger(log_name).setLevel(original_log_level)\n \n # Print initialization results\n backends = anytool.list_backends()\n init_steps = [\n (\"LLM Client\", \"ok\"),\n (f\"Grounding Backends ({len(backends)} available)\", \"ok\"),\n (\"Grounding Agent\", \"ok\"),\n ]\n \n if config.enable_recording:\n init_steps.append((\"Recording Manager\", \"ok\"))\n \n CLIDisplay.print_initialization_progress(init_steps, show_header=True)\n \n return anytool\n\n\nasync def main():\n parser = _create_argument_parser()\n args = parser.parse_args()\n \n # Handle subcommands\n if args.command == 'refresh-cache':\n await refresh_mcp_cache(args.config)\n return 0\n \n # Load configuration\n config = _load_config(args)\n \n # Setup UI\n ui, ui_integration = _setup_ui(args)\n \n # Print configuration\n CLIDisplay.print_configuration(config)\n \n anytool = None\n \n try:\n # Initialize AnyTool\n anytool = await _initialize_anytool(config, args)\n \n # Connect UI (if enabled)\n if ui_integration:\n ui_integration.attach_llm_client(anytool._llm_client)\n ui_integration.attach_grounding_client(anytool._grounding_client)\n CLIDisplay.print_system_ready()\n \n ui_manager = UIManager(ui, ui_integration)\n \n # Run appropriate mode\n if args.query:\n await single_query_mode(anytool, args.query, ui_manager)\n else:\n await interactive_mode(anytool, ui_manager)\n \n except KeyboardInterrupt:\n print(\"\\n\\nInterrupt signal detected\")\n except Exception as e:\n logger.error(f\"Error: {e}\", exc_info=True)\n print(f\"\\nError: {e}\")\n return 1\n finally:\n if anytool:\n print(\"\\nCleaning up resources...\")\n await anytool.cleanup()\n \n print(\"\\nGoodbye!\")\n return 0\n\n\ndef run_main():\n \"\"\"Run main function\"\"\"\n try:\n exit_code = asyncio.run(main())\n sys.exit(exit_code)\n except KeyboardInterrupt:\n print(\"\\n\\nProgram interrupted\")\n sys.exit(0)\n\n\nif __name__ == \"__main__\":\n run_main()"
},
{
"path": "anytool/agents/__init__.py",
"content": "from anytool.agents.base import BaseAgent, AgentStatus, AgentRegistry\nfrom anytool.agents.grounding_agent import GroundingAgent\n\n__all__ = [\n \"BaseAgent\",\n \"AgentStatus\",\n \"AgentRegistry\",\n \"GroundingAgent\",\n]"
},
{
"path": "anytool/agents/base.py",
"content": "from __future__ import annotations\n\nimport json\nfrom abc import ABC, abstractmethod\nfrom typing import TYPE_CHECKING, Dict, List, Optional, Type, Any\n\nfrom anytool.utils.logging import Logger\n\nif TYPE_CHECKING:\n from anytool.llm import LLMClient\n from anytool.grounding.core.grounding_client import GroundingClient\n from anytool.recording import RecordingManager\n\nlogger = Logger.get_logger(__name__)\n\n\nclass BaseAgent(ABC):\n def __init__(\n self,\n name: str,\n backend_scope: Optional[List[str]] = None,\n llm_client: Optional[LLMClient] = None,\n grounding_client: Optional[GroundingClient] = None,\n recording_manager: Optional[RecordingManager] = None,\n ) -> None:\n \"\"\"\n Initialize the BaseAgent.\n \n Args:\n name: Unique name for the agent\n backend_scope: List of backend types this agent can access (e.g., [\"gui\", \"shell\", \"mcp\", \"web\", \"system\"])\n llm_client: LLM client for agent reasoning (optional, can be set later)\n grounding_client: Reference to GroundingClient for tool execution\n recording_manager: RecordingManager for recording execution\n \"\"\"\n self._name = name\n self._grounding_client: Optional[GroundingClient] = grounding_client\n self._backend_scope = backend_scope or []\n self._llm_client = llm_client\n self._recording_manager: Optional[RecordingManager] = recording_manager\n self._step = 0\n self._status = AgentStatus.ACTIVE\n \n self._register_self()\n logger.info(f\"Initialized {self.__class__.__name__}: {name}\")\n\n @property\n def name(self) -> str:\n return self._name\n \n @property\n def grounding_client(self) -> Optional[GroundingClient]:\n \"\"\"Get the grounding client.\"\"\"\n return self._grounding_client\n\n @property\n def backend_scope(self) -> List[str]:\n return self._backend_scope\n\n @property\n def llm_client(self) -> Optional[LLMClient]:\n return self._llm_client\n\n @llm_client.setter\n def llm_client(self, client: LLMClient) -> None:\n self._llm_client = client\n\n @property\n def recording_manager(self) -> Optional[RecordingManager]:\n \"\"\"Get the recording manager.\"\"\"\n return self._recording_manager\n\n @property\n def step(self) -> int:\n return self._step\n\n @property\n def status(self) -> str:\n return self._status\n\n @abstractmethod\n async def process(self, context: Dict[str, Any]) -> Dict[str, Any]:\n pass\n\n @abstractmethod\n def construct_messages(self, context: Dict[str, Any]) -> List[Dict[str, Any]]:\n \"\"\"\n Construct messages for LLM reasoning.\n Context must contain 'instruction' key.\n \"\"\"\n pass\n\n async def get_llm_response(\n self,\n messages: List[Dict[str, Any]],\n tools: Optional[List] = None,\n **kwargs\n ) -> Dict[str, Any]:\n if not self._llm_client:\n raise ValueError(f\"LLM client not initialized for agent {self.name}\")\n \n try:\n response = await self._llm_client.complete(\n messages=messages,\n tools=tools,\n **kwargs\n )\n return response\n except Exception as e:\n logger.error(f\"{self.name}: LLM call failed: {e}\", exc_info=True)\n raise\n\n def response_to_dict(self, response: str) -> Dict[str, Any]:\n try:\n if response.strip().startswith(\"```json\") or response.strip().startswith(\"```\"):\n lines = response.strip().split('\\n')\n if lines and lines[0].startswith('```'):\n lines = lines[1:]\n end_idx = len(lines)\n for i, line in enumerate(lines):\n if line.strip() == '```':\n end_idx = i\n break\n response = '\\n'.join(lines[:end_idx])\n \n return json.loads(response)\n except json.JSONDecodeError as e:\n # If parsing fails, try to find and extract just the JSON object/array\n if \"Extra data\" in str(e):\n try:\n decoder = json.JSONDecoder()\n obj, idx = decoder.raw_decode(response)\n logger.warning(\n f\"{self.name}: Successfully extracted JSON but found extra text after position {idx}. \"\n f\"Extra text: {response[idx:idx+100]}...\"\n )\n return obj\n except Exception as e2:\n logger.error(f\"{self.name}: Failed to extract JSON even with raw_decode: {e2}\")\n \n logger.error(f\"{self.name}: Failed to parse response: {e}\")\n logger.error(f\"{self.name}: Response content: {response[:500]}\")\n return {\"error\": \"Failed to parse response\", \"raw\": response}\n\n def increment_step(self) -> None:\n self._step += 1\n\n @classmethod\n def _register_self(cls) -> None:\n \"\"\"Register the agent class in the registry upon instantiation.\"\"\"\n # Get the actual instance class, not BaseAgent\n if cls.__name__ != \"BaseAgent\" and cls.__name__ not in AgentRegistry._registry:\n AgentRegistry.register(cls.__name__, cls)\n\n def __repr__(self) -> str:\n return f\"<{self.__class__.__name__}(name={self.name}, step={self.step}, status={self.status})>\"\n\n\nclass AgentStatus:\n \"\"\"Constants for agent status.\"\"\"\n ACTIVE = \"active\"\n IDLE = \"idle\"\n WAITING = \"waiting\"\n\n\nclass AgentRegistry:\n \"\"\"\n Registry for managing agent classes.\n Allows dynamic registration and retrieval of agent types.\n \"\"\"\n\n _registry: Dict[str, Type[BaseAgent]] = {}\n\n @classmethod\n def register(cls, name: str, agent_cls: Type[BaseAgent]) -> None:\n if name in cls._registry:\n logger.warning(f\"Agent class '{name}' already registered, overwriting\")\n cls._registry[name] = agent_cls\n logger.debug(f\"Registered agent class: {name}\")\n\n @classmethod\n def get_cls(cls, name: str) -> Type[BaseAgent]:\n if name not in cls._registry:\n raise ValueError(f\"No agent class registered under '{name}'\")\n return cls._registry[name]\n\n @classmethod\n def list_registered(cls) -> List[str]:\n return list(cls._registry.keys())\n\n @classmethod\n def clear(cls) -> None:\n cls._registry.clear()\n logger.debug(\"Agent registry cleared\")"
},
{
"path": "anytool/agents/grounding_agent.py",
"content": "from __future__ import annotations\n\nimport copy\nimport json\nfrom typing import TYPE_CHECKING, Any, Dict, List, Optional\n\nfrom anytool.agents.base import BaseAgent\nfrom anytool.grounding.core.types import BackendType, ToolResult\nfrom anytool.platform.screenshot import ScreenshotClient\nfrom anytool.prompts import GroundingAgentPrompts\nfrom anytool.utils.logging import Logger\n\nif TYPE_CHECKING:\n from anytool.llm import LLMClient\n from anytool.grounding.core.grounding_client import GroundingClient\n from anytool.recording import RecordingManager\n\nlogger = Logger.get_logger(__name__)\n\n\nclass GroundingAgent(BaseAgent):\n def __init__(\n self,\n name: str = \"GroundingAgent\",\n backend_scope: Optional[List[str]] = None,\n llm_client: Optional[LLMClient] = None,\n grounding_client: Optional[GroundingClient] = None,\n recording_manager: Optional[RecordingManager] = None,\n system_prompt: Optional[str] = None,\n max_iterations: int = 15,\n visual_analysis_timeout: float = 30.0,\n tool_retrieval_llm: Optional[LLMClient] = None,\n visual_analysis_model: Optional[str] = None,\n ) -> None:\n \"\"\"\n Initialize the Grounding Agent.\n \n Args:\n name: Agent name\n backend_scope: List of backends this agent can access (None = all available)\n llm_client: LLM client for reasoning\n grounding_client: GroundingClient for tool execution\n recording_manager: RecordingManager for recording execution\n system_prompt: Custom system prompt\n max_iterations: Maximum LLM reasoning iterations for self-correction\n visual_analysis_timeout: Timeout for visual analysis LLM calls in seconds\n tool_retrieval_llm: LLM client for tool retrieval filter (None = use llm_client)\n visual_analysis_model: Model name for visual analysis (None = use llm_client.model)\n \"\"\"\n super().__init__(\n name=name,\n backend_scope=backend_scope or [\"gui\", \"shell\", \"mcp\", \"web\", \"system\"],\n llm_client=llm_client,\n grounding_client=grounding_client,\n recording_manager=recording_manager\n )\n \n self._system_prompt = system_prompt or self._default_system_prompt()\n self._max_iterations = max_iterations\n self._visual_analysis_timeout = visual_analysis_timeout\n self._tool_retrieval_llm = tool_retrieval_llm\n self._visual_analysis_model = visual_analysis_model\n \n logger.info(f\"Grounding Agent initialized: {name}\")\n logger.info(f\"Backend scope: {self._backend_scope}\")\n logger.info(f\"Max iterations: {self._max_iterations}\")\n logger.info(f\"Visual analysis timeout: {self._visual_analysis_timeout}s\")\n if tool_retrieval_llm:\n logger.info(f\"Tool retrieval model: {tool_retrieval_llm.model}\")\n if visual_analysis_model:\n logger.info(f\"Visual analysis model: {visual_analysis_model}\")\n \n def _truncate_messages(\n self, \n messages: List[Dict[str, Any]], \n keep_recent: int = 8,\n max_tokens_estimate: int = 120000\n ) -> List[Dict[str, Any]]:\n if len(messages) <= keep_recent + 2: # +2 for system and initial user\n return messages\n \n total_text = json.dumps(messages, ensure_ascii=False)\n estimated_tokens = len(total_text) // 4\n \n if estimated_tokens < max_tokens_estimate:\n return messages\n \n logger.info(f\"Truncating message history: {len(messages)} messages, \"\n f\"~{estimated_tokens:,} tokens -> keeping recent {keep_recent} rounds\")\n \n system_messages = []\n user_instruction = None\n conversation_messages = []\n \n for msg in messages:\n role = msg.get(\"role\")\n if role == \"system\":\n system_messages.append(msg)\n elif role == \"user\" and user_instruction is None:\n user_instruction = msg\n else:\n conversation_messages.append(msg)\n \n recent_messages = conversation_messages[-(keep_recent * 2):] if conversation_messages else []\n \n truncated = system_messages.copy()\n if user_instruction:\n truncated.append(user_instruction)\n truncated.extend(recent_messages)\n \n logger.info(f\"After truncation: {len(truncated)} messages, \"\n f\"~{len(json.dumps(truncated, ensure_ascii=False))//4:,} tokens (estimated)\")\n \n return truncated\n \n async def process(self, context: Dict[str, Any]) -> Dict[str, Any]:\n \"\"\"\n Process a task execution request with multi-round iteration control.\n \"\"\"\n instruction = context.get(\"instruction\", \"\")\n if not instruction:\n logger.error(\"Grounding Agent: No instruction provided\")\n return {\"error\": \"No instruction provided\", \"status\": \"error\"}\n \n # Store current instruction for visual analysis context\n self._current_instruction = instruction\n \n logger.info(f\"Grounding Agent: Processing instruction at step {self.step}\")\n \n # Exist workspace files check\n workspace_info = await self._check_workspace_artifacts(context)\n if workspace_info[\"has_files\"]:\n context[\"workspace_artifacts\"] = workspace_info\n logger.info(f\"Workspace has {len(workspace_info['files'])} existing files: {workspace_info['files']}\")\n \n # Get available tools (auto-search with cap)\n tools = await self._get_available_tools(instruction)\n \n # Get search debug info (similarity scores, LLM selections)\n search_debug_info = None\n if self.grounding_client:\n search_debug_info = self.grounding_client.get_last_search_debug_info()\n \n # Build retrieved tools list for return value\n retrieved_tools_list = []\n for tool in tools:\n tool_info = {\n \"name\": getattr(tool, \"name\", str(tool)),\n \"description\": getattr(tool, \"description\", \"\"),\n }\n if hasattr(tool, \"backend_type\"):\n tool_info[\"backend\"] = tool.backend_type.value if hasattr(tool.backend_type, \"value\") else str(tool.backend_type)\n if hasattr(tool, \"_runtime_info\") and tool._runtime_info:\n tool_info[\"server_name\"] = tool._runtime_info.server_name\n \n # Add similarity score if available\n if search_debug_info and search_debug_info.get(\"tool_scores\"):\n for score_info in search_debug_info[\"tool_scores\"]:\n if score_info[\"name\"] == tool_info[\"name\"]:\n tool_info[\"similarity_score\"] = score_info[\"score\"]\n break\n \n retrieved_tools_list.append(tool_info)\n \n # Record retrieved tools\n if self._recording_manager:\n from anytool.recording import RecordingManager\n await RecordingManager.record_retrieved_tools(\n task_instruction=instruction,\n tools=tools,\n search_debug_info=search_debug_info,\n )\n \n # Initialize iteration state\n max_iterations = context.get(\"max_iterations\", self._max_iterations)\n current_iteration = 0\n all_tool_results = []\n iteration_contexts = []\n consecutive_empty_responses = 0 # Track consecutive empty LLM responses\n MAX_CONSECUTIVE_EMPTY = 5 # Exit after this many empty responses\n \n # Build initial messages\n messages = self.construct_messages(context)\n \n try:\n while current_iteration < max_iterations:\n current_iteration += 1\n logger.info(f\"Grounding Agent: Iteration {current_iteration}/{max_iterations}\")\n \n # Truncate message history to prevent context length issues\n # Start truncating after 5 iterations to keep context manageable\n if current_iteration >= 5:\n messages = self._truncate_messages(\n messages, \n keep_recent=8, # ไฟ็ๆ่ฟ8่ฝฎๅฏน่ฏ\n max_tokens_estimate=120000 # Claude Sonnet 4.5 ไธไธๆ้ๅถๆฏ200K๏ผไฟๅฎไฝฟ็จ120K\n )\n \n messages_input_snapshot = copy.deepcopy(messages)\n \n # [DISABLED] Iteration summary generation\n # Tool results (including visual analysis) are already in context,\n # LLM can make decisions directly without separate summary.\n # To re-enable, uncomment below and pass iteration_summary_prompt to complete()\n # iteration_summary_prompt = GroundingAgentPrompts.iteration_summary(\n # instruction=instruction,\n # iteration=current_iteration,\n # max_iterations=max_iterations\n # ) if context.get(\"auto_execute\", True) else None\n \n # Call LLMClient for single round\n # LLM will decide whether to call tools or finish with \n llm_response = await self._llm_client.complete(\n messages=messages,\n tools=tools if context.get(\"auto_execute\", True) else None,\n execute_tools=context.get(\"auto_execute\", True),\n summary_prompt=None, # Disabled\n tool_result_callback=self._visual_analysis_callback\n )\n \n # Update messages with LLM response\n messages = llm_response[\"messages\"]\n \n # Collect tool results\n tool_results_this_iteration = llm_response.get(\"tool_results\", [])\n if tool_results_this_iteration:\n all_tool_results.extend(tool_results_this_iteration)\n\n # [DISABLED] Iteration summary logging\n # llm_summary = llm_response.get(\"iteration_summary\")\n # if llm_summary:\n # logger.info(f\"Iteration {current_iteration} summary: {llm_summary[:150]}...\")\n \n assistant_message = llm_response.get(\"message\", {})\n assistant_content = assistant_message.get(\"content\", \"\")\n \n has_tool_calls = llm_response.get('has_tool_calls', False)\n logger.info(f\"Iteration {current_iteration} - Has tool calls: {has_tool_calls}, \"\n f\"Tool results: {len(tool_results_this_iteration)}, \"\n f\"Content length: {len(assistant_content)} chars\")\n \n if len(assistant_content) > 0:\n logger.info(f\"Iteration {current_iteration} - Assistant content preview: {repr(assistant_content[:300])}\")\n consecutive_empty_responses = 0 # Reset counter on valid response\n else:\n if not has_tool_calls:\n consecutive_empty_responses += 1\n logger.warning(f\"Iteration {current_iteration} - NO tool calls and NO content \"\n f\"(empty response {consecutive_empty_responses}/{MAX_CONSECUTIVE_EMPTY})\")\n \n if consecutive_empty_responses >= MAX_CONSECUTIVE_EMPTY:\n logger.error(f\"Exiting due to {MAX_CONSECUTIVE_EMPTY} consecutive empty LLM responses. \"\n \"This may indicate API issues, rate limiting, or context too long.\")\n break\n else:\n consecutive_empty_responses = 0 # Reset if we have tool calls\n \n # Snapshot messages after LLM call (accumulated context)\n messages_output_snapshot = copy.deepcopy(messages)\n \n # Record iteration context\n iteration_context = {\n \"iteration\": current_iteration,\n \"messages_input\": messages_input_snapshot,\n \"messages_output\": messages_output_snapshot,\n \"llm_response_summary\": {\n \"assistant_content\": assistant_content,\n \"has_tool_calls\": has_tool_calls,\n # \"iteration_summary\": llm_summary, # Disabled with iteration summary\n \"tool_calls_count\": len(tool_results_this_iteration),\n },\n }\n iteration_contexts.append(iteration_context)\n \n # Real-time save to conversations.jsonl\n from anytool.recording import RecordingManager\n await RecordingManager.record_iteration_context(\n iteration=current_iteration,\n messages_input=messages_input_snapshot,\n messages_output=messages_output_snapshot,\n llm_response_summary=iteration_context[\"llm_response_summary\"],\n )\n \n # Check for completion token in assistant content\n # [DISABLED] Also check in iteration summary when enabled\n # is_complete = (\n # GroundingAgentPrompts.TASK_COMPLETE in assistant_content or\n # (llm_summary and GroundingAgentPrompts.TASK_COMPLETE in llm_summary)\n # )\n is_complete = GroundingAgentPrompts.TASK_COMPLETE in assistant_content\n \n if is_complete:\n # Task is complete - LLM generated completion token\n logger.info(f\"Task completed at iteration {current_iteration} (found {GroundingAgentPrompts.TASK_COMPLETE})\")\n break\n \n else:\n # LLM didn't generate , continue to next iteration\n if tool_results_this_iteration:\n logger.debug(f\"Task in progress, LLM called {len(tool_results_this_iteration)} tools\")\n else:\n logger.debug(f\"Task in progress, LLM did not generate \")\n \n # Remove previous iteration guidance to avoid accumulation\n messages = [\n msg for msg in messages \n if not (msg.get(\"role\") == \"system\" and \"Iteration\" in msg.get(\"content\", \"\") and \"complete\" in msg.get(\"content\", \"\"))\n ]\n \n guidance_msg = {\n \"role\": \"system\",\n \"content\": f\"Iteration {current_iteration} complete. \"\n f\"Check if task is finished - if yes, output {GroundingAgentPrompts.TASK_COMPLETE}. \"\n f\"If not, continue with next action.\"\n }\n messages.append(guidance_msg)\n \n # [DISABLED] Full iteration feedback with summary\n # self._remove_previous_guidance(messages)\n # feedback_msg = self._build_iteration_feedback(\n # iteration=current_iteration,\n # llm_summary=llm_summary,\n # add_guidance=True\n # )\n # if feedback_msg:\n # messages.append(feedback_msg)\n # logger.debug(f\"Added iteration {current_iteration} feedback with guidance\")\n \n continue\n \n # Build final result\n result = await self._build_final_result(\n instruction=instruction,\n messages=messages,\n all_tool_results=all_tool_results,\n iterations=current_iteration,\n max_iterations=max_iterations,\n iteration_contexts=iteration_contexts,\n retrieved_tools_list=retrieved_tools_list,\n search_debug_info=search_debug_info,\n )\n \n # Record agent action to recording manager\n if self._recording_manager:\n await self._record_agent_execution(result, instruction)\n \n # Increment step\n self.increment_step()\n \n logger.info(f\"Grounding Agent: Execution completed with status: {result.get('status')}\")\n return result\n \n except Exception as e:\n logger.error(f\"Grounding Agent: Execution failed: {e}\")\n result = {\n \"error\": str(e),\n \"status\": \"error\",\n \"instruction\": instruction,\n \"iteration\": current_iteration\n }\n self.increment_step()\n return result\n \n def _default_system_prompt(self) -> str:\n \"\"\"Default system prompt for the grounding agent.\"\"\"\n return GroundingAgentPrompts.SYSTEM_PROMPT\n\n def construct_messages(\n self,\n context: Dict[str, Any]\n ) -> List[Dict[str, Any]]:\n messages = [{\"role\": \"system\", \"content\": self._system_prompt}]\n \n # Get instruction from context\n instruction = context.get(\"instruction\", \"\")\n if not instruction:\n raise ValueError(\"context must contain 'instruction' field\")\n \n # Add workspace directory\n workspace_dir = context.get(\"workspace_dir\")\n if workspace_dir:\n messages.append({\n \"role\": \"system\",\n \"content\": GroundingAgentPrompts.workspace_directory(workspace_dir)\n })\n \n # Add workspace artifacts information\n workspace_artifacts = context.get(\"workspace_artifacts\")\n if workspace_artifacts and workspace_artifacts.get(\"has_files\"):\n files = workspace_artifacts.get(\"files\", [])\n matching_files = workspace_artifacts.get(\"matching_files\", [])\n recent_files = workspace_artifacts.get(\"recent_files\", [])\n \n if matching_files:\n artifact_msg = GroundingAgentPrompts.workspace_matching_files(matching_files)\n elif len(recent_files) >= 2:\n artifact_msg = GroundingAgentPrompts.workspace_recent_files(\n total_files=len(files),\n recent_files=recent_files\n )\n else:\n artifact_msg = GroundingAgentPrompts.workspace_file_list(files)\n \n messages.append({\n \"role\": \"system\",\n \"content\": artifact_msg\n })\n \n # User instruction\n messages.append({\"role\": \"user\", \"content\": instruction})\n \n return messages\n\n async def _get_available_tools(self, task_description: Optional[str]) -> List:\n \"\"\"\n Retrieve tools with auto-search + cap to control prompt bloat.\n Falls back to returning all tools if search fails.\n \"\"\"\n grounding_client = self.grounding_client\n if not grounding_client:\n return []\n\n backends = [BackendType(name) for name in self._backend_scope]\n\n try:\n # Use dedicated tool retrieval LLM if configured, otherwise use main LLM\n retrieval_llm = self._tool_retrieval_llm or self._llm_client\n tools = await grounding_client.get_tools_with_auto_search(\n task_description=task_description,\n backend=backends,\n use_cache=True,\n llm_callable=retrieval_llm,\n )\n logger.info(\n f\"GroundingAgent selected {len(tools)} tools (auto-search) from {len(backends)} backends\"\n )\n return tools\n except Exception as e:\n logger.warning(f\"Auto-search tools failed, falling back to full list: {e}\")\n\n # Fallback: fetch all tools (previous behaviour)\n all_tools = []\n for backend_name in self._backend_scope:\n try:\n backend_type = BackendType(backend_name)\n tools = await grounding_client.list_tools(backend=backend_type)\n all_tools.extend(tools)\n logger.debug(f\"Retrieved {len(tools)} tools from backend: {backend_name}\")\n except Exception as e:\n logger.debug(f\"Could not get tools from {backend_name}: {e}\")\n\n logger.info(\n f\"GroundingAgent fallback retrieved {len(all_tools)} tools from {len(self._backend_scope)} backends\"\n )\n return all_tools\n\n async def _visual_analysis_callback(\n self,\n result: ToolResult,\n tool_name: str,\n tool_call: Dict,\n backend: str\n ) -> ToolResult:\n \"\"\"\n Callback for LLMClient to handle visual analysis after tool execution.\n \"\"\"\n # 1. Check if LLM requested to skip visual analysis\n skip_visual_analysis = False\n try:\n arguments = tool_call.function.arguments\n if isinstance(arguments, str):\n args = json.loads(arguments.strip() or \"{}\")\n else:\n args = arguments\n \n if isinstance(args, dict) and args.get(\"skip_visual_analysis\"):\n skip_visual_analysis = True\n logger.info(f\"Visual analysis skipped for {tool_name} (meta-parameter set by LLM)\")\n except Exception as e:\n logger.debug(f\"Could not parse tool arguments: {e}\")\n \n # 2. If skip requested, return original result\n if skip_visual_analysis:\n return result\n \n # 3. Check if this backend needs visual analysis\n if backend != \"gui\":\n return result\n \n # 4. Check if tool has visual data\n metadata = getattr(result, 'metadata', None)\n has_screenshots = metadata and (metadata.get(\"screenshot\") or metadata.get(\"screenshots\"))\n \n # 5. If no visual data, try to capture a screenshot\n if not has_screenshots:\n try:\n logger.info(f\"No visual data from {tool_name}, capturing screenshot...\")\n screenshot_client = ScreenshotClient()\n screenshot_bytes = await screenshot_client.capture()\n \n if screenshot_bytes:\n # Add screenshot to result metadata\n if metadata is None:\n result.metadata = {}\n metadata = result.metadata\n metadata[\"screenshot\"] = screenshot_bytes\n has_screenshots = True\n logger.info(f\"Screenshot captured for visual analysis\")\n else:\n logger.warning(\"Failed to capture screenshot\")\n except Exception as e:\n logger.warning(f\"Error capturing screenshot: {e}\")\n \n # 6. If still no screenshots, return original result\n if not has_screenshots:\n logger.debug(f\"No visual data available for {tool_name}\")\n return result\n \n # 7. Perform visual analysis\n return await self._enhance_result_with_visual_context(result, tool_name)\n \n async def _enhance_result_with_visual_context(\n self,\n result: ToolResult,\n tool_name: str\n ) -> ToolResult:\n \"\"\"\n Enhance tool result with visual analysis for grounding agent workflows.\n \"\"\"\n import asyncio\n import base64\n import litellm\n \n try:\n metadata = getattr(result, 'metadata', None)\n if not metadata:\n return result\n \n # Collect all screenshots\n screenshots_bytes = []\n \n # Check for multiple screenshots first\n if metadata.get(\"screenshots\"):\n screenshots_list = metadata[\"screenshots\"]\n if isinstance(screenshots_list, list):\n screenshots_bytes = [s for s in screenshots_list if s]\n # Fall back to single screenshot\n elif metadata.get(\"screenshot\"):\n screenshots_bytes = [metadata[\"screenshot\"]]\n \n if not screenshots_bytes:\n return result\n \n # Select key screenshots if there are too many\n selected_screenshots = self._select_key_screenshots(screenshots_bytes, max_count=3)\n \n # Convert to base64\n visual_b64_list = []\n for visual_data in selected_screenshots:\n if isinstance(visual_data, bytes):\n visual_b64_list.append(base64.b64encode(visual_data).decode('utf-8'))\n else:\n visual_b64_list.append(visual_data) # Already base64\n \n # Build prompt based on number of screenshots\n num_screenshots = len(visual_b64_list)\n \n prompt = GroundingAgentPrompts.visual_analysis(\n tool_name=tool_name,\n num_screenshots=num_screenshots,\n task_description=getattr(self, '_current_instruction', '')\n )\n\n # Build content with text prompt + all images\n content = [{\"type\": \"text\", \"text\": prompt}]\n for visual_b64 in visual_b64_list:\n content.append({\n \"type\": \"image_url\",\n \"image_url\": {\n \"url\": f\"data:image/png;base64,{visual_b64}\"\n }\n })\n\n # Use dedicated visual analysis model if configured, otherwise use main LLM model\n visual_model = self._visual_analysis_model or (self._llm_client.model if self._llm_client else \"openrouter/anthropic/claude-sonnet-4.5\")\n response = await asyncio.wait_for(\n litellm.acompletion(\n model=visual_model,\n messages=[{\n \"role\": \"user\",\n \"content\": content\n }],\n timeout=self._visual_analysis_timeout\n ),\n timeout=self._visual_analysis_timeout + 5\n )\n \n analysis = response.choices[0].message.content.strip()\n \n # Inject visual analysis into content\n original_content = result.content or \"(no text output)\"\n enhanced_content = f\"{original_content}\\n\\n**Visual content**: {analysis}\"\n \n # Create enhanced result\n enhanced_result = ToolResult(\n status=result.status,\n content=enhanced_content,\n error=result.error,\n metadata={**metadata, \"visual_analyzed\": True, \"visual_analysis\": analysis},\n execution_time=result.execution_time\n )\n \n logger.info(f\"Enhanced {tool_name} result with visual analysis ({num_screenshots} screenshot(s))\")\n return enhanced_result\n \n except asyncio.TimeoutError:\n logger.warning(f\"Visual analysis timed out for {tool_name}, returning original result\")\n return result\n except Exception as e:\n logger.warning(f\"Failed to analyze visual content for {tool_name}: {e}\")\n return result\n \n def _select_key_screenshots(\n self, \n screenshots: List[bytes], \n max_count: int = 3\n ) -> List[bytes]:\n \"\"\"\n Select key screenshots if there are too many.\n \"\"\"\n if len(screenshots) <= max_count:\n return screenshots\n \n selected_indices = set()\n \n # Always include last (final state)\n selected_indices.add(len(screenshots) - 1)\n \n # If room, include first (initial state)\n if max_count >= 2:\n selected_indices.add(0)\n \n # Fill remaining slots with evenly spaced middle screenshots\n remaining_slots = max_count - len(selected_indices)\n if remaining_slots > 0:\n # Calculate spacing\n available_indices = [\n i for i in range(1, len(screenshots) - 1)\n if i not in selected_indices\n ]\n \n if available_indices:\n step = max(1, len(available_indices) // (remaining_slots + 1))\n for i in range(remaining_slots):\n idx = min((i + 1) * step, len(available_indices) - 1)\n if idx < len(available_indices):\n selected_indices.add(available_indices[idx])\n \n # Return screenshots in original order\n selected = [screenshots[i] for i in sorted(selected_indices)]\n \n logger.debug(\n f\"Selected {len(selected)} screenshots at indices {sorted(selected_indices)} \"\n f\"from total of {len(screenshots)}\"\n )\n \n return selected\n\n def _get_workspace_path(self, context: Dict[str, Any]) -> Optional[str]:\n \"\"\"\n Get workspace directory path from context.\n \"\"\"\n return context.get(\"workspace_dir\")\n \n def _scan_workspace_files(\n self,\n workspace_path: str,\n recent_threshold: int = 600 # seconds\n ) -> Dict[str, Any]:\n \"\"\"\n Scan workspace directory and collect file information.\n \n Args:\n workspace_path: Path to workspace directory\n recent_threshold: Threshold in seconds for recent files\n \n Returns:\n Dictionary with file information:\n - files: List of all filenames\n - file_details: Dict mapping filename to file info (size, modified, age_seconds)\n - recent_files: List of recently modified filenames\n \"\"\"\n import os\n import time\n \n result = {\n \"files\": [],\n \"file_details\": {},\n \"recent_files\": []\n }\n \n if not workspace_path or not os.path.exists(workspace_path):\n return result\n \n # Recording system files to exclude from workspace scanning\n excluded_files = {\"metadata.json\", \"traj.jsonl\"}\n \n try:\n current_time = time.time()\n \n for filename in os.listdir(workspace_path):\n filepath = os.path.join(workspace_path, filename)\n if os.path.isfile(filepath) and filename not in excluded_files:\n result[\"files\"].append(filename)\n \n # Get file stats\n stat = os.stat(filepath)\n file_info = {\n \"size\": stat.st_size,\n \"modified\": stat.st_mtime,\n \"age_seconds\": current_time - stat.st_mtime\n }\n result[\"file_details\"][filename] = file_info\n \n # Track recently created/modified files\n if file_info[\"age_seconds\"] < recent_threshold:\n result[\"recent_files\"].append(filename)\n \n result[\"files\"] = sorted(result[\"files\"])\n \n except Exception as e:\n logger.debug(f\"Error scanning workspace files: {e}\")\n \n return result\n \n async def _check_workspace_artifacts(self, context: Dict[str, Any]) -> Dict[str, Any]:\n \"\"\"\n Check workspace directory for existing artifacts that might be relevant to the task.\n Enhanced to detect if task might already be completed.\n \"\"\"\n import re\n \n workspace_info = {\"has_files\": False, \"files\": [], \"file_details\": {}, \"recent_files\": []}\n \n try:\n # Get workspace path\n workspace_path = self._get_workspace_path(context)\n \n # Scan workspace files\n scan_result = self._scan_workspace_files(workspace_path, recent_threshold=600)\n \n if scan_result[\"files\"]:\n workspace_info[\"has_files\"] = True\n workspace_info[\"files\"] = scan_result[\"files\"]\n workspace_info[\"file_details\"] = scan_result[\"file_details\"]\n workspace_info[\"recent_files\"] = scan_result[\"recent_files\"]\n \n logger.info(f\"Grounding Agent: Found {len(scan_result['files'])} existing files in workspace \"\n f\"({len(scan_result['recent_files'])} recent)\")\n \n # Check if instruction mentions specific filenames\n instruction = context.get(\"instruction\", \"\")\n if instruction:\n # Look for potential file references in instruction\n potential_outputs = []\n # Match common file patterns: filename.ext, \"filename\", 'filename'\n file_patterns = re.findall(r'[\"\\']?([a-zA-Z0-9_\\-]+\\.[a-zA-Z0-9]+)[\"\\']?', instruction)\n for pattern in file_patterns:\n if pattern in scan_result[\"files\"]:\n potential_outputs.append(pattern)\n \n if potential_outputs:\n workspace_info[\"matching_files\"] = potential_outputs\n logger.info(f\"Grounding Agent: Found {len(potential_outputs)} files matching task: {potential_outputs}\")\n \n except Exception as e:\n logger.debug(f\"Could not check workspace artifacts: {e}\")\n \n return workspace_info\n \n def _build_iteration_feedback(\n self,\n iteration: int,\n llm_summary: Optional[str] = None,\n add_guidance: bool = True\n ) -> Optional[Dict[str, str]]:\n \"\"\"\n Build feedback message to add to next iteration.\n \"\"\"\n if not llm_summary:\n return None\n \n feedback_content = GroundingAgentPrompts.iteration_feedback(\n iteration=iteration,\n llm_summary=llm_summary,\n add_guidance=add_guidance\n )\n \n return {\n \"role\": \"system\",\n \"content\": feedback_content\n }\n \n def _remove_previous_guidance(self, messages: List[Dict[str, Any]]) -> None:\n \"\"\"\n Remove guidance section from previous iteration feedback messages.\n \"\"\"\n for msg in messages:\n if msg.get(\"role\") == \"system\":\n content = msg.get(\"content\", \"\")\n # Check if this is an iteration feedback message with guidance\n if \"## Iteration\" in content and \"Summary\" in content and \"---\" in content:\n # Remove everything from \"---\" onwards (the guidance part)\n summary_only = content.split(\"---\")[0].strip()\n msg[\"content\"] = summary_only\n\n async def _generate_final_summary(\n self,\n instruction: str,\n messages: List[Dict],\n iterations: int\n ) -> tuple[str, bool, List[Dict]]:\n \"\"\"\n Generate final summary across all iterations for reporting to upper layer.\n \n Returns:\n tuple[str, bool, List[Dict]]: (summary_text, success_flag, context_used)\n - summary_text: The generated summary or error message\n - success_flag: True if summary was generated successfully, False otherwise\n - context_used: The cleaned messages used for generating summary\n \"\"\"\n final_summary_prompt = {\n \"role\": \"user\",\n \"content\": GroundingAgentPrompts.final_summary(\n instruction=instruction,\n iterations=iterations\n )\n }\n \n clean_messages = []\n for msg in messages:\n # Skip tool result messages\n if msg.get(\"role\") == \"tool\":\n continue\n # Copy message and remove tool_calls if present\n clean_msg = msg.copy()\n if \"tool_calls\" in clean_msg:\n del clean_msg[\"tool_calls\"]\n clean_messages.append(clean_msg)\n \n clean_messages.append(final_summary_prompt)\n \n # Save context for return\n context_for_return = copy.deepcopy(clean_messages)\n \n try:\n # Call LLMClient to generate final summary (without tools)\n summary_response = await self._llm_client.complete(\n messages=clean_messages,\n tools=None,\n execute_tools=False\n )\n \n final_summary = summary_response.get(\"message\", {}).get(\"content\", \"\")\n \n if final_summary:\n logger.info(f\"Generated final summary: {final_summary[:200]}...\")\n return final_summary, True, context_for_return\n else:\n logger.warning(\"LLM returned empty final summary\")\n return f\"Task completed after {iterations} iteration(s). Check execution history for details.\", True, context_for_return\n \n except Exception as e:\n logger.error(f\"Error generating final summary: {e}\")\n return f\"Task completed after {iterations} iteration(s), but failed to generate summary: {str(e)}\", False, context_for_return\n \n\n async def _build_final_result(\n self,\n instruction: str,\n messages: List[Dict],\n all_tool_results: List[Dict],\n iterations: int,\n max_iterations: int,\n iteration_contexts: List[Dict] = None,\n retrieved_tools_list: List[Dict] = None,\n search_debug_info: Dict[str, Any] = None,\n ) -> Dict[str, Any]:\n \"\"\"\n Build final execution result.\n \n Args:\n instruction: Original instruction\n messages: Complete conversation history (including all iteration summaries)\n all_tool_results: All tool execution results\n iterations: Number of iterations performed\n max_iterations: Maximum allowed iterations\n iteration_contexts: Context snapshots for each iteration\n retrieved_tools_list: List of tools retrieved for this task\n search_debug_info: Debug info from tool search (similarity scores, LLM selections)\n \"\"\"\n is_complete = self._check_task_completion(messages)\n \n tool_executions = self._format_tool_executions(all_tool_results)\n \n result = {\n \"instruction\": instruction,\n \"step\": self.step,\n \"iterations\": iterations,\n \"tool_executions\": tool_executions,\n \"messages\": messages,\n \"iteration_contexts\": iteration_contexts or [],\n \"retrieved_tools_list\": retrieved_tools_list or [],\n \"search_debug_info\": search_debug_info,\n \"keep_session\": True\n }\n \n if is_complete:\n logger.info(\"Task completed with marker\")\n # Use LLM's own completion response directly (no extra LLM call needed)\n # LLM already generates a summary before outputting \n last_response = self._extract_last_assistant_message(messages)\n # Remove the token from response for cleaner output\n result[\"response\"] = last_response.replace(GroundingAgentPrompts.TASK_COMPLETE, \"\").strip()\n result[\"status\"] = \"success\"\n \n # [DISABLED] Extra LLM call to generate final summary\n # final_summary, summary_success, final_summary_context = await self._generate_final_summary(\n # instruction=instruction,\n # messages=messages,\n # iterations=iterations\n # )\n # result[\"response\"] = final_summary\n # result[\"final_summary_context\"] = final_summary_context\n else:\n result[\"response\"] = self._extract_last_assistant_message(messages)\n result[\"status\"] = \"incomplete\"\n result[\"warning\"] = (\n f\"Task reached max iterations ({max_iterations}) without completion. \"\n f\"This may indicate the task needs more steps or clarification.\"\n )\n \n return result\n \n def _format_tool_executions(self, all_tool_results: List[Dict]) -> List[Dict]:\n executions = []\n for tr in all_tool_results:\n tool_result_obj = tr.get(\"result\")\n tool_call = tr.get(\"tool_call\")\n \n status = \"unknown\"\n if hasattr(tool_result_obj, 'status'):\n status_obj = tool_result_obj.status\n status = getattr(status_obj, 'value', status_obj)\n \n # Extract tool_name and arguments from tool_call object (litellm format)\n tool_name = \"unknown\"\n arguments = {}\n if tool_call is not None:\n if hasattr(tool_call, 'function'):\n # tool_call is an object with .function attribute\n tool_name = getattr(tool_call.function, 'name', 'unknown')\n args_raw = getattr(tool_call.function, 'arguments', '{}')\n if isinstance(args_raw, str):\n try:\n arguments = json.loads(args_raw) if args_raw.strip() else {}\n except json.JSONDecodeError:\n arguments = {}\n else:\n arguments = args_raw if isinstance(args_raw, dict) else {}\n elif isinstance(tool_call, dict):\n # Fallback: tool_call is a dict\n func = tool_call.get(\"function\", {})\n tool_name = func.get(\"name\", \"unknown\")\n args_raw = func.get(\"arguments\", \"{}\")\n if isinstance(args_raw, str):\n try:\n arguments = json.loads(args_raw) if args_raw.strip() else {}\n except json.JSONDecodeError:\n arguments = {}\n else:\n arguments = args_raw if isinstance(args_raw, dict) else {}\n \n executions.append({\n \"tool_name\": tool_name,\n \"arguments\": arguments,\n \"backend\": tr.get(\"backend\"),\n \"server_name\": tr.get(\"server_name\"),\n \"status\": status,\n \"content\": tool_result_obj.content if hasattr(tool_result_obj, 'content') else None,\n \"error\": tool_result_obj.error if hasattr(tool_result_obj, 'error') else None,\n \"execution_time\": tool_result_obj.execution_time if hasattr(tool_result_obj, 'execution_time') else None,\n \"metadata\": tool_result_obj.metadata if hasattr(tool_result_obj, 'metadata') else {},\n })\n return executions\n \n def _check_task_completion(self, messages: List[Dict]) -> bool:\n for msg in reversed(messages):\n if msg.get(\"role\") == \"assistant\":\n content = msg.get(\"content\", \"\")\n return GroundingAgentPrompts.TASK_COMPLETE in content\n return False\n \n def _extract_last_assistant_message(self, messages: List[Dict]) -> str:\n for msg in reversed(messages):\n if msg.get(\"role\") == \"assistant\":\n return msg.get(\"content\", \"\")\n return \"\"\n \n async def _record_agent_execution(\n self,\n result: Dict[str, Any],\n instruction: str\n ) -> None:\n \"\"\"\n Record agent execution to recording manager.\n \n Args:\n result: Execution result\n instruction: Original instruction\n \"\"\"\n if not self._recording_manager:\n return\n \n # Extract tool execution summary\n tool_summary = []\n if result.get(\"tool_executions\"):\n for exec_info in result[\"tool_executions\"]:\n tool_summary.append({\n \"tool\": exec_info.get(\"tool_name\", \"unknown\"),\n \"backend\": exec_info.get(\"backend\", \"unknown\"),\n \"status\": exec_info.get(\"status\", \"unknown\"),\n })\n \n await self._recording_manager.record_agent_action(\n agent_name=self.name,\n action_type=\"execute\",\n input_data={\"instruction\": instruction},\n reasoning={\n \"response\": result.get(\"response\", \"\"),\n \"tools_selected\": tool_summary,\n },\n output_data={\n \"status\": result.get(\"status\", \"unknown\"),\n \"iterations\": result.get(\"iterations\", 0),\n \"num_tool_executions\": len(result.get(\"tool_executions\", [])),\n },\n metadata={\n \"step\": self.step,\n \"instruction\": instruction,\n }\n )"
},
{
"path": "anytool/config/__init__.py",
"content": "from .grounding import *\nfrom .loader import *\nfrom .constants import * \nfrom .utils import *\nfrom . import constants\n\n__all__ = [\n # Grounding Config\n \"BackendConfig\",\n \"ShellConfig\",\n \"WebConfig\",\n \"MCPConfig\",\n \"GUIConfig\",\n \"ToolSearchConfig\",\n \"SessionConfig\",\n \"SecurityPolicy\",\n \"GroundingConfig\",\n \n # Loader\n \"CONFIG_DIR\",\n \"load_config\",\n \"get_config\",\n \"reset_config\",\n \"save_config\",\n \"load_agents_config\",\n \"get_agent_config\",\n \n # Utils\n \"get_config_value\",\n \"load_json_file\",\n \"save_json_file\",\n] + constants.__all__"
},
{
"path": "anytool/config/config_agents.json",
"content": "{\n \"agents\": [\n {\n \"name\": \"GroundingAgent\",\n \"class_name\": \"GroundingAgent\",\n \"backend_scope\": [\"gui\", \"shell\", \"mcp\", \"system\", \"web\"],\n \"max_iterations\": 15,\n \"visual_analysis_timeout\": 60.0\n }\n ]\n}"
},
{
"path": "anytool/config/config_dev.json.example",
"content": "{\n \"comment\": \"[Optional] Loading grounding.json โ security.json โ dev.json (dev.json overrides the former ones)\",\n \n \"debug\": true,\n \"log_level\": \"DEBUG\",\n \n \"security_policies\": {\n \"global\": {\n \"blocked_commands\": []\n }\n }\n}"
},
{
"path": "anytool/config/config_grounding.json",
"content": "{\n \"shell\": {\n \"mode\": \"local\",\n \"timeout\": 60,\n \"max_retries\": 3,\n \"retry_interval\": 3.0,\n \"default_shell\": \"/bin/bash\",\n \"working_dir\": null,\n \"env\": {},\n \"conda_env\": null,\n \"default_port\": 5000\n },\n \"mcp\": {\n \"timeout\": 30,\n \"max_retries\": 3,\n \"retry_interval\": 2.0,\n \"sandbox\": false,\n \"auto_initialize\": true,\n \"eager_sessions\": false,\n \"sse_read_timeout\": 300.0,\n \"check_dependencies\": true,\n \"auto_install\": true\n },\n \"gui\": {\n \"mode\": \"local\",\n \"timeout\": 90,\n \"max_retries\": 3,\n \"retry_interval\": 5.0,\n \"driver_type\": \"pyautogui\",\n \"failsafe\": false,\n \"screenshot_on_error\": true,\n \"pkgs_prefix\": \"import pyautogui; import time; pyautogui.FAILSAFE = {failsafe}; {command}\"\n },\n \"tool_search\": {\n \"embedding_model\": \"BAAI/bge-small-en-v1.5\",\n \"max_tools\": 40,\n \"search_mode\": \"hybrid\",\n \"enable_llm_filter\": true,\n \"llm_filter_threshold\": 50,\n \"enable_cache_persistence\": true,\n \"cache_dir\": null\n },\n \"tool_quality\": {\n \"enabled\": true,\n \"enable_persistence\": true,\n \"cache_dir\": null,\n \"auto_evaluate_descriptions\": true,\n \"enable_quality_ranking\": true,\n \"evolve_interval\": 5\n },\n \n \"tool_cache_ttl\": 600,\n \"tool_cache_maxsize\": 500,\n\n \"debug\": false,\n \"log_level\": \"INFO\",\n \"enabled_backends\": [\n {\n \"name\": \"shell\",\n \"provider_cls\": \"anytool.grounding.backends.shell.ShellProvider\"\n },\n {\n \"name\": \"web\",\n \"provider_cls\": \"anytool.grounding.backends.web.WebProvider\"\n },\n {\n \"name\": \"mcp\",\n \"provider_cls\": \"anytool.grounding.backends.mcp.MCPProvider\"\n },\n {\n \"name\": \"gui\",\n \"provider_cls\": \"anytool.grounding.backends.gui.GUIProvider\"\n }\n ],\n \n \"_comment_system_backend\": \"Note: 'system' backend is automatically registered and always available. It provides meta-level tools for querying system state. Do not add it to enabled_backends as it requires special initialization.\"\n}"
},
{
"path": "anytool/config/config_mcp.json.example",
"content": ""
},
{
"path": "anytool/config/config_security.json",
"content": "{\n \"security_policies\": {\n \"global\": {\n \"allow_shell_commands\": true,\n \"allow_network_access\": true,\n \"allow_file_access\": true,\n \"blocked_commands\": {\n \"common\": [\"rm\", \"-rf\", \"shutdown\", \"reboot\", \"poweroff\", \"halt\"],\n \"linux\": [\"mkfs\", \"dd\", \"iptables\", \"systemctl\", \"init\", \"kill\", \"-9\", \"pkill\"],\n \"darwin\": [\"diskutil\", \"dd\", \"pfctl\", \"launchctl\", \"killall\"],\n \"windows\": [\"del\", \"format\", \"rd\", \"rmdir\", \"/s\", \"/q\", \"taskkill\", \"/f\"]\n },\n \"sandbox_enabled\": false\n },\n \"backend\": {\n \"shell\": {\n \"allow_shell_commands\": true,\n \"allow_file_access\": true,\n \"blocked_commands\": {\n \"common\": [\"rm\", \"-rf\", \"shutdown\", \"reboot\", \"poweroff\", \"halt\"],\n \"linux\": [\n \"mkfs\", \"mkfs.ext4\", \"mkfs.xfs\",\n \"dd\",\n \"iptables\", \"ip6tables\", \"nftables\",\n \"systemctl\", \"service\",\n \"fdisk\", \"parted\", \"gdisk\",\n \"mount\", \"umount\",\n \"chmod\", \"777\",\n \"chown\", \"root\",\n \"passwd\",\n \"useradd\", \"userdel\", \"usermod\",\n \"kill\", \"-9\", \"pkill\", \"killall\"\n ],\n \"darwin\": [\n \"diskutil\",\n \"dd\",\n \"pfctl\",\n \"launchctl\",\n \"dscl\",\n \"chmod\", \"777\",\n \"chown\", \"root\",\n \"passwd\",\n \"killall\",\n \"pmset\"\n ],\n \"windows\": [\n \"del\", \"erase\",\n \"format\",\n \"rd\", \"rmdir\", \"/s\", \"/q\",\n \"diskpart\",\n \"reg\", \"delete\",\n \"net\", \"user\",\n \"taskkill\", \"/f\",\n \"wmic\"\n ]\n },\n \"sandbox_enabled\": false\n },\n \"mcp\": {\n \"sandbox_enabled\": false\n },\n \"web\": {\n \"allow_network_access\": true,\n \"allowed_domains\": []\n }\n }\n }\n}"
},
{
"path": "anytool/config/constants.py",
"content": "from pathlib import Path\n\nCONFIG_GROUNDING = \"config_grounding.json\"\nCONFIG_SECURITY = \"config_security.json\"\nCONFIG_MCP = \"config_mcp.json\"\nCONFIG_DEV = \"config_dev.json\"\nCONFIG_AGENTS = \"config_agents.json\"\n\nLOG_LEVELS = [\"DEBUG\", \"INFO\", \"WARNING\", \"ERROR\", \"CRITICAL\"]\n\n# Project root directory (AnyTool/)\nPROJECT_ROOT = Path(__file__).parent.parent.parent\n\n\n__all__ = [\n \"CONFIG_GROUNDING\",\n \"CONFIG_SECURITY\",\n \"CONFIG_MCP\",\n \"CONFIG_DEV\",\n \"CONFIG_AGENTS\",\n \"LOG_LEVELS\",\n \"PROJECT_ROOT\",\n]"
},
{
"path": "anytool/config/grounding.py",
"content": "from typing import Dict, Optional, Any, List, Literal\ntry:\n from pydantic import BaseModel, Field, field_validator\n PYDANTIC_V2 = True\nexcept ImportError:\n from pydantic import BaseModel, Field, validator as field_validator\n PYDANTIC_V2 = False\n\nfrom anytool.grounding.core.types import (\n SessionConfig, \n SecurityPolicy,\n BackendType\n)\nfrom .constants import LOG_LEVELS\n\n\nclass ConfigMixin:\n \"\"\"Mixin to add utility methods for config access\"\"\"\n \n def get_value(self, key: str, default=None):\n \"\"\"\n Safely get config value, works with both dict and Pydantic models.\n \n Args:\n key: Configuration key\n default: Default value if key not found\n \"\"\"\n if isinstance(self, dict):\n return self.get(key, default)\n else:\n return getattr(self, key, default)\n\n\nclass BackendConfig(BaseModel, ConfigMixin):\n \"\"\"Base backend configuration\"\"\"\n enabled: bool = Field(True, description=\"Whether the backend is enabled\")\n timeout: int = Field(30, ge=1, le=300, description=\"Timeout in seconds\")\n max_retries: int = Field(3, ge=0, le=10, description=\"Maximum retry attempts\")\n\n\nclass ShellConfig(BackendConfig):\n \"\"\"\n Shell backend configuration\n \n Attributes:\n enabled: Whether shell backend is enabled\n mode: Execution mode - \"local\" runs scripts in-process via subprocess,\n \"server\" connects to a running local_server via HTTP\n timeout: Default timeout for shell operations (seconds)\n max_retries: Maximum number of retry attempts for failed operations\n retry_interval: Wait time between retries (seconds)\n default_shell: Path to default shell executable\n working_dir: Default working directory for bash scripts\n env: Default environment variables for shell operations\n conda_env: Conda environment name to activate before execution (optional)\n default_port: Default port for shell server connection (only used in server mode)\n \"\"\"\n mode: Literal[\"local\", \"server\"] = Field(\"local\", description=\"Execution mode: 'local' (in-process subprocess) or 'server' (HTTP local_server)\")\n retry_interval: float = Field(3.0, ge=0.1, le=60.0, description=\"Wait time between retries in seconds\")\n default_shell: str = Field(\"/bin/bash\", description=\"Default shell path\")\n working_dir: Optional[str] = Field(None, description=\"Default working directory for bash scripts\")\n env: Dict[str, str] = Field(default_factory=dict, description=\"Default environment variables\")\n conda_env: Optional[str] = Field(None, description=\"Conda environment name to activate (e.g., 'myenv')\")\n default_port: int = Field(5000, ge=1, le=65535, description=\"Default port for shell server\")\n \n @field_validator('default_shell')\n @classmethod\n def validate_shell(cls, v):\n if not v or not isinstance(v, str):\n raise ValueError(\"Shell path must be a non-empty string\")\n return v\n \n @field_validator('working_dir')\n @classmethod\n def validate_working_dir(cls, v):\n if v is not None and not isinstance(v, str):\n raise ValueError(\"Working directory must be a string\")\n return v\n\nclass WebConfig(BackendConfig):\n \"\"\"\n Web backend configuration - AI Deep Research\n \n Attributes:\n enabled: Whether web backend is enabled\n timeout: Default timeout for web operations (seconds)\n max_retries: Maximum number of retry attempts\n \n Note:\n All web-specific parameters (API key, base URL) are loaded from \n environment variables or use default values in WebSession:\n - OPENROUTER_API_KEY: API key for deep research (required)\n - Deep research base URL defaults to \"https://openrouter.ai/api/v1\"\n \"\"\"\n pass\n\n\nclass MCPConfig(BackendConfig):\n \"\"\"MCP backend configuration\"\"\"\n sandbox: bool = Field(False, description=\"Whether to enable sandbox\")\n auto_initialize: bool = Field(True, description=\"Whether to auto initialize\")\n eager_sessions: bool = Field(False, description=\"Whether to eagerly create sessions for all servers on initialization\")\n retry_interval: float = Field(2.0, ge=0.1, le=60.0, description=\"Wait time between retries in seconds\")\n servers: Dict[str, Dict[str, Any]] = Field(default_factory=dict, description=\"MCP servers configuration, loaded from config_mcp.json\")\n sse_read_timeout: float = Field(300.0, ge=1.0, le=3600.0, description=\"SSE read timeout in seconds for HTTP/Sandbox connectors\")\n\n\nclass GUIConfig(BackendConfig):\n \"\"\"\n GUI backend configuration\n \n Attributes:\n mode: Execution mode - \"local\" runs GUI operations in-process,\n \"server\" connects to a running local_server via HTTP\n \"\"\"\n mode: Literal[\"local\", \"server\"] = Field(\"local\", description=\"Execution mode: 'local' (in-process) or 'server' (HTTP local_server)\")\n retry_interval: float = Field(5.0, ge=0.1, le=60.0, description=\"Wait time between retries in seconds\")\n driver_type: str = Field(\"pyautogui\", description=\"GUI driver type\")\n failsafe: bool = Field(False, description=\"Whether to enable pyautogui failsafe mode\")\n screenshot_on_error: bool = Field(True, description=\"Whether to capture screenshot on error\")\n pkgs_prefix: str = Field(\n \"import pyautogui; import time; pyautogui.FAILSAFE = {failsafe}; {command}\",\n description=\"Python command prefix for pyautogui setup\"\n )\n\n\nclass ToolSearchConfig(BaseModel):\n \"\"\"Tool search and ranking configuration\"\"\"\n embedding_model: str = Field(\n \"BAAI/bge-small-en-v1.5\",\n description=\"Embedding model name for semantic search\"\n )\n max_tools: int = Field(\n 20,\n ge=1,\n le=1000,\n description=\"Maximum number of tools to return from search\"\n )\n search_mode: str = Field(\n \"hybrid\",\n description=\"Default search mode: semantic, keyword, or hybrid\"\n )\n enable_llm_filter: bool = Field(\n True,\n description=\"Whether to use LLM for backend/server filtering\"\n )\n llm_filter_threshold: int = Field(\n 50,\n ge=1,\n le=1000,\n description=\"Only apply LLM filter when tool count exceeds this threshold\"\n )\n enable_cache_persistence: bool = Field(\n False,\n description=\"Whether to persist embeddings to disk\"\n )\n cache_dir: Optional[str] = Field(\n None,\n description=\"Directory for embedding cache. None means use default /.anytool/embedding_cache\"\n )\n \n @field_validator('search_mode')\n @classmethod\n def validate_search_mode(cls, v):\n valid_modes = ['semantic', 'keyword', 'hybrid']\n if v.lower() not in valid_modes:\n raise ValueError(f\"Search mode must be one of {valid_modes}, got: {v}\")\n return v.lower()\n\n\nclass ToolQualityConfig(BaseModel):\n \"\"\"Tool quality tracking configuration\"\"\"\n enabled: bool = Field(\n True,\n description=\"Whether to enable tool quality tracking\"\n )\n enable_persistence: bool = Field(\n True,\n description=\"Whether to persist quality data to disk\"\n )\n cache_dir: Optional[str] = Field(\n None,\n description=\"Directory for quality cache. None means use default /.anytool/tool_quality\"\n )\n auto_evaluate_descriptions: bool = Field(\n True,\n description=\"Whether to automatically evaluate tool descriptions using LLM\"\n )\n enable_quality_ranking: bool = Field(\n True,\n description=\"Whether to incorporate quality scores in tool ranking\"\n )\n evolve_interval: int = Field(\n 5,\n ge=1,\n le=100,\n description=\"Trigger quality evolution every N tool executions\"\n )\n\n\nclass GroundingConfig(BaseModel):\n \"\"\"\n Main configuration for Grounding module.\n \n Contains configuration for all grounding backends and grounding-level settings.\n Note: Local server connection uses defaults or environment variables (LOCAL_SERVER_URL).\n \"\"\"\n # Backend configurations\n shell: ShellConfig = Field(default_factory=ShellConfig)\n web: WebConfig = Field(default_factory=WebConfig)\n mcp: MCPConfig = Field(default_factory=MCPConfig)\n gui: GUIConfig = Field(default_factory=GUIConfig)\n system: BackendConfig = Field(default_factory=BackendConfig)\n \n # Grounding-level settings\n tool_search: ToolSearchConfig = Field(default_factory=ToolSearchConfig)\n tool_quality: ToolQualityConfig = Field(default_factory=ToolQualityConfig)\n \n enabled_backends: List[Dict[str, str]] = Field(\n default_factory=list,\n description=\"List of enabled backends, each item: {'name': str, 'provider_cls': str}\"\n )\n \n session_defaults: SessionConfig = Field(\n default_factory=lambda: SessionConfig(\n session_name=\"\",\n backend_type=BackendType.SHELL,\n timeout=30,\n auto_reconnect=True,\n health_check_interval=30\n )\n )\n \n tool_cache_ttl: int = Field(\n 300,\n ge=1,\n le=3600,\n description=\"Tool cache time-to-live in seconds\"\n )\n tool_cache_maxsize: int = Field(\n 300,\n ge=1,\n le=10000,\n description=\"Maximum number of tool cache entries\"\n )\n \n debug: bool = Field(False, description=\"Debug mode\")\n log_level: str = Field(\"INFO\", description=\"Log level\")\n security_policies: Dict[str, Any] = Field(default_factory=dict)\n \n @field_validator('log_level')\n @classmethod\n def validate_log_level(cls, v):\n if v.upper() not in LOG_LEVELS:\n raise ValueError(f\"Log level must be one of {LOG_LEVELS}, got: {v}\")\n return v.upper()\n \n def get_backend_config(self, backend_type: str) -> BackendConfig:\n \"\"\"Get configuration for specified backend\"\"\"\n name = backend_type.lower()\n if not hasattr(self, name):\n from anytool.utils.logging import Logger\n logger = Logger.get_logger(__name__)\n logger.warning(f\"Unknown backend type: {backend_type}\")\n return BackendConfig()\n return getattr(self, name)\n \n def get_security_policy(self, backend_type: str) -> SecurityPolicy:\n global_policy = self.security_policies.get(\"global\", {})\n backend_policy = self.security_policies.get(\"backend\", {}).get(backend_type.lower(), {})\n merged_policy = {**global_policy, **backend_policy}\n return SecurityPolicy.from_dict(merged_policy)\n\n\n__all__ = [\n \"BackendConfig\",\n \"ShellConfig\",\n \"WebConfig\",\n \"MCPConfig\",\n \"GUIConfig\",\n \"ToolSearchConfig\",\n \"ToolQualityConfig\",\n \"GroundingConfig\",\n]"
},
{
"path": "anytool/config/loader.py",
"content": "import threading\nfrom pathlib import Path\nfrom typing import Union, Iterable, Dict, Any, Optional\n\nfrom .grounding import GroundingConfig\nfrom .constants import (\n CONFIG_GROUNDING,\n CONFIG_SECURITY,\n CONFIG_DEV,\n CONFIG_MCP,\n CONFIG_AGENTS\n)\nfrom anytool.utils.logging import Logger\nfrom .utils import load_json_file, save_json_file as save_json\n\nlogger = Logger.get_logger(__name__)\n\n\nCONFIG_DIR = Path(__file__).parent\n\n# Global configuration singleton\n_config: GroundingConfig | None = None\n_config_lock = threading.RLock() # Use RLock to support recursive locking\n\n\ndef _deep_merge_dict(base: dict, update: dict) -> dict:\n \"\"\"Deep merge two dictionaries, update's values will override base's values\"\"\"\n result = base.copy()\n for key, value in update.items():\n if key in result and isinstance(result[key], dict) and isinstance(value, dict):\n result[key] = _deep_merge_dict(result[key], value)\n else:\n result[key] = value\n return result\n\ndef _load_json_file(path: Path) -> Dict[str, Any]:\n \"\"\"Load single JSON configuration file.\n \n This function wraps the generic load_json_file and adds global configuration specific error handling and logging.\n \"\"\"\n if not path.exists():\n logger.debug(f\"Configuration file does not exist, skipping: {path}\")\n return {}\n \n try:\n data = load_json_file(path)\n logger.info(f\"Loaded configuration file: {path}\")\n return data\n except Exception as e:\n logger.warning(f\"Failed to load configuration file {path}: {e}\")\n return {}\n\ndef _load_multiple_files(paths: Iterable[Path]) -> Dict[str, Any]:\n \"\"\"Load configuration from multiple files\"\"\"\n merged = {}\n for path in paths:\n data = _load_json_file(path)\n if data:\n merged = _deep_merge_dict(merged, data)\n return merged\n\ndef load_config(*config_paths: Union[str, Path]) -> GroundingConfig:\n \"\"\"\n Load configuration files\n \"\"\"\n global _config\n \n with _config_lock:\n if config_paths:\n paths = [Path(p) for p in config_paths]\n else:\n paths = [\n CONFIG_DIR / CONFIG_GROUNDING,\n CONFIG_DIR / CONFIG_SECURITY,\n CONFIG_DIR / CONFIG_DEV, # Optional: development environment configuration\n ]\n \n # Load and merge configuration\n raw_data = _load_multiple_files(paths)\n \n # Load MCP configuration (separate processing)\n # Check if mcpServers already provided in merged custom configs\n has_custom_mcp_servers = \"mcpServers\" in raw_data\n \n if has_custom_mcp_servers:\n # Use mcpServers from custom config\n if \"mcp\" not in raw_data:\n raw_data[\"mcp\"] = {}\n raw_data[\"mcp\"][\"servers\"] = raw_data.pop(\"mcpServers\")\n logger.debug(f\"Using custom MCP servers from provided config ({len(raw_data['mcp']['servers'])} servers)\")\n else:\n # Load default MCP servers from config_mcp.json\n mcp_data = _load_json_file(CONFIG_DIR / CONFIG_MCP)\n if mcp_data and \"mcpServers\" in mcp_data:\n if \"mcp\" not in raw_data:\n raw_data[\"mcp\"] = {}\n raw_data[\"mcp\"][\"servers\"] = mcp_data[\"mcpServers\"]\n logger.debug(f\"Loaded MCP servers from default config_mcp.json ({len(raw_data['mcp']['servers'])} servers)\")\n \n # Validate and create configuration object\n try:\n _config = GroundingConfig.model_validate(raw_data)\n except Exception as e:\n logger.error(f\"Validation failed, using default configuration: {e}\")\n _config = GroundingConfig()\n \n # Adjust log level according to configuration\n if _config.debug:\n Logger.set_debug(2)\n elif _config.log_level:\n try:\n Logger.configure(level=_config.log_level)\n except Exception as e:\n logger.warning(f\"Failed to set log level {_config.log_level}: {e}\")\n \n return _config\n\ndef get_config() -> GroundingConfig:\n \"\"\"\n Get global configuration instance.\n \n Usage:\n - Get configuration in Provider: get_config().get_backend_config('shell')\n - Get security policy in Tool: get_config().get_security_policy('shell')\n \"\"\"\n global _config\n \n if _config is None:\n with _config_lock:\n if _config is None:\n load_config()\n \n return _config\n\ndef reset_config() -> None:\n \"\"\"Reset configuration (for testing)\"\"\"\n global _config\n with _config_lock:\n _config = None\n\ndef save_config(config: GroundingConfig, path: Union[str, Path]) -> None:\n save_json(config.model_dump(), path)\n logger.info(f\"Configuration saved to: {path}\")\n\n\ndef load_agents_config() -> Dict[str, Any]:\n agents_config_path = CONFIG_DIR / CONFIG_AGENTS\n return _load_json_file(agents_config_path)\n\n\ndef get_agent_config(agent_name: str) -> Optional[Dict[str, Any]]:\n \"\"\"\n Get the configuration of the specified agent\n \"\"\"\n agents_config = load_agents_config()\n \n if \"agents\" not in agents_config:\n logger.warning(f\"No 'agents' key found in {CONFIG_AGENTS}\")\n return None\n \n for agent_cfg in agents_config.get(\"agents\", []):\n if agent_cfg.get(\"name\") == agent_name:\n return agent_cfg\n \n logger.warning(f\"Agent '{agent_name}' not found in {CONFIG_AGENTS}\")\n return None\n\n\n__all__ = [\n \"CONFIG_DIR\",\n \"load_config\",\n \"get_config\",\n \"reset_config\",\n \"save_config\",\n \"load_agents_config\",\n \"get_agent_config\"\n]"
},
{
"path": "anytool/config/utils.py",
"content": "import json\nfrom pathlib import Path\nfrom typing import Any\n\n\ndef get_config_value(config: Any, key: str, default=None):\n if isinstance(config, dict):\n return config.get(key, default)\n else:\n return getattr(config, key, default)\n\n\ndef load_json_file(filepath: str | Path) -> dict[str, Any]:\n filepath = Path(filepath) if isinstance(filepath, str) else filepath\n \n with open(filepath, 'r', encoding='utf-8') as f:\n return json.load(f)\n\n\ndef save_json_file(data: dict[str, Any], filepath: str | Path, indent: int = 2) -> None:\n filepath = Path(filepath) if isinstance(filepath, str) else filepath\n \n # Ensure directory exists\n filepath.parent.mkdir(parents=True, exist_ok=True)\n \n with open(filepath, 'w', encoding='utf-8') as f:\n json.dump(data, f, indent=indent, ensure_ascii=False)\n\n\n__all__ = [\"get_config_value\", \"load_json_file\", \"save_json_file\"]"
},
{
"path": "anytool/grounding/backends/__init__.py",
"content": "# Use lazy imports to avoid loading all backends unconditionally\n\ndef _lazy_import_provider(provider_name: str):\n \"\"\"Lazy import provider class\"\"\"\n if provider_name == 'mcp':\n from .mcp.provider import MCPProvider\n return MCPProvider\n elif provider_name == 'shell':\n from .shell.provider import ShellProvider\n return ShellProvider\n elif provider_name == 'web':\n from .web.provider import WebProvider\n return WebProvider\n elif provider_name == 'gui':\n from .gui.provider import GUIProvider\n return GUIProvider\n else:\n raise ImportError(f\"Unknown provider: {provider_name}\")\n\n\nclass _ProviderRegistry:\n \"\"\"Lazy provider registry\"\"\"\n def __getitem__(self, key):\n return _lazy_import_provider(key)\n \n def __contains__(self, key):\n return key in ['mcp', 'shell', 'web', 'gui']\n\nBACKEND_PROVIDERS = _ProviderRegistry()\n\n__all__ = [\n 'BACKEND_PROVIDERS',\n '_lazy_import_provider'\n]"
},
{
"path": "anytool/grounding/backends/gui/__init__.py",
"content": "from .provider import GUIProvider\nfrom .session import GUISession\nfrom .transport.connector import GUIConnector\nfrom .transport.local_connector import LocalGUIConnector\n\ntry:\n from .anthropic_client import AnthropicGUIClient\n from . import anthropic_utils\n _anthropic_available = True\nexcept ImportError:\n _anthropic_available = False\n\n__all__ = [\n # Core Provider and Session\n \"GUIProvider\",\n \"GUISession\",\n \n # Transport layer\n \"GUIConnector\",\n \"LocalGUIConnector\",\n]\n\n# Add Anthropic modules to exports if available\nif _anthropic_available:\n __all__.extend([\"AnthropicGUIClient\", \"anthropic_utils\"])"
},
{
"path": "anytool/grounding/backends/gui/anthropic_client.py",
"content": "import base64\nimport os\nimport time\nfrom typing import Any, Dict, Optional, Tuple, List\nfrom anytool.utils.logging import Logger\nfrom PIL import Image\nimport io\n\nlogger = Logger.get_logger(__name__)\n\ntry:\n from anthropic import (\n Anthropic,\n AnthropicBedrock,\n AnthropicVertex,\n APIError,\n APIResponseValidationError,\n APIStatusError,\n )\n from anthropic.types.beta import (\n BetaMessageParam,\n BetaTextBlockParam,\n )\n ANTHROPIC_AVAILABLE = True\nexcept ImportError:\n logger.warning(\"Anthropic SDK not available. Install with: pip install anthropic\")\n ANTHROPIC_AVAILABLE = False\n\n# Import utility functions\nfrom .anthropic_utils import (\n APIProvider,\n PROVIDER_TO_DEFAULT_MODEL_NAME,\n COMPUTER_USE_BETA_FLAG,\n PROMPT_CACHING_BETA_FLAG,\n get_system_prompt,\n inject_prompt_caching,\n maybe_filter_to_n_most_recent_images,\n response_to_params,\n)\n\n# API retry configuration\nAPI_RETRY_TIMES = 10\nAPI_RETRY_INTERVAL = 5 # seconds\n\n\nclass AnthropicGUIClient:\n \"\"\"\n Anthropic LLM Client for GUI operations.\n Uses Claude Sonnet 4.5 with computer-use-2025-01-24 API.\n \n Features:\n - Vision-based screen understanding\n - Automatic screenshot resizing (configurable display size)\n - Coordinate scaling between display and actual screen\n \"\"\"\n \n def __init__(\n self,\n model: str = \"claude-sonnet-4-5\",\n platform: str = \"Ubuntu\",\n api_key: Optional[str] = None,\n provider: str = \"anthropic\",\n max_tokens: int = 4096,\n screen_size: Tuple[int, int] = (1920, 1080),\n display_size: Tuple[int, int] = (1024, 768), # Computer use display size\n pyautogui_size: Optional[Tuple[int, int]] = None, # PyAutoGUI working size\n only_n_most_recent_images: int = 3,\n enable_prompt_caching: bool = True,\n backup_api_key: Optional[str] = None,\n ):\n \"\"\"\n Initialize Anthropic GUI Client for Claude Sonnet 4.5.\n \n Args:\n model: Model name (only \"claude-sonnet-4-5\" supported)\n platform: Platform type (Ubuntu, Windows, or macOS)\n api_key: Anthropic API key (defaults to ANTHROPIC_API_KEY env var)\n provider: API provider (only \"anthropic\" supported)\n max_tokens: Maximum tokens for response\n screen_size: Actual screenshot resolution (width, height) - physical pixels\n display_size: Display size for computer use tool (width, height)\n Screenshots will be resized to this size before sending to API\n pyautogui_size: PyAutoGUI working size (logical pixels). If None, assumed same as screen_size.\n On Retina/HiDPI displays, this may be screen_size / 2\n only_n_most_recent_images: Number of recent screenshots to keep in history\n enable_prompt_caching: Whether to enable prompt caching for cost optimization\n backup_api_key: Backup API key (defaults to ANTHROPIC_API_KEY_BACKUP env var)\n \"\"\"\n if not ANTHROPIC_AVAILABLE:\n raise RuntimeError(\"Anthropic SDK not installed. Install with: pip install anthropic\")\n \n # Only support claude-sonnet-4-5\n if model != \"claude-sonnet-4-5\":\n logger.warning(f\"Model '{model}' not supported. Using 'claude-sonnet-4-5'\")\n model = \"claude-sonnet-4-5\"\n \n self.model = model\n self.platform = platform\n self.api_key = api_key or os.environ.get(\"ANTHROPIC_API_KEY\")\n if not self.api_key:\n raise ValueError(\"Anthropic API key not provided. Set ANTHROPIC_API_KEY env var or pass api_key parameter\")\n \n # Backup API key for failover\n self.backup_api_key = backup_api_key or os.environ.get(\"ANTHROPIC_API_KEY_BACKUP\")\n \n # Only support anthropic provider\n if provider != \"anthropic\":\n logger.warning(f\"Provider '{provider}' not supported. Using 'anthropic'\")\n provider = \"anthropic\"\n \n self.provider = APIProvider(provider)\n self.max_tokens = max_tokens\n self.screen_size = screen_size\n self.display_size = display_size\n self.pyautogui_size = pyautogui_size or screen_size # Default to screen_size if not specified\n self.only_n_most_recent_images = only_n_most_recent_images\n self.enable_prompt_caching = enable_prompt_caching\n \n # Message history\n self.messages: List[BetaMessageParam] = []\n \n # Calculate resize factor for coordinate scaling\n # Step 1: LLM coordinates (display_size) -> Physical pixels (screen_size)\n # Step 2: Physical pixels -> PyAutoGUI logical pixels (pyautogui_size)\n self.resize_factor = (\n self.pyautogui_size[0] / display_size[0], # x scale factor\n self.pyautogui_size[1] / display_size[1] # y scale factor\n )\n \n logger.info(\n f\"Initialized AnthropicGUIClient:\\n\"\n f\" Model: {model}\\n\"\n f\" Platform: {platform}\\n\"\n f\" Screen Size (physical): {screen_size}\\n\"\n f\" PyAutoGUI Size (logical): {self.pyautogui_size}\\n\"\n f\" Display Size (LLM): {display_size}\\n\"\n f\" Resize Factor (LLM->PyAutoGUI): {self.resize_factor}\\n\"\n f\" Prompt Caching: {enable_prompt_caching}\"\n )\n \n def _create_client(self, api_key: Optional[str] = None):\n \"\"\"Create Anthropic client (only supports anthropic provider).\"\"\"\n key = api_key or self.api_key\n return Anthropic(api_key=key, max_retries=4)\n \n def _resize_screenshot(self, screenshot_bytes: bytes) -> bytes:\n \"\"\"\n Resize screenshot to display size for Computer Use API.\n \n For computer-use-2025-01-24, the screenshot must be resized to the\n display_width_px x display_height_px specified in the tool definition.\n \"\"\"\n screenshot_image = Image.open(io.BytesIO(screenshot_bytes))\n resized_image = screenshot_image.resize(self.display_size, Image.Resampling.LANCZOS)\n \n output_buffer = io.BytesIO()\n resized_image.save(output_buffer, format='PNG')\n return output_buffer.getvalue()\n \n def _scale_coordinates(self, x: int, y: int) -> Tuple[int, int]:\n \"\"\"\n Scale coordinates from display size to actual screen size.\n \n The API returns coordinates in display_size (e.g., 1024x768).\n We need to scale them to actual screen_size (e.g., 1920x1080) for execution.\n \n Args:\n x, y: Coordinates in display size space\n \n Returns:\n Scaled coordinates in actual screen size space\n \"\"\"\n scaled_x = int(x * self.resize_factor[0])\n scaled_y = int(y * self.resize_factor[1])\n return scaled_x, scaled_y\n \n async def plan_action(\n self,\n task_description: str,\n screenshot: bytes,\n action_history: List[Dict[str, Any]] = None,\n ) -> Tuple[Optional[str], List[str]]:\n \"\"\"\n Plan next action based on task and current screenshot.\n Includes prompt caching, error handling, and backup API key support.\n \n Args:\n task_description: Task to accomplish\n screenshot: Current screenshot (PNG bytes)\n action_history: Previous actions (for context)\n \n Returns:\n Tuple of (reasoning, list of pyautogui commands)\n \"\"\"\n # Resize screenshot\n resized_screenshot = self._resize_screenshot(screenshot)\n screenshot_b64 = base64.b64encode(resized_screenshot).decode('utf-8')\n \n # Initialize messages with first task + screenshot\n if not self.messages:\n # IMPORTANT: Image should come BEFORE text for better model understanding\n # This matches OSWorld's implementation which has proven effectiveness\n self.messages.append({\n \"role\": \"user\",\n \"content\": [\n {\n \"type\": \"image\",\n \"source\": {\n \"type\": \"base64\",\n \"media_type\": \"image/png\",\n \"data\": screenshot_b64,\n },\n },\n {\"type\": \"text\", \"text\": task_description},\n ]\n })\n \n # Filter images BEFORE adding new screenshot to control message size\n # This is critical to avoid exceeding the 25MB API limit\n image_truncation_threshold = 10\n if self.only_n_most_recent_images and len(self.messages) > 1:\n # Reserve 1 slot for the screenshot we're about to add\n maybe_filter_to_n_most_recent_images(\n self.messages,\n max(1, self.only_n_most_recent_images - 1),\n min_removal_threshold=1, # More aggressive filtering\n )\n \n # Add tool result from previous action if exists\n if self.messages and self.messages[-1][\"role\"] == \"assistant\":\n last_content = self.messages[-1][\"content\"]\n if isinstance(last_content, list) and any(\n block.get(\"type\") == \"tool_use\" for block in last_content\n ):\n tool_use_id = next(\n block[\"id\"] for block in last_content \n if block.get(\"type\") == \"tool_use\"\n )\n self._add_tool_result(tool_use_id, \"Success\", resized_screenshot)\n \n # Define tools and betas for claude-sonnet-4-5 with computer-use-2025-01-24\n tools = [{\n 'name': 'computer',\n 'type': 'computer_20250124',\n 'display_width_px': self.display_size[0],\n 'display_height_px': self.display_size[1],\n 'display_number': 1\n }]\n betas = [COMPUTER_USE_BETA_FLAG]\n \n # Prepare system prompt with optional caching\n system = BetaTextBlockParam(\n type=\"text\",\n text=get_system_prompt(self.platform)\n )\n \n # Enable prompt caching if supported and enabled\n if self.enable_prompt_caching:\n betas.append(PROMPT_CACHING_BETA_FLAG)\n inject_prompt_caching(self.messages)\n system[\"cache_control\"] = {\"type\": \"ephemeral\"} # type: ignore\n \n # Model name - use claude-sonnet-4-5 directly\n model_name = \"claude-sonnet-4-5\"\n \n # Enable thinking for complex computer use tasks\n extra_body = {\"thinking\": {\"type\": \"enabled\", \"budget_tokens\": 2048}}\n \n # Log request details for debugging\n # Count current images in messages\n total_images = sum(\n 1\n for message in self.messages\n for item in (message.get(\"content\", []) if isinstance(message.get(\"content\"), list) else [])\n if isinstance(item, dict) and item.get(\"type\") == \"image\"\n )\n tool_result_images = sum(\n 1\n for message in self.messages\n for item in (message.get(\"content\", []) if isinstance(message.get(\"content\"), list) else [])\n if isinstance(item, dict) and item.get(\"type\") == \"tool_result\"\n for content in item.get(\"content\", [])\n if isinstance(content, dict) and content.get(\"type\") == \"image\"\n )\n logger.info(\n f\"Anthropic API request:\\n\"\n f\" Model: {model_name}\\n\"\n f\" Display Size: {self.display_size}\\n\"\n f\" Betas: {betas}\\n\"\n f\" Images: {total_images} ({tool_result_images} in tool_results)\\n\"\n f\" Messages: {len(self.messages)}\"\n )\n \n # Try API call with retry and backup\n client = self._create_client()\n response = None\n \n try:\n # Retry loop with automatic image count reduction on 25MB error\n for attempt in range(API_RETRY_TIMES):\n try:\n response = client.beta.messages.create(\n max_tokens=self.max_tokens,\n messages=self.messages,\n model=model_name,\n system=[system],\n tools=tools,\n betas=betas,\n extra_body=extra_body\n )\n logger.info(f\"API call succeeded on attempt {attempt + 1}\")\n break\n \n except (APIError, APIStatusError, APIResponseValidationError) as e:\n error_msg = str(e)\n logger.warning(f\"Anthropic API error (attempt {attempt+1}/{API_RETRY_TIMES}): {error_msg}\")\n \n # Handle 25MB payload limit error (including HTTP 413)\n if (\"25000000\" in error_msg or \n \"Member must have length less than or equal to\" in error_msg or \n \"request_too_large\" in error_msg or \n \"413\" in str(e)):\n logger.warning(\"Detected 25MB limit error, reducing image count\")\n current_count = self.only_n_most_recent_images\n new_count = max(1, current_count // 2)\n self.only_n_most_recent_images = new_count\n \n maybe_filter_to_n_most_recent_images(\n self.messages,\n new_count,\n min_removal_threshold=1, # Aggressive filtering when hitting limit\n )\n logger.info(f\"Image count reduced from {current_count} to {new_count}\")\n \n if attempt < API_RETRY_TIMES - 1:\n time.sleep(API_RETRY_INTERVAL)\n else:\n raise\n \n except (APIError, APIStatusError, APIResponseValidationError) as e:\n logger.error(f\"Primary API key failed: {e}\")\n \n # Try backup API key if available\n if self.backup_api_key:\n logger.warning(\"Retrying with backup API key...\")\n try:\n backup_client = self._create_client(self.backup_api_key)\n response = backup_client.beta.messages.create(\n max_tokens=self.max_tokens,\n messages=self.messages,\n model=model_name,\n system=[system],\n tools=tools,\n betas=betas,\n extra_body=extra_body\n )\n logger.info(\"Successfully used backup API key\")\n except Exception as backup_e:\n logger.error(f\"Backup API key also failed: {backup_e}\")\n return None, [\"FAIL\"]\n else:\n return None, [\"FAIL\"]\n \n except Exception as e:\n logger.error(f\"Unexpected error: {e}\")\n return None, [\"FAIL\"]\n \n if not response:\n return None, [\"FAIL\"]\n \n # Parse response using utility function\n response_params = response_to_params(response)\n \n # Extract reasoning and commands\n reasoning = \"\"\n commands = []\n \n for block in response_params:\n block_type = block.get(\"type\")\n \n if block_type == \"text\":\n reasoning = block.get(\"text\", \"\")\n elif block_type == \"thinking\":\n reasoning = block.get(\"thinking\", \"\")\n elif block_type == \"tool_use\":\n tool_input = block.get(\"input\", {})\n command = self._parse_computer_tool_use(tool_input)\n if command:\n commands.append(command)\n else:\n logger.warning(f\"Failed to parse tool_use: {tool_input}\")\n \n # Store assistant response\n self.messages.append({\n \"role\": \"assistant\",\n \"content\": response_params\n })\n \n logger.info(f\"Parsed {len(commands)} commands from response\")\n \n return reasoning, commands\n \n def _add_tool_result(\n self,\n tool_use_id: str,\n result: str,\n screenshot_bytes: Optional[bytes] = None\n ):\n \"\"\"\n Add tool result to message history.\n IMPORTANT: Put screenshot BEFORE text for consistency with initial message.\n \"\"\"\n # Build content list with image first (if provided), then text\n content_list = []\n \n # Add screenshot first if provided (consistent with initial message ordering)\n if screenshot_bytes is not None:\n screenshot_b64 = base64.b64encode(screenshot_bytes).decode('utf-8')\n content_list.append({\n \"type\": \"image\",\n \"source\": {\n \"type\": \"base64\",\n \"media_type\": \"image/png\",\n \"data\": screenshot_b64\n }\n })\n \n # Then add text result\n content_list.append({\"type\": \"text\", \"text\": result})\n \n tool_result_content = [{\n \"type\": \"tool_result\",\n \"tool_use_id\": tool_use_id,\n \"content\": content_list\n }]\n \n self.messages.append({\n \"role\": \"user\",\n \"content\": tool_result_content\n })\n \n def _parse_computer_tool_use(self, tool_input: Dict[str, Any]) -> Optional[str]:\n \"\"\"\n Parse Anthropic computer tool use to pyautogui command.\n \n Args:\n tool_input: Tool input from Anthropic (action, coordinate, text, etc.)\n \n Returns:\n PyAutoGUI command string or control command (DONE, FAIL)\n \"\"\"\n action = tool_input.get(\"action\")\n if not action:\n return None\n \n # Action conversion\n action_conversion = {\n \"left click\": \"click\",\n \"right click\": \"right_click\"\n }\n action = action_conversion.get(action, action)\n \n text = tool_input.get(\"text\")\n coordinate = tool_input.get(\"coordinate\")\n scroll_direction = tool_input.get(\"scroll_direction\")\n scroll_amount = tool_input.get(\"scroll_amount\", 5)\n \n # Scale coordinates to actual screen size\n if coordinate:\n coordinate = self._scale_coordinates(coordinate[0], coordinate[1])\n \n # Build commands\n command = \"\"\n \n if action == \"mouse_move\":\n if coordinate:\n x, y = coordinate\n command = f\"pyautogui.moveTo({x}, {y}, duration=0.5)\"\n \n elif action in (\"left_click\", \"click\"):\n if coordinate:\n x, y = coordinate\n command = f\"pyautogui.click({x}, {y})\"\n else:\n command = \"pyautogui.click()\"\n \n elif action == \"right_click\":\n if coordinate:\n x, y = coordinate\n command = f\"pyautogui.rightClick({x}, {y})\"\n else:\n command = \"pyautogui.rightClick()\"\n \n elif action == \"double_click\":\n if coordinate:\n x, y = coordinate\n command = f\"pyautogui.doubleClick({x}, {y})\"\n else:\n command = \"pyautogui.doubleClick()\"\n \n elif action == \"middle_click\":\n if coordinate:\n x, y = coordinate\n command = f\"pyautogui.middleClick({x}, {y})\"\n else:\n command = \"pyautogui.middleClick()\"\n \n elif action == \"left_click_drag\":\n if coordinate:\n x, y = coordinate\n command = f\"pyautogui.dragTo({x}, {y}, duration=0.5)\"\n \n elif action == \"key\":\n if text:\n keys = text.split('+')\n # Key conversion\n key_conversion = {\n \"page_down\": \"pagedown\",\n \"page_up\": \"pageup\",\n \"super_l\": \"win\",\n \"super\": \"command\",\n \"escape\": \"esc\"\n }\n converted_keys = [key_conversion.get(k.strip().lower(), k.strip().lower()) for k in keys]\n \n # Press and release keys\n for key in converted_keys:\n command += f\"pyautogui.keyDown('{key}'); \"\n for key in reversed(converted_keys):\n command += f\"pyautogui.keyUp('{key}'); \"\n # Remove trailing semicolon and space\n command = command.rstrip('; ')\n \n elif action == \"type\":\n if text:\n command = f\"pyautogui.typewrite({repr(text)}, interval=0.01)\"\n \n elif action == \"scroll\":\n if scroll_direction in (\"up\", \"down\"):\n scroll_value = scroll_amount if scroll_direction == \"up\" else -scroll_amount\n if coordinate:\n x, y = coordinate\n command = f\"pyautogui.scroll({scroll_value}, {x}, {y})\"\n else:\n command = f\"pyautogui.scroll({scroll_value})\"\n elif scroll_direction in (\"left\", \"right\"):\n scroll_value = scroll_amount if scroll_direction == \"right\" else -scroll_amount\n if coordinate:\n x, y = coordinate\n command = f\"pyautogui.hscroll({scroll_value}, {x}, {y})\"\n else:\n command = f\"pyautogui.hscroll({scroll_value})\"\n \n elif action == \"screenshot\":\n # Screenshot is automatically handled by the system\n # Return special marker to indicate no action needed\n return \"SCREENSHOT\"\n \n elif action == \"wait\":\n # Wait for specified duration\n duration = tool_input.get(\"duration\", 1)\n command = f\"pyautogui.sleep({duration})\"\n \n elif action == \"done\":\n return \"DONE\"\n \n elif action == \"fail\":\n return \"FAIL\"\n \n return command if command else None\n \n def reset(self):\n \"\"\"Reset message history.\"\"\"\n self.messages = []\n logger.info(\"Reset AnthropicGUIClient message history\")"
},
{
"path": "anytool/grounding/backends/gui/anthropic_utils.py",
"content": "from typing import List, cast\nfrom enum import Enum\nfrom datetime import datetime\nfrom anytool.utils.logging import Logger\n\nlogger = Logger.get_logger(__name__)\n\ntry:\n from anthropic.types.beta import (\n BetaCacheControlEphemeralParam,\n BetaContentBlockParam,\n BetaImageBlockParam,\n BetaMessage,\n BetaMessageParam,\n BetaTextBlock,\n BetaTextBlockParam,\n BetaToolResultBlockParam,\n BetaToolUseBlockParam,\n )\n ANTHROPIC_AVAILABLE = True\nexcept ImportError:\n ANTHROPIC_AVAILABLE = False\n\n\n# Beta flags\n# For claude-sonnet-4-5 with computer-use-2025-01-24\nCOMPUTER_USE_BETA_FLAG = \"computer-use-2025-01-24\"\nPROMPT_CACHING_BETA_FLAG = \"prompt-caching-2024-07-31\"\n\n\nclass APIProvider(Enum):\n \"\"\"API Provider enumeration\"\"\"\n ANTHROPIC = \"anthropic\"\n # BEDROCK = \"bedrock\"\n # VERTEX = \"vertex\"\n\n\n# Provider to model name mapping (simplified for claude-sonnet-4-5 only)\nPROVIDER_TO_DEFAULT_MODEL_NAME: dict = {\n (APIProvider.ANTHROPIC, \"claude-sonnet-4-5\"): \"claude-sonnet-4-5\",\n # (APIProvider.BEDROCK, \"claude-sonnet-4-5\"): \"us.anthropic.claude-sonnet-4-5-v1:0\",\n # (APIProvider.VERTEX, \"claude-sonnet-4-5\"): \"claude-sonnet-4-5-v1\",\n}\n\n\ndef get_system_prompt(platform: str = \"Ubuntu\") -> str:\n \"\"\"\n Get system prompt based on platform.\n \n Args:\n platform: Platform type (Ubuntu, Windows, macOS, or Darwin)\n \n Returns:\n System prompt string\n \"\"\"\n # Normalize platform name\n platform_lower = platform.lower()\n \n if platform_lower in [\"windows\", \"win32\"]:\n return f\"\"\"\n* You are utilising a Windows virtual machine using x86_64 architecture with internet access.\n* You can use the computer tool to interact with the desktop: take screenshots, click, type, and control applications.\n* To accomplish tasks, you MUST use the computer tool to see the screen and take actions.\n* To open browser, please just click on the Chrome icon. Note, Chrome is what is installed on your system.\n* When viewing a page it can be helpful to zoom out so that you can see everything on the page. Either that, or make sure you scroll down to see everything before deciding something isn't available.\n* DO NOT ask users for clarification during task execution. DO NOT stop to request more information from users. Always take action using available tools.\n* When using your computer function calls, they take a while to run and send back to you. Where possible/feasible, try to chain multiple of these calls all into one function calls request.\n* The current date is {datetime.today().strftime('%A, %B %d, %Y')}.\n* Home directory of this Windows system is 'C:\\\\Users\\\\user'.\n* When you want to open some applications on Windows, please use Double Click on it instead of clicking once.\n* After each action, the system will provide you with a new screenshot showing the result.\n* Continue taking actions until the task is complete.\n\"\"\"\n elif platform_lower in [\"macos\", \"darwin\", \"mac\"]:\n return f\"\"\"\n* You are utilising a macOS system with internet access.\n* You can use the computer tool to interact with the desktop: take screenshots, click, type, and control applications.\n* To accomplish tasks, you MUST use the computer tool to see the screen and take actions.\n* To open browser, please just click on the Chrome icon. Note, Chrome is what is installed on your system.\n* When viewing a page it can be helpful to zoom out so that you can see everything on the page. Either that, or make sure you scroll down to see everything before deciding something isn't available.\n* DO NOT ask users for clarification during task execution. DO NOT stop to request more information from users. Always take action using available tools.\n* When using your computer function calls, they take a while to run and send back to you. Where possible/feasible, try to chain multiple of these calls all into one function calls request.\n* The current date is {datetime.today().strftime('%A, %B %d, %Y')}.\n* Home directory of this macOS system is typically '/Users/[username]' or can be accessed via '~'.\n* On macOS, use Command (โ) key combinations instead of Ctrl (e.g., Command+C for copy).\n* After each action, the system will provide you with a new screenshot showing the result.\n* Continue taking actions until the task is complete.\n* When the task is completed, simply describe what you've done in your response WITHOUT using the tool again.\n\"\"\"\n else: # Ubuntu/Linux\n return f\"\"\"\n* You are utilising an Ubuntu virtual machine using x86_64 architecture with internet access.\n* You can use the computer tool to interact with the desktop: take screenshots, click, type, and control applications.\n* To accomplish tasks, you MUST use the computer tool to see the screen and take actions.\n* To open browser, please just click on the Chrome icon. Note, Chrome is what is installed on your system.\n* When viewing a page it can be helpful to zoom out so that you can see everything on the page. Either that, or make sure you scroll down to see everything before deciding something isn't available.\n* DO NOT ask users for clarification during task execution. DO NOT stop to request more information from users. Always take action using available tools.\n* When using your computer function calls, they take a while to run and send back to you. Where possible/feasible, try to chain multiple of these calls all into one function calls request.\n* The current date is {datetime.today().strftime('%A, %B %d, %Y')}.\n* Home directory of this Ubuntu system is '/home/user'.\n* After each action, the system will provide you with a new screenshot showing the result.\n* Continue taking actions until the task is complete.\n\"\"\"\n\n\ndef inject_prompt_caching(messages: List[BetaMessageParam]) -> None:\n \"\"\"\n Set cache breakpoints for the 3 most recent turns.\n One cache breakpoint is left for tools/system prompt, to be shared across sessions.\n \n Args:\n messages: Message history (modified in place)\n \"\"\"\n if not ANTHROPIC_AVAILABLE:\n return\n \n breakpoints_remaining = 3\n for message in reversed(messages):\n if message[\"role\"] == \"user\" and isinstance(\n content := message[\"content\"], list\n ):\n if breakpoints_remaining:\n breakpoints_remaining -= 1\n # Use type ignore to bypass TypedDict check until SDK types are updated\n content[-1][\"cache_control\"] = BetaCacheControlEphemeralParam( # type: ignore\n {\"type\": \"ephemeral\"}\n )\n else:\n content[-1].pop(\"cache_control\", None)\n # we'll only ever have one extra turn per loop\n break\n\n\ndef maybe_filter_to_n_most_recent_images(\n messages: List[BetaMessageParam],\n images_to_keep: int,\n min_removal_threshold: int,\n) -> None:\n \"\"\"\n With the assumption that images are screenshots that are of diminishing value as\n the conversation progresses, remove all but the final `images_to_keep` tool_result\n images in place, with a chunk of min_removal_threshold to reduce the amount we\n break the implicit prompt cache.\n \n Args:\n messages: Message history (modified in place)\n images_to_keep: Number of recent images to keep\n min_removal_threshold: Minimum number of images to remove at once (for cache efficiency)\n \"\"\"\n if not ANTHROPIC_AVAILABLE or images_to_keep is None:\n return\n \n tool_result_blocks = cast(\n list[BetaToolResultBlockParam],\n [\n item\n for message in messages\n for item in (\n message[\"content\"] if isinstance(message[\"content\"], list) else []\n )\n if isinstance(item, dict) and item.get(\"type\") == \"tool_result\"\n ],\n )\n \n total_images = sum(\n 1\n for tool_result in tool_result_blocks\n for content in tool_result.get(\"content\", [])\n if isinstance(content, dict) and content.get(\"type\") == \"image\"\n )\n \n images_to_remove = total_images - images_to_keep\n # for better cache behavior, we want to remove in chunks\n images_to_remove -= images_to_remove % min_removal_threshold\n \n for tool_result in tool_result_blocks:\n if isinstance(tool_result.get(\"content\"), list):\n new_content = []\n for content in tool_result.get(\"content\", []):\n if isinstance(content, dict) and content.get(\"type\") == \"image\":\n if images_to_remove > 0:\n images_to_remove -= 1\n continue\n new_content.append(content)\n tool_result[\"content\"] = new_content\n\n\ndef response_to_params(response: BetaMessage) -> List[BetaContentBlockParam]:\n \"\"\"\n Convert Anthropic response to parameter list.\n Handles both text blocks, tool use blocks, and thinking blocks.\n \n Args:\n response: Anthropic API response\n \n Returns:\n List of content blocks\n \"\"\"\n if not ANTHROPIC_AVAILABLE:\n return []\n \n res: List[BetaContentBlockParam] = []\n if response.content:\n for block in response.content:\n # Check block type using type attribute\n # Note: type may be a string or enum, so convert to string for comparison\n block_type = str(getattr(block, \"type\", \"\"))\n \n if block_type == \"text\":\n # Regular text block\n if isinstance(block, BetaTextBlock) and block.text:\n res.append(BetaTextBlockParam(type=\"text\", text=block.text))\n elif block_type == \"thinking\":\n # Thinking block (for Claude 4 and Sonnet 3.7)\n thinking_block = {\n \"type\": \"thinking\",\n \"thinking\": getattr(block, \"thinking\", \"\"),\n }\n if hasattr(block, \"signature\"):\n thinking_block[\"signature\"] = getattr(block, \"signature\", None)\n res.append(cast(BetaContentBlockParam, thinking_block))\n elif block_type == \"tool_use\":\n # Tool use block - only include required fields to avoid API errors\n # (e.g., 'caller' field is not permitted by Anthropic API)\n tool_use_dict = {\n \"type\": \"tool_use\",\n \"id\": block.id,\n \"name\": block.name,\n \"input\": block.input,\n }\n res.append(cast(BetaToolUseBlockParam, tool_use_dict))\n else:\n # Unknown block type - try to handle generically\n try:\n res.append(cast(BetaContentBlockParam, block.model_dump()))\n except Exception as e:\n logger.warning(f\"Failed to parse block type {block_type}: {e}\")\n return res\n else:\n return []\n\n"
},
{
"path": "anytool/grounding/backends/gui/config.py",
"content": "from typing import Dict, Any, Optional\nimport os\nimport platform as platform_module\nfrom anytool.utils.logging import Logger\n\nlogger = Logger.get_logger(__name__)\n\n\ndef build_llm_config(user_config: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:\n \"\"\"\n Build complete LLM configuration with auto-detection and environment variables.\n \n Auto-detects:\n - API key from environment variables (ANTHROPIC_API_KEY)\n - Platform from system (macOS/Windows/Ubuntu)\n - Provider defaults to 'anthropic'\n \n User-provided config values will override auto-detected values.\n \n Args:\n user_config: User-provided configuration (optional)\n \n Returns:\n Complete LLM configuration dict\n \n Example:\n >>> # Auto-detect everything\n >>> config = build_llm_config()\n \n >>> # Override specific values\n >>> config = build_llm_config({\n ... \"model\": \"claude-3-5-sonnet-20241022\",\n ... \"max_tokens\": 8192\n ... })\n \"\"\"\n if user_config is None:\n user_config = {}\n \n # Auto-detect platform\n system = platform_module.system()\n if system == \"Darwin\":\n detected_platform = \"macOS\"\n elif system == \"Windows\":\n detected_platform = \"Windows\"\n else: # Linux\n detected_platform = \"Ubuntu\"\n \n # Auto-detect API key from environment\n api_key = os.environ.get(\"ANTHROPIC_API_KEY\")\n if not api_key:\n logger.warning(\n \"ANTHROPIC_API_KEY not found in environment. \"\n \"Please set it: export ANTHROPIC_API_KEY='your-key'\"\n )\n \n # Build configuration with precedence: user_config > auto-detected > defaults\n config = {\n \"type\": user_config.get(\"type\", \"anthropic\"),\n \"model\": user_config.get(\"model\", \"claude-sonnet-4-5\"),\n \"platform\": user_config.get(\"platform\", detected_platform),\n \"api_key\": user_config.get(\"api_key\", api_key),\n \"provider\": user_config.get(\"provider\", \"anthropic\"),\n \"max_tokens\": user_config.get(\"max_tokens\", 4096),\n \"only_n_most_recent_images\": user_config.get(\"only_n_most_recent_images\", 3),\n \"enable_prompt_caching\": user_config.get(\"enable_prompt_caching\", True),\n }\n \n # Optional: screen_size (will be auto-detected from screenshot later)\n if \"screen_size\" in user_config:\n config[\"screen_size\"] = user_config[\"screen_size\"]\n \n logger.info(f\"Built LLM config - Platform: {config['platform']}, Model: {config['model']}\")\n if config[\"api_key\"]:\n logger.info(f\"API key loaded: {config['api_key'][:10]}...\")\n \n return config"
},
{
"path": "anytool/grounding/backends/gui/provider.py",
"content": "from typing import Dict, Any, Union\nfrom anytool.grounding.core.types import BackendType, SessionConfig\nfrom anytool.grounding.core.provider import Provider\nfrom anytool.grounding.core.session import BaseSession\nfrom anytool.config import get_config\nfrom anytool.config.utils import get_config_value\nfrom anytool.platform import get_local_server_config\nfrom anytool.utils.logging import Logger\nfrom .transport.connector import GUIConnector\nfrom .transport.local_connector import LocalGUIConnector\nfrom .session import GUISession\n\nlogger = Logger.get_logger(__name__)\n\n\nclass GUIProvider(Provider):\n \"\"\"\n Provider for GUI desktop environment.\n Manages communication with desktop_env through HTTP API or local in-process execution.\n \n Supports two modes:\n - \"local\": Execute GUI operations directly in-process (no server needed)\n - \"server\": Connect to a running local_server via HTTP API\n \n Supports automatic default session creation:\n - If no session exists, a default session will be created on first use\n - Default session uses configuration from config file or environment\n \"\"\"\n \n DEFAULT_SID = BackendType.GUI.value\n \n def __init__(self, config: Dict[str, Any] = None):\n \"\"\"\n Initialize GUI provider.\n \n Args:\n config: Provider configuration\n \"\"\"\n super().__init__(BackendType.GUI, config)\n self.connectors: Dict[str, Union[GUIConnector, LocalGUIConnector]] = {}\n \n async def initialize(self) -> None:\n \"\"\"\n Initialize the provider and create default session.\n \"\"\"\n if not self.is_initialized:\n logger.info(\"Initializing GUI provider\")\n # Auto-create default session\n await self.create_session(SessionConfig(\n session_name=self.DEFAULT_SID,\n backend_type=BackendType.GUI,\n connection_params={}\n ))\n self.is_initialized = True\n \n async def create_session(self, session_config: SessionConfig) -> BaseSession:\n \"\"\"\n Create GUI session.\n \n Args:\n session_config: Session configuration\n \n Returns:\n GUISession instance\n \"\"\"\n # Load GUI backend configuration\n gui_config = get_config().get_backend_config(\"gui\")\n \n # Determine execution mode: \"local\" or \"server\"\n mode = getattr(gui_config, \"mode\", \"local\")\n \n # Extract connection parameters\n conn_params = session_config.connection_params\n timeout = get_config_value(conn_params, 'timeout', gui_config.timeout)\n retry_times = get_config_value(conn_params, 'retry_times', gui_config.max_retries)\n retry_interval = get_config_value(conn_params, 'retry_interval', gui_config.retry_interval)\n \n # Build pkgs_prefix with failsafe setting\n failsafe_str = \"True\" if gui_config.failsafe else \"False\"\n pkgs_prefix = get_config_value(\n conn_params, \n 'pkgs_prefix', \n gui_config.pkgs_prefix.format(failsafe=failsafe_str, command=\"{command}\")\n )\n \n if mode == \"local\":\n # ---------- LOCAL MODE ----------\n logger.info(\"GUI backend using LOCAL mode (no server required)\")\n connector = LocalGUIConnector(\n timeout=timeout,\n retry_times=retry_times,\n retry_interval=retry_interval,\n pkgs_prefix=pkgs_prefix,\n )\n else:\n # ---------- SERVER MODE ----------\n logger.info(\"GUI backend using SERVER mode (connecting to local_server)\")\n local_server_config = get_local_server_config()\n vm_ip = get_config_value(conn_params, 'vm_ip', local_server_config['host'])\n server_port = get_config_value(conn_params, 'server_port', local_server_config['port'])\n \n connector = GUIConnector(\n vm_ip=vm_ip,\n server_port=server_port,\n timeout=timeout,\n retry_times=retry_times,\n retry_interval=retry_interval,\n pkgs_prefix=pkgs_prefix,\n )\n \n # Create session\n session = GUISession(\n connector=connector,\n session_id=session_config.session_name,\n backend_type=BackendType.GUI,\n config=session_config,\n )\n \n # Store connector and session\n self.connectors[session_config.session_name] = connector\n self._sessions[session_config.session_name] = session\n \n logger.info(f\"Created GUI session: {session_config.session_name} (mode={mode})\")\n return session\n \n async def close_session(self, session_name: str) -> None:\n \"\"\"\n Close GUI session.\n \n Args:\n session_name: Name of the session to close\n \"\"\"\n if session_name in self._sessions:\n session = self._sessions[session_name]\n await session.disconnect()\n del self._sessions[session_name]\n \n if session_name in self.connectors:\n connector = self.connectors[session_name]\n await connector.disconnect()\n del self.connectors[session_name]\n \n logger.info(f\"Closed GUI session: {session_name}\")"
},
{
"path": "anytool/grounding/backends/gui/session.py",
"content": "from typing import Dict, Any, Union\nimport os\nfrom anytool.grounding.core.session import BaseSession\nfrom anytool.grounding.core.types import BackendType, SessionStatus, SessionConfig\nfrom anytool.utils.logging import Logger\nfrom .transport.connector import GUIConnector\nfrom .transport.local_connector import LocalGUIConnector\nfrom .tool import GUIAgentTool\nfrom .config import build_llm_config\n\nlogger = Logger.get_logger(__name__)\n\n\nclass GUISession(BaseSession):\n \"\"\"\n Session for GUI desktop environment.\n Manages connection and tools for GUI automation.\n \"\"\"\n \n def __init__(\n self,\n connector: Union[GUIConnector, LocalGUIConnector],\n session_id: str,\n backend_type: BackendType.GUI,\n config: SessionConfig,\n auto_connect: bool = True,\n auto_initialize: bool = True,\n ):\n \"\"\"\n Initialize GUI session.\n \n Args:\n connector: GUI HTTP connector\n session_id: Unique session identifier\n backend_type: Backend type (GUI)\n config: Session configuration\n auto_connect: Auto-connect on context enter\n auto_initialize: Auto-initialize on context enter\n \"\"\"\n super().__init__(\n connector=connector,\n session_id=session_id,\n backend_type=backend_type,\n auto_connect=auto_connect,\n auto_initialize=auto_initialize,\n )\n self.config = config\n self.gui_connector = connector\n \n async def initialize(self) -> Dict[str, Any]:\n \"\"\"\n Initialize session: connect and discover tools.\n \n Returns:\n Session information dict\n \"\"\"\n logger.info(f\"Initializing GUI session: {self.session_id}\")\n \n # Ensure connected\n if not self.connector.is_connected:\n await self.connect()\n \n # Create LLM client if configured\n llm_client = None\n user_llm_config = self.config.connection_params.get(\"llm_config\")\n \n # Build complete LLM config with auto-detection\n # If user provides llm_config, merge with auto-detected values\n # If user doesn't provide llm_config, try to auto-build one if ANTHROPIC_API_KEY exists\n if user_llm_config or os.environ.get(\"ANTHROPIC_API_KEY\"):\n llm_config = build_llm_config(user_llm_config)\n \n if llm_config.get(\"type\") == \"anthropic\":\n # Check if API key is available\n if not llm_config.get(\"api_key\"):\n logger.warning(\n \"Anthropic API key not found. Skipping LLM client initialization. \"\n \"Set ANTHROPIC_API_KEY environment variable or provide api_key in llm_config.\"\n )\n else:\n try:\n from .anthropic_client import AnthropicGUIClient\n \n # Detect actual screen size from screenshot (most accurate)\n # PyAutoGUI may report logical resolution, but we need the actual screenshot size\n try:\n screenshot_bytes = await self.gui_connector.get_screenshot()\n if screenshot_bytes:\n from PIL import Image\n import io\n img = Image.open(io.BytesIO(screenshot_bytes))\n actual_screen_size = img.size\n logger.info(f\"Auto-detected screen size from screenshot: {actual_screen_size}\")\n screen_size = actual_screen_size\n else:\n raise RuntimeError(\"Could not get screenshot\")\n except Exception as e:\n # Fallback to pyautogui detection\n actual_screen_size = await self.gui_connector.get_screen_size()\n if actual_screen_size:\n logger.info(f\"Auto-detected screen size from pyautogui: {actual_screen_size}\")\n screen_size = actual_screen_size\n else:\n # Final fallback to configured value\n screen_size = llm_config.get(\"screen_size\", (1920, 1080))\n logger.warning(f\"Could not auto-detect screen size, using configured: {screen_size}\")\n \n # Detect PyAutoGUI working size (logical pixels)\n pyautogui_size = await self.gui_connector.get_screen_size()\n if pyautogui_size:\n logger.info(f\"PyAutoGUI working size (logical): {pyautogui_size}\")\n else:\n # If we can't detect PyAutoGUI size, assume it's the same as screen size\n pyautogui_size = screen_size\n logger.warning(f\"Could not detect PyAutoGUI size, assuming same as screen: {pyautogui_size}\")\n \n llm_client = AnthropicGUIClient(\n model=llm_config[\"model\"],\n platform=llm_config[\"platform\"],\n api_key=llm_config[\"api_key\"],\n provider=llm_config[\"provider\"],\n screen_size=screen_size,\n pyautogui_size=pyautogui_size,\n max_tokens=llm_config[\"max_tokens\"],\n only_n_most_recent_images=llm_config[\"only_n_most_recent_images\"],\n )\n logger.info(\n f\"Initialized Anthropic LLM client - \"\n f\"Model: {llm_config['model']}, Platform: {llm_config['platform']}\"\n )\n except Exception as e:\n logger.warning(f\"Failed to initialize Anthropic client: {e}\")\n \n # Get recording_manager from connection_params if available\n recording_manager = self.config.connection_params.get(\"recording_manager\")\n \n # Create GUI Agent Tool\n self.tools = [\n GUIAgentTool(\n connector=self.gui_connector, \n llm_client=llm_client,\n recording_manager=recording_manager\n )\n ]\n \n logger.info(f\"Initialized GUI session with {len(self.tools)} tool(s)\")\n \n # Return session info\n session_info = {\n \"session_id\": self.session_id,\n \"backend_type\": self.backend_type.value,\n \"vm_ip\": self.gui_connector.vm_ip,\n \"server_port\": self.gui_connector.server_port,\n \"num_tools\": len(self.tools),\n \"tools\": [tool.name for tool in self.tools],\n \"llm_client\": \"anthropic\" if llm_client else \"none\",\n }\n \n return session_info\n \n async def connect(self) -> None:\n \"\"\"Connect to GUI desktop environment\"\"\"\n if self.connector.is_connected:\n return\n \n self.status = SessionStatus.CONNECTING\n logger.info(f\"Connecting to desktop_env at {self.gui_connector.base_url}\")\n \n await self.connector.connect()\n \n self.status = SessionStatus.CONNECTED\n logger.info(\"Connected to desktop environment\")\n \n async def disconnect(self) -> None:\n \"\"\"Disconnect from GUI desktop environment\"\"\"\n if not self.connector.is_connected:\n return\n \n logger.info(\"Disconnecting from desktop environment\")\n await self.connector.disconnect()\n \n self.status = SessionStatus.DISCONNECTED\n logger.info(\"Disconnected from desktop environment\")\n \n @property\n def is_connected(self) -> bool:\n \"\"\"Check if session is connected\"\"\"\n return self.connector.is_connected"
},
{
"path": "anytool/grounding/backends/gui/tool.py",
"content": "import base64\nfrom typing import Any, Dict\nfrom anytool.grounding.core.tool.base import BaseTool\nfrom anytool.grounding.core.types import BackendType, ToolResult, ToolStatus\nfrom .transport.connector import GUIConnector\nfrom .transport.actions import ACTION_SPACE, KEYBOARD_KEYS\nfrom anytool.utils.logging import Logger\n\nlogger = Logger.get_logger(__name__)\n\n\nclass GUIAgentTool(BaseTool):\n \"\"\"\n LLM-powered GUI Agent Tool.\n \n This tool acts as an intelligent agent that:\n - Takes a task description as input\n - Observes the desktop via screenshot\n - Uses LLM/VLM to understand and plan actions\n - Outputs action space commands\n - Executes actions through the connector\n \"\"\"\n \n _name = \"gui_agent\"\n _description = \"\"\"Vision-based GUI automation agent for tasks requiring graphical interface interaction.\n \n Use this tool when the task involves:\n - Operating desktop applications with graphical interfaces (browsers, editors, design tools, etc.)\n - Tasks that require visual understanding of UI elements, layouts, or content\n - Multi-step workflows that need click, drag, type, or other GUI interactions\n - Scenarios where programmatic APIs or command-line tools are unavailable or insufficient\n \n The agent observes screen state through screenshots, uses vision-language models to understand\n the interface, plans appropriate actions, and executes GUI operations autonomously.\n \n IMPORTANT - max_steps Parameter Guidelines:\n - Simple tasks (1-2 actions): 15-20 steps\n - Medium tasks (3-5 actions): 25-35 steps \n - Complex tasks (6+ actions, like web navigation): 35-50 steps\n - When uncertain, prefer larger values (35+) to avoid premature termination\n - Default is 25, but increase for multi-step workflows\n \n Input: \n - task_description: Natural language task description\n - max_steps: Maximum actions (default 25, increase for complex tasks)\n \n Output: Task execution results with action history and completion status\n \"\"\"\n \n backend_type = BackendType.GUI\n \n def __init__(self, connector: GUIConnector, llm_client=None, recording_manager=None, **kwargs):\n \"\"\"\n Initialize GUI Agent Tool.\n \n Args:\n connector: GUI connector for communication with desktop_env\n llm_client: LLM/VLM client for vision-based planning (optional)\n recording_manager: RecordingManager for recording intermediate steps (optional)\n **kwargs: Additional arguments for BaseTool\n \"\"\"\n super().__init__(**kwargs)\n self.connector = connector\n self.llm_client = llm_client # Will be injected later\n self.recording_manager = recording_manager # For recording intermediate steps\n self.action_history = [] # Track executed actions\n \n async def _arun(\n self,\n task_description: str,\n max_steps: int = 50,\n ) -> ToolResult:\n \"\"\"\n Execute a GUI automation task using LLM planning.\n \n This is the main entry point that:\n 1. Gets current screenshot\n 2. Uses LLM to plan next action based on task and screenshot\n 3. Executes the planned action\n 4. Repeats until task is complete or max_steps reached\n \n Args:\n task_description: Natural language description of the task\n max_steps: Maximum number of actions to execute (default 25)\n Recommended values based on task complexity:\n - Simple (1-2 actions): 15-20\n - Medium (3-5 actions): 25-35\n - Complex (6+ actions, web navigation, multi-app): 35-50\n When in doubt, use higher values to avoid premature termination\n \n Returns:\n ToolResult with task execution status\n \"\"\"\n if not task_description:\n return ToolResult(\n status=ToolStatus.ERROR,\n error=\"task_description is required\"\n )\n \n logger.info(f\"Starting GUI task: {task_description}\")\n self.action_history = []\n \n # Execute task with LLM planning loop\n try:\n result = await self._execute_task_with_planning(\n task_description=task_description,\n max_steps=max_steps,\n )\n return result\n \n except Exception as e:\n logger.error(f\"Task execution failed: {e}\")\n return ToolResult(\n status=ToolStatus.ERROR,\n error=str(e),\n metadata={\n \"task_description\": task_description,\n \"actions_executed\": len(self.action_history),\n \"action_history\": self.action_history,\n }\n )\n \n async def _execute_task_with_planning(\n self,\n task_description: str,\n max_steps: int,\n ) -> ToolResult:\n \"\"\"\n Execute task with LLM-based planning loop.\n \n Planning loop:\n 1. Observe: Get screenshot\n 2. Plan: LLM decides next action\n 3. Execute: Perform the action\n 4. Verify: Check if task is complete\n 5. Repeat until done or max_steps\n \n Args:\n task_description: Task to complete\n max_steps: Maximum planning iterations\n \n Returns:\n ToolResult with execution details\n \"\"\"\n # Collect all screenshots for visual analysis\n all_screenshots = []\n # Collect intermediate steps\n intermediate_steps = []\n \n for step in range(max_steps):\n logger.info(f\"Planning step {step + 1}/{max_steps}\")\n \n # Step 1: Observe current state\n screenshot = await self.connector.get_screenshot()\n if not screenshot:\n return ToolResult(\n status=ToolStatus.ERROR,\n error=\"Failed to get screenshot for planning\",\n metadata={\"step\": step, \"action_history\": self.action_history}\n )\n \n # Collect screenshot for visual analysis\n all_screenshots.append(screenshot)\n \n # Step 2: Plan next action using LLM\n planned_action = await self._plan_next_action(\n task_description=task_description,\n screenshot=screenshot,\n action_history=self.action_history,\n )\n \n # Check if task is complete\n if planned_action[\"action_type\"] == \"DONE\":\n logger.info(\"Task marked as complete by LLM\")\n reasoning = planned_action.get(\"reasoning\", \"Task completed successfully\")\n \n intermediate_steps.append({\n \"step_number\": step + 1,\n \"action\": \"DONE\",\n \"reasoning\": reasoning,\n \"status\": \"done\",\n })\n \n return ToolResult(\n status=ToolStatus.SUCCESS,\n content=f\"Task completed: {task_description}\\n\\nFinal state: {reasoning}\",\n metadata={\n \"steps_taken\": step + 1,\n \"action_history\": self.action_history,\n \"screenshots\": all_screenshots,\n \"intermediate_steps\": intermediate_steps,\n \"final_reasoning\": reasoning,\n }\n )\n \n # Check if task failed\n if planned_action[\"action_type\"] == \"FAIL\":\n logger.warning(\"Task marked as failed by LLM\")\n reason = planned_action.get(\"reason\", \"Task cannot be completed\")\n \n intermediate_steps.append({\n \"step_number\": step + 1,\n \"action\": \"FAIL\",\n \"reasoning\": planned_action.get(\"reasoning\", \"\"),\n \"status\": \"failed\",\n })\n \n return ToolResult(\n status=ToolStatus.ERROR,\n error=reason,\n metadata={\n \"steps_taken\": step + 1,\n \"action_history\": self.action_history,\n \"screenshots\": all_screenshots,\n \"intermediate_steps\": intermediate_steps,\n }\n )\n \n # Check if action is WAIT (screenshot observation, continue to next step)\n if planned_action[\"action_type\"] == \"WAIT\":\n logger.info(\"Screenshot observation step, continuing planning loop\")\n intermediate_steps.append({\n \"step_number\": step + 1,\n \"action\": \"WAIT\",\n \"reasoning\": planned_action.get(\"reasoning\", \"\"),\n \"status\": \"observation\",\n })\n continue\n \n # Step 3: Execute the planned action\n execution_result = await self._execute_planned_action(planned_action)\n \n # Record action in history\n self.action_history.append({\n \"step\": step + 1,\n \"planned_action\": planned_action,\n \"execution_result\": execution_result,\n })\n \n intermediate_steps.append({\n \"step_number\": step + 1,\n \"action\": planned_action.get(\"action_type\", \"unknown\"),\n \"reasoning\": planned_action.get(\"reasoning\", \"\"),\n \"status\": execution_result.get(\"status\", \"unknown\"),\n })\n \n # Check execution result\n if execution_result.get(\"status\") != \"success\":\n logger.warning(f\"Action execution failed: {execution_result.get('error')}\")\n # Continue to next iteration for retry planning\n \n # Max steps reached\n return ToolResult(\n status=ToolStatus.ERROR,\n error=f\"Task incomplete after {max_steps} steps\",\n metadata={\n \"task_description\": task_description,\n \"steps_taken\": max_steps,\n \"action_history\": self.action_history,\n \"screenshots\": all_screenshots,\n \"intermediate_steps\": intermediate_steps,\n }\n )\n \n async def _plan_next_action(\n self,\n task_description: str,\n screenshot: bytes,\n action_history: list,\n ) -> Dict[str, Any]:\n \"\"\"\n Use LLM/VLM to plan the next action.\n \n This method sends:\n - Task description\n - Current screenshot (vision input)\n - Action history (context)\n - Available ACTION_SPACE\n \n And gets back a structured action plan.\n \n Args:\n task_description: The task to accomplish\n screenshot: Current desktop screenshot (PNG/JPEG bytes)\n action_history: Previously executed actions\n \n Returns:\n Dict with action_type and parameters\n \"\"\"\n if self.llm_client is None:\n # Fallback: Simple heuristic or manual mode\n logger.warning(\"No LLM client configured, using fallback mode\")\n return {\n \"action_type\": \"FAIL\",\n \"reason\": \"LLM client not configured\"\n }\n \n # Check if using Anthropic client\n try:\n from .anthropic_client import AnthropicGUIClient\n is_anthropic = isinstance(self.llm_client, AnthropicGUIClient)\n except ImportError:\n is_anthropic = False\n \n if is_anthropic:\n # Use Anthropic client\n try:\n reasoning, commands = await self.llm_client.plan_action(\n task_description=task_description,\n screenshot=screenshot,\n action_history=action_history,\n )\n \n if commands == [\"FAIL\"]:\n return {\n \"action_type\": \"FAIL\",\n \"reason\": \"Anthropic planning failed\"\n }\n \n if commands == [\"DONE\"]:\n return {\n \"action_type\": \"DONE\",\n \"reasoning\": reasoning\n }\n \n if commands == [\"SCREENSHOT\"]:\n # Screenshot is automatically handled by system\n # Continue to next planning step\n logger.info(\"LLM requested screenshot (observation step)\")\n return {\n \"action_type\": \"WAIT\",\n \"reasoning\": reasoning or \"Observing screen state\"\n }\n \n # If no commands but has reasoning, task is complete\n # (Anthropic returns text-only when task is done)\n if not commands and reasoning:\n logger.info(\"LLM returned text-only response, interpreting as task completion\")\n return {\n \"action_type\": \"DONE\",\n \"reasoning\": reasoning\n }\n \n # No commands and no reasoning = error\n if not commands:\n return {\n \"action_type\": \"FAIL\",\n \"reason\": \"No commands generated and no completion message\"\n }\n \n # Return first command (Anthropic returns pyautogui commands directly)\n return {\n \"action_type\": \"PYAUTOGUI_COMMAND\",\n \"command\": commands[0],\n \"reasoning\": reasoning\n }\n \n except Exception as e:\n logger.error(f\"Anthropic planning failed: {e}\")\n return {\n \"action_type\": \"FAIL\",\n \"reason\": f\"Planning error: {str(e)}\"\n }\n \n # Generic LLM client (for future integration with other LLMs)\n # Encode screenshot to base64 for LLM\n screenshot_b64 = base64.b64encode(screenshot).decode('utf-8')\n \n # Prepare prompt for LLM\n prompt = self._build_planning_prompt(\n task_description=task_description,\n action_history=action_history,\n )\n \n # Call LLM with vision input\n try:\n response = await self.llm_client.plan_action(\n prompt=prompt,\n image_base64=screenshot_b64,\n action_space=ACTION_SPACE,\n keyboard_keys=KEYBOARD_KEYS,\n )\n \n # Parse LLM response to action dict\n action = self._parse_llm_response(response)\n \n logger.info(f\"LLM planned action: {action['action_type']}\")\n return action\n \n except Exception as e:\n logger.error(f\"LLM planning failed: {e}\")\n return {\n \"action_type\": \"FAIL\",\n \"reason\": f\"Planning error: {str(e)}\"\n }\n \n def _build_planning_prompt(\n self,\n task_description: str,\n action_history: list,\n ) -> str:\n \"\"\"\n Build prompt for LLM planning.\n \n Args:\n task_description: The task to accomplish\n action_history: Previously executed actions\n \n Returns:\n Formatted prompt string\n \"\"\"\n prompt = f\"\"\"You are a GUI automation agent. Your task is to complete the following:\n\nTask: {task_description}\n\nYou can observe the current desktop state through the provided screenshot.\nYou must plan the next action to take from the available ACTION_SPACE.\n\nAvailable actions:\n- Mouse: MOVE_TO, CLICK, RIGHT_CLICK, DOUBLE_CLICK, DRAG_TO, SCROLL\n- Keyboard: TYPING, PRESS, KEY_DOWN, KEY_UP, HOTKEY\n- Control: WAIT, DONE, FAIL\n\n\"\"\"\n \n if action_history:\n prompt += f\"\\nPrevious actions taken ({len(action_history)}):\\n\"\n for i, action in enumerate(action_history[-5:], 1): # Last 5 actions\n prompt += f\"{i}. {action['planned_action']['action_type']}\"\n if 'parameters' in action['planned_action']:\n prompt += f\" - {action['planned_action']['parameters']}\"\n prompt += \"\\n\"\n \n prompt += \"\"\"\nBased on the screenshot and task, output the next action in JSON format:\n{\n \"action_type\": \"ACTION_TYPE\",\n \"parameters\": {...},\n \"reasoning\": \"Why this action is needed\"\n}\n\nIf the task is complete, output: {\"action_type\": \"DONE\"}\nIf the task cannot be completed, output: {\"action_type\": \"FAIL\", \"reason\": \"explanation\"}\n\"\"\"\n \n return prompt\n \n def _parse_llm_response(self, response: str) -> Dict[str, Any]:\n \"\"\"\n Parse LLM response to extract action.\n \n Args:\n response: LLM response (should be JSON)\n \n Returns:\n Action dict with action_type and parameters\n \"\"\"\n import json\n \n try:\n # Try to parse as JSON\n action = json.loads(response)\n \n # Validate action\n if \"action_type\" not in action:\n raise ValueError(\"Missing action_type in LLM response\")\n \n return action\n \n except json.JSONDecodeError:\n logger.error(f\"Failed to parse LLM response as JSON: {response[:200]}\")\n return {\n \"action_type\": \"FAIL\",\n \"reason\": \"Invalid LLM response format\"\n }\n \n async def _execute_planned_action(\n self,\n action: Dict[str, Any]\n ) -> Dict[str, Any]:\n \"\"\"\n Execute a planned action through the connector.\n \n Args:\n action: Action dict with action_type and parameters\n \n Returns:\n Execution result dict\n \"\"\"\n action_type = action[\"action_type\"]\n \n # Handle Anthropic's direct pyautogui commands\n if action_type == \"PYAUTOGUI_COMMAND\":\n command = action.get(\"command\", \"\")\n logger.info(f\"Executing pyautogui command: {command}\")\n \n try:\n result = await self.connector.execute_python_command(command)\n return {\n \"status\": \"success\" if result else \"error\",\n \"action_type\": action_type,\n \"command\": command,\n \"result\": result\n }\n except Exception as e:\n logger.error(f\"Command execution error: {e}\")\n return {\n \"status\": \"error\",\n \"action_type\": action_type,\n \"error\": str(e)\n }\n \n # Handle standard action space commands\n parameters = action.get(\"parameters\", {})\n logger.info(f\"Executing action: {action_type}\")\n \n try:\n result = await self.connector.execute_action(action_type, parameters)\n return result\n \n except Exception as e:\n logger.error(f\"Action execution error: {e}\")\n return {\n \"status\": \"error\",\n \"action_type\": action_type,\n \"error\": str(e)\n }\n \n # Helper methods for direct action execution\n \n async def execute_action(\n self,\n action_type: str,\n parameters: Dict[str, Any]\n ) -> ToolResult:\n \"\"\"\n Direct action execution (bypass LLM planning).\n \n Args:\n action_type: Action type from ACTION_SPACE\n parameters: Action parameters\n \n Returns:\n ToolResult with execution status\n \"\"\"\n result = await self.connector.execute_action(action_type, parameters)\n \n if result.get(\"status\") == \"success\":\n return ToolResult(\n status=ToolStatus.SUCCESS,\n content=f\"Executed {action_type}\",\n metadata=result\n )\n else:\n return ToolResult(\n status=ToolStatus.ERROR,\n error=result.get(\"error\", \"Unknown error\"),\n metadata=result\n )\n \n async def get_screenshot(self) -> ToolResult:\n \"\"\"Get current desktop screenshot.\"\"\"\n screenshot = await self.connector.get_screenshot()\n if screenshot:\n return ToolResult(\n status=ToolStatus.SUCCESS,\n content=screenshot,\n metadata={\"type\": \"screenshot\", \"size\": len(screenshot)}\n )\n else:\n return ToolResult(\n status=ToolStatus.ERROR,\n error=\"Failed to capture screenshot\"\n )\n \n async def _record_intermediate_step(\n self,\n step_number: int,\n planned_action: Dict[str, Any],\n execution_result: Dict[str, Any],\n screenshot: bytes,\n task_description: str,\n ):\n \"\"\"\n Record an intermediate step of GUI agent execution.\n \n This method records each planning-action cycle to the recording system,\n providing detailed traces of GUI agent's decision-making process.\n \n Args:\n step_number: Step number in the execution sequence\n planned_action: Action planned by LLM\n execution_result: Result of executing the action\n screenshot: Screenshot before executing the action\n task_description: Overall task description\n \"\"\"\n # Try to get recording_manager dynamically if not set at initialization\n recording_manager = self.recording_manager\n if not recording_manager and hasattr(self, '_runtime_info') and self._runtime_info:\n # Try to get from grounding_client\n grounding_client = self._runtime_info.grounding_client\n if grounding_client and hasattr(grounding_client, 'recording_manager'):\n recording_manager = grounding_client.recording_manager\n logger.debug(f\"Step {step_number}: Dynamically retrieved recording_manager from grounding_client\")\n \n if not recording_manager:\n logger.debug(f\"Step {step_number}: No recording_manager available, skipping intermediate step recording\")\n return\n \n # Check if recording is active\n try:\n from anytool.recording.manager import RecordingManager\n if not RecordingManager.is_recording():\n logger.debug(f\"Step {step_number}: RecordingManager not started\")\n return\n except Exception as e:\n logger.debug(f\"Step {step_number}: Failed to check recording status: {e}\")\n return\n \n # Check if recorder is initialized\n if not hasattr(recording_manager, '_recorder') or not recording_manager._recorder:\n logger.warning(f\"Step {step_number}: recording_manager._recorder not initialized\")\n return\n \n # Build command string for display\n action_type = planned_action.get(\"action_type\", \"unknown\")\n command = self._format_action_command(planned_action)\n \n # Build result summary\n status = execution_result.get(\"status\", \"unknown\")\n is_success = status in (\"success\", \"done\", \"observation\")\n \n # Build result content\n if status == \"done\":\n result_content = f\"Task completed at step {step_number}\"\n elif status == \"failed\":\n result_content = execution_result.get(\"message\", \"Task failed\")\n elif status == \"observation\":\n result_content = execution_result.get(\"message\", \"Screenshot observation\")\n else:\n result_content = execution_result.get(\"result\", execution_result.get(\"message\", str(execution_result)))\n \n # Build parameters for recording\n parameters = {\n \"task_description\": task_description,\n \"step_number\": step_number,\n \"action_type\": action_type,\n \"planned_action\": planned_action,\n }\n \n # Record to trajectory recorder (handles screenshot saving)\n try:\n await recording_manager._recorder.record_step(\n backend=\"gui\",\n tool=\"gui_agent_step\",\n command=command,\n result={\n \"status\": \"success\" if is_success else \"error\",\n \"output\": str(result_content)[:200], # Truncate long outputs\n },\n parameters=parameters,\n screenshot=screenshot,\n extra={\n \"gui_step_number\": step_number,\n \"reasoning\": planned_action.get(\"reasoning\", \"\"),\n }\n )\n \n logger.info(f\"โ Recorded GUI intermediate step {step_number}: {command}\")\n \n except Exception as e:\n logger.error(f\"โ Failed to record intermediate step {step_number}: {e}\", exc_info=True)\n \n def _format_action_command(self, planned_action: Dict[str, Any]) -> str:\n \"\"\"\n Format planned action into a human-readable command string.\n \n Args:\n planned_action: Action dictionary from LLM planning\n \n Returns:\n Formatted command string\n \"\"\"\n action_type = planned_action.get(\"action_type\", \"unknown\")\n \n # Handle special action types\n if action_type == \"DONE\":\n return \"DONE (task completed)\"\n elif action_type == \"FAIL\":\n reason = planned_action.get(\"reason\", \"unknown\")\n return f\"FAIL ({reason})\"\n elif action_type == \"WAIT\":\n return \"WAIT (screenshot observation)\"\n \n # Handle PyAutoGUI commands\n elif action_type == \"PYAUTOGUI_COMMAND\":\n command = planned_action.get(\"command\", \"\")\n # Truncate long commands\n if len(command) > 100:\n return command[:100] + \"...\"\n return command\n \n # Handle standard action space commands\n else:\n parameters = planned_action.get(\"parameters\", {})\n if parameters:\n # Format first 2 parameters\n param_items = list(parameters.items())[:2]\n param_str = \", \".join([f\"{k}={v}\" for k, v in param_items])\n return f\"{action_type}({param_str})\"\n else:\n return action_type"
},
{
"path": "anytool/grounding/backends/gui/transport/actions.py",
"content": "\"\"\"\nGUI Action Space Definitions.\n\"\"\"\nfrom typing import Dict, Any\n\n# Screen resolution constants\nX_MAX = 1920\nY_MAX = 1080\n\n# Keyboard keys constants\nKEYBOARD_KEYS = [\n '\\t', '\\n', '\\r', ' ', '!', '\"', '#', '$', '%', '&', \"'\", '(', ')', '*', '+', ',', '-', '.', '/', \n '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', ':', ';', '<', '=', '>', '?', '@', \n '[', '\\\\', ']', '^', '_', '`', \n 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', \n '{', '|', '}', '~', \n 'accept', 'add', 'alt', 'altleft', 'altright', 'apps', 'backspace', \n 'browserback', 'browserfavorites', 'browserforward', 'browserhome', 'browserrefresh', 'browsersearch', 'browserstop', \n 'capslock', 'clear', 'convert', 'ctrl', 'ctrlleft', 'ctrlright', 'decimal', 'del', 'delete', 'divide', \n 'down', 'end', 'enter', 'esc', 'escape', 'execute', \n 'f1', 'f10', 'f11', 'f12', 'f13', 'f14', 'f15', 'f16', 'f17', 'f18', 'f19', \n 'f2', 'f20', 'f21', 'f22', 'f23', 'f24', 'f3', 'f4', 'f5', 'f6', 'f7', 'f8', 'f9', \n 'final', 'fn', 'hanguel', 'hangul', 'hanja', 'help', 'home', 'insert', 'junja', 'kana', 'kanji', \n 'launchapp1', 'launchapp2', 'launchmail', 'launchmediaselect', 'left', 'modechange', 'multiply', \n 'nexttrack', 'nonconvert', 'num0', 'num1', 'num2', 'num3', 'num4', 'num5', 'num6', 'num7', 'num8', 'num9', \n 'numlock', 'pagedown', 'pageup', 'pause', 'pgdn', 'pgup', 'playpause', 'prevtrack', 'print', 'printscreen', \n 'prntscrn', 'prtsc', 'prtscr', 'return', 'right', 'scrolllock', 'select', 'separator', \n 'shift', 'shiftleft', 'shiftright', 'sleep', 'stop', 'subtract', 'tab', 'up', \n 'volumedown', 'volumemute', 'volumeup', 'win', 'winleft', 'winright', 'yen', \n 'command', 'option', 'optionleft', 'optionright'\n]\n\n# Action Space Definition\nACTION_SPACE = [\n {\n \"action_type\": \"MOVE_TO\",\n \"note\": \"move the cursor to the specified position\",\n \"parameters\": {\n \"x\": {\"type\": float, \"range\": [0, X_MAX], \"optional\": False},\n \"y\": {\"type\": float, \"range\": [0, Y_MAX], \"optional\": False},\n }\n },\n {\n \"action_type\": \"CLICK\",\n \"note\": \"click the left button if button not specified, otherwise click the specified button\",\n \"parameters\": {\n \"button\": {\"type\": str, \"range\": [\"left\", \"right\", \"middle\"], \"optional\": True},\n \"x\": {\"type\": float, \"range\": [0, X_MAX], \"optional\": True},\n \"y\": {\"type\": float, \"range\": [0, Y_MAX], \"optional\": True},\n \"num_clicks\": {\"type\": int, \"range\": [1, 2, 3], \"optional\": True},\n }\n },\n {\n \"action_type\": \"MOUSE_DOWN\",\n \"note\": \"press the mouse button\",\n \"parameters\": {\n \"button\": {\"type\": str, \"range\": [\"left\", \"right\", \"middle\"], \"optional\": True}\n }\n },\n {\n \"action_type\": \"MOUSE_UP\",\n \"note\": \"release the mouse button\",\n \"parameters\": {\n \"button\": {\"type\": str, \"range\": [\"left\", \"right\", \"middle\"], \"optional\": True}\n }\n },\n {\n \"action_type\": \"RIGHT_CLICK\",\n \"note\": \"right click at position\",\n \"parameters\": {\n \"x\": {\"type\": float, \"range\": [0, X_MAX], \"optional\": True},\n \"y\": {\"type\": float, \"range\": [0, Y_MAX], \"optional\": True}\n }\n },\n {\n \"action_type\": \"DOUBLE_CLICK\",\n \"note\": \"double click at position\",\n \"parameters\": {\n \"x\": {\"type\": float, \"range\": [0, X_MAX], \"optional\": True},\n \"y\": {\"type\": float, \"range\": [0, Y_MAX], \"optional\": True}\n }\n },\n {\n \"action_type\": \"DRAG_TO\",\n \"note\": \"drag the cursor to position\",\n \"parameters\": {\n \"x\": {\"type\": float, \"range\": [0, X_MAX], \"optional\": False},\n \"y\": {\"type\": float, \"range\": [0, Y_MAX], \"optional\": False}\n }\n },\n {\n \"action_type\": \"SCROLL\",\n \"note\": \"scroll the mouse wheel\",\n \"parameters\": {\n \"dx\": {\"type\": int, \"range\": None, \"optional\": False},\n \"dy\": {\"type\": int, \"range\": None, \"optional\": False}\n }\n },\n {\n \"action_type\": \"TYPING\",\n \"note\": \"type the specified text\",\n \"parameters\": {\n \"text\": {\"type\": str, \"range\": None, \"optional\": False}\n }\n },\n {\n \"action_type\": \"PRESS\",\n \"note\": \"press the specified key\",\n \"parameters\": {\n \"key\": {\"type\": str, \"range\": KEYBOARD_KEYS, \"optional\": False}\n }\n },\n {\n \"action_type\": \"KEY_DOWN\",\n \"note\": \"press down the specified key\",\n \"parameters\": {\n \"key\": {\"type\": str, \"range\": KEYBOARD_KEYS, \"optional\": False}\n }\n },\n {\n \"action_type\": \"KEY_UP\",\n \"note\": \"release the specified key\",\n \"parameters\": {\n \"key\": {\"type\": str, \"range\": KEYBOARD_KEYS, \"optional\": False}\n }\n },\n {\n \"action_type\": \"HOTKEY\",\n \"note\": \"press key combination\",\n \"parameters\": {\n \"keys\": {\"type\": list, \"range\": [KEYBOARD_KEYS], \"optional\": False}\n }\n },\n {\n \"action_type\": \"WAIT\",\n \"note\": \"wait until next action\",\n },\n {\n \"action_type\": \"FAIL\",\n \"note\": \"mark task as failed\",\n },\n {\n \"action_type\": \"DONE\",\n \"note\": \"mark task as done\",\n }\n]\n\n\ndef build_pyautogui_command(action_type: str, parameters: Dict[str, Any]) -> str:\n \"\"\"\n Build pyautogui command from action type and parameters.\n \n Args:\n action_type: Type of action (e.g., 'CLICK', 'TYPING')\n parameters: Action parameters\n \n Returns:\n Python command string\n \"\"\"\n if action_type == \"MOVE_TO\":\n if \"x\" in parameters and \"y\" in parameters:\n x, y = parameters[\"x\"], parameters[\"y\"]\n return f\"pyautogui.moveTo({x}, {y}, 0.5, pyautogui.easeInQuad)\"\n else:\n return \"pyautogui.moveTo()\"\n \n elif action_type == \"CLICK\":\n button = parameters.get(\"button\", \"left\")\n num_clicks = parameters.get(\"num_clicks\", 1)\n \n if \"x\" in parameters and \"y\" in parameters:\n x, y = parameters[\"x\"], parameters[\"y\"]\n return f\"pyautogui.click(button='{button}', x={x}, y={y}, clicks={num_clicks})\"\n else:\n return f\"pyautogui.click(button='{button}', clicks={num_clicks})\"\n \n elif action_type == \"MOUSE_DOWN\":\n button = parameters.get(\"button\", \"left\")\n return f\"pyautogui.mouseDown(button='{button}')\"\n \n elif action_type == \"MOUSE_UP\":\n button = parameters.get(\"button\", \"left\")\n return f\"pyautogui.mouseUp(button='{button}')\"\n \n elif action_type == \"RIGHT_CLICK\":\n if \"x\" in parameters and \"y\" in parameters:\n x, y = parameters[\"x\"], parameters[\"y\"]\n return f\"pyautogui.rightClick(x={x}, y={y})\"\n else:\n return \"pyautogui.rightClick()\"\n \n elif action_type == \"DOUBLE_CLICK\":\n if \"x\" in parameters and \"y\" in parameters:\n x, y = parameters[\"x\"], parameters[\"y\"]\n return f\"pyautogui.doubleClick(x={x}, y={y})\"\n else:\n return \"pyautogui.doubleClick()\"\n \n elif action_type == \"DRAG_TO\":\n if \"x\" in parameters and \"y\" in parameters:\n x, y = parameters[\"x\"], parameters[\"y\"]\n return f\"pyautogui.dragTo({x}, {y}, 1.0, button='left')\"\n \n elif action_type == \"SCROLL\":\n dx = parameters.get(\"dx\", 0)\n dy = parameters.get(\"dy\", 0)\n return f\"pyautogui.scroll({dy})\"\n \n elif action_type == \"TYPING\":\n text = parameters.get(\"text\", \"\")\n # Use repr() for proper string escaping\n return f\"pyautogui.typewrite({repr(text)})\"\n \n elif action_type == \"PRESS\":\n key = parameters.get(\"key\", \"\")\n return f\"pyautogui.press('{key}')\"\n \n elif action_type == \"KEY_DOWN\":\n key = parameters.get(\"key\", \"\")\n return f\"pyautogui.keyDown('{key}')\"\n \n elif action_type == \"KEY_UP\":\n key = parameters.get(\"key\", \"\")\n return f\"pyautogui.keyUp('{key}')\"\n \n elif action_type == \"HOTKEY\":\n keys = parameters.get(\"keys\", [])\n if keys:\n keys_str = \", \".join([f\"'{k}'\" for k in keys])\n return f\"pyautogui.hotkey({keys_str})\"\n \n return None"
},
{
"path": "anytool/grounding/backends/gui/transport/connector.py",
"content": "import asyncio\nimport re\nfrom typing import Any, Dict, Optional\nfrom anytool.grounding.core.transport.connectors import AioHttpConnector\nfrom .actions import build_pyautogui_command, KEYBOARD_KEYS\nfrom anytool.utils.logging import Logger\n\nlogger = Logger.get_logger(__name__)\n\n\nclass GUIConnector(AioHttpConnector):\n \"\"\"\n Connector for desktop_env HTTP API.\n Provides action execution and observation methods.\n \"\"\"\n \n def __init__(\n self,\n vm_ip: str,\n server_port: int = 5000,\n timeout: int = 90,\n retry_times: int = 3,\n retry_interval: float = 5.0,\n pkgs_prefix: str = \"import pyautogui; import time; pyautogui.FAILSAFE = False; {command}\",\n ):\n \"\"\"\n Initialize GUI connector.\n \n Args:\n vm_ip: IP address of the VM running desktop_env\n server_port: Port of the desktop_env HTTP server\n timeout: Request timeout in seconds\n retry_times: Number of retries for failed requests\n retry_interval: Interval between retries in seconds\n pkgs_prefix: Python command prefix for pyautogui setup\n \"\"\"\n base_url = f\"http://{vm_ip}:{server_port}\"\n super().__init__(base_url, timeout=timeout)\n \n self.vm_ip = vm_ip\n self.server_port = server_port\n self.retry_times = retry_times\n self.retry_interval = retry_interval\n self.pkgs_prefix = pkgs_prefix\n self.timeout = timeout\n \n async def _retry_invoke(\n self, \n operation_name: str,\n operation_func,\n *args,\n **kwargs\n ):\n \"\"\"\n Execute operation with retry logic.\n \n Args:\n operation_name: Name of operation for logging\n operation_func: Async function to execute\n *args: Positional arguments for operation_func\n **kwargs: Keyword arguments for operation_func\n \n Returns:\n Operation result\n \n Raises:\n Exception: Last exception after all retries fail\n \"\"\"\n last_exc: Exception | None = None\n \n for attempt in range(1, self.retry_times + 1):\n try:\n result = await operation_func(*args, **kwargs)\n logger.debug(\"%s executed successfully (attempt %d/%d)\", operation_name, attempt, self.retry_times)\n return result\n except asyncio.TimeoutError as exc:\n logger.error(\"%s timed out\", operation_name)\n raise RuntimeError(f\"{operation_name} timed out after {self.timeout} seconds\") from exc\n except Exception as exc:\n last_exc = exc\n if attempt == self.retry_times:\n break\n logger.warning(\n \"%s failed (attempt %d/%d): %s, retrying in %.1f seconds...\", \n operation_name, attempt, self.retry_times, exc, self.retry_interval\n )\n await asyncio.sleep(self.retry_interval)\n \n error_msg = f\"{operation_name} failed after {self.retry_times} retries\"\n logger.error(error_msg)\n raise last_exc or RuntimeError(error_msg)\n \n @staticmethod\n def _is_valid_image_response(content_type: str, data: Optional[bytes]) -> bool:\n \"\"\"Validate image response using magic bytes.\"\"\"\n if not isinstance(data, (bytes, bytearray)) or not data:\n return False\n # PNG magic\n if len(data) >= 8 and data[:8] == b\"\\x89PNG\\r\\n\\x1a\\n\":\n return True\n # JPEG magic\n if len(data) >= 3 and data[:3] == b\"\\xff\\xd8\\xff\":\n return True\n # Fallback to content-type\n if content_type and (\"image/png\" in content_type or \"image/jpeg\" in content_type):\n return True\n return False\n \n @staticmethod\n def _fix_pyautogui_less_than_bug(command: str) -> str:\n \"\"\"\n Fix PyAutoGUI '<' character bug by converting it to hotkey(\"shift\", ',') calls.\n \n This fixes the known PyAutoGUI issue where typing '<' produces '>' instead.\n References:\n - https://github.com/asweigart/pyautogui/issues/198\n - https://github.com/xlang-ai/OSWorld/issues/257\n \n Args:\n command (str): The original pyautogui command\n \n Returns:\n str: The fixed command with '<' characters handled properly\n \"\"\"\n # Pattern to match press('<') or press('\\u003c') calls \n press_pattern = r'pyautogui\\.press\\([\"\\'](?:<|\\\\u003c)[\"\\']\\)'\n\n # Handle press('<') calls\n def replace_press_less_than(match):\n return 'pyautogui.hotkey(\"shift\", \",\")'\n \n # First handle press('<') calls\n command = re.sub(press_pattern, replace_press_less_than, command)\n\n # Pattern to match typewrite calls with quoted strings\n typewrite_pattern = r'pyautogui\\.typewrite\\(([\"\\'])(.*?)\\1\\)'\n \n # Then handle typewrite calls\n def process_typewrite_match(match):\n quote_char = match.group(1)\n content = match.group(2)\n \n # Preprocess: Try to decode Unicode escapes like \\u003c to actual '<'\n # This handles cases where '<' is represented as escaped Unicode\n try:\n # Attempt to decode unicode escapes\n decoded_content = content.encode('utf-8').decode('unicode_escape')\n content = decoded_content\n except UnicodeDecodeError:\n # If decoding fails, proceed with original content to avoid breaking existing logic\n pass # Graceful degradation - fall back to original content if decoding fails\n \n # Check if content contains '<'\n if '<' not in content:\n return match.group(0)\n \n # Split by '<' and rebuild\n parts = content.split('<')\n result_parts = []\n \n for i, part in enumerate(parts):\n if i == 0:\n # First part\n if part:\n result_parts.append(f\"pyautogui.typewrite({quote_char}{part}{quote_char})\")\n else:\n # Add hotkey for '<' and then typewrite for the rest\n result_parts.append('pyautogui.hotkey(\"shift\", \",\")')\n if part:\n result_parts.append(f\"pyautogui.typewrite({quote_char}{part}{quote_char})\")\n \n return '; '.join(result_parts)\n \n command = re.sub(typewrite_pattern, process_typewrite_match, command)\n \n return command\n \n async def get_screen_size(self) -> Optional[tuple[int, int]]:\n \"\"\"\n Get actual screen size from desktop environment using pyautogui.\n \n Returns:\n (width, height) tuple, or None on failure\n \"\"\"\n try:\n command = \"print(pyautogui.size())\"\n result = await self.execute_python_command(command)\n if result and result.get(\"status\") == \"success\":\n output = result.get(\"output\", \"\")\n # Parse output like \"Size(width=2880, height=1800)\"\n import re\n match = re.search(r'width=(\\d+).*height=(\\d+)', output)\n if match:\n width = int(match.group(1))\n height = int(match.group(2))\n logger.info(f\"Detected screen size: {width}x{height}\")\n return (width, height)\n logger.warning(f\"Failed to detect screen size, output: {result}\")\n return None\n except Exception as e:\n logger.error(f\"Failed to get screen size: {e}\")\n return None\n \n async def get_screenshot(self) -> Optional[bytes]:\n \"\"\"\n Get screenshot from desktop environment.\n \n Returns:\n Screenshot image bytes (PNG/JPEG), or None on failure\n \"\"\"\n try:\n async def _get():\n response = await self._request(\"GET\", \"/screenshot\", timeout=10)\n if response.status == 200:\n content_type = response.headers.get(\"Content-Type\", \"\")\n content = await response.read()\n if self._is_valid_image_response(content_type, content):\n return content\n else:\n raise ValueError(\"Invalid screenshot format\")\n else:\n raise RuntimeError(f\"HTTP {response.status}\")\n \n return await self._retry_invoke(\"get_screenshot\", _get)\n except Exception as e:\n logger.error(f\"Failed to get screenshot: {e}\")\n return None\n \n async def execute_python_command(self, command: str) -> Optional[Dict[str, Any]]:\n \"\"\"\n Execute a Python command on desktop environment.\n Used for pyautogui commands.\n \n Args:\n command: Python command to execute\n \n Returns:\n Response dict with execution result, or None on failure\n \"\"\"\n try:\n # Apply '<' character fix for PyAutoGUI bug\n fixed_command = self._fix_pyautogui_less_than_bug(command)\n \n command_list = [\"python\", \"-c\", self.pkgs_prefix.format(command=fixed_command)]\n payload = {\"command\": command_list, \"shell\": False}\n \n async def _execute():\n return await self.post_json(\"/execute\", payload)\n \n return await self._retry_invoke(\"execute_python_command\", _execute)\n except Exception as e:\n logger.error(f\"Failed to execute command: {e}\")\n return None\n \n async def execute_action(self, action_type: str, parameters: Dict[str, Any] = None) -> Dict[str, Any]:\n \"\"\"\n Execute a desktop action.\n This is the main method for action space execution.\n \n Args:\n action_type: Action type (e.g., 'CLICK', 'TYPING')\n parameters: Action parameters\n \n Returns:\n Result dict with execution status\n \"\"\"\n parameters = parameters or {}\n \n # Handle control actions\n if action_type in ['WAIT', 'FAIL', 'DONE']:\n return {\n \"status\": \"success\",\n \"action_type\": action_type,\n \"message\": f\"Control action {action_type} acknowledged\"\n }\n \n # Validate keyboard keys\n if action_type in ['PRESS', 'KEY_DOWN', 'KEY_UP']:\n key = parameters.get('key')\n if key and key not in KEYBOARD_KEYS:\n return {\n \"status\": \"error\",\n \"action_type\": action_type,\n \"error\": f\"Invalid key: {key}. Must be in supported keyboard keys.\"\n }\n \n if action_type == 'HOTKEY':\n keys = parameters.get('keys', [])\n invalid_keys = [k for k in keys if k not in KEYBOARD_KEYS]\n if invalid_keys:\n return {\n \"status\": \"error\",\n \"action_type\": action_type,\n \"error\": f\"Invalid keys: {invalid_keys}\"\n }\n \n # Build pyautogui command\n command = build_pyautogui_command(action_type, parameters)\n \n if command is None:\n return {\n \"status\": \"error\",\n \"action_type\": action_type,\n \"error\": f\"Unsupported action type: {action_type}\"\n }\n \n # Execute command\n result = await self.execute_python_command(command)\n \n if result:\n return {\n \"status\": \"success\",\n \"action_type\": action_type,\n \"parameters\": parameters,\n \"result\": result\n }\n else:\n return {\n \"status\": \"error\",\n \"action_type\": action_type,\n \"parameters\": parameters,\n \"error\": \"Command execution failed\"\n }\n \n async def get_accessibility_tree(self, max_depth: int = 5) -> Optional[Dict[str, Any]]:\n \"\"\"\n Get accessibility tree from desktop environment.\n \n Args:\n max_depth: Maximum depth of accessibility tree traversal\n \n Returns:\n Accessibility tree as dict, or None on failure\n \"\"\"\n try:\n async def _get():\n response = await self._request(\"GET\", \"/accessibility\", timeout=10)\n if response.status == 200:\n data = await response.json()\n return data.get(\"AT\")\n else:\n raise RuntimeError(f\"HTTP {response.status}\")\n \n return await self._retry_invoke(\"get_accessibility_tree\", _get)\n except Exception as e:\n logger.error(f\"Failed to get accessibility tree: {e}\")\n return None\n\n async def get_cursor_position(self) -> Optional[tuple[int, int]]:\n \"\"\"\n Get current mouse cursor position.\n Useful for GUI debugging and relative positioning.\n \n Returns:\n (x, y) tuple, or None on failure\n \"\"\"\n try:\n async def _get():\n result = await self.get_json(\"/cursor_position\")\n return (result.get(\"x\"), result.get(\"y\"))\n \n return await self._retry_invoke(\"get_cursor_position\", _get)\n except Exception as e:\n logger.error(f\"Failed to get cursor position: {e}\")\n return None\n \n async def invoke(self, name: str, params: dict[str, Any]) -> Any:\n \"\"\"\n Unified RPC entry for operations.\n Required by BaseConnector.\n \n Args:\n name: Operation name (action_type or observation method)\n params: Operation parameters\n \n Returns:\n Operation result\n \"\"\"\n # Handle observation methods\n if name == \"screenshot\":\n return await self.get_screenshot()\n elif name == \"accessibility_tree\":\n max_depth = params.get(\"max_depth\", 5) if params else 5\n return await self.get_accessibility_tree(max_depth)\n elif name == \"cursor_position\":\n return await self.get_cursor_position()\n else:\n # Treat as action\n return await self.execute_action(name.upper(), params or {})"
},
{
"path": "anytool/grounding/backends/gui/transport/local_connector.py",
"content": "\"\"\"\nLocal GUI Connector โ execute GUI operations directly in-process.\n\nThis connector has the **same public API** as GUIConnector (HTTP version)\nbut uses local pyautogui / ScreenshotHelper / AccessibilityHelper,\nremoving the need for a local_server.\n\nReturn format is kept identical so that GUISession / GUIAgentTool\nwork without any changes.\n\"\"\"\n\nimport asyncio\nimport os\nimport platform\nimport re\nimport tempfile\nimport uuid\nfrom typing import Any, Dict, Optional\n\nfrom anytool.grounding.core.transport.connectors.base import BaseConnector\nfrom anytool.grounding.core.transport.task_managers.noop import NoOpConnectionManager\nfrom anytool.utils.logging import Logger\n\nlogger = Logger.get_logger(__name__)\n\nplatform_name = platform.system()\n\n\nclass LocalGUIConnector(BaseConnector[Any]):\n \"\"\"\n GUI connector that runs desktop automation **locally** using pyautogui /\n ScreenshotHelper / AccessibilityHelper, bypassing the Flask local_server.\n\n Public API is compatible with ``GUIConnector`` so that ``GUISession``\n works without modification.\n \"\"\"\n\n def __init__(\n self,\n timeout: int = 90,\n retry_times: int = 3,\n retry_interval: float = 5.0,\n pkgs_prefix: str = \"import pyautogui; import time; pyautogui.FAILSAFE = False; {command}\",\n ):\n super().__init__(NoOpConnectionManager())\n self.timeout = timeout\n self.retry_times = retry_times\n self.retry_interval = retry_interval\n self.pkgs_prefix = pkgs_prefix\n\n # Compatibility attributes expected by GUISession\n self.vm_ip = \"localhost\"\n self.server_port = 0\n self.base_url = \"local://localhost\"\n\n # Lazy-initialized helpers (avoid import side effects at class load)\n self._screenshot_helper = None\n self._accessibility_helper = None\n\n def _get_screenshot_helper(self):\n if self._screenshot_helper is None:\n from anytool.local_server.utils import ScreenshotHelper\n self._screenshot_helper = ScreenshotHelper()\n return self._screenshot_helper\n\n def _get_accessibility_helper(self):\n if self._accessibility_helper is None:\n from anytool.local_server.utils import AccessibilityHelper\n self._accessibility_helper = AccessibilityHelper()\n return self._accessibility_helper\n\n # ------------------------------------------------------------------\n # connect / disconnect\n # ------------------------------------------------------------------\n\n async def connect(self) -> None:\n \"\"\"No real connection for local mode.\"\"\"\n if self._connected:\n return\n await super().connect()\n logger.info(\"LocalGUIConnector: ready (local mode, no server required)\")\n\n # ------------------------------------------------------------------\n # Retry wrapper (same interface as GUIConnector._retry_invoke)\n # ------------------------------------------------------------------\n\n async def _retry_invoke(\n self,\n operation_name: str,\n operation_func,\n *args,\n **kwargs,\n ):\n last_exc: Exception | None = None\n for attempt in range(1, self.retry_times + 1):\n try:\n result = await operation_func(*args, **kwargs)\n logger.debug(\n \"%s executed successfully (attempt %d/%d)\",\n operation_name, attempt, self.retry_times,\n )\n return result\n except asyncio.TimeoutError as exc:\n logger.error(\"%s timed out\", operation_name)\n raise RuntimeError(\n f\"{operation_name} timed out after {self.timeout} seconds\"\n ) from exc\n except Exception as exc:\n last_exc = exc\n if attempt == self.retry_times:\n break\n logger.warning(\n \"%s failed (attempt %d/%d): %s, retrying in %.1f seconds...\",\n operation_name, attempt, self.retry_times, exc, self.retry_interval,\n )\n await asyncio.sleep(self.retry_interval)\n\n error_msg = f\"{operation_name} failed after {self.retry_times} retries\"\n logger.error(error_msg)\n raise last_exc or RuntimeError(error_msg)\n\n # ------------------------------------------------------------------\n # PyAutoGUI '<' bug fix (same as GUIConnector)\n # ------------------------------------------------------------------\n\n @staticmethod\n def _fix_pyautogui_less_than_bug(command: str) -> str:\n \"\"\"Fix PyAutoGUI '<' character bug.\"\"\"\n press_pattern = r'pyautogui\\.press\\([\"\\'](?:<|\\\\u003c)[\"\\']\\)'\n\n def replace_press_less_than(match):\n return 'pyautogui.hotkey(\"shift\", \",\")'\n\n command = re.sub(press_pattern, replace_press_less_than, command)\n\n typewrite_pattern = r'pyautogui\\.typewrite\\(([\"\\'])(.*?)\\1\\)'\n\n def process_typewrite_match(match):\n quote_char = match.group(1)\n content = match.group(2)\n try:\n decoded_content = content.encode(\"utf-8\").decode(\"unicode_escape\")\n content = decoded_content\n except UnicodeDecodeError:\n pass\n if \"<\" not in content:\n return match.group(0)\n parts = content.split(\"<\")\n result_parts = []\n for i, part in enumerate(parts):\n if i == 0:\n if part:\n result_parts.append(\n f\"pyautogui.typewrite({quote_char}{part}{quote_char})\"\n )\n else:\n result_parts.append('pyautogui.hotkey(\"shift\", \",\")')\n if part:\n result_parts.append(\n f\"pyautogui.typewrite({quote_char}{part}{quote_char})\"\n )\n return \"; \".join(result_parts)\n\n command = re.sub(typewrite_pattern, process_typewrite_match, command)\n return command\n\n # ------------------------------------------------------------------\n # Image response validation (same as GUIConnector)\n # ------------------------------------------------------------------\n\n @staticmethod\n def _is_valid_image_response(content_type: str, data: Optional[bytes]) -> bool:\n if not isinstance(data, (bytes, bytearray)) or not data:\n return False\n if len(data) >= 8 and data[:8] == b\"\\x89PNG\\r\\n\\x1a\\n\":\n return True\n if len(data) >= 3 and data[:3] == b\"\\xff\\xd8\\xff\":\n return True\n if content_type and (\"image/png\" in content_type or \"image/jpeg\" in content_type):\n return True\n return False\n\n # ------------------------------------------------------------------\n # Public API (same signatures as GUIConnector)\n # ------------------------------------------------------------------\n\n async def get_screen_size(self) -> Optional[tuple[int, int]]:\n \"\"\"Get screen size using pyautogui.\"\"\"\n try:\n command = \"print(pyautogui.size())\"\n result = await self.execute_python_command(command)\n if result and result.get(\"status\") == \"success\":\n output = result.get(\"output\", \"\")\n match = re.search(r\"width=(\\d+).*height=(\\d+)\", output)\n if match:\n width = int(match.group(1))\n height = int(match.group(2))\n logger.info(\"Detected screen size: %dx%d\", width, height)\n return (width, height)\n logger.warning(\"Failed to detect screen size, output: %s\", result)\n return None\n except Exception as e:\n logger.error(\"Failed to get screen size: %s\", e)\n return None\n\n async def get_screenshot(self) -> Optional[bytes]:\n \"\"\"Capture screenshot locally using ScreenshotHelper.\"\"\"\n try:\n async def _get():\n helper = self._get_screenshot_helper()\n tmp_path = os.path.join(\n tempfile.gettempdir(), f\"screenshot_{uuid.uuid4().hex}.png\"\n )\n if helper.capture(tmp_path, with_cursor=True):\n with open(tmp_path, \"rb\") as f:\n data = f.read()\n os.remove(tmp_path)\n return data\n else:\n raise RuntimeError(\"Screenshot capture failed\")\n\n return await self._retry_invoke(\"get_screenshot\", _get)\n except Exception as e:\n logger.error(\"Failed to get screenshot: %s\", e)\n return None\n\n async def execute_python_command(self, command: str) -> Optional[Dict[str, Any]]:\n \"\"\"Execute a pyautogui Python command locally via subprocess.\"\"\"\n try:\n fixed_command = self._fix_pyautogui_less_than_bug(command)\n full_command = self.pkgs_prefix.format(command=fixed_command)\n\n async def _execute():\n python_cmd = \"python\" if platform_name == \"Windows\" else \"python3\"\n proc = await asyncio.create_subprocess_exec(\n python_cmd, \"-c\", full_command,\n stdout=asyncio.subprocess.PIPE,\n stderr=asyncio.subprocess.PIPE,\n )\n stdout_b, stderr_b = await asyncio.wait_for(\n proc.communicate(), timeout=self.timeout\n )\n stdout = stdout_b.decode(\"utf-8\", errors=\"replace\") if stdout_b else \"\"\n stderr = stderr_b.decode(\"utf-8\", errors=\"replace\") if stderr_b else \"\"\n returncode = proc.returncode or 0\n return {\n \"status\": \"success\" if returncode == 0 else \"error\",\n \"output\": stdout + stderr,\n \"error\": stderr if returncode != 0 else \"\",\n \"returncode\": returncode,\n }\n\n return await self._retry_invoke(\"execute_python_command\", _execute)\n except Exception as e:\n logger.error(\"Failed to execute command: %s\", e)\n return None\n\n async def execute_action(\n self, action_type: str, parameters: Dict[str, Any] | None = None\n ) -> Dict[str, Any]:\n \"\"\"Execute a desktop action (same logic as GUIConnector).\"\"\"\n parameters = parameters or {}\n\n if action_type in [\"WAIT\", \"FAIL\", \"DONE\"]:\n return {\n \"status\": \"success\",\n \"action_type\": action_type,\n \"message\": f\"Control action {action_type} acknowledged\",\n }\n\n # Import action builder (same module used by GUIConnector)\n from anytool.grounding.backends.gui.transport.actions import (\n build_pyautogui_command,\n KEYBOARD_KEYS,\n )\n\n if action_type in [\"PRESS\", \"KEY_DOWN\", \"KEY_UP\"]:\n key = parameters.get(\"key\")\n if key and key not in KEYBOARD_KEYS:\n return {\n \"status\": \"error\",\n \"action_type\": action_type,\n \"error\": f\"Invalid key: {key}. Must be in supported keyboard keys.\",\n }\n if action_type == \"HOTKEY\":\n keys = parameters.get(\"keys\", [])\n invalid_keys = [k for k in keys if k not in KEYBOARD_KEYS]\n if invalid_keys:\n return {\n \"status\": \"error\",\n \"action_type\": action_type,\n \"error\": f\"Invalid keys: {invalid_keys}\",\n }\n\n command = build_pyautogui_command(action_type, parameters)\n if command is None:\n return {\n \"status\": \"error\",\n \"action_type\": action_type,\n \"error\": f\"Unsupported action type: {action_type}\",\n }\n\n result = await self.execute_python_command(command)\n if result:\n return {\n \"status\": \"success\",\n \"action_type\": action_type,\n \"parameters\": parameters,\n \"result\": result,\n }\n else:\n return {\n \"status\": \"error\",\n \"action_type\": action_type,\n \"parameters\": parameters,\n \"error\": \"Command execution failed\",\n }\n\n async def get_accessibility_tree(\n self, max_depth: int = 5\n ) -> Optional[Dict[str, Any]]:\n \"\"\"Get accessibility tree locally.\"\"\"\n try:\n async def _get():\n helper = self._get_accessibility_helper()\n return helper.get_tree(max_depth=max_depth)\n\n return await self._retry_invoke(\"get_accessibility_tree\", _get)\n except Exception as e:\n logger.error(\"Failed to get accessibility tree: %s\", e)\n return None\n\n async def get_cursor_position(self) -> Optional[tuple[int, int]]:\n \"\"\"Get cursor position locally.\"\"\"\n try:\n async def _get():\n helper = self._get_screenshot_helper()\n return helper.get_cursor_position()\n\n return await self._retry_invoke(\"get_cursor_position\", _get)\n except Exception as e:\n logger.error(\"Failed to get cursor position: %s\", e)\n return None\n\n # ------------------------------------------------------------------\n # BaseConnector abstract methods\n # ------------------------------------------------------------------\n\n async def invoke(self, name: str, params: dict[str, Any]) -> Any:\n if name == \"screenshot\":\n return await self.get_screenshot()\n elif name == \"accessibility_tree\":\n max_depth = params.get(\"max_depth\", 5) if params else 5\n return await self.get_accessibility_tree(max_depth)\n elif name == \"cursor_position\":\n return await self.get_cursor_position()\n else:\n return await self.execute_action(name.upper(), params or {})\n\n async def request(self, *args: Any, **kwargs: Any) -> Any:\n raise NotImplementedError(\n \"LocalGUIConnector does not support raw HTTP requests\"\n )\n\n"
},
{
"path": "anytool/grounding/backends/mcp/__init__.py",
"content": "\"\"\"\nMCP Backend for AnyTool Grounding.\n\nThis module provides the MCP (Model Context Protocol) backend implementation\nfor the grounding framework. It includes:\n\n- MCPProvider: Manages multiple MCP server sessions\n- MCPSession: Handles individual MCP server connections\n- MCPClient: High-level client for MCP server configuration\n- MCPInstallerManager: Manages automatic installation of MCP dependencies\n- MCPToolCache: Caches tool metadata to avoid starting servers on list_tools\n\"\"\"\n\nfrom .provider import MCPProvider\nfrom .session import MCPSession\nfrom .client import MCPClient\nfrom .installer import (\n MCPInstallerManager,\n get_global_installer,\n set_global_installer,\n MCPDependencyError,\n MCPCommandNotFoundError,\n MCPInstallationCancelledError,\n MCPInstallationFailedError,\n)\nfrom .tool_cache import MCPToolCache, get_tool_cache\n\n__all__ = [\n \"MCPProvider\",\n \"MCPSession\",\n \"MCPClient\",\n \"MCPInstallerManager\",\n \"get_global_installer\",\n \"set_global_installer\",\n \"MCPDependencyError\",\n \"MCPCommandNotFoundError\",\n \"MCPInstallationCancelledError\",\n \"MCPInstallationFailedError\",\n \"MCPToolCache\",\n \"get_tool_cache\",\n]"
},
{
"path": "anytool/grounding/backends/mcp/client.py",
"content": "\"\"\"\nClient for managing MCP servers and sessions.\n\nThis module provides a high-level client that manages MCP servers, connectors,\nand sessions from configuration.\n\"\"\"\nimport asyncio\nimport warnings\nfrom typing import Any, Optional\n\nfrom anytool.grounding.core.types import SandboxOptions\nfrom anytool.config.utils import get_config_value, save_json_file, load_json_file\nfrom .config import create_connector_from_config\nfrom .session import MCPSession\nfrom .installer import MCPInstallerManager, MCPDependencyError\n\nfrom anytool.utils.logging import Logger\n\nlogger = Logger.get_logger(__name__)\n\n\nclass MCPClient:\n \"\"\"Client for managing MCP servers and sessions.\n\n This class provides a unified interface for working with MCP servers,\n handling configuration, connector creation, and session management.\n \"\"\"\n\n def __init__(\n self,\n config: str | dict[str, Any] | None = None,\n sandbox: bool = False,\n sandbox_options: SandboxOptions | None = None,\n timeout: float = 30.0,\n sse_read_timeout: float = 300.0,\n max_retries: int = 3,\n retry_interval: float = 2.0,\n installer: Optional[MCPInstallerManager] = None,\n check_dependencies: bool = True,\n tool_call_max_retries: int = 3,\n tool_call_retry_delay: float = 1.0,\n ) -> None:\n \"\"\"Initialize a new MCP client.\n\n Args:\n config: Either a dict containing configuration or a path to a JSON config file.\n If None, an empty configuration is used.\n sandbox: Whether to use sandboxed execution mode for running MCP servers.\n sandbox_options: Optional sandbox configuration options.\n timeout: Timeout for operations in seconds (default: 30.0)\n sse_read_timeout: SSE read timeout in seconds (default: 300.0)\n max_retries: Maximum number of retry attempts for failed operations (default: 3)\n retry_interval: Wait time between retries in seconds (default: 2.0)\n installer: Optional installer manager for dependency installation\n check_dependencies: Whether to check and install dependencies (default: True)\n tool_call_max_retries: Maximum number of retries for tool calls (default: 3)\n tool_call_retry_delay: Initial delay between tool call retries in seconds (default: 1.0)\n \"\"\"\n self.config: dict[str, Any] = {}\n self.sandbox = sandbox\n self.sandbox_options = sandbox_options\n self.timeout = timeout\n self.sse_read_timeout = sse_read_timeout\n self.max_retries = max_retries\n self.retry_interval = retry_interval\n self.installer = installer\n self.check_dependencies = check_dependencies\n self.tool_call_max_retries = tool_call_max_retries\n self.tool_call_retry_delay = tool_call_retry_delay\n self.sessions: dict[str, MCPSession] = {}\n self.active_sessions: list[str] = []\n\n # Load configuration if provided\n if config is not None:\n if isinstance(config, str):\n self.config = load_json_file(config)\n else:\n self.config = config\n \n def _get_mcp_servers(self) -> dict[str, Any]:\n \"\"\"Internal helper to get mcpServers configuration.\n \n Tries both 'mcpServers' and 'servers' keys for compatibility.\n \n Returns:\n Dictionary of MCP server configurations, empty dict if none found.\n \"\"\"\n servers = get_config_value(self.config, \"mcpServers\", None)\n if servers is None:\n servers = get_config_value(self.config, \"servers\", {})\n return servers or {}\n\n @classmethod\n def from_dict(\n cls,\n config: dict[str, Any],\n sandbox: bool = False,\n sandbox_options: SandboxOptions | None = None,\n timeout: float = 30.0,\n sse_read_timeout: float = 300.0,\n max_retries: int = 3,\n retry_interval: float = 2.0,\n ) -> \"MCPClient\":\n \"\"\"Create a MCPClient from a dictionary.\n\n Args:\n config: The configuration dictionary.\n sandbox: Whether to use sandboxed execution mode for running MCP servers.\n sandbox_options: Optional sandbox configuration options.\n timeout: Timeout for operations in seconds (default: 30.0)\n sse_read_timeout: SSE read timeout in seconds (default: 300.0)\n max_retries: Maximum number of retry attempts (default: 3)\n retry_interval: Wait time between retries in seconds (default: 2.0)\n \"\"\"\n return cls(config=config, sandbox=sandbox, sandbox_options=sandbox_options, \n timeout=timeout, sse_read_timeout=sse_read_timeout,\n max_retries=max_retries, retry_interval=retry_interval)\n\n @classmethod\n def from_config_file(\n cls, filepath: str, sandbox: bool = False, sandbox_options: SandboxOptions | None = None,\n timeout: float = 30.0, sse_read_timeout: float = 300.0,\n max_retries: int = 3, retry_interval: float = 2.0,\n ) -> \"MCPClient\":\n \"\"\"Create a MCPClient from a configuration file.\n\n Args:\n filepath: The path to the configuration file.\n sandbox: Whether to use sandboxed execution mode for running MCP servers.\n sandbox_options: Optional sandbox configuration options.\n timeout: Timeout for operations in seconds (default: 30.0)\n sse_read_timeout: SSE read timeout in seconds (default: 300.0)\n max_retries: Maximum number of retry attempts (default: 3)\n retry_interval: Wait time between retries in seconds (default: 2.0)\n \"\"\"\n return cls(config=load_json_file(filepath), sandbox=sandbox, sandbox_options=sandbox_options,\n timeout=timeout, sse_read_timeout=sse_read_timeout,\n max_retries=max_retries, retry_interval=retry_interval)\n\n def add_server(\n self,\n name: str,\n server_config: dict[str, Any],\n ) -> None:\n \"\"\"Add a server configuration.\n\n Args:\n name: The name to identify this server.\n server_config: The server configuration.\n \"\"\"\n mcp_servers = self._get_mcp_servers()\n if \"mcpServers\" not in self.config:\n self.config[\"mcpServers\"] = {}\n \n self.config[\"mcpServers\"][name] = server_config\n logger.debug(f\"Added MCP server configuration: {name}\")\n\n def remove_server(self, name: str) -> None:\n \"\"\"Remove a server configuration.\n\n Args:\n name: The name of the server to remove.\n \"\"\"\n mcp_servers = self._get_mcp_servers()\n if name in mcp_servers:\n # Remove from config\n if \"mcpServers\" in self.config:\n self.config[\"mcpServers\"].pop(name, None)\n elif \"servers\" in self.config:\n self.config[\"servers\"].pop(name, None)\n\n # If we removed an active session, remove it from active_sessions\n if name in self.active_sessions:\n self.active_sessions.remove(name)\n \n logger.debug(f\"Removed MCP server configuration: {name}\")\n else:\n logger.warning(f\"Server '{name}' not found in configuration\")\n\n def get_server_names(self) -> list[str]:\n \"\"\"Get the list of configured server names.\n\n Returns:\n List of server names.\n \"\"\"\n return list(self._get_mcp_servers().keys())\n\n def save_config(self, filepath: str) -> None:\n \"\"\"Save the current configuration to a file.\n\n Args:\n filepath: The path to save the configuration to.\n \"\"\"\n save_json_file(self.config, filepath)\n\n async def create_session(self, server_name: str, auto_initialize: bool = True) -> MCPSession:\n \"\"\"Create a session for the specified server with retry logic.\n\n Args:\n server_name: The name of the server to create a session for.\n auto_initialize: Whether to automatically initialize the session.\n\n Returns:\n The created MCPSession.\n\n Raises:\n ValueError: If the specified server doesn't exist.\n Exception: If session creation fails after all retries.\n \"\"\"\n # Check if session already exists\n if server_name in self.sessions:\n logger.debug(f\"Session for server '{server_name}' already exists, returning existing session\")\n return self.sessions[server_name]\n \n # Get server config\n servers = self._get_mcp_servers()\n \n if not servers:\n warnings.warn(\"No MCP servers defined in config\", UserWarning, stacklevel=2)\n return None\n\n if server_name not in servers:\n raise ValueError(f\"Server '{server_name}' not found in config. Available: {list(servers.keys())}\")\n\n server_config = servers[server_name]\n\n # Retry logic for session creation\n last_exc: Exception | None = None\n \n for attempt in range(1, self.max_retries + 1):\n try:\n # Create connector with options (now async)\n connector = await create_connector_from_config(\n server_config,\n server_name=server_name,\n sandbox=self.sandbox, \n sandbox_options=self.sandbox_options,\n timeout=self.timeout,\n sse_read_timeout=self.sse_read_timeout,\n installer=self.installer,\n check_dependencies=self.check_dependencies,\n tool_call_max_retries=self.tool_call_max_retries,\n tool_call_retry_delay=self.tool_call_retry_delay,\n )\n\n # Create the session with proper initialization parameters\n session = MCPSession(\n connector=connector,\n session_id=f\"mcp-{server_name}\",\n auto_connect=True,\n auto_initialize=False, # We'll handle initialization explicitly below\n )\n \n # Initialize if requested\n if auto_initialize:\n await session.initialize()\n logger.debug(f\"Initialized session for server '{server_name}'\")\n \n # Store session\n self.sessions[server_name] = session\n\n # Add to active sessions\n if server_name not in self.active_sessions:\n self.active_sessions.append(server_name)\n \n logger.info(f\"Created session for MCP server '{server_name}' (attempt {attempt}/{self.max_retries})\")\n return session\n \n except MCPDependencyError as e:\n # Don't retry dependency errors - they won't succeed on retry\n # Error already shown to user by installer, just re-raise\n logger.debug(f\"Dependency error for server '{server_name}': {type(e).__name__}\")\n raise\n except Exception as e:\n last_exc = e\n if attempt == self.max_retries:\n break\n \n # Use info level for first attempt (common after fresh install), warning for subsequent\n log_level = logger.info if attempt == 1 else logger.warning\n log_level(\n f\"Failed to create session for server '{server_name}' (attempt {attempt}/{self.max_retries}): {e}, \"\n f\"retrying in {self.retry_interval} seconds...\"\n )\n await asyncio.sleep(self.retry_interval)\n \n # All retries failed\n error_msg = f\"Failed to create session for server '{server_name}' after {self.max_retries} retries\"\n logger.error(error_msg)\n raise last_exc or RuntimeError(error_msg)\n\n async def create_all_sessions(\n self,\n auto_initialize: bool = True,\n ) -> dict[str, MCPSession]:\n \"\"\"Create sessions for all configured servers.\n\n Args:\n auto_initialize: Whether to automatically initialize the sessions.\n\n Returns:\n Dictionary mapping server names to their MCPSession instances.\n\n Warns:\n UserWarning: If no servers are configured.\n \"\"\"\n servers = self._get_mcp_servers()\n \n if not servers:\n warnings.warn(\"No MCP servers defined in config\", UserWarning, stacklevel=2)\n return {}\n\n # Create sessions for all servers (create_session already handles initialization)\n logger.debug(f\"Creating sessions for {len(servers)} servers\")\n for name in servers:\n try:\n await self.create_session(name, auto_initialize)\n except Exception as e:\n logger.error(f\"Failed to create session for server '{name}': {e}\")\n\n logger.info(f\"Created {len(self.sessions)} MCP sessions\")\n return self.sessions\n\n def get_session(self, server_name: str) -> MCPSession:\n \"\"\"Get an existing session.\n\n Args:\n server_name: The name of the server to get the session for.\n If None, uses the first active session.\n\n Returns:\n The MCPSession for the specified server.\n\n Raises:\n ValueError: If no active sessions exist or the specified session doesn't exist.\n \"\"\"\n if server_name not in self.sessions:\n raise ValueError(f\"No session exists for server '{server_name}'\")\n\n return self.sessions[server_name]\n\n def get_all_active_sessions(self) -> dict[str, MCPSession]:\n \"\"\"Get all active sessions.\n\n Returns:\n Dictionary mapping server names to their MCPSession instances.\n \"\"\"\n return {name: self.sessions[name] for name in self.active_sessions if name in self.sessions}\n\n async def close_session(self, server_name: str) -> None:\n \"\"\"Close a session.\n\n Args:\n server_name: The name of the server to close the session for.\n\n Raises:\n ValueError: If no active sessions exist or the specified session doesn't exist.\n \"\"\"\n # Check if the session exists\n if server_name not in self.sessions:\n logger.warning(f\"No session exists for server '{server_name}', nothing to close\")\n return\n\n # Get the session\n session = self.sessions[server_name]\n error_occurred = False\n\n try:\n # Disconnect from the session\n logger.debug(f\"Closing session for server '{server_name}'\")\n await session.disconnect()\n logger.info(f\"Successfully closed session for server '{server_name}'\")\n except Exception as e:\n error_occurred = True\n logger.error(f\"Error closing session for server '{server_name}': {e}\")\n finally:\n # Remove the session regardless of whether disconnect succeeded\n self.sessions.pop(server_name, None)\n\n # Remove from active_sessions\n if server_name in self.active_sessions:\n self.active_sessions.remove(server_name)\n \n if error_occurred:\n logger.warning(f\"Session for '{server_name}' removed from tracking despite disconnect error\")\n\n async def close_all_sessions(self) -> None:\n \"\"\"Close all active sessions.\n\n This method ensures all sessions are closed even if some fail.\n \"\"\"\n # Get a list of all session names first to avoid modification during iteration\n server_names = list(self.sessions.keys())\n errors = []\n\n for server_name in server_names:\n try:\n logger.debug(f\"Closing session for server '{server_name}'\")\n await self.close_session(server_name)\n except Exception as e:\n error_msg = f\"Failed to close session for server '{server_name}': {e}\"\n logger.error(error_msg)\n errors.append(error_msg)\n\n # Log summary if there were errors\n if errors:\n logger.error(f\"Encountered {len(errors)} errors while closing sessions\")\n else:\n logger.debug(\"All sessions closed successfully\")\n"
},
{
"path": "anytool/grounding/backends/mcp/config.py",
"content": "\"\"\"\nConfiguration loader for MCP session.\n\nThis module provides functionality to load MCP configuration from JSON files.\n\"\"\"\n\nfrom typing import Any, Optional\n\nfrom anytool.grounding.core.types import SandboxOptions\nfrom anytool.config.utils import get_config_value\nfrom .transport.connectors import (\n MCPBaseConnector,\n HttpConnector,\n SandboxConnector,\n StdioConnector,\n WebSocketConnector,\n)\nfrom .transport.connectors.utils import is_stdio_server\nfrom .installer import MCPInstallerManager\n\n# Import E2BSandbox\ntry:\n from anytool.grounding.core.security import E2BSandbox\n E2B_AVAILABLE = True\nexcept ImportError:\n E2BSandbox = None\n E2B_AVAILABLE = False\n\nasync def create_connector_from_config(\n server_config: dict[str, Any],\n server_name: str = \"unknown\",\n sandbox: bool = False,\n sandbox_options: SandboxOptions | None = None,\n timeout: float = 30.0,\n sse_read_timeout: float = 300.0,\n installer: Optional[MCPInstallerManager] = None,\n check_dependencies: bool = True,\n tool_call_max_retries: int = 3,\n tool_call_retry_delay: float = 1.0,\n) -> MCPBaseConnector:\n \"\"\"Create a connector based on server configuration.\n \n Args:\n server_config: The server configuration section\n server_name: Name of the MCP server (for display purposes)\n sandbox: Whether to use sandboxed execution mode for running MCP servers.\n sandbox_options: Optional sandbox configuration options.\n timeout: Timeout for operations in seconds (default: 30.0)\n sse_read_timeout: SSE read timeout in seconds (default: 300.0)\n installer: Optional installer manager for dependency installation\n check_dependencies: Whether to check and install dependencies (default: True)\n tool_call_max_retries: Maximum number of retries for tool calls (default: 3)\n tool_call_retry_delay: Initial delay between retries in seconds (default: 1.0)\n\n Returns:\n A configured connector instance\n \n Raises:\n RuntimeError: If dependencies are not installed and user declines installation\n \"\"\"\n \n # Get original command and args from config\n original_command = get_config_value(server_config, \"command\")\n original_args = get_config_value(server_config, \"args\", [])\n\n # Check and install dependencies if needed (only for stdio servers)\n if is_stdio_server(server_config) and check_dependencies:\n # Use provided installer or get global instance\n if installer is None:\n from .installer import get_global_installer\n installer = get_global_installer()\n\n # Ensure dependencies are installed (using original command/args)\n await installer.ensure_dependencies(server_name, original_command, original_args)\n\n # Stdio connector (command-based)\n if is_stdio_server(server_config) and not sandbox:\n return StdioConnector(\n command=get_config_value(server_config, \"command\"),\n args=get_config_value(server_config, \"args\"),\n env=get_config_value(server_config, \"env\", None),\n )\n\n # Sandboxed connector\n elif is_stdio_server(server_config) and sandbox:\n if not E2B_AVAILABLE:\n raise ImportError(\n \"E2B sandbox support not available. Please install e2b-code-interpreter: \"\n \"'pip install e2b-code-interpreter'\"\n )\n \n # Create E2B sandbox instance\n _sandbox_options = sandbox_options or {}\n e2b_sandbox = E2BSandbox(_sandbox_options)\n \n # Extract timeout values from sandbox_options or use defaults\n connector_timeout = _sandbox_options.get(\"timeout\", timeout)\n connector_sse_timeout = _sandbox_options.get(\"sse_read_timeout\", sse_read_timeout)\n \n # Create and return sandbox connector\n return SandboxConnector(\n sandbox=e2b_sandbox,\n command=get_config_value(server_config, \"command\"),\n args=get_config_value(server_config, \"args\"),\n env=get_config_value(server_config, \"env\", None),\n supergateway_command=_sandbox_options.get(\"supergateway_command\", \"npx -y supergateway\"),\n port=_sandbox_options.get(\"port\", 3000),\n timeout=connector_timeout,\n sse_read_timeout=connector_sse_timeout,\n )\n\n # HTTP connector\n elif \"url\" in server_config:\n return HttpConnector(\n base_url=get_config_value(server_config, \"url\"),\n headers=get_config_value(server_config, \"headers\", None),\n auth_token=get_config_value(server_config, \"auth_token\", None),\n timeout=timeout,\n sse_read_timeout=sse_read_timeout,\n tool_call_max_retries=tool_call_max_retries,\n tool_call_retry_delay=tool_call_retry_delay,\n )\n\n # WebSocket connector\n elif \"ws_url\" in server_config:\n return WebSocketConnector(\n url=get_config_value(server_config, \"ws_url\"),\n headers=get_config_value(server_config, \"headers\", None),\n auth_token=get_config_value(server_config, \"auth_token\", None),\n )\n\n raise ValueError(\"Cannot determine connector type from config\")"
},
{
"path": "anytool/grounding/backends/mcp/installer.py",
"content": "import asyncio\nimport sys\nimport shutil\nfrom typing import Callable, Awaitable, Optional, Dict, List\nfrom anytool.utils.logging import Logger\n\nlogger = Logger.get_logger(__name__)\n\nPromptFunc = Callable[[str], Awaitable[bool]]\n\n# Global lock to prevent concurrent user prompts\n_prompt_lock = asyncio.Lock()\n\n\nclass MCPDependencyError(RuntimeError):\n \"\"\"Base exception for MCP dependency errors.\"\"\"\n pass\n\n\nclass MCPCommandNotFoundError(MCPDependencyError):\n \"\"\"Raised when a required command is not available.\"\"\"\n pass\n\n\nclass MCPInstallationCancelledError(MCPDependencyError):\n \"\"\"Raised when user cancels installation.\"\"\"\n pass\n\n\nclass MCPInstallationFailedError(MCPDependencyError):\n \"\"\"Raised when installation fails.\"\"\"\n pass\n\n\nclass Colors:\n RESET = \"\\033[0m\"\n BOLD = \"\\033[1m\"\n RED = \"\\033[91m\"\n YELLOW = \"\\033[93m\"\n GREEN = \"\\033[92m\"\n CYAN = \"\\033[96m\"\n GRAY = \"\\033[90m\"\n WHITE = \"\\033[97m\"\n BLUE = \"\\033[94m\"\n\n\nclass MCPInstallerManager:\n \"\"\"\n MCP dependencies package installer manager.\n \n Responsible for detecting if the MCP server dependencies are installed, and if not, asking the user whether to install them.\n \"\"\"\n \n def __init__(self, prompt: PromptFunc | None = None, auto_install: bool = False, verbose: bool = False):\n \"\"\"Initialize the installer manager.\n \n Args:\n prompt: Custom user prompt function, if None, the default CLI prompt is used\n auto_install: If True, automatically install dependencies without asking the user\n verbose: If True, show detailed installation logs; if False, only show progress indicator\n \"\"\"\n self._prompt: PromptFunc | None = prompt or self._default_cli_prompt\n self._auto_install = auto_install\n self._verbose = verbose\n self._installed_cache: Dict[str, bool] = {} # Cache for checked packages\n self._failed_installations: Dict[str, str] = {} # Track failed installations to avoid retry\n \n async def _default_cli_prompt(self, message: str) -> bool:\n \"\"\"Default CLI prompt function (called within lock by ensure_dependencies).\"\"\"\n from anytool.utils.display import print_separator, colorize\n \n print()\n print_separator(70, 'c', 2)\n print(f\" {colorize('MCP dependencies installation prompt', color=Colors.BLUE, bold=True)}\")\n print_separator(70, 'c', 2)\n print(f\" {message}\")\n print_separator(70, 'gr', 2)\n print(f\" {colorize('[y/yes]', color=Colors.GREEN)} Install | {colorize('[n/no]', color=Colors.RED)} Cancel\")\n print_separator(70, 'gr', 2)\n print(f\" {colorize('Your choice:', bold=True)} \", end=\"\", flush=True)\n \n answer = await asyncio.get_running_loop().run_in_executor(None, sys.stdin.readline)\n response = answer.strip().lower() in {\"y\", \"yes\"}\n \n if response:\n print(f\"{Colors.GREEN}โ Installation confirmed{Colors.RESET}\\n\")\n else:\n print(f\"{Colors.RED}โ Installation cancelled{Colors.RESET}\\n\")\n \n return response\n \n async def _ask_user(self, message: str) -> bool:\n \"\"\"Ask the user whether to install.\"\"\"\n if self._auto_install:\n logger.info(\"Automatic installation mode enabled, will automatically install dependencies\")\n return True\n \n if self._prompt:\n try:\n return await self._prompt(message)\n except Exception as e:\n logger.error(f\"Error asking user: {e}\")\n return False\n return False\n \n def _check_command_available(self, command: str) -> bool:\n \"\"\"Check if the command is available.\n \n Args:\n command: The command to check (e.g. \"npx\", \"uvx\")\n \n Returns:\n bool: Whether the command is available\n \"\"\"\n return shutil.which(command) is not None\n \n async def _check_package_installed(self, command: str, args: List[str]) -> bool:\n \"\"\"Check if the package is installed.\n \n Args:\n command: The command to check (e.g. \"npx\", \"uvx\")\n args: The arguments list\n \n Returns:\n bool: Whether the package is installed\n \"\"\"\n # Build cache key\n cache_key = f\"{command}:{':'.join(args)}\"\n \n # Check cache\n if cache_key in self._installed_cache:\n return self._installed_cache[cache_key]\n \n # For different types of commands, use different check methods\n try:\n if command == \"npx\":\n # For npx, check if the npm package exists\n package_name = self._extract_npm_package(args)\n if package_name:\n result = await self._check_npm_package(package_name)\n self._installed_cache[cache_key] = result\n return result\n elif command == \"uvx\":\n # For uvx, check if the Python package exists\n package_name = self._extract_python_package(args)\n if package_name:\n result = await self._check_python_package(package_name)\n self._installed_cache[cache_key] = result\n return result\n elif command == \"uv\":\n # For \"uv run --with package ...\", check if the Python package exists\n package_name = self._extract_uv_package(args)\n if package_name:\n result = await self._check_uv_pip_package(package_name)\n self._installed_cache[cache_key] = result\n return result\n except Exception as e:\n logger.debug(f\"Error checking package installation status: {e}\")\n \n # Default to assuming not installed\n return False\n \n def _extract_npm_package(self, args: List[str]) -> Optional[str]:\n \"\"\"Extract package name from npx arguments.\n \n Args:\n args: npx arguments list, e.g. [\"-y\", \"mcp-excalidraw-server\"] or [\"bazi-mcp\"]\n \n Returns:\n Package name (without version tag) or None\n \"\"\"\n for i, arg in enumerate(args):\n # Skip option parameters\n if arg.startswith(\"-\"):\n continue\n \n # Found package name, now strip version tag\n package_name = arg\n \n # Handle scoped packages: @scope/package@version -> @scope/package\n if package_name.startswith(\"@\"):\n # Scoped package like @rtuin/mcp-mermaid-validator@latest\n parts = package_name.split(\"/\", 1)\n if len(parts) == 2:\n scope = parts[0]\n name_with_version = parts[1]\n # Remove version tag from name part (e.g., \"pkg@latest\" -> \"pkg\")\n name = name_with_version.split(\"@\")[0] if \"@\" in name_with_version else name_with_version\n return f\"{scope}/{name}\"\n return package_name\n else:\n # Regular package like mcp-deepwiki@latest -> mcp-deepwiki\n return package_name.split(\"@\")[0] if \"@\" in package_name else package_name\n \n return None\n \n def _extract_python_package(self, args: List[str]) -> Optional[str]:\n \"\"\"Extract package name from uvx arguments.\n \n Args:\n args: uvx arguments list, e.g. [\"--from\", \"office-powerpoint-mcp-server\", \"ppt_mcp_server\"]\n or [\"--with\", \"mcp==1.9.0\", \"sitemap-mcp-server\"]\n or [\"arxiv-mcp-server\", \"--storage-path\", \"./path\"]\n \n Returns:\n Package name or None\n \"\"\"\n # Find --from parameter (this is the package to install)\n for i, arg in enumerate(args):\n if arg == \"--from\" and i + 1 < len(args):\n return args[i + 1]\n \n # Skip option flags and their values, find the main package (FIRST positional arg)\n # Options that take a value: --with, --python, --from, --storage-path, etc.\n options_with_value = {\"--with\", \"--from\", \"--python\", \"-p\", \"--storage-path\"}\n skip_next = False\n \n for arg in args:\n if skip_next:\n skip_next = False\n continue\n if arg in options_with_value:\n skip_next = True\n continue\n if arg.startswith(\"-\"):\n # Other flags without values (or unknown options with values)\n # Also skip the next arg if it looks like an option value (doesn't start with -)\n continue\n # First non-option argument is the package name\n return arg\n \n return None\n \n def _extract_uv_package(self, args: List[str]) -> Optional[str]:\n \"\"\"Extract package name from uv run arguments.\n \n Args:\n args: uv arguments list, e.g. [\"run\", \"--with\", \"biomcp-python\", \"biomcp\", \"run\"]\n \n Returns:\n Package name or None\n \"\"\"\n # Find --with parameter (this specifies the package to install)\n for i, arg in enumerate(args):\n if arg == \"--with\" and i + 1 < len(args):\n package_name = args[i + 1]\n # Remove version specifier if present (e.g., \"mcp==1.9.0\" -> \"mcp\")\n if \"==\" in package_name:\n return package_name.split(\"==\")[0]\n if \">=\" in package_name:\n return package_name.split(\">=\")[0]\n return package_name\n \n return None\n \n async def _check_npm_package(self, package_name: str) -> bool:\n \"\"\"Check if the npm package is globally installed.\n \n Args:\n package_name: npm package name\n \n Returns:\n bool: Whether the npm package is installed\n \"\"\"\n try:\n process = await asyncio.create_subprocess_exec(\n \"npm\", \"list\", \"-g\", package_name,\n stdout=asyncio.subprocess.PIPE,\n stderr=asyncio.subprocess.PIPE\n )\n stdout, stderr = await process.communicate()\n \n # npm list returns 0 if the package is installed\n return process.returncode == 0\n except Exception as e:\n logger.debug(f\"Error checking npm package {package_name}: {e}\")\n return False\n \n async def _check_python_package(self, package_name: str) -> bool:\n \"\"\"Check if the Python package is installed as a uvx tool.\n \n uvx tools are installed in ~/.local/share/uv/tools/ directory,\n not in the current pip environment.\n \n Args:\n package_name: Python package/tool name\n \n Returns:\n bool: Whether the uvx tool is installed\n \"\"\"\n import os\n from pathlib import Path\n \n # Strip version specifier if present (e.g., \"mcp==1.9.0\" -> \"mcp\")\n clean_name = package_name.split(\"==\")[0].split(\">=\")[0].split(\"<=\")[0].split(\">\")[0].split(\"<\")[0]\n \n # Check if uvx tool exists in the standard uv tools directory\n uv_tools_dir = Path.home() / \".local\" / \"share\" / \"uv\" / \"tools\"\n tool_dir = uv_tools_dir / clean_name\n \n if tool_dir.exists():\n logger.debug(f\"uvx tool '{clean_name}' found at {tool_dir}\")\n return True\n \n # Fallback: try running uvx with --help to check if it's available\n try:\n process = await asyncio.create_subprocess_exec(\n \"uvx\", clean_name, \"--help\",\n stdout=asyncio.subprocess.PIPE,\n stderr=asyncio.subprocess.PIPE\n )\n # Just wait briefly, don't need the full output\n try:\n await asyncio.wait_for(process.communicate(), timeout=5.0)\n except asyncio.TimeoutError:\n process.kill()\n await process.wait()\n \n # If it didn't error immediately, the tool likely exists\n return process.returncode == 0\n except Exception as e:\n logger.debug(f\"Error checking uvx tool {clean_name}: {e}\")\n \n return False\n \n async def _check_uv_pip_package(self, package_name: str) -> bool:\n \"\"\"Check if a Python package is installed via uv pip.\n \n Args:\n package_name: Python package name\n \n Returns:\n bool: Whether the package is installed\n \"\"\"\n # Strip version specifier if present\n clean_name = package_name.split(\"==\")[0].split(\">=\")[0].split(\"<=\")[0].split(\">\")[0].split(\"<\")[0]\n \n try:\n # Try using uv pip show to check if package is installed\n process = await asyncio.create_subprocess_exec(\n \"uv\", \"pip\", \"show\", clean_name,\n stdout=asyncio.subprocess.PIPE,\n stderr=asyncio.subprocess.PIPE\n )\n stdout, stderr = await process.communicate()\n \n if process.returncode == 0:\n logger.debug(f\"uv pip package '{clean_name}' found\")\n return True\n except Exception as e:\n logger.debug(f\"Error checking uv pip package {clean_name}: {e}\")\n \n # Fallback: check with regular pip\n try:\n process = await asyncio.create_subprocess_exec(\n \"pip\", \"show\", clean_name,\n stdout=asyncio.subprocess.PIPE,\n stderr=asyncio.subprocess.PIPE\n )\n stdout, stderr = await process.communicate()\n \n return process.returncode == 0\n except Exception as e:\n logger.debug(f\"Error checking pip package {clean_name}: {e}\")\n \n return False\n \n async def _install_package(self, command: str, args: List[str], use_sudo: bool = False) -> bool:\n \"\"\"Execute the install command.\n \n Args:\n command: The command to execute (e.g. \"npx\", \"uvx\")\n args: The arguments list\n use_sudo: Whether to use sudo for installation\n \n Returns:\n bool: Whether the installation is successful\n \"\"\"\n install_command = self._get_install_command(command, args)\n \n if not install_command:\n logger.error(\"Cannot determine install command\")\n return False\n \n # Add sudo if requested\n if use_sudo:\n install_command = [\"sudo\"] + install_command\n \n logger.info(f\"Executing install command: {' '.join(install_command)}\")\n \n try:\n # For sudo commands, always show verbose output so password prompt is visible\n if self._verbose or use_sudo:\n # Verbose mode: show all installation logs\n from anytool.utils.display import print_separator, colorize\n \n print_separator(70, 'c', 2)\n if use_sudo:\n print(f\" {colorize('Installing with administrator privileges...', color=Colors.BLUE)}\")\n print(f\" {colorize('>> You will be prompted for your password below <<', color=Colors.YELLOW)}\")\n else:\n print(f\" {colorize('Installing dependencies...', color=Colors.BLUE)}\")\n print(f\" {colorize('Command: ' + ' '.join(install_command), color=Colors.GRAY)}\")\n print_separator(70, 'c', 2)\n print()\n \n # For sudo, don't redirect stdin so password prompt works\n if use_sudo:\n process = await asyncio.create_subprocess_exec(\n *install_command,\n stdout=asyncio.subprocess.PIPE,\n stderr=asyncio.subprocess.STDOUT,\n stdin=None # Let sudo use terminal for password\n )\n else:\n process = await asyncio.create_subprocess_exec(\n *install_command,\n stdout=asyncio.subprocess.PIPE,\n stderr=asyncio.subprocess.STDOUT\n )\n \n # Real-time output of installation logs\n output_lines = []\n while True:\n line = await process.stdout.readline()\n if not line:\n break\n line_str = line.decode().rstrip()\n output_lines.append(line_str)\n print(f\"{Colors.GRAY}{line_str}{Colors.RESET}\")\n \n await process.wait()\n full_output = '\\n'.join(output_lines)\n else:\n # Quiet mode: only show progress indicator\n print(f\"\\n{Colors.BLUE}Installing dependencies...{Colors.RESET} \", end=\"\", flush=True)\n \n process = await asyncio.create_subprocess_exec(\n *install_command,\n stdout=asyncio.subprocess.PIPE,\n stderr=asyncio.subprocess.PIPE\n )\n \n # Show spinner animation while installing\n spinner = ['โ ', 'โ ', 'โ น', 'โ ธ', 'โ ผ', 'โ ด', 'โ ฆ', 'โ ง', 'โ ', 'โ ']\n spinner_idx = 0\n \n while True:\n try:\n await asyncio.wait_for(process.wait(), timeout=0.1)\n break\n except asyncio.TimeoutError:\n print(f\"\\r{Colors.BLUE}Installing dependencies...{Colors.RESET} {Colors.CYAN}{spinner[spinner_idx]}{Colors.RESET}\", end=\"\", flush=True)\n spinner_idx = (spinner_idx + 1) % len(spinner)\n \n # Clear the spinner line\n print(f\"\\r{' ' * 100}\\r\", end=\"\", flush=True)\n \n # Collect output\n stdout, stderr = await process.communicate()\n full_output = (stdout or stderr).decode() if (stdout or stderr) else \"\"\n \n if process.returncode == 0:\n print(f\"{Colors.GREEN}โ Dependencies installed successfully{Colors.RESET}\")\n if not use_sudo:\n print(f\"{Colors.GRAY}(Note: First connection may take a moment to initialize){Colors.RESET}\")\n # Update cache\n cache_key = f\"{command}:{':'.join(args)}\"\n self._installed_cache[cache_key] = True\n return True\n else:\n # Check if it's a permission error\n is_permission_error = \"EACCES\" in full_output or \"permission denied\" in full_output.lower()\n \n if is_permission_error and not use_sudo:\n print(f\"\\n{Colors.YELLOW}Permission denied{Colors.RESET}\")\n print(f\"{Colors.GRAY}The installation requires administrator privileges.{Colors.RESET}\\n\")\n \n # Ask user if they want to use sudo\n message = (\n f\"\\n{Colors.WHITE}Administrator privileges required{Colors.RESET}\\n\\n\"\n f\"Command: {Colors.GRAY}{' '.join(install_command)}{Colors.RESET}\\n\\n\"\n f\"{Colors.YELLOW}Do you want to retry with sudo (requires password)?{Colors.RESET}\"\n )\n \n if await self._ask_user(message):\n # No extra print needed, the verbose mode will show clear instructions\n return await self._install_package(command, args, use_sudo=True)\n else:\n print(f\"\\n{Colors.RED}โ Installation cancelled{Colors.RESET}\")\n return False\n else:\n print(f\"{Colors.RED}โ Dependencies installation failed (return code: {process.returncode}){Colors.RESET}\")\n # Show error output if not already shown\n if not self._verbose and full_output:\n # Limit error output to last 20 lines\n error_lines = full_output.split('\\n')\n if len(error_lines) > 20:\n error_lines = ['...(truncated)...'] + error_lines[-20:]\n print(f\"{Colors.GRAY}Error output:\\n{chr(10).join(error_lines)}{Colors.RESET}\")\n \n # Add general guidance for manual installation\n print(f\"\\n{Colors.YELLOW}Tip:{Colors.RESET} {Colors.GRAY}If automatic installation fails, please refer to the\")\n print(f\"official documentation of the MCP server for manual installation instructions.{Colors.RESET}\\n\")\n \n return False\n \n except Exception as e:\n logger.error(f\"Error installing dependencies: {e}\")\n print(f\"{Colors.RED}โ Error occurred during installation: {e}{Colors.RESET}\")\n return False\n \n def _get_install_command(self, command: str, args: List[str]) -> Optional[List[str]]:\n \"\"\"Generate install command based on command type.\n \n Args:\n command: The command to execute (e.g. \"npx\", \"uvx\", \"uv\")\n args: The original arguments list\n \n Returns:\n Install command list or None\n \"\"\"\n if command == \"npx\":\n package_name = self._extract_npm_package(args)\n if package_name:\n return [\"npm\", \"install\", \"-g\", package_name]\n elif command == \"uvx\":\n package_name = self._extract_python_package(args)\n if package_name:\n return [\"pip\", \"install\", package_name]\n elif command == \"uv\":\n # Handle \"uv run --with package_name ...\" format\n package_name = self._extract_uv_package(args)\n if package_name:\n return [\"uv\", \"pip\", \"install\", package_name]\n \n return None\n \n async def ensure_dependencies(\n self, \n server_name: str,\n command: str, \n args: List[str]\n ) -> bool:\n \"\"\"Ensure the dependencies of the MCP server are installed.\n \n This method checks if the dependencies are installed, and if not, asks the user whether to install them.\n \n Args:\n server_name: MCP server name (for display purposes)\n command: The command to execute (e.g. \"npx\", \"uvx\")\n args: The arguments list\n \n Returns:\n bool: Whether the dependencies are installed (installed or successfully installed)\n \n Raises:\n RuntimeError: When the command is not available or the user refuses to install\n \"\"\"\n # Use lock to ensure entire installation process is atomic\n async with _prompt_lock:\n return await self._ensure_dependencies_impl(server_name, command, args)\n \n async def _ensure_dependencies_impl(\n self, \n server_name: str,\n command: str, \n args: List[str]\n ) -> bool:\n \"\"\"Internal implementation of ensure_dependencies (called within lock).\"\"\"\n # Skip dependency checking for direct script execution commands\n # These commands run scripts directly and don't need package installation\n SKIP_COMMANDS = {\"node\", \"python\", \"python3\", \"bash\", \"sh\", \"deno\", \"bun\"}\n \n if command.lower() in SKIP_COMMANDS:\n logger.debug(f\"Skipping dependency check for direct script execution command: {command}\")\n return True\n \n # Skip dependency checking for GitHub-based npx packages\n # These packages are handled directly by npx which downloads, builds, and runs them\n # npm install -g doesn't work properly for GitHub packages that require building\n if command == \"npx\":\n package_name = self._extract_npm_package(args)\n if package_name and package_name.startswith(\"github:\"):\n logger.debug(f\"Skipping dependency check for GitHub-based npx package: {package_name}\")\n return True\n \n # Check if this server has already failed installation\n cache_key = f\"{server_name}:{command}:{':'.join(args)}\"\n if cache_key in self._failed_installations:\n error_msg = self._failed_installations[cache_key]\n logger.debug(f\"Skipping installation for '{server_name}' - previously failed\")\n raise MCPDependencyError(error_msg)\n \n # Special handling for uvx - check if uv is installed\n if command == \"uvx\":\n if not self._check_command_available(\"uv\"):\n # Only show once to user, no verbose logging\n print(f\"\\n{Colors.RED}โ Server '{server_name}' requires 'uv' to be installed{Colors.RESET}\")\n print(f\"{Colors.YELLOW}Please install uv first:\")\n print(f\" โข macOS/Linux: curl -LsSf https://astral.sh/uv/install.sh | sh\")\n print(f\" โข Or with pip: pip install uv\")\n print(f\" โข Or with brew: brew install uv{Colors.RESET}\\n\")\n \n error_msg = f\"uvx requires 'uv' to be installed (server: {server_name})\"\n self._failed_installations[cache_key] = error_msg\n raise MCPCommandNotFoundError(error_msg)\n \n # Check if the command is available\n if not self._check_command_available(command):\n error_msg = (\n f\"Command '{command}' is not available.\\n\"\n f\"Please install the necessary tools first.\"\n )\n logger.error(error_msg)\n self._failed_installations[cache_key] = error_msg\n raise MCPCommandNotFoundError(error_msg)\n \n # Check if the package is installed\n if await self._check_package_installed(command, args):\n logger.debug(f\"The dependencies of the MCP server '{server_name}' are installed\")\n return True\n \n # Extract package name for display\n if command == \"npx\":\n package_name = self._extract_npm_package(args)\n package_type = \"npm\"\n elif command == \"uvx\":\n package_name = self._extract_python_package(args)\n package_type = \"Python\"\n elif command == \"uv\":\n package_name = self._extract_uv_package(args)\n package_type = \"Python\"\n else:\n package_name = f\"{command} {' '.join(args)}\"\n package_type = \"package\"\n \n # Build the message for displaying the install command\n install_cmd = self._get_install_command(command, args)\n \n # If we can't determine an install command, show helpful message\n if not install_cmd:\n print(f\"\\n{Colors.YELLOW}Cannot automatically install dependencies for '{server_name}'{Colors.RESET}\")\n print(f\"{Colors.GRAY}Command: {command} {' '.join(args)}{Colors.RESET}\")\n print(f\"\\n{Colors.WHITE}This MCP server may require manual installation or configuration.{Colors.RESET}\")\n print(f\"{Colors.GRAY}Please refer to the MCP server's official documentation for installation instructions.{Colors.RESET}\\n\")\n \n error_msg = f\"Manual installation required for '{server_name}' (command: {command})\"\n self._failed_installations[cache_key] = error_msg\n raise MCPDependencyError(error_msg)\n \n install_cmd_str = ' '.join(install_cmd)\n \n # Build the message\n message = (\n f\"\\n{Colors.WHITE}The MCP server needs to install dependencies{Colors.RESET}\\n\\n\"\n f\"Server name: {Colors.CYAN}{server_name}{Colors.RESET}\\n\"\n f\"Package type: {Colors.YELLOW}{package_type}{Colors.RESET}\\n\"\n f\"Package name: {Colors.YELLOW}{package_name or 'Unknown'}{Colors.RESET}\\n\"\n f\"Install command: {Colors.GRAY}{install_cmd_str}{Colors.RESET}\\n\\n\"\n f\"{Colors.YELLOW}Whether to install this dependency package?{Colors.RESET}\"\n )\n \n # Ask the user\n if not await self._ask_user(message):\n error_msg = f\"User cancelled the dependency installation for '{server_name}'\"\n logger.warning(error_msg)\n self._failed_installations[cache_key] = error_msg\n raise MCPInstallationCancelledError(error_msg)\n \n # Execute installation\n success = await self._install_package(command, args)\n \n if not success:\n error_msg = f\"Dependency installation failed for '{server_name}'\"\n logger.error(error_msg)\n self._failed_installations[cache_key] = error_msg\n raise MCPInstallationFailedError(error_msg)\n \n return True\n\n\n# Global singleton instance\n_global_installer: Optional[MCPInstallerManager] = None\n\n\ndef get_global_installer() -> MCPInstallerManager:\n \"\"\"Get the global installer manager instance.\"\"\"\n global _global_installer\n if _global_installer is None:\n _global_installer = MCPInstallerManager()\n return _global_installer\n\ndef set_global_installer(installer: MCPInstallerManager) -> None:\n \"\"\"Set the global installer manager instance.\"\"\"\n global _global_installer\n _global_installer = installer"
},
{
"path": "anytool/grounding/backends/mcp/provider.py",
"content": "\"\"\"\nMCP Provider implementation.\n\nThis module provides a provider for managing MCP server sessions.\n\"\"\"\nimport asyncio\nfrom typing import Dict, List, Optional\n\nfrom anytool.grounding.backends.mcp.session import MCPSession\nfrom anytool.grounding.core.provider import Provider\nfrom anytool.grounding.core.types import SessionConfig, BackendType, ToolSchema\nfrom anytool.grounding.backends.mcp.client import MCPClient\nfrom anytool.grounding.backends.mcp.installer import MCPInstallerManager, MCPDependencyError\nfrom anytool.grounding.backends.mcp.tool_cache import get_tool_cache\nfrom anytool.grounding.backends.mcp.tool_converter import _sanitize_mcp_schema\nfrom anytool.grounding.core.tool import BaseTool, RemoteTool\nfrom anytool.utils.logging import Logger\nfrom anytool.config.utils import get_config_value\n\nlogger = Logger.get_logger(__name__)\n\n\nclass MCPProvider(Provider[MCPSession]):\n \"\"\"\n MCP Provider manages multiple MCP server sessions.\n \n Each MCP server defined in config corresponds to one session.\n The provider handles lazy/eager session creation and tool aggregation.\n \"\"\"\n \n def __init__(self, config: Dict | None = None, installer: Optional[MCPInstallerManager] = None):\n \"\"\"Initialize MCP Provider.\n \n Args:\n config: Configuration dict with MCP server definitions.\n Example: {\"mcpServers\": {\"server1\": {...}, \"server2\": {...}}}\n installer: Optional installer manager for dependency installation\n \"\"\"\n super().__init__(BackendType.MCP, config)\n \n # Extract MCP-specific configuration\n sandbox = get_config_value(config, \"sandbox\", False)\n timeout = get_config_value(config, \"timeout\", 30)\n sse_read_timeout = get_config_value(config, \"sse_read_timeout\", 300.0)\n max_retries = get_config_value(config, \"max_retries\", 3)\n retry_interval = get_config_value(config, \"retry_interval\", 2.0)\n check_dependencies = get_config_value(config, \"check_dependencies\", True)\n auto_install = get_config_value(config, \"auto_install\", False)\n # Tool call retry settings (for transient errors like 400, 500, etc.)\n tool_call_max_retries = get_config_value(config, \"tool_call_max_retries\", 3)\n tool_call_retry_delay = get_config_value(config, \"tool_call_retry_delay\", 1.0)\n \n # Create sandbox options if sandbox is enabled\n sandbox_options = None\n if sandbox:\n sandbox_options = {\n \"timeout\": timeout,\n \"sse_read_timeout\": sse_read_timeout,\n }\n \n # Create installer with auto_install setting if not provided\n if installer is None and auto_install:\n installer = MCPInstallerManager(auto_install=True)\n \n # Initialize MCPClient with configuration\n self._client = MCPClient(\n config=config or {},\n sandbox=sandbox,\n sandbox_options=sandbox_options,\n timeout=timeout,\n sse_read_timeout=sse_read_timeout,\n max_retries=max_retries,\n retry_interval=retry_interval,\n installer=installer,\n check_dependencies=check_dependencies,\n tool_call_max_retries=tool_call_max_retries,\n tool_call_retry_delay=tool_call_retry_delay,\n )\n \n # Map server name to session for quick lookup\n self._server_sessions: Dict[str, MCPSession] = {}\n\n async def initialize(self) -> None:\n \"\"\"Initialize the MCP provider.\n \n If config[\"eager_sessions\"] is True, creates sessions for all configured servers.\n Otherwise, sessions are created lazily on first access.\n \"\"\"\n if self.is_initialized:\n return\n\n # config can be dict or Pydantic model, use utility function\n eager = get_config_value(self.config, \"eager_sessions\", False)\n if eager:\n servers = self.list_servers()\n logger.debug(f\"Eagerly initializing {len(servers)} MCP server sessions\")\n for srv in servers:\n if srv not in self._server_sessions:\n cfg = SessionConfig(\n session_name=f\"mcp-{srv}\",\n backend_type=BackendType.MCP,\n connection_params={\"server\": srv},\n )\n await self.create_session(cfg)\n\n self.is_initialized = True\n logger.info(\n f\"MCPProvider initialized with {len(self.list_servers())} servers (eager={eager})\"\n )\n\n def list_servers(self) -> List[str]:\n \"\"\"Return all configured MCP server names from MCPClient config.\n \n Returns:\n List of server names\n \"\"\"\n return self._client.get_server_names()\n\n async def create_session(self, session_config: SessionConfig) -> MCPSession:\n \"\"\"Create a new MCP session for a specific server.\n \n Args:\n session_config: Must contain 'server' in connection_params\n \n Returns:\n MCPSession instance\n \n Raises:\n ValueError: If 'server' not in connection_params\n Exception: If session creation or initialization fails\n \"\"\"\n server = get_config_value(session_config.connection_params, \"server\")\n if not server:\n raise ValueError(\"MCPProvider.create_session requires 'server' in connection_params\")\n\n # Generate session_id: mcp-\n session_id = f\"{self.backend_type.value}-{server}\"\n\n # Check if session already exists\n if server in self._server_sessions:\n logger.debug(f\"Session for server '{server}' already exists, returning existing session\")\n return self._server_sessions[server]\n\n # Create session through MCPClient\n try:\n logger.debug(f\"Creating new session for MCP server: {server}\")\n session = await self._client.create_session(server, auto_initialize=True)\n session.session_id = session_id\n\n # Store in both maps\n self._server_sessions[server] = session\n self._sessions[session_id] = session\n \n logger.info(f\"Created MCP session '{session_id}' for server '{server}'\")\n return session\n except MCPDependencyError as e:\n # Dependency errors already shown to user, just debug log\n logger.debug(f\"Dependency error for server '{server}': {type(e).__name__}\")\n raise\n except Exception as e:\n logger.error(f\"Failed to create session for server '{server}': {e}\")\n raise\n\n async def close_session(self, session_name: str) -> None:\n \"\"\"Close an MCP session by session name.\n \n Args:\n session_name: Session name in format 'mcp-'\n \"\"\"\n # Parse server name from session_name (format: mcp-)\n try:\n prefix, server_name = session_name.split(\"-\", 1)\n if prefix != self.backend_type.value:\n raise ValueError(f\"Invalid MCP session name format: {session_name}, expected 'mcp-'\")\n except ValueError as e:\n logger.warning(f\"Invalid session_name format: {session_name} - {e}\")\n return\n\n # Check if session exists\n if session_name not in self._sessions and server_name not in self._server_sessions:\n logger.warning(f\"Session '{session_name}' not found, nothing to close\")\n return\n\n error_occurred = False\n try:\n logger.debug(f\"Closing MCP session '{session_name}' (server: {server_name})\")\n await self._client.close_session(server_name)\n logger.info(f\"Successfully closed MCP session '{session_name}'\")\n except Exception as e:\n error_occurred = True\n logger.error(f\"Error closing MCP session '{session_name}': {e}\")\n finally:\n # Clean up both maps regardless of errors\n self._server_sessions.pop(server_name, None)\n self._sessions.pop(session_name, None)\n \n if error_occurred:\n logger.warning(f\"Session '{session_name}' removed from tracking despite close error\")\n\n async def list_tools(self, session_name: str | None = None, use_cache: bool = True) -> List[BaseTool]:\n \"\"\"List tools from MCP sessions.\n \n Args:\n session_name: If provided, only list tools from that session.\n If None, list tools from all sessions.\n use_cache: If True, try to load from cache first (no server startup).\n If False, start servers and get live tools.\n \n Returns:\n List of BaseTool instances\n \"\"\"\n await self.ensure_initialized()\n \n # Case 1: List tools from specific session (always live, no cache)\n if session_name:\n sess = self._sessions.get(session_name)\n if sess:\n try:\n tools = await sess.list_tools()\n server_name = session_name.replace(f\"{self.backend_type.value}-\", \"\", 1)\n for tool in tools:\n tool.bind_runtime_info(\n backend=self.backend_type,\n session_name=session_name,\n server_name=server_name,\n )\n return tools\n except Exception as e:\n logger.error(f\"Error listing tools from session '{session_name}': {e}\")\n return []\n else:\n logger.warning(f\"Session '{session_name}' not found\")\n return []\n\n # Case 2: List tools from all servers\n # Try cache first if enabled\n if use_cache:\n cache = get_tool_cache()\n if cache.has_cache():\n tools = self._load_tools_from_cache()\n if tools:\n logger.info(f\"Loaded {len(tools)} tools from cache (no server startup)\")\n return tools\n \n # No cache or cache disabled, start servers\n return await self._list_tools_live()\n \n def _load_tools_from_cache(self) -> List[BaseTool]:\n \"\"\"Load tools from cache file without starting servers.\n \n Priority:\n 1. Try to load from sanitized cache (mcp_tool_cache_sanitized.json)\n 2. If not exists, load from raw cache and sanitize, then save sanitized version\n \"\"\"\n cache = get_tool_cache()\n config_servers = self.list_servers()\n \n # Try sanitized cache first\n if cache.has_sanitized_cache():\n logger.debug(\"Loading from sanitized cache\")\n all_cached_tools = cache.get_all_sanitized_tools()\n return self._build_tools_from_cache(all_cached_tools, config_servers)\n \n # Fall back to raw cache, sanitize and save\n if cache.has_cache():\n logger.info(\"Sanitized cache not found, building from raw cache...\")\n all_cached_tools = cache.get_all_tools()\n sanitized_servers = self._sanitize_and_save_cache(all_cached_tools, cache)\n return self._build_tools_from_cache(sanitized_servers, config_servers)\n \n return []\n \n def _sanitize_and_save_cache(\n self, \n raw_tools: Dict[str, List[Dict]], \n cache\n ) -> Dict[str, List[Dict]]:\n \"\"\"Sanitize raw cache and save to sanitized cache file.\"\"\"\n sanitized_servers: Dict[str, List[Dict]] = {}\n \n for server_name, tool_list in raw_tools.items():\n sanitized_tools = []\n for tool_meta in tool_list:\n raw_params = tool_meta.get(\"parameters\", {})\n sanitized_params = _sanitize_mcp_schema(raw_params)\n sanitized_tools.append({\n \"name\": tool_meta[\"name\"],\n \"description\": tool_meta.get(\"description\", \"\"),\n \"parameters\": sanitized_params,\n })\n sanitized_servers[server_name] = sanitized_tools\n \n # Save sanitized cache for future use\n cache.save_sanitized(sanitized_servers)\n logger.info(f\"Created sanitized cache with {len(sanitized_servers)} servers\")\n \n return sanitized_servers\n \n def _build_tools_from_cache(\n self, \n all_cached_tools: Dict[str, List[Dict]], \n config_servers: List[str]\n ) -> List[BaseTool]:\n \"\"\"Build BaseTool instances from cached tool metadata.\"\"\"\n tools: List[BaseTool] = []\n \n for server_name in config_servers:\n tool_list = all_cached_tools.get(server_name)\n if not tool_list:\n continue\n \n session_name = f\"{self.backend_type.value}-{server_name}\"\n for tool_meta in tool_list:\n schema = ToolSchema(\n name=tool_meta[\"name\"],\n description=tool_meta.get(\"description\", \"\"),\n parameters=tool_meta.get(\"parameters\", {}),\n backend_type=BackendType.MCP,\n )\n tool = RemoteTool(schema=schema, connector=None)\n tool.bind_runtime_info(\n backend=self.backend_type,\n session_name=session_name,\n server_name=server_name,\n )\n tools.append(tool)\n \n return tools\n \n async def _list_tools_live(self) -> List[BaseTool]:\n \"\"\"List tools by starting all servers.\n \n Uses a semaphore to serialize session creation, avoiding TaskGroup race conditions\n that occur when multiple MCP connections are initialized concurrently.\n \"\"\"\n servers = self.list_servers()\n \n if not servers:\n logger.warning(\"No MCP servers configured\")\n return []\n \n # Find servers that don't have sessions yet\n to_create = [s for s in servers if s not in self._server_sessions]\n\n # Create missing sessions with serialized execution using semaphore\n if to_create:\n logger.info(f\"Creating {len(to_create)} MCP sessions (serialized to avoid race conditions)\")\n \n # Use semaphore with limit=1 to serialize session creation\n # This avoids TaskGroup race conditions in concurrent HTTP connection setup\n semaphore = asyncio.Semaphore(1)\n \n async def _create_with_semaphore(server: str):\n async with semaphore:\n logger.debug(f\"Creating session for '{server}'\")\n return await self._lazy_create(server)\n \n tasks = [_create_with_semaphore(s) for s in to_create]\n results = await asyncio.gather(*tasks, return_exceptions=True)\n \n # Log errors\n for i, result in enumerate(results):\n if isinstance(result, MCPDependencyError):\n logger.debug(f\"Dependency error for '{to_create[i]}': {type(result).__name__}\")\n elif isinstance(result, Exception):\n logger.error(f\"Failed to create session for '{to_create[i]}': {result}\")\n\n # Aggregate tools from all sessions\n uniq: Dict[tuple[str, str], BaseTool] = {}\n failed_servers = []\n \n logger.debug(f\"Listing tools from {len(self._server_sessions)} sessions\")\n for server, sess in self._server_sessions.items():\n try:\n tools = await sess.list_tools()\n session_name = f\"{self.backend_type.value}-{server}\"\n for tool in tools:\n key = (server, tool.schema.name)\n if key not in uniq:\n tool.bind_runtime_info(\n backend=self.backend_type,\n session_name=session_name,\n server_name=server,\n )\n uniq[key] = tool\n except Exception as e:\n failed_servers.append(server)\n logger.error(f\"Error listing tools from server '{server}': {e}\")\n \n if failed_servers:\n logger.warning(f\"Failed to list tools from {len(failed_servers)} server(s): {failed_servers}\")\n \n tools_list = list(uniq.values())\n logger.debug(f\"Listed {len(tools_list)} unique tools from {len(self._server_sessions)} MCP servers\")\n \n # Save to cache for next time\n await self._save_tools_to_cache(tools_list)\n \n return tools_list\n \n async def _save_tools_to_cache(self, tools: List[BaseTool]) -> None:\n \"\"\"Save tools metadata to cache file.\"\"\"\n cache = get_tool_cache()\n \n # Group tools by server\n servers: Dict[str, List[Dict]] = {}\n for tool in tools:\n server_name = tool.runtime_info.server_name if tool.is_bound else \"unknown\"\n if server_name not in servers:\n servers[server_name] = []\n servers[server_name].append({\n \"name\": tool.schema.name,\n \"description\": tool.schema.description or \"\",\n \"parameters\": tool.schema.parameters or {},\n })\n \n cache.save(servers)\n \n async def ensure_server_session(self, server_name: str) -> Optional[MCPSession]:\n \"\"\"Ensure a server session exists, creating it if needed.\n \n This is used for on-demand server startup when executing tools.\n \"\"\"\n if server_name in self._server_sessions:\n return self._server_sessions[server_name]\n \n # Server not running, start it\n logger.info(f\"Starting MCP server on-demand: {server_name}\")\n cfg = SessionConfig(\n session_name=f\"mcp-{server_name}\",\n backend_type=BackendType.MCP,\n connection_params={\"server\": server_name},\n )\n \n try:\n session = await self.create_session(cfg)\n return session\n except Exception as e:\n logger.error(f\"Failed to start server '{server_name}': {e}\")\n return None\n\n async def _lazy_create(self, server: str) -> None:\n \"\"\"Internal helper for lazy session creation.\n \n Args:\n server: Server name to create session for\n \n Raises:\n Exception: Re-raises any exception from session creation for error tracking\n \"\"\"\n # Double-check to avoid race conditions\n if server in self._server_sessions:\n logger.debug(f\"Session for server '{server}' already exists, skipping lazy creation\")\n return\n \n cfg = SessionConfig(\n session_name=f\"mcp-{server}\",\n backend_type=BackendType.MCP,\n connection_params={\"server\": server},\n )\n \n try:\n await self.create_session(cfg)\n logger.debug(f\"Lazily created session for server '{server}'\")\n except MCPDependencyError as e:\n # Dependency errors already shown to user\n logger.debug(f\"Dependency error for server '{server}': {type(e).__name__}\")\n # Re-raise so that asyncio.gather can track the error\n raise\n except Exception as e:\n logger.error(f\"Failed to lazily create session for server '{server}': {e}\")\n # Re-raise so that asyncio.gather can track the error\n raise"
},
{
"path": "anytool/grounding/backends/mcp/session.py",
"content": "\"\"\"\nSession manager for MCP connections.\n\nThis module provides a session manager for MCP connections,\nwhich handles authentication, initialization, and tool discovery.\n\"\"\"\n\nfrom typing import Any, Dict\n\nfrom anytool.grounding.backends.mcp.transport.connectors import MCPBaseConnector\nfrom anytool.grounding.backends.mcp.tool_converter import convert_mcp_tool_to_base_tool\nfrom anytool.grounding.core.session import BaseSession\nfrom anytool.grounding.core.types import BackendType\nfrom anytool.utils.logging import Logger\n\nlogger = Logger.get_logger(__name__)\n\n\nclass MCPSession(BaseSession):\n \"\"\"Session manager for MCP connections.\n\n This class manages the lifecycle of an MCP connection, including\n authentication, initialization, and tool discovery.\n \"\"\"\n\n def __init__(\n self,\n connector: MCPBaseConnector,\n *,\n session_id: str = \"\",\n auto_connect: bool = True,\n auto_initialize: bool = True,\n ) -> None:\n \"\"\"Initialize a new MCP session.\n\n Args:\n connector: The connector to use for communicating with the MCP implementation.\n session_id: Unique identifier for this session\n auto_connect: Whether to automatically connect to the MCP implementation.\n auto_initialize: Whether to automatically initialize the session.\n \"\"\"\n super().__init__(\n connector=connector,\n session_id=session_id,\n backend_type=BackendType.MCP,\n auto_connect=auto_connect,\n auto_initialize=auto_initialize,\n )\n\n async def initialize(self) -> Dict[str, Any]:\n \"\"\"Initialize the MCP session and discover available tools.\n\n Returns:\n The session information returned by the MCP implementation.\n \"\"\"\n # Make sure we're connected\n if not self.is_connected and self.auto_connect:\n await self.connect()\n\n # Initialize the session through connector\n logger.debug(f\"Initializing MCP session {self.session_id}\")\n session_info = await self.connector.initialize()\n\n # List tools from MCP server and convert to BaseTool\n mcp_tools = self.connector.tools # MCPBaseConnector caches tools after initialize\n logger.debug(f\"Converting {len(mcp_tools)} MCP tools to BaseTool\")\n \n self.tools = [\n convert_mcp_tool_to_base_tool(mcp_tool, self.connector)\n for mcp_tool in mcp_tools\n ]\n \n logger.debug(f\"MCP session {self.session_id} initialized with {len(self.tools)} tools\")\n\n return session_info"
},
{
"path": "anytool/grounding/backends/mcp/tool_cache.py",
"content": "import json\nfrom pathlib import Path\nfrom datetime import datetime\nfrom typing import Any, Dict, List, Optional\n\nfrom anytool.utils.logging import Logger\n\nlogger = Logger.get_logger(__name__)\n\n# Cache path in project root directory (AnyTool/)\n# __file__ = .../AnyTool/anytool/grounding/backends/mcp/tool_cache.py\n# parent x5 = .../AnyTool/\nDEFAULT_CACHE_PATH = Path(__file__).parent.parent.parent.parent.parent / \"mcp_tool_cache.json\"\n# Sanitized cache path (Claude API compatible JSON Schema)\nDEFAULT_SANITIZED_CACHE_PATH = Path(__file__).parent.parent.parent.parent.parent / \"mcp_tool_cache_sanitized.json\"\n\n\nclass MCPToolCache:\n \"\"\"Simple file-based cache for MCP tool metadata.\"\"\"\n \n CACHE_VERSION = 1\n \n def __init__(self, cache_path: Optional[Path] = None, sanitized_cache_path: Optional[Path] = None):\n self.cache_path = cache_path or DEFAULT_CACHE_PATH\n self.sanitized_cache_path = sanitized_cache_path or DEFAULT_SANITIZED_CACHE_PATH\n self._cache: Optional[Dict] = None\n self._sanitized_cache: Optional[Dict] = None\n self._server_order: Optional[List[str]] = None\n \n def set_server_order(self, order: List[str]):\n \"\"\"Set expected server order (from config). Used when saving to disk.\"\"\"\n self._server_order = order\n \n def _reorder_servers(self, servers: Dict[str, List[Dict]]) -> Dict[str, List[Dict]]:\n \"\"\"Reorder servers dict according to _server_order.\"\"\"\n if not self._server_order:\n return servers\n \n ordered = {}\n # First add servers in config order\n for name in self._server_order:\n if name in servers:\n ordered[name] = servers[name]\n # Then add any remaining servers (not in config)\n for name in servers:\n if name not in ordered:\n ordered[name] = servers[name]\n return ordered\n \n def _ensure_dir(self):\n \"\"\"Ensure cache directory exists.\"\"\"\n self.cache_path.parent.mkdir(parents=True, exist_ok=True)\n \n def load(self) -> Dict[str, Any]:\n \"\"\"Load cache from disk. Returns empty dict if not exists.\"\"\"\n if self._cache is not None:\n return self._cache\n \n if not self.cache_path.exists():\n self._cache = {\"version\": self.CACHE_VERSION, \"servers\": {}}\n return self._cache\n \n try:\n with open(self.cache_path, \"r\", encoding=\"utf-8\") as f:\n self._cache = json.load(f)\n logger.info(f\"Loaded MCP tool cache: {len(self._cache.get('servers', {}))} servers\")\n return self._cache\n except Exception as e:\n logger.warning(f\"Failed to load cache: {e}\")\n self._cache = {\"version\": self.CACHE_VERSION, \"servers\": {}}\n return self._cache\n \n def save(self, servers: Dict[str, List[Dict]]):\n \"\"\"\n Save tool metadata to disk (overwrites existing cache).\n \n Args:\n servers: Dict mapping server_name -> list of tool metadata dicts\n Each tool dict should have: name, description, parameters\n \"\"\"\n self._ensure_dir()\n \n cache_data = {\n \"version\": self.CACHE_VERSION,\n \"updated_at\": datetime.now().isoformat(),\n \"servers\": servers,\n }\n \n try:\n with open(self.cache_path, \"w\", encoding=\"utf-8\") as f:\n json.dump(cache_data, f, indent=2, ensure_ascii=False)\n self._cache = cache_data\n logger.info(f\"Saved MCP tool cache: {len(servers)} servers\")\n except Exception as e:\n logger.error(f\"Failed to save cache: {e}\")\n \n def save_server(self, server_name: str, tools: List[Dict]):\n \"\"\"\n Save/update a single server's tools to cache (incremental append).\n \n Args:\n server_name: Name of the MCP server\n tools: List of tool metadata dicts for this server\n \"\"\"\n self._ensure_dir()\n \n # Load existing cache\n cache = self.load()\n \n # Update server entry\n if \"servers\" not in cache:\n cache[\"servers\"] = {}\n cache[\"servers\"][server_name] = tools\n cache[\"servers\"] = self._reorder_servers(cache[\"servers\"])\n cache[\"updated_at\"] = datetime.now().isoformat()\n \n # Save back\n try:\n with open(self.cache_path, \"w\", encoding=\"utf-8\") as f:\n json.dump(cache, f, indent=2, ensure_ascii=False)\n self._cache = cache\n logger.debug(f\"Saved {len(tools)} tools for server '{server_name}'\")\n except Exception as e:\n logger.error(f\"Failed to save cache for server '{server_name}': {e}\")\n \n def get_server_tools(self, server_name: str) -> Optional[List[Dict]]:\n \"\"\"Get cached tools for a specific server.\"\"\"\n cache = self.load()\n return cache.get(\"servers\", {}).get(server_name)\n \n def get_all_tools(self) -> Dict[str, List[Dict]]:\n \"\"\"Get all cached tools, grouped by server.\"\"\"\n cache = self.load()\n return cache.get(\"servers\", {})\n \n def has_cache(self) -> bool:\n \"\"\"Check if cache exists and has data.\"\"\"\n cache = self.load()\n return bool(cache.get(\"servers\"))\n \n def clear(self):\n \"\"\"Clear the cache.\"\"\"\n if self.cache_path.exists():\n self.cache_path.unlink()\n self._cache = None\n logger.info(\"MCP tool cache cleared\")\n \n def save_failed_server(self, server_name: str, error: str):\n \"\"\"\n Record a failed server to cache.\n \n Args:\n server_name: Name of the failed MCP server\n error: Error message\n \"\"\"\n self._ensure_dir()\n \n # Load existing cache\n cache = self.load()\n \n # Add to failed_servers list\n if \"failed_servers\" not in cache:\n cache[\"failed_servers\"] = {}\n cache[\"failed_servers\"][server_name] = {\n \"error\": error,\n \"failed_at\": datetime.now().isoformat(),\n }\n cache[\"updated_at\"] = datetime.now().isoformat()\n \n # Save back\n try:\n with open(self.cache_path, \"w\", encoding=\"utf-8\") as f:\n json.dump(cache, f, indent=2, ensure_ascii=False)\n self._cache = cache\n except Exception as e:\n logger.error(f\"Failed to save failed server '{server_name}': {e}\")\n \n def get_failed_servers(self) -> Dict[str, Dict]:\n \"\"\"Get list of failed servers from cache.\"\"\"\n cache = self.load()\n return cache.get(\"failed_servers\", {})\n \n def load_sanitized(self) -> Dict[str, Any]:\n \"\"\"Load sanitized cache from disk. Returns empty dict if not exists.\"\"\"\n if self._sanitized_cache is not None:\n return self._sanitized_cache\n \n if not self.sanitized_cache_path.exists():\n self._sanitized_cache = {\"version\": self.CACHE_VERSION, \"servers\": {}}\n return self._sanitized_cache\n \n try:\n with open(self.sanitized_cache_path, \"r\", encoding=\"utf-8\") as f:\n self._sanitized_cache = json.load(f)\n logger.info(f\"Loaded sanitized MCP tool cache: {len(self._sanitized_cache.get('servers', {}))} servers\")\n return self._sanitized_cache\n except Exception as e:\n logger.warning(f\"Failed to load sanitized cache: {e}\")\n self._sanitized_cache = {\"version\": self.CACHE_VERSION, \"servers\": {}}\n return self._sanitized_cache\n \n def save_sanitized(self, servers: Dict[str, List[Dict]]):\n \"\"\"\n Save sanitized tool metadata to disk.\n \n Args:\n servers: Dict mapping server_name -> list of sanitized tool metadata dicts\n \"\"\"\n self._ensure_dir()\n \n cache_data = {\n \"version\": self.CACHE_VERSION,\n \"updated_at\": datetime.now().isoformat(),\n \"sanitized\": True,\n \"servers\": servers,\n }\n \n try:\n with open(self.sanitized_cache_path, \"w\", encoding=\"utf-8\") as f:\n json.dump(cache_data, f, indent=2, ensure_ascii=False)\n self._sanitized_cache = cache_data\n logger.info(f\"Saved sanitized MCP tool cache: {len(servers)} servers\")\n except Exception as e:\n logger.error(f\"Failed to save sanitized cache: {e}\")\n \n def get_all_sanitized_tools(self) -> Dict[str, List[Dict]]:\n \"\"\"Get all sanitized cached tools, grouped by server.\"\"\"\n cache = self.load_sanitized()\n return cache.get(\"servers\", {})\n \n def has_sanitized_cache(self) -> bool:\n \"\"\"Check if sanitized cache exists and has data.\"\"\"\n cache = self.load_sanitized()\n return bool(cache.get(\"servers\"))\n \n def clear_sanitized(self):\n \"\"\"Clear the sanitized cache.\"\"\"\n if self.sanitized_cache_path.exists():\n self.sanitized_cache_path.unlink()\n self._sanitized_cache = None\n logger.info(\"Sanitized MCP tool cache cleared\")\n\n\n# Global instance\n_tool_cache: Optional[MCPToolCache] = None\n\n\ndef get_tool_cache() -> MCPToolCache:\n \"\"\"Get global tool cache instance.\"\"\"\n global _tool_cache\n if _tool_cache is None:\n _tool_cache = MCPToolCache()\n return _tool_cache\n\n"
},
{
"path": "anytool/grounding/backends/mcp/tool_converter.py",
"content": "\"\"\"\nTool converter for MCP.\n\nThis module provides utilities to convert MCP tools to BaseTool instances.\n\"\"\"\n\nimport copy\nfrom typing import Any, Dict\nfrom mcp.types import Tool as MCPTool\n\nfrom anytool.grounding.core.tool import BaseTool, RemoteTool\nfrom anytool.grounding.core.types import BackendType, ToolSchema\nfrom anytool.grounding.core.transport.connectors import BaseConnector\nfrom anytool.utils.logging import Logger\n\nlogger = Logger.get_logger(__name__)\n\n\ndef _sanitize_mcp_schema(params: Dict[str, Any]) -> Dict[str, Any]:\n \"\"\"\n Sanitize MCP tool schema to ensure Claude API compatibility (JSON Schema draft 2020-12).\n \n Fixes:\n - Empty schemas -> valid object schema\n - Missing required fields (type, properties, required)\n - Removes non-standard fields (title, examples, nullable, default, etc.)\n - Recursively cleans nested properties and items\n - Ensures every property has a valid type\n - Ensures top-level type is 'object' (Anthropic API requirement)\n \"\"\"\n if not params:\n return {\"type\": \"object\", \"properties\": {}, \"required\": []}\n \n sanitized = copy.deepcopy(params)\n sanitized = _deep_sanitize(sanitized)\n \n # Anthropic API requires top-level type to be 'object'\n # If it's not an object, wrap the schema as a property of an object\n top_level_type = sanitized.get(\"type\")\n if top_level_type and top_level_type != \"object\":\n logger.debug(f\"[MCP_SCHEMA_SANITIZE] Wrapping non-object schema (type={top_level_type}) into object\")\n wrapped = {\n \"type\": \"object\",\n \"properties\": {\n \"value\": sanitized # The original schema becomes a property\n },\n \"required\": [\"value\"] # Make it required\n }\n sanitized = wrapped\n \n return sanitized\n\n\ndef _deep_sanitize(schema: Dict[str, Any]) -> Dict[str, Any]:\n \"\"\"\n Recursively sanitize a JSON schema to conform to JSON Schema draft 2020-12.\n Removes non-standard fields and ensures valid structure.\n \"\"\"\n if not isinstance(schema, dict):\n return {\"type\": \"string\"}\n \n # Allowed top-level keys for Claude API compatibility\n allowed_keys = {\n \"type\", \"properties\", \"required\", \"items\", \n \"description\", \"enum\", \"const\",\n \"minimum\", \"maximum\", \"minLength\", \"maxLength\",\n \"minItems\", \"maxItems\", \"pattern\",\n \"additionalProperties\", \"anyOf\", \"oneOf\", \"allOf\"\n }\n \n # Remove disallowed keys\n keys_to_remove = [k for k in schema if k not in allowed_keys]\n for k in keys_to_remove:\n schema.pop(k, None)\n \n # Ensure type exists\n if \"type\" not in schema:\n # Type is defined via anyOf/oneOf/allOf - don't add default type\n # These combination keywords define the type themselves\n if \"anyOf\" in schema or \"oneOf\" in schema or \"allOf\" in schema:\n pass # Type is defined through combination keywords, do not add default type\n # Try to infer type\n elif \"properties\" in schema:\n schema[\"type\"] = \"object\"\n elif \"items\" in schema:\n schema[\"type\"] = \"array\"\n elif \"enum\" in schema:\n # For enum, try to infer from values\n enum_vals = schema.get(\"enum\", [])\n if enum_vals and all(isinstance(v, str) for v in enum_vals):\n schema[\"type\"] = \"string\"\n elif enum_vals and all(isinstance(v, (int, float)) for v in enum_vals):\n schema[\"type\"] = \"number\"\n else:\n schema[\"type\"] = \"string\"\n elif not schema:\n # Empty schema (e.g., only had $schema which was removed) -> no parameters needed\n schema[\"type\"] = \"object\"\n schema[\"properties\"] = {}\n schema[\"required\"] = []\n else:\n schema[\"type\"] = \"object\"\n \n # Handle object type\n if schema.get(\"type\") == \"object\":\n if \"properties\" not in schema:\n schema[\"properties\"] = {}\n if \"required\" not in schema:\n schema[\"required\"] = []\n \n # Recursively sanitize properties\n if isinstance(schema.get(\"properties\"), dict):\n for prop_name, prop_schema in list(schema[\"properties\"].items()):\n if isinstance(prop_schema, dict):\n schema[\"properties\"][prop_name] = _deep_sanitize(prop_schema)\n else:\n # Invalid property schema, replace with string\n schema[\"properties\"][prop_name] = {\"type\": \"string\"}\n \n # Sanitize additionalProperties if present\n if \"additionalProperties\" in schema and isinstance(schema[\"additionalProperties\"], dict):\n schema[\"additionalProperties\"] = _deep_sanitize(schema[\"additionalProperties\"])\n \n # Handle array type\n elif schema.get(\"type\") == \"array\":\n if \"items\" in schema:\n if isinstance(schema[\"items\"], dict):\n schema[\"items\"] = _deep_sanitize(schema[\"items\"])\n elif isinstance(schema[\"items\"], list):\n # Tuple validation - sanitize each item\n schema[\"items\"] = [_deep_sanitize(item) if isinstance(item, dict) else {\"type\": \"string\"} for item in schema[\"items\"]]\n else:\n schema[\"items\"] = {\"type\": \"string\"}\n else:\n # Default items to string if not specified\n schema[\"items\"] = {\"type\": \"string\"}\n \n # Handle anyOf/oneOf/allOf\n for combo_key in [\"anyOf\", \"oneOf\", \"allOf\"]:\n if combo_key in schema and isinstance(schema[combo_key], list):\n schema[combo_key] = [\n _deep_sanitize(sub) if isinstance(sub, dict) else {\"type\": \"string\"}\n for sub in schema[combo_key]\n ]\n \n return schema\n\n\ndef convert_mcp_tool_to_base_tool(\n mcp_tool: MCPTool, \n connector: BaseConnector\n) -> BaseTool:\n \"\"\"\n Convert an MCP Tool to a BaseTool (RemoteTool) instance.\n \n This function extracts the tool schema from an MCP tool object and creates\n a RemoteTool that can be used within the grounding framework.\n \n Args:\n mcp_tool: MCP Tool object from the MCP SDK\n connector: Connector instance for communicating with the MCP server\n \n Returns:\n RemoteTool instance wrapping the MCP tool\n \"\"\"\n # Extract tool metadata\n tool_name = mcp_tool.name\n tool_description = getattr(mcp_tool, 'description', None) or \"\"\n \n # Convert MCP input schema to our parameter schema format (with sanitization)\n input_schema: Dict[str, Any] = {}\n if hasattr(mcp_tool, 'inputSchema') and mcp_tool.inputSchema:\n input_schema = _sanitize_mcp_schema(mcp_tool.inputSchema)\n else:\n input_schema = {\"type\": \"object\", \"properties\": {}, \"required\": []}\n \n # Create ToolSchema\n schema = ToolSchema(\n name=tool_name,\n description=tool_description,\n parameters=input_schema,\n backend_type=BackendType.MCP,\n )\n \n # Create and return RemoteTool\n remote_tool = RemoteTool(\n connector=connector,\n remote_name=tool_name,\n schema=schema,\n backend=BackendType.MCP,\n )\n \n logger.debug(f\"Converted MCP tool '{tool_name}' to RemoteTool\")\n return remote_tool"
},
{
"path": "anytool/grounding/backends/mcp/transport/connectors/__init__.py",
"content": "\"\"\"\nConnectors for various MCP transports.\n\nThis module provides interfaces for connecting to MCP implementations\nthrough different transport mechanisms.\n\"\"\"\n\nfrom .base import MCPBaseConnector # noqa: F401\nfrom .http import HttpConnector # noqa: F401\nfrom .sandbox import SandboxConnector # noqa: F401\nfrom .stdio import StdioConnector # noqa: F401\nfrom .websocket import WebSocketConnector # noqa: F401\n\n__all__ = [\n \"MCPBaseConnector\",\n \"StdioConnector\",\n \"HttpConnector\",\n \"WebSocketConnector\",\n \"SandboxConnector\",\n]\n"
},
{
"path": "anytool/grounding/backends/mcp/transport/connectors/base.py",
"content": "\"\"\"\nBase connector for MCP implementations.\n\nThis module provides the base connector interface that all MCP connectors must implement.\n\"\"\"\n\nimport asyncio\nfrom abc import abstractmethod\nfrom typing import Any\n\nfrom mcp import ClientSession\nfrom mcp.shared.exceptions import McpError\nfrom mcp.types import CallToolResult, GetPromptResult, Prompt, ReadResourceResult, Resource, Tool\n\nfrom anytool.grounding.core.transport.task_managers import BaseConnectionManager\nfrom anytool.grounding.core.transport.connectors import BaseConnector\nfrom anytool.utils.logging import Logger\n\nlogger = Logger.get_logger(__name__)\n\n# Default retry settings for tool calls\nDEFAULT_TOOL_CALL_MAX_RETRIES = 3\nDEFAULT_TOOL_CALL_RETRY_DELAY = 1.0\n\n\nclass MCPBaseConnector(BaseConnector[ClientSession]):\n \"\"\"Base class for MCP connectors.\n\n This class defines the interface that all MCP connectors must implement.\n \"\"\"\n\n def __init__(\n self, \n connection_manager: BaseConnectionManager[ClientSession],\n tool_call_max_retries: int = DEFAULT_TOOL_CALL_MAX_RETRIES,\n tool_call_retry_delay: float = DEFAULT_TOOL_CALL_RETRY_DELAY,\n ):\n \"\"\"Initialize base connector with common attributes.\n \n Args:\n connection_manager: The connection manager to use for the connection.\n tool_call_max_retries: Maximum number of retries for tool calls (default: 3)\n tool_call_retry_delay: Initial delay between retries in seconds (default: 1.0)\n \"\"\"\n super().__init__(connection_manager)\n self.client_session: ClientSession | None = None\n self._tools: list[Tool] | None = None\n self._resources: list[Resource] | None = None\n self._prompts: list[Prompt] | None = None\n self.auto_reconnect = True # Whether to automatically reconnect on connection loss (not configurable for now)\n self.tool_call_max_retries = tool_call_max_retries\n self.tool_call_retry_delay = tool_call_retry_delay\n\n @property\n @abstractmethod\n def public_identifier(self) -> str:\n \"\"\"Get the identifier for the connector.\"\"\"\n pass\n \n async def _get_streams_from_connection(self):\n \"\"\"Get read and write streams from the connection. Override in subclasses if needed.\"\"\"\n # Default implementation for most MCP connectors (stdio, HTTP)\n # Returns the connection directly as it should be a tuple of (read_stream, write_stream)\n return self._connection\n \n async def _after_connect(self) -> None:\n \"\"\"Create ClientSession after connection is established.\n \n Some connectors (like WebSocket) don't use ClientSession and may override this method.\n \"\"\"\n # Get streams from the connection\n streams = await self._get_streams_from_connection()\n \n if streams is None:\n # Some connectors (like WebSocket) don't use ClientSession\n # They should override this method to set up their own resources\n logger.debug(\"No streams returned, ClientSession creation skipped\")\n return\n \n if isinstance(streams, tuple) and len(streams) == 2:\n read_stream, write_stream = streams\n # Create the client session\n self.client_session = ClientSession(read_stream, write_stream, sampling_callback=None)\n await self.client_session.__aenter__()\n logger.debug(\"MCP ClientSession created successfully\")\n else:\n raise RuntimeError(f\"Invalid streams format: expected tuple of 2 elements, got {type(streams)}\")\n\n async def _before_disconnect(self) -> None:\n \"\"\"Clean up MCP-specific resources before disconnection.\"\"\"\n errors = []\n\n # Close the client session\n if self.client_session:\n try:\n logger.debug(\"Closing MCP client session\")\n await self.client_session.__aexit__(None, None, None)\n except Exception as e:\n error_msg = f\"Error closing client session: {e}\"\n logger.warning(error_msg)\n errors.append(error_msg)\n finally:\n self.client_session = None\n\n # Reset tools, resources, and prompts\n self._tools = None\n self._resources = None\n self._prompts = None\n\n if errors:\n logger.warning(f\"Encountered {len(errors)} errors during MCP resource cleanup\")\n \n async def _cleanup_on_connect_failure(self) -> None:\n \"\"\"Override to add MCP-specific cleanup on connection failure.\"\"\"\n # Clean up client session if it was created\n if self.client_session:\n try:\n await self.client_session.__aexit__(None, None, None)\n except Exception:\n pass\n finally:\n self.client_session = None\n \n # Call parent cleanup\n await super()._cleanup_on_connect_failure()\n\n async def initialize(self) -> dict[str, Any]:\n \"\"\"Initialize the MCP session and return session information.\"\"\"\n if not self.client_session:\n raise RuntimeError(\"MCP client is not connected\")\n\n logger.debug(\"Initializing MCP session\")\n\n # Initialize the session\n result = await self.client_session.initialize()\n\n server_capabilities = result.capabilities\n\n if server_capabilities.tools:\n # Get available tools\n tools_result = await self.list_tools()\n self._tools = tools_result or []\n else:\n self._tools = []\n\n if server_capabilities.resources:\n # Get available resources\n resources_result = await self.list_resources()\n self._resources = resources_result or []\n else:\n self._resources = []\n\n if server_capabilities.prompts:\n # Get available prompts\n prompts_result = await self.list_prompts()\n self._prompts = prompts_result or []\n else:\n self._prompts = []\n\n logger.debug(\n f\"MCP session initialized with {len(self._tools)} tools, \"\n f\"{len(self._resources)} resources, \"\n f\"and {len(self._prompts)} prompts\"\n )\n\n return result\n\n @property\n def tools(self) -> list[Tool]:\n \"\"\"Get the list of available tools.\"\"\"\n if self._tools is None:\n raise RuntimeError(\"MCP client is not initialized\")\n return self._tools\n\n @property\n def resources(self) -> list[Resource]:\n \"\"\"Get the list of available resources.\"\"\"\n if self._resources is None:\n raise RuntimeError(\"MCP client is not initialized\")\n return self._resources\n\n @property\n def prompts(self) -> list[Prompt]:\n \"\"\"Get the list of available prompts.\"\"\"\n if self._prompts is None:\n raise RuntimeError(\"MCP client is not initialized\")\n return self._prompts\n\n @property\n def is_connected(self) -> bool:\n \"\"\"Check if the connector is actually connected and the connection is alive.\n\n This property checks not only the connected flag but also verifies that\n the client session exists and the underlying connection is still active.\n\n Returns:\n True if the connector is connected and the connection is alive, False otherwise.\n \"\"\"\n # First check the basic connected flag\n if not self._connected:\n return False\n\n # Check if we have a client session\n if not self.client_session:\n self._connected = False\n return False\n\n # Check if connection manager task is still running (if applicable)\n if self._connection_manager and hasattr(self._connection_manager, \"_task\"):\n task = self._connection_manager._task\n if task and task.done():\n logger.debug(\"Connection manager task is done, marking as disconnected\")\n self._connected = False\n return False\n\n return True\n\n async def _ensure_connected(self) -> None:\n \"\"\"Ensure the connector is connected, reconnecting if necessary.\n\n Raises:\n RuntimeError: If connection cannot be established and auto_reconnect is False.\n \"\"\"\n if not self.client_session:\n raise RuntimeError(\"MCP client is not connected\")\n\n if not self.is_connected:\n if self.auto_reconnect:\n logger.debug(\"Connection lost, attempting to reconnect...\")\n try:\n await self.connect()\n logger.debug(\"Reconnection successful\")\n except Exception as e:\n raise RuntimeError(f\"Failed to reconnect to MCP server: {e}\") from e\n else:\n raise RuntimeError(\n \"Connection to MCP server has been lost. Auto-reconnection is disabled. Please reconnect manually.\"\n )\n\n async def call_tool(self, name: str, arguments: dict[str, Any]) -> CallToolResult:\n \"\"\"Call an MCP tool with automatic reconnection handling and retry logic.\n\n Args:\n name: The name of the tool to call.\n arguments: The arguments to pass to the tool.\n\n Returns:\n The result of the tool call.\n\n Raises:\n RuntimeError: If the connection is lost and cannot be reestablished.\n Exception: If the tool call fails after all retries.\n \"\"\"\n last_error: Exception | None = None\n \n for attempt in range(self.tool_call_max_retries):\n # Ensure we're connected\n await self._ensure_connected()\n\n logger.debug(f\"Calling tool '{name}' with arguments: {arguments} (attempt {attempt + 1}/{self.tool_call_max_retries})\")\n try:\n result = await self.client_session.call_tool(name, arguments)\n logger.debug(f\"Tool '{name}' called successfully\")\n return result\n except Exception as e:\n last_error = e\n error_str = str(e).lower()\n \n # Check if the error might be due to connection loss\n if not self.is_connected:\n logger.warning(f\"Tool call '{name}' failed due to connection loss: {e}\")\n # Try to reconnect on next iteration\n continue\n \n # Check for retryable HTTP errors (400, 500, 502, 503, 504)\n is_retryable = any(code in error_str for code in ['400', '500', '502', '503', '504', 'bad request', 'internal server error', 'service unavailable', 'gateway timeout'])\n \n if is_retryable and attempt < self.tool_call_max_retries - 1:\n delay = self.tool_call_retry_delay * (2 ** attempt) # Exponential backoff\n logger.warning(\n f\"Tool call '{name}' failed with retryable error: {e}, \"\n f\"retrying in {delay:.1f}s (attempt {attempt + 1}/{self.tool_call_max_retries})\"\n )\n await asyncio.sleep(delay)\n continue\n \n # Non-retryable error or max retries reached, re-raise\n raise\n \n # All retries exhausted\n error_msg = f\"Tool call '{name}' failed after {self.tool_call_max_retries} retries\"\n logger.error(error_msg)\n raise RuntimeError(error_msg) from last_error\n\n async def list_tools(self) -> list[Tool]:\n \"\"\"List all available tools from the MCP implementation.\"\"\"\n\n # Ensure we're connected\n await self._ensure_connected()\n\n logger.debug(\"Listing tools\")\n try:\n result = await self.client_session.list_tools()\n return result.tools\n except McpError as e:\n logger.error(f\"Error listing tools: {e}\")\n return []\n\n async def list_resources(self) -> list[Resource]:\n \"\"\"List all available resources from the MCP implementation.\"\"\"\n # Ensure we're connected\n await self._ensure_connected()\n\n logger.debug(\"Listing resources\")\n try:\n result = await self.client_session.list_resources()\n return result.resources\n except McpError as e:\n logger.error(f\"Error listing resources: {e}\")\n return []\n\n async def read_resource(self, uri: str) -> ReadResourceResult:\n \"\"\"Read a resource by URI.\"\"\"\n if not self.client_session:\n raise RuntimeError(\"MCP client is not connected\")\n\n logger.debug(f\"Reading resource: {uri}\")\n result = await self.client_session.read_resource(uri)\n return result\n\n async def list_prompts(self) -> list[Prompt]:\n \"\"\"List all available prompts from the MCP implementation.\"\"\"\n # Ensure we're connected\n await self._ensure_connected()\n\n logger.debug(\"Listing prompts\")\n try:\n result = await self.client_session.list_prompts()\n return result.prompts\n except McpError as e:\n logger.error(f\"Error listing prompts: {e}\")\n return []\n\n async def get_prompt(self, name: str, arguments: dict[str, Any] | None = None) -> GetPromptResult:\n \"\"\"Get a prompt by name.\"\"\"\n # Ensure we're connected\n await self._ensure_connected()\n\n logger.debug(f\"Getting prompt: {name}\")\n result = await self.client_session.get_prompt(name, arguments)\n return result\n\n async def request(self, method: str, params: dict[str, Any] | None = None) -> Any:\n \"\"\"Send a raw request to the MCP implementation.\"\"\"\n # Ensure we're connected\n await self._ensure_connected()\n\n logger.debug(f\"Sending request: {method} with params: {params}\")\n return await self.client_session.request({\"method\": method, \"params\": params or {}})\n\n async def invoke(self, name: str, params: dict[str, Any]) -> Any:\n await self._ensure_connected()\n\n if not name.startswith(\"__\"):\n return await self.call_tool(name, params)\n\n if name == \"__read_resource__\":\n return await self.read_resource(params[\"uri\"])\n if name == \"__list_prompts__\":\n return await self.list_prompts()\n if name == \"__get_prompt__\":\n return await self.get_prompt(params[\"name\"], params.get(\"args\"))\n\n raise ValueError(f\"Unsupported MCP invoke name: {name}\")"
},
{
"path": "anytool/grounding/backends/mcp/transport/connectors/http.py",
"content": "\"\"\"\nHTTP connector for MCP implementations.\n\nThis module provides a connector for communicating with MCP implementations\nthrough HTTP APIs with SSE, Streamable HTTP, or simple JSON-RPC for transport.\n\"\"\"\n\nimport asyncio\nimport anyio\nimport httpx\nfrom typing import Any, Dict, List\nfrom mcp import ClientSession\nfrom mcp.types import (\n CallToolResult,\n TextContent,\n ImageContent,\n EmbeddedResource,\n Tool,\n Resource,\n Prompt,\n GetPromptResult,\n ReadResourceResult,\n)\n\nfrom anytool.utils.logging import Logger\nfrom anytool.grounding.core.transport.task_managers.base import BaseConnectionManager\nfrom anytool.grounding.backends.mcp.transport.task_managers import SseConnectionManager, StreamableHttpConnectionManager\nfrom anytool.grounding.backends.mcp.transport.connectors.base import MCPBaseConnector, DEFAULT_TOOL_CALL_MAX_RETRIES, DEFAULT_TOOL_CALL_RETRY_DELAY\n\nlogger = Logger.get_logger(__name__)\n\n\nclass HttpConnector(MCPBaseConnector):\n \"\"\"Connector for MCP implementations using HTTP transport.\n\n This connector uses HTTP/SSE or streamable HTTP to communicate with remote MCP implementations,\n using a connection manager to handle the proper lifecycle management.\n \"\"\"\n\n def __init__(\n self,\n base_url: str,\n auth_token: str | None = None,\n headers: dict[str, str] | None = None,\n timeout: float = 5,\n sse_read_timeout: float = 60 * 5,\n tool_call_max_retries: int = DEFAULT_TOOL_CALL_MAX_RETRIES,\n tool_call_retry_delay: float = DEFAULT_TOOL_CALL_RETRY_DELAY,\n ):\n \"\"\"Initialize a new HTTP connector.\n\n Args:\n base_url: The base URL of the MCP HTTP API.\n auth_token: Optional authentication token.\n headers: Optional additional headers.\n timeout: Timeout for HTTP operations in seconds.\n sse_read_timeout: Timeout for SSE read operations in seconds.\n tool_call_max_retries: Maximum number of retries for tool calls (default: 3)\n tool_call_retry_delay: Initial delay between retries in seconds (default: 1.0)\n \"\"\"\n self.base_url = base_url.rstrip(\"/\")\n self.auth_token = auth_token\n self.headers = headers or {}\n if auth_token:\n self.headers[\"Authorization\"] = f\"Bearer {auth_token}\"\n self.timeout = timeout\n self.sse_read_timeout = sse_read_timeout\n \n # JSON-RPC HTTP mode fields\n self._use_jsonrpc = False\n self._jsonrpc_client: httpx.AsyncClient | None = None\n self._jsonrpc_request_id = 0\n \n # Create a placeholder connection manager (will be set up later in connect())\n # We use a placeholder here because the actual transport type (SSE vs Streamable HTTP)\n # can only be determined at runtime through server negotiation as per MCP specification\n from anytool.grounding.core.transport.task_managers import PlaceholderConnectionManager\n connection_manager = PlaceholderConnectionManager()\n super().__init__(\n connection_manager, \n tool_call_max_retries=tool_call_max_retries,\n tool_call_retry_delay=tool_call_retry_delay,\n )\n\n async def connect(self) -> None:\n \"\"\"Create the underlying session/connection.\n \n For JSON-RPC mode, we don't use a connection manager.\n \"\"\"\n if self._connected:\n return\n \n try:\n # Hook: before connection - this sets up transport type\n await self._before_connect()\n \n if self._use_jsonrpc:\n # JSON-RPC mode doesn't use connection manager\n # Just call _after_connect to set up the HTTP client\n await self._after_connect()\n self._connected = True\n else:\n # Use normal connection flow with connection manager\n # If _before_connect() already established a connection, reuse it\n if self._connection is None:\n self._connection = await self._connection_manager.start()\n await self._after_connect()\n self._connected = True\n except Exception:\n await self._cleanup_on_connect_failure()\n raise\n\n async def disconnect(self) -> None:\n \"\"\"Close the session/connection and reset state.\"\"\"\n if not self._connected:\n return\n \n # Hook: before disconnection\n await self._before_disconnect()\n \n if not self._use_jsonrpc:\n # Stop the connection manager only for non-JSON-RPC modes\n if self._connection_manager:\n await self._connection_manager.stop()\n self._connection = None\n \n # Hook: after disconnection\n await self._after_disconnect()\n \n self._connected = False\n\n async def _before_connect(self) -> None:\n \"\"\"Negotiate transport type and set up the appropriate connection manager.\n \n Tries transports in order:\n 1. Streamable HTTP (new MCP transport)\n 2. SSE (legacy MCP transport)\n 3. Simple JSON-RPC HTTP (for custom servers)\n \n This implements backwards compatibility per MCP specification.\n \"\"\"\n self.transport_type = None\n self._use_jsonrpc = False\n connection_manager = None\n streamable_error = None\n sse_error = None\n\n # First, try the new streamable HTTP transport\n try:\n logger.debug(f\"Attempting streamable HTTP connection to: {self.base_url}\")\n connection_manager = StreamableHttpConnectionManager(\n self.base_url, self.headers, self.timeout, self.sse_read_timeout\n )\n\n # Test the connection by starting it with built-in timeout\n read_stream, write_stream = await connection_manager.start(timeout=self.timeout)\n\n # Create and verify ClientSession\n test_client = ClientSession(read_stream, write_stream, sampling_callback=None)\n \n # Add timeout to __aenter__ - use asyncio.wait_for instead of anyio.fail_after\n # to avoid cancel scope conflicts with background tasks\n try:\n await asyncio.wait_for(test_client.__aenter__(), timeout=self.timeout)\n except asyncio.TimeoutError:\n raise TimeoutError(f\"ClientSession enter timed out after {self.timeout}s\")\n\n try:\n # Add timeout to initialize() using asyncio.wait_for to prevent hanging\n try:\n await asyncio.wait_for(test_client.initialize(), timeout=self.timeout)\n except asyncio.TimeoutError:\n raise TimeoutError(f\"initialize() timed out after {self.timeout}s\")\n \n try:\n await asyncio.wait_for(test_client.list_tools(), timeout=self.timeout)\n except asyncio.TimeoutError:\n raise TimeoutError(f\"list_tools() timed out after {self.timeout}s\")\n \n # SUCCESS! Keep the client session (don't close it, closing destroys the streams)\n # Store it directly as the client_session for later use\n self.transport_type = \"streamable HTTP\"\n self._connection_manager = connection_manager\n self._connection = connection_manager.get_streams()\n self.client_session = test_client # Reuse the working session\n logger.debug(\"Streamable HTTP transport selected\")\n return\n except TimeoutError:\n try:\n await asyncio.wait_for(test_client.__aexit__(None, None, None), timeout=2)\n except (asyncio.TimeoutError, Exception):\n pass\n raise\n except Exception as init_error:\n # Clean up the test client only on error\n try:\n await asyncio.wait_for(test_client.__aexit__(None, None, None), timeout=2)\n except (asyncio.TimeoutError, Exception):\n pass\n raise init_error\n\n except Exception as e:\n streamable_error = e\n logger.debug(f\"Streamable HTTP failed: {e}\")\n\n # Clean up the failed connection manager\n if connection_manager:\n try:\n await asyncio.wait_for(connection_manager.stop(), timeout=2)\n except (asyncio.TimeoutError, Exception):\n pass\n\n # Try SSE fallback\n try:\n logger.debug(f\"Attempting SSE fallback connection to: {self.base_url}\")\n connection_manager = SseConnectionManager(\n self.base_url, self.headers, self.timeout, self.sse_read_timeout\n )\n\n # Test the connection by starting it with built-in timeout\n read_stream, write_stream = await connection_manager.start(timeout=self.timeout)\n\n # Create and verify ClientSession\n test_client = ClientSession(read_stream, write_stream, sampling_callback=None)\n \n # Add timeout to __aenter__ - use asyncio.wait_for instead of anyio.fail_after\n # to avoid cancel scope conflicts with background tasks\n try:\n await asyncio.wait_for(test_client.__aenter__(), timeout=self.timeout)\n except asyncio.TimeoutError:\n raise TimeoutError(f\"ClientSession enter timed out after {self.timeout}s\")\n\n try:\n try:\n await asyncio.wait_for(test_client.initialize(), timeout=self.timeout)\n except asyncio.TimeoutError:\n raise TimeoutError(f\"initialize() timed out after {self.timeout}s\")\n \n try:\n await asyncio.wait_for(test_client.list_tools(), timeout=self.timeout)\n except asyncio.TimeoutError:\n raise TimeoutError(f\"list_tools() timed out after {self.timeout}s\")\n \n # SUCCESS! Keep the client session (don't close it, closing destroys the streams)\n # Store it directly as the client_session for later use\n self.transport_type = \"SSE\"\n self._connection_manager = connection_manager\n self._connection = connection_manager.get_streams()\n self.client_session = test_client # Reuse the working session\n logger.debug(\"SSE transport selected\")\n return\n except TimeoutError:\n try:\n await asyncio.wait_for(test_client.__aexit__(None, None, None), timeout=2)\n except (asyncio.TimeoutError, Exception):\n pass\n raise\n except Exception as init_error:\n # Clean up the test client only on error\n try:\n await asyncio.wait_for(test_client.__aexit__(None, None, None), timeout=2)\n except (asyncio.TimeoutError, Exception):\n pass\n raise init_error\n\n except Exception as e:\n sse_error = e\n logger.debug(f\"SSE failed: {e}\")\n\n # Clean up the failed connection manager\n if connection_manager:\n try:\n await asyncio.wait_for(connection_manager.stop(), timeout=2)\n except (asyncio.TimeoutError, Exception):\n pass\n\n # Both MCP transports failed, try simple JSON-RPC HTTP as last resort\n # This is useful for custom MCP servers that don't implement proper MCP transports\n logger.debug(f\"Attempting JSON-RPC HTTP fallback to: {self.base_url}\")\n try:\n # Test JSON-RPC connection\n await self._try_jsonrpc_connection()\n \n self.transport_type = \"JSON-RPC HTTP\"\n self._use_jsonrpc = True\n logger.info(f\"JSON-RPC HTTP transport selected for: {self.base_url}\")\n return\n \n except Exception as jsonrpc_error:\n # All transports failed\n logger.error(\n f\"All transport methods failed for {self.base_url}. \"\n f\"Streamable HTTP: {streamable_error}, SSE: {sse_error}, JSON-RPC: {jsonrpc_error}\"\n )\n # Raise the most relevant error - prefer the original streamable error\n raise streamable_error or sse_error or jsonrpc_error\n\n async def _try_jsonrpc_connection(self) -> None:\n \"\"\"Test JSON-RPC HTTP connection by sending an initialize request.\"\"\"\n headers = {**self.headers, \"Content-Type\": \"application/json\"}\n \n async with httpx.AsyncClient(timeout=httpx.Timeout(self.timeout), headers=headers) as client:\n payload = {\n \"jsonrpc\": \"2.0\",\n \"id\": 1,\n \"method\": \"initialize\",\n \"params\": {\n \"protocolVersion\": \"2024-11-05\",\n \"capabilities\": {},\n \"clientInfo\": {\"name\": \"AnyTool\", \"version\": \"1.0.0\"},\n }\n }\n \n response = await client.post(self.base_url, json=payload)\n response.raise_for_status()\n \n data = response.json()\n \n # Check for JSON-RPC error\n if \"error\" in data:\n error = data[\"error\"]\n raise RuntimeError(f\"JSON-RPC error: {error.get('message', str(error))}\")\n \n # Success - server supports JSON-RPC\n logger.debug(f\"JSON-RPC test succeeded: {data.get('result', {})}\")\n\n async def _after_connect(self) -> None:\n \"\"\"Create ClientSession (or set up JSON-RPC client) and log success.\"\"\"\n if self._use_jsonrpc:\n # Set up JSON-RPC HTTP client\n headers = {**self.headers, \"Content-Type\": \"application/json\"}\n self._jsonrpc_client = httpx.AsyncClient(\n timeout=httpx.Timeout(self.timeout),\n headers=headers,\n )\n logger.debug(f\"JSON-RPC HTTP client set up for: {self.base_url}\")\n else:\n # Skip creating ClientSession if _before_connect() already created one\n if self.client_session is None:\n await super()._after_connect()\n else:\n logger.debug(\"Reusing ClientSession from _before_connect()\")\n \n logger.debug(f\"Successfully connected to MCP implementation via {self.transport_type}: {self.base_url}\")\n\n async def _before_disconnect(self) -> None:\n \"\"\"Clean up resources before disconnection.\"\"\"\n # Clean up JSON-RPC client if used\n if self._jsonrpc_client:\n try:\n await self._jsonrpc_client.aclose()\n except Exception as e:\n logger.warning(f\"Error closing JSON-RPC client: {e}\")\n finally:\n self._jsonrpc_client = None\n \n # Call parent cleanup for MCP resources\n await super()._before_disconnect()\n\n @property\n def public_identifier(self) -> str:\n \"\"\"Get the identifier for the connector.\"\"\"\n return {\"type\": self.transport_type, \"base_url\": self.base_url}\n\n # =====================\n # JSON-RPC HTTP Methods\n # =====================\n\n def _next_jsonrpc_id(self) -> int:\n \"\"\"Get next JSON-RPC request ID.\"\"\"\n self._jsonrpc_request_id += 1\n return self._jsonrpc_request_id\n\n async def _send_jsonrpc_request(\n self, \n method: str, \n params: Dict[str, Any] = None,\n max_retries: int = 3,\n retry_delay: float = 1.0,\n ) -> Any:\n \"\"\"Send a JSON-RPC request and return the result.\n \n Args:\n method: The JSON-RPC method name (e.g., \"tools/list\", \"tools/call\")\n params: The method parameters\n max_retries: Maximum number of retries for transient errors (400, 503, etc.)\n retry_delay: Initial delay between retries (doubles each retry)\n \n Returns:\n The result field from the JSON-RPC response\n \"\"\"\n if not self._jsonrpc_client:\n raise RuntimeError(\"JSON-RPC client not initialized\")\n\n last_error = None\n \n for attempt in range(max_retries):\n request_id = self._next_jsonrpc_id()\n payload = {\n \"jsonrpc\": \"2.0\",\n \"id\": request_id,\n \"method\": method,\n \"params\": params or {},\n }\n\n logger.debug(f\"Sending JSON-RPC request: {method} (id={request_id}, attempt {attempt + 1}/{max_retries})\")\n \n try:\n response = await self._jsonrpc_client.post(self.base_url, json=payload)\n response.raise_for_status()\n \n data = response.json()\n \n if \"error\" in data:\n error = data[\"error\"]\n error_msg = error.get(\"message\", str(error))\n raise RuntimeError(f\"JSON-RPC error: {error_msg}\")\n \n return data.get(\"result\", {})\n \n except httpx.HTTPStatusError as e:\n last_error = e\n status_code = e.response.status_code\n \n # Retry on 400 (Bad Request) and 5xx errors\n # 400 can happen when MCP server is temporarily not ready\n if status_code in (400, 500, 502, 503, 504) and attempt < max_retries - 1:\n delay = retry_delay * (2 ** attempt)\n logger.warning(\n f\"HTTP {status_code} error on {method}, retrying in {delay:.1f}s \"\n f\"(attempt {attempt + 1}/{max_retries})\"\n )\n await asyncio.sleep(delay)\n continue\n \n raise RuntimeError(f\"HTTP error: {status_code}\") from e\n \n except httpx.RequestError as e:\n last_error = e\n # Retry on connection errors\n if attempt < max_retries - 1:\n delay = retry_delay * (2 ** attempt)\n logger.warning(\n f\"Request error on {method}: {e}, retrying in {delay:.1f}s \"\n f\"(attempt {attempt + 1}/{max_retries})\"\n )\n await asyncio.sleep(delay)\n continue\n \n raise RuntimeError(f\"Request error: {e}\") from e\n \n # Should not reach here, but just in case\n raise RuntimeError(f\"Max retries exceeded for {method}\") from last_error\n\n def _parse_tools_from_json(self, tools_data: List[Dict]) -> List[Tool]:\n \"\"\"Parse tool data into Tool objects.\"\"\"\n tools = []\n for tool_dict in tools_data:\n try:\n tool = Tool(\n name=tool_dict.get(\"name\", \"\"),\n description=tool_dict.get(\"description\", \"\"),\n inputSchema=tool_dict.get(\"inputSchema\", {}),\n )\n tools.append(tool)\n except Exception as e:\n logger.warning(f\"Failed to parse tool: {e}\")\n return tools\n\n def _parse_resources_from_json(self, resources_data: List[Dict]) -> List[Resource]:\n \"\"\"Parse resource data into Resource objects.\"\"\"\n resources = []\n for res_dict in resources_data:\n try:\n resource = Resource(\n uri=res_dict.get(\"uri\", \"\"),\n name=res_dict.get(\"name\", \"\"),\n description=res_dict.get(\"description\"),\n mimeType=res_dict.get(\"mimeType\"),\n )\n resources.append(resource)\n except Exception as e:\n logger.warning(f\"Failed to parse resource: {e}\")\n return resources\n\n def _parse_prompts_from_json(self, prompts_data: List[Dict]) -> List[Prompt]:\n \"\"\"Parse prompt data into Prompt objects.\"\"\"\n prompts = []\n for prompt_dict in prompts_data:\n try:\n prompt = Prompt(\n name=prompt_dict.get(\"name\", \"\"),\n description=prompt_dict.get(\"description\"),\n arguments=prompt_dict.get(\"arguments\"),\n )\n prompts.append(prompt)\n except Exception as e:\n logger.warning(f\"Failed to parse prompt: {e}\")\n return prompts\n\n # =====================\n # Override MCP Methods for JSON-RPC Support\n # =====================\n\n async def initialize(self) -> Dict[str, Any]:\n \"\"\"Initialize the MCP session.\"\"\"\n if not self._use_jsonrpc:\n return await super().initialize()\n \n # JSON-RPC mode\n logger.debug(\"Initializing JSON-RPC HTTP MCP session\")\n \n result = await self._send_jsonrpc_request(\"initialize\", {\n \"protocolVersion\": \"2024-11-05\",\n \"capabilities\": {},\n \"clientInfo\": {\"name\": \"AnyTool\", \"version\": \"1.0.0\"},\n })\n \n capabilities = result.get(\"capabilities\", {})\n \n # List tools\n if capabilities.get(\"tools\"):\n try:\n tools_result = await self._send_jsonrpc_request(\"tools/list\", {})\n self._tools = self._parse_tools_from_json(tools_result.get(\"tools\", []))\n except Exception:\n self._tools = []\n else:\n # Try anyway - some servers don't advertise capabilities correctly\n try:\n tools_result = await self._send_jsonrpc_request(\"tools/list\", {})\n self._tools = self._parse_tools_from_json(tools_result.get(\"tools\", []))\n except Exception:\n self._tools = []\n \n # List resources\n if capabilities.get(\"resources\"):\n try:\n resources_result = await self._send_jsonrpc_request(\"resources/list\", {})\n self._resources = self._parse_resources_from_json(resources_result.get(\"resources\", []))\n except Exception:\n self._resources = []\n else:\n self._resources = []\n \n # List prompts\n if capabilities.get(\"prompts\"):\n try:\n prompts_result = await self._send_jsonrpc_request(\"prompts/list\", {})\n self._prompts = self._parse_prompts_from_json(prompts_result.get(\"prompts\", []))\n except Exception:\n self._prompts = []\n else:\n self._prompts = []\n \n logger.info(\n f\"JSON-RPC HTTP MCP session initialized with {len(self._tools)} tools, \"\n f\"{len(self._resources)} resources, {len(self._prompts)} prompts\"\n )\n \n return result\n\n @property\n def is_connected(self) -> bool:\n \"\"\"Check if the connector is connected.\"\"\"\n if self._use_jsonrpc:\n return self._connected and self._jsonrpc_client is not None\n return super().is_connected\n\n async def _ensure_connected(self) -> None:\n \"\"\"Ensure the connector is connected.\"\"\"\n if self._use_jsonrpc:\n if not self._connected or not self._jsonrpc_client:\n raise RuntimeError(\"JSON-RPC HTTP connector is not connected\")\n else:\n await super()._ensure_connected()\n\n async def list_tools(self) -> List[Tool]:\n \"\"\"List all available tools.\"\"\"\n if not self._use_jsonrpc:\n return await super().list_tools()\n \n await self._ensure_connected()\n try:\n tools_result = await self._send_jsonrpc_request(\"tools/list\", {})\n self._tools = self._parse_tools_from_json(tools_result.get(\"tools\", []))\n return self._tools\n except Exception as e:\n logger.error(f\"Error listing tools: {e}\")\n return []\n\n async def call_tool(self, name: str, arguments: Dict[str, Any]) -> CallToolResult:\n \"\"\"Call an MCP tool.\"\"\"\n if not self._use_jsonrpc:\n return await super().call_tool(name, arguments)\n \n await self._ensure_connected()\n logger.debug(f\"Calling tool '{name}' with arguments: {arguments}\")\n \n result = await self._send_jsonrpc_request(\"tools/call\", {\n \"name\": name,\n \"arguments\": arguments,\n })\n \n # Parse the result into CallToolResult\n content = []\n for item in result.get(\"content\", []):\n item_type = item.get(\"type\", \"text\")\n if item_type == \"text\":\n content.append(TextContent(type=\"text\", text=item.get(\"text\", \"\")))\n elif item_type == \"image\":\n content.append(ImageContent(\n type=\"image\",\n data=item.get(\"data\", \"\"),\n mimeType=item.get(\"mimeType\", \"image/png\"),\n ))\n elif item_type == \"resource\":\n content.append(EmbeddedResource(\n type=\"resource\",\n resource=item.get(\"resource\", {}),\n ))\n \n if not content and result:\n content.append(TextContent(type=\"text\", text=str(result)))\n \n return CallToolResult(\n content=content,\n isError=result.get(\"isError\", False),\n )\n\n async def list_resources(self) -> List[Resource]:\n \"\"\"List all available resources.\"\"\"\n if not self._use_jsonrpc:\n return await super().list_resources()\n \n await self._ensure_connected()\n try:\n resources_result = await self._send_jsonrpc_request(\"resources/list\", {})\n self._resources = self._parse_resources_from_json(resources_result.get(\"resources\", []))\n return self._resources\n except Exception as e:\n logger.error(f\"Error listing resources: {e}\")\n return []\n\n async def read_resource(self, uri: str) -> ReadResourceResult:\n \"\"\"Read a resource by URI.\"\"\"\n if not self._use_jsonrpc:\n return await super().read_resource(uri)\n \n await self._ensure_connected()\n result = await self._send_jsonrpc_request(\"resources/read\", {\"uri\": uri})\n return ReadResourceResult(**result)\n\n async def list_prompts(self) -> List[Prompt]:\n \"\"\"List all available prompts.\"\"\"\n if not self._use_jsonrpc:\n return await super().list_prompts()\n \n await self._ensure_connected()\n try:\n prompts_result = await self._send_jsonrpc_request(\"prompts/list\", {})\n self._prompts = self._parse_prompts_from_json(prompts_result.get(\"prompts\", []))\n return self._prompts\n except Exception as e:\n logger.error(f\"Error listing prompts: {e}\")\n return []\n\n async def get_prompt(self, name: str, arguments: Dict[str, Any] | None = None) -> GetPromptResult:\n \"\"\"Get a prompt by name.\"\"\"\n if not self._use_jsonrpc:\n return await super().get_prompt(name, arguments)\n \n await self._ensure_connected()\n result = await self._send_jsonrpc_request(\"prompts/get\", {\n \"name\": name,\n \"arguments\": arguments or {},\n })\n return GetPromptResult(**result)\n\n async def request(self, method: str, params: Dict[str, Any] | None = None) -> Any:\n \"\"\"Send a raw request to the MCP implementation.\"\"\"\n if not self._use_jsonrpc:\n return await super().request(method, params)\n \n await self._ensure_connected()\n return await self._send_jsonrpc_request(method, params or {})\n\n async def invoke(self, name: str, params: Dict[str, Any]) -> Any:\n \"\"\"Invoke a tool or special method.\"\"\"\n if not self._use_jsonrpc:\n return await super().invoke(name, params)\n \n await self._ensure_connected()\n\n if not name.startswith(\"__\"):\n return await self.call_tool(name, params)\n\n if name == \"__read_resource__\":\n return await self.read_resource(params[\"uri\"])\n if name == \"__list_prompts__\":\n return await self.list_prompts()\n if name == \"__get_prompt__\":\n return await self.get_prompt(params[\"name\"], params.get(\"args\"))\n\n raise ValueError(f\"Unsupported MCP invoke name: {name}\")\n"
},
{
"path": "anytool/grounding/backends/mcp/transport/connectors/sandbox.py",
"content": "\"\"\"\nSandbox connector for MCP implementations.\n\nThis module provides a connector for communicating with MCP implementations\nthat are executed inside a sandbox environment (supports any BaseSandbox implementation).\n\"\"\"\n\nimport asyncio\nimport sys\nimport time\n\nimport aiohttp\nfrom mcp import ClientSession\n\nfrom anytool.utils.logging import Logger\nfrom anytool.grounding.backends.mcp.transport.task_managers import SseConnectionManager\nfrom anytool.grounding.core.security import BaseSandbox\nfrom anytool.grounding.backends.mcp.transport.connectors.base import MCPBaseConnector\n\nlogger = Logger.get_logger(__name__)\n\n\nclass SandboxConnector(MCPBaseConnector):\n \"\"\"Connector for MCP implementations running in a sandbox environment.\n\n This connector runs a user-defined stdio command within a sandbox environment\n through a BaseSandbox implementation (e.g., E2BSandbox), potentially wrapped \n by a utility like 'supergateway' to expose its stdio.\n \"\"\"\n\n def __init__(\n self,\n sandbox: BaseSandbox,\n command: str,\n args: list[str],\n env: dict[str, str] | None = None,\n supergateway_command: str = \"npx -y supergateway\",\n port: int = 3000,\n timeout: float = 5,\n sse_read_timeout: float = 60 * 5,\n ):\n \"\"\"Initialize a new sandbox connector.\n\n Args:\n sandbox: A BaseSandbox implementation (e.g., E2BSandbox) to run commands in.\n command: The user's MCP server command to execute in the sandbox.\n args: Command line arguments for the user's MCP server command.\n env: Environment variables for the user's MCP server command.\n supergateway_command: Command to run supergateway (default: \"npx -y supergateway\").\n port: Port number for the sandbox server (default: 3000).\n timeout: Timeout for the sandbox process in seconds.\n sse_read_timeout: Timeout for the SSE connection in seconds.\n \"\"\"\n # Store user command configuration\n self.user_command = command\n self.user_args = args or []\n self.user_env = env or {}\n self.port = port\n \n # Create a placeholder connection manager (will be set up in connect())\n # We need the sandbox to start first to get the base_url, so we can't create\n # the real SseConnectionManager until connect() is called\n from anytool.grounding.core.transport.task_managers import PlaceholderConnectionManager\n connection_manager = PlaceholderConnectionManager()\n super().__init__(connection_manager)\n\n # Sandbox configuration\n self._sandbox = sandbox\n self.supergateway_cmd_parts = supergateway_command\n \n # Runtime state\n self.process = None\n self.client_session: ClientSession | None = None\n self.errlog = sys.stderr\n self.base_url: str | None = None\n self._connected = False\n self._connection_manager: SseConnectionManager | None = None\n\n # SSE connection parameters\n self.headers = {}\n self.timeout = timeout\n self.sse_read_timeout = sse_read_timeout\n\n self.stdout_lines: list[str] = []\n self.stderr_lines: list[str] = []\n self._server_ready = asyncio.Event()\n\n def _handle_stdout(self, data: str) -> None:\n \"\"\"Handle stdout data from the sandbox process.\"\"\"\n self.stdout_lines.append(data)\n logger.debug(f\"[SANDBOX STDOUT] {data}\", end=\"\", flush=True)\n\n def _handle_stderr(self, data: str) -> None:\n \"\"\"Handle stderr data from the sandbox process.\"\"\"\n self.stderr_lines.append(data)\n logger.debug(f\"[SANDBOX STDERR] {data}\", file=self.errlog, end=\"\", flush=True)\n\n async def wait_for_server_response(self, base_url: str, timeout: int = 30) -> bool:\n \"\"\"Wait for the server to respond to HTTP requests.\n \n Args:\n base_url: The base URL to check for server readiness\n timeout: Maximum time to wait in seconds\n \n Returns:\n True if server is responding, raises TimeoutError otherwise\n \"\"\"\n logger.info(f\"Waiting for server at {base_url} to respond...\")\n sys.stdout.flush()\n\n start_time = time.time()\n ping_url = f\"{base_url}/sse\"\n\n # Try to connect to the server\n while time.time() - start_time < timeout:\n try:\n async with aiohttp.ClientSession() as session:\n try:\n # First try the endpoint\n async with session.get(ping_url, timeout=2) as response:\n if response.status == 200:\n elapsed = time.time() - start_time\n logger.info(f\"Server is ready! SSE endpoint responded with 200 after {elapsed:.1f}s\")\n return True\n except Exception:\n # If sse endpoint doesn't work, try the base URL\n async with session.get(base_url, timeout=2) as response:\n if response.status < 500: # Accept any non-server error\n elapsed = time.time() - start_time\n logger.info(\n f\"Server is ready! Base URL responded with {response.status} after {elapsed:.1f}s\"\n )\n return True\n except Exception:\n # Wait a bit before trying again\n await asyncio.sleep(0.5)\n continue\n\n # If we get here, the request failed\n await asyncio.sleep(0.5)\n\n # Log status every 5 seconds\n elapsed = time.time() - start_time\n if int(elapsed) % 5 == 0:\n logger.info(f\"Still waiting for server to respond... ({elapsed:.1f}s elapsed)\")\n sys.stdout.flush()\n\n # If we get here, we timed out\n raise TimeoutError(f\"Timeout waiting for server to respond (waited {timeout} seconds)\")\n\n async def _before_connect(self) -> None:\n \"\"\"Set up the sandbox and prepare the connection manager.\"\"\"\n logger.debug(\"Connecting to MCP implementation in sandbox\")\n\n # Start the sandbox if not already active\n if not self._sandbox.is_active:\n logger.debug(\"Starting sandbox...\")\n await self._sandbox.start()\n\n # Get the host for the sandbox\n # Note: This assumes the sandbox implementation has a get_host method\n # For E2BSandbox, this is available\n host = self._sandbox.get_host(self.port)\n self.base_url = f\"https://{host}\".rstrip(\"/\")\n\n # Append command with args\n command = f\"{self.user_command} {' '.join(self.user_args)}\"\n\n # Construct the full command with supergateway\n full_command = f'{self.supergateway_cmd_parts} \\\n --base-url {self.base_url} \\\n --port {self.port} \\\n --cors \\\n --stdio \"{command}\"'\n\n logger.debug(f\"Full command: {full_command}\")\n\n # Execute the command in the sandbox\n self.process = await self._sandbox.execute_safe(\n full_command,\n envs=self.user_env,\n timeout=1000 * 60 * 10, # 10 minutes timeout\n background=True,\n on_stdout=self._handle_stdout,\n on_stderr=self._handle_stderr,\n )\n\n # Wait for the server to be ready\n await self.wait_for_server_response(self.base_url, timeout=30)\n logger.debug(\"Initializing connection manager...\")\n\n # Create the SSE connection URL\n sse_url = f\"{self.base_url}/sse\"\n\n # Create and set up the connection manager\n self._connection_manager = SseConnectionManager(sse_url, self.headers, self.timeout, self.sse_read_timeout)\n\n async def _after_connect(self) -> None:\n \"\"\"Create ClientSession and log success.\"\"\"\n await super()._after_connect()\n logger.debug(f\"Successfully connected to MCP implementation via HTTP/SSE in sandbox: {self.base_url}\")\n\n async def _before_disconnect(self) -> None:\n \"\"\"Clean up sandbox-specific resources before disconnection.\"\"\"\n logger.debug(\"Cleaning up sandbox resources\")\n\n # Stop the sandbox (which will clean up processes)\n if self._sandbox and self._sandbox.is_active:\n try:\n logger.debug(\"Stopping sandbox instance\")\n await self._sandbox.stop()\n logger.debug(\"Sandbox instance stopped successfully\")\n except Exception as e:\n logger.warning(f\"Error stopping sandbox: {e}\")\n\n self.process = None\n\n # Call the parent method to clean up MCP resources\n await super()._before_disconnect()\n\n # Clear any collected output\n self.stdout_lines = []\n self.stderr_lines = []\n self.base_url = None\n \n async def _cleanup_on_connect_failure(self) -> None:\n \"\"\"Clean up sandbox resources on connection failure.\"\"\"\n # Stop the sandbox if it was started\n if self._sandbox and self._sandbox.is_active:\n try:\n await self._sandbox.stop()\n except Exception as e:\n logger.warning(f\"Error stopping sandbox during cleanup: {e}\")\n \n self.process = None\n self.stdout_lines = []\n self.stderr_lines = []\n self.base_url = None\n \n # Call parent cleanup\n await super()._cleanup_on_connect_failure()\n\n @property\n def sandbox(self) -> BaseSandbox:\n \"\"\"Get the underlying sandbox instance.\"\"\"\n return self._sandbox\n\n @property\n def public_identifier(self) -> str:\n \"\"\"Get the identifier for the connector.\"\"\"\n return {\"type\": \"sandbox\", \"command\": self.user_command, \"args\": self.user_args}\n"
},
{
"path": "anytool/grounding/backends/mcp/transport/connectors/stdio.py",
"content": "\"\"\"\nStdIO connector for MCP implementations.\n\nThis module provides a connector for communicating with MCP implementations\nthrough the standard input/output streams.\n\"\"\"\n\nimport sys\n\nfrom mcp import ClientSession, StdioServerParameters\n\nfrom anytool.utils.logging import Logger\nfrom ..task_managers import StdioConnectionManager\nfrom .base import MCPBaseConnector\n\nlogger = Logger.get_logger(__name__)\n\n\nclass StdioConnector(MCPBaseConnector):\n \"\"\"Connector for MCP implementations using stdio transport.\n\n This connector uses the stdio transport to communicate with MCP implementations\n that are executed as child processes. It uses a connection manager to handle\n the proper lifecycle management of the stdio client.\n \"\"\"\n\n def __init__(\n self,\n command: str = \"npx\",\n args: list[str] | None = None,\n env: dict[str, str] | None = None,\n errlog=None,\n ):\n \"\"\"Initialize a new stdio connector.\n\n Args:\n command: The command to execute.\n args: Optional command line arguments.\n env: Optional environment variables.\n errlog: Stream to write error output to (defaults to filtered stderr).\n StdioConnectionManager will wrap this to filter harmless errors.\n \"\"\"\n self.command = command\n self.args = args or [] # Ensure args is never None\n \n # Ensure env is not None and add settings to suppress non-JSON output from servers\n self.env = env or {}\n # Add environment variables to encourage MCP servers to suppress non-JSON output\n # Many Node.js-based servers respect NODE_ENV=production\n if \"NODE_ENV\" not in self.env:\n self.env[\"NODE_ENV\"] = \"production\"\n # Add flag to suppress informational messages (some servers respect this)\n if \"MCP_SILENT\" not in self.env:\n self.env[\"MCP_SILENT\"] = \"true\"\n \n self.errlog = errlog\n \n # Create server parameters and connection manager\n # StdioConnectionManager will wrap errlog in FilteredStderrWrapper\n server_params = StdioServerParameters(command=self.command, args=self.args, env=self.env)\n connection_manager = StdioConnectionManager(server_params, self.errlog)\n super().__init__(connection_manager)\n\n async def _before_connect(self) -> None:\n \"\"\"Log connection attempt.\"\"\"\n logger.debug(f\"Connecting to MCP implementation: {self.command}\")\n\n async def _after_connect(self) -> None:\n \"\"\"Create ClientSession and log success.\"\"\"\n # Call parent's _after_connect to create the ClientSession\n await super()._after_connect()\n logger.debug(f\"Successfully connected to MCP implementation: {self.command}\")\n\n @property\n def public_identifier(self) -> dict[str, str]:\n return {\"type\": \"stdio\", \"command&args\": f\"{self.command} {' '.join(self.args)}\"}"
},
{
"path": "anytool/grounding/backends/mcp/transport/connectors/utils.py",
"content": "from typing import Any\n\n\ndef is_stdio_server(server_config: dict[str, Any]) -> bool:\n \"\"\"Check if the server configuration is for a stdio server.\n\n Args:\n server_config: The server configuration section\n\n Returns:\n True if the server is a stdio server, False otherwise\n \"\"\"\n return \"command\" in server_config and \"args\" in server_config"
},
{
"path": "anytool/grounding/backends/mcp/transport/connectors/websocket.py",
"content": "\"\"\"\nWebSocket connector for MCP implementations.\n\nThis module provides a connector for communicating with MCP implementations\nthrough WebSocket connections.\n\"\"\"\n\nimport asyncio\nimport json\nimport uuid\nfrom typing import Any\n\nfrom mcp.types import Tool\nfrom websockets import ClientConnection\n\nfrom anytool.utils.logging import Logger\nfrom anytool.grounding.core.transport.task_managers.base import BaseConnectionManager\nfrom ..task_managers import WebSocketConnectionManager\nfrom .base import MCPBaseConnector\n\nlogger = Logger.get_logger(__name__)\n\n\nclass WebSocketConnector(MCPBaseConnector):\n \"\"\"Connector for MCP implementations using WebSocket transport.\n\n This connector uses WebSockets to communicate with remote MCP implementations,\n using a connection manager to handle the proper lifecycle management.\n \"\"\"\n\n def __init__(\n self,\n url: str,\n auth_token: str | None = None,\n headers: dict[str, str] | None = None,\n ):\n \"\"\"Initialize a new WebSocket connector.\n\n Args:\n url: The WebSocket URL to connect to.\n auth_token: Optional authentication token.\n headers: Optional additional headers.\n \"\"\"\n self.url = url\n self.auth_token = auth_token\n self.headers = headers or {}\n if auth_token:\n self.headers[\"Authorization\"] = f\"Bearer {auth_token}\"\n\n self.ws: ClientConnection | None = None\n self._receiver_task: asyncio.Task | None = None\n self.pending_requests: dict[str, asyncio.Future] = {}\n self._tools: list[Tool] | None = None\n \n # Create connection manager with actual parameters\n connection_manager = WebSocketConnectionManager(self.url, self.headers)\n super().__init__(connection_manager)\n self._connected = False\n\n async def _get_streams_from_connection(self):\n \"\"\"WebSocket doesn't use streams, return None to skip ClientSession creation.\"\"\"\n return None\n \n async def _after_connect(self) -> None:\n \"\"\"Set up WebSocket-specific resources after connection.\n \n WebSocket doesn't use ClientSession, so we skip the parent's implementation\n and set up WebSocket-specific resources instead.\n \"\"\"\n # Store the WebSocket connection\n self.ws = self._connection\n \n # Start the message receiver task\n self._receiver_task = asyncio.create_task(self._receive_messages(), name=\"websocket_receiver_task\")\n \n logger.debug(f\"Successfully connected to MCP implementation via WebSocket: {self.url}\")\n\n async def _receive_messages(self) -> None:\n \"\"\"Continuously receive and process messages from the WebSocket.\"\"\"\n if not self.ws:\n raise RuntimeError(\"WebSocket is not connected\")\n\n try:\n async for message in self.ws:\n # Parse the message\n data = json.loads(message)\n\n # Check if this is a response to a pending request\n request_id = data.get(\"id\")\n if request_id and request_id in self.pending_requests:\n future = self.pending_requests.pop(request_id)\n if \"result\" in data:\n future.set_result(data[\"result\"])\n elif \"error\" in data:\n future.set_exception(Exception(data[\"error\"]))\n\n logger.debug(f\"Received response for request {request_id}\")\n else:\n logger.debug(f\"Received message: {data}\")\n except Exception as e:\n logger.error(f\"Error in WebSocket message receiver: {e}\")\n # If the websocket connection was closed or errored,\n # reject all pending requests\n for future in self.pending_requests.values():\n if not future.done():\n future.set_exception(e)\n\n async def _before_disconnect(self) -> None:\n \"\"\"Clean up WebSocket-specific resources before disconnection.\"\"\"\n errors = []\n\n # First cancel the receiver task\n if self._receiver_task and not self._receiver_task.done():\n try:\n logger.debug(\"Cancelling WebSocket receiver task\")\n self._receiver_task.cancel()\n try:\n await self._receiver_task\n except asyncio.CancelledError:\n logger.debug(\"WebSocket receiver task cancelled successfully\")\n except Exception as e:\n logger.warning(f\"Error during WebSocket receiver task cancellation: {e}\")\n except Exception as e:\n error_msg = f\"Error cancelling WebSocket receiver task: {e}\"\n logger.warning(error_msg)\n errors.append(error_msg)\n finally:\n self._receiver_task = None\n\n # Reject any pending requests\n if self.pending_requests:\n logger.debug(f\"Rejecting {len(self.pending_requests)} pending requests\")\n for future in self.pending_requests.values():\n if not future.done():\n future.set_exception(ConnectionError(\"WebSocket disconnected\"))\n self.pending_requests.clear()\n\n # Reset WebSocket and tools\n self.ws = None\n self._tools = None\n\n if errors:\n logger.warning(f\"Encountered {len(errors)} errors during WebSocket resource cleanup\")\n \n async def _cleanup_on_connect_failure(self) -> None:\n \"\"\"Clean up WebSocket resources on connection failure.\"\"\"\n # Cancel receiver task if it was started\n if self._receiver_task and not self._receiver_task.done():\n try:\n self._receiver_task.cancel()\n await self._receiver_task\n except asyncio.CancelledError:\n pass\n except Exception:\n pass\n finally:\n self._receiver_task = None\n \n # Reject pending requests\n for future in self.pending_requests.values():\n if not future.done():\n future.set_exception(ConnectionError(\"Connection failed\"))\n self.pending_requests.clear()\n \n # Call parent cleanup\n await super()._cleanup_on_connect_failure()\n self.ws = None\n\n async def _send_request(self, method: str, params: dict[str, Any] | None = None) -> Any:\n \"\"\"Send a request and wait for a response.\"\"\"\n if not self.ws:\n raise RuntimeError(\"WebSocket is not connected\")\n\n # Create a request ID\n request_id = str(uuid.uuid4())\n\n # Create a future to receive the response\n future = asyncio.Future()\n self.pending_requests[request_id] = future\n\n # Send the request\n await self.ws.send(json.dumps({\"id\": request_id, \"method\": method, \"params\": params or {}}))\n\n logger.debug(f\"Sent request {request_id} method: {method}\")\n\n # Wait for the response\n try:\n return await future\n except Exception as e:\n # Remove the request from pending requests\n self.pending_requests.pop(request_id, None)\n logger.error(f\"Error waiting for response to request {request_id}: {e}\")\n raise\n\n async def initialize(self) -> dict[str, Any]:\n \"\"\"Initialize the MCP session and return session information.\"\"\"\n logger.debug(\"Initializing MCP session\")\n result = await self._send_request(\"initialize\")\n\n # Get available tools\n tools_result = await self.list_tools()\n self._tools = [Tool(**tool) for tool in tools_result]\n\n logger.debug(f\"MCP session initialized with {len(self._tools)} tools\")\n return result\n\n async def list_tools(self) -> list[dict[str, Any]]:\n \"\"\"List all available tools from the MCP implementation.\"\"\"\n logger.debug(\"Listing tools\")\n result = await self._send_request(\"tools/list\")\n return result.get(\"tools\", [])\n\n @property\n def tools(self) -> list[Tool]:\n \"\"\"Get the list of available tools.\"\"\"\n if not self._tools:\n raise RuntimeError(\"MCP client is not initialized\")\n return self._tools\n\n async def call_tool(self, name: str, arguments: dict[str, Any]) -> Any:\n \"\"\"Call an MCP tool with the given arguments.\"\"\"\n logger.debug(f\"Calling tool '{name}' with arguments: {arguments}\")\n return await self._send_request(\"tools/call\", {\"name\": name, \"arguments\": arguments})\n\n async def list_resources(self) -> list[dict[str, Any]]:\n \"\"\"List all available resources from the MCP implementation.\"\"\"\n logger.debug(\"Listing resources\")\n result = await self._send_request(\"resources/list\")\n return result\n\n async def read_resource(self, uri: str) -> tuple[bytes, str]:\n \"\"\"Read a resource by URI.\"\"\"\n logger.debug(f\"Reading resource: {uri}\")\n result = await self._send_request(\"resources/read\", {\"uri\": uri})\n return result.get(\"content\", b\"\"), result.get(\"mimeType\", \"\")\n\n async def request(self, method: str, params: dict[str, Any] | None = None) -> Any:\n \"\"\"Send a raw request to the MCP implementation.\"\"\"\n logger.debug(f\"Sending request: {method} with params: {params}\")\n return await self._send_request(method, params)\n\n @property\n def public_identifier(self) -> str:\n \"\"\"Get the identifier for the connector.\"\"\"\n return {\"type\": \"websocket\", \"url\": self.url}\n"
},
{
"path": "anytool/grounding/backends/mcp/transport/task_managers/__init__.py",
"content": "\"\"\"\nConnectors for various MCP transports.\n\nThis module provides interfaces for connecting to MCP implementations\nthrough different transport mechanisms.\n\"\"\"\n\nfrom .sse import SseConnectionManager\nfrom .stdio import StdioConnectionManager\nfrom .streamable_http import StreamableHttpConnectionManager\nfrom .websocket import WebSocketConnectionManager\n\n__all__ = [\n \"StdioConnectionManager\",\n \"WebSocketConnectionManager\",\n \"SseConnectionManager\",\n \"StreamableHttpConnectionManager\",\n]"
},
{
"path": "anytool/grounding/backends/mcp/transport/task_managers/sse.py",
"content": "\"\"\"\nSSE connection management for MCP implementations.\n\nThis module provides a connection manager for SSE-based MCP connections\nthat ensures proper task isolation and resource cleanup.\n\"\"\"\n\nfrom typing import Any, Tuple\nfrom mcp.client.sse import sse_client\nfrom anytool.utils.logging import Logger\nfrom anytool.grounding.core.transport.task_managers import (\n AsyncContextConnectionManager,\n)\n\nlogger = Logger.get_logger(__name__)\n\n\nclass SseConnectionManager(AsyncContextConnectionManager[Tuple[Any, Any], ...]):\n \"\"\"Connection manager for SSE-based MCP connections.\n\n This class handles the proper task isolation for sse_client context managers\n to prevent the \"cancel scope in different task\" error. It runs the sse_client\n in a dedicated task and manages its lifecycle.\n \"\"\"\n\n def __init__(\n self,\n url: str,\n headers: dict[str, str] | None = None,\n timeout: float = 5,\n sse_read_timeout: float = 60 * 5,\n ):\n \"\"\"Initialize a new SSE connection manager.\n\n Args:\n url: The SSE endpoint URL\n headers: Optional HTTP headers\n timeout: Timeout for HTTP operations in seconds\n sse_read_timeout: Timeout for SSE read operations in seconds\n \"\"\"\n super().__init__(\n sse_client,\n url=url,\n headers=headers or {},\n timeout=timeout,\n sse_read_timeout=sse_read_timeout,\n )\n self.url = url\n self.headers = headers or {}\n logger.debug(\"SseConnectionManager init url=%s\", url)\n"
},
{
"path": "anytool/grounding/backends/mcp/transport/task_managers/stdio.py",
"content": "\"\"\"\nStdIO connection management for MCP implementations.\n\nThis module provides a connection manager for stdio-based MCP connections\nthat ensures proper task isolation and resource cleanup.\n\"\"\"\n\nimport asyncio\nimport io\nimport logging\nimport sys\nfrom typing import Any, TextIO, Tuple\n\nfrom mcp import StdioServerParameters\nfrom mcp.client.stdio import stdio_client\n\nfrom anytool.utils.logging import Logger\nfrom anytool.grounding.core.transport.task_managers import (\n AsyncContextConnectionManager,\n)\n\nlogger = Logger.get_logger(__name__)\n\n\nclass FilteredStderrWrapper(io.TextIOBase):\n \"\"\"Wrapper for stderr that filters out harmless MCP server shutdown messages.\n \n This wrapper suppresses error messages from MCP servers during shutdown\n that are harmless but create noise in the logs.\n \"\"\"\n \n def __init__(self, wrapped_stream: TextIO):\n \"\"\"Initialize the wrapper.\n \n Args:\n wrapped_stream: The underlying stderr stream\n \"\"\"\n self._stream = wrapped_stream\n self._buffer = \"\"\n self._in_traceback = False\n self._traceback_lines = []\n self._in_rich_traceback = False # Track rich-formatted tracebacks\n self._rich_traceback_needs_error_line = False # After โฐ, need one more line\n \n def write(self, s: str) -> int:\n \"\"\"Write to stderr, filtering out harmless error messages.\n \n Args:\n s: The string to write\n \n Returns:\n Number of characters written\n \"\"\"\n # Buffer the input for line-by-line processing\n self._buffer += s\n \n # Process complete lines\n while '\\n' in self._buffer:\n line, self._buffer = self._buffer.split('\\n', 1)\n self._process_line(line + '\\n')\n \n return len(s)\n \n def _process_line(self, line: str):\n \"\"\"Process a single line and decide whether to output it.\"\"\"\n # Detect start of traceback or exception group\n if line.lstrip().startswith((\"โญ\", \"โ\")):\n self._in_traceback = True\n self._in_rich_traceback = True\n self._rich_traceback_needs_error_line = False \n self._traceback_lines = [line]\n return\n\n if (line.strip().startswith('Traceback (most recent call last)') or\n line.strip().startswith('Exception Group Traceback (most recent call last)') or\n line.strip().startswith('BaseExceptionGroup:') or\n line.strip().startswith('ExceptionGroup:')):\n self._in_traceback = True\n self._traceback_lines = [line]\n self._in_rich_traceback = False\n self._rich_traceback_needs_error_line = False\n return\n \n # Collect traceback lines\n if self._in_traceback:\n self._traceback_lines.append(line)\n\n # If not in rich traceback mode, but current line contains rich border characters, switch to rich mode\n if not self._in_rich_traceback and any(ch in line for ch in (\"โญ\", \"โ\")):\n self._in_rich_traceback = True\n \n # Check for end of rich-formatted traceback (line with โฐ)\n if self._in_rich_traceback and 'โฐ' in line:\n # Rich traceback box ended, but we need to collect the error line that follows\n self._rich_traceback_needs_error_line = True\n return\n \n # If we just ended a rich traceback, this should be the error line\n if self._rich_traceback_needs_error_line:\n # Now we have the complete rich traceback including the error line\n if self._is_harmless_error():\n logger.debug(f\"Suppressed harmless rich-formatted MCP server error\")\n else:\n # Output the full traceback\n for tb_line in self._traceback_lines:\n self._stream.write(tb_line)\n self._stream.flush()\n \n # Reset traceback collection\n self._in_traceback = False\n self._in_rich_traceback = False\n self._rich_traceback_needs_error_line = False\n self._traceback_lines = []\n return\n \n # For exception groups, we need to collect more lines\n # Check if we've collected enough to determine if it's harmless\n if len(self._traceback_lines) > 5 and not self._in_rich_traceback:\n # Check periodically if this is a harmless error\n if self._is_harmless_error():\n # Suppress this traceback\n logger.debug(f\"Suppressed harmless MCP server shutdown error\")\n self._in_traceback = False\n self._in_rich_traceback = False\n self._rich_traceback_needs_error_line = False\n self._traceback_lines = []\n return\n \n # Check if this is the error line (last line of regular traceback)\n # But not for rich tracebacks which use box characters\n # A final traceback line is typically unindented and contains \"ErrorType: message\"\n if not self._in_rich_traceback and line and not line[0].isspace() and ':' in line:\n # Check if this is a harmless cleanup error\n if self._is_harmless_error():\n # Suppress this traceback\n logger.debug(f\"Suppressed harmless MCP server shutdown error\")\n else:\n # Output the full traceback\n for tb_line in self._traceback_lines:\n self._stream.write(tb_line)\n self._stream.flush()\n \n # Reset traceback collection\n self._in_traceback = False\n self._in_rich_traceback = False\n self._rich_traceback_needs_error_line = False\n self._traceback_lines = []\n return\n \n # If we've collected too many lines without finding the end, output and reset\n if len(self._traceback_lines) > 100:\n # Output what we have\n for tb_line in self._traceback_lines:\n self._stream.write(tb_line)\n self._stream.flush()\n self._in_traceback = False\n self._in_rich_traceback = False\n self._rich_traceback_needs_error_line = False\n self._traceback_lines = []\n return\n else:\n # Normal line - check if it's a harmless error log\n line_lower = line.lower()\n harmless_log_patterns = [\n 'an error occurred during closing of asynchronous generator',\n 'asyncgen:',\n 'service stopped.',\n ]\n \n # Check if this is a harmless log line\n is_harmless_log = any(pattern in line_lower for pattern in harmless_log_patterns)\n \n if not is_harmless_log:\n # Output normal lines\n self._stream.write(line)\n self._stream.flush()\n else:\n # Suppress harmless log messages\n logger.debug(f\"Suppressed harmless log line: {line.strip()}\")\n \n def _is_harmless_error(self) -> bool:\n \"\"\"Check if the collected traceback is a harmless error.\"\"\"\n traceback_text = ''.join(self._traceback_lines).lower()\n \n # List of harmless error patterns (case-insensitive)\n harmless_patterns = [\n 'valueerror: i/o operation on closed file',\n 'oserror: [errno 9] bad file descriptor',\n 'brokenpipeerror',\n 'runtimeerror: attempted to exit cancel scope in a different task',\n 'baseexceptiongroup: unhandled errors in a taskgroup',\n 'generatorexit',\n 'an error occurred during closing of asynchronous generator',\n ]\n \n # Check if any pattern matches and it's related to shutdown\n for pattern in harmless_patterns:\n if pattern in traceback_text:\n # Also check if it's related to shutdown/cleanup\n shutdown_keywords = ['finally:', 'stopped', 'cleanup', '__exit__', '__aexit__', 'stdio_client', 'service stopped']\n if any(keyword in traceback_text for keyword in shutdown_keywords):\n return True\n \n return False\n \n def flush(self):\n \"\"\"Flush any remaining buffered content and the underlying stream.\"\"\"\n if self._buffer:\n self._process_line(self._buffer)\n self._buffer = \"\"\n \n if self._traceback_lines:\n # Flush incomplete traceback\n for line in self._traceback_lines:\n self._stream.write(line)\n self._traceback_lines = []\n \n self._stream.flush()\n \n def fileno(self) -> int:\n \"\"\"Return the file descriptor of the underlying stream.\"\"\"\n if hasattr(self._stream, 'fileno'):\n return self._stream.fileno()\n return -1\n \n @property\n def closed(self) -> bool:\n \"\"\"Check if the stream is closed.\"\"\"\n return self._stream.closed\n\n\nclass StdioConnectionManager(AsyncContextConnectionManager[Tuple[Any, Any], ...]):\n \"\"\"Connection manager for stdio-based MCP connections.\n\n This class handles the proper task isolation for stdio_client context managers\n to prevent the \"cancel scope in different task\" error. It runs the stdio_client\n in a dedicated task and manages its lifecycle.\n \n Note: Error handling during cleanup (e.g., I/O operations on closed files) is \n handled by the parent AsyncContextConnectionManager class in _close_connection().\n \"\"\"\n\n def __init__(\n self,\n server_params: StdioServerParameters,\n errlog: TextIO | None = None,\n ):\n \"\"\"Initialize a new stdio connection manager.\n\n Args:\n server_params: The parameters for the stdio server\n errlog: The error log stream (defaults to filtered sys.stderr)\n \"\"\"\n # Wrap stderr to filter out harmless shutdown errors\n if errlog is None:\n errlog = FilteredStderrWrapper(sys.stderr)\n elif not isinstance(errlog, FilteredStderrWrapper):\n errlog = FilteredStderrWrapper(errlog)\n \n super().__init__(stdio_client, server_params, errlog)\n self.server_params = server_params\n self.errlog = errlog\n self._mcp_logger_filter = None\n self._stop_event: asyncio.Event | None = None # Signal for background task\n self._runner_task: asyncio.Task | None = None # Background runner task\n self._conn_future: asyncio.Future | None = None # Future for the established connection\n logger.debug(\"StdioConnectionManager init with params=%s\", server_params)\n \n async def _establish_connection(self) -> Tuple[Any, Any]:\n \"\"\"Establish connection in a dedicated task to avoid cancel-scope issues.\"\"\"\n # Suppress MCP SDK's noisy JSON parse errors **before** starting the runner\n self._suppress_mcp_json_errors()\n\n # Lazily create primitives the first time we connect\n if self._stop_event is None:\n self._stop_event = asyncio.Event()\n if self._conn_future is None or self._conn_future.done():\n self._conn_future = asyncio.get_event_loop().create_future()\n\n async def _runner(): # Runs in its *own* task (same task for enter/exit)\n try:\n async with stdio_client(self.server_params, self.errlog) as conn:\n # Pass connection back to the caller\n if not self._conn_future.done():\n self._conn_future.set_result(conn)\n # Wait until close is requested\n await self._stop_event.wait()\n finally:\n # Make sure the future is set even on error so awaiters donโt hang\n if not self._conn_future.done():\n self._conn_future.set_exception(RuntimeError(\"Connection failed\"))\n\n # Start background runner if not already active\n if self._runner_task is None or self._runner_task.done():\n self._runner_task = asyncio.create_task(_runner(), name=\"stdio_client_runner\")\n\n # Wait for the connection tuple from the future\n conn: Tuple[Any, Any] = await self._conn_future # type: ignore\n return conn\n\n async def _close_connection(self) -> None:\n \"\"\"Request the background task to exit its context and wait for it.\"\"\"\n try:\n # Restore original logging configuration *before* shutdown\n self._restore_mcp_logging()\n\n # Signal the runner to exit its context manager\n if self._stop_event and not self._stop_event.is_set():\n self._stop_event.set()\n\n # Await the runner task so that __aexit__ executes in *its* task\n if self._runner_task:\n try:\n await asyncio.wait_for(self._runner_task, timeout=2.0)\n except asyncio.TimeoutError:\n logger.warning(\"Timeout while waiting for stdio_client to shut down\")\n finally:\n # Clean up helpers so next connect() creates new ones\n self._runner_task = None\n self._stop_event = None\n self._conn_future = None\n \n def _suppress_mcp_json_errors(self):\n \"\"\"Suppress MCP SDK's JSON parsing error logs.\n \n The MCP SDK logs errors when it receives non-JSON messages from servers.\n These are harmless (the SDK continues working), so we filter them out.\n \"\"\"\n mcp_logger = logging.getLogger(\"mcp.client.stdio\")\n \n class JSONErrorFilter(logging.Filter):\n \"\"\"Filter out JSON parsing errors from MCP SDK.\"\"\"\n def filter(self, record):\n # Suppress \"Failed to parse JSONRPC message\" errors\n if \"Failed to parse JSONRPC message\" in str(record.msg):\n return False\n return True\n \n self._mcp_logger_filter = JSONErrorFilter()\n mcp_logger.addFilter(self._mcp_logger_filter)\n \n def _restore_mcp_logging(self):\n \"\"\"Restore MCP SDK logging to normal.\"\"\"\n if self._mcp_logger_filter:\n mcp_logger = logging.getLogger(\"mcp.client.stdio\")\n mcp_logger.removeFilter(self._mcp_logger_filter)\n self._mcp_logger_filter = None\n\nif not isinstance(sys.stderr, FilteredStderrWrapper):\n sys.stderr = FilteredStderrWrapper(sys.stderr)\n logger.debug(\"Applied global FilteredStderrWrapper to sys.stderr\")"
},
{
"path": "anytool/grounding/backends/mcp/transport/task_managers/streamable_http.py",
"content": "\"\"\"\nStreamable HTTP connection management for MCP implementations.\n\nThis module provides a connection manager for streamable HTTP-based MCP connections\nthat ensures proper task isolation and resource cleanup.\n\"\"\"\n\nfrom datetime import timedelta\nfrom typing import Any, Tuple\nfrom contextlib import asynccontextmanager\n\nfrom mcp.client.streamable_http import streamablehttp_client\nfrom anytool.utils.logging import Logger\nfrom anytool.grounding.core.transport.task_managers import (\n AsyncContextConnectionManager,\n)\n\nlogger = Logger.get_logger(__name__)\n\n\ndef _make_shim():\n \"\"\"\n Create a shim that wraps streamablehttp_client with improved error handling.\n \"\"\"\n @asynccontextmanager\n async def _shim(**kw):\n client_streams = None\n ctx_manager = None\n \n try:\n # Enter the context - this may raise ExceptionGroup during concurrent init\n ctx_manager = streamablehttp_client(**kw)\n try:\n r, w, _sid_cb = await ctx_manager.__aenter__()\n client_streams = (r, w)\n except Exception as conn_error:\n # Handle connection errors during __aenter__\n error_msg = str(conn_error).lower()\n if \"unhandled errors in a taskgroup\" in error_msg:\n logger.debug(f\"TaskGroup race condition during connection: {type(conn_error).__name__}\")\n # Clean up and re-raise to trigger retry\n if ctx_manager:\n try:\n await ctx_manager.__aexit__(None, None, None)\n except Exception:\n pass # Ignore cleanup errors\n raise\n else:\n # Other connection errors - log and re-raise\n logger.warning(f\"Connection error: {conn_error}\")\n raise\n \n # Yield to caller\n yield client_streams\n \n except GeneratorExit:\n # Normal generator exit - this happens during cleanup\n logger.debug(\"StreamableHTTP generator exit (normal cleanup)\")\n \n finally:\n # Always try to exit the context manager\n if ctx_manager is not None:\n try:\n await ctx_manager.__aexit__(None, None, None)\n except (GeneratorExit, RuntimeError, OSError, Exception) as e:\n # Cleanup errors are expected during concurrent shutdown\n # Log at debug level and suppress\n error_type = type(e).__name__\n if \"ExceptionGroup\" in error_type or \"TaskGroup\" in str(e):\n logger.debug(f\"Benign TaskGroup cleanup error: {error_type}\")\n else:\n logger.debug(f\"Benign cleanup error: {error_type}\")\n \n return _shim\n\n\nclass StreamableHttpConnectionManager(\n AsyncContextConnectionManager[Tuple[Any, Any], ...]\n):\n \"\"\"\n MCP Streamable-HTTP connection manager based on the generic\n AsyncContextConnectionManager. Extra session-id callback returned by the\n SDK is discarded by the shim above.\n \"\"\"\n\n def __init__(\n self,\n url: str,\n headers: dict[str, str] | None = None,\n timeout: float = 5,\n read_timeout: float = 60 * 5,\n ):\n shim = _make_shim() \n super().__init__(\n shim,\n url=url,\n headers=headers or {},\n timeout=timedelta(seconds=timeout),\n sse_read_timeout=timedelta(seconds=read_timeout),\n )\n self.url = url\n self.headers = headers or {}\n logger.debug(\"StreamableHttpConnectionManager init url=%s\", url)"
},
{
"path": "anytool/grounding/backends/mcp/transport/task_managers/websocket.py",
"content": "\"\"\"\nWebSocket connection management for MCP implementations.\n\nThis module provides a connection manager for WebSocket-based MCP connections.\n\"\"\"\n\nfrom typing import Any, Tuple\nfrom mcp.client.websocket import websocket_client\nfrom anytool.utils.logging import Logger\nfrom anytool.grounding.core.transport.task_managers import (\n AsyncContextConnectionManager,\n)\n\nlogger = Logger.get_logger(__name__)\n\nclass WebSocketConnectionManager(\n AsyncContextConnectionManager[Tuple[Any, Any], ...]\n):\n\n def __init__(self, url: str, headers: dict[str, str] | None = None):\n # Note: The current MCP websocket_client implementation doesn't support headers\n # If headers need to be passed, this would need to be updated when MCP supports it\n super().__init__(websocket_client, url)\n self.url = url\n self.headers = headers or {}\n logger.debug(\"WebSocketConnectionManager init url=%s\", url)"
},
{
"path": "anytool/grounding/backends/shell/__init__.py",
"content": "from .provider import ShellProvider\nfrom .session import ShellSession\nfrom .transport.connector import ShellConnector\nfrom .transport.local_connector import LocalShellConnector\n\n__all__ = [\n \"ShellProvider\",\n \"ShellSession\",\n \"ShellConnector\",\n \"LocalShellConnector\",\n]"
},
{
"path": "anytool/grounding/backends/shell/provider.py",
"content": "from anytool.grounding.core.provider import Provider\nfrom anytool.grounding.core.types import BackendType, SessionConfig\nfrom .session import ShellSession\nfrom .transport.connector import ShellConnector\nfrom .transport.local_connector import LocalShellConnector\nfrom anytool.config import get_config\nfrom anytool.config.utils import get_config_value\nfrom anytool.platform.config import get_local_server_config\nfrom anytool.utils.logging import Logger\n\nlogger = Logger.get_logger(__name__)\n\n\nclass ShellProvider(Provider[ShellSession]):\n \n DEFAULT_SID = BackendType.SHELL.value\n \n def __init__(self, config: dict | None = None):\n super().__init__(BackendType.SHELL, config)\n # Note: _setup_security_policy() is already called by parent class __init__\n\n def _setup_security_policy(self, config: dict | None = None):\n security_policy = get_config().get_security_policy(self.backend_type.value)\n \n if config:\n security_config = get_config_value(config, \"security\", None)\n if security_config:\n for key, value in security_config.items():\n if hasattr(security_policy, key):\n setattr(security_policy, key, value)\n \n sandbox_enabled = get_config_value(config, \"sandbox_enabled\", None)\n if sandbox_enabled is not None:\n security_policy.sandbox_enabled = sandbox_enabled\n \n logger.info(f\"Shell security policy: allow_shell_commands={security_policy.allow_shell_commands}, \"\n f\"blocked_commands={security_policy.blocked_commands}\")\n \n self.security_manager.set_backend_policy(BackendType.SHELL, security_policy)\n\n async def initialize(self) -> None:\n if not self.is_initialized:\n await self.create_session(SessionConfig(\n session_name=self.DEFAULT_SID,\n backend_type=BackendType.SHELL,\n connection_params={}\n ))\n self.is_initialized = True\n\n async def create_session(self, session_config: SessionConfig) -> ShellSession:\n sid = self.DEFAULT_SID\n if sid in self._sessions:\n return self._sessions[sid]\n \n # Load shell backend configuration\n shell_config = get_config().get_backend_config(\"shell\")\n \n # Determine execution mode: \"local\" or \"server\"\n mode = getattr(shell_config, \"mode\", \"local\")\n \n if mode == \"local\":\n # ---------- LOCAL MODE ----------\n # Execute scripts directly via subprocess, no server required.\n logger.info(\"Shell backend using LOCAL mode (no server required)\")\n connector = LocalShellConnector(\n retry_times=shell_config.max_retries,\n retry_interval=shell_config.retry_interval,\n security_manager=self.security_manager,\n )\n else:\n # ---------- SERVER MODE ----------\n # Connect to a running local_server via HTTP.\n logger.info(\"Shell backend using SERVER mode (connecting to local_server)\")\n local_server_config = get_local_server_config()\n default_port = local_server_config.get('port', shell_config.default_port)\n \n connector = ShellConnector(\n vm_ip=get_config_value(session_config.connection_params, \"vm_ip\", local_server_config['host']),\n port=get_config_value(session_config.connection_params, \"port\", default_port),\n retry_times=shell_config.max_retries,\n retry_interval=shell_config.retry_interval,\n security_manager=self.security_manager,\n )\n \n # Create session with config parameters\n session = ShellSession(\n connector=connector,\n session_id=sid,\n security_manager=self.security_manager,\n default_working_dir=shell_config.working_dir,\n default_env=shell_config.env,\n default_conda_env=shell_config.conda_env\n )\n \n await session.initialize()\n self._sessions[sid] = session\n return session\n\n async def close_session(self, session_id: str) -> None:\n sess = self._sessions.pop(session_id, None)\n if sess:\n await sess.disconnect()"
},
{
"path": "anytool/grounding/backends/shell/session.py",
"content": "import re\nfrom typing import Union\nfrom anytool.grounding.core.types import BackendType\nfrom anytool.grounding.core.session import BaseSession\nfrom anytool.grounding.backends.shell.transport.connector import ShellConnector\nfrom anytool.grounding.backends.shell.transport.local_connector import LocalShellConnector\nfrom anytool.grounding.core.tool import BaseTool\nfrom anytool.grounding.core.security.policies import SecurityPolicyManager\nfrom anytool.llm import LLMClient\nfrom anytool.utils.logging import Logger\n\nlogger = Logger.get_logger(__name__)\n\n\nclass ShellSession(BaseSession):\n backend_type = BackendType.SHELL\n\n def __init__(\n self, \n connector: Union[ShellConnector, LocalShellConnector], \n *, \n session_id: str, \n security_manager: SecurityPolicyManager = None,\n default_working_dir: str = None,\n default_env: dict = None,\n default_conda_env: str = None\n ):\n super().__init__(connector=connector, session_id=session_id,\n backend_type=BackendType.SHELL)\n self.security_manager = security_manager\n self.default_working_dir = default_working_dir\n self.default_env = default_env or {}\n self.default_conda_env = default_conda_env\n\n async def initialize(self):\n self.tools = [ShellAgentTool(\n self, \n security_manager=self.security_manager,\n default_working_dir=self.default_working_dir,\n default_env=self.default_env,\n default_conda_env=self.default_conda_env\n )]\n return {\"tools\": [t.name for t in self.tools]}\n\nclass PythonScriptTool(BaseTool):\n _name = \"_python_exec\"\n _description = \"Internal helper: run python code.\"\n\n def __init__(self, session: \"ShellSession\", default_working_dir: str = None, default_env: dict = None, default_conda_env: str = None):\n self._session = session\n self._default_working_dir = default_working_dir\n self._default_env = default_env or {}\n self._default_conda_env = default_conda_env\n super().__init__()\n\n async def _arun(self, code: str, timeout: int = 90, working_dir: str | None = None, env: dict | None = None, conda_env: str | None = None):\n # Use provided params, or fall back to session defaults\n effective_working_dir = working_dir or self._default_working_dir\n effective_env = {**self._default_env, **(env or {})} # Merge default and provided env\n effective_conda_env = conda_env or self._default_conda_env\n return await self._session.connector.run_python_script(\n code, \n timeout=timeout, \n working_dir=effective_working_dir,\n env=effective_env if effective_env else None,\n conda_env=effective_conda_env\n )\n\nclass BashScriptTool(BaseTool):\n _name = \"_bash_exec\"\n _description = \"Internal helper: run bash script.\"\n\n def __init__(self, session: \"ShellSession\", default_working_dir: str = None, default_env: dict = None, default_conda_env: str = None):\n self._session = session\n self._default_working_dir = default_working_dir\n self._default_env = default_env or {}\n self._default_conda_env = default_conda_env\n super().__init__()\n\n async def _arun(self, script: str, timeout: int = 30, working_dir: str | None = None, env: dict | None = None, conda_env: str | None = None):\n # Use provided params, or fall back to session defaults\n effective_working_dir = working_dir or self._default_working_dir\n effective_env = {**self._default_env, **(env or {})} # Merge default and provided env\n effective_conda_env = conda_env or self._default_conda_env\n return await self._session.connector.run_bash_script(\n script, \n timeout=timeout, \n working_dir=effective_working_dir,\n env=effective_env if effective_env else None,\n conda_env=effective_conda_env\n )\n\nclass ShellAgentTool(BaseTool):\n _name = \"shell_agent\"\n _description = \"\"\"Execute commands or scripts directly in the computer's terminal. \nThis tool uses an internal agent that will write and run Python or Bash code to accomplish tasks or inspect the current system state. The internal agent will automatically retry and fix errors when possible.\n\nUse this tool when you need to:\n- Execute any terminal-based task that requires code\n- Check the current environment (files, processes, system info)\n- Run calculations or data processing\n- Install packages or modify system settings\n\nThe tool will keep trying until the task succeeds or determines it cannot be completed.\"\"\"\n \n backend_type = BackendType.SHELL\n _CODE_RGX = re.compile(\n r\"```(?Ppython|py|bash|shell|sh)[^\\n]*\\n(?P.*?)```\",\n re.S | re.I,\n )\n\n def __init__(\n self, \n session: \"ShellSession\", \n client_password: str = \"\", \n max_steps: int = 5,\n security_manager: SecurityPolicyManager = None,\n default_working_dir: str = None,\n default_env: dict = None,\n default_conda_env: str = None\n ):\n self._session = session\n self._llm = LLMClient()\n self.client_password = client_password\n self.max_steps = max_steps\n self._system_info = None\n self.security_manager = security_manager\n self._default_working_dir = default_working_dir\n self._default_env = default_env or {}\n self._default_conda_env = default_conda_env\n self._py_tool = PythonScriptTool(session, default_working_dir=default_working_dir, default_env=default_env, default_conda_env=default_conda_env)\n self._bash_tool = BashScriptTool(session, default_working_dir=default_working_dir, default_env=default_env, default_conda_env=default_conda_env)\n super().__init__()\n\n async def _get_system_info(self):\n \"\"\"\n Get system information for shell agent.\n \n First tries to get comprehensive info from local server's /platform endpoint.\n Falls back to simple bash commands if that fails.\n \n Returns:\n Dict with at least 'platform' and 'username' keys\n \"\"\"\n if self._system_info is None:\n try:\n # Try to get system info from server via HTTP API\n try:\n from anytool.platform import SystemInfoClient\n \n # Get base_url from connector\n base_url = self._session.connector.base_url\n \n # Create temporary client\n async with SystemInfoClient(base_url=base_url, timeout=5) as client:\n info = await client.get_system_info(use_cache=False)\n \n if info:\n # Use comprehensive info from server\n self._system_info = {\n \"platform\": info.get(\"system\", \"Linux\"),\n \"username\": info.get(\"username\", \"user\"),\n \"machine\": info.get(\"machine\"),\n \"release\": info.get(\"release\"),\n \"full_info\": info # Keep full info for reference\n }\n logger.debug(f\"Got system info from server: {info.get('system')}\")\n return self._system_info\n \n except ImportError:\n logger.debug(\"SystemInfoClient not available, using bash commands\")\n \n # Fallback: use simple bash commands (original method)\n platform_result = await self._session.connector.run_bash_script(\"uname -s\", timeout=5)\n username_result = await self._session.connector.run_bash_script(\"whoami\", timeout=5)\n \n platform = self._extract_output(platform_result).strip()\n username = self._extract_output(username_result).strip()\n \n self._system_info = {\n \"platform\": platform,\n \"username\": username\n }\n logger.debug(f\"Got system info from bash: {platform}\")\n \n except Exception as e:\n logger.warning(f\"Failed to get system info: {e}, using defaults\")\n self._system_info = {\"platform\": \"Linux\", \"username\": \"user\"}\n \n return self._system_info\n\n async def _arun(self, task: str, timeout: int = 300):\n from anytool.grounding.core.types import ToolResult, ToolStatus\n \n sys_info = await self._get_system_info()\n conversation_history = []\n iteration = 0\n last_error = None\n \n # record the code history\n code_history = []\n \n # Build environment context\n env_context = []\n if self._default_working_dir:\n env_context.append(f\"Working Directory: {self._default_working_dir}\")\n if self._default_conda_env:\n env_context.append(f\"Conda Environment: {self._default_conda_env}\")\n if self._default_env:\n env_vars = \", \".join([f\"{k}={v}\" for k, v in list(self._default_env.items())[:3]])\n if len(self._default_env) > 3:\n env_vars += f\", ... (+{len(self._default_env)-3} more)\"\n env_context.append(f\"Custom Environment Variables: {env_vars}\")\n \n env_section = \"\\n\".join([f\"# {ctx}\" for ctx in env_context]) if env_context else \"\"\n \n SHELL_AGENT_SYSTEM_PROMPT = f\"\"\"You are an expert system administrator and programmer focused on executing tasks efficiently.\n\n# System: {sys_info[\"platform\"]}, User: {sys_info[\"username\"]}\n{env_section}\n\n# Your task: {task}\n\n# IMPORTANT: You MUST provide exactly ONE code block in EVERY response\n# Either ```bash or ```python - never respond without code\n\n# Available actions:\n1. Execute bash commands: ```bash ```\n2. Write Python code: ```python ```\n\n# Rules:\n- ALWAYS include a code block in your response\n- Write EXACTLY ONE code block per response\n- If you need to understand the current environment, start with bash commands like: pwd, ls, ps, df, etc.\n- If you get errors, analyze and fix them in the next iteration\n- For sudo: use 'echo {self.client_password} | sudo -S '\n- The environment (working directory, conda env) is managed automatically\n\n# CRITICAL: Avoid quote escaping errors in bash:\n- For complex string operations (JSON, multi-line text, special chars): ALWAYS use Python with heredoc\n- Good: ```python ```\n- Bad: bash commands with nested quotes like: echo \"$(cat 'file' | grep \"pattern\")\"\n- When reading/writing files with complex content: prefer Python over bash\n- When processing JSON: ALWAYS use Python's json module, never bash string manipulation\n\n# Before executing, check if task output already exists:\n- Use 'ls -la ' to check for existing files\n- If files exist, read and verify them first before recreating\n- Avoid redundant work - reuse existing valid outputs\n\n# Task completion marking:\nWhen you believe the task is COMPLETED, end your response with:\n[TASK_COMPLETED: brief explanation of what was accomplished]\n\nWhen you encounter an UNRECOVERABLE error that you cannot fix, end your response with:\n[TASK_FAILED: brief explanation of why it cannot be completed]\"\"\"\n\n conversation_history.append({\"role\": \"system\", \"content\": SHELL_AGENT_SYSTEM_PROMPT})\n \n no_code_counter = 0\n final_message = \"\"\n \n while iteration < self.max_steps:\n iteration += 1\n \n logger.info(f\"[ShellAgent] Step {iteration}/{self.max_steps}: Processing task\")\n \n try: \n messages_text = LLMClient.format_messages_to_text(conversation_history)\n response = await self._llm.complete(messages_text)\n\n assistant_content = response[\"message\"][\"content\"]\n logger.debug(f\"[ShellAgent] Step {iteration} LLM response: {assistant_content[:200]}...\")\n\n # extract and execute the code, and track the code block\n code_info, execution_result = await self._execute_code_from_response(assistant_content)\n if code_info:\n code_history.append(code_info)\n \n logger.info(f\"[ShellAgent] Step {iteration} execution result: {execution_result[:100]}...\")\n if execution_result == \"ERROR: No valid code block found\":\n no_code_counter += 1\n if no_code_counter >= 3:\n final_message = f\"Task failed after {iteration} steps: LLM failed to provide code blocks repeatedly\"\n return ToolResult(\n status=ToolStatus.ERROR,\n content=final_message,\n metadata={\"tool\": self._name, \"code_history\": code_history}\n )\n else:\n no_code_counter = 0\n \n completion_status = self._check_task_status(assistant_content, execution_result, last_error)\n \n if completion_status[\"completed\"]:\n content_parts = [f\"Task completed successfully after {iteration} steps\"]\n content_parts.append(f\"\\n{'='*60}\")\n content_parts.append(f\"\\nFinal Result:\")\n content_parts.append(execution_result)\n \n if len(code_history) > 1:\n content_parts.append(f\"\\n{'='*60}\")\n content_parts.append(f\"\\nExecution Summary ({len(code_history)} steps):\")\n for i, code_info in enumerate(code_history, 1):\n lang = code_info.get(\"language\", \"unknown\")\n output = code_info.get(\"output\", \"\")\n output_preview = output[:200].replace('\\n', ' ')\n if len(output) > 200:\n output_preview += \"...\"\n content_parts.append(f\"\\n Step {i} [{lang}]: {output_preview}\")\n \n content_parts.append(f\"\\n{'='*60}\")\n content_parts.append(f\"\\nSummary: {completion_status['reason']}\")\n \n final_message = \"\\n\".join(content_parts)\n return ToolResult(\n status=ToolStatus.SUCCESS,\n content=final_message,\n metadata={\"tool\": self._name, \"code_history\": code_history}\n )\n elif completion_status[\"failed\"]:\n final_message = f\"Task failed after {iteration} steps: {completion_status['reason']}\\nLast result: {execution_result}\"\n return ToolResult(\n status=ToolStatus.ERROR,\n content=final_message,\n metadata={\"tool\": self._name, \"code_history\": code_history}\n )\n\n feedback = self._generate_feedback(execution_result, iteration, last_error)\n \n conversation_history.extend([\n {\"role\": \"assistant\", \"content\": assistant_content},\n {\"role\": \"user\", \"content\": feedback}\n ])\n \n last_error = execution_result if \"ERROR\" in execution_result else None\n \n except Exception as e:\n final_message = f\"Tool execution failed at step {iteration}: {str(e)}\"\n return ToolResult(\n status=ToolStatus.ERROR,\n content=final_message,\n metadata={\"tool\": self._name, \"code_history\": code_history}\n )\n \n final_message = f\"Reached maximum steps ({self.max_steps}). Task may be too complex or impossible.\"\n return ToolResult(\n status=ToolStatus.ERROR,\n content=final_message,\n metadata={\"tool\": self._name, \"code_history\": code_history}\n )\n\n async def _execute_code_from_response(self, response: str):\n \"\"\"\n execute the code and track the code block\n \n Returns:\n Tuple[Optional[Dict], str]: (code_info, execution_result)\n - code_info: {\"lang\": \"python/bash\", \"code\": \"...\", \"status\": \"success/error\"}\n - execution_result: the execution result string\n \"\"\"\n matches = list(self._CODE_RGX.finditer(response))\n if not matches:\n return None, \"ERROR: No valid code block found\"\n \n lang, code = matches[0][\"lang\"].lower(), matches[0][\"code\"].strip()\n \n # standardize the language name\n lang_normalized = \"python\" if lang in [\"python\", \"py\"] else \"bash\"\n \n code_info = {\n \"lang\": lang_normalized,\n \"code\": code,\n }\n\n # Security check is only done at the Connector layer to avoid duplicate prompts\n \n try:\n if lang in [\"python\", \"py\"]:\n helper = self._py_tool\n result = await helper._arun(code)\n elif lang in [\"bash\", \"shell\", \"sh\"]:\n helper = self._bash_tool\n result = await helper._arun(code)\n else:\n execution_result = f\"ERROR: Unsupported language: {lang}\"\n code_info[\"status\"] = \"error\"\n return code_info, execution_result\n \n execution_result = self._extract_output(result)\n code_info[\"status\"] = \"success\" if \"ERROR\" not in execution_result else \"error\"\n return code_info, execution_result\n \n except Exception as e:\n execution_result = f\"EXECUTION ERROR: {str(e)}\"\n code_info[\"status\"] = \"error\"\n return code_info, execution_result\n\n def _generate_feedback(self, result: str, iteration: int, last_error: str) -> str:\n feedback = f\"Step {iteration} result:\\n{result}\\n\\n\"\n \n if \"ERROR\" in result:\n if last_error and last_error == result:\n feedback += \"Same error as previous step. Try a different approach.\\n\"\n else:\n feedback += \"Error occurred. Analyze the error and fix it.\\n\"\n else:\n feedback += \"Execution successful. Continue to next step if needed.\\n\"\n \n feedback += \"\\nWhat's your next action? (Remember: provide exactly ONE code block)\"\n return feedback\n\n def _extract_output(self, result):\n if isinstance(result, dict):\n # Check for execution errors\n stderr = result.get(\"error\") or result.get(\"stderr\") or \"\"\n returncode = result.get(\"returncode\", 0)\n stdout = result.get(\"content\") or result.get(\"output\") or result.get(\"stdout\") or \"\"\n \n # If there's a non-zero return code or stderr with actual errors, report it\n if returncode != 0 or (stderr and len(stderr.strip()) > 0):\n error_msg = f\"EXECUTION ERROR (exit code {returncode}):\\n\"\n if stderr:\n error_msg += f\"stderr: {stderr}\\n\"\n if stdout:\n error_msg += f\"stdout: {stdout}\"\n return error_msg\n \n return stdout or str(result)\n return str(result)\n\n def _check_task_status(self, response: str, execution_result: str, last_error: str) -> dict:\n if \"[TASK_COMPLETED:\" in response:\n reason = response.split(\"[TASK_COMPLETED:\")[1].split(\"]\")[0].strip()\n return {\"completed\": True, \"failed\": False, \"reason\": reason}\n\n if \"[TASK_FAILED:\" in response:\n reason = response.split(\"[TASK_FAILED:\")[1].split(\"]\")[0].strip()\n return {\"completed\": False, \"failed\": True, \"reason\": reason}\n\n # Extended error pattern detection\n error_patterns = [\n \"ERROR:\",\n \"EXECUTION ERROR:\",\n \"CommandNotFoundError\",\n \"Traceback (most recent call last)\",\n \"Exception:\",\n \"PermissionError\",\n \"FileNotFoundError\",\n \"SyntaxError:\",\n \"ImportError:\",\n \"ModuleNotFoundError\",\n \"No such file or directory\",\n \"command not found\",\n ]\n \n has_error = any(pattern in execution_result for pattern in error_patterns)\n \n if has_error:\n if last_error and last_error == execution_result:\n return {\"completed\": False, \"failed\": True, \"reason\": \"Same error repeated - unable to resolve\"}\n return {\"completed\": False, \"failed\": False, \"reason\": \"Execution error occurred\"}\n\n return {\"completed\": False, \"failed\": False, \"reason\": \"Task in progress\"}"
},
{
"path": "anytool/grounding/backends/shell/transport/connector.py",
"content": "import asyncio\nfrom typing import Any, Optional, Dict\n\nfrom anytool.grounding.core.transport.connectors import AioHttpConnector\nfrom anytool.grounding.core.security import SecurityPolicyManager\nfrom anytool.utils.logging import Logger\n\nlogger = Logger.get_logger(__name__)\n\n\nclass ShellConnector(AioHttpConnector):\n \"\"\"\n Shell backend HTTP connector\n Basic routes:\n POST /run_python {\"code\": str}\n POST /run_bash_script {\"script\": str, \"timeout\": int, \"working_dir\": str | None}\n \"\"\"\n\n def __init__(\n self,\n vm_ip: str,\n port: int = 5000,\n *,\n retry_times: int = 3,\n retry_interval: float = 5,\n security_manager: \"SecurityPolicyManager | None\" = None,\n ) -> None:\n base_url = f\"http://{vm_ip}:{port}\"\n super().__init__(base_url)\n self.retry_times = retry_times\n self.retry_interval = retry_interval\n self._security_manager = security_manager\n\n async def _retry_invoke(\n self, \n name: str, \n payload: Dict[str, Any], \n script_timeout: int,\n *,\n break_on_timeout: bool = False\n ):\n \"\"\"\n Execute HTTP request and retry\n \n Args:\n name: RPC method name\n payload: Request payload\n script_timeout: Script execution timeout\n break_on_timeout: Whether to exit immediately on timeout (default False)\n \n Returns:\n Server response result\n \n Raises:\n Exception: Last exception thrown after all retries fail\n \"\"\"\n last_exc: Exception | None = None\n # HTTP request timeout should be longer than script execution timeout, leaving buffer time\n http_timeout = script_timeout + 60\n \n for attempt in range(1, self.retry_times + 1):\n try:\n # Pass timeout parameter to server\n result = await self.invoke(name, payload | {\"timeout\": script_timeout})\n logger.info(\"%s executed successfully (attempt %d/%d)\", name, attempt, self.retry_times)\n return result\n except asyncio.TimeoutError as exc:\n # Timeout exception usually does not need to be retried (script execution time too long)\n if break_on_timeout:\n logger.error(\"%s timed out after %d seconds, aborting retry\", name, script_timeout)\n raise RuntimeError(\n f\"Script execution timed out after {script_timeout} seconds\"\n ) from exc\n last_exc = exc\n if attempt == self.retry_times:\n break\n logger.warning(\n \"%s timed out (attempt %d/%d), retrying in %.1f seconds...\", \n name, attempt, self.retry_times, self.retry_interval\n )\n await asyncio.sleep(self.retry_interval)\n except Exception as exc:\n last_exc = exc\n if attempt == self.retry_times:\n break\n logger.warning(\n \"%s failed (attempt %d/%d): %s, retrying in %.1f seconds...\", \n name, attempt, self.retry_times, exc, self.retry_interval\n )\n await asyncio.sleep(self.retry_interval)\n \n error_msg = f\"{name} failed after {self.retry_times} retries\"\n logger.error(error_msg)\n raise last_exc or RuntimeError(error_msg)\n\n async def run_python_script(\n self, \n code: str, \n *, \n timeout: int = 90,\n working_dir: Optional[str] = None,\n env: Optional[Dict[str, str]] = None,\n conda_env: Optional[str] = None\n ) -> Any:\n \"\"\"\n Execute Python script on remote server\n \n Args:\n code: Python code string\n timeout: Execution timeout in seconds (default 90 seconds)\n working_dir: Working directory for script execution (optional)\n env: Environment variables for script execution (optional)\n conda_env: Conda environment name to activate (optional)\n \n Returns:\n Server response result\n \n Raises:\n PermissionError: Security policy blocked execution\n RuntimeError: Execution failed or timed out\n \"\"\"\n if self._security_manager:\n from anytool.grounding.core.types import BackendType\n allowed = await self._security_manager.check_command_allowed(BackendType.SHELL, code)\n if not allowed:\n logger.error(\"SecurityPolicy blocked python code execution\")\n raise PermissionError(\"SecurityPolicy: python code execution blocked\")\n \n payload = {\"code\": code, \"working_dir\": working_dir, \"env\": env, \"conda_env\": conda_env}\n logger.info(\n \"Executing python script with timeout=%d seconds%s%s%s\",\n timeout,\n f\", working_dir={working_dir}\" if working_dir else \"\",\n f\", env={list(env.keys())}\" if env else \"\",\n f\", conda_env={conda_env}\" if conda_env else \"\"\n )\n # Python script timed out, exit immediately without retry (timeout usually means script logic problem)\n return await self._retry_invoke(\n \"POST /run_python\", \n payload, \n timeout,\n break_on_timeout=True\n )\n\n async def run_bash_script(\n self,\n script: str,\n *,\n timeout: int = 90,\n working_dir: Optional[str] = None,\n env: Optional[Dict[str, str]] = None,\n conda_env: Optional[str] = None\n ) -> Any:\n \"\"\"\n Execute Bash script on remote server\n \n Args:\n script: Bash script content (can be multi-line)\n timeout: Execution timeout in seconds (default 90 seconds)\n working_dir: Working directory for script execution (optional)\n env: Environment variables for script execution (optional)\n conda_env: Conda environment name to activate (optional)\n \n Returns:\n Server response result, containing status, output, error, returncode, etc.\n \n Raises:\n PermissionError: Security policy blocked execution\n RuntimeError: Execution failed or timed out\n \"\"\"\n if self._security_manager:\n from anytool.grounding.core.types import BackendType\n allowed = await self._security_manager.check_command_allowed(BackendType.SHELL, script)\n if not allowed:\n logger.error(\"SecurityPolicy blocked bash script execution\")\n raise PermissionError(\"SecurityPolicy: bash script execution blocked\")\n \n payload = {\"script\": script, \"working_dir\": working_dir, \"env\": env, \"conda_env\": conda_env}\n logger.info(\n \"Executing bash script with timeout=%d seconds%s%s%s\", \n timeout,\n f\", working_dir={working_dir}\" if working_dir else \"\",\n f\", env={list(env.keys())}\" if env else \"\",\n f\", conda_env={conda_env}\" if conda_env else \"\"\n )\n \n # Bash script timed out, exit immediately without retry (timeout usually means script logic problem)\n result = await self._retry_invoke(\n \"POST /run_bash_script\", \n payload, \n timeout,\n break_on_timeout=True\n )\n \n # Record execution result\n if isinstance(result, dict) and \"returncode\" in result:\n logger.info(\"Bash script executed with return code: %d\", result.get(\"returncode\", -1))\n \n return result"
},
{
"path": "anytool/grounding/backends/shell/transport/local_connector.py",
"content": "\"\"\"\nLocal Shell Connector โ execute Python / Bash scripts directly via subprocess.\n\nThis connector has the **same public API** as ShellConnector (HTTP version)\nbut runs everything in-process, removing the need for a local_server.\n\nReturn format is kept identical so that ShellSession / ShellAgentTool\nwork without any changes.\n\"\"\"\n\nimport asyncio\nimport os\nimport platform\nimport tempfile\nimport uuid\nfrom typing import Any, Optional, Dict\n\nfrom anytool.grounding.core.transport.connectors.base import BaseConnector\nfrom anytool.grounding.core.transport.task_managers.noop import NoOpConnectionManager\nfrom anytool.grounding.core.security import SecurityPolicyManager\nfrom anytool.utils.logging import Logger\n\nlogger = Logger.get_logger(__name__)\n\nplatform_name = platform.system()\n\n\n# ---------------------------------------------------------------------------\n# Conda helpers (mirrored from local_server/main.py)\n# ---------------------------------------------------------------------------\n\ndef _get_conda_activation_prefix(conda_env: str | None) -> str:\n \"\"\"Generate platform-specific conda activation prefix.\"\"\"\n if not conda_env:\n return \"\"\n if platform_name == \"Windows\":\n conda_paths = [\n os.path.expandvars(r\"%USERPROFILE%\\miniconda3\\Scripts\\activate.bat\"),\n os.path.expandvars(r\"%USERPROFILE%\\anaconda3\\Scripts\\activate.bat\"),\n r\"C:\\ProgramData\\Miniconda3\\Scripts\\activate.bat\",\n r\"C:\\ProgramData\\Anaconda3\\Scripts\\activate.bat\",\n ]\n for p in conda_paths:\n if os.path.exists(p):\n return f'call \"{p}\" {conda_env} && '\n return f\"conda activate {conda_env} && \"\n else:\n conda_paths = [\n os.path.expanduser(\"~/miniconda3/etc/profile.d/conda.sh\"),\n os.path.expanduser(\"~/anaconda3/etc/profile.d/conda.sh\"),\n \"/opt/conda/etc/profile.d/conda.sh\",\n \"/usr/local/miniconda3/etc/profile.d/conda.sh\",\n \"/usr/local/anaconda3/etc/profile.d/conda.sh\",\n ]\n for p in conda_paths:\n if os.path.exists(p):\n return f'source \"{p}\" && conda activate {conda_env} && '\n return f\"conda activate {conda_env} && \"\n\n\ndef _wrap_script_with_conda(script: str, conda_env: str | None) -> str:\n \"\"\"Wrap bash script with conda activation if needed.\"\"\"\n if not conda_env:\n return script\n if platform_name == \"Windows\":\n prefix = _get_conda_activation_prefix(conda_env)\n return f\"{prefix}{script}\"\n else:\n conda_paths = [\n os.path.expanduser(\"~/miniconda3/etc/profile.d/conda.sh\"),\n os.path.expanduser(\"~/anaconda3/etc/profile.d/conda.sh\"),\n os.path.expanduser(\"~/opt/anaconda3/etc/profile.d/conda.sh\"),\n \"/opt/conda/etc/profile.d/conda.sh\",\n ]\n conda_sh = None\n for p in conda_paths:\n if os.path.exists(p):\n conda_sh = p\n break\n if conda_sh:\n return (\n f'#!/bin/bash\\n'\n f'if [ -f \"{conda_sh}\" ]; then\\n'\n f' . \"{conda_sh}\"\\n'\n f' conda activate {conda_env} 2>/dev/null || true\\n'\n f'fi\\n\\n'\n f'{script}\\n'\n )\n else:\n logger.warning(\n \"Conda environment '%s' requested but conda not found. \"\n \"Executing with system Python.\", conda_env\n )\n return script\n\n\nclass LocalShellConnector(BaseConnector[Any]):\n \"\"\"\n Shell connector that runs scripts **locally** using asyncio subprocesses,\n bypassing the Flask local_server entirely.\n \n Public API is compatible with ``ShellConnector`` so that ``ShellSession``\n works without modification.\n \"\"\"\n\n def __init__(\n self,\n *,\n retry_times: int = 3,\n retry_interval: float = 5,\n security_manager: \"SecurityPolicyManager | None\" = None,\n ) -> None:\n super().__init__(NoOpConnectionManager())\n self.retry_times = retry_times\n self.retry_interval = retry_interval\n self._security_manager = security_manager\n # Provide base_url = None so ShellSession._get_system_info falls back\n # to bash-based detection instead of HTTP.\n self.base_url: str | None = None\n\n # ------------------------------------------------------------------\n # connect / disconnect (mostly no-ops for local execution)\n # ------------------------------------------------------------------\n\n async def connect(self) -> None:\n \"\"\"No real connection to establish for local mode.\"\"\"\n if self._connected:\n return\n await super().connect()\n logger.info(\"LocalShellConnector: ready (local mode, no server required)\")\n\n # ------------------------------------------------------------------\n # Core execution helpers\n # ------------------------------------------------------------------\n\n async def _run_subprocess(\n self,\n cmd: list[str],\n *,\n timeout: int = 90,\n working_dir: str | None = None,\n env: dict[str, str] | None = None,\n ) -> Dict[str, Any]:\n \"\"\"Run a command via asyncio subprocess and return a result dict\n matching the format returned by the local_server endpoints.\"\"\"\n exec_env = os.environ.copy()\n if env:\n exec_env.update(env)\n\n cwd = working_dir or os.getcwd()\n\n try:\n proc = await asyncio.create_subprocess_exec(\n *cmd,\n stdout=asyncio.subprocess.PIPE,\n stderr=asyncio.subprocess.PIPE,\n cwd=cwd,\n env=exec_env,\n )\n stdout_b, stderr_b = await asyncio.wait_for(\n proc.communicate(), timeout=timeout\n )\n stdout = stdout_b.decode(\"utf-8\", errors=\"replace\") if stdout_b else \"\"\n stderr = stderr_b.decode(\"utf-8\", errors=\"replace\") if stderr_b else \"\"\n returncode = proc.returncode or 0\n\n return {\n \"status\": \"success\" if returncode == 0 else \"error\",\n \"output\": stdout,\n \"content\": stdout or \"Code executed successfully (no output)\",\n \"error\": stderr,\n \"returncode\": returncode,\n }\n except asyncio.TimeoutError:\n return {\n \"status\": \"error\",\n \"output\": f\"Execution timed out after {timeout} seconds\",\n \"content\": f\"Execution timed out after {timeout} seconds\",\n \"error\": \"\",\n \"returncode\": -1,\n }\n except Exception as e:\n return {\n \"status\": \"error\",\n \"output\": \"\",\n \"content\": \"\",\n \"error\": str(e),\n \"returncode\": -1,\n }\n\n async def _run_shell_command(\n self,\n shell_cmd: str,\n *,\n timeout: int = 90,\n working_dir: str | None = None,\n env: dict[str, str] | None = None,\n ) -> Dict[str, Any]:\n \"\"\"Run a shell command string (used for conda-wrapped scripts).\"\"\"\n exec_env = os.environ.copy()\n if env:\n exec_env.update(env)\n\n cwd = working_dir or os.getcwd()\n\n try:\n proc = await asyncio.create_subprocess_shell(\n shell_cmd,\n stdout=asyncio.subprocess.PIPE,\n stderr=asyncio.subprocess.STDOUT,\n cwd=cwd,\n env=exec_env,\n )\n stdout_b, _ = await asyncio.wait_for(\n proc.communicate(), timeout=timeout\n )\n stdout = stdout_b.decode(\"utf-8\", errors=\"replace\") if stdout_b else \"\"\n returncode = proc.returncode or 0\n\n return {\n \"status\": \"success\" if returncode == 0 else \"error\",\n \"output\": stdout,\n \"content\": stdout or \"Code executed successfully (no output)\",\n \"error\": \"\",\n \"returncode\": returncode,\n }\n except asyncio.TimeoutError:\n return {\n \"status\": \"error\",\n \"output\": f\"Script execution timed out after {timeout} seconds\",\n \"content\": f\"Script execution timed out after {timeout} seconds\",\n \"error\": \"\",\n \"returncode\": -1,\n }\n except Exception as e:\n return {\n \"status\": \"error\",\n \"output\": \"\",\n \"content\": \"\",\n \"error\": str(e),\n \"returncode\": -1,\n }\n\n # ------------------------------------------------------------------\n # Public API (same signatures as ShellConnector)\n # ------------------------------------------------------------------\n\n async def run_python_script(\n self,\n code: str,\n *,\n timeout: int = 90,\n working_dir: Optional[str] = None,\n env: Optional[Dict[str, str]] = None,\n conda_env: Optional[str] = None,\n ) -> Any:\n \"\"\"Execute a Python script locally.\n\n Return format matches the server's ``/run_python`` endpoint.\n \"\"\"\n # Security check\n if self._security_manager:\n from anytool.grounding.core.types import BackendType\n allowed = await self._security_manager.check_command_allowed(\n BackendType.SHELL, code\n )\n if not allowed:\n logger.error(\"SecurityPolicy blocked python code execution\")\n raise PermissionError(\"SecurityPolicy: python code execution blocked\")\n\n # Write code to temp file (same as local_server)\n suffix = uuid.uuid4().hex\n if platform_name == \"Windows\":\n temp_filename = os.path.join(tempfile.gettempdir(), f\"python_exec_{suffix}.py\")\n else:\n temp_filename = f\"/tmp/python_exec_{suffix}.py\"\n\n try:\n with open(temp_filename, \"w\") as f:\n f.write(code)\n\n logger.info(\n \"Executing python script locally with timeout=%d seconds%s%s%s\",\n timeout,\n f\", working_dir={working_dir}\" if working_dir else \"\",\n f\", env={list(env.keys())}\" if env else \"\",\n f\", conda_env={conda_env}\" if conda_env else \"\",\n )\n\n if conda_env:\n activation = _get_conda_activation_prefix(conda_env)\n if activation:\n python_cmd = \"python\" if platform_name == \"Windows\" else \"python3\"\n full_cmd = f'{activation}{python_cmd} \"{temp_filename}\"'\n result = await self._run_shell_command(\n full_cmd, timeout=timeout, working_dir=working_dir, env=env\n )\n else:\n python_cmd = \"python\" if platform_name == \"Windows\" else \"python3\"\n result = await self._run_subprocess(\n [python_cmd, temp_filename],\n timeout=timeout,\n working_dir=working_dir,\n env=env,\n )\n else:\n python_cmd = \"python\" if platform_name == \"Windows\" else \"python3\"\n result = await self._run_subprocess(\n [python_cmd, temp_filename],\n timeout=timeout,\n working_dir=working_dir,\n env=env,\n )\n\n return result\n\n finally:\n if os.path.exists(temp_filename):\n os.remove(temp_filename)\n\n async def run_bash_script(\n self,\n script: str,\n *,\n timeout: int = 90,\n working_dir: Optional[str] = None,\n env: Optional[Dict[str, str]] = None,\n conda_env: Optional[str] = None,\n ) -> Any:\n \"\"\"Execute a Bash script locally.\n\n Return format matches the server's ``/run_bash_script`` endpoint.\n \"\"\"\n # Security check\n if self._security_manager:\n from anytool.grounding.core.types import BackendType\n allowed = await self._security_manager.check_command_allowed(\n BackendType.SHELL, script\n )\n if not allowed:\n logger.error(\"SecurityPolicy blocked bash script execution\")\n raise PermissionError(\"SecurityPolicy: bash script execution blocked\")\n\n # Wrap with conda if needed\n final_script = _wrap_script_with_conda(script, conda_env)\n\n # Write to temp file (same as local_server)\n suffix = uuid.uuid4().hex\n if platform_name == \"Windows\":\n temp_filename = os.path.join(tempfile.gettempdir(), f\"bash_exec_{suffix}.sh\")\n else:\n temp_filename = f\"/tmp/bash_exec_{suffix}.sh\"\n\n try:\n with open(temp_filename, \"w\") as f:\n f.write(final_script)\n os.chmod(temp_filename, 0o755)\n\n logger.info(\n \"Executing bash script locally with timeout=%d seconds%s%s%s\",\n timeout,\n f\", working_dir={working_dir}\" if working_dir else \"\",\n f\", env={list(env.keys())}\" if env else \"\",\n f\", conda_env={conda_env}\" if conda_env else \"\",\n )\n\n shell_cmd = [\"bash\", temp_filename] if platform_name == \"Windows\" else [\"/bin/bash\", temp_filename]\n result = await self._run_subprocess(\n shell_cmd,\n timeout=timeout,\n working_dir=working_dir,\n env=env,\n )\n return result\n\n finally:\n if os.path.exists(temp_filename):\n os.unlink(temp_filename)\n\n # ------------------------------------------------------------------\n # BaseConnector abstract methods\n # ------------------------------------------------------------------\n\n async def invoke(self, name: str, params: dict[str, Any]) -> Any:\n \"\"\"Dispatch by name โ same routing as ShellConnector via AioHttpConnector.\"\"\"\n name_upper = name.strip().upper()\n if \"/RUN_PYTHON\" in name_upper:\n return await self.run_python_script(\n params.get(\"code\", \"\"),\n timeout=params.get(\"timeout\", 90),\n working_dir=params.get(\"working_dir\"),\n env=params.get(\"env\"),\n conda_env=params.get(\"conda_env\"),\n )\n elif \"/RUN_BASH_SCRIPT\" in name_upper:\n return await self.run_bash_script(\n params.get(\"script\", \"\"),\n timeout=params.get(\"timeout\", 90),\n working_dir=params.get(\"working_dir\"),\n env=params.get(\"env\"),\n conda_env=params.get(\"conda_env\"),\n )\n else:\n raise NotImplementedError(f\"LocalShellConnector does not support: {name}\")\n\n async def request(self, *args: Any, **kwargs: Any) -> Any:\n \"\"\"Not used in local mode.\"\"\"\n raise NotImplementedError(\n \"LocalShellConnector does not support raw HTTP requests\"\n )\n\n"
},
{
"path": "anytool/grounding/backends/web/__init__.py",
"content": "from .provider import WebProvider\nfrom .session import WebSession\n\n__all__ = [\n \"WebProvider\",\n \"WebSession\"\n]"
},
{
"path": "anytool/grounding/backends/web/provider.py",
"content": "from typing import Dict, Any\nfrom anytool.grounding.core.types import BackendType, SessionConfig\nfrom anytool.grounding.core.provider import Provider\nfrom .session import WebSession\nfrom anytool.utils.logging import Logger\n\nlogger = Logger.get_logger(__name__)\n\n\nclass WebProvider(Provider[WebSession]):\n \n DEFAULT_SID = BackendType.WEB.value\n \n def __init__(self, config: Dict[str, Any] = None):\n super().__init__(BackendType.WEB, config)\n \n async def initialize(self) -> None:\n \"\"\"Initialize Web Provider and create default session\"\"\"\n if not self.is_initialized:\n logger.info(\"Initializing Web provider (Knowledge Research)\")\n # Auto-create default session\n await self.create_session(SessionConfig(\n session_name=self.DEFAULT_SID,\n backend_type=BackendType.WEB,\n connection_params={}\n ))\n self.is_initialized = True\n \n async def create_session(self, session_config: SessionConfig) -> WebSession:\n \"\"\"Create Web session\"\"\"\n session_name = session_config.session_name\n \n if session_name in self._sessions:\n logger.warning(f\"Session {session_name} already exists, returning existing session\")\n return self._sessions[session_name]\n \n # Create WebSession with auto-connect and auto-initialize enabled\n session = WebSession(\n session_id=session_name,\n config=session_config,\n auto_connect=True,\n auto_initialize=True\n )\n \n self._sessions[session_name] = session\n \n logger.info(f\"Created Web session (Knowledge Research): {session_name}\")\n return session\n \n async def close_session(self, session_name: str) -> None:\n \"\"\"Close Web session\"\"\"\n session = self._sessions.pop(session_name, None)\n if session:\n await session.disconnect()\n logger.info(f\"Closed Web session: {session_name}\")"
},
{
"path": "anytool/grounding/backends/web/session.py",
"content": "import os\nfrom typing import Dict, Any, Optional\nfrom anytool.grounding.core.session import BaseSession\nfrom anytool.grounding.core.types import BackendType, SessionConfig\nfrom anytool.grounding.core.tool import BaseTool\nfrom anytool.grounding.core.transport.connectors import BaseConnector\nfrom anytool.llm import LLMClient\nfrom anytool.utils.logging import Logger\nfrom dotenv import load_dotenv\n\nload_dotenv()\nlogger = Logger.get_logger(__name__)\n\n\ntry:\n from openai import AsyncOpenAI\n OPENAI_AVAILABLE = True\nexcept ImportError:\n OPENAI_AVAILABLE = False\n\n\nclass WebConnector(BaseConnector):\n def __init__(self, api_key: str, base_url: str):\n self.api_key = api_key\n self.base_url = base_url\n self.client: Optional[AsyncOpenAI] = None\n self._connected = False\n \n async def connect(self) -> None:\n if self._connected:\n return\n \n if not OPENAI_AVAILABLE:\n raise RuntimeError(\n \"OpenAI library not available. Install with: pip install openai\"\n )\n \n if not self.api_key:\n raise RuntimeError(\n \"API key not provided. Set OPENROUTER_API_KEY environment variable \"\n \"or provide deep_research_api_key in config.\"\n )\n \n self.client = AsyncOpenAI(\n base_url=self.base_url,\n api_key=self.api_key\n )\n self._connected = True\n logger.info(f\"Web connector connected to {self.base_url}\")\n \n async def disconnect(self) -> None:\n if not self._connected:\n return\n \n self.client = None\n self._connected = False\n logger.info(\"Web connector disconnected\")\n \n @property\n def is_connected(self) -> bool:\n return self._connected\n \n async def invoke(self, name: str, params: dict) -> Any:\n if name == \"chat_completion\":\n if not self.client:\n raise RuntimeError(\"Client not connected\")\n return await self.client.chat.completions.create(**params)\n raise NotImplementedError(f\"Unknown method: {name}\")\n \n async def request(self, *args: Any, **kwargs: Any) -> Any:\n raise NotImplementedError(\"Web backend uses invoke() instead of request()\")\n\n\nclass WebSession(BaseSession):\n \n backend_type = BackendType.WEB\n \n def __init__(\n self,\n *,\n session_id: str,\n config: SessionConfig,\n deep_research_api_key: Optional[str] = None,\n deep_research_base_url: str = \"https://openrouter.ai/api/v1\",\n auto_connect: bool = True,\n auto_initialize: bool = True\n ):\n api_key = deep_research_api_key or os.getenv(\"OPENROUTER_API_KEY\")\n connector = WebConnector(\n api_key=api_key or \"\", # Empty string will raise an error when connect\n base_url=deep_research_base_url\n )\n \n super().__init__(\n connector=connector,\n session_id=session_id,\n backend_type=BackendType.WEB,\n auto_connect=auto_connect,\n auto_initialize=auto_initialize\n )\n self.config = config\n \n @property\n def web_connector(self) -> WebConnector:\n return self.connector\n \n async def initialize(self) -> Dict[str, Any]:\n \"\"\"Connect to WebConnector and register tools.\n\n BaseSession in __aenter__ will call connect() according to auto_connect,\n but in provider.create_session directly instantiating Session will not trigger this logic.\n Therefore, we need to explicitly ensure that the connection is established, avoiding AttributeError\n when DeepResearchTool is called and `self.web_connector.client` is still None.\n \"\"\"\n\n # If the connection is not established, connect explicitly\n if not self.is_connected:\n try:\n await self.connect()\n except Exception as e:\n logger.error(f\"Failed to connect WebSession {self.session_id}: {e}\")\n raise\n\n if self.tools:\n logger.debug(f\"Web session {self.session_id} already initialized, skipping\")\n return {\n \"tools\": [t.name for t in self.tools],\n \"backend\": BackendType.WEB.value\n }\n\n self.tools = [DeepResearchTool(session=self)]\n \n logger.info(f\"Initialized Web session {self.session_id} with AI Deep Research tool\")\n \n return {\n \"tools\": [t.name for t in self.tools],\n \"backend\": BackendType.WEB.value\n }\n\n\nclass DeepResearchTool(BaseTool):\n \n backend_type = BackendType.WEB\n _name = \"deep_research_agent\"\n _description = \"\"\"Knowledge Research Tool - Primary tool for acquiring external knowledge\n\nPURPOSE:\nAcquires comprehensive knowledge from the web through deep research and analysis.\nPowered by Perplexity AI's sonar-deep-research model, then post-processed to extract\nactionable insights and concise summaries. The main tool for gathering information\nbeyond existing knowledge base.\n\nWHEN TO USE:\n- Information needed on professional/technical topics\n- Research on technical problems, concepts, or implementations \n- Understanding of latest developments, trends, or news\n- Comparison of different approaches, tools, or solutions\n- Factual information, definitions, or explanations required\n- Synthesis from multiple authoritative sources needed\n\nHOW IT WORKS:\n1. Conducts deep web search using Perplexity's sonar-deep-research\n2. Analyzes and synthesizes information from multiple sources\n3. Post-processes to distill knowledge-dense summary retaining critical details\n4. Returns comprehensive summary ready for immediate use\n\nRETURNS:\nKnowledge-dense comprehensive summary (400-600 words) that:\n- Retains important details and technical specifics\n- Focuses on substantive knowledge without losing critical information\n- Organized and structured for clarity\n- Directly usable by agents for decision-making and task execution\n\nNOT DESIGNED FOR:\n- Tasks requiring browser interaction or UI manipulation\n- Direct file downloads or web scraping operations\n- Real-time system operations or executions\n\nUSAGE GUIDELINES:\n- Frame clear, specific questions (e.g., \"Explain the architecture of Transformer models\")\n- Specify context when needed (e.g., \"Compare PostgreSQL vs MySQL for high-concurrency scenarios\")\n- Suitable for any knowledge or information acquisition needs\n\"\"\"\n \n def __init__(\n self,\n session: WebSession\n ):\n super().__init__()\n self._session = session\n self._llm = LLMClient()\n \n async def _arun(self, query: str) -> str:\n if not query:\n return \"ERROR: Missing required parameter: query\"\n \n try:\n # Step 1: Deep research\n logger.info(f\"Start deep research: {query}\")\n \n completion = await self._session.web_connector.client.chat.completions.create(\n model=\"perplexity/sonar-deep-research\",\n messages=[{\"role\": \"user\", \"content\": query}]\n )\n \n full_answer = completion.choices[0].message.content\n logger.info(f\"Research completed, length: {len(full_answer)} characters\")\n \n # Step 2: Use LLMClient to generate summary and distill key points\n logger.info(f\"Begin to distill key points...\")\n \n SUMMARY_AGENT_PROMPT = f\"\"\"Please distill the following deep research results into a knowledge-dense summary. Requirements:\n\nProvide a comprehensive yet concise summary (400-600 words):\n- Focus on SUBSTANTIVE knowledge and key information\n- Retain important details, technical specifics, and concrete facts\n- Do NOT sacrifice critical information for brevity\n- Organize information clearly and logically with proper structure\n- Remove only redundancy and verbose explanations\n- Include actionable insights and decision-relevant information\n- Make it directly usable for task execution and decision-making\n\nOutput ONLY the summary text, no additional formatting or JSON structure needed.\n\nDeep Research Results:\n{full_answer}\n\"\"\"\n \n summary_response = await self._llm.complete(SUMMARY_AGENT_PROMPT)\n summary = summary_response[\"message\"][\"content\"].strip()\n \n logger.info(f\"Summary generation completed\")\n \n return summary\n \n except Exception as e:\n logger.error(f\"Deep research failed: {e}\")\n return f\"ERROR: AI research failed: {e}\""
},
{
"path": "anytool/grounding/core/exceptions.py",
"content": "\"\"\"\nUnified exception & error-code definitions for the grounding framework\n\"\"\"\nfrom enum import Enum, auto\nfrom typing import Any, Dict\n\n\nclass ErrorCode(str, Enum):\n # generic\n UNKNOWN = auto()\n CONFIG_INVALID = auto()\n\n # provider / session / connector\n PROVIDER_ERROR = auto()\n SESSION_NOT_FOUND = auto()\n\n # connection\n CONNECTION_FAILED = auto()\n CONNECTION_TIMEOUT = auto()\n\n # tool\n TOOL_NOT_FOUND = auto()\n TOOL_EXECUTION_FAIL = auto()\n AMBIGUOUS_TOOL = auto()\n\n\nclass GroundingError(Exception):\n \"\"\"\n Framework-wide base exception.\n\n Parameters\n ----------\n message : str\n Human readable error message.\n code : ErrorCode\n One of the error codes defined above.\n retryable : bool\n Whether the caller may retry the operation automatically.\n context : Dict[str, Any]\n Extra key-value pairs (e.g. tool_name, session_id) for logging / metrics.\n \"\"\"\n\n __slots__ = (\"message\", \"code\", \"retryable\", \"context\")\n\n def __init__(\n self,\n message: str,\n *,\n code: ErrorCode = ErrorCode.UNKNOWN,\n retryable: bool = False,\n **context: Any,\n ):\n super().__init__(f\"[{code}] {message}\")\n self.message: str = message\n self.code: ErrorCode = code\n self.retryable: bool = retryable\n self.context: Dict[str, Any] = context\n\n def to_dict(self) -> Dict[str, Any]:\n \"\"\"Serialize error for structured logging / JSON response.\"\"\"\n return {\n \"code\": self.code.value,\n \"message\": self.message,\n \"retryable\": self.retryable,\n \"context\": self.context,\n }\n\n def __str__(self) -> str: \n return f\"[{self.code}] {self.message}\"\n\n def __repr__(self) -> str: \n return f\"GroundingError(code={self.code}, msg={self.message!r})\""
},
{
"path": "anytool/grounding/core/grounding_client.py",
"content": "import asyncio\nimport time\nfrom collections import OrderedDict\nfrom datetime import datetime\nfrom typing import Any, Dict, List, Optional\n\nfrom .types import BackendType, SessionConfig, SessionInfo, SessionStatus, ToolResult\nfrom .exceptions import ErrorCode, GroundingError\nfrom .tool import BaseTool\nfrom .provider import Provider, ProviderRegistry\nfrom .session import BaseSession\nfrom .search_tools import SearchCoordinator\nfrom anytool.config import GroundingConfig, get_config\nfrom anytool.config.utils import get_config_value\nfrom anytool.utils.logging import Logger\nimport importlib\n\n\nclass GroundingClient:\n \"\"\"\n Global Entry, Facing Agent/Application, only concerned with Provider & Session\n \"\"\"\n def __init__(self, config: Optional[GroundingConfig] = None, recording_manager=None) -> None:\n # Initialize logger first (needed by other initialization steps)\n self._logger = Logger.get_logger(__name__)\n \n self._config: GroundingConfig = config or get_config()\n self._registry: ProviderRegistry = ProviderRegistry()\n \n # Register providers from config\n self._register_providers_from_config()\n\n # Session\n self._sessions: Dict[str, BaseSession] = {}\n self._session_info: Dict[str, SessionInfo] = {}\n self._server_session_map: dict[tuple[BackendType, str], str] = {} # (backend, server) -> session_name\n\n # Tool cache\n self._tool_cache: \"OrderedDict[str, tuple[List[BaseTool], float]]\" = OrderedDict()\n self._tool_cache_ttl: int = get_config_value(self._config, \"tool_cache_ttl\", 300)\n self._tool_cache_maxsize: int = get_config_value(self._config, \"tool_cache_maxsize\", 300)\n\n # Concurrent control\n self._lock = asyncio.Lock()\n self._cache_lock = asyncio.Lock()\n\n # Tool search coordinator\n self._search_coordinator: Optional[SearchCoordinator] = None\n \n # Recording manager (optional, for GUI intermediate step recording)\n self._recording_manager = recording_manager\n \n # Tool quality manager\n self._quality_manager = self._init_quality_manager()\n \n # Register SystemProvider (requires GroundingClient instance, so must be done after __init__)\n self._register_system_provider()\n \n def _register_providers_from_config(self) -> None:\n \"\"\"\n Based on GroundingConfig.enabled_backends, register Provider instances to\n self._registry. Here only do *instantiation*, not await initialize(),\n to avoid blocking the event loop in the import stage; Provider will be lazily initialized when it is first used.\n \n Note: SystemProvider is skipped here and registered separately in _register_system_provider()\n because it requires a GroundingClient instance.\n \"\"\"\n if not self._config.enabled_backends:\n self._logger.warning(\"No enabled_backends defined in config\")\n return\n\n for item in self._config.enabled_backends:\n be_name: str | None = item.get(\"name\")\n cls_path: str | None = item.get(\"provider_cls\")\n if not (be_name and cls_path):\n self._logger.warning(\"Invalid backend entry: %s\", item)\n continue\n\n backend = BackendType(be_name.lower())\n \n # Skip system backend - it will be registered separately\n if backend == BackendType.SYSTEM:\n self._logger.debug(\"Skipping system backend in config registration (will be registered separately)\")\n continue\n \n if backend in self._registry.list():\n continue # Already registered\n\n # Dynamically import Provider class\n try:\n module_path, _, cls_name = cls_path.rpartition(\".\")\n module = importlib.import_module(module_path)\n prov_cls = getattr(module, cls_name)\n except (ModuleNotFoundError, AttributeError) as e:\n self._logger.error(\"Import provider failed: %s (%s)\", cls_path, e)\n continue\n\n backend_cfg = self._config.get_backend_config(be_name)\n provider: Provider = prov_cls(backend_cfg)\n self._registry.register(provider)\n \n def _register_system_provider(self) -> None:\n \"\"\"\n Register SystemProvider separately because it requires GroundingClient instance.\n SystemProvider provides meta-level tools for querying system state (list providers, tools, etc.)\n and is always available regardless of configuration.\n \"\"\"\n try:\n from .system import SystemProvider\n system_provider = SystemProvider(self)\n self._registry.register(system_provider)\n self._logger.debug(\"SystemProvider registered successfully\")\n except Exception as e:\n self._logger.warning(f\"Failed to register SystemProvider: {e}\")\n \n def _init_quality_manager(self):\n \"\"\"Initialize tool quality manager based on config.\"\"\"\n try:\n # Check if quality tracking is enabled in config\n quality_config = getattr(self._config, 'tool_quality', None)\n if not quality_config or not getattr(quality_config, 'enabled', True):\n self._logger.debug(\"Tool quality tracking disabled\")\n return None\n \n from .quality import ToolQualityManager, set_quality_manager\n from pathlib import Path\n \n cache_dir = getattr(quality_config, 'cache_dir', None)\n if cache_dir:\n cache_dir = Path(cache_dir)\n \n manager = ToolQualityManager(\n cache_dir=cache_dir,\n enable_persistence=getattr(quality_config, 'enable_persistence', True),\n auto_save=True,\n evolve_interval=getattr(quality_config, 'evolve_interval', 5),\n )\n \n # Set as global manager for BaseTool access\n set_quality_manager(manager)\n \n self._logger.info(\n f\"ToolQualityManager initialized \"\n f\"(records={len(manager._records)})\"\n )\n return manager\n \n except Exception as e:\n self._logger.warning(f\"Failed to initialize ToolQualityManager: {e}\")\n return None\n \n @property\n def quality_manager(self):\n \"\"\"Get the tool quality manager.\"\"\"\n return self._quality_manager\n \n # Quality API for Upper Layer\n def get_quality_report(self) -> Dict[str, Any]:\n \"\"\"\n Get comprehensive tool quality report.\n \"\"\"\n if not self._quality_manager:\n return {\"status\": \"disabled\", \"message\": \"Quality tracking not enabled\"}\n return self._quality_manager.get_quality_report()\n \n async def evolve_quality(self) -> Dict[str, Any]:\n \"\"\"\n Run quality self-evolution cycle.\n \n This triggers:\n - Tool change detection\n - Description re-evaluation for updated tools\n - Adaptive quality weight computation\n \n Call this periodically or after tool set changes.\n \"\"\"\n if not self._quality_manager:\n return {\"status\": \"disabled\"}\n \n # Get all tools\n all_tools = await self.list_tools()\n return await self._quality_manager.evolve(all_tools)\n \n def get_tool_insights(self, tool: BaseTool) -> Dict[str, Any]:\n \"\"\"\n Get detailed quality insights for a specific tool.\n \"\"\"\n if not self._quality_manager:\n return {\"status\": \"disabled\"}\n return self._quality_manager.get_tool_insights(tool)\n\n def register_provider(self, provider: Provider) -> None:\n self._registry.register(provider)\n \n def get_provider(self, backend: BackendType) -> Provider:\n return self._registry.get(backend)\n\n def list_providers(self) -> Dict[BackendType, Provider]:\n return self._registry.list()\n \n @property\n def recording_manager(self):\n \"\"\"Get the recording manager.\"\"\"\n return self._recording_manager\n \n @recording_manager.setter\n def recording_manager(self, manager):\n \"\"\"\n Set or update the recording manager.\n This allows coordinator to inject recording_manager after GroundingClient creation.\n \"\"\"\n self._recording_manager = manager\n self._logger.info(\"GroundingClient: RecordingManager updated\")\n \n async def initialize_all_providers(self) -> None:\n await asyncio.gather(*[provider.initialize() for provider in self._registry.list().values() if not provider.is_initialized])\n\n\n async def create_session(\n self,\n *,\n backend: BackendType,\n name: str | None = None,\n connection_params: Dict[str, Any] | None = None,\n server: str | None = None,\n **options,\n ) -> str:\n \"\"\"\n Create and initialize Session, return \"session_name\" (external visible)\n name is auto generated when it's None: -\n MCP backend needs to provide server\n \"\"\"\n async with self._lock:\n # Check concurrent sessions limit\n max_sessions = get_config_value(self._config, \"max_concurrent_sessions\", 100)\n if len(self._sessions) >= max_sessions:\n raise GroundingError(f\"Reached maximum session limit: {max_sessions}\")\n\n # Session naming strategy\n if server: # Only MCP will pass in server\n name = name or f\"{backend.value}-{server}\"\n else:\n name = name or backend.value # Other backends have a fixed 1 session\n \n if name in self._sessions:\n # Reuse existing session\n self._logger.warning(\"Session '%s' exists, reusing.\", name)\n return name\n\n # Get Provider (initialize if first time)\n provider = self._registry.get(backend)\n if not provider.is_initialized:\n await provider.initialize()\n \n if backend == BackendType.MCP:\n if server is None:\n raise GroundingError(\"Must specify 'server' when creating MCP session\")\n\n # Construct SessionConfig, pass to Provider to create\n connection_params = connection_params or {}\n if server:\n connection_params.setdefault(\"server\", server)\n \n # Inject recording_manager for GUI backend (for intermediate step recording)\n if backend == BackendType.GUI and self._recording_manager is not None:\n connection_params.setdefault(\"recording_manager\", self._recording_manager)\n\n sess_cfg = SessionConfig(\n session_name=name, # Use external visible name\n backend_type=backend,\n connection_params=connection_params,\n **options,\n )\n session_obj = await provider.create_session(sess_cfg)\n\n # Store session and monitoring info\n async with self._lock:\n self._sessions[name] = session_obj\n now = datetime.utcnow()\n self._session_info[name] = SessionInfo(\n session_name=name,\n backend_type=backend,\n status=SessionStatus.CONNECTED,\n created_at=now,\n last_activity=now,\n )\n if server:\n self._server_session_map[(backend, server)] = name\n\n self._logger.info(\"Session created: %s\", name)\n return name\n \n def list_sessions(self) -> List[str]:\n return list(self._sessions.keys())\n\n async def close_session(self, name: str) -> None:\n async with self._lock:\n session = self._sessions.pop(name, None)\n info = self._session_info.pop(name, None)\n self._tool_cache.pop(name, None)\n\n for k, v in list(self._server_session_map.items()):\n if v == name:\n self._server_session_map.pop(k)\n\n if not session:\n self._logger.warning(\"Session '%s' not found\", name)\n return\n\n try:\n provider = self._registry.get(info.backend_type) if info else None\n if provider:\n await provider.close_session(name)\n else:\n # Fallback: if no provider, disconnect directly\n await session.disconnect()\n finally:\n self._logger.info(\"Session closed: %s\", name)\n\n async def close_all_sessions(self) -> None:\n for sid in list(self._sessions.keys()):\n await self.close_session(sid)\n \n async def ensure_session(self, backend: BackendType, server: str | None = None) -> str:\n sid = backend.value if server is None else f\"{backend.value}-{server}\"\n if sid not in self._sessions:\n await self.create_session(backend=backend, name=sid, server=server)\n return sid\n \n def get_session_info(self, name: str) -> SessionInfo:\n \"\"\"Get session monitoring info\"\"\"\n if name not in self._session_info:\n raise ErrorCode.SESSION_NOT_FOUND(name)\n return self._session_info[name]\n \n def get_session(self, name: str) -> BaseSession:\n \"\"\"Get session\"\"\"\n if name not in self._sessions:\n raise ErrorCode.SESSION_NOT_FOUND(name)\n return self._sessions[name]\n \n \n async def _fetch_tools(\n self,\n backend: BackendType,\n *,\n session_name: str | None = None,\n use_cache: bool = False,\n bind_runtime_info: bool = True, \n ) -> List[BaseTool]:\n \"\"\"\n Fetch tools from provider.\n \n Args:\n backend: Backend type\n session_name: \n - None: fetch all tools from all sessions of this backend\n - str: fetch tools from specific session\n use_cache: Whether to use cache\n bind_runtime_info: Whether to bind runtime info to tool instances\n \"\"\"\n now = time.time()\n \n # Auto-generate cache_scope from parameters\n if session_name:\n cache_scope = session_name\n else:\n cache_scope = f\"backend-{backend.value}\"\n\n # Check cache\n if use_cache:\n async with self._cache_lock:\n if cache_scope in self._tool_cache:\n tools, ts = self._tool_cache[cache_scope]\n if now - ts < self._tool_cache_ttl:\n self._tool_cache.move_to_end(cache_scope)\n return tools\n\n provider = self._registry.get(backend)\n if not provider.is_initialized:\n await provider.initialize()\n\n tools = await provider.list_tools(session_name=session_name)\n\n if bind_runtime_info:\n # If session_name is specified, bind all tools to that session\n if session_name:\n server_name = None\n if backend == BackendType.MCP:\n server_name = session_name.replace(f\"{backend.value}-\", \"\", 1)\n \n for tool in tools:\n tool.bind_runtime_info(\n backend=backend,\n session_name=session_name,\n server_name=server_name,\n grounding_client=self,\n )\n else:\n # No session_name specified - get tools from all sessions\n # For each backend, find the default/primary session\n # For Shell/Web/GUI: use the default session (backend.value)\n # For MCP: tools should already be bound by the provider\n default_session_name = None\n \n # Try to find an existing session for this backend\n for sid, info in self._session_info.items():\n if info.backend_type == backend:\n default_session_name = sid\n break\n \n # Fallback: use backend default naming\n if not default_session_name:\n default_session_name = backend.value\n \n server_name = None\n if backend == BackendType.MCP and default_session_name:\n server_name = default_session_name.replace(f\"{backend.value}-\", \"\", 1)\n \n for tool in tools:\n # Only bind if tool doesn't have runtime info already\n # (some providers like MCP bind runtime info during list_tools)\n if not tool.is_bound:\n tool.bind_runtime_info(\n backend=backend,\n session_name=default_session_name,\n server_name=server_name,\n grounding_client=self,\n )\n elif not tool.runtime_info.grounding_client:\n # Tool has runtime info but no grounding_client, add it\n tool.bind_runtime_info(\n backend=tool.runtime_info.backend,\n session_name=tool.runtime_info.session_name,\n server_name=tool.runtime_info.server_name,\n grounding_client=self,\n )\n\n # Save to cache\n if use_cache:\n async with self._cache_lock:\n self._tool_cache[cache_scope] = (tools, now)\n self._tool_cache.move_to_end(cache_scope)\n while len(self._tool_cache) > self._tool_cache_maxsize:\n self._tool_cache.popitem(last=False)\n\n return tools\n\n async def list_tools(\n self,\n backend: BackendType | list[BackendType] | None = None,\n session_name: str | None = None,\n *,\n use_cache: bool = False,\n ) -> List[BaseTool]:\n \"\"\"\n List tools from backend(s) or session.\n \n 1. session_name is provided โ return tools from that session\n 2. backend is list โ return tools from multiple backends\n 3. backend is single โ return tools from that backend\n 4. backend is None โ return tools from all backends\n \n Args:\n backend: Single backend, list of backends, or None for all\n session_name: Specific session name (overrides backend parameter)\n use_cache: Whether to use cache\n \n Returns:\n List of tools\n \"\"\"\n # Session-level\n if session_name: \n if session_name not in self._sessions:\n raise ErrorCode.SESSION_NOT_FOUND(session_name)\n backend_type = self._session_info[session_name].backend_type\n return await self._fetch_tools(\n backend_type,\n session_name=session_name,\n use_cache=use_cache,\n )\n \n # Multiple backends\n if isinstance(backend, list):\n tools: List[BaseTool] = []\n for be in backend:\n backend_tools = await self._fetch_tools(\n be,\n session_name=None, # Provider aggregates all sessions\n use_cache=use_cache,\n )\n tools.extend(backend_tools)\n return tools\n \n # Single backend\n if backend is not None:\n return await self._fetch_tools(\n backend,\n session_name=None,\n use_cache=use_cache,\n )\n\n # All backends\n tools: List[BaseTool] = []\n for backend_type in self._registry.list().keys():\n backend_tools = await self._fetch_tools(\n backend_type,\n session_name=None,\n use_cache=use_cache,\n )\n tools.extend(backend_tools)\n return tools\n\n async def list_backend_tools(\n self, \n backend: BackendType | list[BackendType] | None = None,\n use_cache: bool = False\n ) -> list[BaseTool]:\n return await self.list_tools(backend=backend, session_name=None, use_cache=use_cache)\n\n async def list_session_tools(\n self, \n session_name: str, \n use_cache: bool = False\n ) -> list[BaseTool]:\n if session_name not in self._session_info:\n raise ErrorCode.SESSION_NOT_FOUND(session_name)\n backend = self._session_info[session_name].backend_type\n return await self.list_tools(backend, session_name, use_cache)\n\n async def list_all_backend_tools(\n self,\n use_cache: bool = False\n ) -> Dict[BackendType, list[BaseTool]]:\n \"\"\"List static tools for every registered backend.\"\"\"\n result = {}\n for backend_type in self.list_providers().keys():\n tools = await self.list_backend_tools(backend=backend_type, use_cache=use_cache)\n result[backend_type] = tools\n return result\n\n async def search_tools(\n self,\n task_description: str,\n *,\n backend: BackendType | list[BackendType] | None = None,\n session_name: str | None = None,\n max_tools: int | None = None,\n search_mode: str | None = None,\n use_cache: bool = True,\n llm_callable = None,\n enable_llm_filter: bool | None = None,\n llm_filter_threshold: int | None = None,\n enable_cache_persistence: bool | None = None,\n cache_dir: str | None = None,\n ) -> list[BaseTool]:\n \"\"\"\n Search tools from backend(s) or session.\n \n Args:\n task_description: Task description for searching relevant tools\n backend: Backend type(s) to search\n session_name: Specific session to search\n max_tools: Maximum number of tools to return\n search_mode: Search mode (\"semantic\", \"keyword\", \"hybrid\")\n use_cache: Whether to use cached tool list\n llm_callable: LLM client for intelligent filtering\n enable_llm_filter: Whether to use LLM pre-filtering\n llm_filter_threshold: Threshold for applying LLM filter\n enable_cache_persistence: Whether to persist embeddings to disk. If None, uses config value.\n cache_dir: Directory for persistent cache. If None, uses config value or default.\n \"\"\"\n candidate_tools = await self.list_tools(\n backend=backend,\n session_name=session_name,\n use_cache=use_cache,\n )\n \n if not candidate_tools:\n self._logger.warning(\"No candidate tools found for search\")\n return []\n \n # lazy initialize SearchCoordinator (or recreate if parameters changed)\n if self._search_coordinator is None:\n # Get quality ranking settings from config\n quality_config = getattr(self._config, 'tool_quality', None)\n enable_quality_ranking = getattr(quality_config, 'enable_quality_ranking', True) if quality_config else True\n \n self._search_coordinator = SearchCoordinator(\n max_tools=max_tools,\n llm=llm_callable,\n enable_llm_filter=enable_llm_filter,\n llm_filter_threshold=llm_filter_threshold,\n enable_cache_persistence=enable_cache_persistence,\n cache_dir=cache_dir,\n quality_manager=self._quality_manager,\n enable_quality_ranking=enable_quality_ranking,\n )\n \n # execute search and sort\n try:\n filtered_tools = await self._search_coordinator._arun(\n task_prompt=task_description,\n candidate_tools=candidate_tools,\n max_tools=max_tools,\n mode=search_mode,\n )\n return filtered_tools\n except Exception as exc:\n self._logger.error(f\"Tool search failed: {exc}\")\n # fallback: return top N tools\n fallback_max = max_tools or self._config.tool_search.max_tools\n return candidate_tools[:fallback_max]\n \n def get_last_search_debug_info(self) -> Optional[Dict[str, Any]]:\n \"\"\"Get debug info from the last tool search operation.\n \n Returns:\n Dict containing search debug info, or None if no search has been performed.\n \"\"\"\n if self._search_coordinator is None:\n return None\n return self._search_coordinator.get_last_search_debug_info()\n \n async def get_tools_with_auto_search(\n self,\n *,\n task_description: str | None = None,\n backend: BackendType | list[BackendType] | None = None,\n session_name: str | None = None,\n max_tools: int | None = None,\n search_mode: str | None = None,\n use_cache: bool = True,\n llm_callable = None,\n enable_llm_filter: bool | None = None,\n llm_filter_threshold: int | None = None,\n enable_cache_persistence: bool | None = None,\n cache_dir: str | None = None,\n ) -> list[BaseTool]:\n \"\"\"\n Intelligent tool retrieval: automatically decides whether to return all tools or trigger search.\n \n Logic:\n - If tool_count <= max_tools: return all tools directly\n - If tool_count > max_tools: trigger search and return top max_tools\n \n Args:\n task_description: Task description (required for search if triggered). \n If None, search will not be triggered even if tool count exceeds max_tools.\n backend: Backend type(s) to query\n session_name: Specific session name\n max_tools: Maximum number of tools to return. Also acts as the threshold for triggering search.\n - None: Use value from config (default: 30)\n search_mode: Search mode (\"semantic\", \"keyword\", \"hybrid\")\n use_cache: Whether to use cache\n llm_callable: LLM client (for intelligent filtering)\n enable_llm_filter: Whether to use LLM for backend/server pre-filtering.\n - None: Use config default\n - False: Disable LLM filter, use tool-level search only\n - True: Enable LLM filter\n llm_filter_threshold: Only apply LLM filter when tool count > this threshold.\n - None: Use default (50)\n - N: Only apply LLM filter when > N tools\n enable_cache_persistence: Whether to persist embeddings to disk. If None, uses config value.\n cache_dir: Directory for persistent cache. If None, uses config value or default.\n \n Returns:\n List of tools (at most max_tools)\n \n Examples:\n # Scenario 1: Auto-detect whether search is needed\n tools = await gc.get_tools_with_auto_search(\n task_description=\"Create a flowchart\",\n backend=BackendType.MCP\n )\n \n # Scenario 2: Custom max_tools\n tools = await gc.get_tools_with_auto_search(\n task_description=\"Edit file\",\n backend=BackendType.SHELL,\n max_tools=30 # Return at most 30 tools\n )\n \n # Scenario 3: Disable search (return all tools regardless of count)\n tools = await gc.get_tools_with_auto_search(\n backend=BackendType.MCP # No task_description = no search\n )\n \"\"\"\n # Fetch all candidate tools\n all_tools = await self.list_tools(\n backend=backend,\n session_name=session_name,\n use_cache=use_cache,\n )\n \n if not all_tools:\n self._logger.warning(\"No tools found\")\n return []\n \n # Determine max_tools from config if not provided\n if max_tools is None:\n max_tools = self._config.tool_search.max_tools\n \n # Decide whether search is needed\n tools_count = len(all_tools)\n need_search = tools_count > max_tools and task_description is not None\n \n if need_search:\n self._logger.info(\n f\"Tool count ({tools_count}) > max_tools ({max_tools}), \"\n f\"triggering search to filter relevant tools...\"\n )\n return await self.search_tools(\n task_description=task_description,\n backend=backend,\n session_name=session_name,\n max_tools=max_tools,\n search_mode=search_mode,\n use_cache=use_cache,\n llm_callable=llm_callable,\n enable_llm_filter=enable_llm_filter,\n llm_filter_threshold=llm_filter_threshold,\n enable_cache_persistence=enable_cache_persistence,\n cache_dir=cache_dir,\n )\n else:\n if task_description is None:\n self._logger.debug(\n f\"No task description provided, returning all {tools_count} tools\"\n )\n else:\n self._logger.debug(\n f\"Tool count ({tools_count}) โค max_tools ({max_tools}), \"\n f\"returning all tools without search\"\n )\n return all_tools\n\n async def invoke_tool(\n self,\n tool: BaseTool | str,\n parameters: Dict[str, Any] | None = None,\n *,\n backend: BackendType | None = None,\n session_name: str | None = None,\n server: str | None = None,\n keep_session: bool = False,\n **kwargs\n ) -> ToolResult:\n \"\"\"\n Universal tool invocation method.\n Supports multiple calling patterns:\n \n 1. Using BaseTool instance with bound runtime info\n 2. Using BaseTool instance with explicit backend/session\n 3. Using tool name with automatic lookup\n 4. Using tool name with explicit backend/session/server\n \n Args:\n tool: BaseTool instance or tool name string\n parameters: Tool parameters as dict\n backend: Backend type (optional for BaseTool with runtime_info)\n session_name: Session name (optional for BaseTool with runtime_info)\n server: Server name (for MCP, optional for BaseTool with runtime_info)\n keep_session: Whether to keep session alive after invocation\n **kwargs: Alternative parameter passing\n \n Returns:\n ToolResult\n \n Examples:\n # Pattern 1: Tool instance with runtime info (from list_tools)\n tools = await gc.list_tools()\n tool = next(t for t in tools if t.name == \"read_file\")\n result = await gc.invoke_tool(tool, {\"path\": \"/tmp/a.txt\"})\n \n # Pattern 2: Tool instance with explicit backend/session\n my_tool = MyTool()\n result = await gc.invoke_tool(\n my_tool, \n {\"arg\": \"value\"}, \n backend=BackendType.SHELL\n )\n \n # Pattern 3: Tool name with automatic lookup\n result = await gc.invoke_tool(\"read_file\", {\"path\": \"/tmp/a.txt\"})\n \n # Pattern 4: Tool name with explicit backend/server\n result = await gc.invoke_tool(\n \"read_file\",\n {\"path\": \"/tmp/a.txt\"},\n backend=BackendType.MCP,\n server=\"filesystem\"\n )\n \"\"\"\n params = parameters or kwargs\n \n # BaseTool instance\n if isinstance(tool, BaseTool):\n tool_name = tool.schema.name\n \n # Try to use bound runtime info first\n if tool.is_bound and not (backend or session_name or server):\n # Use runtime info\n runtime_backend = tool.runtime_info.backend\n runtime_session = tool.runtime_info.session_name\n runtime_server = tool.runtime_info.server_name\n else:\n # Use provided or tool's default backend\n runtime_backend = backend or tool.backend_type\n runtime_session = session_name\n runtime_server = server\n \n if runtime_backend == BackendType.NOT_SET:\n raise GroundingError(\n f\"Cannot invoke tool '{tool_name}': no backend specified. \"\n f\"Either bind runtime info or provide backend parameter.\",\n code=ErrorCode.TOOL_EXECUTION_FAIL\n )\n \n # Tool name string\n elif isinstance(tool, str):\n tool_name = tool\n \n # If explicit backend/session provided, use them\n if backend or session_name:\n runtime_session = session_name\n runtime_server = server\n \n # Infer backend: prefer explicit backend; otherwise get from session\n if backend is not None:\n runtime_backend = backend\n else:\n if runtime_session not in self._session_info:\n raise ErrorCode.SESSION_NOT_FOUND(runtime_session)\n runtime_backend = self._session_info[\n runtime_session\n ].backend_type\n else:\n # Auto-lookup: search for the tool\n all_tools = await self.list_tools(use_cache=True)\n matching = [t for t in all_tools if t.name == tool_name]\n \n if not matching:\n raise GroundingError(\n f\"Tool '{tool_name}' not found\",\n code=ErrorCode.TOOL_NOT_FOUND\n )\n \n if len(matching) > 1:\n sources = [\n f\"{t.runtime_info.backend.value}/{t.runtime_info.session_name}\" \n for t in matching if t.is_bound\n ]\n raise GroundingError(\n f\"Multiple tools named '{tool_name}' found in: {sources}. \"\n f\"Please specify 'backend' or 'session_name' parameter.\",\n code=ErrorCode.AMBIGUOUS_TOOL\n )\n \n # Use the found tool's runtime info\n found_tool = matching[0]\n runtime_backend = found_tool.runtime_info.backend\n runtime_session = found_tool.runtime_info.session_name\n runtime_server = found_tool.runtime_info.server_name\n \n # Execute the tool\n # Ensure session exists (except for SYSTEM backend which doesn't use sessions)\n # Check if session really exists - cached tools have session_name but session may not be running\n if runtime_backend != BackendType.SYSTEM:\n if not runtime_session or runtime_session not in self._sessions:\n runtime_session = await self.ensure_session(runtime_backend, runtime_server)\n \n try:\n provider = self._registry.get(runtime_backend)\n # SystemProvider doesn't use sessions, pass a dummy value\n session_param = runtime_session if runtime_session else \"system\"\n result = await provider.call_tool(session_param, tool_name, params)\n \n # Update last_activity in session_info (skip for SYSTEM backend)\n if runtime_backend != BackendType.SYSTEM and runtime_session and runtime_session in self._session_info:\n async with self._lock:\n old_info = self._session_info[runtime_session]\n self._session_info[runtime_session] = old_info.model_copy(\n update={\"last_activity\": datetime.utcnow()}\n )\n \n return result\n finally:\n # Auto-close session if requested (skip for SYSTEM backend)\n if runtime_backend != BackendType.SYSTEM and not keep_session and runtime_session:\n if runtime_server or runtime_session.startswith(runtime_backend.value):\n await self.close_session(runtime_session)"
},
{
"path": "anytool/grounding/core/provider.py",
"content": "\"\"\"\nprovider is to manage sessions of a backend, if the backend is mcp, then provider will manage sessions through servers\n\"\"\"\nfrom abc import ABC, abstractmethod\nfrom typing import List, Dict, Any, Optional, Generic, TypeVar\n\nfrom .tool import BaseTool\nfrom .types import BackendType, SessionConfig, ToolResult, ToolStatus\nfrom .session import BaseSession\nfrom .security.policies import SecurityPolicyManager\nfrom anytool.config import get_config\nfrom anytool.utils.logging import Logger\n\nlogger = Logger.get_logger(__name__)\nTSession = TypeVar('TSession', bound=BaseSession)\n\n\nclass Provider(ABC, Generic[TSession]):\n \"\"\"Backend provider base class\"\"\" \n def __init__(self, backend_type: BackendType, config: Dict[str, Any] = None):\n self.backend_type = backend_type\n self.config = config or {}\n self.is_initialized = False\n self._sessions: Dict[str, TSession] = {} # session management\n self._session_counter: int = 0\n self.security_manager = SecurityPolicyManager()\n \n self._setup_security_policy(config)\n \n def _setup_security_policy(self, config: dict | None = None): \n security_policy = get_config().get_security_policy(self.backend_type.value)\n self.security_manager.set_backend_policy(BackendType.SHELL, security_policy)\n \n async def ensure_initialized(self) -> None:\n \"\"\"\n Internal helper. Guarantee that `initialize()` has been executed\n \"\"\"\n if not self.is_initialized:\n await self.initialize()\n \n @abstractmethod\n async def initialize(self) -> None:\n \"\"\"Initialize provider, call `create_session` to create all sessions if not exist \n Subclasses should set `self.is_initialized = True` after successful initialization\n \"\"\"\n pass\n \n @abstractmethod\n async def create_session(self, session_config: SessionConfig) -> TSession:\n \"\"\"Create session, update _sessions\"\"\"\n pass\n\n @abstractmethod\n async def close_session(self, session_name: str) -> None:\n \"\"\"Close session\"\"\"\n pass\n \n def list_sessions(self) -> List[str]:\n \"\"\"Get all session IDs\"\"\"\n return list(self._sessions.keys())\n \n def get_session(self, session_name: str) -> Optional[TSession]:\n \"\"\"Get session object by ID\"\"\"\n return self._sessions.get(session_name)\n \n async def close_all_sessions(self) -> None:\n \"\"\"Provider shutdown cleanup\"\"\"\n for session_name in list(self._sessions.keys()):\n try:\n await self.close_session(session_name)\n except Exception as e:\n print(f\"Error closing session {session_name}: {e}\")\n \n self._sessions.clear()\n self.is_initialized = False\n\n def __repr__(self) -> str:\n return (f\"Provider(backend={self.backend_type.value}, \"\n f\"initialized={self.is_initialized}, \"\n f\"sessions={len(self._sessions)}, \"\n f\"config_items={len(self.config)})\")\n \n async def list_tools(self, session_name: Optional[str] = None) -> List[BaseTool]:\n \"\"\"\n Return BaseTool list.\n If session_name is specified, only return the tools of the specified session. \n If session_name is not specified, return all tools of all sessions.\n \"\"\"\n await self.ensure_initialized()\n \n if session_name:\n session = self._sessions.get(session_name)\n return await session.list_tools() if session else []\n\n tools: list[BaseTool] = []\n for sess in self._sessions.values():\n tools.extend(await sess.list_tools())\n return tools\n \n async def call_tool(\n self,\n session_name: str,\n tool_name: str,\n parameters: Dict[str, Any] | None = None,\n ) -> ToolResult:\n \n await self.ensure_initialized()\n parameters = parameters or {}\n\n session = self._sessions.get(session_name)\n if session is None:\n return ToolResult(\n status=ToolStatus.ERROR,\n content=\"\",\n error=f\"Session '{session_name}' not found\",\n metadata={\"session_name\": session_name, \"tool_name\": tool_name},\n )\n\n try:\n return await session.call_tool(tool_name, parameters)\n except Exception as e:\n logger.error(\"Execute tool error: %s @%s - %s\", tool_name, session_name, e)\n return ToolResult(\n status=ToolStatus.ERROR,\n content=\"\",\n error=str(e),\n metadata={\"session_name\": session_name, \"tool_name\": tool_name},\n )\n\n\nclass ProviderRegistry:\n \"\"\"\n Maintain mapping of BackendType -> Provider, and provide dynamic registration / retrieval capabilities\n \"\"\"\n def __init__(self) -> None:\n self._providers: dict[BackendType, Provider] = {}\n\n def register(self, provider: \"Provider\") -> None:\n self._providers[provider.backend_type] = provider\n logger.debug(\"Provider for %s registered\", provider.backend_type)\n\n def get(self, backend: BackendType) -> \"Provider\":\n if backend not in self._providers: \n raise KeyError(f\"Provider for '{backend.value}' not registered\")\n return self._providers[backend]\n\n def list(self) -> dict[BackendType, \"Provider\"]:\n return dict(self._providers)"
},
{
"path": "anytool/grounding/core/quality/__init__.py",
"content": "from .types import ToolQualityRecord, ExecutionRecord, DescriptionQuality\nfrom .manager import ToolQualityManager\nfrom .store import QualityStore\n\n# Global manager instance\n_global_manager: \"ToolQualityManager | None\" = None\n\n\ndef get_quality_manager() -> \"ToolQualityManager | None\":\n \"\"\"Get the global quality manager instance.\"\"\"\n return _global_manager\n\n\ndef set_quality_manager(manager: \"ToolQualityManager\") -> None:\n \"\"\"Set the global quality manager instance.\"\"\"\n global _global_manager\n _global_manager = manager\n\n\n__all__ = [\n \"ToolQualityRecord\",\n \"ExecutionRecord\",\n \"DescriptionQuality\",\n \"ToolQualityManager\",\n \"QualityStore\",\n \"get_quality_manager\",\n \"set_quality_manager\",\n]\n"
},
{
"path": "anytool/grounding/core/quality/manager.py",
"content": "\"\"\"\nTool Quality Manager\n\nCore API (called by main flow):\n- record_execution(): Called by BaseTool after execution\n- adjust_ranking(): Called by SearchCoordinator for quality-aware sorting\n- evolve(): Called periodically by ToolLayer for self-evolution\n\nQuery API (for inspection/debugging):\n- get_quality_report(), get_tool_insights()\n\"\"\"\n\nimport hashlib\nfrom datetime import datetime\nfrom pathlib import Path\nfrom typing import Dict, List, Optional, Tuple, TYPE_CHECKING\n\nfrom .types import ToolQualityRecord, ExecutionRecord, DescriptionQuality\nfrom .store import QualityStore\nfrom anytool.utils.logging import Logger\nfrom anytool.config.constants import PROJECT_ROOT\n\nif TYPE_CHECKING:\n from anytool.grounding.core.tool import BaseTool\n from anytool.grounding.core.types import ToolResult\n from anytool.llm import LLMClient\n\nlogger = Logger.get_logger(__name__)\n\n\nclass ToolQualityManager:\n \"\"\"\n Manages tool quality tracking and quality-aware ranking.\n \n Features:\n - Track execution success rate and latency\n - LLM-based description quality evaluation (optional, requires llm_client)\n - Persistent memory across sessions\n - Quality-integrated tool ranking\n - Incremental update detection\n \"\"\"\n \n def __init__(\n self,\n *,\n cache_dir: Optional[Path] = None,\n llm_client: Optional[\"LLMClient\"] = None,\n enable_persistence: bool = True,\n auto_save: bool = True,\n evolve_interval: int = 5,\n ):\n self._cache_dir = cache_dir or PROJECT_ROOT / \".anytool\" / \"tool_quality\"\n self._llm_client = llm_client\n self._enable_persistence = enable_persistence\n self._auto_save = auto_save\n self._evolve_interval = evolve_interval\n \n # In-memory cache\n self._records: Dict[str, ToolQualityRecord] = {}\n self._global_execution_count: int = 0\n self._last_evolve_count: int = 0\n \n # Persistent store\n self._store = QualityStore(self._cache_dir) if enable_persistence else None\n \n # Load from disk\n if self._store:\n self._records, self._global_execution_count = self._store.load_all()\n self._last_evolve_count = (self._global_execution_count // self._evolve_interval) * self._evolve_interval\n \n logger.info(\n f\"ToolQualityManager initialized \"\n f\"(persistence={enable_persistence}, records={len(self._records)}, \"\n f\"global_count={self._global_execution_count}, evolve_interval={self._evolve_interval})\"\n )\n\n def get_tool_key(self, tool: \"BaseTool\") -> str:\n \"\"\"Generate unique key for a tool.\"\"\"\n from anytool.grounding.core.types import BackendType\n \n if tool.is_bound:\n backend = tool.runtime_info.backend.value\n server = tool.runtime_info.server_name or \"default\"\n else:\n backend = tool.backend_type.value if tool.backend_type != BackendType.NOT_SET else \"unknown\"\n server = \"default\"\n \n return f\"{backend}:{server}:{tool.name}\"\n \n def _compute_description_hash(self, tool: \"BaseTool\") -> str:\n \"\"\"Compute hash of tool description for change detection.\"\"\"\n content = f\"{tool.name}|{tool.description or ''}|{tool.schema.parameters}\"\n return hashlib.md5(content.encode()).hexdigest()[:16]\n\n def get_record(self, tool: \"BaseTool\") -> ToolQualityRecord:\n \"\"\"Get or create quality record for a tool.\"\"\"\n key = self.get_tool_key(tool)\n \n if key not in self._records:\n backend, server, name = key.split(\":\", 2)\n self._records[key] = ToolQualityRecord(\n tool_key=key,\n backend=backend,\n server=server,\n tool_name=name,\n description_hash=self._compute_description_hash(tool),\n )\n \n return self._records[key]\n \n def get_quality_score(self, tool: \"BaseTool\") -> float:\n \"\"\"Get quality score for a tool (0-1).\"\"\"\n return self.get_record(tool).quality_score\n \n # Execution Tracking\n async def record_execution(\n self,\n tool: \"BaseTool\",\n result: \"ToolResult\",\n execution_time_ms: float,\n ) -> None:\n \"\"\"Record tool execution result and increment global counter.\"\"\"\n record = self.get_record(tool)\n \n # Extract error message if failed\n error_message = None\n if result.is_error and result.error:\n error_message = str(result.error)[:500]\n \n # Add execution record\n record.add_execution(ExecutionRecord(\n timestamp=datetime.now(),\n success=result.is_success,\n execution_time_ms=execution_time_ms,\n error_message=error_message,\n ))\n \n # Increment global execution count\n self._global_execution_count += 1\n \n # Auto-save\n if self._auto_save and self._store:\n await self._store.save_record(record, self._records, self._global_execution_count)\n \n logger.debug(\n f\"Recorded execution: {record.tool_key} \"\n f\"success={result.is_success} time={execution_time_ms:.0f}ms \"\n f\"(global_count={self._global_execution_count})\"\n )\n \n async def evaluate_description(\n self,\n tool: \"BaseTool\",\n force: bool = False,\n ) -> Optional[DescriptionQuality]:\n \"\"\"\n Evaluate tool description quality using LLM.\n \"\"\"\n if not self._llm_client:\n logger.debug(\"LLM client not available for description evaluation\")\n return None\n \n record = self.get_record(tool)\n \n # Skip if already evaluated and not forced\n if record.description_quality and not force:\n # Check if description changed\n current_hash = self._compute_description_hash(tool)\n if current_hash == record.description_hash:\n return record.description_quality\n \n # Build evaluation prompt\n desc = tool.description or \"No description provided\"\n if len(desc) > 4000:\n desc = desc[:4000] + \"\\n... (truncated for length)\"\n \n params = tool.schema.parameters or {}\n if params:\n param_lines = []\n # Extract parameter names and types from JSON schema\n if \"properties\" in params:\n for param_name, param_info in params.get(\"properties\", {}).items():\n param_type = param_info.get(\"type\", \"unknown\")\n param_desc = param_info.get(\"description\", \"\")\n param_lines.append(f\"- {param_name} ({param_type}): {param_desc}\" if param_desc else f\"- {param_name} ({param_type})\")\n param_text = \"\\n\".join(param_lines) if param_lines else \"No parameter descriptions available\"\n else:\n param_text = \"No parameters\"\n \n prompt = f\"\"\"# Task: Evaluate this tool's documentation quality\n\n## Tool Information\n\nName: {tool.name}\n\nDescription:\n{desc}\n\nParameters:\n{param_text}\n\n## Evaluation Task\n\nRate the documentation on two dimensions (0.0 to 1.0 scale):\n\n### 1. Clarity\nHow clear is the tool's purpose and usage?\n\n- 0.0-0.3: No description or completely unclear\n- 0.4-0.6: Basic purpose but vague\n- 0.7-0.8: Clear purpose and functionality\n- 0.9-1.0: Very clear with usage examples or context\n\n### 2. Completeness\nAre inputs/outputs properly documented?\n\n- 0.0-0.3: Missing critical information\n- 0.4-0.6: Basic info but lacks details\n- 0.7-0.8: Well documented with types\n- 0.9-1.0: Comprehensive with constraints and examples\n\n## Scoring Guidelines\n\n- Short descriptions can score high if clear and accurate\n- If parameters exist but aren't explained in description, reduce completeness score\n- Missing description means clarity = 0.0\n\n## Output\n\nRespond with JSON only:\n\n```json\n{{\n \"reasoning\": \"Brief 1-2 sentence analysis\",\n \"clarity\": 0.8,\n \"completeness\": 0.7\n}}\n```\"\"\"\n\n try:\n response = await self._llm_client.complete(prompt)\n content = response[\"message\"][\"content\"]\n \n # Parse JSON response\n import json\n \n # Extract complete JSON object\n def extract_json_object(text: str) -> str | None:\n \"\"\"Extract first complete JSON object from text by counting braces.\"\"\"\n start = text.find('{')\n if start == -1:\n return None\n \n count = 0\n in_string = False\n escape_next = False\n \n for i, char in enumerate(text[start:], start):\n if escape_next:\n escape_next = False\n continue\n \n if char == '\\\\':\n escape_next = True\n continue\n \n if char == '\"' and not escape_next:\n in_string = not in_string\n continue\n \n if not in_string:\n if char == '{':\n count += 1\n elif char == '}':\n count -= 1\n if count == 0:\n return text[start:i+1]\n return None\n \n json_str = extract_json_object(content)\n if not json_str:\n logger.warning(f\"Could not find JSON in LLM response for {tool.name}\")\n return None\n \n data = json.loads(json_str)\n \n # Extract and validate scores with robust error handling\n def safe_float(value, default=0.5, min_val=0.0, max_val=1.0):\n \"\"\"Safely convert to float and clamp to valid range.\"\"\"\n try:\n if value is None:\n return default\n f = float(value)\n return max(min_val, min(max_val, f))\n except (ValueError, TypeError):\n logger.warning(f\"Invalid score value: {value}, using default {default}\")\n return default\n \n clarity = safe_float(data.get(\"clarity\"), default=0.5)\n completeness = safe_float(data.get(\"completeness\"), default=0.5)\n reasoning = str(data.get(\"reasoning\", \"\"))[:500] # Limit reasoning length\n \n quality = DescriptionQuality(\n clarity=clarity,\n completeness=completeness,\n evaluated_at=datetime.now(),\n reasoning=reasoning,\n )\n \n # Update record\n record.description_quality = quality\n record.description_hash = self._compute_description_hash(tool)\n record.last_updated = datetime.now()\n \n # Save\n if self._auto_save and self._store:\n await self._store.save_record(record, self._records, self._global_execution_count)\n \n logger.info(f\"Evaluated description: {tool.name} score={quality.overall_score:.2f}\")\n return quality\n \n except Exception as e:\n logger.error(f\"Description evaluation failed for {tool.name}: {e}\")\n return None\n \n # Quality-Aware Ranking\n def adjust_ranking(\n self,\n tools_with_scores: List[Tuple[\"BaseTool\", float]],\n ) -> List[Tuple[\"BaseTool\", float]]:\n \"\"\"\n Adjust tool ranking using penalty-based approach.\n \n Args:\n tools_with_scores: List of (tool, semantic_score) tuples\n \"\"\"\n adjusted = []\n for tool, semantic_score in tools_with_scores:\n penalty = self.get_penalty(tool)\n \n adjusted_score = semantic_score * penalty\n \n adjusted.append((tool, adjusted_score))\n \n # Sort by adjusted score (descending)\n adjusted.sort(key=lambda x: x[1], reverse=True)\n \n return adjusted\n \n def get_penalty(self, tool: \"BaseTool\") -> float:\n \"\"\"Get penalty factor for a tool (0.2-1.0).\"\"\"\n return self.get_record(tool).penalty\n \n # Change Detection\n def check_changes(self, tools: List[\"BaseTool\"]) -> Dict[str, str]:\n \"\"\"\n Check for tool changes (new/updated/unchanged).\n \n Returns dict: {tool_key: \"new\"|\"updated\"|\"unchanged\"}\n \"\"\"\n changes = {}\n \n for tool in tools:\n key = self.get_tool_key(tool)\n current_hash = self._compute_description_hash(tool)\n \n if key not in self._records:\n changes[key] = \"new\"\n elif self._records[key].description_hash != current_hash:\n changes[key] = \"updated\"\n # Clear old evaluation on description change\n self._records[key].description_quality = None\n self._records[key].description_hash = current_hash\n else:\n changes[key] = \"unchanged\"\n \n new_count = sum(1 for v in changes.values() if v == \"new\")\n updated_count = sum(1 for v in changes.values() if v == \"updated\")\n \n if new_count or updated_count:\n logger.info(f\"Tool changes: {new_count} new, {updated_count} updated\")\n \n return changes\n \n async def save(self) -> None:\n \"\"\"\n Manually save all records to disk.\n \n Note: Usually not needed - auto_save handles persistence in\n record_execution(), evaluate_description(), and evolve().\n Provided as public API for explicit save when needed.\n \"\"\"\n if self._store:\n await self._store.save_all(self._records)\n \n def clear_cache(self) -> None:\n \"\"\"Clear all cached data.\"\"\"\n self._records.clear()\n if self._store:\n self._store.clear()\n \n def get_stats(self) -> Dict:\n \"\"\"\n Get quality tracking statistics.\n \n Note: Query API for inspection, may not be called in main flow.\n \"\"\"\n if not self._records:\n return {\"total_tools\": 0}\n \n records = list(self._records.values())\n \n return {\n \"total_tools\": len(records),\n \"total_executions\": sum(r.total_calls for r in records),\n \"avg_success_rate\": (\n sum(r.success_rate for r in records) / len(records)\n if records else 0\n ),\n \"avg_quality_score\": (\n sum(r.quality_score for r in records) / len(records)\n if records else 0\n ),\n \"tools_with_description_eval\": sum(\n 1 for r in records if r.description_quality\n ),\n }\n\n def get_top_tools(\n self,\n n: int = 10,\n backend: Optional[str] = None,\n min_calls: int = 3,\n ) -> List[ToolQualityRecord]:\n \"\"\"\n Get top N tools by quality score.\n \n Args:\n n: Number of tools to return\n backend: Filter by backend type (optional)\n min_calls: Minimum calls required (to filter untested tools)\n \"\"\"\n records = [\n r for r in self._records.values()\n if r.total_calls >= min_calls\n and (backend is None or r.backend == backend)\n ]\n \n records.sort(key=lambda r: r.quality_score, reverse=True)\n return records[:n]\n \n def get_problematic_tools(\n self,\n success_rate_threshold: float = 0.5,\n min_calls: int = 5,\n ) -> List[ToolQualityRecord]:\n \"\"\"\n Get tools with low success rate (candidates for review/removal).\n \n Args:\n success_rate_threshold: Tools below this rate are flagged\n min_calls: Minimum calls required (avoid flagging new tools)\n \"\"\"\n return [\n r for r in self._records.values()\n if r.total_calls >= min_calls\n and r.recent_success_rate < success_rate_threshold\n ]\n \n def get_quality_report(self) -> Dict:\n \"\"\"\n Generate comprehensive quality report for upper layer.\n \n Returns structured report with:\n - Overall stats\n - Per-backend breakdown\n - Top/problematic tools\n - Improvement suggestions\n \"\"\"\n if not self._records:\n return {\"status\": \"no_data\", \"message\": \"No quality data collected yet\"}\n \n records = list(self._records.values())\n tested_records = [r for r in records if r.total_calls >= 3]\n \n # Per-backend stats\n backends = {}\n for r in records:\n if r.backend not in backends:\n backends[r.backend] = {\n \"tools\": 0,\n \"total_calls\": 0,\n \"success_count\": 0,\n \"servers\": set()\n }\n backends[r.backend][\"tools\"] += 1\n backends[r.backend][\"total_calls\"] += r.total_calls\n backends[r.backend][\"success_count\"] += r.success_count\n backends[r.backend][\"servers\"].add(r.server)\n \n # Convert sets to counts\n for b in backends:\n backends[b][\"servers\"] = len(backends[b][\"servers\"])\n backends[b][\"success_rate\"] = (\n backends[b][\"success_count\"] / backends[b][\"total_calls\"]\n if backends[b][\"total_calls\"] > 0 else 0\n )\n \n # Top and problematic tools\n top_tools = self.get_top_tools(5)\n problematic = self.get_problematic_tools()\n \n return {\n \"summary\": {\n \"total_tools\": len(records),\n \"tested_tools\": len(tested_records),\n \"total_executions\": sum(r.total_calls for r in records),\n \"overall_success_rate\": (\n sum(r.success_count for r in records) /\n max(1, sum(r.total_calls for r in records))\n ),\n \"avg_quality_score\": (\n sum(r.quality_score for r in tested_records) / len(tested_records)\n if tested_records else 0\n ),\n },\n \"by_backend\": backends,\n \"top_tools\": [\n {\"key\": r.tool_key, \"score\": r.quality_score, \"success_rate\": r.success_rate}\n for r in top_tools\n ],\n \"problematic_tools\": [\n {\"key\": r.tool_key, \"success_rate\": r.success_rate, \"calls\": r.total_calls}\n for r in problematic\n ],\n \"recommendations\": self._generate_recommendations(records, problematic),\n }\n \n def _generate_recommendations(\n self,\n records: List[ToolQualityRecord],\n problematic: List[ToolQualityRecord],\n ) -> List[str]:\n \"\"\"Generate actionable recommendations based on quality data.\"\"\"\n recommendations = []\n \n # Check for problematic tools\n if problematic:\n tool_names = [r.tool_name for r in problematic[:3]]\n recommendations.append(\n f\"Review low-success tools: {', '.join(tool_names)}\"\n )\n \n # Check for tools needing description evaluation\n unevaluated = [r for r in records if not r.description_quality and r.total_calls >= 3]\n if unevaluated:\n recommendations.append(\n f\"{len(unevaluated)} tools need description quality evaluation\"\n )\n \n # Check for low description quality\n poor_docs = [\n r for r in records\n if r.description_quality and r.description_quality.overall_score < 0.5\n ]\n if poor_docs:\n recommendations.append(\n f\"{len(poor_docs)} tools have poor documentation quality\"\n )\n \n return recommendations\n\n def compute_adaptive_quality_weight(self) -> float:\n \"\"\"\n Compute adaptive quality weight based on data confidence.\n \n Returns higher weight when we have more reliable quality data,\n lower weight when data is sparse.\n \"\"\"\n if not self._records:\n return 0.1 # Low weight when no data\n \n records = list(self._records.values())\n tested_count = sum(1 for r in records if r.total_calls >= 3)\n \n if tested_count == 0:\n return 0.1\n \n # More tested tools -> higher confidence -> higher weight\n coverage = tested_count / len(records)\n \n # Average calls per tested tool -> data richness\n avg_calls = sum(r.total_calls for r in records) / len(records)\n richness = min(1.0, avg_calls / 20) # Cap at 20 calls average\n \n # Combine coverage and richness\n confidence = (coverage * 0.5 + richness * 0.5)\n \n # Map to weight range [0.1, 0.5]\n weight = 0.1 + confidence * 0.4\n \n return round(weight, 2)\n \n def should_reevaluate_description(self, tool: \"BaseTool\") -> bool:\n \"\"\"\n Check if a tool's description should be re-evaluated.\n \n Triggers re-evaluation when:\n - Description hash changed\n - Success rate dropped significantly\n - No evaluation yet but enough calls\n \"\"\"\n record = self._records.get(self.get_tool_key(tool))\n if not record:\n return True\n \n # Check hash change\n current_hash = self._compute_description_hash(tool)\n if current_hash != record.description_hash:\n return True\n \n # No evaluation yet but enough data\n if not record.description_quality and record.total_calls >= 5:\n return True\n \n # Success rate dropped significantly (maybe description is misleading)\n if record.description_quality and record.total_calls >= 10:\n if record.recent_success_rate < 0.5 and record.description_quality.overall_score > 0.7:\n # High doc quality but low success -> mismatch\n return True\n \n return False\n \n async def evolve(self, tools: List[\"BaseTool\"]) -> Dict:\n \"\"\"\n Run self-evolution cycle on given tools.\n \n This method:\n 1. Detects tool changes\n 2. Re-evaluates descriptions where needed\n 3. Updates quality weights\n 4. Returns evolution report\n \"\"\"\n report = {\n \"changes_detected\": {},\n \"descriptions_evaluated\": 0,\n \"adaptive_weight\": 0.0,\n \"recommendations\": [],\n }\n \n # 1. Detect changes\n report[\"changes_detected\"] = self.check_changes(tools)\n \n # 2. Find tools needing re-evaluation\n needs_eval = [t for t in tools if self.should_reevaluate_description(t)]\n \n # 3. Evaluate descriptions (limit to avoid too many LLM calls)\n if needs_eval and self._llm_client:\n for tool in needs_eval[:5]: # Max 5 per cycle\n result = await self.evaluate_description(tool, force=True)\n if result:\n report[\"descriptions_evaluated\"] += 1\n \n # 4. Compute adaptive weight\n report[\"adaptive_weight\"] = self.compute_adaptive_quality_weight()\n \n # 5. Generate recommendations\n problematic = self.get_problematic_tools()\n report[\"recommendations\"] = self._generate_recommendations(\n list(self._records.values()), problematic\n )\n \n # 6. Update last evolve count\n self._last_evolve_count = self._global_execution_count\n \n # Save\n if self._store:\n await self._store.save_all(self._records, self._global_execution_count)\n \n logger.info(\n f\"Evolution cycle complete: \"\n f\"changes={len([v for v in report['changes_detected'].values() if v != 'unchanged'])}, \"\n f\"evaluated={report['descriptions_evaluated']}, \"\n f\"weight={report['adaptive_weight']}, \"\n f\"global_count={self._global_execution_count}\"\n )\n \n return report\n \n def should_evolve(self) -> bool:\n \"\"\"Check if evolution should be triggered based on global execution count.\"\"\"\n return self._global_execution_count >= self._last_evolve_count + self._evolve_interval\n \n def get_tool_insights(self, tool: \"BaseTool\") -> Dict:\n \"\"\"\n Get detailed insights for a specific tool (for debugging/analysis).\n \n Returns comprehensive info about tool's quality history.\n \"\"\"\n record = self._records.get(self.get_tool_key(tool))\n if not record:\n return {\"status\": \"not_tracked\", \"tool\": tool.name}\n \n # Count recent failures\n recent_failures_count = sum(\n 1 for e in record.recent_executions[-20:]\n if not e.success\n )\n \n return {\n \"tool_key\": record.tool_key,\n \"total_calls\": record.total_calls,\n \"success_rate\": record.success_rate,\n \"recent_success_rate\": record.recent_success_rate,\n \"avg_execution_time_ms\": record.avg_execution_time_ms,\n \"quality_score\": record.quality_score,\n \"description_quality\": {\n \"overall_score\": record.description_quality.overall_score,\n \"clarity\": record.description_quality.clarity,\n \"completeness\": record.description_quality.completeness,\n \"reasoning\": record.description_quality.reasoning,\n } if record.description_quality else None,\n \"recent_failures_count\": recent_failures_count,\n \"first_seen\": record.first_seen.isoformat(),\n \"last_updated\": record.last_updated.isoformat(),\n }\n\n"
},
{
"path": "anytool/grounding/core/quality/store.py",
"content": "\"\"\"\nPersistent storage for tool quality data.\n\"\"\"\n\nimport json\nimport asyncio\nfrom pathlib import Path\nfrom typing import Dict, List, Optional\n\nfrom .types import ToolQualityRecord\nfrom anytool.utils.logging import Logger\nfrom anytool.config.constants import PROJECT_ROOT\n\nlogger = Logger.get_logger(__name__)\n\n\nclass QualityStore:\n \"\"\"\n Persistent storage for tool quality records.\n \n Storage structure:\n /.anytool/tool_quality/\n โโโ records.json # All quality records\n โโโ records_backup.json # Backup on save\n \"\"\"\n \n VERSION = 1\n \n def __init__(self, cache_dir: Optional[Path] = None):\n if cache_dir is None:\n cache_dir = PROJECT_ROOT / \".anytool\" / \"tool_quality\"\n \n self._cache_dir = Path(cache_dir)\n self._cache_dir.mkdir(parents=True, exist_ok=True)\n \n self._records_file = self._cache_dir / \"records.json\"\n self._backup_file = self._cache_dir / \"records_backup.json\"\n \n self._write_lock = asyncio.Lock()\n \n logger.debug(f\"QualityStore initialized at {self._cache_dir}\")\n \n def load_all(self) -> tuple[Dict[str, ToolQualityRecord], int]:\n \"\"\"Load all quality records and global execution count from disk.\n \n Returns:\n Tuple of (records_dict, global_execution_count)\n \"\"\"\n if not self._records_file.exists():\n return {}, 0\n \n try:\n with open(self._records_file, \"r\", encoding=\"utf-8\") as f:\n data = json.load(f)\n \n # Version check\n if data.get(\"version\") != self.VERSION:\n logger.warning(f\"Cache version mismatch, clearing cache\")\n return {}, 0\n \n records = {}\n for key, record_data in data.get(\"records\", {}).items():\n try:\n records[key] = ToolQualityRecord.from_dict(record_data)\n except Exception as e:\n logger.warning(f\"Failed to load record {key}: {e}\")\n \n global_count = data.get(\"global_execution_count\", 0)\n logger.info(f\"Loaded {len(records)} quality records from cache (global_count={global_count})\")\n return records, global_count\n \n except Exception as e:\n logger.error(f\"Failed to load quality cache: {e}\")\n return {}, 0\n \n async def save_all(self, records: Dict[str, ToolQualityRecord], global_execution_count: int = 0) -> None:\n \"\"\"Save all quality records and global execution count to disk.\"\"\"\n async with self._write_lock:\n try:\n # Backup existing file\n if self._records_file.exists():\n import shutil\n shutil.copy(self._records_file, self._backup_file)\n \n data = {\n \"version\": self.VERSION,\n \"global_execution_count\": global_execution_count,\n \"records\": {\n key: record.to_dict()\n for key, record in records.items()\n }\n }\n \n with open(self._records_file, \"w\", encoding=\"utf-8\") as f:\n json.dump(data, f, indent=2, ensure_ascii=False)\n \n logger.debug(f\"Saved {len(records)} quality records to cache (global_count={global_execution_count})\")\n \n except Exception as e:\n logger.error(f\"Failed to save quality cache: {e}\")\n \n async def save_record(self, record: ToolQualityRecord, all_records: Dict[str, ToolQualityRecord], global_execution_count: int = 0) -> None:\n \"\"\"Save a single record (saves all for simplicity).\"\"\"\n all_records[record.tool_key] = record\n await self.save_all(all_records, global_execution_count)\n \n def clear(self) -> None:\n \"\"\"Clear all cached data.\"\"\"\n if self._records_file.exists():\n self._records_file.unlink()\n if self._backup_file.exists():\n self._backup_file.unlink()\n logger.info(\"Quality cache cleared\")\n"
},
{
"path": "anytool/grounding/core/quality/types.py",
"content": "\"\"\"\nData types for tool quality tracking.\n\"\"\"\n\nfrom dataclasses import dataclass, field\nfrom datetime import datetime\nfrom typing import ClassVar, Dict, List, Optional, Any\n\n\n@dataclass\nclass ExecutionRecord:\n \"\"\"Single execution record.\"\"\"\n timestamp: datetime\n success: bool\n execution_time_ms: float\n error_message: Optional[str] = None\n\n\n@dataclass\nclass DescriptionQuality:\n \"\"\"LLM-evaluated description quality.\"\"\"\n clarity: float # 0-1: Is the purpose and usage clear?\n completeness: float # 0-1: Are inputs/outputs documented?\n evaluated_at: datetime\n reasoning: str = \"\" # LLM's reasoning for the scores\n \n @property\n def overall_score(self) -> float:\n \"\"\"Computed overall score (average of all dimensions).\"\"\"\n return (self.clarity + self.completeness) / 2\n\n\n@dataclass\nclass ToolQualityRecord:\n \"\"\"\n Complete quality record for a tool.\n \n Key: \"{backend}:{server}:{tool_name}\"\n \"\"\"\n tool_key: str\n backend: str\n server: str\n tool_name: str\n \n # Execution stats\n total_calls: int = 0\n success_count: int = 0\n total_execution_time_ms: float = 0.0\n \n # Recent execution history (rolling window)\n recent_executions: List[ExecutionRecord] = field(default_factory=list)\n \n # Description quality (LLM-evaluated)\n description_quality: Optional[DescriptionQuality] = None\n \n # Metadata\n description_hash: Optional[str] = None\n first_seen: datetime = field(default_factory=datetime.now)\n last_updated: datetime = field(default_factory=datetime.now)\n \n # Keep only recent N executions\n MAX_RECENT_EXECUTIONS: ClassVar[int] = 100\n \n # Penalty threshold: only penalize tools with success rate below this value\n # Tools with success rate >= this threshold get penalty = 1.0 (no penalty)\n PENALTY_THRESHOLD: ClassVar[float] = 0.4\n \n @property\n def success_rate(self) -> float:\n \"\"\"Overall success rate.\"\"\"\n if self.total_calls == 0:\n return 0.0\n return self.success_count / self.total_calls\n \n @property\n def avg_execution_time_ms(self) -> float:\n \"\"\"Average execution time.\"\"\"\n if self.total_calls == 0:\n return 0.0\n return self.total_execution_time_ms / self.total_calls\n \n @property\n def recent_success_rate(self) -> float:\n \"\"\"Success rate from recent executions.\"\"\"\n if not self.recent_executions:\n return self.success_rate\n successes = sum(1 for e in self.recent_executions if e.success)\n return successes / len(self.recent_executions)\n \n @property\n def consecutive_failures(self) -> int:\n \"\"\"Count consecutive failures from the most recent execution.\"\"\"\n count = 0\n for exec_record in reversed(self.recent_executions):\n if not exec_record.success:\n count += 1\n else:\n break\n return count\n \n @property\n def penalty(self) -> float:\n \"\"\"\n Compute penalty factor based on failure rate.\n \n Design principles:\n - Only penalize tools with success rate < PENALTY_THRESHOLD (default 40%)\n - New tools (< 3 calls) get no penalty to allow fair evaluation\n \n Returns value between 0.2-1.0:\n - 1.0: No penalty (success rate >= threshold or insufficient data)\n - 0.2: Maximum penalty (consistently failing tool)\n \"\"\"\n if self.total_calls < 3:\n return 1.0\n \n success_rate = self.recent_success_rate\n threshold = self.PENALTY_THRESHOLD\n \n if success_rate >= threshold:\n return 1.0\n \n # Linear mapping: penalty = 0.3 + (success_rate / threshold) * 0.7\n base_penalty = 0.3 + (success_rate / threshold) * 0.7\n \n # Extra penalty for consecutive failures (indicates systematic issues)\n consec = self.consecutive_failures\n if consec >= 3:\n # 3 consecutive โ extra 0.1, 5 consecutive โ extra 0.3\n extra_penalty = min(0.3, (consec - 2) * 0.1)\n base_penalty -= extra_penalty\n \n # Clamp to [0.2, 1.0]\n return max(0.2, min(1.0, base_penalty))\n \n @property\n def quality_score(self) -> float:\n \"\"\"\n Legacy quality score for backward compatibility.\n Now delegates to penalty property.\n \"\"\"\n return self.penalty\n \n def add_execution(self, record: ExecutionRecord) -> None:\n \"\"\"Add execution record and update stats.\"\"\"\n self.total_calls += 1\n self.total_execution_time_ms += record.execution_time_ms\n \n if record.success:\n self.success_count += 1\n \n self.recent_executions.append(record)\n \n # Trim to max size\n if len(self.recent_executions) > self.MAX_RECENT_EXECUTIONS:\n self.recent_executions = self.recent_executions[-self.MAX_RECENT_EXECUTIONS:]\n \n self.last_updated = datetime.now()\n \n def to_dict(self) -> Dict[str, Any]:\n \"\"\"Serialize to dict for persistence.\"\"\"\n return {\n \"tool_key\": self.tool_key,\n \"backend\": self.backend,\n \"server\": self.server,\n \"tool_name\": self.tool_name,\n \"total_calls\": self.total_calls,\n \"success_count\": self.success_count,\n \"total_execution_time_ms\": self.total_execution_time_ms,\n \"recent_executions\": [\n {\n \"timestamp\": e.timestamp.isoformat(),\n \"success\": e.success,\n \"execution_time_ms\": e.execution_time_ms,\n \"error_message\": e.error_message,\n }\n for e in self.recent_executions\n ],\n \"description_quality\": {\n \"clarity\": self.description_quality.clarity,\n \"completeness\": self.description_quality.completeness,\n \"evaluated_at\": self.description_quality.evaluated_at.isoformat(),\n \"reasoning\": self.description_quality.reasoning,\n } if self.description_quality else None,\n \"description_hash\": self.description_hash,\n \"first_seen\": self.first_seen.isoformat(),\n \"last_updated\": self.last_updated.isoformat(),\n }\n \n @classmethod\n def from_dict(cls, data: Dict[str, Any]) -> \"ToolQualityRecord\":\n \"\"\"Deserialize from dict.\"\"\"\n record = cls(\n tool_key=data[\"tool_key\"],\n backend=data[\"backend\"],\n server=data[\"server\"],\n tool_name=data[\"tool_name\"],\n total_calls=data.get(\"total_calls\", 0),\n success_count=data.get(\"success_count\", 0),\n total_execution_time_ms=data.get(\"total_execution_time_ms\", 0.0),\n description_hash=data.get(\"description_hash\"),\n first_seen=datetime.fromisoformat(data[\"first_seen\"]),\n last_updated=datetime.fromisoformat(data[\"last_updated\"]),\n )\n \n # Parse recent executions\n for e in data.get(\"recent_executions\", []):\n record.recent_executions.append(ExecutionRecord(\n timestamp=datetime.fromisoformat(e[\"timestamp\"]),\n success=e[\"success\"],\n execution_time_ms=e[\"execution_time_ms\"],\n error_message=e.get(\"error_message\"),\n ))\n \n # Parse description quality\n dq = data.get(\"description_quality\")\n if dq:\n record.description_quality = DescriptionQuality(\n clarity=dq.get(\"clarity\", 0.5), # Fallback for old data\n completeness=dq.get(\"completeness\", 0.5),\n evaluated_at=datetime.fromisoformat(dq[\"evaluated_at\"]),\n reasoning=dq.get(\"reasoning\", \"\"), # Optional field\n )\n \n return record\n"
},
{
"path": "anytool/grounding/core/search_tools.py",
"content": "from anytool.grounding.core.tool.base import BaseTool\nimport re\nimport os\nimport numpy as np\nimport httpx\nfrom typing import Iterable, List, Tuple, Dict, Optional, Any, TYPE_CHECKING\nfrom enum import Enum\nimport json\nimport pickle\nfrom pathlib import Path\nfrom datetime import datetime\n\nfrom .tool import BaseTool\nfrom .types import BackendType\nfrom anytool.llm import LLMClient\nfrom anytool.utils.logging import Logger\nfrom anytool.config.constants import PROJECT_ROOT\n\nif TYPE_CHECKING:\n from .quality import ToolQualityManager\n\nlogger = Logger.get_logger(__name__)\n\n\nclass SearchMode(str, Enum):\n SEMANTIC = \"semantic\"\n KEYWORD = \"keyword\"\n HYBRID = \"hybrid\"\n\n\nclass ToolRanker:\n \"\"\"\n ToolRanker: rank tools by keyword, semantic or hybrid\n \"\"\"\n # Cache version for persistent storage - increment when cache format changes\n CACHE_VERSION = 1\n \n def __init__(\n self, \n model_name: Optional[str] = None,\n cache_dir: Optional[str | Path] = None,\n enable_cache_persistence: bool = False\n ):\n \"\"\"Initialize ToolRanker.\n \n Args:\n model_name: Embedding model name. If None, will use env or config value.\n cache_dir: Directory to store persistent embedding cache.\n enable_cache_persistence: Whether to persist embeddings to disk.\n \"\"\"\n # Check for remote API config from environment\n self._api_base_url = os.getenv(\"EMBEDDING_BASE_URL\")\n self._api_key = os.getenv(\"EMBEDDING_API_KEY\")\n self._use_remote_api = bool(self._api_key and self._api_base_url)\n \n # Get model name: env > param > config > default\n if model_name is None:\n model_name = os.getenv(\"EMBEDDING_MODEL\")\n \n if model_name is None:\n try:\n from anytool.config import get_config\n config = get_config()\n model_name = config.tool_search.embedding_model\n except Exception as exc:\n logger.warning(f\"Failed to load config, using default model: {exc}\")\n model_name = \"BAAI/bge-small-en-v1.5\"\n \n self._model_name = model_name\n self._embed_model = None # lazy load\n self._embedding_fn = None\n \n if self._use_remote_api:\n logger.info(f\"Using remote embedding API: {self._api_base_url}, model: {model_name}\")\n \n # Persistent cache settings\n self._enable_cache_persistence = enable_cache_persistence\n if cache_dir is None:\n cache_dir = PROJECT_ROOT / \".anytool\" / \"embedding_cache\"\n self._cache_dir = Path(cache_dir)\n \n # Log cache settings\n logger.info(\n f\"ToolRanker initialized: enable_cache_persistence={enable_cache_persistence}, \"\n f\"cache_dir={self._cache_dir}\"\n )\n \n # Structured in-memory cache\n # Structure: {backend: {server: {tool_name: {\"embedding\": np.ndarray, \"description\": str, \"cached_at\": str}}}}\n self._structured_cache: Dict[str, Dict[str, Dict[str, Dict[str, Any]]]] = {}\n \n # For backward compatibility and quick lookup: {text -> (backend, server, tool_name)}\n self._text_to_key: Dict[str, Tuple[str, str, str]] = {}\n \n # Load persistent cache if enabled\n if self._enable_cache_persistence:\n logger.info(f\"Loading persistent cache from {self._cache_dir}\")\n self._load_persistent_cache()\n \n def _get_cache_key(self, tool: BaseTool) -> Tuple[str, str, str]:\n \"\"\"Get structured cache key (backend, server, tool_name) from tool.\"\"\"\n if tool.is_bound:\n backend = tool.runtime_info.backend.value\n server = tool.runtime_info.server_name or \"default\"\n else:\n if not tool.backend_type or tool.backend_type == BackendType.NOT_SET:\n backend = \"UNKNOWN\"\n else:\n backend = tool.backend_type.value\n server = \"default\"\n \n return (backend, server, tool.name)\n \n def _get_cache_file_path(self) -> Path:\n \"\"\"Get the cache file path for the current model.\"\"\"\n # Use model name in filename to support multiple models\n safe_model_name = self._model_name.replace(\"/\", \"_\").replace(\"\\\\\", \"_\")\n return self._cache_dir / f\"embeddings_{safe_model_name}_v{self.CACHE_VERSION}.pkl\"\n \n def _load_persistent_cache(self) -> None:\n \"\"\"Load embeddings from disk cache.\"\"\"\n cache_file = self._get_cache_file_path()\n \n if not cache_file.exists():\n logger.debug(f\"No persistent cache found at {cache_file}\")\n return\n \n try:\n with open(cache_file, 'rb') as f:\n data = pickle.load(f)\n \n # Validate cache version\n if isinstance(data, dict) and data.get(\"version\") == self.CACHE_VERSION:\n self._structured_cache = data.get(\"embeddings\", {})\n self._rebuild_text_index()\n \n # Count total embeddings\n total = sum(\n len(tools) \n for backend in self._structured_cache.values() \n for tools in backend.values()\n )\n logger.info(f\"Loaded {total} embeddings from cache: {cache_file}\")\n else:\n logger.warning(f\"Cache version mismatch or invalid format, starting fresh\")\n self._structured_cache = {}\n except Exception as exc:\n logger.warning(f\"Failed to load persistent cache: {exc}\")\n self._structured_cache = {}\n \n def _rebuild_text_index(self) -> None:\n \"\"\"Rebuild text-to-key mapping for quick lookup.\"\"\"\n self._text_to_key.clear()\n for backend, servers in self._structured_cache.items():\n for server, tools in servers.items():\n for tool_name, tool_data in tools.items():\n desc = tool_data.get(\"description\", \"\")\n text = f\"{tool_name}: {desc}\"\n self._text_to_key[text] = (backend, server, tool_name)\n \n def _save_persistent_cache(self) -> None:\n \"\"\"Save embeddings to disk cache.\"\"\"\n if not self._enable_cache_persistence or not self._structured_cache:\n return\n \n cache_file = self._get_cache_file_path()\n \n try:\n # Create directory if it doesn't exist\n cache_file.parent.mkdir(parents=True, exist_ok=True)\n \n # Build cache data with metadata\n cache_data = {\n \"version\": self.CACHE_VERSION,\n \"model_name\": self._model_name,\n \"last_updated\": datetime.now().isoformat(),\n \"embeddings\": self._structured_cache\n }\n \n # Save cache\n with open(cache_file, 'wb') as f:\n pickle.dump(cache_data, f, protocol=pickle.HIGHEST_PROTOCOL)\n \n # Count total embeddings\n total = sum(\n len(tools) \n for backend in self._structured_cache.values() \n for tools in backend.values()\n )\n logger.debug(f\"Saved {total} embeddings to cache: {cache_file}\")\n except Exception as exc:\n logger.warning(f\"Failed to save persistent cache: {exc}\")\n\n def rank(\n self,\n query: str,\n tools: List[BaseTool],\n *,\n top_k: int = 50,\n mode: SearchMode = SearchMode.SEMANTIC,\n ) -> List[Tuple[BaseTool, float]]:\n if mode == SearchMode.KEYWORD:\n return self._keyword_search(query, tools, top_k)\n if mode == SearchMode.SEMANTIC:\n return self._semantic_search(query, tools, top_k)\n # hybrid\n return self._hybrid_search(query, tools, top_k)\n\n @staticmethod\n def _tokenize(text: str) -> list[str]:\n tokens = re.split(r\"[^\\w]+\", text.lower())\n tokens = [tok for tok in tokens if tok]\n return tokens\n\n def _keyword_search(\n self, query: str, tools: Iterable[BaseTool], top_k: int\n ) -> List[Tuple[BaseTool, float]]:\n try:\n from rank_bm25 import BM25Okapi # type: ignore\n except ImportError:\n BM25Okapi = None # fallback below\n\n tool_list = list(tools)\n if not tool_list:\n return []\n \n corpus_tokens: list[list[str]] = [self._tokenize(f\"{t.name} {t.description}\") for t in tool_list]\n query_tokens = self._tokenize(query)\n\n if BM25Okapi and corpus_tokens:\n bm25 = BM25Okapi(corpus_tokens)\n scores = bm25.get_scores(query_tokens)\n scored = [(t, float(s)) for t, s in zip(tool_list, scores, strict=True)]\n else:\n # fallback: simple term overlap ratio\n q_set = set(query_tokens)\n scored = []\n for t, toks in zip(tool_list, corpus_tokens, strict=True):\n if not toks:\n scored.append((t, 0.0)) # Include tool with 0 score\n continue\n overlap = q_set.intersection(toks)\n score = len(overlap) / len(q_set) if len(q_set) > 0 else 0.0\n scored.append((t, score))\n\n scored.sort(key=lambda x: x[1], reverse=True)\n result = scored[:top_k]\n \n # If no matches found (all scores are 0), return all tools\n if not result or all(score == 0.0 for _, score in result):\n logger.debug(f\"Keyword search found no matches, returning all {len(tool_list)} tools\")\n return [(t, 0.0) for t in tool_list]\n \n return result\n\n def _ensure_model(self) -> bool:\n \"\"\"Ensure embedding model is ready (local or remote).\"\"\"\n if self._embedding_fn is not None:\n return True\n \n if self._use_remote_api:\n return self._init_remote_embedding()\n return self._init_local_embedding()\n\n def _init_remote_embedding(self) -> bool:\n \"\"\"Initialize remote embedding API (OpenRouter/OpenAI compatible).\"\"\"\n try:\n def embed_texts(texts: List[str]) -> List[np.ndarray]:\n with httpx.Client(timeout=60.0) as client:\n response = client.post(\n f\"{self._api_base_url}/embeddings\",\n headers={\n \"Authorization\": f\"Bearer {self._api_key}\",\n \"Content-Type\": \"application/json\"\n },\n json={\"model\": self._model_name, \"input\": texts}\n )\n response.raise_for_status()\n data = response.json()\n return [np.array(item[\"embedding\"]) for item in data[\"data\"]]\n \n self._embedding_fn = embed_texts\n logger.info(f\"Remote embedding API initialized: {self._model_name}\")\n return True\n except Exception as exc:\n logger.error(f\"Failed to initialize remote embedding API: {exc}\")\n return False\n\n def _init_local_embedding(self) -> bool:\n \"\"\"Initialize local fastembed model.\"\"\"\n try:\n from fastembed import TextEmbedding \n logger.debug(f\"fastembed imported successfully, loading model: {self._model_name}\")\n except ImportError as e:\n logger.warning(\n f\"fastembed not installed (ImportError: {e}), semantic search unavailable. \"\n f\"Install with: pip install fastembed\"\n )\n return False\n \n try:\n logger.info(f\"Loading embedding model: {self._model_name}...\")\n self._embed_model = TextEmbedding(model_name=self._model_name)\n self._embedding_fn = lambda txts: list(self._embed_model.embed(txts))\n logger.info(f\"Embedding model '{self._model_name}' loaded successfully\")\n return True\n except Exception as exc:\n logger.error(f\"Embedding model '{self._model_name}' loading failed: {exc}\")\n return False\n\n def _get_embedding(self, tool: BaseTool) -> Optional[np.ndarray]:\n \"\"\"Get embedding from structured cache.\"\"\"\n backend, server, tool_name = self._get_cache_key(tool)\n \n if backend not in self._structured_cache:\n return None\n if server not in self._structured_cache[backend]:\n return None\n if tool_name not in self._structured_cache[backend][server]:\n return None\n \n return self._structured_cache[backend][server][tool_name].get(\"embedding\")\n \n def _set_embedding(self, tool: BaseTool, embedding: np.ndarray) -> None:\n \"\"\"Store embedding in structured cache.\"\"\"\n backend, server, tool_name = self._get_cache_key(tool)\n \n # Initialize nested structure if needed\n if backend not in self._structured_cache:\n self._structured_cache[backend] = {}\n if server not in self._structured_cache[backend]:\n self._structured_cache[backend][server] = {}\n \n # Store embedding with metadata\n self._structured_cache[backend][server][tool_name] = {\n \"embedding\": embedding,\n \"description\": tool.description or \"\",\n \"cached_at\": datetime.now().isoformat()\n }\n \n # Update text index for backward compatibility\n text = f\"{tool.name}: {tool.description}\"\n self._text_to_key[text] = (backend, server, tool_name)\n \n def _semantic_search(\n self, query: str, tools: Iterable[BaseTool], top_k: int\n ) -> List[Tuple[BaseTool, float]]:\n if not self._ensure_model():\n logger.debug(\"Semantic search unavailable, returning empty list\")\n return []\n \n tools_list = list(tools)\n \n # Collect embeddings with cache reuse\n missing_tools = [t for t in tools_list if self._get_embedding(t) is None]\n cache_updated = False\n \n if missing_tools:\n try:\n # Generate embeddings for missing tools\n missing_texts = [f\"{t.name}: {t.description}\" for t in missing_tools]\n new_embs = self._embedding_fn(missing_texts)\n \n for tool, emb in zip(missing_tools, new_embs, strict=True):\n self._set_embedding(tool, emb)\n \n cache_updated = True\n logger.debug(f\"Computed embeddings for {len(missing_tools)} new tools\")\n except Exception as exc:\n logger.error(\"Failed to generate embeddings: %s\", exc)\n return []\n \n # Save to persistent cache if updated\n if cache_updated:\n self._save_persistent_cache()\n\n try:\n q_emb = self._embedding_fn([query])[0]\n except Exception as exc:\n logger.error(\"Failed to embed query: %s\", exc)\n return []\n\n scored: list[tuple[BaseTool, float]] = []\n for t in tools_list:\n emb = self._get_embedding(t)\n if emb is None:\n # Should not happen, but handle gracefully\n logger.warning(f\"No embedding found for tool: {t.name}\")\n scored.append((t, 0.0))\n continue\n \n # Calculate cosine similarity with zero-division protection\n q_norm = np.linalg.norm(q_emb)\n emb_norm = np.linalg.norm(emb)\n if q_norm == 0 or emb_norm == 0:\n sim = 0.0\n else:\n sim = float(np.dot(q_emb, emb) / (q_norm * emb_norm))\n scored.append((t, sim))\n \n scored.sort(key=lambda x: x[1], reverse=True)\n return scored[:top_k]\n\n def _hybrid_search(\n self, query: str, tools: Iterable[BaseTool], top_k: int\n ) -> List[Tuple[BaseTool, float]]:\n # keyword filter\n kw_top = self._keyword_search(query, tools, top_k * 3)\n if not kw_top:\n # No keyword matches, try semantic search\n semantic_results = self._semantic_search(query, tools, top_k)\n if semantic_results:\n return semantic_results\n # Both failed, return top N tools\n logger.warning(\"Both keyword and semantic search failed, returning top N tools\")\n return [(t, 0.0) for t in list(tools)[:top_k]]\n \n # semantic ranking on keyword results\n semantic_results = self._semantic_search(query, [t for t, _ in kw_top], top_k)\n if semantic_results:\n return semantic_results\n \n # Semantic unavailable, return keyword results\n logger.debug(\"Semantic search unavailable, using keyword results only\")\n return kw_top[:top_k]\n \n def get_cache_stats(self) -> Dict[str, Any]:\n \"\"\"Get statistics about the embedding cache.\n \n Returns:\n Dict with structure: {\n \"total_embeddings\": int,\n \"backends\": {\n \"backend_name\": {\n \"total\": int,\n \"servers\": {\n \"server_name\": int # count of tools\n }\n }\n }\n }\n \"\"\"\n stats = {\n \"total_embeddings\": 0,\n \"backends\": {}\n }\n \n for backend, servers in self._structured_cache.items():\n backend_total = 0\n server_stats = {}\n \n for server, tools in servers.items():\n tool_count = len(tools)\n backend_total += tool_count\n server_stats[server] = tool_count\n \n stats[\"backends\"][backend] = {\n \"total\": backend_total,\n \"servers\": server_stats\n }\n stats[\"total_embeddings\"] += backend_total\n \n return stats\n \n def clear_cache(self, backend: Optional[str] = None, server: Optional[str] = None) -> int:\n \"\"\"Clear embeddings from cache.\n \n Args:\n backend: If provided, only clear this backend. If None, clear all.\n server: If provided (and backend is provided), only clear this server.\n \n Returns:\n Number of embeddings cleared.\n \"\"\"\n cleared_count = 0\n \n if backend is None:\n # Clear everything\n for b in self._structured_cache.values():\n for s in b.values():\n cleared_count += len(s)\n self._structured_cache.clear()\n self._text_to_key.clear()\n elif server is None:\n # Clear specific backend\n if backend in self._structured_cache:\n for s in self._structured_cache[backend].values():\n cleared_count += len(s)\n del self._structured_cache[backend]\n # Rebuild text index\n self._rebuild_text_index()\n else:\n # Clear specific backend+server\n if backend in self._structured_cache and server in self._structured_cache[backend]:\n cleared_count = len(self._structured_cache[backend][server])\n del self._structured_cache[backend][server]\n # Clean up empty backend\n if not self._structured_cache[backend]:\n del self._structured_cache[backend]\n # Rebuild text index\n self._rebuild_text_index()\n \n # Save after clearing\n if cleared_count > 0 and self._enable_cache_persistence:\n self._save_persistent_cache()\n logger.info(f\"Cleared {cleared_count} embeddings from cache\")\n \n return cleared_count\n\n\nclass SearchDebugInfo:\n \"\"\"Debug information from tool search process.\"\"\"\n \n def __init__(self):\n self.search_mode: str = \"\"\n self.total_candidates: int = 0\n self.mcp_count: int = 0\n self.non_mcp_count: int = 0\n \n # LLM filter info\n self.llm_filter_used: bool = False\n self.llm_brief_plan: str = \"\"\n self.llm_utility_tools: Dict[str, List[str]] = {} # server -> tool names\n self.llm_domain_servers: List[str] = []\n self.llm_utility_count: int = 0\n self.llm_domain_count: int = 0\n \n # Semantic search scores\n self.tool_scores: List[Dict[str, Any]] = [] # [{name, server, score, selected}]\n \n # Final selected tools\n self.selected_tools: List[Dict[str, Any]] = [] # [{name, server, backend}]\n \n def to_dict(self) -> Dict[str, Any]:\n return {\n \"search_mode\": self.search_mode,\n \"total_candidates\": self.total_candidates,\n \"mcp_count\": self.mcp_count,\n \"non_mcp_count\": self.non_mcp_count,\n \"llm_filter\": {\n \"used\": self.llm_filter_used,\n \"brief_plan\": self.llm_brief_plan,\n \"utility_tools\": self.llm_utility_tools,\n \"domain_servers\": self.llm_domain_servers,\n \"utility_count\": self.llm_utility_count,\n \"domain_count\": self.llm_domain_count,\n },\n \"tool_scores\": self.tool_scores,\n \"selected_tools\": self.selected_tools,\n }\n\n\nclass SearchCoordinator(BaseTool):\n _name = \"_filter_tools\"\n _description = \"Internal helper: filter & rank tools from a given list.\"\n \n # Fallback defaults when config loading fails\n DEFAULT_MAX_TOOLS: int = 20\n DEFAULT_LLM_FILTER: bool = True\n DEFAULT_LLM_THRESHOLD: int = 50\n DEFAULT_CACHE_PERSISTENCE: bool = False\n DEFAULT_SEARCH_MODE: str = \"hybrid\"\n\n @classmethod\n def get_parameters_schema(cls) -> Dict[str, Any]:\n \"\"\"Override to avoid JSON schema generation for list[BaseTool] parameter.\n \n The _arun method uses `candidate_tools: list[BaseTool]` which cannot be\n converted to JSON Schema because BaseTool is an ABC class, not a Pydantic model.\n Since this is an internal tool, we return an empty schema.\n \"\"\"\n return {}\n\n def __init__(\n self,\n *,\n max_tools: Optional[int] = None,\n llm: LLMClient = LLMClient(),\n enable_llm_filter: Optional[bool] = None,\n llm_filter_threshold: Optional[int] = None,\n enable_cache_persistence: Optional[bool] = None,\n cache_dir: Optional[str | Path] = None,\n quality_manager: Optional[\"ToolQualityManager\"] = None,\n enable_quality_ranking: bool = True,\n ):\n \"\"\"Create a SearchCoordinator.\n\n Args:\n max_tools: max number of tools to return. If None, will use the value from config.\n llm: optional async LLM, used to filter backend/server first\n enable_llm_filter: whether to use LLM to pre-filter by backend/server. \n If None, uses config value.\n llm_filter_threshold: only apply LLM filter when tool count > this threshold.\n If None, always apply (when enabled).\n enable_cache_persistence: whether to persist embeddings to disk. If None, uses config value.\n cache_dir: directory to store persistent embedding cache. If None, uses config value or default.\n \"\"\"\n super().__init__()\n \n # Load config (may be None if loading fails)\n tool_search_config = None\n try:\n from anytool.config import get_config\n tool_search_config = getattr(get_config(), 'tool_search', None)\n except Exception as exc:\n logger.warning(f\"Failed to load config: {exc}\")\n \n def resolve(user_value, config_attr: str, default):\n \"\"\"Priority: user_value โ config โ default\"\"\"\n if user_value is not None:\n return user_value\n if tool_search_config is not None:\n config_value = getattr(tool_search_config, config_attr, None)\n if config_value is not None:\n return config_value\n return default\n \n # Resolve each setting with priority: user โ config โ default\n self.max_tools = resolve(max_tools, 'max_tools', self.DEFAULT_MAX_TOOLS)\n enable_llm_filter = resolve(enable_llm_filter, 'enable_llm_filter', self.DEFAULT_LLM_FILTER)\n llm_filter_threshold = resolve(llm_filter_threshold, 'llm_filter_threshold', self.DEFAULT_LLM_THRESHOLD)\n enable_cache_persistence = resolve(enable_cache_persistence, 'enable_cache_persistence', self.DEFAULT_CACHE_PERSISTENCE)\n cache_dir = resolve(cache_dir, 'cache_dir', None)\n self._default_mode = resolve(None, 'search_mode', self.DEFAULT_SEARCH_MODE)\n \n # Log cache settings for debugging\n logger.info(\n f\"SearchCoordinator initialized with cache settings: \"\n f\"enable_cache_persistence={enable_cache_persistence}, cache_dir={cache_dir}\"\n )\n \n self._ranker = ToolRanker(\n enable_cache_persistence=enable_cache_persistence,\n cache_dir=cache_dir\n )\n self._llm: LLMClient | None = llm if llm is not None else LLMClient()\n \n # LLM filter settings\n self._enable_llm_filter = enable_llm_filter\n self._llm_filter_threshold = llm_filter_threshold\n \n # Quality-aware ranking settings\n self._quality_manager = quality_manager\n self._enable_quality_ranking = enable_quality_ranking\n \n # Debug info from last search\n self._last_search_debug_info: Optional[SearchDebugInfo] = None\n\n async def _arun(\n self,\n task_prompt: str,\n candidate_tools: list[BaseTool],\n *,\n max_tools: int | None = None,\n mode: str | None = None, # \"semantic\" | \"keyword\" | \"hybrid\"\n ) -> list[BaseTool]:\n max_tools = self.max_tools if max_tools is None else max_tools\n mode = self._default_mode if mode is None else mode\n\n # Initialize debug info\n debug_info = SearchDebugInfo()\n debug_info.search_mode = mode\n debug_info.total_candidates = len(candidate_tools)\n self._last_search_debug_info = debug_info\n\n # Cache check\n cache_key = (id(candidate_tools), task_prompt, mode, max_tools)\n if not hasattr(self, \"_query_cache\"):\n self._query_cache: Dict[tuple, list[BaseTool]] = {}\n if cache_key in self._query_cache:\n return self._query_cache[cache_key]\n\n # Split MCP tools and non-MCP tools\n # Non-MCP tools (shell, gui, web, etc.) are always included, skip all filtering\n mcp_tools = []\n non_mcp_tools = []\n \n for t in candidate_tools:\n if t.is_bound:\n backend = t.runtime_info.backend.value\n else:\n backend = t.backend_type.value if t.backend_type else \"UNKNOWN\"\n \n if backend.lower() == \"mcp\":\n mcp_tools.append(t)\n else:\n non_mcp_tools.append(t)\n \n debug_info.mcp_count = len(mcp_tools)\n debug_info.non_mcp_count = len(non_mcp_tools)\n logger.info(f\"Tool split: {len(mcp_tools)} MCP, {len(non_mcp_tools)} non-MCP (always included)\")\n \n # If MCP tools within limit, return all\n if len(mcp_tools) <= max_tools:\n result = mcp_tools + non_mcp_tools\n self._query_cache[cache_key] = result\n self._populate_selected_tools(debug_info, result)\n return result\n\n mcp_count = len(mcp_tools)\n should_use_llm_filter = (\n self._llm and \n self._enable_llm_filter and \n mcp_count > self._llm_filter_threshold\n )\n \n # Path 1: LLM pre-filter (large MCP tool set)\n if should_use_llm_filter:\n logger.info(f\"Path 1: MCP count ({mcp_count}) > threshold, using LLM filter...\")\n debug_info.llm_filter_used = True\n \n try:\n utility_tools, domain_tools, llm_filter_info = await self._llm_filter_with_planning(\n task_prompt, mcp_tools\n )\n \n # Record LLM filter results\n debug_info.llm_brief_plan = llm_filter_info.get(\"brief_plan\", \"\")\n debug_info.llm_utility_tools = llm_filter_info.get(\"utility_tools\", {})\n debug_info.llm_domain_servers = llm_filter_info.get(\"domain_servers\", [])\n \n utility_count = len(utility_tools)\n domain_count = len(domain_tools)\n debug_info.llm_utility_count = utility_count\n debug_info.llm_domain_count = domain_count\n total_count = utility_count + domain_count\n \n if total_count <= max_tools:\n mcp_result = utility_tools + domain_tools\n else:\n # Exceeds limit: keep utility, search domain\n domain_quota = max(max_tools - utility_count, 5)\n logger.info(\n f\"Total ({total_count}) > max_tools ({max_tools}), \"\n f\"keeping {utility_count} utility, searching {domain_count} domain (quota: {domain_quota})\"\n )\n \n # Compute scores for utility tools (marked as LLM-selected)\n if utility_tools:\n utility_ranked = self._ranker.rank(\n task_prompt, utility_tools,\n top_k=len(utility_tools), mode=SearchMode(mode)\n )\n self._record_tool_scores(debug_info, utility_ranked, is_selected=True)\n \n if domain_tools:\n # Rank all domain tools to see all scores for debugging\n all_domain_ranked = self._ranker.rank(\n task_prompt, domain_tools, \n top_k=len(domain_tools), mode=SearchMode(mode)\n )\n # Save scores for all domain tools (mark which ones are selected)\n for i, (tool, score) in enumerate(all_domain_ranked):\n server_name = None\n if tool.is_bound and tool.runtime_info:\n server_name = tool.runtime_info.server_name\n debug_info.tool_scores.append({\n \"name\": tool.name,\n \"server\": server_name,\n \"score\": round(score, 4),\n \"selected\": i < domain_quota,\n })\n searched_domain = [t for t, _ in all_domain_ranked[:domain_quota]]\n else:\n searched_domain = []\n \n mcp_result = utility_tools + searched_domain\n \n except Exception as exc:\n logger.warning(f\"LLM filter failed ({exc}), fallback to direct ranking\")\n ranked = self._ranker.rank(task_prompt, mcp_tools, top_k=max_tools, mode=SearchMode(mode))\n self._record_tool_scores(debug_info, ranked, is_selected=True)\n mcp_result = [t for t, _ in ranked]\n \n # Path 2: Plan-enhanced search (small MCP tool set)\n else:\n logger.info(f\"Path 2: MCP count ({mcp_count}) <= threshold, using enhanced search...\")\n debug_info.llm_filter_used = False\n \n if self._llm:\n try:\n enhanced_query = await self._generate_search_query(task_prompt)\n except Exception:\n enhanced_query = task_prompt\n else:\n enhanced_query = task_prompt\n \n try:\n ranked = self._ranker.rank(\n enhanced_query, mcp_tools, \n top_k=max_tools, mode=SearchMode(mode)\n )\n # Record all scores from semantic search\n self._record_tool_scores(debug_info, ranked, is_selected=True)\n mcp_result = [t for t, _ in ranked]\n except Exception:\n ranked = self._ranker._keyword_search(\n enhanced_query, mcp_tools, max_tools\n )\n self._record_tool_scores(debug_info, ranked, is_selected=True)\n mcp_result = [t for t, _ in ranked]\n\n # Apply quality ranking on MCP results\n if self._enable_quality_ranking and self._quality_manager and mcp_result:\n try:\n ranked_with_scores = [(t, 1.0) for t in mcp_result]\n ranked_with_scores = self._quality_manager.adjust_ranking(ranked_with_scores)\n mcp_result = [t for t, _ in ranked_with_scores]\n except Exception:\n pass\n\n # Limit MCP tools, then combine with non-MCP tools\n mcp_result = mcp_result[:max_tools]\n result = mcp_result + non_mcp_tools\n \n # Populate final selected tools in debug info\n self._populate_selected_tools(debug_info, result)\n \n self._log_search_results(candidate_tools, result, mode)\n self._query_cache[cache_key] = result\n return result\n \n def _record_tool_scores(\n self, \n debug_info: SearchDebugInfo, \n ranked: List[Tuple[BaseTool, float]], \n is_selected: bool = False\n ) -> None:\n \"\"\"Record tool scores from ranking results.\"\"\"\n for tool, score in ranked:\n server_name = None\n if tool.is_bound and tool.runtime_info:\n server_name = tool.runtime_info.server_name\n \n debug_info.tool_scores.append({\n \"name\": tool.name,\n \"server\": server_name,\n \"score\": round(score, 4),\n \"selected\": is_selected,\n })\n \n def _populate_selected_tools(\n self, \n debug_info: SearchDebugInfo, \n tools: List[BaseTool]\n ) -> None:\n \"\"\"Populate selected tools in debug info.\"\"\"\n for tool in tools:\n backend = \"UNKNOWN\"\n server_name = None\n \n if tool.is_bound and tool.runtime_info:\n backend = tool.runtime_info.backend.value\n server_name = tool.runtime_info.server_name\n elif tool.backend_type:\n backend = tool.backend_type.value\n \n debug_info.selected_tools.append({\n \"name\": tool.name,\n \"server\": server_name,\n \"backend\": backend,\n })\n\n async def _llm_filter_with_planning(\n self, \n task_prompt: str, \n tools: list[BaseTool]\n ) -> tuple[list[BaseTool], list[BaseTool], Dict[str, Any]]:\n \"\"\"\n LLM pre-filter for MCP servers.\n Returns (utility_tools, domain_tools, llm_filter_info).\n \"\"\"\n from collections import defaultdict\n \n # Group tools by server name\n server_tools: Dict[str, list[BaseTool]] = defaultdict(list)\n for t in tools:\n if t.is_bound and t.runtime_info:\n server = t.runtime_info.server_name or \"default\"\n else:\n server = \"unknown\"\n server_tools[server].append(t)\n\n # Build tool name -> tool object mapping\n tool_name_map: Dict[str, BaseTool] = {t.name: t for t in tools}\n\n # Build server description with tool names\n lines: list[str] = [\"Available MCP servers:\"]\n lines.append(\"\")\n \n for server, tool_list in server_tools.items():\n lines.append(f\"### Server: {server} ({len(tool_list)} tools)\")\n tool_names = [t.name for t in tool_list]\n lines.append(f\" All tools: {', '.join(tool_names)}\")\n if tool_list:\n lines.append(f\" Example capabilities:\")\n for tool in tool_list[:5]:\n tool_desc = tool.description or \"No description\"\n if len(tool_desc) > 100:\n tool_desc = tool_desc[:97] + \"...\"\n lines.append(f\" - {tool.name}: {tool_desc}\")\n lines.append(\"\")\n\n servers_block = \"\\n\".join(lines)\n\n TOOL_FILTER_SYSTEM_PROMPT = f\"\"\"You are an expert tool selection assistant.\n\n# Your task\nAnalyze the given task and determine which MCP servers and tools are needed.\nThink about how you would accomplish this task step by step, then classify needed servers and tools.\n\n# Important guidelines\n- **Focus on tool names and capabilities**: Carefully examine the tool names to understand what each server can do\n- **Be inclusive for domain servers**: If a server has tools that might be relevant to the core task, include it\n- **Be precise for utility tools**: Only select the specific auxiliary tools needed (e.g., file save, time query)\n- **When in doubt, include in domain_servers**: It's better to include a server than miss relevant tools\n\n{servers_block}\n\n# Output format\nReturn ONLY a JSON object (no markdown, no explanation):\n{{\n \"brief_plan\": \"1-2 sentence execution plan\",\n \"utility_tools\": {{\n \"server1\": [\"tool1\", \"tool2\"]\n }},\n \"domain_servers\": [\"server2\", \"server3\"]\n}}\n\n- **utility_tools**: Dict mapping server name to list of specific tool names.\n These are auxiliary tools for supporting operations (e.g., filesystem: [\"write_file\"], time-server: [\"get_time\"]).\n Only include the specific tools needed, NOT the entire server.\n- **domain_servers**: Server names that directly provide the main capabilities for the task.\n All tools from these servers will be considered. Be inclusive here.\"\"\"\n\n user_query = f\"Task: {task_prompt}\\n\\nClassify the needed servers and tools.\"\n\n messages_text = LLMClient.format_messages_to_text([\n {\"role\": \"system\", \"content\": TOOL_FILTER_SYSTEM_PROMPT},\n {\"role\": \"user\", \"content\": user_query}\n ])\n resp = await self._llm.complete(messages_text)\n content = resp[\"message\"][\"content\"].strip()\n \n # Extract JSON\n code_block_pattern = r'```(?:json)?\\s*\\n?(.*?)\\n?```'\n match = re.search(code_block_pattern, content, re.DOTALL)\n if match:\n content = match.group(1).strip()\n else:\n json_match = re.search(r'\\{.*\\}', content, re.DOTALL)\n if json_match:\n content = json_match.group()\n \n try:\n result = json.loads(content)\n except json.JSONDecodeError as e:\n logger.warning(f\"Failed to parse LLM response: {e}\")\n return [], tools\n \n # Parse utility_tools: {server: [tool_names]}\n utility_tools_config = result.get(\"utility_tools\", {})\n domain_servers = set(result.get(\"domain_servers\", []))\n brief_plan = result.get(\"brief_plan\", \"N/A\")\n \n logger.info(f\"LLM Planning: {brief_plan}\")\n logger.info(f\"Utility tools: {utility_tools_config}\")\n logger.info(f\"Domain servers: {domain_servers}\")\n \n # Collect utility tools (specific tools only)\n utility_tools = []\n for server_name, tool_names in utility_tools_config.items():\n if server_name in server_tools:\n server_tool_names = {t.name for t in server_tools[server_name]}\n for tool_name in tool_names:\n if tool_name in server_tool_names and tool_name in tool_name_map:\n utility_tools.append(tool_name_map[tool_name])\n \n # Collect domain tools (entire servers)\n domain_tools = []\n for server, tool_list in server_tools.items():\n if server in domain_servers:\n domain_tools.extend(tool_list)\n \n logger.info(f\"LLM filter result: {len(utility_tools)} utility tools, {len(domain_tools)} domain tools\")\n \n # Build LLM filter info for debugging\n llm_filter_info = {\n \"brief_plan\": brief_plan,\n \"utility_tools\": utility_tools_config,\n \"domain_servers\": list(domain_servers),\n }\n \n # Fallback if no match\n if not utility_tools and not domain_tools:\n logger.warning(f\"LLM filter matched 0 tools, returning all as domain\")\n return [], tools, llm_filter_info\n \n return utility_tools, domain_tools, llm_filter_info\n\n async def _generate_search_query(self, task_prompt: str) -> str:\n prompt = f\"\"\"Task: {task_prompt}\n\nList keywords for the capabilities needed (comma-separated, brief):\"\"\"\n\n resp = await self._llm.complete(prompt)\n capabilities = resp[\"message\"][\"content\"].strip().replace(\"\\n\", \" \")\n \n enhanced_query = f\"{task_prompt} {capabilities}\"\n logger.debug(f\"Enhanced search query: {enhanced_query[:150]}...\")\n \n return enhanced_query\n\n def _log_search_results(self, all_tools: list[BaseTool], filtered_tools: list[BaseTool], mode: str) -> None:\n \"\"\"\n Log search results in a concise, grouped format.\n Shows backend/server breakdown and tool names (truncated if too many).\n \"\"\"\n from collections import defaultdict\n \n # Group filtered tools by backend and server\n grouped: Dict[str, Dict[str | None, list[str]]] = defaultdict(lambda: defaultdict(list))\n \n for t in filtered_tools:\n # Get backend and server info\n if t.is_bound:\n backend = t.runtime_info.backend.value\n server = t.runtime_info.server_name if backend.lower() == \"mcp\" else None\n else:\n if not t.backend_type or t.backend_type == BackendType.NOT_SET:\n backend = \"UNKNOWN\"\n server = None\n else:\n backend = t.backend_type.value\n server = None\n \n grouped[backend][server].append(t.name)\n \n # Build concise summary\n lines = [f\"\\n{'='*60}\"]\n lines.append(f\"๐ Tool Search Results (mode: {mode})\")\n lines.append(f\" {len(all_tools)} candidates โ {len(filtered_tools)} selected tools\")\n lines.append(f\"{'='*60}\")\n \n for backend, srv_map in sorted(grouped.items()):\n backend_total = sum(len(tools) for tools in srv_map.values())\n lines.append(f\"\\n๐ฆ {backend} ({backend_total} tools)\")\n \n for server, tool_names in sorted(srv_map.items()):\n if backend.lower() == \"mcp\" and server:\n prefix = f\" โโ {server}: \"\n else:\n prefix = f\" โโ \"\n \n # Limit display to avoid overwhelming output\n if len(tool_names) <= 8:\n tools_display = \", \".join(tool_names)\n else:\n tools_display = \", \".join(tool_names[:8]) + f\" ... (+{len(tool_names)-8} more)\"\n \n lines.append(f\"{prefix}{tools_display}\")\n \n lines.append(f\"{'='*60}\\n\")\n \n # Use info level so users can see it\n logger.info(\"\\n\".join(lines))\n\n @staticmethod\n def _format_tool_list(tools: list[BaseTool]) -> str:\n rows = [f\"{i}. **{t.name}**: {t.description}\" for i, t in enumerate(tools, 1)]\n return f\"Total {len(tools)} tools, list out directly:\\n\\n\" + \"\\n\".join(rows)\n\n @staticmethod\n def _format_ranked(results: list[tuple[BaseTool, float]], mode: SearchMode) -> str:\n lines = [f\"Search results (mode={mode}) total {len(results)}:\\n\"]\n for i, (tool, score) in enumerate(results, 1):\n lines.append(f\"{i}. {tool.name} (score: {score:.3f})\\n {tool.description}\")\n return \"\\n\".join(lines)\n\n def _run(self, *args, **kwargs):\n raise NotImplementedError(\"SearchCoordinator only supports asynchronous calls. Use _arun instead.\")\n \n def get_embedding_cache_stats(self) -> Dict[str, Any]:\n \"\"\"Get statistics about the embedding cache.\n \n Returns:\n Dict with cache statistics including total embeddings and breakdown by backend/server.\n \"\"\"\n return self._ranker.get_cache_stats()\n \n def clear_embedding_cache(self, backend: Optional[str] = None, server: Optional[str] = None) -> int:\n \"\"\"Clear embeddings from cache.\n \n Args:\n backend: If provided, only clear this backend. If None, clear all.\n server: If provided (and backend is provided), only clear this server.\n \n Returns:\n Number of embeddings cleared.\n \"\"\"\n return self._ranker.clear_cache(backend=backend, server=server)\n \n def get_last_search_debug_info(self) -> Optional[Dict[str, Any]]:\n \"\"\"Get debug info from the last search operation.\n \n Returns:\n Dict containing search debug info, or None if no search has been performed.\n Includes:\n - search_mode: The search mode used\n - total_candidates: Total number of candidate tools\n - mcp_count/non_mcp_count: Tool counts by type\n - llm_filter: LLM filter information if used\n - tool_scores: Similarity scores for each tool\n - selected_tools: Final selected tools\n \"\"\"\n if self._last_search_debug_info is None:\n return None\n return self._last_search_debug_info.to_dict()"
},
{
"path": "anytool/grounding/core/security/__init__.py",
"content": "from .sandbox import BaseSandbox, SandboxManager\nfrom .policies import SecurityPolicyManager, SecurityPolicy\n\n# Try to import E2BSandbox (optional dependency)\ntry:\n from .e2b_sandbox import E2BSandbox\n E2B_AVAILABLE = True\nexcept ImportError:\n E2BSandbox = None\n E2B_AVAILABLE = False\n\n__all__ = [\n \"BaseSandbox\",\n \"SandboxManager\",\n \"SecurityPolicyManager\",\n \"SecurityPolicy\"\n]\n\nif E2B_AVAILABLE:\n __all__.append(\"E2BSandbox\")"
},
{
"path": "anytool/grounding/core/security/e2b_sandbox.py",
"content": "\"\"\"\nE2B Sandbox implementation.\n\nThis module provides a concrete implementation of BaseSandbox using E2B.\n\"\"\"\n\nimport os\nfrom typing import Any, Dict, Optional, TYPE_CHECKING\n\nfrom anytool.utils.logging import Logger\nfrom .sandbox import BaseSandbox\nfrom ..types import SandboxOptions\n\nlogger = Logger.get_logger(__name__)\n\n# Import E2B SDK components (optional dependency)\nif TYPE_CHECKING:\n # For type checking purposes only\n try:\n from e2b_code_interpreter import CommandHandle, Sandbox\n except ImportError:\n CommandHandle = None # type: ignore\n Sandbox = None # type: ignore\n\ntry:\n logger.debug(\"Attempting to import e2b_code_interpreter...\")\n from e2b_code_interpreter import ( # type: ignore\n CommandHandle,\n Sandbox,\n )\n logger.debug(\"Successfully imported e2b_code_interpreter\")\n E2B_AVAILABLE = True\nexcept ImportError as e:\n logger.debug(f\"Failed to import e2b_code_interpreter: {e}\")\n CommandHandle = None # type: ignore\n Sandbox = None # type: ignore\n E2B_AVAILABLE = False\n\n\nclass E2BSandbox(BaseSandbox):\n \"\"\"E2B sandbox implementation for secure code execution.\"\"\"\n \n def __init__(self, options: SandboxOptions):\n \"\"\"Initialize E2B sandbox.\n \n Args:\n options: Sandbox configuration options including:\n - api_key: E2B API key (or use E2B_API_KEY env var)\n - sandbox_template_id: Template ID for the sandbox (default: \"base\")\n - timeout: Command execution timeout in seconds\n \"\"\"\n super().__init__(options)\n \n if not E2B_AVAILABLE:\n raise ImportError(\n \"E2B SDK (e2b-code-interpreter) not found. Please install it with \"\n \"'pip install e2b-code-interpreter'.\"\n )\n \n # Get API key from options or environment\n self.api_key = options.get(\"api_key\") or os.environ.get(\"E2B_API_KEY\")\n if not self.api_key:\n raise ValueError(\n \"E2B API key is required. Provide it via 'options.api_key'\"\n \" or the E2B_API_KEY environment variable.\"\n )\n \n # Get sandbox configuration\n self.sandbox_template_id = options.get(\"sandbox_template_id\", \"base\")\n self.timeout = options.get(\"timeout\", 600) # Default 10 minutes\n \n # Sandbox instance (using Any to avoid import issues with optional dependency)\n self._sandbox: Any = None\n self._process: Any = None\n \n async def start(self) -> bool:\n \"\"\"Start the E2B sandbox instance.\n \n Returns:\n True if sandbox started successfully, False otherwise.\n \"\"\"\n if self._active:\n logger.debug(\"E2B sandbox already active\")\n return True\n \n try:\n logger.debug(f\"Creating E2B sandbox with template: {self.sandbox_template_id}\")\n self._sandbox = Sandbox(\n template=self.sandbox_template_id,\n api_key=self.api_key,\n )\n self._active = True\n logger.info(f\"E2B sandbox started successfully (template: {self.sandbox_template_id})\")\n return True\n \n except Exception as e:\n logger.error(f\"Failed to start E2B sandbox: {e}\")\n self._active = False\n return False\n \n async def stop(self) -> None:\n \"\"\"Stop the E2B sandbox instance.\"\"\"\n if not self._active:\n logger.debug(\"E2B sandbox not active\")\n return\n \n try:\n # Terminate any running process\n if self._process:\n try:\n logger.debug(\"Terminating sandbox process\")\n self._process.kill()\n except Exception as e:\n logger.warning(f\"Error terminating sandbox process: {e}\")\n finally:\n self._process = None\n \n # Close the sandbox\n if self._sandbox:\n try:\n logger.debug(\"Closing E2B sandbox instance\")\n self._sandbox.kill()\n logger.info(\"E2B sandbox stopped successfully\")\n except Exception as e:\n logger.warning(f\"Error closing E2B sandbox: {e}\")\n finally:\n self._sandbox = None\n \n self._active = False\n \n except Exception as e:\n logger.error(f\"Error stopping E2B sandbox: {e}\")\n raise\n \n async def execute_safe(self, command: str, **kwargs) -> Any:\n \"\"\"Execute a command safely in the E2B sandbox.\n \n Args:\n command: The command to execute\n **kwargs: Additional options:\n - envs: Environment variables (dict)\n - timeout: Command timeout in milliseconds\n - background: Run in background (bool)\n - on_stdout: Stdout callback function\n - on_stderr: Stderr callback function\n \n Returns:\n CommandHandle object representing the running process\n \"\"\"\n if not self._active or not self._sandbox:\n raise RuntimeError(\"E2B sandbox is not active. Call start() first.\")\n \n try:\n # Extract execution options\n envs = kwargs.get(\"envs\", {})\n timeout = kwargs.get(\"timeout\", self.timeout * 1000) # Convert to ms\n background = kwargs.get(\"background\", False)\n on_stdout = kwargs.get(\"on_stdout\")\n on_stderr = kwargs.get(\"on_stderr\")\n \n logger.debug(f\"Executing command in E2B sandbox: {command}\")\n \n # Execute the command\n self._process = self._sandbox.commands.run(\n command,\n envs=envs,\n timeout=timeout,\n background=background,\n on_stdout=on_stdout,\n on_stderr=on_stderr,\n )\n \n return self._process\n \n except Exception as e:\n logger.error(f\"Failed to execute command in E2B sandbox: {e}\")\n raise\n \n def get_connector(self) -> Any:\n \"\"\"Get the underlying E2B sandbox connector.\n \n Returns:\n The E2B Sandbox instance, or None if not active.\n \"\"\"\n return self._sandbox\n \n def get_host(self, port: int) -> str:\n \"\"\"Get the host URL for a specific port.\n \n Args:\n port: The port number to get the host for\n \n Returns:\n The host URL string\n \n Raises:\n RuntimeError: If sandbox is not active\n \"\"\"\n if not self._active or not self._sandbox:\n raise RuntimeError(\"E2B sandbox is not active. Call start() first.\")\n \n return self._sandbox.get_host(port)\n \n @property\n def sandbox(self) -> Any:\n \"\"\"Get the underlying E2B Sandbox instance.\"\"\"\n return self._sandbox\n \n @property\n def process(self) -> Any:\n \"\"\"Get the current running process handle.\"\"\"\n return self._process\n\n"
},
{
"path": "anytool/grounding/core/security/policies.py",
"content": "import asyncio\nimport sys\nfrom typing import Callable, Awaitable, Dict, Optional\nfrom ..types import SecurityPolicy, BackendType\n\nPromptFunc = Callable[[str], Awaitable[bool]] \n\n\n# ANSI color codes\nclass Colors:\n RESET = \"\\033[0m\"\n BOLD = \"\\033[1m\"\n RED = \"\\033[91m\"\n YELLOW = \"\\033[93m\"\n GREEN = \"\\033[92m\"\n CYAN = \"\\033[96m\"\n GRAY = \"\\033[90m\"\n WHITE = \"\\033[97m\"\n\n\nclass SecurityPolicyManager:\n def __init__(self, prompt: PromptFunc | None = None):\n self._policies: Dict[BackendType, SecurityPolicy] = {}\n self._global_policy: Optional[SecurityPolicy] = None\n self._prompt: PromptFunc | None = prompt or self._default_cli_prompt\n \n async def _default_cli_prompt(self, message: str) -> bool:\n # Clean and professional prompt using unified display\n from anytool.utils.display import Box, BoxStyle, colorize, print_separator\n \n print()\n print_separator(70, 'y', 2)\n print(f\" {colorize('โ ๏ธ Security Policy Warning', color=Colors.RED, bold=True)}\")\n print_separator(70, 'y', 2)\n print(f\" {message}\")\n print_separator(70, 'gr', 2)\n print(f\" {colorize('[y/yes]', color=Colors.GREEN)} Allow | {colorize('[n/no]', color=Colors.RED)} Deny\")\n print_separator(70, 'gr', 2)\n print(f\" {colorize('Your choice:', bold=True)} \", end=\"\", flush=True)\n \n answer = await asyncio.get_running_loop().run_in_executor(None, sys.stdin.readline)\n response = answer.strip().lower() in {\"y\", \"yes\"}\n \n if response:\n print(f\" {colorize('โ Allowed', color=Colors.GREEN)}\\n\")\n else:\n print(f\" {colorize('โ Denied', color=Colors.RED)}\\n\")\n \n return response\n \n def set_global_policy(self, policy: SecurityPolicy) -> None:\n self._global_policy = policy\n \n def set_backend_policy(self, backend_type: BackendType, policy: SecurityPolicy) -> None:\n self._policies[backend_type] = policy\n \n def get_policy(self, backend_type: BackendType) -> SecurityPolicy:\n policy = self._policies.get(backend_type) \n if policy:\n return policy\n \n if self._global_policy:\n return self._global_policy\n \n return SecurityPolicy()\n \n async def _ask_user(self, message: str) -> bool:\n \"\"\"If prompt is provided, ask user for confirmation, otherwise default to deny\"\"\"\n if self._prompt:\n try:\n return await self._prompt(message)\n except Exception:\n return False\n return False\n\n async def check_command_allowed(self, backend_type: BackendType, command: str) -> bool:\n policy = self.get_policy(backend_type)\n\n if policy.check(command=command):\n return True\n\n # Find dangerous tokens\n dangerous_tokens = policy.find_dangerous_tokens(command)\n \n # Extract only lines containing dangerous commands\n lines = command.split('\\n')\n dangerous_lines = []\n for i, line in enumerate(lines):\n line_lower = line.lower()\n if any(token in line_lower for token in dangerous_tokens):\n # Add line number and the line itself\n dangerous_lines.append((i + 1, line.strip()))\n \n # If no specific dangerous lines found but policy failed, show first few lines\n if not dangerous_lines:\n dangerous_lines = [(i + 1, line.strip()) for i, line in enumerate(lines[:5])]\n \n # Format dangerous lines for display (limit to 10 lines)\n max_display_lines = 10\n if len(dangerous_lines) > max_display_lines:\n display_lines = dangerous_lines[:max_display_lines]\n truncated = True\n else:\n display_lines = dangerous_lines\n truncated = False\n \n # Build formatted command display\n formatted_cmd_lines = []\n for line_num, line in display_lines:\n # Truncate very long lines\n if len(line) > 80:\n line = line[:77] + \"...\"\n formatted_cmd_lines.append(f\" L{line_num}: {line}\")\n \n if truncated:\n formatted_cmd_lines.append(\" ... (more lines)\")\n \n formatted_command = '\\n'.join(formatted_cmd_lines)\n \n # Show which dangerous commands were detected\n dangerous_list = ', '.join([f\"{Colors.RED}{tok}{Colors.RESET}\" for tok in dangerous_tokens[:5]])\n \n from anytool.utils.display import Box, BoxStyle, colorize\n \n # Build command box\n box = Box(width=66, style=BoxStyle.SQUARE, color='gr')\n cmd_box = [\n box.top_line(2),\n box.empty_line(2),\n ]\n for line in formatted_cmd_lines:\n cmd_box.append(box.text_line(line, indent=2))\n cmd_box.extend([\n box.empty_line(2),\n box.bottom_line(2)\n ])\n \n message = (\n f\"\\n{colorize('Potentially dangerous command detected', color=Colors.WHITE)}\\n\\n\"\n f\"Backend: {colorize(backend_type.value, color=Colors.CYAN)}\\n\"\n f\"Dangerous commands: {dangerous_list}\\n\\n\"\n f\"Affected lines:\\n\"\n + \"\\n\".join(cmd_box) + \"\\n\\n\"\n f\"{colorize('This command may contain risky operations. Continue?', color=Colors.YELLOW)}\"\n )\n\n return await self._ask_user(message)\n \n async def check_domain_allowed(self, backend_type: BackendType, domain: str) -> bool:\n policy = self.get_policy(backend_type)\n\n if policy.check(domain=domain):\n return True\n\n message = (\n f\"\\n{Colors.WHITE}Unauthorized domain access detected{Colors.RESET}\\n\\n\"\n f\"Backend: {Colors.CYAN}{backend_type.value}{Colors.RESET}\\n\"\n f\"Domain: {Colors.YELLOW}{domain}{Colors.RESET}\\n\\n\"\n f\"{Colors.YELLOW}This domain is not in the allowed list. Continue?{Colors.RESET}\"\n )\n\n return await self._ask_user(message)"
},
{
"path": "anytool/grounding/core/security/sandbox.py",
"content": "from typing import Any, Dict, Optional\nfrom abc import ABC, abstractmethod\n\nfrom ..types import SandboxOptions, BackendType\n\n\nclass BaseSandbox(ABC): \n def __init__(self, options: SandboxOptions):\n self.options = options\n self._active = False\n \n @abstractmethod\n async def start(self) -> bool:\n \"\"\"Set self._active to True\"\"\"\n pass\n \n @abstractmethod\n async def stop(self) -> None:\n \"\"\"Set self._active to False\"\"\"\n pass\n \n @abstractmethod\n async def execute_safe(self, command: str, **kwargs) -> Any:\n pass\n \n @abstractmethod\n def get_connector(self) -> Any:\n pass\n \n @property\n def is_active(self) -> bool:\n return self._active\n\n\nclass SandboxManager:\n def __init__(self):\n self._sandboxes: Dict[BackendType, BaseSandbox] = {}\n \n def register_sandbox(self, backend_type: BackendType, sandbox: BaseSandbox) -> None:\n self._sandboxes[backend_type] = sandbox\n \n def get_sandbox(self, backend_type: BackendType) -> Optional[BaseSandbox]:\n return self._sandboxes.get(backend_type)\n \n async def start_all(self) -> None:\n for sandbox in self._sandboxes.values():\n await sandbox.start()\n \n async def stop_all(self) -> None:\n for sandbox in self._sandboxes.values():\n await sandbox.stop()"
},
{
"path": "anytool/grounding/core/session.py",
"content": "from abc import ABC, abstractmethod\nfrom typing import Any, Dict, List\nfrom datetime import datetime\n\nfrom .tool import BaseTool\nfrom .transport.connectors import BaseConnector\nfrom .types import SessionInfo, SessionStatus, BackendType, ToolResult\nfrom anytool.utils.logging import Logger\n\nlogger = Logger.get_logger(__name__)\n\n\nclass BaseSession(ABC):\n \"\"\"\n Session manager for all backends.\n \"\"\"\n def __init__(\n self,\n connector: BaseConnector,\n *,\n session_id: str,\n backend_type: BackendType | None = None,\n auto_connect: bool = True,\n auto_initialize: bool = True,\n ) -> None:\n self.connector = connector\n self.session_id = session_id\n self.backend_type = backend_type or BackendType.NOT_SET\n self.auto_connect = auto_connect\n self.auto_initialize = auto_initialize\n\n self.status: SessionStatus = SessionStatus.DISCONNECTED\n self.session_info: Dict[str, Any] | None = None\n self._created_at = datetime.utcnow()\n self._last_activity = self._created_at\n self.tools: List[BaseTool] = []\n\n async def __aenter__(self) -> \"BaseSession\":\n if self.auto_connect:\n await self.connect()\n if self.auto_initialize:\n self.session_info = await self.initialize()\n return self\n\n async def __aexit__(self, exc_type, exc_val, exc_tb) -> None:\n \"\"\"Exit the async context manager.\n\n Args:\n exc_type: The exception type, if an exception was raised.\n exc_val: The exception value, if an exception was raised.\n exc_tb: The exception traceback, if an exception was raised.\n \"\"\"\n await self.disconnect()\n\n async def connect(self) -> None:\n if self.connector.is_connected:\n return\n self.status = SessionStatus.CONNECTING\n await self.connector.connect()\n self.status = SessionStatus.CONNECTED\n\n async def disconnect(self) -> None:\n if not self.connector.is_connected:\n return\n await self.connector.disconnect()\n self.status = SessionStatus.DISCONNECTED\n\n @property\n def is_connected(self) -> bool:\n return self.connector.is_connected\n\n @abstractmethod\n async def initialize(self) -> Dict[str, Any]:\n \"\"\"\n Negotiate with the backend, discover tools, etc.\n Return session information (can be an empty dict).\n \n `self.tools` need to be set in this method.\n \"\"\"\n raise NotImplementedError(\"Sub-class must implement this method\")\n \n async def list_tools(self) -> List[BaseTool]:\n \"\"\"\n Return tools discovered during `initialize()`.\n \"\"\"\n if not self.tools:\n self.session_info = await self.initialize()\n return self.tools\n \n async def call_tool(self, tool_name: str, parameters=None) -> ToolResult:\n parameters = parameters or {}\n \n # Ensure tools are initialized before calling\n if not self.tools:\n logger.debug(f\"Tools not initialized for session {self.session_id}, initializing now...\")\n self.session_info = await self.initialize()\n \n tool_map = {t.schema.name: t for t in self.tools}\n if tool_name not in tool_map:\n raise ValueError(f\"Unknown tool: {tool_name}\")\n result = await tool_map[tool_name].arun(**parameters)\n self._touch()\n return result\n \n # Update when a successful call is made\n def _touch(self):\n self._last_activity = datetime.utcnow()\n\n @property\n def info(self) -> SessionInfo:\n return SessionInfo(\n session_id=self.session_id,\n backend_type=getattr(self, \"backend_type\", BackendType.NOT_SET),\n status=self.status,\n created_at=self._created_at,\n last_activity=self._last_activity,\n metadata=self.session_info or {},\n )"
},
{
"path": "anytool/grounding/core/system/__init__.py",
"content": "from .provider import SystemProvider\nfrom .tool import SYSTEM_TOOLS\n\n__all__ = [\n \"SystemProvider\",\n \"SYSTEM_TOOLS\",\n]"
},
{
"path": "anytool/grounding/core/system/provider.py",
"content": "from typing import List, Dict, Any\nfrom ..provider import Provider\nfrom ..types import BackendType, SessionConfig\nfrom ..grounding_client import GroundingClient\nfrom .tool import SYSTEM_TOOLS, _BaseSystemTool\nfrom ..exceptions import GroundingError, ErrorCode\n\n\nclass SystemProvider(Provider):\n \"\"\"\n Provider for system-level query tools\n \"\"\"\n def __init__(self, client: GroundingClient):\n super().__init__(BackendType.SYSTEM, {})\n # Instantiates all system tools\n self._tools: List[_BaseSystemTool] = [tool_cls(client) for tool_cls in SYSTEM_TOOLS]\n\n async def initialize(self): \n self.is_initialized = True\n\n async def create_session(self, session_config: SessionConfig):\n raise GroundingError(\n \"SystemProvider does not support sessions\",\n code=ErrorCode.CONFIG_INVALID,\n )\n\n async def list_tools(self, session_name: str | None = None):\n return self._tools\n\n async def call_tool(\n self,\n session_name: str,\n tool_name: str,\n parameters: Dict[str, Any] | None = None,\n ):\n tool_map = {t.schema.name: t for t in self._tools}\n if tool_name not in tool_map:\n raise GroundingError(\n f\"System tool '{tool_name}' not found\",\n code=ErrorCode.TOOL_NOT_FOUND,\n )\n return await tool_map[tool_name].arun(**(parameters or {}))\n\n async def close_session(self, session_name: str) -> None:\n return"
},
{
"path": "anytool/grounding/core/system/tool.py",
"content": "from ..tool.local_tool import LocalTool\nfrom ..types import BackendType, ToolResult, ToolStatus\nfrom ..grounding_client import GroundingClient\n\n\nclass _BaseSystemTool(LocalTool):\n backend_type = BackendType.SYSTEM\n\n def __init__(self, client: GroundingClient):\n super().__init__(verbose=False, handle_errors=True)\n self._client = client\n\n @property\n def client(self) -> GroundingClient:\n return self._client\n\n\nclass ListProvidersTool(_BaseSystemTool):\n _name = \"list_providers\"\n _description = \"List all registered backend providers\"\n\n async def _arun(self) -> ToolResult:\n prov = list(self.client.list_providers().keys())\n return ToolResult(\n status=ToolStatus.SUCCESS,\n content=\", \".join(prov),\n )\n\n\nclass ListBackendToolsTool(_BaseSystemTool):\n _name = \"list_backend_tools\"\n _description = \"List static tools for a backend\"\n\n async def _arun(self, backend: str) -> ToolResult:\n try:\n be = BackendType(backend.lower())\n except ValueError:\n return ToolResult(ToolStatus.ERROR, error=f\"Unknown backend '{backend}'\")\n\n tools = await self.client.list_backend_tools(be)\n names = [t.schema.name for t in tools]\n return ToolResult(\n status=ToolStatus.SUCCESS,\n content=\", \".join(names),\n )\n\n\nclass ListSessionToolsTool(_BaseSystemTool):\n _name = \"list_session_tools\"\n _description = \"List tools (incl. dynamic) for a session\"\n\n async def _arun(self, session_id: str) -> ToolResult:\n tools = await self.client.list_session_tools(session_id)\n names = [t.schema.name for t in tools]\n return ToolResult(\n status=ToolStatus.SUCCESS,\n content=\", \".join(names),\n )\n\n\nclass ListAllBackendToolsTool(_BaseSystemTool):\n _name = \"list_all_backend_tools\"\n _description = \"List static tools for every registered backend\"\n\n async def _arun(self, use_cache: bool = False) -> ToolResult:\n all_tools = await self.client.list_all_backend_tools(use_cache=use_cache)\n lines = [\n f\"{backend.value}: {', '.join(t.schema.name for t in tools)}\"\n for backend, tools in all_tools.items()\n ]\n return ToolResult(\n status=ToolStatus.SUCCESS,\n content=\"\\n\".join(lines),\n )\n\n\nSYSTEM_TOOLS: list[type[_BaseSystemTool]] = [\n ListProvidersTool,\n ListBackendToolsTool,\n ListSessionToolsTool,\n ListAllBackendToolsTool,\n]"
},
{
"path": "anytool/grounding/core/tool/__init__.py",
"content": "from .base import BaseTool\nfrom .local_tool import LocalTool\nfrom .remote_tool import RemoteTool\n\n__all__ = [\"BaseTool\", \"LocalTool\", \"RemoteTool\"]"
},
{
"path": "anytool/grounding/core/tool/base.py",
"content": "\"\"\"\nBaseTool.\nAll pre-defined grounding atomic operations will inherit this tool class.\nRemoteTool needs to pass in connector.\n\"\"\"\nimport asyncio, time, inspect\nfrom abc import ABC, abstractmethod\nfrom functools import lru_cache\nfrom typing import Any, ClassVar, Dict, Optional, TYPE_CHECKING\nfrom pydantic import BaseModel, ConfigDict, Field, create_model\n\nfrom ..types import BackendType, ToolResult, ToolSchema, ToolStatus\nfrom ..exceptions import GroundingError, ErrorCode\nfrom anytool.utils.logging import Logger\nimport jsonschema\n\nif TYPE_CHECKING:\n from ..grounding_client import GroundingClient\n\nlogger = Logger.get_logger(__name__)\n\n\nclass ToolRuntimeInfo:\n \"\"\"Runtime information for a tool instance\"\"\"\n def __init__(\n self,\n backend: BackendType,\n session_name: str,\n server_name: Optional[str] = None,\n grounding_client: Optional['GroundingClient'] = None,\n ):\n self.backend = backend\n self.session_name = session_name\n self.server_name = server_name\n self.grounding_client = grounding_client\n \n def __repr__(self):\n return f\"\"\n \n\nclass BaseTool(ABC):\n _name: ClassVar[str] = \"\"\n _description: ClassVar[str] = \"\"\n backend_type: ClassVar[BackendType] = BackendType.NOT_SET\n\n def __init__(self,\n schema: Optional[ToolSchema] = None,\n *,\n verbose: bool = False,\n handle_errors: bool = True) -> None:\n self.verbose = verbose\n self.handle_errors = handle_errors\n self.schema: ToolSchema = schema or ToolSchema(\n name=self._name or self.__class__.__name__.lower(),\n description=self._description,\n parameters=self.get_parameters_schema(),\n backend_type=self.backend_type,\n )\n \n self._runtime_info: Optional[ToolRuntimeInfo] = None\n self._disable_outer_recording = True\n \n @property\n def name(self) -> str:\n \"\"\"Get tool name from schema (supports both class-defined and runtime-injected names)\"\"\"\n return self.schema.name if hasattr(self, 'schema') and self.schema else self._name\n \n @property\n def description(self) -> str:\n \"\"\"Get tool description from schema (supports both class-defined and runtime-injected descriptions)\"\"\"\n return self.schema.description if hasattr(self, 'schema') and self.schema else self._description\n\n @classmethod\n @lru_cache\n def get_parameters_schema(cls) -> Dict[str, Any]:\n \"\"\"Auto-generate JSON-schema from _run() or _arun() signature.\n \n Returns empty dict for tools with no parameters.\n Priority: prefer _arun if overridden, otherwise use _run.\n \"\"\"\n # Priority: prefer _arun if it's overridden by subclass, else use _run\n # This allows async-first tools to define their signature via _arun\n sig_src = None\n \n # Check if _arun is overridden (not from BaseTool)\n if cls._arun is not BaseTool._arun:\n sig_src = cls._arun\n # Otherwise check if _run is overridden\n elif cls._run is not BaseTool._run:\n sig_src = cls._run\n # If neither is overridden, raise error\n else:\n raise ValueError(\n f\"{cls.__name__} must implement _run() or _arun() to define its parameters schema\"\n )\n \n sig = inspect.signature(sig_src)\n fields: dict[str, Any] = {}\n for name, p in sig.parameters.items():\n # Skip 'self' and **kwargs / *args\n if name == \"self\" or p.kind in (inspect.Parameter.VAR_KEYWORD, inspect.Parameter.VAR_POSITIONAL):\n continue\n typ = p.annotation if p.annotation is not inspect._empty else str\n default = p.default if p.default is not inspect._empty else ...\n fields[name] = (typ, Field(default))\n \n if not fields:\n return {}\n \n PModel: type[BaseModel] = create_model(\n f\"{cls.__name__}Params\",\n __config__=ConfigDict(arbitrary_types_allowed=True),\n **fields\n )\n return PModel.model_json_schema()\n\n def validate_parameters(self, params: Dict[str, Any]) -> None:\n try:\n self.schema.validate_parameters(params, raise_exc=True)\n except jsonschema.ValidationError as ve:\n raise GroundingError(\n f\"Invalid parameters: {ve.message}\",\n code=ErrorCode.TOOL_EXECUTION_FAIL,\n tool_name=self.schema.name,\n ) from ve\n\n def run(self, **kwargs):\n try:\n return asyncio.run(self.invoke(**kwargs))\n except RuntimeError: # already in running loop\n loop = asyncio.get_running_loop()\n return loop.create_task(self.invoke(**kwargs))\n\n def __call__(self, **kwargs):\n return self.run(**kwargs)\n\n async def __acall__(self, **kwargs):\n return await self.arun(**kwargs)\n \n async def arun(self, **kwargs) -> ToolResult:\n start = time.time()\n try:\n self.validate_parameters(kwargs)\n raw = await self._arun(**kwargs)\n result = self._wrap_result(raw, time.time() - start)\n \n # Auto-record (if enabled)\n await self._auto_record_execution(kwargs, result, time.time() - start)\n \n return result\n except Exception as e: \n if self.handle_errors:\n return ToolResult(\n status=ToolStatus.ERROR,\n error=str(e),\n metadata={\"tool\": self.schema.name},\n )\n raise\n\n # to be implemented by subclasses\n @abstractmethod\n async def _arun(self, **kwargs): ...\n \n def bind_runtime_info(\n self,\n backend: BackendType,\n session_name: str,\n server_name: Optional[str] = None,\n grounding_client: Optional['GroundingClient'] = None,\n ) -> 'BaseTool':\n \"\"\"\n Bind runtime information to the tool instance.\n Allow the tool to be invoked directly without specifying backend/session/server.\n \n Args:\n backend: Backend type\n session_name: Session name\n server_name: Server name (for MCP)\n grounding_client: Optional reference to GroundingClient for direct invocation\n \"\"\"\n self._runtime_info = ToolRuntimeInfo(\n backend=backend,\n session_name=session_name,\n server_name=server_name,\n grounding_client=grounding_client,\n )\n return self\n \n @property\n def runtime_info(self) -> Optional['ToolRuntimeInfo']:\n \"\"\"Get runtime information if bound\"\"\"\n return self._runtime_info\n \n @property\n def is_bound(self) -> bool:\n \"\"\"Check if tool has runtime information bound\"\"\"\n return self._runtime_info is not None\n \n async def invoke(\n self, \n parameters: Dict[str, Any] | None = None, \n keep_session: bool = True,\n **kwargs\n ) -> ToolResult:\n \"\"\"\n Invoke this tool using bound runtime information.\n Requires runtime info to be bound via bind_runtime_info().\n If no runtime info is bound, the tool will be executed locally. \n \"\"\"\n params = parameters or kwargs\n\n if self.is_bound and self._runtime_info.grounding_client:\n return await self._runtime_info.grounding_client.invoke_tool(\n tool=self,\n parameters=params,\n keep_session=keep_session,\n )\n\n return await self.arun(**params)\n\n def _wrap_result(self, obj: Any, elapsed: float) -> ToolResult:\n if isinstance(obj, ToolResult):\n obj.execution_time = elapsed\n return obj\n if self.verbose:\n logger.debug(\"[%s] done in %.2f s\", self.schema.name, elapsed)\n if isinstance(obj, (bytes, bytearray)):\n obj = obj.decode(\"utf-8\", errors=\"replace\")\n return ToolResult(\n status=ToolStatus.SUCCESS,\n content=str(obj),\n execution_time=elapsed,\n metadata={\"tool\": self.schema.name},\n )\n \n async def _auto_record_execution(\n self,\n parameters: Dict[str, Any],\n result: ToolResult,\n execution_time: float,\n ):\n \"\"\"Auto-record tool execution to recording manager and quality manager.\"\"\"\n # Record to quality manager (for quality tracking)\n await self._record_to_quality_manager(result, execution_time * 1000)\n \n # Record to recording manager (for trajectory recording)\n try:\n from anytool.recording import RecordingManager\n \n if not RecordingManager.is_recording():\n return\n \n # Check if tool has disabled outer recording (e.g., GUI agent with intermediate steps)\n if hasattr(self, '_disable_outer_recording') and self._disable_outer_recording:\n logger.debug(f\"Skipping outer recording for {self.schema.name} (intermediate steps recorded)\")\n return\n \n # Get backend and server_name from runtime_info (if bound)\n backend = self.backend_type.value\n server_name = None\n \n if self.is_bound and self._runtime_info:\n # Prefer runtime_info information (more accurate)\n backend = self._runtime_info.backend.value\n server_name = self._runtime_info.server_name\n \n # Get screenshot (if GUI backend)\n screenshot = None\n if self.backend_type == BackendType.GUI and hasattr(self, 'connector'):\n try:\n screenshot = await self.connector.get_screenshot()\n except Exception as e:\n logger.debug(f\"Failed to capture screenshot: {e}\")\n \n # Record tool execution with complete runtime information\n await RecordingManager.record_tool_execution(\n tool_name=self.schema.name,\n backend=backend,\n parameters=parameters,\n result=result.content,\n server_name=server_name,\n is_success=result.is_success, # Pass actual success status from ToolResult\n )\n except Exception as e:\n # Recording failure should not affect tool execution\n logger.debug(f\"Failed to auto-record tool execution: {e}\")\n \n async def _record_to_quality_manager(\n self,\n result: ToolResult,\n execution_time_ms: float,\n ):\n \"\"\"Record execution result to quality manager for quality tracking.\"\"\"\n try:\n from anytool.grounding.core.quality import get_quality_manager\n \n manager = get_quality_manager()\n if manager:\n await manager.record_execution(self, result, execution_time_ms)\n except Exception as e:\n # Quality recording failure should not affect tool execution\n logger.debug(f\"Failed to record to quality manager: {e}\")\n\n # keep _run for backward-compatibility / thread-pool fallback\n def _run(self, **kwargs):\n raise NotImplementedError\n\n def __repr__(self):\n base = f\"\"\n\n def __init_subclass__(cls, **kwargs):\n \"\"\"\n - at least implement _run or _arun\n - backend_type is NOT_SET, only give a warning, allow RemoteTool to inject at runtime\n \"\"\"\n super().__init_subclass__(**kwargs)\n\n if cls._arun is BaseTool._arun and cls._run is BaseTool._run:\n raise ValueError(f\"{cls.__name__} must implement _run() or _arun()\")\n\n if cls.backend_type is BackendType.NOT_SET:\n logger.debug(\n \"%s.backend_type is NOT_SET; remember to override or set at runtime.\",\n cls.__name__,\n )"
},
{
"path": "anytool/grounding/core/tool/local_tool.py",
"content": "\"\"\"\nLocalTool.\nExecutes entirely inside this Python process.\n\"\"\"\nimport asyncio\nfrom typing import Any\nfrom .base import BaseTool\n\n\nclass LocalTool(BaseTool):\n def _run(self, **kwargs): \n raise NotImplementedError\n \n async def _dispatch_run(self, **kwargs) -> Any:\n # Prefer subclass's own _arun if it was overridden\n if self.__class__._arun is not LocalTool._arun:\n return await super()._arun(**kwargs)\n\n # Else fall back to thread-pooled _run if provided\n if self.__class__._run is not LocalTool._run:\n loop = asyncio.get_running_loop()\n return await loop.run_in_executor(None, lambda: self._run(**kwargs))\n\n raise NotImplementedError(\n f\"{self.__class__.__name__} must implement _run() or _arun()\"\n )\n\n async def _arun(self, **kwargs):\n return await self._dispatch_run(**kwargs)"
},
{
"path": "anytool/grounding/core/tool/remote_tool.py",
"content": "\"\"\"\nRemoteTool.\nWrapper around a connector that calls a remote tool.\n\"\"\"\nfrom typing import Optional\nfrom anytool.utils.logging import Logger\nfrom ..types import BackendType, ToolResult, ToolSchema, ToolStatus\nfrom .base import BaseTool\nfrom anytool.grounding.core.transport.connectors import BaseConnector\n\nlogger = Logger.get_logger(__name__)\n\n\nclass RemoteTool(BaseTool):\n backend_type = BackendType.NOT_SET\n\n def __init__(\n self,\n schema: ToolSchema | None = None,\n connector: Optional[BaseConnector] = None,\n remote_name: str = \"\",\n *,\n verbose: bool = False,\n backend: BackendType = BackendType.NOT_SET,\n ):\n self._conn = connector\n self._remote_name = remote_name or (schema.name if schema else \"\")\n self.backend_type = backend\n super().__init__(schema=schema, verbose=verbose)\n\n async def _arun(self, **kwargs):\n # If no connector, tool must be invoked via grounding_client (on-demand startup)\n if self._conn is None:\n raise RuntimeError(\n f\"Tool '{self.name}' has no connector. \"\n \"Use grounding_client.invoke_tool() to execute it with on-demand server startup.\"\n )\n \n raw = await self._conn.invoke(self._remote_name, kwargs)\n \n if hasattr(raw, 'content') and hasattr(raw, 'isError'):\n content_parts = []\n for item in (raw.content or []):\n # Extract text from TextContent\n if hasattr(item, 'text') and item.text:\n content_parts.append(item.text)\n # Handle ImageContent (just note its presence)\n elif hasattr(item, 'data'):\n content_parts.append(f\"[Image data: {len(item.data) if item.data else 0} bytes]\")\n # Handle EmbeddedResource\n elif hasattr(item, 'resource'):\n content_parts.append(f\"[Embedded resource: {getattr(item.resource, 'uri', 'unknown')}]\")\n \n content = \"\\n\".join(content_parts) if content_parts else \"\"\n is_error = getattr(raw, 'isError', False)\n \n return ToolResult(\n status=ToolStatus.ERROR if is_error else ToolStatus.SUCCESS,\n content=content,\n error=content if is_error else None,\n )\n \n # Handle dict response\n if isinstance(raw, dict):\n import json\n try:\n content = json.dumps(raw, ensure_ascii=False, indent=2)\n except (TypeError, ValueError):\n content = str(raw)\n # Handle list/tuple response\n elif isinstance(raw, (list, tuple)):\n import json\n try:\n content = json.dumps(raw, ensure_ascii=False, indent=2)\n except (TypeError, ValueError):\n content = str(raw)\n # Handle primitive types\n elif isinstance(raw, (int, float, bool)):\n content = str(raw)\n elif isinstance(raw, str):\n content = raw\n # Fallback for unknown types\n else:\n content = str(raw)\n \n return ToolResult(\n status=ToolStatus.SUCCESS,\n content=content,\n )"
},
{
"path": "anytool/grounding/core/transport/connectors/__init__.py",
"content": "from .base import BaseConnector\nfrom .aiohttp_connector import AioHttpConnector\n\n__all__ = [\n \"BaseConnector\", \n \"AioHttpConnector\",\n]"
},
{
"path": "anytool/grounding/core/transport/connectors/aiohttp_connector.py",
"content": "from typing import Any\nfrom yarl import URL\nimport aiohttp\n\nfrom ..task_managers import AioHttpConnectionManager\nfrom .base import BaseConnector\nfrom anytool.utils.logging import Logger\nfrom pydantic import BaseModel\n\nlogger = Logger.get_logger(__name__)\n\n\nclass AioHttpConnector(BaseConnector[aiohttp.ClientSession]):\n \"\"\"Generic HTTP-based connector with auto-reconnect & helper methods.\"\"\"\n\n def __init__(self, base_url: str, **session_kw):\n connection_manager = AioHttpConnectionManager(base_url, **session_kw)\n super().__init__(connection_manager)\n self.base_url = base_url.rstrip(\"/\")\n \n async def connect(self) -> None:\n await super().connect()\n try:\n async with self._connection.get(self.base_url, timeout=5) as resp:\n if resp.status >= 500:\n raise ConnectionError(f\"HTTP {resp.status}\")\n except Exception as e:\n await self.disconnect()\n raise ConnectionError(f\"Ping {self.base_url} failed: {e}\")\n\n async def _request(\n self,\n method: str,\n path: str,\n *,\n json: Any | BaseModel | None = None, \n data: Any | None = None,\n params: dict[str, Any] | None = None,\n **kw,\n ) -> aiohttp.ClientResponse:\n if not self.is_connected:\n await self.connect()\n\n assert self._connection is not None # for mypy\n url = URL(self.base_url) / path.lstrip(\"/\")\n logger.debug(\"%s %s\", method.upper(), url)\n return await self._connection.request(\n method.upper(),\n url,\n json=self._to_json_compatible(json), \n data=data,\n params=params,\n **kw,\n )\n\n async def get_json(self, path: str, **kw) -> Any:\n response_model: type[BaseModel] | None = kw.pop(\"response_model\", None)\n resp = await self._request(\"GET\", path, **kw)\n resp.raise_for_status()\n data = await resp.json()\n return self._parse_as(data, response_model)\n\n async def get_bytes(self, path: str, **kw) -> bytes:\n resp = await self._request(\"GET\", path, **kw)\n resp.raise_for_status()\n return await resp.read()\n \n async def post_json(\n self,\n path: str,\n payload: Any | BaseModel,\n *,\n response_model: type[BaseModel] | None = None,\n **kw,\n ) -> Any | BaseModel:\n resp = await self._request(\"POST\", path, json=payload, **kw)\n \n try:\n data = await resp.json()\n except Exception:\n data = None\n \n if resp.status >= 400:\n # Extract detailed error from response body\n detail = \"\"\n if data:\n detail = data.get(\"output\") or data.get(\"message\") or data.get(\"error\") or \"\"\n error_msg = f\"{resp.status}, message='{resp.reason}'\"\n if detail:\n error_msg += f\", detail='{detail}'\"\n raise aiohttp.ClientResponseError(\n resp.request_info,\n resp.history,\n status=resp.status,\n message=error_msg,\n )\n \n return self._parse_as(data, response_model)\n\n async def request(self, method: str, path: str, **kw) -> aiohttp.ClientResponse:\n return await self._request(method, path, **kw)\n\n async def invoke(self, name: str, params: dict[str, Any]) -> Any:\n \"\"\"\n Generic tool-invocation mapping for HTTP back-ends.\n\n name rule (case-insensitive):\n - \"GET /path\" -> GET, return JSON\n - \"GET_TEXT /path\" -> GET, return str\n - \"GET_BYTES /path\" -> GET, return bytes\n - \"POST /path\" -> POST, payload = params (JSON)\n - other -> default POST /{name}, payload = params\n \n If PUT/PATCH/DELETE is needed in the future, it can be reused in _handle_other_json.\n \"\"\"\n verb_path = name.strip().split(maxsplit=1)\n verb = verb_path[0].upper()\n path = verb_path[1] if len(verb_path) == 2 else verb_path[0]\n\n if verb == \"GET_BYTES\":\n return await self.get_bytes(path, params=params)\n\n if verb == \"GET_TEXT\":\n resp = await self._request(\"GET\", path, params=params)\n resp.raise_for_status()\n return await resp.text()\n\n if verb in {\"GET\", \"POST\"} and len(verb_path) == 2:\n if verb == \"GET\":\n return await self.get_json(path, params=params)\n return await self.post_json(path, payload=params)\n\n if verb in {\"PUT\", \"PATCH\", \"DELETE\"} and len(verb_path) == 2:\n return await self._handle_other_json(verb, path, params)\n\n return await self.post_json(name, payload=params)\n\n async def _handle_other_json(self, method: str, path: str, params: dict[str, Any]):\n \"\"\"Fallback implementation for PUT/PATCH/DELETE returning JSON/text, can be overridden by subclasses.\"\"\"\n resp = await self._request(method, path, json=params)\n resp.raise_for_status()\n try:\n return await resp.json()\n except Exception:\n return await resp.text()"
},
{
"path": "anytool/grounding/core/transport/connectors/base.py",
"content": "\"\"\"\nBase connector abstraction.\n\nA connector is a very thin wrapper-class that owns a *connection manager*\n(e.g. AioHttpConnectionManager, AsyncContextConnectionManager, โฆ).\nIt exposes a unified `connect / disconnect / is_connected` lifecycle and\ndefines an abstract `request()` method which concrete back-ends must\nimplement.\n\"\"\"\nimport asyncio\nfrom abc import ABC, abstractmethod\nfrom typing import Any, Generic, TypeVar, Type\nfrom pydantic import BaseModel\nfrom ..task_managers import BaseConnectionManager\n\nT = TypeVar(\"T\") # The object returned by manager.start(): session / connection\n\n\nclass BaseConnector(ABC, Generic[T]):\n \"\"\"\n Generic connector that delegates the heavy lifting to the supplied\n *connection manager*. Concrete subclasses only need to implement\n their own `request()` method.\n \"\"\"\n\n def __init__(self, connection_manager: BaseConnectionManager[T]):\n self._connection_manager = connection_manager # e.g. AioHttpConnectionManager instance\n # The raw connection object returned by the manager, for reusing the established long-term connection\n self._connection: T | None = None\n self._connected = False\n\n async def connect(self) -> None:\n \"\"\"Create the underlying session/connection via the manager.\"\"\"\n if self._connected:\n return\n \n try:\n # Hook: before connection\n await self._before_connect()\n \n # Start the connection manager\n self._connection = await self._connection_manager.start()\n \n # Hook: after connection established\n await self._after_connect()\n \n # Mark as connected\n self._connected = True\n except Exception:\n # Clean up on failure\n await self._cleanup_on_connect_failure()\n raise\n\n async def disconnect(self) -> None:\n \"\"\"Close the session/connection and reset state.\n \n Ensures proper cleanup of all resources including aiohttp sessions.\n \"\"\"\n if not self._connected:\n return\n \n # Hook: before disconnection\n await self._before_disconnect()\n \n # Stop the connection manager\n if self._connection_manager:\n await self._connection_manager.stop()\n self._connection = None\n \n # Hook: after disconnection\n await self._after_disconnect()\n \n self._connected = False\n\n async def _before_connect(self) -> None:\n \"\"\"Hook called before establishing connection. Override in subclasses if needed.\"\"\"\n pass\n\n async def _after_connect(self) -> None:\n \"\"\"Hook called after connection is established. Override in subclasses if needed.\"\"\"\n pass\n\n async def _cleanup_on_connect_failure(self) -> None:\n \"\"\"Hook called when connection fails. Override in subclasses if needed.\"\"\"\n if self._connection_manager:\n try:\n await self._connection_manager.stop()\n except Exception:\n pass\n self._connection = None\n\n async def _before_disconnect(self) -> None:\n \"\"\"Hook called before disconnection. Override in subclasses if needed.\"\"\"\n pass\n\n async def _after_disconnect(self) -> None:\n \"\"\"Hook called after disconnection. Override in subclasses if needed.\"\"\"\n pass\n\n @property\n def is_connected(self) -> bool:\n \"\"\"Return True iff `connect()` has completed successfully.\"\"\"\n return self._connected\n\n @staticmethod\n def _to_json_compatible(obj: Any) -> Any:\n \"\"\"\n Convert a Pydantic BaseModel to a JSON-serialisable dict (by_alias=True).\n Leave all other types unchanged.\n \"\"\"\n if isinstance(obj, BaseModel):\n return obj.model_dump(by_alias=True)\n return obj\n\n @staticmethod\n def _parse_as(data: Any, model_cls: \"Type[BaseModel] | None\" = None) -> Any:\n \"\"\"\n Try to parse *data* into *model_cls* (a subclass of BaseModel). \n If `model_cls` is None or not a subclass of BaseModel, return the original data.\n \"\"\"\n if model_cls is None:\n return data\n if isinstance(model_cls, type) and issubclass(model_cls, BaseModel):\n return model_cls.model_validate(data)\n return data\n \n @abstractmethod\n async def invoke(self, name: str, params: dict[str, Any]) -> Any:\n \"\"\"\n Unified RPC entry for all tools.\n Sub-class maps this to its actual RPC like call_tool / run_cmd.\n \"\"\"\n raise NotImplementedError\n\n @abstractmethod\n async def request(self, *args: Any, **kwargs: Any) -> Any:\n \"\"\"Abstract RPC / HTTP / WS request method to be implemented by child classes.\"\"\"\n raise NotImplementedError(\"This connector has not implemented 'request'\")"
},
{
"path": "anytool/grounding/core/transport/task_managers/__init__.py",
"content": "from .base import BaseConnectionManager\nfrom .aiohttp_connection_manager import AioHttpConnectionManager\nfrom .async_ctx import AsyncContextConnectionManager\nfrom .placeholder import PlaceholderConnectionManager\nfrom .noop import NoOpConnectionManager\n\n__all__ = [\n \"BaseConnectionManager\", \n \"AioHttpConnectionManager\",\n \"AsyncContextConnectionManager\",\n \"PlaceholderConnectionManager\",\n \"NoOpConnectionManager\",\n]"
},
{
"path": "anytool/grounding/core/transport/task_managers/aiohttp_connection_manager.py",
"content": "\"\"\"\nLong-lived aiohttp ClientSession manager based on AsyncContextConnectionManager.\n\nIt keeps a single ClientSession open during the lifetime of a backend\nsession, saving the overhead of creating and closing a TCP connection\nfor every request.\n\"\"\"\nfrom typing import Optional\nimport aiohttp\n\nfrom .async_ctx import AsyncContextConnectionManager\n\n\nclass AioHttpConnectionManager(\n AsyncContextConnectionManager[aiohttp.ClientSession, ...]\n):\n \"\"\"Manage a persistent aiohttp.ClientSession.\"\"\"\n\n def __init__(\n self,\n base_url: str,\n headers: Optional[dict[str, str]] = None,\n timeout: float = 30,\n ):\n self.base_url = base_url.rstrip(\"/\")\n timeout_cfg = aiohttp.ClientTimeout(total=timeout)\n super().__init__(\n aiohttp.ClientSession,\n timeout=timeout_cfg,\n headers=headers or {},\n )\n self._logger.debug(\n \"Init AioHttpConnectionManager base_url=%s timeout=%s\", self.base_url, timeout\n )\n\n async def _establish_connection(self) -> aiohttp.ClientSession:\n \"\"\"Create and enter the aiohttp.ClientSession context.\"\"\"\n session = await super()._establish_connection()\n self._logger.debug(\"aiohttp ClientSession created\")\n return session\n\n async def _close_connection(self) -> None:\n \"\"\"Close the session and then call the parent cleanup.\n \n Ensures proper cleanup even if close() fails.\n \"\"\"\n if self._ctx:\n try:\n self._logger.debug(\"Closing aiohttp ClientSession\")\n await self._ctx.close()\n # Give aiohttp time to finish its internal cleanup callbacks\n import asyncio\n await asyncio.sleep(0.1)\n except Exception as e:\n self._logger.warning(f\"Error closing aiohttp ClientSession: {e}\")\n await super()._close_connection()"
},
{
"path": "anytool/grounding/core/transport/task_managers/async_ctx.py",
"content": "\"\"\"\nGeneric connection manager based on an *async context manager*.\nGive it any factory that returns an asyncโcontext-manager. \n\"\"\"\nimport sys\nfrom typing import Any, Callable, Generic, Optional, ParamSpec, TypeVar\nfrom .base import BaseConnectionManager\n\n# BaseExceptionGroup only exists in Python 3.11+\nif sys.version_info >= (3, 11):\n _BaseExceptionGroup = BaseExceptionGroup\nelse:\n # Dummy class for older Python versions\n class _BaseExceptionGroup(Exception):\n pass\n\nT = TypeVar(\"T\") # Return type of the async context\nP = ParamSpec(\"P\") # Parameter specification of the factory\n\n\nclass AsyncContextConnectionManager(Generic[T, P], BaseConnectionManager[T]):\n def __init__(self,\n ctx_factory: Callable[P, Any],\n *args: P.args,\n **kwargs: P.kwargs):\n super().__init__()\n self._factory = ctx_factory\n self._factory_args = args\n self._factory_kwargs = kwargs\n self._ctx: Optional[Any] = None \n\n async def _establish_connection(self) -> T:\n \"\"\"Create the context manager and enter it.\"\"\"\n self._logger.debug(\"Creating context via %s\", self._factory.__name__)\n try:\n self._ctx = self._factory(*self._factory_args, **self._factory_kwargs)\n result: T = await self._ctx.__aenter__()\n self._logger.debug(\"Context %s entered successfully\", self._factory.__name__)\n return result\n except Exception as e:\n # Check if this is a benign ExceptionGroup/TaskGroup error\n # These occur during concurrent initialization and cleanup\n error_msg = str(e).lower()\n is_taskgroup_error = (\n \"unhandled errors in a taskgroup\" in error_msg or\n \"cancel scope in a different task\" in error_msg or\n \"exceptiongroup\" in type(e).__name__.lower()\n )\n \n if is_taskgroup_error:\n # This is a benign race condition during concurrent connection setup\n # Log at debug level and re-raise to trigger retry logic\n self._logger.debug(\n f\"Benign TaskGroup race condition during {self._factory.__name__} connection: {type(e).__name__}\"\n )\n # Clean up the partially created context\n if self._ctx is not None:\n try:\n await self._ctx.__aexit__(None, None, None)\n except Exception:\n pass # Ignore cleanup errors\n self._ctx = None\n raise\n else:\n # Real error - log at error level\n self._logger.error(f\"Error establishing connection via {self._factory.__name__}: {e}\")\n raise\n\n async def _close_connection(self) -> None:\n \"\"\"Exit the context manager if it exists.\n \n Uses try-finally to ensure ctx is cleared even if __aexit__ fails.\n This prevents resource leaks when cleanup encounters errors.\n \"\"\"\n if self._ctx is not None:\n try:\n self._logger.debug(\"Exiting context %s\", self._factory.__name__)\n \n # Give subprocesses a moment to flush buffers before closing\n import asyncio\n await asyncio.sleep(0.05)\n \n # Try to exit the context, but catch all possible exceptions\n try:\n await self._ctx.__aexit__(None, None, None)\n except BaseException as e:\n # Catch absolutely everything including SystemExit, KeyboardInterrupt, etc.\n # Check if it's a benign error\n benign_error_types = (\n BrokenPipeError, ConnectionResetError, ValueError, \n OSError, IOError, ProcessLookupError, RuntimeError,\n GeneratorExit\n )\n \n is_benign = False\n \n # Check direct exception type\n if isinstance(e, benign_error_types):\n is_benign = True\n # Check for BaseExceptionGroup (Python 3.11+)\n elif hasattr(e, 'exceptions'):\n # It's an exception group, check all sub-exceptions\n is_benign = all(isinstance(sub_e, benign_error_types) for sub_e in e.exceptions)\n \n if is_benign:\n self._logger.debug(f\"Benign cleanup error for {self._factory.__name__}: {type(e).__name__}\")\n else:\n self._logger.warning(f\"Error during context exit for {self._factory.__name__}: {type(e).__name__}: {e}\")\n \n # Don't re-raise - we want cleanup to complete\n \n except Exception as e:\n # Catch any other unexpected errors in the outer try block\n self._logger.warning(f\"Unexpected error during cleanup for {self._factory.__name__}: {e}\")\n finally:\n self._ctx = None "
},
{
"path": "anytool/grounding/core/transport/task_managers/base.py",
"content": "\"\"\"\nBase connection manager for all backend connectors.\n\nThis module provides an abstract base class for different types of connection\nmanagers used in all backend connectors.\n\nFlow: start() โ launch_connection_task() โ call subclass _establish_connection() โ notify ready โ maintain connection until stop() โ call subclass _close_connection() โ cleanup\n\"\"\"\nimport asyncio\nfrom abc import ABC, abstractmethod\nfrom typing import Generic, TypeVar\nfrom anytool.utils.logging import Logger\n\nT = TypeVar(\"T\")\n\n\nclass BaseConnectionManager(Generic[T], ABC):\n \"\"\"Abstract base class for connection managers.\n\n This class defines the interface for different types of connection managers\n used with all backend connectors.\n \"\"\"\n\n def __init__(self):\n \"\"\"Initialize a new connection manager.\"\"\"\n self._ready_event = asyncio.Event()\n self._done_event = asyncio.Event()\n self._exception: Exception | None = None\n self._connection: T | None = None\n self._task: asyncio.Task | None = None\n self._logger = Logger.get_logger(f\"{__name__}.{self.__class__.__name__}\")\n\n @abstractmethod\n async def _establish_connection(self) -> T:\n \"\"\"Establish the connection.\n\n This method should be implemented by subclasses to establish\n the specific type of connection needed.\n\n Returns:\n The established connection.\n\n Raises:\n Exception: If connection cannot be established.\n \"\"\"\n pass\n\n @abstractmethod\n async def _close_connection(self) -> None:\n \"\"\"Close the connection.\n\n This method should be implemented by subclasses to close\n the specific type of connection.\n\n \"\"\"\n pass\n\n async def start(self, timeout: float | None = None) -> T:\n \"\"\"Start the connection manager and establish a connection.\n\n Args:\n timeout: Optional timeout in seconds. If None, waits indefinitely.\n If specified, will cancel the background task on timeout.\n\n Returns:\n The established connection.\n\n Raises:\n TimeoutError: If connection establishment times out.\n Exception: If connection cannot be established.\n \"\"\"\n # Reset state\n self._ready_event.clear()\n self._done_event.clear()\n self._exception = None\n\n # Create a task to establish and maintain the connection\n self._task = asyncio.create_task(self._connection_task(), name=f\"{self.__class__.__name__}_task\")\n\n # Wait for the connection to be ready or fail (with optional timeout)\n try:\n if timeout is not None:\n await asyncio.wait_for(self._ready_event.wait(), timeout=timeout)\n else:\n await self._ready_event.wait()\n except asyncio.TimeoutError:\n # Timeout! Cancel the background task\n self._logger.warning(f\"Connection establishment timed out after {timeout}s, cancelling...\")\n if self._task and not self._task.done():\n self._task.cancel()\n try:\n await asyncio.wait_for(self._task, timeout=2.0) # Give it 2s to cleanup\n except (asyncio.CancelledError, asyncio.TimeoutError):\n pass\n except Exception as e:\n self._logger.debug(f\"Error during task cancellation: {e}\")\n raise TimeoutError(f\"Connection establishment timed out after {timeout}s\")\n\n # If there was an exception, raise it\n if self._exception:\n # Check if this is a benign TaskGroup race condition\n error_msg = str(self._exception).lower()\n is_benign_taskgroup_error = (\n \"unhandled errors in a taskgroup\" in error_msg or\n \"cancel scope in a different task\" in error_msg or\n \"exceptiongroup\" in type(self._exception).__name__.lower()\n )\n \n if is_benign_taskgroup_error:\n # Log as debug - this is expected and will be retried\n self._logger.debug(f\"Benign TaskGroup race condition, will retry: {type(self._exception).__name__}\")\n else:\n # Real error - log at error level\n self._logger.error(f\"Failed to start connection: {self._exception}\")\n \n raise self._exception\n\n # Return the connection\n if self._connection is None:\n error_msg = \"Connection was not established\"\n self._logger.error(error_msg)\n raise RuntimeError(error_msg)\n \n self._logger.info(\"Connection manager started successfully\")\n return self._connection\n\n async def stop(self, timeout: float = 5.0) -> None:\n \"\"\"Stop the connection manager and close the connection.\n \n Args:\n timeout: Maximum time to wait for cleanup (default 5s).\n \n Ensures all async resources (including aiohttp sessions) are properly closed.\n \"\"\"\n if self._task and not self._task.done():\n self._task.cancel()\n try:\n await asyncio.wait_for(self._task, timeout=timeout)\n except asyncio.TimeoutError:\n self._logger.warning(f\"Task cleanup timed out after {timeout}s\")\n except asyncio.CancelledError:\n pass # Expected\n except Exception as e:\n self._logger.warning(f\"Error stopping task: {e}\")\n\n # Wait for the connection to be done (with timeout)\n try:\n await asyncio.wait_for(self._done_event.wait(), timeout=timeout)\n except asyncio.TimeoutError:\n self._logger.warning(f\"Done event wait timed out after {timeout}s\")\n \n self._logger.info(\"Connection manager stopped\")\n\n def get_streams(self) -> T | None:\n \"\"\"Get the current connection streams.\n\n Returns:\n The current connection (typically a tuple of read_stream, write_stream) or None if not connected.\n \"\"\"\n return self._connection\n\n async def _connection_task(self) -> None:\n \"\"\"Run the connection task.\n\n This task establishes and maintains the connection until cancelled.\n \"\"\"\n try:\n # Establish the connection\n self._connection = await self._establish_connection()\n self._logger.debug(\"Connection established\")\n\n # Signal that the connection is ready\n self._ready_event.set()\n\n # Wait indefinitely until cancelled\n try:\n await asyncio.Event().wait()\n except asyncio.CancelledError:\n raise\n\n except asyncio.CancelledError:\n raise\n except Exception as e:\n # Store the exception\n self._exception = e\n \n # Check if this is a benign TaskGroup race condition\n error_msg = str(e).lower()\n is_benign_taskgroup_error = (\n \"unhandled errors in a taskgroup\" in error_msg or\n \"cancel scope in a different task\" in error_msg or\n \"exceptiongroup\" in type(e).__name__.lower()\n )\n \n if is_benign_taskgroup_error:\n # Log as debug - this is expected during concurrent connection setup\n self._logger.debug(f\"Benign TaskGroup race condition in connection task: {type(e).__name__}\")\n else:\n # Real error - log at error level\n self._logger.error(f\"Connection task failed: {e}\")\n \n # Signal that the connection is ready (with error)\n self._ready_event.set()\n\n finally:\n # Close the connection if it was established\n if self._connection is not None:\n try:\n await self._close_connection()\n except Exception as e:\n self._logger.warning(f\"Error closing connection: {e}\")\n self._connection = None\n\n # Signal that the connection is done\n self._done_event.set()"
},
{
"path": "anytool/grounding/core/transport/task_managers/noop.py",
"content": "\"\"\"No-op connection manager for local (in-process) connectors.\n\nLocal connectors execute commands directly via subprocess, so they don't\nneed a real network connection. This manager satisfies the\nBaseConnectionManager interface that BaseConnector requires.\n\"\"\"\nimport asyncio\nfrom typing import Any\nfrom .base import BaseConnectionManager\n\n\nclass NoOpConnectionManager(BaseConnectionManager[Any]):\n \"\"\"Connection manager that immediately reports 'ready' without\n establishing any real connection.\n \n Used by LocalShellConnector and LocalGUIConnector.\n \"\"\"\n\n async def _establish_connection(self) -> Any:\n \"\"\"No-op: return a sentinel value.\"\"\"\n return True\n\n async def _close_connection(self) -> None:\n \"\"\"No-op: nothing to close.\"\"\"\n pass\n\n"
},
{
"path": "anytool/grounding/core/transport/task_managers/placeholder.py",
"content": "from typing import Any\nfrom .base import BaseConnectionManager\n\n\nclass PlaceholderConnectionManager(BaseConnectionManager[Any]):\n \"\"\"A placeholder connection manager that does nothing.\n \n This is used by connectors that set up their real connection manager\n during the connect() phase.\n \"\"\"\n \n async def _establish_connection(self) -> Any:\n \"\"\"Establish the connection (placeholder implementation).\"\"\"\n raise NotImplementedError(\"PlaceholderConnectionManager should be replaced before use\")\n \n async def _close_connection(self) -> None:\n \"\"\"Close the connection (placeholder implementation).\"\"\"\n pass"
},
{
"path": "anytool/grounding/core/types.py",
"content": "from enum import Enum\nfrom datetime import datetime\nfrom typing import Any, Dict, Generic, List, TypeVar, Optional\nimport jsonschema\nfrom pydantic import BaseModel, Field, ConfigDict\n\n# Pydantic v2 compatibility\ntry:\n from pydantic import RootModel\n PYDANTIC_V2 = True\nexcept ImportError:\n PYDANTIC_V2 = False\n\n\nclass BackendType(str, Enum):\n MCP = \"mcp\"\n SHELL = \"shell\"\n WEB = \"web\"\n GUI = \"gui\"\n SYSTEM = \"system\"\n NOT_SET = \"not_set\"\n\n\nclass ToolStatus(str, Enum):\n SUCCESS = \"success\"\n ERROR = \"error\"\n\n\nclass SessionStatus(str, Enum):\n CONNECTED = \"connected\"\n DISCONNECTED = \"disconnected\"\n CONNECTING = \"connecting\"\n \n \nProgressToken = str | int\nRequestId = str | int\n\nRequestParamsT = TypeVar(\"RequestParamsT\", bound=BaseModel | Dict[str, Any] | None)\nNotificationParamsT = TypeVar(\"NotificationParamsT\", bound=BaseModel | Dict[str, Any] | None)\nMethodT = TypeVar(\"MethodT\", bound=str)\n\n\nclass BaseEntity(BaseModel):\n metadata: Dict[str, Any] = Field(default_factory=dict)\n model_config = ConfigDict(extra=\"allow\")\n\n\nclass JsonRpcBase(BaseEntity):\n jsonrpc: str = \"2.0\"\n\n\nclass RpcMessage(JsonRpcBase, Generic[MethodT, RequestParamsT]):\n method: MethodT\n params: RequestParamsT\n\n\nclass Request(RpcMessage[MethodT, RequestParamsT]):\n id: RequestId | None = None # id is None means Notification\n\n\nclass Notification(RpcMessage[MethodT, NotificationParamsT]):\n pass\n\n\nclass Result(JsonRpcBase):\n pass\n\n\nclass ErrorData(BaseEntity):\n code: int\n message: str\n data: Any | None = None\n\n\nclass ToolResult(Result):\n \"\"\"Tool execution result\"\"\"\n status: ToolStatus\n content: Any = \"\"\n error: ErrorData | str | None = None\n execution_time: float | None = None\n\n @property\n def is_success(self) -> bool: return self.status == ToolStatus.SUCCESS\n \n @property\n def is_error(self) -> bool: return self.status == ToolStatus.ERROR\n\n\nclass SecurityPolicy(BaseEntity):\n allow_shell_commands: bool = True\n allow_network_access: bool = True\n allow_file_access: bool = True\n allowed_domains: List[str] = Field(default_factory=list)\n blocked_commands: List[str] = Field(default_factory=list)\n sandbox_enabled: bool = False\n \n @classmethod\n def from_dict(cls, data: Dict) -> \"SecurityPolicy\":\n \"\"\"\n Create SecurityPolicy from configuration dict.\n \n Supports two formats for blocked_commands:\n 1. List format (applies to all OS): [\"cmd1\", \"cmd2\"]\n 2. Dict format (OS-specific):\n {\n \"common\": [\"cmd1\", \"cmd2\"],\n \"linux\": [\"cmd3\"],\n \"darwin\": [\"cmd4\"],\n \"windows\": [\"cmd5\"]\n }\n \n When using dict format, merges 'common' commands with current OS-specific commands.\n \"\"\"\n import sys\n import platform\n \n processed_data = {}\n for k, v in data.items():\n if k not in cls.model_fields:\n continue\n \n # Special handling for blocked_commands\n if k == \"blocked_commands\":\n if isinstance(v, dict):\n # Dict format: merge common + OS-specific\n blocked_list = list(v.get(\"common\", []))\n \n # Determine current OS\n system = sys.platform\n if system.startswith(\"linux\"):\n os_key = \"linux\"\n elif system == \"darwin\":\n os_key = \"darwin\"\n elif system.startswith(\"win\"):\n os_key = \"windows\"\n else:\n os_key = None\n \n # Merge OS-specific commands\n if os_key and os_key in v:\n blocked_list.extend(v[os_key])\n \n processed_data[k] = blocked_list\n elif isinstance(v, list):\n # List format: use as-is\n processed_data[k] = v\n else:\n # Invalid format, use empty list\n processed_data[k] = []\n else:\n processed_data[k] = v\n \n return cls(**processed_data)\n\n def check(self, *, command: str | None = None, domain: str | None = None) -> bool:\n \"\"\"\n return True if allowed, False if denied.\n Command check uses token-level matching to prevent simple space/escape bypasses.\n \"\"\"\n import shlex\n\n # Shell / Python command check\n if command:\n if not self.allow_shell_commands:\n return False\n\n tokens = [t.lower() for t in shlex.split(command, posix=True)]\n blocked_set = {b.lower() for b in self.blocked_commands}\n if any(tok in blocked_set for tok in tokens):\n return False\n\n # Network access check\n if domain:\n if not self.allow_network_access:\n return False\n if self.allowed_domains and domain not in self.allowed_domains:\n return False\n\n return True\n\n def find_dangerous_tokens(self, command: str) -> List[str]:\n \"\"\"\n Find and return all dangerous tokens in the command.\n Returns empty list if no dangerous tokens found.\n \"\"\"\n import shlex\n \n if not command:\n return []\n \n try:\n tokens = [t.lower() for t in shlex.split(command, posix=True)]\n except ValueError:\n # If shlex.split fails, fall back to simple split\n tokens = [t.lower() for t in command.split()]\n \n blocked_set = {b.lower() for b in self.blocked_commands}\n dangerous = [tok for tok in tokens if tok in blocked_set]\n \n return dangerous\n\n\nclass ToolSchema(BaseEntity):\n name: str\n description: str | None = None\n parameters: Dict[str, Any] = Field(default_factory=dict) # JSON Schema, optional\n return_schema: Dict[str, Any] = Field(default_factory=dict)\n examples: List[dict] = Field(default_factory=list)\n usage_hint: str | None = None\n latency_hint: str | None = None\n backend_type: BackendType\n security_policy: SecurityPolicy | None = None\n\n def validate_parameters(self, params: Dict[str, Any], *, raise_exc: bool = False) -> bool:\n \"\"\"use jsonschema to validate parameters\n \n Returns True if parameters are valid or if tool has no parameters.\n \"\"\"\n # If tool has no parameters defined and no parameters are provided, validation passes\n if not self.parameters and not params:\n return True\n \n # If tool has no parameters defined but parameters are provided, validation fails\n if not self.parameters and params:\n if raise_exc:\n raise ValueError(f\"Tool '{self.name}' does not accept any parameters, but got: {list(params.keys())}\")\n return False\n \n try:\n jsonschema.validate(params, self.parameters)\n return True\n except jsonschema.ValidationError:\n if raise_exc:\n raise\n return False\n\n def is_allowed(self, *, command: str | None = None, domain: str | None = None) -> bool:\n \"\"\"check security policy\"\"\"\n return self.security_policy.check(command=command, domain=domain) if self.security_policy else True\n\n\nclass SessionConfig(BaseEntity):\n session_name: str\n backend_type: BackendType\n connection_params: Dict[str, Any] = Field(default_factory=dict)\n timeout: int = 30\n max_retries: int = 3\n auto_reconnect: bool = True\n auto_connect: bool = True\n health_check_interval: int = 5\n custom_settings: Dict[str, Any] = Field(default_factory=dict)\n\n\nclass SessionInfo(SessionConfig):\n status: SessionStatus\n created_at: datetime\n last_activity: datetime\n\n\nclass SandboxOptions(BaseEntity):\n api_key: str\n \"\"\"Direct API key for sandbox provider (e.g., E2B API key).\n If not provided, will use E2B_API_KEY environment variable.\"\"\"\n \n sandbox_template_id: Optional[str] = None\n \"\"\"Template ID for the sandbox environment.\n Default: 'base'\"\"\"\n\n supergateway_command: Optional[str] = None\n \"\"\"Command to run supergateway.\n Default: 'npx -y supergateway'\"\"\"\n\n\n# ClientMessage: Only available in Pydantic v2\nif PYDANTIC_V2:\n class ClientMessage(\n RootModel[\n Request[Any, str] | Notification[Any, str]\n ]\n ):\n \"\"\"\n Unified deserialization entry: `ClientMessage.model_validate_json(raw_bytes)`\n \"\"\"\nelse:\n # Pydantic v1 fallback: not used in current codebase\n ClientMessage = None # type: ignore"
},
{
"path": "anytool/llm/__init__.py",
"content": "from .client import LLMClient"
},
{
"path": "anytool/llm/client.py",
"content": "import litellm\nimport json\nimport asyncio\nimport time\nfrom typing import List, Sequence, Union, Dict, Optional\nfrom dotenv import load_dotenv\nfrom openai.types.chat import ChatCompletionToolParam\n\nfrom anytool.grounding.core.types import ToolSchema, ToolResult, ToolStatus\nfrom anytool.grounding.core.tool import BaseTool\nfrom anytool.utils.logging import Logger\n\nload_dotenv()\n\n# Disable LiteLLM verbose logging to prevent stdout blocking with large tool schemas\nlitellm.set_verbose = False\nlitellm.suppress_debug_info = True\n\nlogger = Logger.get_logger(__name__)\n\n\ndef _sanitize_schema(params: Dict) -> Dict:\n \"\"\"Sanitize tool parameter schema to comply with Claude API requirements.\n \n Fixes common issues:\n - Empty object schemas (no properties, no required)\n - Missing required fields for Claude compatibility\n \"\"\"\n if not params:\n return {\"type\": \"object\", \"properties\": {}, \"required\": []}\n \n # Deep copy to avoid modifying the original\n import copy\n sanitized = copy.deepcopy(params)\n \n # Anthropic API requires top-level type to be 'object'\n # If it's not an object, wrap the schema as a property of an object\n top_level_type = sanitized.get(\"type\")\n if top_level_type and top_level_type != \"object\":\n # Wrap non-object schema as a single property called \"value\"\n logger.debug(f\"[SCHEMA_SANITIZE] Wrapping non-object schema (type={top_level_type}) into object\")\n wrapped = {\n \"type\": \"object\",\n \"properties\": {\n \"value\": sanitized # The original schema becomes a property\n },\n \"required\": [\"value\"] # Make it required\n }\n sanitized = wrapped\n \n # If type is object but missing properties/required, add them\n if sanitized.get(\"type\") == \"object\":\n if \"properties\" not in sanitized:\n sanitized[\"properties\"] = {}\n if \"required\" not in sanitized:\n sanitized[\"required\"] = []\n \n # Remove non-standard fields that may cause issues (like 'title')\n sanitized.pop(\"title\", None)\n \n # Recursively sanitize nested properties\n if \"properties\" in sanitized and isinstance(sanitized[\"properties\"], dict):\n for prop_name, prop_schema in list(sanitized[\"properties\"].items()):\n if isinstance(prop_schema, dict):\n # Remove title from nested properties\n prop_schema.pop(\"title\", None)\n \n return sanitized\n\n\ndef _schema_to_openai(schema: ToolSchema) -> ChatCompletionToolParam:\n \"\"\"Convert ToolSchema to OpenAI ChatCompletion tool format\"\"\"\n function_def = {\n \"name\": schema.name,\n \"description\": schema.description or \"\",\n }\n \n # Sanitize and add parameters\n if schema.parameters:\n sanitized = _sanitize_schema(schema.parameters)\n function_def[\"parameters\"] = sanitized\n # Debug: verify sanitization worked\n if \"title\" in schema.parameters and \"title\" not in sanitized:\n logger.debug(f\"Sanitized tool '{schema.name}': removed title\")\n else:\n # Claude requires parameters field even if empty\n function_def[\"parameters\"] = {\"type\": \"object\", \"properties\": {}, \"required\": []}\n \n return { \n \"type\": \"function\",\n \"function\": function_def\n }\n \ndef _prepare_tools_for_llmclient(\n tools: List[BaseTool] | None,\n fmt: str = \"openai\",\n) -> tuple[Sequence[Union[ToolSchema, ChatCompletionToolParam]], Dict[str, BaseTool]]:\n \"\"\"Convert BaseTool list to LLMClient usable format, with deduplication.\n \n Args:\n tools: BaseTool instance list (should be obtained from GroundingClient and bound to runtime_info)\n if None or empty list, return empty list\n fmt: output format, \"openai\" for OpenAI format\n \"\"\"\n if not tools:\n return [], {}\n \n if fmt == \"openai\":\n result = []\n tool_map = {} # llm_name -> BaseTool\n name_count = {}\n \n for tool in tools:\n name = tool.schema.name\n name_count[name] = name_count.get(name, 0) + 1\n \n\n seen_names = set()\n for tool in tools:\n original_name = tool.schema.name\n \n if name_count[original_name] > 1:\n server_name = \"unknown\"\n if tool.is_bound and tool.runtime_info and tool.runtime_info.server_name:\n server_name = tool.runtime_info.server_name\n llm_name = f\"{server_name}__{original_name}\"\n else:\n llm_name = original_name\n \n if llm_name in seen_names:\n logger.warning(f\"[TOOL_DEDUP] Skipping duplicate tool: {llm_name}\")\n continue\n seen_names.add(llm_name)\n \n tool_param = _schema_to_openai(tool.schema)\n tool_param[\"function\"][\"name\"] = llm_name \n result.append(tool_param)\n \n tool_map[llm_name] = tool\n \n if llm_name != original_name:\n logger.info(f\"[TOOL_RENAME] {original_name} -> {llm_name}\")\n \n logger.info(f\"[SCHEMA_SANITIZE] Prepared {len(result)} tools for LLM (from {len(tools)} total)\")\n return result, tool_map\n \n tool_map = {tool.schema.name: tool for tool in tools}\n return [tool.schema for tool in tools], tool_map\n\nDEFAULT_SUMMARIZE_THRESHOLD_CHARS = 200000 # ~50K tokens, lowered from 400K to prevent context overflow\nMAX_TOOL_RESULT_CHARS = 200000 # Fallback truncation limit when summarization fails (~50K tokens)\n\nasync def _summarize_tool_result(\n content: str,\n tool_name: str,\n task: str = \"\",\n model: str = \"openrouter/anthropic/claude-sonnet-4.5\",\n timeout: float = 60.0\n) -> str:\n \"\"\"Use LLM to summarize large tool results.\"\"\"\n try:\n logger.info(f\"Summarizing tool result from '{tool_name}': {len(content):,} chars\")\n \n # Pre-truncate if content is too large for the model (leave room for prompt + output)\n # Assuming ~4 chars per token, 200K tokens limit, 8K output, ~500 tokens for prompt\n # Safe input limit: (200K - 8K - 0.5K) * 4 = ~766K chars, but be conservative at 400K\n max_input_chars = 400000\n if len(content) > max_input_chars:\n logger.warning(f\"Pre-truncating content for summarization: {len(content):,} -> {max_input_chars:,} chars\")\n content = content[:max_input_chars] + f\"\\n\\n[TRUNCATED for summarization: original was {len(content):,} chars]\"\n \n task_hint = f\"\\n\\nUser's task: {task}\\nSummarize with focus on information relevant to this task.\" if task else \"\"\n \n prompt = f\"\"\"Tool '{tool_name}' returned a large result ({len(content):,} chars). Summarize it concisely.{task_hint}\n\n**Guidelines:**\n- Structured data (coordinates, steps, etc.): Keep key summary (totals, start/end), omit repetitive details.\n- Markup content (HTML, XML): Extract text and key data only, ignore tags/scripts.\n- Long documents: Keep structure outline and essential sections.\n- Lists/arrays: Summarize count and most relevant items.\n- Always preserve: numbers, URLs, file paths, IDs, key identifiers.\n\nContent:\n{content}\n\nConcise summary:\"\"\"\n \n response = await asyncio.wait_for(\n litellm.acompletion(\n model=model,\n messages=[{\"role\": \"user\", \"content\": prompt}],\n timeout=timeout\n ),\n timeout=timeout + 5\n )\n \n summary = response.choices[0].message.content.strip()\n result = f\"[SUMMARY of {len(content):,} chars]\\n{summary}\"\n \n logger.info(f\"Tool result summarized: {len(content):,} -> {len(result):,} chars\")\n return result\n \n except Exception as e:\n logger.warning(f\"Summarization failed for '{tool_name}': {e}\")\n return None\n\n\nasync def _tool_result_to_message_async(\n result: ToolResult, \n *, \n tool_call_id: str, \n tool_name: str,\n task: str = \"\",\n summarize_threshold: int = DEFAULT_SUMMARIZE_THRESHOLD_CHARS,\n summarize_model: str = \"openrouter/anthropic/claude-sonnet-4.5\",\n enable_summarization: bool = True\n) -> Dict:\n \"\"\"Convert ToolResult to LLMClient usable message format with LLM summarization for large results.\n\n Args:\n result: Tool execution result\n tool_call_id: OpenAI tool_call ID\n tool_name: Tool name\n task: User's original task for context-aware summarization\n summarize_threshold: If content exceeds this, use LLM summarization\n summarize_model: Model to use for summarization\n enable_summarization: Whether to enable LLM summarization\n \n Returns:\n OpenAI ChatCompletion tool message (text only)\n \"\"\"\n if result.is_error:\n text_content = f\"[ERROR] {result.error or 'unknown error'}\"\n else:\n text_content = (\n result.content\n if isinstance(result.content, str)\n else json.dumps(result.content, ensure_ascii=False, default=str)\n )\n \n original_len = len(text_content)\n \n # Use LLM summarization if content exceeds threshold\n if original_len > summarize_threshold and enable_summarization:\n summary = await _summarize_tool_result(text_content, tool_name, task, summarize_model)\n if summary:\n text_content = summary\n elif original_len > MAX_TOOL_RESULT_CHARS:\n # Fallback: truncate if summarization failed and content is too large\n truncate_msg = f\"\\n\\n[TRUNCATED: Original content was {original_len:,} chars, showing first {MAX_TOOL_RESULT_CHARS:,}]\"\n text_content = text_content[:MAX_TOOL_RESULT_CHARS - len(truncate_msg)] + truncate_msg\n logger.warning(f\"Tool result truncated for '{tool_name}': {original_len:,} -> {len(text_content):,} chars (summarization failed)\")\n \n return {\n \"role\": \"tool\",\n \"name\": tool_name,\n \"content\": text_content,\n \"tool_call_id\": tool_call_id,\n }\n\nasync def _execute_tool_call(\n tool: BaseTool,\n openai_tool_call: Dict,\n) -> ToolResult:\n \"\"\"Execute LLMClient returned tool_call\n\n Args:\n tool: BaseTool instance (must be obtained from GroundingClient and bound to runtime_info)\n openai_tool_call: LLMClient usable tool_call object, contains id, type, function etc. fields\n \"\"\"\n if not tool.is_bound:\n raise ValueError(\n f\"Tool '{tool.schema.name}' is not bound to runtime_info. \"\n f\"Please ensure tools are obtained from GroundingClient.list_tools() \"\n f\"with bind_runtime_info=True\"\n )\n \n func = openai_tool_call[\"function\"]\n arguments = func.get(\"arguments\", \"{}\")\n if isinstance(arguments, str):\n arguments = json.loads(arguments or \"{}\")\n \n # Filter out parameters that are not in the tool's schema\n if isinstance(arguments, dict) and tool.schema.parameters:\n # Get valid parameter names from tool schema (JSON Schema format)\n schema_params = tool.schema.parameters\n valid_params = set()\n \n if isinstance(schema_params, dict) and \"properties\" in schema_params:\n valid_params = set(schema_params[\"properties\"].keys())\n \n # Check for invalid parameters\n invalid_params = []\n for param_name in list(arguments.keys()):\n if param_name == \"skip_visual_analysis\":\n invalid_params.append(param_name)\n continue\n \n # Check if parameter is in the tool's schema\n if valid_params and param_name not in valid_params:\n invalid_params.append(param_name)\n \n # Remove invalid parameters\n for param in invalid_params:\n arguments.pop(param)\n logger.debug(\n f\"Removed parameter '{param}' from {tool.schema.name} \"\n f\"(not in tool schema)\"\n )\n\n return await tool.invoke(\n parameters=arguments,\n keep_session=True\n )\n\n\nclass LLMClient:\n \"\"\"LLMClient class for single round call\"\"\"\n def __init__(\n self, \n model: str = \"openrouter/anthropic/claude-sonnet-4.5\", \n enable_thinking: bool = False,\n rate_limit_delay: float = 0.0,\n max_retries: int = 3,\n retry_delay: float = 1.0,\n timeout: float = 120.0,\n summarize_threshold_chars: int = DEFAULT_SUMMARIZE_THRESHOLD_CHARS,\n enable_tool_result_summarization: bool = True,\n **litellm_kwargs\n ):\n \"\"\"\n Args:\n model: LLM model identifier\n enable_thinking: Whether to enable extended thinking mode\n rate_limit_delay: Minimum delay between API calls in seconds (0 = no delay)\n max_retries: Maximum number of retries on rate limit errors\n retry_delay: Initial delay between retries in seconds (exponential backoff)\n timeout: Request timeout in seconds (default: 120s)\n summarize_threshold_chars: If tool result exceeds this threshold, use LLM to \n summarize the result (default: 50000 chars โ 12.5K tokens)\n enable_tool_result_summarization: Whether to enable LLM-based summarization for \n large tool results (default: True)\n **litellm_kwargs: Additional litellm parameters\n \"\"\"\n self.model = model\n self.enable_thinking = enable_thinking\n self.rate_limit_delay = rate_limit_delay\n self.max_retries = max_retries\n self.retry_delay = retry_delay\n self.timeout = timeout\n self.summarize_threshold_chars = summarize_threshold_chars\n self.enable_tool_result_summarization = enable_tool_result_summarization\n self.litellm_kwargs = litellm_kwargs\n self._logger = Logger.get_logger(__name__)\n self._last_call_time = 0.0\n \n async def _rate_limit(self):\n \"\"\"Apply rate limiting by adding delay between API calls\"\"\"\n if self.rate_limit_delay > 0:\n current_time = time.time()\n time_since_last_call = current_time - self._last_call_time\n \n if time_since_last_call < self.rate_limit_delay:\n sleep_time = self.rate_limit_delay - time_since_last_call\n self._logger.debug(f\"Rate limiting: waiting {sleep_time:.2f}s before next API call\")\n await asyncio.sleep(sleep_time)\n \n self._last_call_time = time.time()\n \n async def _call_with_retry(self, **completion_kwargs):\n \"\"\"Call LLM with backoff retry on rate limit errors\n \n Timeout and retry strategy:\n - Single call timeout: self.timeout (default 120s)\n - Rate limit retry delays: 60s, 90s, 120s\n - Total max time: timeout * max_retries + sum(retry_delays)\n \"\"\"\n last_exception = None\n \n for attempt in range(self.max_retries):\n try:\n # Add timeout to the completion call\n response = await asyncio.wait_for(\n litellm.acompletion(**completion_kwargs),\n timeout=self.timeout\n )\n return response\n except asyncio.TimeoutError:\n self._logger.error(\n f\"LLM call timed out after {self.timeout}s (attempt {attempt + 1}/{self.max_retries})\"\n )\n last_exception = TimeoutError(f\"LLM call timed out after {self.timeout}s\")\n if attempt < self.max_retries - 1:\n # Retry on timeout with shorter delay\n self._logger.info(f\"Retrying after {self.retry_delay}s delay...\")\n await asyncio.sleep(self.retry_delay)\n continue\n else:\n raise last_exception\n except Exception as e:\n last_exception = e\n error_str = str(e).lower()\n \n # Check if it's a retryable error\n is_rate_limit = any(\n keyword in error_str \n for keyword in ['rate limit', 'rate_limit', 'too many requests', '429']\n )\n \n is_overloaded = any(\n keyword in error_str\n for keyword in ['overloaded', '500', '502', '503', '504', 'internal server error', 'service unavailable']\n )\n \n if attempt < self.max_retries - 1 and (is_rate_limit or is_overloaded):\n # Determine backoff delay based on error type\n if is_rate_limit:\n # Use longer backoff for rate limits to cross rate limit windows\n backoff_delay = 60 + (attempt * 30) # 60s, 90s, 120s\n error_type = \"Rate limit\"\n else: # is_overloaded\n # Use exponential backoff for server errors\n backoff_delay = min(5 * (2 ** attempt), 60) # 5s, 10s, 20s, max 60s\n error_type = \"Server overload\"\n \n self._logger.warning(\n f\"{error_type} error (attempt {attempt + 1}/{self.max_retries}), \"\n f\"waiting {backoff_delay}s before retry...\"\n )\n await asyncio.sleep(backoff_delay)\n continue\n else:\n # Not a retryable error, or max retries reached\n if attempt >= self.max_retries - 1:\n self._logger.error(f\"Max retries ({self.max_retries}) reached, giving up\")\n raise\n \n raise last_exception\n \n async def complete(\n self,\n messages: List[Dict] | str, \n tools: List[BaseTool] | None = None,\n execute_tools: bool = True,\n summary_prompt: Optional[str] = None,\n tool_result_callback: Optional[callable] = None,\n **kwargs\n ) -> Dict:\n \"\"\"\n Single-round LLM call with optional tool execution.\n \n Args:\n messages: conversation history (List[Dict] for standard OpenAI format, or str for text format)\n tools: BaseTool instance list (must be obtained from GroundingClient and bound to runtime_info)\n if None or empty list, only perform conversation, no tools\n execute_tools: if LLM returns tool_calls, whether to automatically execute tools\n summary_prompt: Optional custom prompt for requesting iteration summary. \n If provided, will request summary after tool execution.\n If None, no summary will be requested.\n tool_result_callback: Optional async callback to process tool results after execution.\n Signature: async def callback(result: ToolResult, tool_name: str, tool_call: Dict, backend: str) -> ToolResult\n **kwargs: additional parameters for litellm completion\n \"\"\"\n # 1. Process messages\n if isinstance(messages, str):\n current_messages = [{\"role\": \"user\", \"content\": messages}]\n user_task = messages\n elif isinstance(messages, list):\n current_messages = messages.copy()\n # Extract first user message as task for context-aware summarization\n user_task = next(\n (m.get(\"content\", \"\") for m in messages if m.get(\"role\") == \"user\"),\n \"\"\n )\n else:\n raise ValueError(\"messages must be List[Dict] or str\")\n \n # 2. prepare base litellm completion kwargs\n completion_kwargs = {\n \"model\": kwargs.get(\"model\", self.model),\n **self.litellm_kwargs,\n }\n \n # Add thinking/reasoning_effort only if explicitly enabled and not using tools\n enable_thinking = kwargs.get(\"enable_thinking\", self.enable_thinking)\n \n # 3. if tools are provided, add them to the request\n llm_tools = None\n tool_map = {} # llm_name -> BaseTool\n if tools:\n llm_tools, tool_map = _prepare_tools_for_llmclient(tools, fmt=\"openai\")\n if llm_tools:\n completion_kwargs[\"tools\"] = llm_tools\n completion_kwargs[\"tool_choice\"] = kwargs.get(\"tool_choice\", \"auto\")\n # Disable thinking when using tools to avoid format conflicts\n enable_thinking = False\n self._logger.debug(f\"Prepared {len(llm_tools)} tools for LLM\")\n else:\n self._logger.warning(\"Tools provided but none could be prepared for LLM\")\n \n # Add thinking parameters if enabled\n if enable_thinking:\n completion_kwargs[\"reasoning_effort\"] = kwargs.get(\"reasoning_effort\", \"medium\")\n \n # 4. Apply rate limiting\n await self._rate_limit()\n \n # 5. Call LLM with retry (single round)\n completion_kwargs[\"messages\"] = current_messages\n response = await self._call_with_retry(**completion_kwargs)\n \n if not response.choices:\n raise ValueError(\"LLM response has no choices\")\n \n response_message = response.choices[0].message\n \n # 6. Build assistant message\n assistant_message = {\n \"role\": \"assistant\",\n \"content\": response_message.content or \"\",\n }\n \n tool_calls = getattr(response_message, 'tool_calls', None)\n if tool_calls:\n assistant_message[\"tool_calls\"] = [\n {\n \"id\": tc.id,\n \"type\": \"function\",\n \"function\": {\n \"name\": tc.function.name,\n \"arguments\": tc.function.arguments\n }\n }\n for tc in tool_calls\n ]\n \n # Add assistant message to conversation\n current_messages.append(assistant_message)\n \n # 7. Execute tools if requested\n tool_results = []\n if execute_tools and tool_calls and tools:\n self._logger.info(f\"Executing {len(tool_calls)} tool calls...\")\n \n for tool_call in tool_calls:\n tool_name = tool_call.function.name\n \n # Extract tool metadata and check visual analysis request\n tool_obj = tool_map.get(tool_name)\n backend = None\n server_name = None\n \n if tool_obj:\n try:\n # Prefer runtime_info if bound\n if getattr(tool_obj, 'is_bound', False) and getattr(tool_obj, 'runtime_info', None):\n backend = tool_obj.runtime_info.backend.value\n server_name = tool_obj.runtime_info.server_name\n else:\n backend = tool_obj.backend_type.value if hasattr(tool_obj, 'backend_type') else None\n except Exception:\n pass\n \n # Log tool execution\n try:\n if isinstance(tool_call.function.arguments, str):\n safe_args_str = tool_call.function.arguments.strip() or \"{}\"\n args = json.loads(safe_args_str)\n else:\n args = tool_call.function.arguments\n \n args_str = json.dumps(args, ensure_ascii=False)[:200]\n self._logger.info(f\"Calling {tool_name} with args: {args_str}\")\n except:\n pass\n \n if tool_name not in tool_map:\n result = ToolResult(\n status=ToolStatus.ERROR,\n error=f\"Tool '{tool_name}' not found\"\n )\n else:\n try:\n result = await _execute_tool_call(\n tool=tool_map[tool_name],\n openai_tool_call={\n \"id\": tool_call.id,\n \"type\": \"function\",\n \"function\": {\n \"name\": tool_call.function.name,\n \"arguments\": tool_call.function.arguments\n }\n }\n )\n\n # Apply tool result callback if provided\n if tool_result_callback and not result.is_error:\n try:\n result = await tool_result_callback(\n result=result,\n tool_name=tool_name,\n tool_call=tool_call,\n backend=backend\n )\n except Exception as e:\n self._logger.warning(f\"Tool result callback failed for {tool_name}: {e}\")\n except Exception as e:\n result = ToolResult(\n status=ToolStatus.ERROR,\n error=str(e)\n )\n \n # Use async version with LLM summarization for large results\n tool_message = await _tool_result_to_message_async(\n result, \n tool_call_id=tool_call.id, \n tool_name=tool_name,\n task=user_task,\n summarize_threshold=self.summarize_threshold_chars,\n summarize_model=self.model,\n enable_summarization=self.enable_tool_result_summarization\n )\n current_messages.append(tool_message)\n \n # Store result\n tool_results.append({\n \"tool_call\": tool_call,\n \"result\": result,\n \"message\": tool_message,\n \"backend\": backend,\n \"server_name\": server_name,\n })\n \n self._logger.info(f\"Tool execution completed, {len(tool_results)} tools executed\")\n \n # 8. Request summary if provided and tools were executed\n iteration_summary = None\n \n if summary_prompt and tool_results:\n self._logger.debug(\"Requesting iteration summary from LLM\")\n summary_message = {\n \"role\": \"system\",\n \"content\": summary_prompt\n }\n current_messages.append(summary_message)\n \n # Apply rate limiting before summary call\n await self._rate_limit()\n \n # Call LLM to generate summary (without tools)\n summary_kwargs = {\n **self.litellm_kwargs,\n \"model\": self.model,\n \"messages\": current_messages,\n \"tools\": [], \n \"tool_choice\": \"none\",\n }\n \n summary_response = await self._call_with_retry(**summary_kwargs)\n \n if summary_response.choices:\n summary_message = summary_response.choices[0].message\n iteration_summary = summary_message.content or \"\"\n \n # Add summary response to messages\n current_messages.append({\n \"role\": \"assistant\",\n \"content\": iteration_summary\n })\n \n self._logger.debug(f\"Generated iteration summary: {iteration_summary[:100]}...\")\n \n # 9. Return single-round result \n return {\n \"message\": assistant_message,\n \"tool_results\": tool_results,\n \"messages\": current_messages,\n \"has_tool_calls\": bool(tool_calls),\n \"iteration_summary\": iteration_summary\n }\n \n @staticmethod\n def format_messages_to_text(messages: List[Dict]) -> str:\n \"\"\"Format conversation history to readable text (for logging/debugging)\"\"\"\n formatted = \"\"\n for msg in messages:\n role = msg.get(\"role\", \"unknown\").upper()\n content = msg.get(\"content\", \"\")\n formatted += f\"[{role}]\\n{content}\\n\\n\"\n return formatted"
},
{
"path": "anytool/local_server/README.md",
"content": "# AnyTool Local Server (Desktop Version)\n\n## 1. Introduction\n\nThe AnyTool Local Server is a **lightweight, cross-platform** Flask service that launches on the host workstation and exposes a uniform HTTP interface for controlling the native desktop environment. By translating REST calls into deterministic GUI actionsโmouse and keyboard synthesis, window management, screenshot capture, file I/Oโit enables higher-level AnyTool agents to interact with real software instead of simulated environments.\n\n**Supported platforms:** Windows 10/11, macOS 11+ (Intel & Apple Silicon) and mainstream Linux distributions (X11/Wayland).\n\n## 2. System Architecture\n\n* **PlatformAdapter** abstracts OS-specific primitives (Windows, macOS, Linux).\n* **Accessibility Helper** queries the UI accessibility tree for semantic information.\n* **Screenshot Helper** captures full or partial screenshots (PNG).\n* **Recorder** streams screen recordings for offline analysis.\n* **Health / Feature Checker** validates runtime capabilities and permissions.\n\n## 3. REST Endpoints\n\n| Path | Method | Semantics |\n|------|--------|-----------|\n| `/` | GET | Liveness probe |\n| `/platform` | GET | Return host OS metadata |\n| `/execute` | POST | Execute a PyAutoGUI script fragment |\n| `/execute_with_verification` | POST | Execute fragment and verify via template matching |\n| `/run_python` | POST | Run arbitrary Python within a sandbox |\n| `/run_bash_script` | POST | Run shell script (optional conda activation) |\n| `/screenshot` | GET | Return PNG screenshot (full or ROI) |\n| `/cursor_position` | GET | Current mouse coordinates |\n| `/screen_size` | GET/POST | Query or set virtual screen resolution |\n| `/list_directory` | POST | List directory contents |\n\n*see* `main.py` *for ~20 additional endpoints.*\n\n## 4. Setup & Launch\n\n> [!NOTE] \n> python=3.12 \n> Accessibility / screen-record permissions (macOS: *System Settings โธ Privacy & Security*).\n\n### Dependency Installation\n```bash\ncd anytool/local_server\npip install -r requirements.txt\n```\n\n### Launching the Server\n*Python entry point*\n```bash\npython -m anytool.local_server.main \\\n --host 127.0.0.1 --port 5000 # flags optional; defaults read from config.json\n```\n\n*Bash helper script*\n```bash\n./run.sh # reads config.json then starts the service\n```\n\nPress `Ctrl+C` at any time to gracefully stop the server.\n\n---\n\n## 5. Configuration\nRuntime options live in `config.json`:\n```json\n{\n \"server\": {\n \"host\": \"127.0.0.1\", // listening address (0.0.0.0 for all interfaces)\n \"port\": 5000, // default port\n \"debug\": false // verbose Flask logs\n }\n}\n```"
},
{
"path": "anytool/local_server/__init__.py",
"content": "from .main import app, run_server\n\n__all__ = [\"app\", \"run_server\"]"
},
{
"path": "anytool/local_server/config.json",
"content": "{\n \"server\": {\n \"host\": \"127.0.0.1\",\n \"port\": 5000,\n \"debug\": false,\n \"threaded\": true\n }\n}\n"
},
{
"path": "anytool/local_server/feature_checker.py",
"content": "import platform\nimport subprocess\nimport tempfile\nfrom typing import Dict, Any\n\nfrom anytool.utils.logging import Logger\nlogger = Logger.get_logger(__name__)\n\nplatform_name = platform.system()\n\n\nclass FeatureChecker:\n def __init__(self, platform_adapter=None, accessibility_helper=None):\n self.platform_adapter = platform_adapter\n self.accessibility_helper = accessibility_helper\n self.platform = platform_name\n self._cache = {} \n \n def check_screenshot_available(self, use_cache: bool = True) -> bool:\n if use_cache and 'screenshot' in self._cache:\n return self._cache['screenshot']\n \n try:\n import pyautogui\n from PIL import Image\n \n size = pyautogui.size()\n result = size.width > 0 and size.height > 0\n \n self._cache['screenshot'] = result\n logger.info(f\"Screenshot check: {'available' if result else 'unavailable'}\")\n return result\n \n except ImportError as e:\n logger.warning(f\"Screenshot unavailable - missing dependency: {e}\")\n self._cache['screenshot'] = False\n return False\n except Exception as e:\n logger.error(f\"Screenshot check failed: {e}\")\n self._cache['screenshot'] = False\n return False\n \n def check_shell_available(self, use_cache: bool = True) -> bool:\n if use_cache and 'shell' in self._cache:\n return self._cache['shell']\n \n try:\n if self.platform == \"Windows\":\n cmd = ['cmd', '/c', 'echo', 'test']\n else:\n cmd = ['echo', 'test']\n \n result = subprocess.run(\n cmd,\n capture_output=True,\n timeout=2,\n text=True\n )\n \n available = result.returncode == 0\n self._cache['shell'] = available\n logger.info(f\"Shell check: {'available' if available else 'unavailable'}\")\n return available\n \n except FileNotFoundError as e:\n logger.warning(f\"Shell check failed - command not found: {e}\")\n self._cache['shell'] = False\n return False\n except Exception as e:\n logger.error(f\"Shell check failed: {e}\")\n self._cache['shell'] = False\n return False\n \n def check_python_available(self, use_cache: bool = True) -> bool:\n if use_cache and 'python' in self._cache:\n return self._cache['python']\n \n python_commands = []\n if self.platform == \"Windows\":\n python_commands = ['py', 'python', 'python3']\n else:\n python_commands = ['python3', 'python']\n \n for python_cmd in python_commands:\n try:\n result = subprocess.run(\n [python_cmd, '--version'],\n capture_output=True,\n timeout=2,\n text=True\n )\n \n if result.returncode == 0:\n version = result.stdout.strip() or result.stderr.strip()\n self._cache['python'] = True\n logger.info(f\"Python check: available ({python_cmd} - {version})\")\n return True\n \n except FileNotFoundError:\n continue\n except Exception as e:\n logger.debug(f\"Error testing {python_cmd}: {e}\")\n continue\n \n logger.warning(\"Python check failed - no valid Python interpreter found\")\n self._cache['python'] = False\n return False\n \n def check_file_ops_available(self, use_cache: bool = True) -> bool:\n if use_cache and 'file_ops' in self._cache:\n return self._cache['file_ops']\n \n try:\n with tempfile.NamedTemporaryFile(mode='w+b', delete=True) as tmp:\n test_data = b'test data'\n tmp.write(test_data)\n tmp.flush()\n \n tmp.seek(0)\n read_data = tmp.read()\n \n available = read_data == test_data\n self._cache['file_ops'] = available\n logger.info(f\"File operations check: {'available' if available else 'unavailable'}\")\n return available\n \n except PermissionError as e:\n logger.warning(f\"File operations check failed - permission denied: {e}\")\n self._cache['file_ops'] = False\n return False\n except Exception as e:\n logger.error(f\"File operations check failed: {e}\")\n self._cache['file_ops'] = False\n return False\n \n def check_window_mgmt_available(self, use_cache: bool = True) -> bool:\n if use_cache and 'window_mgmt' in self._cache:\n return self._cache['window_mgmt']\n \n try:\n if not self.platform_adapter:\n logger.warning(\"Window management check failed - no platform adapter loaded\")\n self._cache['window_mgmt'] = False\n return False\n \n required_methods = ['activate_window', 'close_window', 'list_windows']\n available_methods = [\n method for method in required_methods \n if hasattr(self.platform_adapter, method)\n ]\n \n available = len(available_methods) > 0\n self._cache['window_mgmt'] = available\n \n if available:\n logger.info(f\"Window management check: {'available' if available else 'unavailable'} - supported methods: {', '.join(available_methods)}\")\n else:\n logger.warning(f\"Window management check failed - platform adapter missing required methods\")\n \n return available\n \n except Exception as e:\n logger.error(f\"Window management check failed: {e}\")\n self._cache['window_mgmt'] = False\n return False\n \n def check_recording_available(self, use_cache: bool = True) -> bool:\n if use_cache and 'recording' in self._cache:\n return self._cache['recording']\n \n try:\n if not self.platform_adapter:\n logger.warning(\"Recording check failed - no platform adapter loaded\")\n self._cache['recording'] = False\n return False\n \n available = (\n hasattr(self.platform_adapter, 'start_recording') and \n hasattr(self.platform_adapter, 'stop_recording')\n )\n \n self._cache['recording'] = available\n logger.info(f\"Recording check: {'available' if available else 'unavailable'}\")\n return available\n \n except Exception as e:\n logger.error(f\"Recording check failed: {e}\")\n self._cache['recording'] = False\n return False\n \n def check_accessibility_available(self, use_cache: bool = True) -> bool:\n if use_cache and 'accessibility' in self._cache:\n return self._cache['accessibility']\n \n try:\n if not self.accessibility_helper:\n logger.warning(\"Accessibility check failed - no accessibility helper loaded\")\n self._cache['accessibility'] = False\n return False\n \n available = self.accessibility_helper.is_available()\n self._cache['accessibility'] = available\n logger.info(f\"Accessibility check: {'available' if available else 'unavailable'}\")\n return available\n \n except Exception as e:\n logger.error(f\"Accessibility check failed: {e}\")\n self._cache['accessibility'] = False\n return False\n \n def check_platform_adapter_available(self, use_cache: bool = True) -> bool:\n if use_cache and 'platform_adapter' in self._cache:\n return self._cache['platform_adapter']\n \n available = self.platform_adapter is not None\n self._cache['platform_adapter'] = available\n logger.info(f\"Platform adapter check: {'available' if available else 'unavailable'}\")\n return available\n \n def check_all_features(self, use_cache: bool = True) -> Dict[str, bool]:\n logger.info(f\"Checking all features (platform: {self.platform})\")\n \n results = {\n 'accessibility': self.check_accessibility_available(use_cache),\n 'screenshot': self.check_screenshot_available(use_cache),\n 'recording': self.check_recording_available(use_cache),\n 'shell': self.check_shell_available(use_cache),\n 'python': self.check_python_available(use_cache),\n 'file_ops': self.check_file_ops_available(use_cache),\n 'window_mgmt': self.check_window_mgmt_available(use_cache),\n 'platform_adapter': self.check_platform_adapter_available(use_cache),\n }\n \n available_count = sum(1 for v in results.values() if v)\n total_count = len(results)\n logger.info(f\"Feature check completed: {available_count}/{total_count} features available\")\n \n return results\n \n def clear_cache(self):\n self._cache.clear()\n logger.debug(\"Feature check cache cleared\")\n \n def get_feature_report(self) -> Dict[str, Any]:\n results = self.check_all_features()\n \n return {\n 'platform': {\n 'system': self.platform,\n 'release': platform.release(),\n 'version': platform.version(),\n 'machine': platform.machine(),\n 'processor': platform.processor(),\n },\n 'features': results,\n 'summary': {\n 'total': len(results),\n 'available': sum(1 for v in results.values() if v),\n 'unavailable': sum(1 for v in results.values() if not v),\n }\n }"
},
{
"path": "anytool/local_server/health_checker.py",
"content": "import requests\nimport os\nfrom pathlib import Path\nfrom typing import Dict, Tuple, Optional\nfrom anytool.utils.logging import Logger\nfrom anytool.local_server.feature_checker import FeatureChecker\n\nlogger = Logger.get_logger(__name__)\n\nfrom anytool.utils.display import colorize as _c\n\n\nclass HealthStatus:\n \"\"\"Health status\"\"\"\n def __init__(self, feature_available: bool, endpoint_available: Optional[bool], \n endpoint_detail: str = \"\"):\n self.feature_available = feature_available\n self.endpoint_available = endpoint_available\n self.endpoint_detail = endpoint_detail\n \n @property\n def fully_available(self) -> bool:\n \"\"\"Fully available: feature and endpoint are available\"\"\"\n return self.feature_available and (self.endpoint_available == True)\n \n def __str__(self):\n if not self.feature_available:\n return \"Feature N/A\"\n elif self.endpoint_available is None:\n return \"Feature OK (endpoint not tested)\"\n elif self.endpoint_available:\n return f\"OK ({self.endpoint_detail})\"\n else:\n return f\"Endpoint failed: {self.endpoint_detail}\"\n\n\nclass HealthChecker:\n \"\"\"Health checker with functional testing\"\"\"\n \n def __init__(self, feature_checker: FeatureChecker, \n base_url: str = \"http://127.0.0.1:5000\",\n auto_cleanup: bool = True,\n test_output_dir: str = None):\n self.feature_checker = feature_checker\n self.base_url = base_url\n self.results = {}\n self.auto_cleanup = auto_cleanup\n \n # set the test output directory\n if test_output_dir:\n self.test_output_dir = Path(test_output_dir)\n else:\n current_dir = Path(__file__).parent\n self.test_output_dir = current_dir / \"temp\"\n \n # create the directory\n self.test_output_dir.mkdir(exist_ok=True)\n \n self.temp_files = [] # Track temporary files for cleanup\n \n logger.info(f\"Health checker initialized. Test output: {self.test_output_dir}, Auto-cleanup: {auto_cleanup}\")\n \n def _get_test_file_path(self, filename: str) -> str:\n \"\"\"Get path for a test file\"\"\"\n filepath = str(self.test_output_dir / filename)\n self._register_temp_file(filepath)\n return filepath\n \n def _register_temp_file(self, filepath: str):\n \"\"\"Register a temporary file for later cleanup\"\"\"\n if filepath and filepath not in self.temp_files:\n self.temp_files.append(filepath)\n \n def cleanup_temp_files(self):\n \"\"\"Clean up all temporary test files\"\"\"\n if not self.auto_cleanup:\n logger.info(f\"Auto-cleanup disabled. Test files kept in: {self.test_output_dir}\")\n return\n \n cleaned = 0\n for filepath in self.temp_files:\n try:\n if os.path.exists(filepath):\n os.remove(filepath)\n cleaned += 1\n logger.debug(f\"Cleaned up: {filepath}\")\n except Exception as e:\n logger.warning(f\"Failed to clean up {filepath}: {e}\")\n \n self.temp_files.clear()\n \n # if the directory is empty, delete it\n try:\n if self.test_output_dir.exists() and not any(self.test_output_dir.iterdir()):\n self.test_output_dir.rmdir()\n logger.debug(f\"Removed empty directory: {self.test_output_dir}\")\n except:\n pass\n \n if cleaned > 0:\n logger.info(f\"Cleaned up {cleaned} test files\")\n \n def check_screenshot(self) -> Tuple[bool, str]:\n \"\"\"Functionally test screenshot - actually take a screenshot and verify\"\"\"\n # 1. Check feature first\n if not self.feature_checker.check_screenshot_available():\n return False, \"Feature N/A\"\n \n # 2. Save screenshot to test directory\n screenshot_path = self._get_test_file_path(\"test_screenshot.png\")\n \n try:\n response = requests.get(f\"{self.base_url}/screenshot\", timeout=10)\n \n if response.status_code != 200:\n return False, f\"HTTP {response.status_code}\"\n \n # 3. Save to file\n with open(screenshot_path, 'wb') as f:\n f.write(response.content)\n \n # 4. Verify it's actually an image\n content_type = response.headers.get('Content-Type', '')\n if 'image' not in content_type:\n return False, f\"Invalid content type: {content_type}\"\n \n # 5. Check file size (should be > 1KB)\n size_kb = len(response.content) / 1024\n if size_kb < 1:\n return False, \"Image too small\"\n \n logger.info(f\"Screenshot saved: {screenshot_path} ({size_kb:.1f}KB)\")\n return True, f\"OK ({size_kb:.1f}KB)\"\n \n except requests.exceptions.Timeout:\n return False, \"Timeout\"\n except Exception as e:\n return False, f\"Error: {str(e)[:30]}\"\n \n def check_cursor_position(self) -> Tuple[bool, str]:\n \"\"\"Test cursor position\"\"\"\n if not self.feature_checker.check_screenshot_available():\n return False, \"Feature N/A\"\n \n try:\n response = requests.get(f\"{self.base_url}/cursor_position\", timeout=5)\n if response.status_code != 200:\n return False, f\"HTTP {response.status_code}\"\n \n data = response.json()\n if 'x' in data and 'y' in data:\n return True, f\"({data['x']}, {data['y']})\"\n return False, \"Invalid response\"\n except Exception as e:\n return False, str(e)[:30]\n \n def check_screen_size(self) -> Tuple[bool, str]:\n \"\"\"Test screen size\"\"\"\n if not self.feature_checker.check_screenshot_available():\n return False, \"Feature N/A\"\n \n try:\n response = requests.get(f\"{self.base_url}/screen_size\", timeout=5)\n if response.status_code != 200:\n return False, f\"HTTP {response.status_code}\"\n \n data = response.json()\n if 'width' in data and 'height' in data:\n return True, f\"{data['width']}x{data['height']}\"\n return False, \"Invalid response\"\n except Exception as e:\n return False, str(e)[:30]\n \n def check_shell_command(self) -> Tuple[bool, str]:\n \"\"\"Functionally test shell command execution\"\"\"\n if not self.feature_checker.check_shell_available():\n return False, \"Feature N/A\"\n \n try:\n response = requests.post(\n f\"{self.base_url}/execute\",\n json={\"command\": \"echo hello_test\", \"shell\": True},\n timeout=5\n )\n \n if response.status_code != 200:\n return False, f\"HTTP {response.status_code}\"\n \n data = response.json()\n output = data.get('output', '').strip()\n \n # Verify the command actually executed\n if 'hello_test' in output:\n return True, \"Command executed\"\n return False, \"Command failed\"\n \n except Exception as e:\n return False, str(e)[:30]\n \n def check_python_execution(self) -> Tuple[bool, str]:\n \"\"\"Functionally test Python code execution\"\"\"\n if not self.feature_checker.check_python_available():\n return False, \"Feature N/A\"\n \n try:\n test_code = 'print(\"test_output_123\")'\n response = requests.post(\n f\"{self.base_url}/run_python\",\n json={\"code\": test_code},\n timeout=5\n )\n \n if response.status_code != 200:\n return False, f\"HTTP {response.status_code}\"\n \n data = response.json()\n content = data.get('content', '')\n \n # Verify Python executed correctly\n if 'test_output_123' in content:\n return True, \"Python executed\"\n return False, \"Execution failed\"\n \n except Exception as e:\n return False, str(e)[:30]\n \n def check_bash_script(self) -> Tuple[bool, str]:\n \"\"\"Functionally test Bash script execution\"\"\"\n if not self.feature_checker.check_shell_available():\n return False, \"Feature N/A\"\n \n try:\n response = requests.post(\n f\"{self.base_url}/run_bash_script\",\n json={\"script\": \"echo bash_test_456\"},\n timeout=5\n )\n \n if response.status_code != 200:\n return False, f\"HTTP {response.status_code}\"\n \n data = response.json()\n output = data.get('output', '')\n \n if 'bash_test_456' in output:\n return True, \"Bash executed\"\n return False, \"Execution failed\"\n \n except Exception as e:\n return False, str(e)[:30]\n \n def check_file_operations(self) -> Tuple[bool, str]:\n \"\"\"Test file operations\"\"\"\n if not self.feature_checker.check_file_ops_available():\n return False, \"Feature N/A\"\n \n try:\n # Test list directory\n response = requests.post(\n f\"{self.base_url}/list_directory\",\n json={\"path\": \".\"},\n timeout=5\n )\n \n if response.status_code != 200:\n return False, f\"HTTP {response.status_code}\"\n \n data = response.json()\n if 'items' in data and isinstance(data['items'], list):\n return True, f\"{len(data['items'])} items\"\n return False, \"Invalid response\"\n \n except Exception as e:\n return False, str(e)[:30]\n \n def check_desktop_path(self) -> Tuple[bool, str]:\n \"\"\"Test desktop path\"\"\"\n if not self.feature_checker.check_file_ops_available():\n return False, \"Feature N/A\"\n \n try:\n response = requests.get(f\"{self.base_url}/desktop_path\", timeout=5)\n if response.status_code != 200:\n return False, f\"HTTP {response.status_code}\"\n \n data = response.json()\n path = data.get('path', '')\n if path and os.path.exists(path):\n return True, \"Path valid\"\n return False, \"Path not found\"\n except Exception as e:\n return False, str(e)[:30]\n \n def check_window_management(self) -> Tuple[bool, str]:\n \"\"\"Test window management\"\"\"\n if not self.feature_checker.check_window_mgmt_available():\n return False, \"Feature N/A\"\n \n try:\n # Just test if endpoint responds (window may not exist)\n response = requests.post(\n f\"{self.base_url}/setup/activate_window\",\n json={\"window_name\": \"NonExistentWindow\"},\n timeout=5\n )\n \n # 200 (success), 404 (not found), 501 (not supported) are all acceptable\n if response.status_code in [200, 404, 501]:\n return True, f\"API available\"\n return False, f\"HTTP {response.status_code}\"\n except Exception as e:\n return False, str(e)[:30]\n \n def check_recording(self) -> Tuple[bool, str]:\n \"\"\"Functionally test recording - actually start and stop recording\"\"\"\n if not self.feature_checker.check_recording_available():\n return False, \"Feature N/A\"\n \n recording_path = self._get_test_file_path(\"test_recording.mp4\")\n \n try:\n # 1. Start recording\n response = requests.post(f\"{self.base_url}/start_recording\", json={}, timeout=10)\n \n if response.status_code == 501:\n return False, \"Not supported\"\n \n if response.status_code != 200:\n return False, f\"Start failed: {response.status_code}\"\n \n # 2. Wait a bit\n import time\n time.sleep(3.0) # Record for 3 seconds\n \n # 3. Stop recording\n response = requests.post(f\"{self.base_url}/end_recording\", json={}, timeout=15)\n \n if response.status_code == 200:\n # Save the recording file\n with open(recording_path, 'wb') as f:\n f.write(response.content)\n \n size_kb = len(response.content) / 1024\n logger.info(f\"Recording saved: {recording_path} ({size_kb:.1f}KB)\")\n return True, f\"OK ({size_kb:.1f}KB)\"\n else:\n return False, f\"Stop failed: {response.status_code}\"\n \n except Exception as e:\n # Try to stop recording in case of error\n try:\n requests.post(f\"{self.base_url}/end_recording\", json={}, timeout=5)\n except:\n pass\n return False, str(e)[:30]\n \n def check_accessibility(self) -> Tuple[bool, str]:\n \"\"\"Test accessibility tree\"\"\"\n if not self.feature_checker.check_accessibility_available():\n return False, \"Feature N/A\"\n \n try:\n response = requests.get(f\"{self.base_url}/accessibility?max_depth=1\", timeout=10)\n \n if response.status_code != 200:\n return False, f\"HTTP {response.status_code}\"\n \n data = response.json()\n if 'error' in data:\n return False, \"Permission denied\"\n \n # Should have some tree structure\n if 'platform' in data or 'children' in data:\n return True, \"Tree available\"\n return False, \"Invalid response\"\n \n except Exception as e:\n return False, str(e)[:30]\n \n def check_health_endpoint(self) -> Tuple[bool, str]:\n \"\"\"Test health check endpoint\"\"\"\n try:\n response = requests.get(f\"{self.base_url}/\", timeout=5)\n if response.status_code == 200:\n data = response.json()\n if data.get('status') == 'ok':\n return True, \"OK\"\n return False, f\"HTTP {response.status_code}\"\n except Exception as e:\n return False, str(e)[:30]\n \n def check_platform_info(self) -> Tuple[bool, str]:\n \"\"\"Test platform info endpoint\"\"\"\n try:\n response = requests.get(f\"{self.base_url}/platform\", timeout=5)\n if response.status_code == 200:\n data = response.json()\n if 'system' in data:\n return True, data['system']\n return False, f\"HTTP {response.status_code}\"\n except Exception as e:\n return False, str(e)[:30]\n \n def check_all(self, test_endpoints: bool = True) -> Dict[str, HealthStatus]:\n \"\"\"\n Check all features with functional testing\n \n Args:\n test_endpoints: Whether to test endpoints (False only checks features)\n \n Returns:\n {Feature name: HealthStatus}\n \"\"\"\n results = {}\n \n if not test_endpoints:\n # Only check features, not endpoints\n feature_results = self.feature_checker.check_all_features()\n for name, available in feature_results.items():\n results[name] = HealthStatus(available, None, \"\")\n self.results = results\n return results\n \n # Functional tests\n test_functions = {\n 'Health Check': self.check_health_endpoint,\n 'Platform Info': self.check_platform_info,\n 'Screenshot': self.check_screenshot,\n 'Cursor Position': self.check_cursor_position,\n 'Screen Size': self.check_screen_size,\n 'Shell Command': self.check_shell_command,\n 'Python Execution': self.check_python_execution,\n 'Bash Script': self.check_bash_script,\n 'File Operations': self.check_file_operations,\n 'Desktop Path': self.check_desktop_path,\n 'Window Management': self.check_window_management,\n 'Recording': self.check_recording,\n 'Accessibility': self.check_accessibility,\n }\n \n for name, test_func in test_functions.items():\n success, detail = test_func()\n \n # Determine feature availability\n if detail == \"Feature N/A\":\n feature_available = False\n endpoint_available = None\n else:\n feature_available = True\n endpoint_available = success\n \n results[name] = HealthStatus(feature_available, endpoint_available, detail)\n \n # Clean up temporary files\n self.cleanup_temp_files()\n \n self.results = results\n return results\n \n def print_results(self, results: Dict[str, HealthStatus] = None, \n show_endpoint_details: bool = False):\n \"\"\"Print check results\"\"\"\n if results is None:\n results = self.results\n \n if not results:\n return\n \n total = len(results)\n feature_available = sum(1 for s in results.values() if s.feature_available)\n fully_available = sum(1 for s in results.values() if s.fully_available)\n \n # Categorize\n basic = ['Health Check', 'Platform Info']\n \n # Basic Features\n print()\n print(_c(\" - Basic\", 'c', bold=True))\n basic_items = []\n for name in basic:\n if name in results:\n status = results[name]\n # Use colored dot instead of emoji\n if status.fully_available:\n icon = _c(\"โ\", 'g')\n elif not status.feature_available:\n icon = _c(\"โ\", 'rd')\n elif status.endpoint_available is None:\n icon = _c(\"โ\", 'y')\n else:\n icon = _c(\"โ\", 'y')\n \n text = _c(name, 'gr' if not status.feature_available else '')\n basic_items.append((icon, text, status))\n \n # Display in rows of 4\n for i in range(0, len(basic_items), 4):\n line_items = []\n for j in range(4):\n if i + j < len(basic_items):\n icon, text, status = basic_items[i + j]\n line_items.append(f\"{icon} {text:<15}\")\n print(\" \" + \" \".join(line_items))\n \n # Show details if requested\n if show_endpoint_details:\n for name in basic:\n if name in results:\n status = results[name]\n print(f\" {_c('ยท', 'gr')} {name}: {_c(str(status), 'gr')}\")\n \n # Advanced Features\n print()\n print(_c(\" - Advanced\", 'c', bold=True))\n advanced_items = []\n for name, status in results.items():\n if name not in basic:\n # Use colored dot instead of emoji\n if status.fully_available:\n icon = _c(\"โ\", 'g')\n elif not status.feature_available:\n icon = _c(\"โ\", 'rd')\n elif status.endpoint_available is None:\n icon = _c(\"โ\", 'y')\n else:\n icon = _c(\"โ\", 'y')\n \n text = _c(name, 'gr' if not status.feature_available else '')\n advanced_items.append((icon, text, status))\n \n # Display in rows of 4\n for i in range(0, len(advanced_items), 4):\n line_items = []\n for j in range(4):\n if i + j < len(advanced_items):\n icon, text, _ = advanced_items[i + j]\n line_items.append(f\"{icon} {text:<15}\")\n print(\" \" + \" \".join(line_items))\n \n # Show details if requested\n if show_endpoint_details:\n for name, status in results.items():\n if name not in basic:\n print(f\" {_c('ยท', 'gr')} {name}: {_c(str(status), 'gr')}\")\n \n # Summary\n from anytool.utils.display import print_separator\n print()\n print_separator()\n print(f\" {_c('Summary:', 'c', bold=True)} {_c(str(feature_available) + '/' + str(total), 'g' if feature_available == total else 'y')} features available\", end='')\n if any(s.endpoint_available is not None for s in results.values()):\n print(f\", {_c(str(fully_available) + '/' + str(total), 'g' if fully_available == total else 'y')} fully functional\")\n else:\n print()\n print_separator()\n \n # Legend\n print(f\" {_c('Legend:', 'gr')} {_c('โ', 'g')} Available {_c('โ', 'y')} Partial/Untested {_c('โ', 'rd')} Unavailable\")\n \n # Test files info\n if self.temp_files and not self.auto_cleanup:\n print()\n print(f\" {_c('Test files saved:', 'y')} {self.test_output_dir}\")\n print(f\" {_c(str(len(self.temp_files)) + ' file(s) available for inspection', 'gr')}\")\n \n print()\n \n def get_summary(self) -> dict:\n \"\"\"Get summary\"\"\"\n if not self.results:\n return {}\n \n total = len(self.results)\n feature_available = sum(1 for s in self.results.values() if s.feature_available)\n fully_available = sum(1 for s in self.results.values() if s.fully_available)\n \n return {\n 'total': total,\n 'feature_available': feature_available,\n 'fully_available': fully_available,\n 'details': {k: str(v) for k, v in self.results.items()}\n }\n \n def get_simple_features_dict(self) -> Dict[str, bool]:\n \"\"\"Get simple feature dict (for banner display)\"\"\"\n return self.feature_checker.check_all_features()"
},
{
"path": "anytool/local_server/main.py",
"content": "import os\nimport platform\nimport shlex\nimport subprocess\nimport signal\nimport time\nimport json\nimport uuid\nfrom datetime import datetime\nfrom flask import Flask, request, jsonify, send_file, abort\nimport pyautogui\nimport threading\nfrom io import BytesIO\nimport tempfile\n\nfrom anytool.utils.logging import Logger\nfrom anytool.local_server.utils import AccessibilityHelper, ScreenshotHelper\nfrom anytool.local_server.platform_adapters import get_platform_adapter\nfrom anytool.local_server.health_checker import HealthChecker\nfrom anytool.local_server.feature_checker import FeatureChecker\n\nplatform_name = platform.system()\n\napp = Flask(__name__)\napp.config['MAX_CONTENT_LENGTH'] = 500 * 1024 * 1024 # 500MB\n\npyautogui.PAUSE = 0\nif platform_name == \"Darwin\":\n pyautogui.DARWIN_CATCH_UP_TIME = 0\n\nlogger = Logger.get_logger(__name__)\n\nTIMEOUT = 1800\nrecording_process = None\n\nif platform_name == \"Windows\":\n recording_path = os.path.join(os.environ.get('TEMP', 'C:\\\\Temp'), 'recording.mp4')\nelse:\n recording_path = \"/tmp/recording.mp4\"\n\naccessibility_helper = AccessibilityHelper()\nscreenshot_helper = ScreenshotHelper()\nplatform_adapter = get_platform_adapter()\n\nfeature_checker = FeatureChecker(\n platform_adapter=platform_adapter,\n accessibility_helper=accessibility_helper\n)\n\n\ndef get_conda_activation_prefix(conda_env: str = None) -> str:\n \"\"\"\n Generate platform-specific conda activation command prefix\n \n Args:\n conda_env: Conda environment name (e.g., 'myenv')\n \n Returns:\n Activation command prefix string, empty if no conda_env\n \"\"\"\n if not conda_env:\n return \"\"\n \n if platform_name == \"Windows\":\n # Windows: use conda.bat or conda.exe\n # Try common conda installation paths\n conda_paths = [\n os.path.expandvars(\"%USERPROFILE%\\\\miniconda3\\\\Scripts\\\\activate.bat\"),\n os.path.expandvars(\"%USERPROFILE%\\\\anaconda3\\\\Scripts\\\\activate.bat\"),\n \"C:\\\\ProgramData\\\\Miniconda3\\\\Scripts\\\\activate.bat\",\n \"C:\\\\ProgramData\\\\Anaconda3\\\\Scripts\\\\activate.bat\",\n ]\n \n # Find first existing conda activate script\n activate_script = None\n for path in conda_paths:\n if os.path.exists(path):\n activate_script = path\n break\n \n if activate_script:\n return f'call \"{activate_script}\" {conda_env} && '\n else:\n # Fallback: assume conda is in PATH\n return f'conda activate {conda_env} && '\n \n else:\n # Linux/macOS: source conda.sh then activate\n conda_paths = [\n os.path.expanduser(\"~/miniconda3/etc/profile.d/conda.sh\"),\n os.path.expanduser(\"~/anaconda3/etc/profile.d/conda.sh\"),\n \"/opt/conda/etc/profile.d/conda.sh\",\n \"/usr/local/miniconda3/etc/profile.d/conda.sh\",\n \"/usr/local/anaconda3/etc/profile.d/conda.sh\",\n ]\n \n # Find first existing conda.sh\n conda_sh = None\n for path in conda_paths:\n if os.path.exists(path):\n conda_sh = path\n break\n \n if conda_sh:\n return f'source \"{conda_sh}\" && conda activate {conda_env} && '\n else:\n # Fallback: assume conda is already initialized in shell\n return f'conda activate {conda_env} && '\n\n\ndef wrap_script_with_conda(script: str, conda_env: str = None) -> str:\n \"\"\"\n Wrap script with conda activation command.\n If conda is not available, returns original script without conda activation.\n \"\"\"\n if not conda_env:\n return script\n \n if platform_name == \"Windows\":\n activation_prefix = get_conda_activation_prefix(conda_env)\n return f\"{activation_prefix}{script}\"\n else:\n conda_paths = [\n os.path.expanduser(\"~/miniconda3/etc/profile.d/conda.sh\"),\n os.path.expanduser(\"~/anaconda3/etc/profile.d/conda.sh\"),\n os.path.expanduser(\"~/opt/anaconda3/etc/profile.d/conda.sh\"),\n \"/opt/conda/etc/profile.d/conda.sh\",\n ]\n \n conda_sh = None\n for path in conda_paths:\n if os.path.exists(path):\n conda_sh = path\n break\n \n if conda_sh:\n # Use bash -i -c to run interactively, or directly source conda.sh\n wrapped_script = f\"\"\"#!/bin/bash\n# Initialize conda\nif [ -f \"{conda_sh}\" ]; then\n . \"{conda_sh}\"\n conda activate {conda_env} 2>/dev/null || true\nfi\n\n# Run user script\n{script}\n\"\"\"\n return wrapped_script\n else:\n # Conda not found - log warning and execute script directly without conda\n logger.warning(f\"Conda environment '{conda_env}' requested but conda not found. Executing with system Python.\")\n return script\n\n\nhealth_checker = None\n\n@app.route('/', methods=['GET'])\ndef health_check():\n \"\"\"Health check interface - return features information\"\"\"\n # Get features from health_checker\n if health_checker:\n features = health_checker.get_simple_features_dict()\n else:\n # Initial startup of health_checker may not have been initialized, fallback to feature_checker\n features = feature_checker.check_all_features(use_cache=True)\n \n return jsonify({\n 'status': 'ok',\n 'service': 'AnyTool Desktop Server',\n 'version': '1.0.0',\n 'platform': platform_name,\n 'features': features,\n 'timestamp': datetime.now().isoformat()\n })\n\n@app.route('/platform', methods=['GET'])\ndef get_platform():\n info = {\n 'system': platform_name,\n 'release': platform.release(),\n 'version': platform.version(),\n 'machine': platform.machine(),\n 'processor': platform.processor()\n }\n \n if platform_adapter and hasattr(platform_adapter, 'get_system_info'):\n info.update(platform_adapter.get_system_info())\n \n return jsonify(info)\n\n@app.route('/execute', methods=['POST'])\n@app.route('/setup/execute', methods=['POST'])\ndef execute_command():\n data = request.json\n # The 'command' key in the JSON request should contain the command to be executed.\n shell = data.get('shell', False)\n command = data.get('command', \"\" if shell else [])\n timeout = data.get('timeout', 120)\n \n if isinstance(command, str) and not shell:\n command = shlex.split(command)\n \n # Expand user directory\n if isinstance(command, list):\n for i, arg in enumerate(command):\n if arg.startswith(\"~/\"):\n command[i] = os.path.expanduser(arg)\n \n try:\n if platform_name == \"Windows\":\n result = subprocess.run(\n command,\n stdout=subprocess.PIPE,\n stderr=subprocess.PIPE,\n shell=shell,\n text=True,\n timeout=timeout,\n creationflags=subprocess.CREATE_NO_WINDOW,\n )\n else:\n result = subprocess.run(\n command,\n stdout=subprocess.PIPE,\n stderr=subprocess.PIPE,\n shell=shell,\n text=True,\n timeout=timeout,\n )\n \n return jsonify({\n 'status': 'success',\n 'output': result.stdout,\n 'error': result.stderr,\n 'returncode': result.returncode\n })\n except subprocess.TimeoutExpired:\n return jsonify({\n 'status': 'error',\n 'message': f'Command timeout after {timeout} seconds'\n }), 408\n except Exception as e:\n return jsonify({\n 'status': 'error',\n 'message': str(e)\n }), 500\n\n@app.route('/execute_with_verification', methods=['POST'])\n@app.route('/setup/execute_with_verification', methods=['POST'])\ndef execute_command_with_verification():\n \"\"\"Execute command and verify the result based on provided verification criteria\"\"\"\n data = request.json\n shell = data.get('shell', False)\n command = data.get('command', \"\" if shell else [])\n verification = data.get('verification', {})\n max_wait_time = data.get('max_wait_time', 10) # Maximum wait time in seconds\n check_interval = data.get('check_interval', 1) # Check interval in seconds\n \n if isinstance(command, str) and not shell:\n command = shlex.split(command)\n \n # Expand user directory\n if isinstance(command, list):\n for i, arg in enumerate(command):\n if arg.startswith(\"~/\"):\n command[i] = os.path.expanduser(arg)\n \n # Execute the main command\n try:\n if platform_name == \"Windows\":\n result = subprocess.run(\n command,\n stdout=subprocess.PIPE,\n stderr=subprocess.PIPE,\n shell=shell,\n text=True,\n timeout=120,\n creationflags=subprocess.CREATE_NO_WINDOW,\n )\n else:\n result = subprocess.run(\n command,\n stdout=subprocess.PIPE,\n stderr=subprocess.PIPE,\n shell=shell,\n text=True,\n timeout=120,\n )\n \n # If no verification is needed, return immediately\n if not verification:\n return jsonify({\n 'status': 'success',\n 'output': result.stdout,\n 'error': result.stderr,\n 'returncode': result.returncode\n })\n \n # Wait and verify the result\n start_time = time.time()\n while time.time() - start_time < max_wait_time:\n verification_passed = True\n \n # Check window existence if specified\n if 'window_exists' in verification:\n window_name = verification['window_exists']\n try:\n if platform_name == 'Linux':\n wmctrl_result = subprocess.run(\n ['wmctrl', '-l'],\n capture_output=True,\n text=True,\n check=True\n )\n if window_name.lower() not in wmctrl_result.stdout.lower():\n verification_passed = False\n elif platform_adapter:\n # Use platform adapter to check window existence\n windows = platform_adapter.list_windows() if hasattr(platform_adapter, 'list_windows') else []\n if not any(window_name.lower() in str(w).lower() for w in windows):\n verification_passed = False\n except:\n verification_passed = False\n \n # Check command execution if specified\n if 'command_success' in verification:\n verify_cmd = verification['command_success']\n try:\n verify_result = subprocess.run(\n verify_cmd,\n shell=True,\n capture_output=True,\n text=True,\n timeout=5\n )\n if verify_result.returncode != 0:\n verification_passed = False\n except:\n verification_passed = False\n \n if verification_passed:\n return jsonify({\n 'status': 'success',\n 'output': result.stdout,\n 'error': result.stderr,\n 'returncode': result.returncode,\n 'verification': 'passed',\n 'wait_time': time.time() - start_time\n })\n \n time.sleep(check_interval)\n \n # Verification failed\n return jsonify({\n 'status': 'verification_failed',\n 'output': result.stdout,\n 'error': result.stderr,\n 'returncode': result.returncode,\n 'verification': 'failed',\n 'wait_time': max_wait_time\n }), 500\n \n except Exception as e:\n return jsonify({\n 'status': 'error',\n 'message': str(e)\n }), 500\n\ndef _get_machine_architecture() -> str:\n \"\"\"Get the machine architecture, e.g., x86_64, arm64, aarch64, i386, etc.\n Returns 'amd' for x86/AMD architectures, 'arm' for ARM architectures, or 'unknown'.\n \"\"\"\n architecture = platform.machine().lower()\n if architecture in ['amd32', 'amd64', 'x86', 'x86_64', 'x86-64', 'x64', 'i386', 'i686']:\n return 'amd'\n elif architecture in ['arm64', 'aarch64', 'aarch32']:\n return 'arm'\n else:\n return 'unknown'\n\n@app.route('/setup/launch', methods=[\"POST\"])\ndef launch_app():\n data = request.json\n shell = data.get(\"shell\", False)\n command = data.get(\"command\", \"\" if shell else [])\n \n if isinstance(command, str) and not shell:\n command = shlex.split(command)\n \n # Expand user directory\n if isinstance(command, list):\n for i, arg in enumerate(command):\n if arg.startswith(\"~/\"):\n command[i] = os.path.expanduser(arg)\n \n try:\n # ARM architecture compatibility: replace google-chrome with chromium\n # ARM64 Chrome is not available yet, can only use Chromium\n if isinstance(command, list) and 'google-chrome' in command and _get_machine_architecture() == 'arm':\n index = command.index('google-chrome')\n command[index] = 'chromium'\n logger.info(\"ARM architecture detected: replacing 'google-chrome' with 'chromium'\")\n \n subprocess.Popen(command, shell=shell)\n cmd_str = command if shell else \" \".join(command)\n logger.info(f\"Application launched successfully: {cmd_str}\")\n return jsonify({\n 'status': 'success',\n 'message': f'{cmd_str} launched successfully'\n })\n except Exception as e:\n logger.error(f\"Application launch failed: {str(e)}\")\n return jsonify({\n 'status': 'error',\n 'message': str(e)\n }), 500\n\n@app.route(\"/run_python\", methods=['POST'])\ndef run_python():\n data = request.json\n code = data.get('code', None)\n timeout = data.get('timeout', 30)\n working_dir = data.get('working_dir', None)\n env = data.get('env', None)\n conda_env = data.get('conda_env', None)\n \n if not code:\n return jsonify({'status': 'error', 'message': 'Code not supplied!'}), 400\n \n # Generate unique filename\n if platform_name == \"Windows\":\n temp_filename = os.path.join(tempfile.gettempdir(), f\"python_exec_{uuid.uuid4().hex}.py\")\n else:\n temp_filename = f\"/tmp/python_exec_{uuid.uuid4().hex}.py\"\n \n try:\n with open(temp_filename, 'w') as f:\n f.write(code)\n \n # Prepare environment variables\n exec_env = os.environ.copy()\n if env:\n exec_env.update(env)\n \n # If conda_env is specified, try to use bash/cmd to activate and run\n # If conda is not available, fall back to system Python\n if conda_env:\n activation_cmd = get_conda_activation_prefix(conda_env)\n # Check if conda activation command is empty (conda not found)\n if not activation_cmd:\n logger.warning(f\"Conda environment '{conda_env}' requested but conda not found. Using system Python.\")\n conda_env = None # Disable conda and use default path\n \n if conda_env and get_conda_activation_prefix(conda_env):\n if platform_name == \"Windows\":\n # Windows: use cmd with activation\n activation_cmd = get_conda_activation_prefix(conda_env)\n full_cmd = f'{activation_cmd}python \"{temp_filename}\"'\n result = subprocess.run(\n ['cmd', '/c', full_cmd],\n stdout=subprocess.PIPE,\n stderr=subprocess.PIPE,\n text=True,\n timeout=timeout,\n cwd=working_dir or os.getcwd(),\n env=exec_env\n )\n else:\n # Linux/macOS: use bash with activation\n activation_cmd = get_conda_activation_prefix(conda_env)\n full_cmd = f'{activation_cmd}python3 \"{temp_filename}\"'\n result = subprocess.run(\n ['/bin/bash', '-c', full_cmd],\n stdout=subprocess.PIPE,\n stderr=subprocess.PIPE,\n text=True,\n timeout=timeout,\n cwd=working_dir or os.getcwd(),\n env=exec_env\n )\n else:\n # No conda activation needed\n python_cmd = 'python' if platform_name == \"Windows\" else 'python3'\n result = subprocess.run(\n [python_cmd, temp_filename],\n stdout=subprocess.PIPE,\n stderr=subprocess.PIPE,\n text=True,\n timeout=timeout,\n cwd=working_dir or os.getcwd(),\n env=exec_env\n )\n \n os.remove(temp_filename)\n \n output = result.stdout + result.stderr\n \n return jsonify({\n 'status': 'success' if result.returncode == 0 else 'error',\n 'content': output or \"Code executed successfully (no output)\",\n 'returncode': result.returncode\n })\n \n except subprocess.TimeoutExpired:\n if os.path.exists(temp_filename):\n os.remove(temp_filename)\n return jsonify({\n 'status': 'error',\n 'message': f'Execution timeout after {timeout} seconds'\n }), 408\n except Exception as e:\n if os.path.exists(temp_filename):\n os.remove(temp_filename)\n return jsonify({\n 'status': 'error',\n 'message': str(e)\n }), 500\n\n@app.route(\"/run_bash_script\", methods=['POST'])\ndef run_bash_script():\n data = request.json\n script = data.get('script', None)\n timeout = data.get('timeout', 30)\n working_dir = data.get('working_dir', None)\n env = data.get('env', None)\n conda_env = data.get('conda_env', None)\n \n if not script:\n return jsonify({'status': 'error', 'message': 'Script not supplied!'}), 400\n \n # Generate unique filename\n if platform_name == \"Windows\":\n temp_filename = os.path.join(tempfile.gettempdir(), f\"bash_exec_{uuid.uuid4().hex}.sh\")\n else:\n temp_filename = f\"/tmp/bash_exec_{uuid.uuid4().hex}.sh\"\n \n try:\n # Wrap script with conda activation if needed\n final_script = wrap_script_with_conda(script, conda_env)\n \n with open(temp_filename, 'w') as f:\n f.write(final_script)\n \n os.chmod(temp_filename, 0o755)\n \n if platform_name == \"Windows\":\n shell_cmd = ['bash', temp_filename]\n else:\n shell_cmd = ['/bin/bash', temp_filename]\n \n # Prepare environment variables\n exec_env = os.environ.copy()\n if env:\n exec_env.update(env)\n \n result = subprocess.run(\n shell_cmd,\n stdout=subprocess.PIPE,\n stderr=subprocess.STDOUT,\n text=True,\n timeout=timeout,\n cwd=working_dir or os.getcwd(),\n env=exec_env\n )\n \n os.unlink(temp_filename)\n \n return jsonify({\n 'status': 'success' if result.returncode == 0 else 'error',\n 'output': result.stdout,\n 'error': \"\",\n 'returncode': result.returncode\n })\n \n except subprocess.TimeoutExpired:\n if os.path.exists(temp_filename):\n os.unlink(temp_filename)\n return jsonify({\n 'status': 'error',\n 'output': f'Script execution timed out after {timeout} seconds',\n 'error': \"\",\n 'returncode': -1\n }), 500\n except Exception as e:\n if os.path.exists(temp_filename):\n try:\n os.unlink(temp_filename)\n except:\n pass\n return jsonify({\n 'status': 'error',\n 'output': f'Failed to execute script: {str(e)}',\n 'error': \"\",\n 'returncode': -1\n }), 500\n \n@app.route('/screenshot', methods=['GET'])\ndef capture_screen_with_cursor():\n \"\"\"Capture screenshot (including mouse cursor)\"\"\"\n try:\n buf = BytesIO()\n tmp_path = os.path.join(tempfile.gettempdir(), f\"screenshot_{uuid.uuid4().hex}.png\")\n if screenshot_helper.capture(tmp_path, with_cursor=True):\n with open(tmp_path, 'rb') as f:\n buf.write(f.read())\n os.remove(tmp_path) \n buf.seek(0)\n return send_file(buf, mimetype='image/png')\n else:\n return jsonify({'status':'error','message':'Screenshot failed'}), 500\n \n except Exception as e:\n logger.error(f\"Screenshot failed: {str(e)}\")\n return jsonify({\n 'status': 'error',\n 'message': str(e)\n }), 500\n\n@app.route('/cursor_position', methods=['GET'])\ndef get_cursor_position():\n \"\"\"Get cursor position\"\"\"\n try:\n x, y = screenshot_helper.get_cursor_position()\n return jsonify({'x': x, 'y': y, 'status': 'success'})\n except Exception as e:\n return jsonify({'status': 'error', 'message': str(e)}), 500\n\n@app.route('/screen_size', methods=['POST', 'GET'])\ndef get_screen_size():\n \"\"\"Get screen size\"\"\"\n try:\n width, height = screenshot_helper.get_screen_size()\n return jsonify({'width': width, 'height': height, 'status': 'success'})\n except Exception as e:\n return jsonify({'status': 'error', 'message': str(e)}), 500\n\n# Accessibility Tree\n@app.route(\"/accessibility\", methods=[\"GET\"])\ndef get_accessibility_tree():\n \"\"\"Get accessibility tree\"\"\"\n try:\n max_depth = request.args.get('max_depth', 10, type=int)\n tree = accessibility_helper.get_tree(max_depth=max_depth)\n return jsonify(tree)\n except Exception as e:\n logger.error(f\"Failed to get accessibility tree: {str(e)}\")\n return jsonify({\n 'status': 'error',\n 'message': str(e)\n }), 500\n\n# File Operations\n@app.route('/list_directory', methods=['POST'])\ndef list_directory():\n \"\"\"List directory contents\"\"\"\n data = request.json\n path = data.get('path', '.')\n \n try:\n path = os.path.expanduser(path)\n items = []\n \n for item in os.listdir(path):\n item_path = os.path.join(path, item)\n items.append({\n 'name': item,\n 'is_dir': os.path.isdir(item_path),\n 'is_file': os.path.isfile(item_path),\n 'size': os.path.getsize(item_path) if os.path.isfile(item_path) else None\n })\n \n return jsonify({\n 'status': 'success',\n 'path': path,\n 'items': items\n })\n except Exception as e:\n return jsonify({\n 'status': 'error',\n 'message': str(e)\n }), 500\n\n@app.route('/file', methods=['POST'])\ndef file_operation():\n \"\"\"File operations\"\"\"\n data = request.json\n operation = data.get('operation', 'read')\n path = data.get('path')\n \n if not path:\n return jsonify({'status': 'error', 'message': 'Path required'}), 400\n \n path = os.path.expanduser(path)\n \n try:\n if operation == 'read':\n with open(path, 'r') as f:\n content = f.read()\n return jsonify({\n 'status': 'success',\n 'content': content\n })\n elif operation == 'exists':\n exists = os.path.exists(path)\n return jsonify({\n 'status': 'success',\n 'exists': exists\n })\n else:\n return jsonify({\n 'status': 'error',\n 'message': f'Unknown operation: {operation}'\n }), 400\n except Exception as e:\n return jsonify({\n 'status': 'error',\n 'message': str(e)\n }), 500\n\n@app.route('/desktop_path', methods=['POST', 'GET'])\ndef get_desktop_path():\n \"\"\"Get desktop path\"\"\"\n try:\n desktop = os.path.expanduser(\"~/Desktop\")\n return jsonify({\n 'status': 'success',\n 'path': desktop\n })\n except Exception as e:\n return jsonify({\n 'status': 'error',\n 'message': str(e)\n }), 500\n\n@app.route(\"/setup/activate_window\", methods=['POST'])\ndef activate_window():\n \"\"\"Activate window\"\"\"\n data = request.json\n window_name = data.get(\"window_name\")\n strict = data.get(\"strict\", False)\n by_class_name = data.get(\"by_class\", False)\n \n if not window_name:\n return jsonify({'status': 'error', 'message': 'window_name required'}), 400\n \n try:\n if platform_adapter and hasattr(platform_adapter, 'activate_window'):\n result = platform_adapter.activate_window(window_name, strict=strict)\n if result['status'] == 'success':\n return jsonify(result)\n else:\n return jsonify(result), 400\n else:\n return jsonify({\n 'status': 'error',\n 'message': f'Window activation not supported on {platform_name}'\n }), 501\n except Exception as e:\n logger.error(f\"Window activation failed: {str(e)}\")\n return jsonify({'status': 'error', 'message': str(e)}), 500\n\n@app.route(\"/setup/close_window\", methods=[\"POST\"])\ndef close_window():\n \"\"\"Close window\"\"\"\n data = request.json\n window_name = data.get(\"window_name\")\n strict = data.get(\"strict\", False)\n by_class_name = data.get(\"by_class\", False)\n \n if not window_name:\n return jsonify({'status': 'error', 'message': 'window_name required'}), 400\n \n try:\n if platform_adapter and hasattr(platform_adapter, 'close_window'):\n result = platform_adapter.close_window(window_name, strict=strict)\n if result['status'] == 'success':\n return jsonify(result)\n else:\n return jsonify(result), 404\n else:\n return jsonify({\n 'status': 'error',\n 'message': f'Window closing not supported on {platform_name}'\n }), 501\n except Exception as e:\n logger.error(f\"Window closing failed: {str(e)}\")\n return jsonify({'status': 'error', 'message': str(e)}), 500\n\n@app.route('/window_size', methods=['POST'])\ndef get_window_size():\n \"\"\"Get window size\"\"\"\n try:\n width, height = screenshot_helper.get_screen_size()\n return jsonify({\n 'status': 'success',\n 'width': width,\n 'height': height\n })\n except Exception as e:\n return jsonify({'status': 'error', 'message': str(e)}), 500\n\n@app.route('/wallpaper', methods=['POST'])\n@app.route('/setup/change_wallpaper', methods=['POST'])\ndef set_wallpaper():\n \"\"\"Set wallpaper\"\"\"\n data = request.json\n image_path = data.get('path')\n \n if not image_path:\n return jsonify({'status': 'error', 'message': 'path required'}), 400\n \n try:\n if platform_adapter and hasattr(platform_adapter, 'set_wallpaper'):\n result = platform_adapter.set_wallpaper(image_path)\n if result['status'] == 'success':\n return jsonify(result)\n else:\n return jsonify(result), 400\n else:\n return jsonify({\n 'status': 'error',\n 'message': f'Wallpaper setting not supported on {platform_name}'\n }), 501\n except Exception as e:\n logger.error(f\"Failed to set wallpaper: {str(e)}\")\n return jsonify({'status': 'error', 'message': str(e)}), 500\n\n# Screen Recording\n@app.route('/start_recording', methods=['POST'])\ndef start_recording():\n \"\"\"Start screen recording (supports Linux, macOS, Windows)\"\"\"\n global recording_process\n \n # Check if platform adapter supports recording\n if not platform_adapter or not hasattr(platform_adapter, 'start_recording'):\n return jsonify({\n 'status': 'error',\n 'message': f'Recording not supported on {platform_name}'\n }), 501\n \n # Check if recording is already in progress\n if recording_process and recording_process.poll() is None:\n return jsonify({\n 'status': 'error',\n 'message': 'Recording is already in progress.'\n }), 400\n \n # Clean up old recording file\n if os.path.exists(recording_path):\n try:\n os.remove(recording_path)\n except OSError as e:\n logger.error(f\"Cannot delete old recording file: {e}\")\n \n try:\n # Use platform adapter to start recording\n result = platform_adapter.start_recording(recording_path)\n \n if result['status'] == 'success':\n recording_process = result.get('process')\n logger.info(\"Recording started successfully\")\n return jsonify({\n 'status': 'success',\n 'message': 'Recording started'\n })\n else:\n logger.error(f\"Failed to start recording: {result.get('message', 'Unknown error')}\")\n return jsonify({\n 'status': 'error',\n 'message': result.get('message', 'Failed to start recording')\n }), 500\n \n except Exception as e:\n logger.error(f\"Failed to start recording: {str(e)}\")\n return jsonify({\n 'status': 'error',\n 'message': str(e)\n }), 500\n\n@app.route('/end_recording', methods=['POST'])\ndef end_recording():\n \"\"\"End screen recording (supports Linux, macOS, Windows)\"\"\"\n global recording_process\n \n # Check if recording is in progress\n if not recording_process or recording_process.poll() is not None:\n recording_process = None\n return jsonify({\n 'status': 'error',\n 'message': 'No recording in progress'\n }), 400\n \n try:\n # Use platform adapter to stop recording\n if platform_adapter and hasattr(platform_adapter, 'stop_recording'):\n result = platform_adapter.stop_recording(recording_process)\n recording_process = None\n \n if result['status'] != 'success':\n logger.error(f\"Failed to stop recording: {result.get('message', 'Unknown error')}\")\n return jsonify(result), 500\n else:\n # Fallback: terminate process directly\n recording_process.send_signal(signal.SIGINT)\n try:\n recording_process.wait(timeout=15)\n except subprocess.TimeoutExpired:\n logger.warning(\"ffmpeg not responding, force terminating\")\n recording_process.kill()\n recording_process.wait()\n recording_process = None\n \n # Check if recording file exists\n # wait for ffmpeg to write the file header\n for _ in range(10):\n if os.path.exists(recording_path) and os.path.getsize(recording_path) > 0:\n break\n time.sleep(0.5)\n\n if os.path.exists(recording_path) and os.path.getsize(recording_path) > 0:\n logger.info(\"Recording ended, file saved\")\n return send_file(recording_path, as_attachment=True)\n else:\n logger.error(\"Recording file is missing or empty\")\n return abort(500, description=\"Recording file is missing or empty\")\n \n except Exception as e:\n logger.error(f\"Failed to end recording: {str(e)}\")\n if recording_process:\n try:\n recording_process.kill()\n recording_process.wait()\n except:\n pass\n recording_process = None\n return jsonify({\n 'status': 'error',\n 'message': str(e)\n }), 500\n\n@app.route('/terminal', methods=['GET'])\ndef get_terminal_output():\n \"\"\"Get terminal output (supports Linux, macOS, Windows)\"\"\"\n try:\n if platform_adapter and hasattr(platform_adapter, 'get_terminal_output'):\n output = platform_adapter.get_terminal_output()\n if output:\n return jsonify({'output': output, 'status': 'success'})\n else:\n return jsonify({\n 'status': 'error',\n 'message': f'No terminal output available on {platform_name}',\n 'platform_note': 'Make sure a terminal window is open and active'\n }), 404\n else:\n return jsonify({\n 'status': 'error',\n 'message': f'Terminal output not supported on {platform_name}'\n }), 501\n except Exception as e:\n logger.error(f\"Failed to get terminal output: {str(e)}\")\n return jsonify({'status': 'error', 'message': str(e)}), 500\n\n\n@app.route(\"/setup/upload\", methods=[\"POST\"])\ndef upload_file():\n \"\"\"Upload file\"\"\"\n if 'file' not in request.files:\n return jsonify({'status': 'error', 'message': 'No file provided'}), 400\n \n file = request.files['file']\n if file.filename == '':\n return jsonify({'status': 'error', 'message': 'No file selected'}), 400\n \n try:\n # Get target path\n target_path = request.form.get('path', os.path.expanduser('~/Desktop'))\n target_path = os.path.expanduser(target_path)\n \n # Ensure directory exists\n os.makedirs(target_path, exist_ok=True)\n \n # Save file\n file_path = os.path.join(target_path, file.filename)\n file.save(file_path)\n \n logger.info(f\"File uploaded successfully: {file_path}\")\n return jsonify({\n 'status': 'success',\n 'path': file_path,\n 'message': 'File uploaded successfully'\n })\n except Exception as e:\n logger.error(f\"File upload failed: {str(e)}\")\n return jsonify({'status': 'error', 'message': str(e)}), 500\n\n@app.route(\"/setup/download_file\", methods=[\"POST\"])\ndef download_file():\n \"\"\"Download file\"\"\"\n data = request.json\n path = data.get('path')\n \n if not path:\n return jsonify({'status': 'error', 'message': 'path required'}), 400\n \n try:\n path = os.path.expanduser(path)\n \n if not os.path.exists(path):\n return jsonify({'status': 'error', 'message': f'File not found: {path}'}), 404\n \n return send_file(path, as_attachment=True)\n except Exception as e:\n logger.error(f\"File download failed: {str(e)}\")\n return jsonify({'status': 'error', 'message': str(e)}), 500\n\n@app.route(\"/setup/open_file\", methods=['POST'])\ndef open_file():\n \"\"\"Open file (using system default application)\"\"\"\n data = request.json\n path = data.get('path')\n \n if not path:\n return jsonify({'status': 'error', 'message': 'path required'}), 400\n \n try:\n path = os.path.expanduser(path)\n \n if not os.path.exists(path):\n return jsonify({'status': 'error', 'message': f'File not found: {path}'}), 404\n \n if platform_name == \"Darwin\":\n subprocess.Popen(['open', path])\n elif platform_name == \"Linux\":\n subprocess.Popen(['xdg-open', path])\n elif platform_name == \"Windows\":\n os.startfile(path)\n \n logger.info(f\"File opened successfully: {path}\")\n return jsonify({\n 'status': 'success',\n 'message': f'File opened: {path}'\n })\n except Exception as e:\n logger.error(f\"File opening failed: {str(e)}\")\n return jsonify({'status': 'error', 'message': str(e)}), 500\n\ndef print_banner(host: str = \"127.0.0.1\", port: int = 5000, debug: bool = False):\n \"\"\"Print startup banner with server information\"\"\"\n from anytool.utils.display import print_banner as display_banner, print_section, print_separator, colorize\n \n # STARTUP INFORMATION\n display_banner(\"AnyTool ยท Local Server\")\n \n server_url = f\"http://{host}:{port}\"\n \n # Server section\n info_lines = [\n colorize(server_url, 'g', bold=True),\n ]\n if host == '0.0.0.0':\n info_lines.append(f\"{colorize('Listening on all interfaces', 'gr')} {colorize('(0.0.0.0:' + str(port) + ')', 'y')}\")\n info_lines.append(f\"{colorize(platform_name, 'gr')} ยท {colorize('Debug' if debug else 'Production', 'y' if debug else 'g')}\")\n \n print_section(\"Server\", info_lines)\n \n print()\n print_separator()\n print(f\" {colorize('Press Ctrl+C to stop', 'gr')}\")\n print()\n\ndef run_health_check_async():\n \"\"\"Asynchronous running health check\"\"\"\n def _run():\n from anytool.utils.display import colorize\n time.sleep(2)\n \n print(colorize(\"\\n - Starting health check...\\n\", 'c', bold=True))\n \n results = health_checker.check_all(test_endpoints=True)\n \n health_checker.print_results(results, show_endpoint_details=False)\n \n summary = health_checker.get_summary()\n logger.info(f\"Health check completed: {summary['fully_available']}/{summary['total']} fully available\")\n \n thread = threading.Thread(target=_run, daemon=True)\n thread.start()\n\ndef run_server(host: str = \"127.0.0.1\", port: int = 5000, debug: bool = False):\n \"\"\"\n Start desktop control server\n \n Args:\n host: Listening address (127.0.0.1 for local, 0.0.0.0 for all interfaces)\n port: Listening port\n debug: Debug mode (display detailed logs)\n \"\"\"\n global health_checker\n \n # Initialize health_checker\n base_url = f\"http://{host if host != '0.0.0.0' else '127.0.0.1'}:{port}\"\n health_checker = HealthChecker(feature_checker, base_url, auto_cleanup=False)\n \n print_banner(host, port, debug)\n\n if not debug:\n run_health_check_async()\n \n app.run(host=host, port=port, debug=debug, threaded=True)\n\ndef main():\n import argparse\n from anytool.config.utils import get_config_value\n \n parser = argparse.ArgumentParser(\n description='AnyTool Local Server - Desktop Control Server'\n )\n parser.add_argument('--host', type=str, default='127.0.0.1',\n help='Server host (default: 127.0.0.1)')\n parser.add_argument('--port', type=int, default=5000,\n help='Server port (default: 5000)')\n parser.add_argument('--debug', action='store_true',\n help='Enable debug mode')\n parser.add_argument('--config', type=str,\n help='Path to config.json file')\n \n args = parser.parse_args()\n \n config_path = args.config\n if not config_path:\n config_path = os.path.join(os.path.dirname(__file__), 'config.json')\n \n if os.path.exists(config_path):\n try:\n with open(config_path, 'r') as f:\n config = json.load(f)\n server_config = get_config_value(config, 'server', {})\n \n host = args.host if args.host != '127.0.0.1' else get_config_value(server_config, 'host', '127.0.0.1')\n port = args.port if args.port != 5000 else get_config_value(server_config, 'port', 5000)\n debug = args.debug or get_config_value(server_config, 'debug', False)\n \n run_server(host=host, port=port, debug=debug)\n except Exception as e:\n logger.error(f\"Failed to load config: {e}\")\n run_server(host=args.host, port=args.port, debug=args.debug)\n else:\n run_server(host=args.host, port=args.port, debug=args.debug)\n\n\nif __name__ == \"__main__\":\n main()"
},
{
"path": "anytool/local_server/platform_adapters/__init__.py",
"content": "import platform\nfrom typing import Optional, Any\n\nplatform_name = platform.system()\n\nif platform_name == \"Darwin\":\n try:\n from .macos_adapter import MacOSAdapter as PlatformAdapter\n ADAPTER_AVAILABLE = True\n except ImportError:\n PlatformAdapter = None\n ADAPTER_AVAILABLE = False\nelif platform_name == \"Linux\":\n try:\n from .linux_adapter import LinuxAdapter as PlatformAdapter\n ADAPTER_AVAILABLE = True\n except ImportError:\n PlatformAdapter = None\n ADAPTER_AVAILABLE = False\nelif platform_name == \"Windows\":\n try:\n from .windows_adapter import WindowsAdapter as PlatformAdapter\n ADAPTER_AVAILABLE = True\n except ImportError:\n PlatformAdapter = None\n ADAPTER_AVAILABLE = False\nelse:\n PlatformAdapter = None\n ADAPTER_AVAILABLE = False\n\ndef get_platform_adapter() -> Optional[Any]:\n if ADAPTER_AVAILABLE and PlatformAdapter:\n return PlatformAdapter()\n return None\n\n__all__ = [\"PlatformAdapter\", \"get_platform_adapter\", \"ADAPTER_AVAILABLE\"]\n\n"
},
{
"path": "anytool/local_server/platform_adapters/linux_adapter.py",
"content": "import subprocess\nimport os\nfrom typing import Dict, Any, Optional, List\nfrom anytool.utils.logging import Logger\nfrom PIL import Image\nimport pyautogui\n\ntry:\n import pyatspi\n from pyatspi import Accessible, StateType, STATE_SHOWING\n import Xlib\n from Xlib import display, X\n LINUX_LIBS_AVAILABLE = True\nexcept ImportError:\n LINUX_LIBS_AVAILABLE = False\n\nlogger = Logger.get_logger(__name__)\n\n\nclass LinuxAdapter:\n \n def __init__(self):\n if not LINUX_LIBS_AVAILABLE:\n logger.warning(\"Linux libraries are not fully installed, some features may not be available\")\n self.available = LINUX_LIBS_AVAILABLE\n \n def capture_screenshot_with_cursor(self, output_path: str) -> bool:\n \"\"\"\n Use pyautogui + pyxcursor to capture screenshot (including cursor)\n \n Args:\n output_path: Output file path\n \n Returns:\n Whether the screenshot is successful\n \"\"\"\n try:\n # Use pyautogui to capture screenshot\n screenshot = pyautogui.screenshot()\n \n # Try to add cursor\n try:\n # Import pyxcursor (should be in the same directory)\n import sys\n import os\n sys.path.insert(0, os.path.dirname(__file__))\n \n from pyxcursor import Xcursor\n \n cursor_obj = Xcursor()\n imgarray = cursor_obj.getCursorImageArrayFast()\n cursor_img = Image.fromarray(imgarray)\n cursor_x, cursor_y = pyautogui.position()\n screenshot.paste(cursor_img, (cursor_x, cursor_y), cursor_img)\n logger.info(\"Linux screenshot successfully (with cursor)\")\n except Exception as e:\n logger.warning(f\"Failed to add cursor to screenshot: {e}\")\n logger.info(\"Linux screenshot successfully (without cursor)\")\n \n screenshot.save(output_path)\n return True\n \n except Exception as e:\n logger.error(f\"Linux screenshot failed: {e}\")\n return False\n \n def activate_window(self, window_name: str, strict: bool = False, by_class: bool = False) -> Dict[str, Any]:\n \"\"\"\n Activate window (Linux uses wmctrl)\n \n Args:\n window_name: Window name\n strict: Whether to strictly match\n by_class: Whether to match by class name\n \n Returns:\n Result dictionary\n \"\"\"\n try:\n # Build wmctrl command\n flags = f\"-{'x' if by_class else ''}{'F' if strict else ''}a\"\n cmd = [\"wmctrl\", flags, window_name]\n \n subprocess.run(cmd, check=True, timeout=5)\n logger.info(f\"Linux window activated successfully: {window_name}\")\n return {'status': 'success', 'message': 'Window activated'}\n \n except subprocess.CalledProcessError as e:\n logger.warning(f\"wmctrl command execution failed: {e}\")\n return {'status': 'error', 'message': f'Window {window_name} not found or wmctrl failed'}\n except FileNotFoundError:\n logger.error(\"wmctrl not installed, please install: sudo apt install wmctrl\")\n return {'status': 'error', 'message': 'wmctrl not installed'}\n except Exception as e:\n logger.error(f\"Linux window activation failed: {e}\")\n return {'status': 'error', 'message': str(e)}\n \n def close_window(self, window_name: str, strict: bool = False, by_class: bool = False) -> Dict[str, Any]:\n \"\"\"\n Close window (Linux uses wmctrl)\n \n Args:\n window_name: Window name\n strict: Whether to strictly match\n by_class: Whether to match by class name\n \n Returns:\n Result dictionary\n \"\"\"\n try:\n # Build wmctrl command\n flags = f\"-{'x' if by_class else ''}{'F' if strict else ''}c\"\n cmd = [\"wmctrl\", flags, window_name]\n \n subprocess.run(cmd, check=True, timeout=5)\n logger.info(f\"Linux window closed successfully: {window_name}\")\n return {'status': 'success', 'message': 'Window closed'}\n \n except subprocess.CalledProcessError as e:\n logger.warning(f\"wmctrl command execution failed: {e}\")\n return {'status': 'error', 'message': f'Window {window_name} not found or wmctrl failed'}\n except FileNotFoundError:\n logger.error(\"wmctrl not installed\")\n return {'status': 'error', 'message': 'wmctrl not installed'}\n except Exception as e:\n logger.error(f\"Linux window close failed: {e}\")\n return {'status': 'error', 'message': str(e)}\n \n def get_accessibility_tree(self, max_depth: int = 10, max_width: int = 50) -> Dict[str, Any]:\n \"\"\"\n Get Linux accessibility tree (using AT-SPI)\n \n Args:\n max_depth: Maximum depth\n max_width: Maximum number of child elements per level\n \n Returns:\n Accessibility tree data\n \"\"\"\n if not LINUX_LIBS_AVAILABLE:\n return {'error': 'Linux accessibility libraries not available'}\n \n try:\n # Get desktop root node\n desktop = pyatspi.Registry.getDesktop(0)\n \n # Serialize accessibility tree\n tree = self._serialize_atspi_element(\n desktop, \n depth=0, \n max_depth=max_depth,\n max_width=max_width\n )\n \n return {\n 'tree': tree,\n 'platform': 'Linux'\n }\n \n except Exception as e:\n logger.error(f\"Linux get accessibility tree failed: {e}\")\n return {'error': str(e)}\n \n def _serialize_atspi_element(\n self, \n element: Accessible, \n depth: int = 0, \n max_depth: int = 10,\n max_width: int = 50\n ) -> Optional[Dict[str, Any]]:\n \"\"\"\n Serialize AT-SPI element to dictionary\n \n Args:\n element: AT-SPI accessible element\n depth: Current depth\n max_depth: Maximum depth\n max_width: Maximum width\n \n Returns:\n Serialized dictionary\n \"\"\"\n if depth > max_depth:\n return None\n \n try:\n result = {\n 'depth': depth,\n 'role': element.getRoleName(),\n 'name': element.name,\n }\n \n # Get states\n try:\n states = element.getState().get_states()\n result['states'] = [StateType._enum_lookup[st].split('_', 1)[1].lower() \n for st in states if st in StateType._enum_lookup]\n except:\n result['states'] = []\n \n # Get attributes\n try:\n attributes = element.get_attributes()\n if attributes:\n result['attributes'] = dict(attributes)\n except:\n result['attributes'] = {}\n \n # Get position and size (if visible)\n if STATE_SHOWING in element.getState().get_states():\n try:\n component = element.queryComponent()\n bbox = component.getExtents(pyatspi.XY_SCREEN)\n result['position'] = {'x': bbox[0], 'y': bbox[1]}\n result['size'] = {'width': bbox[2], 'height': bbox[3]}\n except:\n pass\n \n # Get text content\n try:\n text_obj = element.queryText()\n text = text_obj.getText(0, text_obj.characterCount)\n if text:\n result['text'] = text.replace(\"\\ufffc\", \"\").replace(\"\\ufffd\", \"\")\n except:\n pass\n \n # Recursively get child elements\n result['children'] = []\n try:\n child_count = min(element.childCount, max_width)\n for i in range(child_count):\n try:\n child = element.getChildAtIndex(i)\n child_data = self._serialize_atspi_element(\n child, \n depth + 1, \n max_depth,\n max_width\n )\n if child_data:\n result['children'].append(child_data)\n except Exception as e:\n logger.debug(f\"Cannot serialize child element {i}: {e}\")\n continue\n except Exception as e:\n logger.debug(f\"Cannot get child elements: {e}\")\n \n return result\n \n except Exception as e:\n logger.debug(f\"Failed to serialize element (depth={depth}): {e}\")\n return None\n \n def get_screen_size(self) -> Dict[str, int]:\n \"\"\"\n Get screen size\n \n Returns:\n Screen size dictionary\n \"\"\"\n try:\n if LINUX_LIBS_AVAILABLE:\n d = display.Display()\n screen = d.screen()\n return {\n 'width': screen.width_in_pixels,\n 'height': screen.height_in_pixels\n }\n else:\n # Use pyautogui as fallback\n size = pyautogui.size()\n return {'width': size.width, 'height': size.height}\n \n except Exception as e:\n logger.error(f\"Failed to get screen size: {e}\")\n return {'width': 1920, 'height': 1080} # Default value\n \n def list_windows(self) -> List[Dict[str, Any]]:\n \"\"\"\n List all windows\n \n Returns:\n Window list\n \"\"\"\n try:\n result = subprocess.run(\n ['wmctrl', '-l'],\n capture_output=True,\n text=True,\n check=True\n )\n \n windows = []\n for line in result.stdout.strip().split('\\n'):\n if line:\n parts = line.split(None, 3)\n if len(parts) >= 4:\n windows.append({\n 'id': parts[0],\n 'desktop': parts[1],\n 'hostname': parts[2],\n 'title': parts[3]\n })\n \n return windows\n \n except FileNotFoundError:\n logger.error(\"wmctrl not installed\")\n return []\n except Exception as e:\n logger.error(f\"List windows failed: {e}\")\n return []\n \n def get_terminal_output(self) -> Optional[str]:\n \"\"\"\n Get terminal output (GNOME Terminal)\n \n Returns:\n Terminal output content\n \"\"\"\n if not LINUX_LIBS_AVAILABLE:\n return None\n \n try:\n desktop = pyatspi.Registry.getDesktop(0)\n \n # Find gnome-terminal-server\n for app in desktop:\n if app.getRoleName() == \"application\" and app.name == \"gnome-terminal-server\":\n for frame in app:\n if frame.getRoleName() == \"frame\" and frame.getState().contains(pyatspi.STATE_ACTIVE):\n # Find terminal component\n for component in self._find_terminals(frame):\n try:\n text_obj = component.queryText()\n output = text_obj.getText(0, text_obj.characterCount)\n return output.rstrip() if output else None\n except:\n continue\n \n return None\n \n except Exception as e:\n logger.error(f\"Failed to get terminal output: {e}\")\n return None\n \n def _find_terminals(self, element) -> List[Accessible]:\n \"\"\"Recursively find terminal components\"\"\"\n terminals = []\n try:\n if element.getRoleName() == \"terminal\":\n terminals.append(element)\n \n for i in range(element.childCount):\n child = element.getChildAtIndex(i)\n terminals.extend(self._find_terminals(child))\n except:\n pass\n \n return terminals\n \n def set_wallpaper(self, image_path: str) -> Dict[str, Any]:\n \"\"\"\n Set desktop wallpaper (GNOME)\n \n Args:\n image_path: Image path\n \n Returns:\n Result dictionary\n \"\"\"\n try:\n image_path = os.path.expanduser(image_path)\n image_path = os.path.abspath(image_path)\n \n if not os.path.exists(image_path):\n return {'status': 'error', 'message': f'Image not found: {image_path}'}\n \n # Use gsettings to set wallpaper (GNOME)\n subprocess.run([\n 'gsettings', 'set', \n 'org.gnome.desktop.background', \n 'picture-uri', \n f'file://{image_path}'\n ], check=True, timeout=5)\n \n logger.info(f\"Linux wallpaper set successfully: {image_path}\")\n return {'status': 'success', 'message': 'Wallpaper set successfully'}\n \n except Exception as e:\n logger.error(f\"Linux set wallpaper failed: {e}\")\n return {'status': 'error', 'message': str(e)}\n \n def get_system_info(self) -> Dict[str, Any]:\n \"\"\"\n Get Linux system information\n \n Returns:\n System information dictionary\n \"\"\"\n try:\n # Get distribution information\n try:\n with open('/etc/os-release', 'r') as f:\n os_info = {}\n for line in f:\n if '=' in line:\n key, value = line.strip().split('=', 1)\n os_info[key] = value.strip('\"')\n distro = os_info.get('PRETTY_NAME', 'Unknown Linux')\n except:\n distro = 'Unknown Linux'\n \n # Get kernel version\n kernel = subprocess.run(\n ['uname', '-r'],\n capture_output=True,\n text=True\n ).stdout.strip()\n \n return {\n 'platform': 'Linux',\n 'distro': distro,\n 'kernel': kernel,\n 'available': self.available\n }\n \n except Exception as e:\n logger.error(f\"Failed to get system information: {e}\")\n return {\n 'platform': 'Linux',\n 'error': str(e)\n }\n \n def start_recording(self, output_path: str) -> Dict[str, Any]:\n try:\n try:\n subprocess.run(['ffmpeg', '-version'], \n capture_output=True, \n check=True,\n timeout=5)\n except (subprocess.CalledProcessError, FileNotFoundError):\n return {\n 'status': 'error',\n 'message': 'ffmpeg not installed. Install with: sudo apt install ffmpeg'\n }\n \n try:\n if LINUX_LIBS_AVAILABLE:\n from Xlib import display as xdisplay\n d = xdisplay.Display()\n screen_width = d.screen().width_in_pixels\n screen_height = d.screen().height_in_pixels\n else:\n # use pyautogui as fallback\n size = pyautogui.size()\n screen_width = size.width\n screen_height = size.height\n except:\n screen_width, screen_height = 1920, 1080\n \n command = [\n 'ffmpeg',\n '-y', \n '-f', 'x11grab',\n '-draw_mouse', '1',\n '-s', f'{screen_width}x{screen_height}', \n '-i', ':0.0', \n '-c:v', 'libx264', \n '-preset', 'ultrafast', \n '-r', '30', \n output_path\n ]\n \n process = subprocess.Popen(\n command,\n stdout=subprocess.DEVNULL,\n stderr=subprocess.PIPE,\n text=True\n )\n \n import time\n time.sleep(1)\n \n if process.poll() is not None:\n error_output = process.stderr.read() if process.stderr else \"Unknown error\"\n return {\n 'status': 'error',\n 'message': f'Failed to start recording: {error_output}'\n }\n \n logger.info(f\"Linux recording started: {output_path}\")\n return {\n 'status': 'success',\n 'message': 'Recording started',\n 'process': process\n }\n \n except Exception as e:\n logger.error(f\"Linux start recording failed: {e}\")\n return {\n 'status': 'error',\n 'message': str(e)\n }\n \n def stop_recording(self, process) -> Dict[str, Any]:\n try:\n import signal\n \n if not process or process.poll() is not None:\n return {\n 'status': 'error',\n 'message': 'No recording in progress'\n }\n \n process.send_signal(signal.SIGINT)\n \n try:\n process.wait(timeout=15)\n except subprocess.TimeoutExpired:\n logger.warning(\"ffmpeg did not respond to SIGINT, killing process\")\n process.kill()\n process.wait()\n \n logger.info(\"Linux recording stopped successfully\")\n return {\n 'status': 'success',\n 'message': 'Recording stopped'\n }\n \n except Exception as e:\n logger.error(f\"Linux stop recording failed: {e}\")\n return {\n 'status': 'error',\n 'message': str(e)\n }\n \n def get_running_applications(self) -> List[Dict[str, str]]:\n \"\"\"\n Get list of all running applications\n \n Returns:\n Application list\n \"\"\"\n try:\n import psutil\n \n apps = []\n seen_names = set()\n \n for proc in psutil.process_iter(['pid', 'name', 'exe', 'cmdline']):\n try:\n pinfo = proc.info\n name = pinfo['name']\n exe = pinfo['exe']\n \n # Skip kernel processes and system daemons\n if not exe or name.startswith('['):\n continue\n \n # Skip duplicates\n if name in seen_names:\n continue\n \n seen_names.add(name)\n \n apps.append({\n 'name': name,\n 'pid': pinfo['pid'],\n 'path': exe or '',\n 'cmdline': ' '.join(pinfo.get('cmdline', []))\n })\n \n except (psutil.NoSuchProcess, psutil.AccessDenied, psutil.ZombieProcess):\n pass\n \n return apps\n \n except ImportError:\n logger.warning(\"psutil not installed, cannot get running applications\")\n return []\n except Exception as e:\n logger.error(f\"Failed to get running applications list: {e}\")\n return []"
},
{
"path": "anytool/local_server/platform_adapters/macos_adapter.py",
"content": "import subprocess\nimport os\nfrom typing import Dict, Any, Optional, List\nfrom anytool.utils.logging import Logger\n\ntry:\n import AppKit\n import atomacos\n MACOS_LIBS_AVAILABLE = True\nexcept ImportError:\n MACOS_LIBS_AVAILABLE = False\n\nlogger = Logger.get_logger(__name__)\n\n_warning_shown = False\n\n\nclass MacOSAdapter:\n def __init__(self):\n global _warning_shown\n if not MACOS_LIBS_AVAILABLE and not _warning_shown:\n logger.warning(\"macOS libraries are not fully installed, some features may not be available\")\n logger.info(\"To install missing libraries, run: pip install pyobjc-framework-Cocoa atomacos\")\n _warning_shown = True\n self.available = MACOS_LIBS_AVAILABLE\n \n def capture_screenshot_with_cursor(self, output_path: str) -> bool:\n \"\"\"\n Capture screenshot with cursor using macOS native screencapture command\n \n Args:\n output_path: Output file path\n \n Returns:\n Whether successful\n \"\"\"\n try:\n # -C parameter includes cursor, -x disables sound, -m captures main display\n subprocess.run([\"screencapture\", \"-C\", \"-x\", \"-m\", output_path], check=True)\n logger.info(f\"macOS screenshot successfully: {output_path}\")\n return True\n except Exception as e:\n logger.error(f\"macOS screenshot failed: {e}\")\n return False\n \n def activate_window(self, window_name: str, strict: bool = False) -> Dict[str, Any]:\n \"\"\"\n Activate window (macOS uses AppleScript)\n \n Args:\n window_name: Window name or application name\n strict: Whether to strictly match\n \n Returns:\n Result dictionary\n \"\"\"\n try:\n # Try to activate application\n script = f'''\n tell application \"System Events\"\n set appName to \"{window_name}\"\n try\n -- Try to activate application by name\n set frontmost of first process whose name is appName to true\n return \"success\"\n on error\n -- Try to find window by title\n set foundWindow to false\n repeat with theProcess in (every process whose visible is true)\n try\n tell theProcess\n repeat with theWindow in windows\n if name of theWindow contains appName then\n set frontmost of theProcess to true\n set foundWindow to true\n exit repeat\n end if\n end repeat\n end tell\n end try\n if foundWindow then exit repeat\n end repeat\n \n if foundWindow then\n return \"success\"\n else\n return \"not found\"\n end if\n end try\n end tell\n '''\n \n result = subprocess.run(\n ['osascript', '-e', script],\n capture_output=True,\n text=True,\n timeout=10\n )\n \n if \"success\" in result.stdout:\n logger.info(f\"macOS window activated successfully: {window_name}\")\n return {'status': 'success', 'message': 'Window activated'}\n else:\n logger.warning(f\"macOS window not found: {window_name}\")\n return {'status': 'error', 'message': f'Window {window_name} not found'}\n \n except Exception as e:\n logger.error(f\"macOS window activation failed: {e}\")\n return {'status': 'error', 'message': str(e)}\n \n def close_window(self, window_name: str, strict: bool = False) -> Dict[str, Any]:\n \"\"\"\n Close window or application (macOS uses AppleScript)\n \n Args:\n window_name: Window name or application name\n strict: Whether to strictly match\n \n Returns:\n Result dictionary\n \"\"\"\n try:\n # Try to exit application\n script = f'''\n tell application \"{window_name}\"\n quit\n end tell\n '''\n \n subprocess.run(['osascript', '-e', script], check=True, timeout=5)\n logger.info(f\"macOS window/application closed successfully: {window_name}\")\n return {'status': 'success', 'message': 'Window/Application closed'}\n \n except subprocess.TimeoutExpired:\n # If timeout, try to force terminate\n try:\n script_force = f'''\n tell application \"{window_name}\"\n quit\n end tell\n do shell script \"killall '{window_name}'\"\n '''\n subprocess.run(['osascript', '-e', script_force], timeout=5)\n logger.info(f\"macOS application force closed: {window_name}\")\n return {'status': 'success', 'message': 'Application force closed'}\n except Exception as e2:\n logger.error(f\"macOS force close failed: {e2}\")\n return {'status': 'error', 'message': str(e2)}\n \n except Exception as e:\n logger.error(f\"macOS close window failed: {e}\")\n return {'status': 'error', 'message': str(e)}\n \n def get_accessibility_tree(self, max_depth: int = 10) -> Dict[str, Any]:\n \"\"\"\n Get macOS accessibility tree\n \n Args:\n max_depth: Maximum depth\n \n Returns:\n Accessibility tree data\n \"\"\"\n if not MACOS_LIBS_AVAILABLE:\n return {'error': 'macOS accessibility libraries not available'}\n \n try:\n # Get frontmost application\n workspace = AppKit.NSWorkspace.sharedWorkspace()\n active_app = workspace.activeApplication()\n \n if not active_app:\n return {'error': 'No active application'}\n \n app_name = active_app.get('NSApplicationName', 'Unknown')\n bundle_id = active_app.get('NSApplicationBundleIdentifier', '')\n \n logger.info(f\"Getting accessibility tree: {app_name} ({bundle_id})\")\n \n # Use atomacos to get application reference\n try:\n if bundle_id:\n app_ref = atomacos.getAppRefByBundleId(bundle_id)\n else:\n # If no bundle_id, try to find by name\n return {'error': 'Cannot find application without bundle ID'}\n \n # Serialize accessibility tree\n tree = self._serialize_ax_element(app_ref, depth=0, max_depth=max_depth)\n \n return {\n 'app_name': app_name,\n 'bundle_id': bundle_id,\n 'tree': tree,\n 'platform': 'macOS'\n }\n \n except Exception as e:\n logger.error(f\"Cannot get app reference: {e}\")\n return {\n 'error': f'Cannot get app reference: {e}',\n 'app_name': app_name,\n 'bundle_id': bundle_id\n }\n \n except Exception as e:\n logger.error(f\"macOS get accessibility tree failed: {e}\")\n return {'error': str(e)}\n \n def _serialize_ax_element(self, element, depth: int = 0, max_depth: int = 10) -> Optional[Dict[str, Any]]:\n \"\"\"\n Serialize macOS accessibility element to dictionary\n \n Args:\n element: AX element\n depth: Current depth\n max_depth: Maximum depth\n \n Returns:\n Serialized dictionary\n \"\"\"\n if depth > max_depth:\n return None\n \n try:\n result = {\n 'depth': depth\n }\n \n # Get common attributes\n try:\n result['role'] = element.AXRole if hasattr(element, 'AXRole') else 'unknown'\n except:\n result['role'] = 'unknown'\n \n try:\n result['title'] = element.AXTitle if hasattr(element, 'AXTitle') else ''\n except:\n result['title'] = ''\n \n try:\n result['description'] = element.AXDescription if hasattr(element, 'AXDescription') else ''\n except:\n result['description'] = ''\n \n try:\n result['value'] = str(element.AXValue) if hasattr(element, 'AXValue') else ''\n except:\n result['value'] = ''\n \n try:\n result['enabled'] = element.AXEnabled if hasattr(element, 'AXEnabled') else False\n except:\n result['enabled'] = False\n \n try:\n result['focused'] = element.AXFocused if hasattr(element, 'AXFocused') else False\n except:\n result['focused'] = False\n \n # Position and size\n try:\n if hasattr(element, 'AXPosition'):\n pos = element.AXPosition\n result['position'] = {'x': pos.x, 'y': pos.y}\n except:\n pass\n \n try:\n if hasattr(element, 'AXSize'):\n size = element.AXSize\n result['size'] = {'width': size.width, 'height': size.height}\n except:\n pass\n \n # Recursively get child elements (with limit)\n result['children'] = []\n try:\n if hasattr(element, 'AXChildren') and element.AXChildren:\n for i, child in enumerate(element.AXChildren[:30]): # Limit to max 30 child elements\n try:\n child_data = self._serialize_ax_element(child, depth + 1, max_depth)\n if child_data:\n result['children'].append(child_data)\n except Exception as e:\n logger.debug(f\"Cannot serialize child element {i}: {e}\")\n continue\n except Exception as e:\n logger.debug(f\"Cannot get child elements: {e}\")\n \n return result\n \n except Exception as e:\n logger.debug(f\"Failed to serialize element (depth={depth}): {e}\")\n return None\n \n def get_running_applications(self) -> List[Dict[str, str]]:\n \"\"\"\n Get list of all running applications\n \n Returns:\n Application list\n \"\"\"\n try:\n workspace = AppKit.NSWorkspace.sharedWorkspace()\n running_apps = workspace.runningApplications()\n \n apps = []\n for app in running_apps:\n if app.activationPolicy() == AppKit.NSApplicationActivationPolicyRegular:\n apps.append({\n 'name': app.localizedName() or 'Unknown',\n 'bundle_id': app.bundleIdentifier() or '',\n 'pid': app.processIdentifier(),\n 'active': app.isActive()\n })\n \n return apps\n \n except Exception as e:\n logger.error(f\"Failed to get running applications list: {e}\")\n return []\n \n def set_wallpaper(self, image_path: str) -> Dict[str, Any]:\n \"\"\"\n Set desktop wallpaper\n \n Args:\n image_path: Image path\n \n Returns:\n Result dictionary\n \"\"\"\n try:\n image_path = os.path.expanduser(image_path)\n \n if not os.path.exists(image_path):\n return {'status': 'error', 'message': f'Image not found: {image_path}'}\n \n # Use AppleScript to set wallpaper\n script = f'''\n tell application \"System Events\"\n tell every desktop\n set picture to \"{image_path}\"\n end tell\n end tell\n '''\n \n subprocess.run(['osascript', '-e', script], check=True, timeout=10)\n logger.info(f\"macOS wallpaper set successfully: {image_path}\")\n return {'status': 'success', 'message': 'Wallpaper set successfully'}\n \n except Exception as e:\n logger.error(f\"macOS set wallpaper failed: {e}\")\n return {'status': 'error', 'message': str(e)}\n \n def get_system_info(self) -> Dict[str, Any]:\n \"\"\"\n Get macOS system information\n \n Returns:\n System information dictionary\n \"\"\"\n try:\n # Get macOS version\n version = subprocess.run(\n ['sw_vers', '-productVersion'],\n capture_output=True,\n text=True\n ).stdout.strip()\n \n # Get hardware information\n model = subprocess.run(\n ['sysctl', '-n', 'hw.model'],\n capture_output=True,\n text=True\n ).stdout.strip()\n \n return {\n 'platform': 'macOS',\n 'version': version,\n 'model': model,\n 'available': self.available\n }\n \n except Exception as e:\n logger.error(f\"Failed to get system information: {e}\")\n return {\n 'platform': 'macOS',\n 'error': str(e)\n }\n \n def _detect_screen_device(self) -> str:\n \"\"\"\n Return the screen device number of avfoundation, like '1:none'\n \n On macOS, ffmpeg -f avfoundation -list_devices true -i \"\" will list all devices:\n - AVFoundation video devices (usually the camera is [0])\n - AVFoundation audio devices \n - The screen capture device usually displays as \"Capture screen X\", numbered from [1]\n \"\"\"\n try:\n probe = subprocess.run(\n ['ffmpeg', '-f', 'avfoundation', '-list_devices', 'true', '-i', ''],\n stderr=subprocess.PIPE, text=True, timeout=5\n )\n \n # Find all \"Capture screen\" devices\n screen_devices = []\n for line in probe.stderr.splitlines():\n # Match lines like \"[AVFoundation indev @ 0x...] [1] Capture screen 0\"\n if 'Capture screen' in line and '[AVFoundation' in line:\n # Extract device number from square brackets\n import re\n # Find pattern like \"] [number] Capture screen\"\n match = re.search(r'\\]\\s*\\[(\\d+)\\]\\s*Capture screen', line)\n if match:\n device_id = match.group(1)\n screen_devices.append(device_id)\n logger.info(f\"Found screen capture device: {device_id} - {line.strip()}\")\n \n # Use first found screen capture device\n if screen_devices:\n device = f'{screen_devices[0]}:none'\n logger.info(f\"Using screen capture device: {device}\")\n return device\n else:\n logger.warning(\"No screen capture device found, using default '1:none'\")\n return '1:none' # Usually screen capture is device 1\n \n except Exception as e:\n logger.warning(f\"Failed to detect screen device: {e}, using default '1:none'\")\n return '1:none'\n\n def start_recording(self, output_path: str) -> Dict[str, Any]:\n try:\n # Check if libx264 encoder is available\n result = subprocess.run(\n ['ffmpeg', '-encoders'],\n capture_output=True,\n text=True,\n timeout=5\n )\n has_libx264 = 'libx264' in result.stdout\n \n # Get screen resolution\n try:\n if MACOS_LIBS_AVAILABLE:\n from AppKit import NSScreen\n screen = NSScreen.mainScreen()\n frame = screen.frame()\n width = int(frame.size.width)\n height = int(frame.size.height)\n logger.info(f\"Screen resolution: {width}x{height}\")\n else:\n width, height = 1920, 1080\n logger.info(f\"Using default resolution: {width}x{height}\")\n except:\n width, height = 1920, 1080\n logger.info(f\"Using default resolution: {width}x{height}\")\n \n # Detect screen capture device\n screen_dev = self._detect_screen_device()\n logger.info(f\"Screen capture device: {screen_dev}\")\n \n # Build ffmpeg command\n command = [\n 'ffmpeg', '-y',\n '-f', 'avfoundation',\n '-capture_cursor', '1',\n '-capture_mouse_clicks', '1',\n '-framerate', '30',\n '-i', screen_dev, # Use detected screen device\n ]\n \n if has_libx264:\n command.extend(['-c:v', 'libx264', '-pix_fmt', 'yuv420p'])\n logger.info(\"Using libx264 encoder\")\n else:\n command.extend(['-c:v', 'mpeg4'])\n logger.info(\"Using mpeg4 encoder\")\n \n command.extend(['-r', '30', output_path])\n \n logger.info(f\"Starting recording with command: {' '.join(command)}\")\n \n process = subprocess.Popen(\n command,\n stdin=subprocess.PIPE, \n stdout=subprocess.DEVNULL,\n stderr=subprocess.PIPE,\n text=True\n )\n \n import time\n time.sleep(1.5) # Wait for a longer time to ensure ffmpeg starts\n\n # Check if process exited early\n if process.poll() is not None:\n err = process.stderr.read() if process.stderr else \"\"\n logger.error(f\"FFmpeg exited early with stderr: {err}\")\n \n if \"Operation not permitted\" in err or \"Screen Recording\" in err:\n return {\n \"status\": \"error\",\n \"message\": \"Screen-recording permission denied. Please grant permission in System Settings โ Privacy & Security โ Screen Recording.\"\n }\n \n # Check if it's a device error\n if \"Input/output error\" in err or \"Invalid argument\" in err or \"does not exist\" in err:\n return {\n \"status\": \"error\",\n \"message\": f\"Invalid screen capture device. Please ensure screen recording is enabled. Error: {err[:200]}\"\n }\n \n error_output = err or \"Unknown error\"\n return {\n 'status': 'error',\n 'message': f'Failed to start recording: {error_output[:300]}'\n }\n \n logger.info(f\"macOS recording started successfully: {output_path}\")\n return {\n 'status': 'success',\n 'message': 'Recording started',\n 'process': process\n }\n \n except Exception as e:\n logger.error(f\"macOS start recording failed: {e}\")\n return {\n 'status': 'error',\n 'message': str(e)\n }\n \n def stop_recording(self, process) -> Dict[str, Any]:\n try:\n import signal\n import time\n \n if not process or process.poll() is not None:\n return {\n 'status': 'error',\n 'message': 'No recording in progress'\n }\n\n try:\n process.stdin.write('q')\n process.stdin.flush()\n logger.info(\"Sent 'q' command to ffmpeg\")\n \n process.wait(timeout=5)\n logger.info(\"ffmpeg exited gracefully\")\n time.sleep(0.2) # give ffmpeg time to flush the file\n \n except subprocess.TimeoutExpired:\n logger.warning(\"ffmpeg did not respond to 'q', trying SIGINT\")\n \n process.send_signal(signal.SIGINT)\n try:\n process.wait(timeout=20)\n logger.info(\"ffmpeg responded to SIGINT\")\n except subprocess.TimeoutExpired:\n logger.warning(\"ffmpeg did not respond to SIGINT, killing process\")\n process.kill()\n process.wait()\n \n except Exception as e:\n logger.warning(f\"Failed to send 'q': {e}, trying SIGINT\")\n process.send_signal(signal.SIGINT)\n try:\n process.wait(timeout=20)\n except subprocess.TimeoutExpired:\n logger.warning(\"Killing ffmpeg\")\n process.kill()\n process.wait()\n \n time.sleep(0.5)\n \n logger.info(\"macOS recording stopped successfully\")\n return {\n 'status': 'success',\n 'message': 'Recording stopped'\n }\n \n except Exception as e:\n logger.error(f\"macOS stop recording failed: {e}\")\n return {\n 'status': 'error',\n 'message': str(e)\n }\n \n def list_windows(self) -> List[Dict[str, Any]]:\n \"\"\"\n List all windows\n \n Returns:\n Window list\n \"\"\"\n try:\n # Use AppleScript to get window list\n script = '''\n tell application \"System Events\"\n set windowList to {}\n repeat with theProcess in (every process whose visible is true)\n try\n set processName to name of theProcess\n tell theProcess\n repeat with theWindow in windows\n try\n set windowTitle to name of theWindow\n set windowInfo to {processName, windowTitle}\n set end of windowList to windowInfo\n end try\n end repeat\n end tell\n end try\n end repeat\n return windowList\n end tell\n '''\n \n result = subprocess.run(\n ['osascript', '-e', script],\n capture_output=True,\n text=True,\n timeout=10\n )\n \n windows = []\n if result.returncode == 0 and result.stdout:\n # Parse AppleScript output: \"app1, window1, app2, window2\"\n output = result.stdout.strip()\n if output:\n # AppleScript returns comma-separated list\n items = [item.strip() for item in output.split(',')]\n # Group by pairs (app, window)\n for i in range(0, len(items), 2):\n if i + 1 < len(items):\n windows.append({\n 'app_name': items[i],\n 'window_title': items[i + 1]\n })\n \n return windows\n \n except Exception as e:\n logger.error(f\"List windows failed: {e}\")\n return []\n \n def get_terminal_output(self) -> Optional[str]:\n \"\"\"\n Get terminal output (macOS Terminal.app or iTerm2)\n \n Returns:\n Terminal output content\n \"\"\"\n try:\n # Try to get Terminal.app output first\n script = '''\n tell application \"Terminal\"\n if (count of windows) > 0 then\n try\n set currentTab to selected tab of front window\n set terminalOutput to contents of currentTab\n return terminalOutput\n on error\n return \"\"\n end try\n else\n return \"\"\n end if\n end tell\n '''\n \n result = subprocess.run(\n ['osascript', '-e', script],\n capture_output=True,\n text=True,\n timeout=5\n )\n \n if result.returncode == 0 and result.stdout:\n output = result.stdout.strip()\n if output:\n return output\n \n # Try iTerm2 if Terminal.app failed\n iterm_script = '''\n tell application \"iTerm\"\n if (count of windows) > 0 then\n try\n tell current session of current window\n set terminalOutput to contents\n return terminalOutput\n end tell\n on error\n return \"\"\n end try\n else\n return \"\"\n end if\n end tell\n '''\n \n result = subprocess.run(\n ['osascript', '-e', iterm_script],\n capture_output=True,\n text=True,\n timeout=5\n )\n \n if result.returncode == 0 and result.stdout:\n output = result.stdout.strip()\n if output:\n return output\n \n return None\n \n except Exception as e:\n logger.error(f\"Failed to get terminal output: {e}\")\n return None"
},
{
"path": "anytool/local_server/platform_adapters/pyxcursor.py",
"content": "import os\nimport ctypes\nimport ctypes.util\nimport numpy as np\n\n# A helper function to convert data from Xlib to byte array.\nimport struct, array\n\n# Define ctypes version of XFixesCursorImage structure.\nPIXEL_DATA_PTR = ctypes.POINTER(ctypes.c_ulong)\nAtom = ctypes.c_ulong\n\n\nclass XFixesCursorImage(ctypes.Structure):\n \"\"\"\n See /usr/include/X11/extensions/Xfixes.h\n\n typedef struct {\n short\t x, y;\n unsigned short width, height;\n unsigned short xhot, yhot;\n unsigned long cursor_serial;\n unsigned long *pixels;\n if XFIXES_MAJOR >= 2\n Atom\t atom;\t/* Version >= 2 only */\n const char\t*name;\t/* Version >= 2 only */\n endif\n } XFixesCursorImage;\n \"\"\"\n _fields_ = [('x', ctypes.c_short),\n ('y', ctypes.c_short),\n ('width', ctypes.c_ushort),\n ('height', ctypes.c_ushort),\n ('xhot', ctypes.c_ushort),\n ('yhot', ctypes.c_ushort),\n ('cursor_serial', ctypes.c_ulong),\n ('pixels', PIXEL_DATA_PTR),\n ('atom', Atom),\n ('name', ctypes.c_char_p)]\n\n\nclass Display(ctypes.Structure):\n pass\n\n\nclass Xcursor:\n display = None\n\n def __init__(self, display=None):\n if not display:\n try:\n display = os.environ[\"DISPLAY\"].encode(\"utf-8\")\n except KeyError:\n raise Exception(\"$DISPLAY not set.\")\n\n # XFixeslib = ctypes.CDLL('libXfixes.so')\n XFixes = ctypes.util.find_library(\"Xfixes\")\n if not XFixes:\n raise Exception(\"No XFixes library found.\")\n self.XFixeslib = ctypes.cdll.LoadLibrary(XFixes)\n\n # xlib = ctypes.CDLL('libX11.so.6')\n x11 = ctypes.util.find_library(\"X11\")\n if not x11:\n raise Exception(\"No X11 library found.\")\n self.xlib = ctypes.cdll.LoadLibrary(x11)\n\n # Define ctypes' version of XFixesGetCursorImage function\n XFixesGetCursorImage = self.XFixeslib.XFixesGetCursorImage\n XFixesGetCursorImage.restype = ctypes.POINTER(XFixesCursorImage)\n XFixesGetCursorImage.argtypes = [ctypes.POINTER(Display)]\n self.XFixesGetCursorImage = XFixesGetCursorImage\n\n XOpenDisplay = self.xlib.XOpenDisplay\n XOpenDisplay.restype = ctypes.POINTER(Display)\n XOpenDisplay.argtypes = [ctypes.c_char_p]\n\n if not self.display:\n self.display = self.xlib.XOpenDisplay(display) # (display) or (None)\n\n def argbdata_to_pixdata(self, data, len):\n if data == None or len < 1: return None\n\n # Create byte array\n b = array.array('b', b'\\x00' * 4 * len)\n\n offset, i = 0, 0\n while i < len:\n argb = data[i] & 0xffffffff\n rgba = (argb << 8) | (argb >> 24)\n b1 = (rgba >> 24) & 0xff\n b2 = (rgba >> 16) & 0xff\n b3 = (rgba >> 8) & 0xff\n b4 = rgba & 0xff\n\n struct.pack_into(\"=BBBB\", b, offset, b1, b2, b3, b4)\n offset = offset + 4\n i = i + 1\n\n return b\n\n def getCursorImageData(self):\n # Call the function. Read data of cursor/mouse-pointer.\n cursor_data = self.XFixesGetCursorImage(self.display)\n\n if not (cursor_data and cursor_data[0]):\n raise Exception(\"Cannot read XFixesGetCursorImage()\")\n\n # Note: cursor_data is a pointer, take cursor_data[0]\n return cursor_data[0]\n\n def getCursorImageArray(self):\n data = self.getCursorImageData()\n # x, y = data.x, data.y\n height, width = data.height, data.width\n\n bytearr = self.argbdata_to_pixdata(data.pixels, height * width)\n\n imgarray = np.array(bytearr, dtype=np.uint8)\n imgarray = imgarray.reshape(height, width, 4)\n del bytearr\n\n return imgarray\n\n def getCursorImageArrayFast(self):\n data = self.getCursorImageData()\n # x, y = data.x, data.y\n height, width = data.height, data.width\n\n bytearr = ctypes.cast(data.pixels, ctypes.POINTER(ctypes.c_ulong * height * width))[0]\n imgarray = np.array(bytearray(bytearr))\n imgarray = imgarray.reshape(height, width, 8)[:, :, (0, 1, 2, 3)]\n del bytearr\n\n return imgarray\n\n def saveImage(self, imgarray, text):\n from PIL import Image\n img = Image.fromarray(imgarray)\n img.save(text)\n\n\nif __name__ == \"__main__\":\n cursor = Xcursor()\n imgarray = cursor.getCursorImageArrayFast()\n cursor.saveImage(imgarray, 'cursor_image.png')\n"
},
{
"path": "anytool/local_server/platform_adapters/windows_adapter.py",
"content": "import os\nimport ctypes\nimport subprocess\nfrom typing import Dict, Any, Optional, List\nfrom anytool.utils.logging import Logger\nfrom PIL import Image, ImageGrab\n\ntry:\n from pywinauto import Desktop\n import win32ui\n import win32gui\n import win32con\n import pygetwindow as gw\n WINDOWS_LIBS_AVAILABLE = True\nexcept ImportError:\n WINDOWS_LIBS_AVAILABLE = False\n\nlogger = Logger.get_logger(__name__)\n\n\nclass WindowsAdapter:\n \"\"\"Windows platform-specific functionality adapter\"\"\"\n \n def __init__(self):\n if not WINDOWS_LIBS_AVAILABLE:\n logger.warning(\"Windows libraries are not fully installed, some features may not be available\")\n self.available = WINDOWS_LIBS_AVAILABLE\n \n def capture_screenshot_with_cursor(self, output_path: str) -> bool:\n \"\"\"\n Capture screenshot using ImageGrab (including cursor)\n \n Args:\n output_path: Output file path\n \n Returns:\n Whether successful\n \"\"\"\n try:\n # Use ImageGrab to capture screenshot\n img = ImageGrab.grab(bbox=None, include_layered_windows=True)\n \n # Try to add cursor\n try:\n if WINDOWS_LIBS_AVAILABLE:\n cursor, hotspot = self._get_cursor()\n if cursor:\n # Get scaling ratio\n ratio = ctypes.windll.shcore.GetScaleFactorForDevice(0) / 100\n pos_win = win32gui.GetCursorPos()\n pos = (\n round(pos_win[0] * ratio - hotspot[0]),\n round(pos_win[1] * ratio - hotspot[1])\n )\n img.paste(cursor, pos, cursor)\n logger.info(\"Windows screenshot successfully (with cursor)\")\n else:\n logger.info(\"Windows screenshot successfully (without cursor)\")\n except Exception as e:\n logger.warning(f\"Cannot add cursor to screenshot: {e}\")\n logger.info(\"Windows screenshot successfully (without cursor)\")\n \n img.save(output_path)\n return True\n \n except Exception as e:\n logger.error(f\"Windows screenshot failed: {e}\")\n return False\n \n def _get_cursor(self) -> tuple:\n \"\"\"\n Get current cursor image and hotspot\n \n Returns:\n (cursor_image, (hotspot_x, hotspot_y))\n \"\"\"\n try:\n hcursor = win32gui.GetCursorInfo()[1]\n hdc = win32ui.CreateDCFromHandle(win32gui.GetDC(0))\n hbmp = win32ui.CreateBitmap()\n hbmp.CreateCompatibleBitmap(hdc, 36, 36)\n hdc_compatible = hdc.CreateCompatibleDC()\n hdc_compatible.SelectObject(hbmp)\n hdc_compatible.DrawIcon((0, 0), hcursor)\n \n bmpinfo = hbmp.GetInfo()\n bmpstr = hbmp.GetBitmapBits(True)\n cursor = Image.frombuffer(\n 'RGB',\n (bmpinfo['bmWidth'], bmpinfo['bmHeight']),\n bmpstr, 'raw', 'BGRX', 0, 1\n ).convert(\"RGBA\")\n \n win32gui.DestroyIcon(hcursor)\n win32gui.DeleteObject(hbmp.GetHandle())\n hdc_compatible.DeleteDC()\n \n # Make black pixels transparent\n pixdata = cursor.load()\n width, height = cursor.size\n for y in range(height):\n for x in range(width):\n if pixdata[x, y] == (0, 0, 0, 255):\n pixdata[x, y] = (0, 0, 0, 0)\n \n hotspot = win32gui.GetIconInfo(hcursor)[1:3]\n \n return (cursor, hotspot)\n \n except Exception as e:\n logger.debug(f\"Failed to get cursor image: {e}\")\n return (None, (0, 0))\n \n def activate_window(self, window_name: str, strict: bool = False) -> Dict[str, Any]:\n \"\"\"\n Activate window (Windows uses pygetwindow)\n \n Args:\n window_name: Window title\n strict: Whether to strictly match\n \n Returns:\n Result dictionary\n \"\"\"\n if not WINDOWS_LIBS_AVAILABLE:\n return {'status': 'error', 'message': 'Windows libraries not available'}\n \n try:\n windows = gw.getWindowsWithTitle(window_name)\n \n if not windows:\n logger.warning(f\"Window not found: {window_name}\")\n return {'status': 'error', 'message': f'Window {window_name} not found'}\n \n window = None\n if strict:\n # Strict match\n for wnd in windows:\n if wnd.title == window_name:\n window = wnd\n break\n if not window:\n return {'status': 'error', 'message': f'Window {window_name} not found (strict mode)'}\n else:\n window = windows[0]\n \n window.activate()\n logger.info(f\"Windows window activated successfully: {window_name}\")\n return {'status': 'success', 'message': 'Window activated'}\n \n except Exception as e:\n logger.error(f\"Windows window activation failed: {e}\")\n return {'status': 'error', 'message': str(e)}\n \n def close_window(self, window_name: str, strict: bool = False) -> Dict[str, Any]:\n \"\"\"\n Close window (Windows uses pygetwindow)\n \n Args:\n window_name: Window title\n strict: Whether to strictly match\n \n Returns:\n Result dictionary\n \"\"\"\n if not WINDOWS_LIBS_AVAILABLE:\n return {'status': 'error', 'message': 'Windows libraries not available'}\n \n try:\n windows = gw.getWindowsWithTitle(window_name)\n \n if not windows:\n logger.warning(f\"Window not found: {window_name}\")\n return {'status': 'error', 'message': f'Window {window_name} not found'}\n \n window = None\n if strict:\n for wnd in windows:\n if wnd.title == window_name:\n window = wnd\n break\n if not window:\n return {'status': 'error', 'message': f'Window {window_name} not found (strict mode)'}\n else:\n window = windows[0]\n \n window.close()\n logger.info(f\"Windows window closed successfully: {window_name}\")\n return {'status': 'success', 'message': 'Window closed'}\n \n except Exception as e:\n logger.error(f\"Windows window close failed: {e}\")\n return {'status': 'error', 'message': str(e)}\n \n def get_accessibility_tree(self, max_depth: int = 10, max_width: int = 50) -> Dict[str, Any]:\n \"\"\"\n Get Windows accessibility tree (using pywinauto)\n \n Args:\n max_depth: Maximum depth\n max_width: Maximum number of child elements per level\n \n Returns:\n Accessibility tree data\n \"\"\"\n if not WINDOWS_LIBS_AVAILABLE:\n return {'error': 'Windows accessibility libraries not available'}\n \n try:\n # Get desktop\n desktop = Desktop(backend=\"uia\")\n \n # Serialize accessibility tree\n tree = self._serialize_uia_element(\n desktop, \n depth=0, \n max_depth=max_depth,\n max_width=max_width,\n visited=set()\n )\n \n return {\n 'tree': tree,\n 'platform': 'Windows'\n }\n \n except Exception as e:\n logger.error(f\"Windows get accessibility tree failed: {e}\")\n return {'error': str(e)}\n \n def _serialize_uia_element(\n self, \n element, \n depth: int = 0, \n max_depth: int = 10,\n max_width: int = 50,\n visited: set = None\n ) -> Optional[Dict[str, Any]]:\n \"\"\"\n Serialize Windows UIA element to dictionary\n \n Args:\n element: UIA element\n depth: Current depth\n max_depth: Maximum depth\n max_width: Maximum width\n visited: Set of visited elements\n \n Returns:\n Serialized dictionary\n \"\"\"\n if visited is None:\n visited = set()\n \n if depth > max_depth or element in visited:\n return None\n \n visited.add(element)\n \n try:\n result = {\n 'depth': depth\n }\n \n # Get basic attributes\n try:\n result['class_name'] = element.class_name()\n except:\n result['class_name'] = 'unknown'\n \n try:\n result['name'] = element.window_text()\n except:\n result['name'] = ''\n \n # Get states\n states = {}\n state_methods = [\n 'is_enabled', 'is_visible', 'is_minimized', 'is_maximized',\n 'is_focused', 'is_checked', 'is_selected'\n ]\n \n for method_name in state_methods:\n if hasattr(element, method_name):\n try:\n method = getattr(element, method_name)\n states[method_name] = method()\n except:\n pass\n \n if states:\n result['states'] = states\n \n # Get position and size\n try:\n rectangle = element.rectangle()\n result['position'] = {\n 'left': rectangle.left,\n 'top': rectangle.top\n }\n result['size'] = {\n 'width': rectangle.width(),\n 'height': rectangle.height()\n }\n except:\n pass\n \n # Recursively get child elements\n result['children'] = []\n try:\n children = element.children()\n for i, child in enumerate(children[:max_width]):\n try:\n child_data = self._serialize_uia_element(\n child, \n depth + 1, \n max_depth,\n max_width,\n visited\n )\n if child_data:\n result['children'].append(child_data)\n except Exception as e:\n logger.debug(f\"Cannot serialize child element {i}: {e}\")\n continue\n except Exception as e:\n logger.debug(f\"Cannot get child elements: {e}\")\n \n return result\n \n except Exception as e:\n logger.debug(f\"Failed to serialize element (depth={depth}): {e}\")\n return None\n \n def list_windows(self) -> List[Dict[str, Any]]:\n \"\"\"\n List all windows\n \n Returns:\n Window list\n \"\"\"\n if not WINDOWS_LIBS_AVAILABLE:\n return []\n \n try:\n windows = gw.getAllWindows()\n \n return [\n {\n 'title': win.title,\n 'left': win.left,\n 'top': win.top,\n 'width': win.width,\n 'height': win.height,\n 'visible': win.visible,\n 'active': win.isActive\n }\n for win in windows\n if win.title # Only return windows with titles\n ]\n \n except Exception as e:\n logger.error(f\"List windows failed: {e}\")\n return []\n \n def set_wallpaper(self, image_path: str) -> Dict[str, Any]:\n \"\"\"\n Set desktop wallpaper\n \n Args:\n image_path: Image path\n \n Returns:\n Result dictionary\n \"\"\"\n try:\n image_path = os.path.expanduser(image_path)\n image_path = os.path.abspath(image_path)\n \n if not os.path.exists(image_path):\n return {'status': 'error', 'message': f'Image not found: {image_path}'}\n \n # Use Windows API to set wallpaper\n SPI_SETDESKWALLPAPER = 20\n ctypes.windll.user32.SystemParametersInfoW(\n SPI_SETDESKWALLPAPER,\n 0,\n image_path,\n 3 # SPIF_UPDATEINIFILE | SPIF_SENDCHANGE\n )\n \n logger.info(f\"Windows wallpaper set successfully: {image_path}\")\n return {'status': 'success', 'message': 'Wallpaper set successfully'}\n \n except Exception as e:\n logger.error(f\"Windows set wallpaper failed: {e}\")\n return {'status': 'error', 'message': str(e)}\n \n def get_system_info(self) -> Dict[str, Any]:\n \"\"\"\n Get Windows system information\n \n Returns:\n System information dictionary\n \"\"\"\n try:\n import platform as plat\n \n return {\n 'platform': 'Windows',\n 'version': plat.version(),\n 'release': plat.release(),\n 'edition': plat.win32_edition() if hasattr(plat, 'win32_edition') else 'Unknown',\n 'available': self.available\n }\n \n except Exception as e:\n logger.error(f\"Failed to get system information: {e}\")\n return {\n 'platform': 'Windows',\n 'error': str(e)\n }\n \n def start_recording(self, output_path: str) -> Dict[str, Any]:\n try:\n try:\n result = subprocess.run(['ffmpeg', '-version'], \n capture_output=True, \n check=True,\n timeout=5,\n creationflags=subprocess.CREATE_NO_WINDOW)\n except (subprocess.CalledProcessError, FileNotFoundError):\n return {\n 'status': 'error',\n 'message': 'ffmpeg not installed. Download from: https://ffmpeg.org/download.html'\n }\n try:\n user32 = ctypes.windll.user32\n width = user32.GetSystemMetrics(0) # SM_CXSCREEN\n height = user32.GetSystemMetrics(1) # SM_CYSCREEN\n except:\n width, height = 1920, 1080\n \n command = [\n 'ffmpeg',\n '-y', \n '-f', 'gdigrab', \n '-draw_mouse', '1', \n '-framerate', '30',\n '-video_size', f'{width}x{height}',\n '-i', 'desktop', \n '-c:v', 'libx264',\n '-preset', 'ultrafast', \n '-pix_fmt', 'yuv420p', \n '-r', '30', \n output_path\n ]\n \n process = subprocess.Popen(\n command,\n stdout=subprocess.DEVNULL,\n stderr=subprocess.PIPE,\n text=True,\n creationflags=subprocess.CREATE_NO_WINDOW\n )\n \n import time\n time.sleep(1)\n \n if process.poll() is not None:\n error_output = process.stderr.read() if process.stderr else \"Unknown error\"\n return {\n 'status': 'error',\n 'message': f'Failed to start recording: {error_output}'\n }\n \n logger.info(f\"Windows recording started: {output_path}\")\n return {\n 'status': 'success',\n 'message': 'Recording started',\n 'process': process\n }\n \n except Exception as e:\n logger.error(f\"Windows start recording failed: {e}\")\n return {\n 'status': 'error',\n 'message': str(e)\n }\n \n def stop_recording(self, process) -> Dict[str, Any]:\n try:\n if not process or process.poll() is not None:\n return {\n 'status': 'error',\n 'message': 'No recording in progress'\n }\n \n import signal\n try:\n process.send_signal(signal.CTRL_C_EVENT)\n except:\n process.terminate()\n \n try:\n process.wait(timeout=15)\n except subprocess.TimeoutExpired:\n logger.warning(\"ffmpeg did not respond, killing process\")\n process.kill()\n process.wait()\n \n logger.info(\"Windows recording stopped successfully\")\n return {\n 'status': 'success',\n 'message': 'Recording stopped'\n }\n \n except Exception as e:\n logger.error(f\"Windows stop recording failed: {e}\")\n return {\n 'status': 'error',\n 'message': str(e)\n }\n \n def get_running_applications(self) -> List[Dict[str, str]]:\n \"\"\"\n Get list of all running applications\n \n Returns:\n Application list\n \"\"\"\n if not WINDOWS_LIBS_AVAILABLE:\n return []\n \n try:\n import psutil\n \n apps = []\n seen_names = set()\n \n for proc in psutil.process_iter(['pid', 'name', 'exe']):\n try:\n pinfo = proc.info\n name = pinfo['name']\n exe = pinfo['exe']\n \n # Skip system processes\n if not exe or name in ['System', 'Registry', 'svchost.exe', 'csrss.exe']:\n continue\n \n # Skip duplicates\n if name in seen_names:\n continue\n \n seen_names.add(name)\n \n apps.append({\n 'name': name,\n 'pid': pinfo['pid'],\n 'path': exe or ''\n })\n \n except (psutil.NoSuchProcess, psutil.AccessDenied, psutil.ZombieProcess):\n pass\n \n return apps\n \n except ImportError:\n logger.warning(\"psutil not installed, cannot get running applications\")\n return []\n except Exception as e:\n logger.error(f\"Failed to get running applications list: {e}\")\n return []\n \n def get_screen_size(self) -> Dict[str, int]:\n \"\"\"\n Get screen size\n \n Returns:\n Screen size dictionary\n \"\"\"\n try:\n user32 = ctypes.windll.user32\n width = user32.GetSystemMetrics(0) # SM_CXSCREEN\n height = user32.GetSystemMetrics(1) # SM_CYSCREEN\n return {'width': width, 'height': height}\n except Exception as e:\n logger.error(f\"Failed to get screen size: {e}\")\n return {'width': 1920, 'height': 1080} # Default value\n \n def get_terminal_output(self) -> Optional[str]:\n \"\"\"\n Get terminal output (Windows Command Prompt, PowerShell, or Windows Terminal)\n \n Note: Due to Windows architecture, getting terminal output is complex.\n This method attempts to find active console windows.\n \n Returns:\n Terminal output content (limited functionality on Windows)\n \"\"\"\n try:\n # Windows doesn't provide easy access to terminal content like Linux/macOS\n # This is a limitation of the Windows platform\n # We can try to use PowerShell to get recent command history\n \n # Try to get PowerShell history\n try:\n history_path = os.path.expanduser(\n '~\\\\AppData\\\\Roaming\\\\Microsoft\\\\Windows\\\\PowerShell\\\\PSReadLine\\\\ConsoleHost_history.txt'\n )\n if os.path.exists(history_path):\n with open(history_path, 'r', encoding='utf-8', errors='ignore') as f:\n # Get last 50 lines\n lines = f.readlines()\n recent_history = ''.join(lines[-50:])\n if recent_history:\n return f\"PowerShell History (last 50 commands):\\n{recent_history}\"\n except Exception as e:\n logger.debug(f\"Cannot read PowerShell history: {e}\")\n \n # Try to get Command Prompt history using doskey\n try:\n result = subprocess.run(\n ['doskey', '/history'],\n capture_output=True,\n text=True,\n timeout=2,\n creationflags=subprocess.CREATE_NO_WINDOW\n )\n if result.returncode == 0 and result.stdout:\n return f\"Command Prompt History:\\n{result.stdout}\"\n except Exception as e:\n logger.debug(f\"Cannot get Command Prompt history: {e}\")\n \n logger.warning(\"Windows terminal output is limited - only command history available\")\n return None\n \n except Exception as e:\n logger.error(f\"Failed to get terminal output: {e}\")\n return None\n\n"
},
{
"path": "anytool/local_server/requirements.txt",
"content": "# Local server dependencies (cross-platform)\nflask>=3.1.0\npyautogui>=0.9.54\npydantic>=2.12.0\nrequests>=2.32.0\n\n# # macOS-specific dependencies (local server)\n# pyobjc-core>=12.0; sys_platform == 'darwin'\n# pyobjc-framework-cocoa>=12.0; sys_platform == 'darwin'\n# pyobjc-framework-quartz>=12.0; sys_platform == 'darwin'\n# atomacos>=3.2.0; sys_platform == 'darwin'\n\n# # Linux-specific dependencies (local server)\n# python-xlib>=0.33; sys_platform == 'linux'\n# pyatspi>=2.38.0; sys_platform == 'linux'\n# numpy>=1.24.0; sys_platform == 'linux'\n\n# # Windows-specific dependencies (local server)\n# pywinauto>=0.6.8; sys_platform == 'win32'\n# pywin32>=306; sys_platform == 'win32'\n# PyGetWindow>=0.0.9; sys_platform == 'win32'"
},
{
"path": "anytool/local_server/run.sh",
"content": "#!/bin/bash\nSCRIPT_DIR=\"$( cd \"$( dirname \"${BASH_SOURCE[0]}\" )\" && pwd )\"\nPROJECT_ROOT=\"$( cd \"$SCRIPT_DIR/../..\" && pwd )\"\n\n# Check Python\nif ! command -v python3 &> /dev/null; then\n echo \"Error: python3 not installed\"\n exit 1\nfi\n\n# Check if dependencies are installed\nif ! python3 -c \"import flask\" &> /dev/null; then\n echo \"Installing dependencies...\"\n pip3 install -q -r \"$SCRIPT_DIR/requirements.txt\" || {\n echo \"Failed to install dependencies\"\n exit 1\n }\nfi\n\n# Set PYTHONPATH and start server\nexport PYTHONPATH=\"$PROJECT_ROOT:$PYTHONPATH\"\ncd \"$PROJECT_ROOT\"\npython3 -m anytool.local_server.main"
},
{
"path": "anytool/local_server/utils/__init__.py",
"content": "from .accessibility import AccessibilityHelper\nfrom .screenshot import ScreenshotHelper\n\n__all__ = [\"AccessibilityHelper\", \"ScreenshotHelper\"]"
},
{
"path": "anytool/local_server/utils/accessibility.py",
"content": "import platform\nfrom anytool.utils.logging import Logger\nfrom typing import Dict, Any, Optional\n\nlogger = Logger.get_logger(__name__)\n\nplatform_name = platform.system()\n\n\nclass AccessibilityHelper:\n def __init__(self):\n self.platform = platform_name\n self.adapter = None\n \n try:\n if platform_name == \"Darwin\":\n from ..platform_adapters.macos_adapter import MacOSAdapter\n self.adapter = MacOSAdapter()\n elif platform_name == \"Linux\":\n from ..platform_adapters.linux_adapter import LinuxAdapter\n self.adapter = LinuxAdapter()\n elif platform_name == \"Windows\":\n from ..platform_adapters.windows_adapter import WindowsAdapter\n self.adapter = WindowsAdapter()\n except ImportError as e:\n logger.warning(f\"Failed to import platform adapter: {e}\")\n \n def get_tree(self, max_depth: int = 10) -> Dict[str, Any]:\n if not self.adapter:\n return {\n 'error': f'No adapter available for {self.platform}',\n 'platform': self.platform\n }\n \n try:\n return self.adapter.get_accessibility_tree(max_depth=max_depth)\n except Exception as e:\n logger.error(f\"Failed to get accessibility tree: {e}\")\n return {\n 'error': str(e),\n 'platform': self.platform\n }\n \n def is_available(self) -> bool:\n return self.adapter is not None and hasattr(self.adapter, 'available') and self.adapter.available\n \n def find_element_by_name(self, tree: Dict[str, Any], name: str) -> Optional[Dict[str, Any]]:\n if not tree or 'tree' not in tree:\n return None\n \n return self._search_tree(tree['tree'], 'name', name)\n \n def find_element_by_role(self, tree: Dict[str, Any], role: str) -> Optional[Dict[str, Any]]:\n if not tree or 'tree' not in tree:\n return None\n \n return self._search_tree(tree['tree'], 'role', role)\n \n def _search_tree(self, node: Dict[str, Any], key: str, value: str) -> Optional[Dict[str, Any]]:\n if not node:\n return None\n \n # Check current node\n if key in node and node[key] == value:\n return node\n \n # Recursively search child nodes\n if 'children' in node:\n for child in node['children']:\n result = self._search_tree(child, key, value)\n if result:\n return result\n \n return None\n \n def flatten_tree(self, tree: Dict[str, Any]) -> list:\n if not tree or 'tree' not in tree:\n return []\n \n result = []\n self._flatten_node(tree['tree'], result)\n return result\n \n def _flatten_node(self, node: Dict[str, Any], result: list):\n \"\"\"Recursively flatten nodes\"\"\"\n if not node:\n return\n \n # Add current node (remove children)\n node_copy = {k: v for k, v in node.items() if k != 'children'}\n result.append(node_copy)\n \n # Recursively process child nodes\n if 'children' in node:\n for child in node['children']:\n self._flatten_node(child, result)\n \n def get_visible_elements(self, tree: Dict[str, Any]) -> list:\n all_elements = self.flatten_tree(tree)\n \n visible = []\n for element in all_elements:\n if self.platform == \"Linux\":\n if 'states' in element and 'showing' in element.get('states', []):\n visible.append(element)\n elif self.platform == \"Darwin\":\n if element.get('enabled', False):\n visible.append(element)\n elif self.platform == \"Windows\":\n if element.get('states', {}).get('is_visible', False):\n visible.append(element)\n \n return visible\n \n def get_clickable_elements(self, tree: Dict[str, Any]) -> list:\n all_elements = self.flatten_tree(tree)\n \n clickable_roles = [\n 'button', 'push-button', 'toggle-button', 'radio-button',\n 'link', 'menu-item', 'AXButton', 'AXLink', 'AXMenuItem'\n ]\n \n clickable = []\n for element in all_elements:\n role = element.get('role', '').lower()\n if any(cr in role for cr in clickable_roles):\n clickable.append(element)\n \n return clickable\n \n def get_statistics(self, tree: Dict[str, Any]) -> Dict[str, Any]:\n all_elements = self.flatten_tree(tree)\n \n # Count roles\n roles = {}\n for element in all_elements:\n role = element.get('role', 'unknown')\n roles[role] = roles.get(role, 0) + 1\n \n return {\n 'total_elements': len(all_elements),\n 'visible_elements': len(self.get_visible_elements(tree)),\n 'clickable_elements': len(self.get_clickable_elements(tree)),\n 'roles': roles,\n 'platform': self.platform\n }\n\n"
},
{
"path": "anytool/local_server/utils/screenshot.py",
"content": "import platform\nimport os\nimport logging\nfrom typing import Optional, Tuple\nfrom PIL import Image\nimport pyautogui\n\nlogger = logging.getLogger(__name__)\n\nplatform_name = platform.system()\n\n\nclass ScreenshotHelper:\n def __init__(self):\n self.platform = platform_name\n self.adapter = None\n \n try:\n if platform_name == \"Darwin\":\n from ..platform_adapters.macos_adapter import MacOSAdapter\n self.adapter = MacOSAdapter()\n elif platform_name == \"Linux\":\n from ..platform_adapters.linux_adapter import LinuxAdapter\n self.adapter = LinuxAdapter()\n elif platform_name == \"Windows\":\n from ..platform_adapters.windows_adapter import WindowsAdapter\n self.adapter = WindowsAdapter()\n except ImportError as e:\n logger.warning(f\"Failed to import platform adapter: {e}\")\n \n def capture(self, output_path: str, with_cursor: bool = True) -> bool:\n try:\n # Ensure directory exists\n os.makedirs(os.path.dirname(output_path), exist_ok=True)\n \n if with_cursor and self.adapter:\n # Use platform-specific method to capture screenshot (with cursor)\n return self.adapter.capture_screenshot_with_cursor(output_path)\n else:\n # Use pyautogui to capture screenshot (without cursor)\n screenshot = pyautogui.screenshot()\n screenshot.save(output_path)\n logger.info(f\"Screenshot successfully (without cursor): {output_path}\")\n return True\n \n except Exception as e:\n logger.error(f\"Screenshot failed: {e}\")\n return False\n \n def capture_region(\n self, \n output_path: str, \n x: int, \n y: int, \n width: int, \n height: int\n ) -> bool:\n \"\"\"\n Capture specified screen region\n \n Args:\n output_path: Output path\n x: Starting x coordinate\n y: Starting y coordinate\n width: Width\n height: Height\n \n Returns:\n Whether successful\n \"\"\"\n try:\n os.makedirs(os.path.dirname(output_path), exist_ok=True)\n \n screenshot = pyautogui.screenshot(region=(x, y, width, height))\n screenshot.save(output_path)\n logger.info(f\"Region screenshot successfully: {output_path}\")\n return True\n \n except Exception as e:\n logger.error(f\"Region screenshot failed: {e}\")\n return False\n \n def get_screen_size(self) -> Tuple[int, int]:\n \"\"\"\n Get screen size\n \n Returns:\n (width, height)\n \"\"\"\n try:\n size = pyautogui.size()\n return (size.width, size.height)\n except Exception as e:\n logger.error(f\"Failed to get screen size: {e}\")\n return (1920, 1080) # Default value\n \n def get_cursor_position(self) -> Tuple[int, int]:\n \"\"\"\n Get cursor position\n \n Returns:\n (x, y)\n \"\"\"\n try:\n pos = pyautogui.position()\n return (pos.x, pos.y)\n except Exception as e:\n logger.error(f\"Failed to get cursor position: {e}\")\n return (0, 0)\n \n def capture_to_base64(self, with_cursor: bool = True) -> Optional[str]:\n \"\"\"\n Capture screenshot and convert to base64\n \n Args:\n with_cursor: Whether to include cursor\n \n Returns:\n Base64 encoded image string\n \"\"\"\n import tempfile\n import base64\n \n try:\n # Create temporary file\n with tempfile.NamedTemporaryFile(suffix='.png', delete=False) as tmp:\n tmp_path = tmp.name\n \n # Capture screenshot\n if self.capture(tmp_path, with_cursor):\n # Read and encode\n with open(tmp_path, 'rb') as f:\n img_data = f.read()\n img_base64 = base64.b64encode(img_data).decode('utf-8')\n \n # Delete temporary file\n os.remove(tmp_path)\n \n return img_base64\n else:\n if os.path.exists(tmp_path):\n os.remove(tmp_path)\n return None\n \n except Exception as e:\n logger.error(f\"Failed to convert screenshot to base64: {e}\")\n return None\n \n def compare_screenshots(self, path1: str, path2: str) -> float:\n \"\"\"\n Compare similarity between two screenshots\n \n Args:\n path1: First image path\n path2: Second image path\n \n Returns:\n Similarity (0-1), 1 means identical\n \"\"\"\n try:\n from PIL import ImageChops\n import math\n import operator\n from functools import reduce\n \n img1 = Image.open(path1)\n img2 = Image.open(path2)\n \n # Ensure same size\n if img1.size != img2.size:\n # Resize to same size\n img2 = img2.resize(img1.size)\n \n # Calculate difference\n diff = ImageChops.difference(img1, img2)\n \n # Calculate statistics\n stat = diff.histogram()\n sum_of_squares = reduce(\n operator.add,\n map(lambda h, i: h * (i ** 2), stat, range(len(stat)))\n )\n \n # Calculate RMS\n rms = math.sqrt(sum_of_squares / float(img1.size[0] * img1.size[1]))\n \n # Normalize to 0-1, RMS max value is approximately 441 (for RGB)\n similarity = 1 - (rms / 441.0)\n \n return max(0, min(1, similarity))\n \n except Exception as e:\n logger.error(f\"Failed to compare screenshots: {e}\")\n return 0.0\n \n def annotate_screenshot(\n self, \n input_path: str, \n output_path: str, \n annotations: list\n ) -> bool:\n \"\"\"\n Add annotations to screenshot\n \n Args:\n input_path: Input image path\n output_path: Output image path\n annotations: List of annotations, each annotation is a dict:\n {'type': 'rectangle'/'text', 'x': int, 'y': int, \n 'width': int, 'height': int, 'text': str, 'color': tuple}\n \n Returns:\n Whether successful\n \"\"\"\n try:\n from PIL import ImageDraw, ImageFont\n \n img = Image.open(input_path)\n draw = ImageDraw.Draw(img)\n \n for annotation in annotations:\n ann_type = annotation.get('type', 'rectangle')\n color = annotation.get('color', (255, 0, 0))\n \n if ann_type == 'rectangle':\n x = annotation.get('x', 0)\n y = annotation.get('y', 0)\n width = annotation.get('width', 100)\n height = annotation.get('height', 100)\n \n draw.rectangle(\n [(x, y), (x + width, y + height)],\n outline=color,\n width=2\n )\n \n elif ann_type == 'text':\n x = annotation.get('x', 0)\n y = annotation.get('y', 0)\n text = annotation.get('text', '')\n \n try:\n font = ImageFont.truetype(\"Arial.ttf\", 20)\n except:\n font = ImageFont.load_default()\n \n draw.text((x, y), text, fill=color, font=font)\n \n img.save(output_path)\n logger.info(f\"Annotated screenshot successfully: {output_path}\")\n return True\n \n except Exception as e:\n logger.error(f\"Failed to annotate screenshot: {e}\")\n return False"
},
{
"path": "anytool/platform/__init__.py",
"content": "from .system_info import SystemInfoClient, get_system_info, get_screen_size\nfrom .recording import RecordingClient, RecordingContextManager\nfrom .screenshot import ScreenshotClient, AutoScreenshotWrapper\nfrom .config import get_local_server_config, get_client_base_url\n\n__all__ = [\n # System Info\n \"SystemInfoClient\",\n \"get_system_info\",\n \"get_screen_size\",\n \n # Recording\n \"RecordingClient\",\n \"RecordingContextManager\",\n \n # Screenshot\n \"ScreenshotClient\",\n \"AutoScreenshotWrapper\",\n \n # Config\n \"get_local_server_config\",\n \"get_client_base_url\",\n]"
},
{
"path": "anytool/platform/config.py",
"content": "import os\nimport json\nfrom typing import Dict, Any\nfrom anytool.utils.logging import Logger\n\nlogger = Logger.get_logger(__name__)\n\ndef get_local_server_config() -> Dict[str, Any]:\n \"\"\"\n Read local server configuration.\n \n Priority:\n 1. Environment variable LOCAL_SERVER_URL (parsed into host/port)\n 2. Config file local_server/config.json\n 3. Defaults (127.0.0.1:5000)\n \n Returns:\n Dict with 'host' and 'port' from server config\n \"\"\"\n # Check environment variable first (for OSWorld/remote VM integration)\n env_url = os.getenv(\"LOCAL_SERVER_URL\")\n if env_url:\n try:\n # Parse URL like \"http://localhost:5000\"\n from urllib.parse import urlparse\n parsed = urlparse(env_url)\n host = parsed.hostname or '127.0.0.1'\n port = parsed.port or 5000\n logger.debug(f\"Using LOCAL_SERVER_URL: {host}:{port}\")\n return {\n 'host': host,\n 'port': port,\n 'debug': False,\n }\n except Exception as e:\n logger.warning(f\"Failed to parse LOCAL_SERVER_URL: {e}\")\n \n # Find local_server config file\n try:\n # Try relative path from this file\n current_dir = os.path.dirname(__file__)\n config_path = os.path.join(current_dir, '../local_server/config.json')\n config_path = os.path.abspath(config_path)\n \n if os.path.exists(config_path):\n with open(config_path, 'r') as f:\n config = json.load(f)\n server_config = config.get('server', {})\n return {\n 'host': server_config.get('host', '127.0.0.1'),\n 'port': server_config.get('port', 5000),\n 'debug': server_config.get('debug', False),\n }\n except Exception as e:\n logger.debug(f\"Failed to read local server config: {e}\")\n \n # Return defaults\n return {\n 'host': '127.0.0.1',\n 'port': 5000,\n 'debug': False,\n }\n\n\ndef get_client_base_url() -> str:\n \"\"\"\n Get base URL for connecting to local server.\n \n Priority:\n 1. Environment variable LOCAL_SERVER_URL\n 2. Read from local_server/config.json\n 3. Default http://localhost:5000\n \n Returns:\n Base URL string\n \"\"\"\n # Check environment variable first\n env_url = os.getenv(\"LOCAL_SERVER_URL\")\n if env_url:\n return env_url\n \n # Read from config file\n config = get_local_server_config()\n host = config['host']\n port = config['port']\n \n # Convert 0.0.0.0 to localhost for client\n if host == '0.0.0.0':\n host = 'localhost'\n \n return f\"http://{host}:{port}\""
},
{
"path": "anytool/platform/recording.py",
"content": "import aiohttp\nfrom typing import Optional\nfrom anytool.utils.logging import Logger\nfrom .config import get_client_base_url\n\nlogger = Logger.get_logger(__name__)\n\n\nclass RecordingClient:\n \"\"\"\n Client for screen recording via HTTP API.\n \n This client directly calls the local server's recording endpoints:\n - POST /start_recording\n - POST /end_recording\n \"\"\"\n \n def __init__(\n self,\n base_url: Optional[str] = None,\n timeout: int = 30\n ):\n \"\"\"\n Initialize recording client.\n \n Args:\n base_url: Base URL of the local server\n (default: read from local_server/config.json or env LOCAL_SERVER_URL)\n timeout: Request timeout in seconds\n \"\"\"\n # Get base_url: priority is explicit > env > config file\n if base_url is None:\n base_url = get_client_base_url()\n \n self.base_url = base_url.rstrip(\"/\")\n self.timeout = timeout\n self._session: Optional[aiohttp.ClientSession] = None\n \n async def _get_session(self) -> aiohttp.ClientSession:\n \"\"\"Get or create aiohttp session.\"\"\"\n if self._session is None or self._session.closed:\n self._session = aiohttp.ClientSession(\n timeout=aiohttp.ClientTimeout(total=self.timeout)\n )\n return self._session\n \n async def start_recording(self, auto_cleanup: bool = True) -> bool:\n \"\"\"\n Start screen recording.\n \n Args:\n auto_cleanup: If True, automatically end previous recording if one is in progress\n \"\"\"\n try:\n session = await self._get_session()\n url = f\"{self.base_url}/start_recording\"\n \n async with session.post(url) as response:\n if response.status == 200:\n logger.info(\"Screen recording started\")\n return True\n elif response.status == 400 and auto_cleanup:\n # Check if error is due to recording already in progress\n error_text = await response.text()\n if \"already in progress\" in error_text.lower():\n logger.warning(\"Recording already in progress, stopping previous recording...\")\n \n # Try to end the previous recording\n video_bytes = await self.end_recording()\n if video_bytes:\n logger.info(\"Previous recording ended successfully, retrying start...\")\n else:\n logger.warning(\"Failed to end previous recording, but will retry start anyway...\")\n \n # Retry starting recording (without auto_cleanup to avoid infinite loop)\n return await self.start_recording(auto_cleanup=False)\n else:\n logger.error(f\"Failed to start recording: HTTP {response.status} - {error_text}\")\n return False\n else:\n error_text = await response.text()\n logger.error(f\"Failed to start recording: HTTP {response.status} - {error_text}\")\n return False\n \n except Exception as e:\n logger.error(f\"Failed to start recording: {e}\")\n return False\n \n async def end_recording(self, dest: Optional[str] = None) -> Optional[bytes]:\n \"\"\"\n End screen recording and optionally save to file.\n \"\"\"\n try:\n session = await self._get_session()\n url = f\"{self.base_url}/end_recording\"\n \n # Use longer timeout for end_recording (file may be large)\n async with session.post(url, timeout=aiohttp.ClientTimeout(total=60)) as response:\n if response.status == 200:\n video_bytes = await response.read()\n \n # Save to file if destination provided\n if dest:\n try:\n with open(dest, \"wb\") as f:\n f.write(video_bytes)\n logger.info(f\"Recording saved to: {dest}\")\n except Exception as e:\n logger.error(f\"Failed to save recording file: {e}\")\n return None\n \n logger.info(\"Screen recording ended\")\n return video_bytes\n else:\n error_text = await response.text()\n logger.error(f\"Failed to end recording: HTTP {response.status} - {error_text}\")\n return None\n \n except Exception as e:\n logger.error(f\"Failed to end recording: {e}\")\n return None\n \n async def close(self):\n \"\"\"Close the HTTP session.\"\"\"\n if self._session and not self._session.closed:\n await self._session.close()\n # Give aiohttp time to finish cleanup callbacks\n import asyncio\n await asyncio.sleep(0.25)\n logger.debug(\"Recording client session closed\")\n \n async def __aenter__(self):\n \"\"\"Context manager entry.\"\"\"\n return self\n \n async def __aexit__(self, exc_type, exc_val, exc_tb):\n \"\"\"Context manager exit.\"\"\"\n await self.close()\n return False\n\n\nclass RecordingContextManager:\n\n def __init__(\n self,\n base_url: Optional[str] = None,\n output_path: Optional[str] = None,\n timeout: Optional[int] = None\n ):\n \"\"\"\n Initialize recording context manager.\n \n Args:\n base_url: Base URL of the local server (default: from config)\n output_path: Path to save recording (default: from config)\n timeout: Request timeout in seconds (default: from config)\n \"\"\"\n # Load output_path from config if not provided\n if output_path is None:\n try:\n from anytool.config import get_config\n config = get_config()\n if config.recording.screen_recording_path:\n output_path = config.recording.screen_recording_path\n except Exception:\n pass\n \n self.client = RecordingClient(base_url=base_url, timeout=timeout)\n self.output_path = output_path\n self.recording_started = False\n \n async def __aenter__(self) -> RecordingClient:\n \"\"\"Start recording on context entry.\"\"\"\n success = await self.client.start_recording()\n if success:\n self.recording_started = True\n logger.info(\"Recording context started\")\n else:\n logger.warning(\"Failed to start recording in context\")\n \n return self.client\n \n async def __aexit__(self, exc_type, exc_val, exc_tb):\n \"\"\"Stop recording on context exit.\"\"\"\n if self.recording_started:\n try:\n await self.client.end_recording(dest=self.output_path)\n logger.info(\"Recording context ended\")\n except Exception as e:\n logger.error(f\"Failed to end recording in context: {e}\")\n \n await self.client.close()\n return False"
},
{
"path": "anytool/platform/screenshot.py",
"content": "\"\"\"\nScreenshot client for capturing screens via HTTP API.\n\nThis module provides a screenshot client that captures screenshots by calling\nthe local_server's /screenshot endpoint.\n\nAlways uses HTTP API (like RecordingClient):\n- Local: http://127.0.0.1:5000/screenshot\n- Remote: http://remote-vm:5000/screenshot\n\"\"\"\nimport aiohttp\nfrom typing import Optional\nfrom anytool.utils.logging import Logger\nfrom .config import get_client_base_url\n\nlogger = Logger.get_logger(__name__)\n\n\nclass ScreenshotClient:\n \n def __init__(\n self, \n base_url: Optional[str] = None,\n timeout: int = 10\n ):\n \"\"\"\n Initialize screenshot client.\n \n Args:\n base_url: Base URL of local_server \n (default: read from config/env, typically http://127.0.0.1:5000)\n timeout: Request timeout (seconds)\n \"\"\"\n # Get base_url from config if not provided\n if base_url is None:\n base_url = get_client_base_url()\n \n self.base_url = base_url.rstrip(\"/\")\n self.timeout = timeout\n self._session = None\n \n logger.debug(f\"ScreenshotClient initialized: {self.base_url}\")\n \n async def _get_session(self) -> aiohttp.ClientSession:\n \"\"\"Get or create aiohttp session.\"\"\"\n if self._session is None or self._session.closed:\n self._session = aiohttp.ClientSession(\n timeout=aiohttp.ClientTimeout(total=self.timeout)\n )\n return self._session\n \n @staticmethod\n def _is_valid_image_response(content_type: str, data: Optional[bytes]) -> bool:\n \"\"\"\n Validate image response using magic bytes.\n \n Args:\n content_type: HTTP Content-Type header\n data: Response data bytes\n \n Returns:\n True if data is valid PNG/JPEG image\n \"\"\"\n if not isinstance(data, (bytes, bytearray)) or not data:\n return False\n \n # PNG magic bytes: \\x89PNG\\r\\n\\x1a\\n\n if len(data) >= 8 and data[:8] == b\"\\x89PNG\\r\\n\\x1a\\n\":\n return True\n \n # JPEG magic bytes: \\xff\\xd8\\xff\n if len(data) >= 3 and data[:3] == b\"\\xff\\xd8\\xff\":\n return True\n \n # Fallback to content-type check\n if content_type and (\"image/png\" in content_type or \"image/jpeg\" in content_type):\n return True\n \n return False\n \n async def capture(self) -> Optional[bytes]:\n \"\"\"\n Capture screenshot via HTTP API.\n \n Calls: GET {base_url}/screenshot\n \n Returns:\n PNG image bytes, or None on failure\n \"\"\"\n try:\n session = await self._get_session()\n url = f\"{self.base_url}/screenshot\"\n \n logger.debug(f\"Requesting screenshot: {url}\")\n \n async with session.get(url) as response:\n if response.status == 200:\n content_type = response.headers.get(\"Content-Type\", \"\")\n screenshot_bytes = await response.read()\n \n # Validate image format\n if self._is_valid_image_response(content_type, screenshot_bytes):\n logger.debug(f\"Screenshot captured: {len(screenshot_bytes)} bytes\")\n return screenshot_bytes\n else:\n logger.error(\"Invalid screenshot format received\")\n return None\n else:\n error_text = await response.text()\n logger.error(f\"Failed to capture screenshot: HTTP {response.status} - {error_text}\")\n return None\n \n except Exception as e:\n logger.error(f\"Failed to capture screenshot: {e}\")\n return None\n \n async def capture_to_file(self, output_path: str) -> bool:\n try:\n screenshot = await self.capture()\n if screenshot:\n import os\n os.makedirs(os.path.dirname(output_path) or '.', exist_ok=True)\n with open(output_path, 'wb') as f:\n f.write(screenshot)\n logger.info(f\"Screenshot saved to: {output_path}\")\n return True\n return False\n except Exception as e:\n logger.error(f\"Failed to save screenshot to file: {e}\")\n return False\n \n async def get_screen_size(self) -> tuple[int, int]:\n \"\"\"\n Get screen size via HTTP API.\n \n Calls: GET {base_url}/screen_size\n \n Returns:\n (width, height)\n \"\"\"\n try:\n session = await self._get_session()\n url = f\"{self.base_url}/screen_size\"\n \n async with session.get(url) as response:\n if response.status == 200:\n data = await response.json()\n width = data.get('width', 1920)\n height = data.get('height', 1080)\n logger.debug(f\"Screen size: {width}x{height}\")\n return (width, height)\n else:\n logger.warning(\"Failed to get screen size, using default\")\n return (1920, 1080)\n \n except Exception as e:\n logger.error(f\"Failed to get screen size: {e}\")\n return (1920, 1080)\n \n async def close(self):\n \"\"\"Close HTTP session.\"\"\"\n if self._session and not self._session.closed:\n await self._session.close()\n logger.debug(\"Screenshot client session closed\")\n \n async def __aenter__(self):\n \"\"\"Context manager entry.\"\"\"\n return self\n \n async def __aexit__(self, exc_type, exc_val, exc_tb):\n \"\"\"Context manager exit.\"\"\"\n await self.close()\n return False\n\n\nclass AutoScreenshotWrapper:\n \"\"\"\n Wrapper that automatically captures screenshots after backend calls.\n \n This wrapper can be used to wrap any backend tool/session and automatically\n capture screenshots after each operation.\n \n Usage:\n # Wrap a backend tool\n wrapped_tool = AutoScreenshotWrapper(\n tool=gui_tool,\n screenshot_client=screenshot_client,\n on_screenshot=lambda screenshot: recorder.record_step(...)\n )\n \n # Use wrapped tool normally\n result = await wrapped_tool.execute(...)\n # Screenshot is automatically captured and handled\n \"\"\"\n \n def __init__(\n self,\n tool,\n screenshot_client: Optional[ScreenshotClient] = None,\n on_screenshot=None,\n enabled: bool = True\n ):\n \"\"\"\n Initialize auto-screenshot wrapper.\n \n Args:\n tool: The tool/session to wrap\n screenshot_client: Screenshot client to use (created if None)\n on_screenshot: Callback function(screenshot_bytes) called after each screenshot\n enabled: Whether auto-screenshot is enabled\n \"\"\"\n self._tool = tool\n self._screenshot_client = screenshot_client or ScreenshotClient()\n self._on_screenshot = on_screenshot\n self._enabled = enabled\n \n def __getattr__(self, name):\n \"\"\"Delegate attribute access to wrapped tool.\"\"\"\n return getattr(self._tool, name)\n \n async def _capture_and_notify(self):\n \"\"\"Capture screenshot and notify callback.\"\"\"\n if not self._enabled:\n return\n \n try:\n screenshot = await self._screenshot_client.capture()\n if screenshot and self._on_screenshot:\n await self._on_screenshot(screenshot)\n except Exception as e:\n logger.warning(f\"Failed to auto-capture screenshot: {e}\")\n \n async def execute(self, *args, **kwargs):\n \"\"\"\n Execute tool and auto-capture screenshot.\n \"\"\"\n # Execute original method\n result = await self._tool.execute(*args, **kwargs)\n \n # Capture screenshot after execution\n await self._capture_and_notify()\n \n return result\n \n async def _arun(self, *args, **kwargs):\n \"\"\"\n Run tool and auto-capture screenshot.\n \"\"\"\n # Execute original method\n result = await self._tool._arun(*args, **kwargs)\n \n # Capture screenshot after execution\n await self._capture_and_notify()\n \n return result\n \n def enable(self):\n \"\"\"Enable auto-screenshot.\"\"\"\n self._enabled = True\n \n def disable(self):\n \"\"\"Disable auto-screenshot.\"\"\"\n self._enabled = False"
},
{
"path": "anytool/platform/system_info.py",
"content": "import aiohttp\nfrom typing import Optional, Dict, Any\nfrom anytool.utils.logging import Logger\nfrom .config import get_client_base_url\n\nlogger = Logger.get_logger(__name__)\n\n\nclass SystemInfoClient:\n \"\"\"\n This client provides simple methods to get:\n - Platform info (OS, architecture, version, etc.)\n - Screen size\n - Cursor position \n \"\"\"\n \n def __init__(\n self,\n base_url: Optional[str] = None,\n timeout: int = 10\n ):\n \"\"\"\n Initialize system info client.\n \n Args:\n base_url: Base URL of the local server\n (default: read from local_server/config.json or env LOCAL_SERVER_URL)\n timeout: Request timeout in seconds\n \"\"\"\n # Get base_url: priority is explicit > env > config file\n if base_url is None:\n base_url = get_client_base_url()\n \n self.base_url = base_url.rstrip(\"/\")\n self.timeout = timeout\n self._session: Optional[aiohttp.ClientSession] = None\n self._cached_info: Optional[Dict[str, Any]] = None\n \n async def _get_session(self) -> aiohttp.ClientSession:\n \"\"\"Get or create aiohttp session.\"\"\"\n if self._session is None or self._session.closed:\n self._session = aiohttp.ClientSession(\n timeout=aiohttp.ClientTimeout(total=self.timeout)\n )\n return self._session\n \n async def get_system_info(self, use_cache: bool = True) -> Optional[Dict[str, Any]]:\n \"\"\"\n Get comprehensive system information.\n \n Returns information including:\n - system: OS name (Linux, Darwin, Windows)\n - release: OS release version\n - version: Detailed version string\n - machine: Architecture (x86_64, arm64, etc.)\n - processor: Processor type\n - Additional platform-specific info\n \n Args:\n use_cache: Whether to use cached info (default: True)\n \"\"\"\n # Check cache\n if use_cache and self._cached_info:\n logger.debug(\"Using cached system info\")\n return self._cached_info\n \n try:\n session = await self._get_session()\n url = f\"{self.base_url}/platform\"\n \n async with session.get(url) as response:\n if response.status == 200:\n info = await response.json()\n \n # Cache the result\n if use_cache:\n self._cached_info = info\n \n logger.debug(f\"System info retrieved: {info.get('system')}\")\n return info\n else:\n error_text = await response.text()\n logger.error(f\"Failed to get system info: HTTP {response.status} - {error_text}\")\n return None\n \n except Exception as e:\n logger.error(f\"Failed to get system info: {e}\")\n return None\n \n async def get_screen_size(self) -> Optional[Dict[str, int]]:\n \"\"\"\n Get screen size.\n \n Returns:\n Dict with 'width' and 'height', or None on failure\n \"\"\"\n try:\n session = await self._get_session()\n url = f\"{self.base_url}/screen_size\"\n \n async with session.get(url) as response:\n if response.status == 200:\n size = await response.json()\n logger.debug(f\"Screen size: {size.get('width')}x{size.get('height')}\")\n return {\n \"width\": size.get(\"width\"),\n \"height\": size.get(\"height\")\n }\n else:\n error_text = await response.text()\n logger.error(f\"Failed to get screen size: HTTP {response.status} - {error_text}\")\n return None\n \n except Exception as e:\n logger.error(f\"Failed to get screen size: {e}\")\n return None\n \n async def get_cursor_position(self) -> Optional[Dict[str, int]]:\n \"\"\"\n Get current cursor position.\n \n Returns:\n Dict with 'x' and 'y', or None on failure\n \"\"\"\n try:\n session = await self._get_session()\n url = f\"{self.base_url}/cursor_position\"\n \n async with session.get(url) as response:\n if response.status == 200:\n pos = await response.json()\n return {\n \"x\": pos.get(\"x\"),\n \"y\": pos.get(\"y\")\n }\n else:\n error_text = await response.text()\n logger.error(f\"Failed to get cursor position: HTTP {response.status} - {error_text}\")\n return None\n \n except Exception as e:\n logger.error(f\"Failed to get cursor position: {e}\")\n return None\n \n def clear_cache(self):\n \"\"\"Clear cached system information.\"\"\"\n self._cached_info = None\n logger.debug(\"System info cache cleared\")\n \n async def close(self):\n \"\"\"Close the HTTP session.\"\"\"\n if self._session and not self._session.closed:\n await self._session.close()\n logger.debug(\"System info client session closed\")\n \n async def __aenter__(self):\n \"\"\"Context manager entry.\"\"\"\n return self\n \n async def __aexit__(self, exc_type, exc_val, exc_tb):\n \"\"\"Context manager exit.\"\"\"\n await self.close()\n return False\n\nasync def get_system_info(base_url: Optional[str] = None) -> Optional[Dict[str, Any]]:\n async with SystemInfoClient(base_url=base_url) as client:\n return await client.get_system_info(use_cache=False)\n\n\nasync def get_screen_size(base_url: Optional[str] = None) -> Optional[Dict[str, int]]:\n async with SystemInfoClient(base_url=base_url) as client:\n return await client.get_screen_size()"
},
{
"path": "anytool/prompts/__init__.py",
"content": "from anytool.prompts.grounding_agent_prompts import GroundingAgentPrompts\n\n__all__ = [\"GroundingAgentPrompts\"]"
},
{
"path": "anytool/prompts/grounding_agent_prompts.py",
"content": "from typing import List\n\n\nclass GroundingAgentPrompts:\n \n TASK_COMPLETE = \"\"\n \n SYSTEM_PROMPT = f\"\"\"You are a Grounding Agent. Execute tasks using tools.\n\n# Tool Execution\n\n- Select appropriate tools from descriptions and schemas\n- Provide correct parameters\n- Call multiple tools if needed\n- Tools execute immediately, results appear in next iteration\n- If you need results to decide next action, wait for next iteration\n\n# Tool Selection Tips\n\n- **MCP tools** and **Shell tools** are typically faster and more accurate when applicable\n- **GUI tools** offer finer-grained control and can handle tasks not covered by MCP/shell tools\n- Choose based on the task requirements and tool availability; prefer MCP/shell when they fit well\n\n# Visual Analysis Control\n\nGUI tools auto-analyze screenshots to extract information.\n\nTo skip analysis when NOT needed, add parameter:\n```json\n{{\"task_description\": \"...\", \"skip_visual_analysis\": true}}\n```\n\n**Decision Rule:**\n- Task goal is OPERATIONAL (open/navigate/click/show): Skip analysis\n- Task goal requires KNOWLEDGE EXTRACTION (read/extract/save data): Keep analysis\n\n**Examples:**\n- \"Open settings page\": Operational only, skip analysis\n- \"Open settings and record all values\": Needs knowledge, keep analysis\n- \"Navigate to GitHub homepage\": Operational only, skip analysis\n- \"Search Python tutorials and save top 5 titles\": Needs knowledge, keep analysis\n\n**Key principle:** If you need to extract information FROM the screen for subsequent steps or user reporting, keep analysis (don't skip).\n**Note:** Only GUI tools support this parameter. Other backend tools ignore it.\n\n# Task Completion\n\nAfter each iteration, evaluate if the task is complete:\n\n**If task is COMPLETE:**\n- Write a response summarizing what was accomplished\n- Include the completion token `{TASK_COMPLETE}` on a new line at the end of your response\n- Example response format:\n ```\n I have successfully completed the task. The file has been created at /path/to/file.txt with the requested content.\n \n {TASK_COMPLETE}\n ```\n\n**If task is NOT complete:**\n- Continue by calling the appropriate tools\n- Do NOT output `{TASK_COMPLETE}`\n- Tool results will appear in the next iteration\n\nThe token `{TASK_COMPLETE}` signals that no further iterations are needed.\"\"\"\n \n @staticmethod\n def iteration_summary(\n instruction: str,\n iteration: int,\n max_iterations: int\n ) -> str:\n \"\"\"\n Build iteration summary prompt for LLMClient auto-summary.\n LLM extracts information directly from tool results in conversation history.\n \"\"\"\n return f\"\"\"Based on the original task and the tool execution results in the conversation above, generate a structured iteration summary.\n\n**Original Task:**\n{instruction}\n\n**Progress:** Iteration {iteration} of {max_iterations}\n\n**Generate Summary in This Format:**\n\n## Iteration {iteration} Progress\n\nActions taken: \n\nKnowledge obtained (COMPLETE and SPECIFIC):\n- File locations: \n- Visual content: \n- Data retrieved: \n- URLs/Links: \n- System state: \n\nErrors encountered: \n\nCRITICAL GUIDELINES:\n- This summary is for preserving knowledge for subsequent iterations\n- Extract ALL concrete information from tool outputs in the conversation above\n- Filenames, paths, URLs - use exact values from tool outputs\n- Visual content - extract actual text/data visible, not just \"saw something\"\n- Search results - include specific data, not vague descriptions\n- The next iteration cannot see current tool outputs - this summary is the ONLY source of knowledge\"\"\"\n \n @staticmethod\n def visual_analysis(\n tool_name: str,\n num_screenshots: int,\n task_description: str = \"\"\n ) -> str:\n \"\"\"\n Build prompt for visual analysis of screenshots.\n \n Args:\n tool_name: Tool name that generated the screenshots\n num_screenshots: Number of screenshots\n task_description: Original task description for context\n \"\"\"\n screenshot_text = \"screenshot\" if num_screenshots == 1 else f\"{num_screenshots} screenshots\"\n these_text = \"this screenshot\" if num_screenshots == 1 else \"these screenshots\"\n \n task_context = f\"\"\"\n**Original Task**: {task_description}\n\nFocus on extracting information RELEVANT to this task. Prioritize content that helps accomplish the goal.\n\"\"\" if task_description else \"\"\n \n return f\"\"\"Extract the KNOWLEDGE and INFORMATION from {these_text}. This will be passed to the next iteration so it can continue working with the information (search, analyze, save, etc.). Without this extraction, the visual content would only be viewable by humans and unusable for subsequent operations.\n{task_context}\n**EXTRACT all visible knowledge content** (prioritize task-relevant information):\n1. **Text content**: Articles, documentation, code, messages, descriptions - extract the actual text\n2. **Data points**: Numbers, statistics, measurements, values, percentages - be specific\n3. **List items**: Names, titles, entries in lists/search results/files - list them out\n4. **Structured data**: Information from tables, charts, forms - describe what they contain\n5. **Key information**: URLs, paths, names, IDs, dates, labels - anything useful for next steps\n\n**IGNORE interface elements**:\n- Buttons, menus, toolbars, navigation bars\n- UI design, layout, colors, styling\n- Non-informational visual elements\n\n**Goal**: Extract usable knowledge that enables the next agent to work with this information programmatically. Be SPECIFIC and COMPLETE, but FOCUS on what's relevant to the task.\n\n{screenshot_text.capitalize()} from tool '{tool_name}'\"\"\"\n \n @staticmethod\n def final_summary(\n instruction: str,\n iterations: int\n ) -> str:\n \"\"\"\n Build prompt for generating final summary across all iterations.\n \"\"\" \n return f\"\"\"Based on the complete conversation history above (including all {iterations} iteration summaries and tool executions), generate a comprehensive final summary.\n\n## Final Task Summary\n\nTask: {instruction}\n\nWhat was accomplished: \n\nKey information obtained: \n- Files: \n- Data: \n- Findings: \n\nIssues encountered: \n\nResult: <\"Success\" or \"Incomplete\">\n\nGuidelines:\n- Consolidate information from ALL iteration summaries\n- Include concrete deliverables (file paths, data, etc.)\n- Be comprehensive but concise\n- Focus on what the user cares about\"\"\"\n \n @staticmethod\n def workspace_directory(workspace_dir: str) -> str:\n \"\"\"\n Build workspace directory information for cross-iteration/cross-backend data sharing.\n \"\"\"\n # Check if this is a benchmark scenario (LiveMCPBench /root mapping)\n # In benchmark mode, paths in query are already converted by caller (e.g., map_path_to_local)\n is_benchmark = \"/root\" in workspace_dir or \"LiveMCPBench/root\" in workspace_dir\n \n if is_benchmark:\n # Benchmark mode: all task files are in workspace directory\n return f\"\"\"**Working Directory**: `{workspace_dir}`\n- All task files (input/output) are located in this directory\n- Read from and write to this directory for all file operations\"\"\"\n else:\n # Normal mode: workspace is for intermediate results\n return f\"\"\"**Working Directory**: `{workspace_dir}`\n- Persist intermediate results here; later iterations/backends can read what you saved earlier\n- Note: User's personal files are NOT here - search in ~/Desktop, ~/Documents, ~/Downloads, etc.\"\"\"\n \n @staticmethod\n def workspace_matching_files(matching_files: List[str]) -> str:\n \"\"\"\n Build alert for files matching task requirements.\n \"\"\"\n files_str = ', '.join([f\"`{f}`\" for f in matching_files])\n return f\"\"\"**Workspace Alert**: Files matching task requirements found: {files_str}\n- Read these files to verify if they satisfy the task\n- If satisfied, mark task as completed\n- If not satisfied, modify or recreate as needed\"\"\"\n \n @staticmethod\n def workspace_recent_files(total_files: int, recent_files: List[str]) -> str:\n \"\"\"\n Build info for recently modified files.\n \"\"\"\n recent_list = ', '.join([f\"`{f}`\" for f in recent_files[:15]])\n return f\"\"\"**Workspace Info**: {total_files} files exist, {len(recent_files)} recently modified\nRecent files: {recent_list}\nConsider checking recent files before creating new ones\"\"\"\n \n @staticmethod\n def workspace_file_list(files: List[str]) -> str:\n \"\"\"\n Build list of all existing files.\n \"\"\"\n files_list = ', '.join([f\"`{f}`\" for f in files[:15]])\n if len(files) > 15:\n files_list += f\" (and {len(files) - 15} more)\"\n return f\"**Workspace Info**: {len(files)} existing file(s): {files_list}\"\n \n @staticmethod\n def iteration_feedback(\n iteration: int,\n llm_summary: str,\n add_guidance: bool = True\n ) -> str:\n \"\"\"\n Build feedback message to pass iteration summary to next iteration.\n \"\"\"\n content = f\"\"\"## Iteration {iteration} Summary\n\n{llm_summary}\"\"\"\n \n if add_guidance:\n content += f\"\"\"\n---\nNow continue with iteration {iteration + 1}. You can see the full conversation history above. Based on all progress so far, decide whether to:\n- Call more tools if the task is not yet complete\n- Output {GroundingAgentPrompts.TASK_COMPLETE} if the task is fully accomplished\"\"\"\n \n return content"
},
{
"path": "anytool/recording/__init__.py",
"content": "\"\"\"\n RecordingManager\n โโโ internal management of platform.RecordingClient\n โโโ internal management of platform.ScreenshotClient \n โโโ internal management of TrajectoryRecorder\n โโโ internal management of ActionRecorder\n\"\"\"\n\n# Auto-record the tool execution\nfrom .manager import RecordingManager\n\n# Low-level components (advanced users)\nfrom .recorder import TrajectoryRecorder\nfrom .action_recorder import ActionRecorder\n\n# Utility functions\nfrom .utils import (\n load_trajectory_from_jsonl,\n load_metadata,\n format_trajectory_for_export,\n analyze_trajectory,\n load_recording_session,\n filter_trajectory,\n extract_errors,\n generate_summary_report,\n)\n\nfrom .action_recorder import (\n load_agent_actions,\n analyze_agent_actions,\n format_agent_actions,\n)\n\n__all__ = [\n # Manager\n 'RecordingManager',\n \n # Recorders\n 'TrajectoryRecorder',\n 'ActionRecorder',\n \n # Trajectory utils\n 'load_trajectory_from_jsonl',\n 'load_metadata',\n 'format_trajectory_for_export',\n 'analyze_trajectory',\n 'load_recording_session',\n 'filter_trajectory',\n 'extract_errors',\n 'generate_summary_report',\n \n # Agent action utils\n 'load_agent_actions',\n 'analyze_agent_actions',\n 'format_agent_actions',\n]"
},
{
"path": "anytool/recording/action_recorder.py",
"content": "\"\"\"\nAgent Action Recorder\n\nRecords agent decision-making processes, reasoning, and outputs.\nFocuses on high-level agent behaviors rather than low-level tool executions.\n\"\"\"\n\nimport datetime\nimport json\nfrom typing import Any, Dict, Optional\nfrom pathlib import Path\n\nfrom anytool.utils.logging import Logger\n\nlogger = Logger.get_logger(__name__)\n\n\nclass ActionRecorder:\n \"\"\"\n Records agent actions and decision-making processes.\n \n This recorder captures the 'thinking' layer of the agent:\n - Task planning and decomposition\n - Tool selection reasoning\n - Evaluation decisions\n \"\"\"\n \n def __init__(self, trajectory_dir: Path):\n \"\"\"\n Initialize action recorder.\n \n Args:\n trajectory_dir: Directory to save action records\n \"\"\"\n self.trajectory_dir = trajectory_dir\n self.actions_file = trajectory_dir / \"agent_actions.jsonl\"\n self.step_counter = 0\n \n # Ensure directory exists\n self.trajectory_dir.mkdir(parents=True, exist_ok=True)\n \n async def record_action(\n self,\n agent_name: str,\n action_type: str,\n input_data: Optional[Dict[str, Any]] = None,\n reasoning: Optional[Dict[str, Any]] = None,\n output_data: Optional[Dict[str, Any]] = None,\n metadata: Optional[Dict[str, Any]] = None,\n related_tool_steps: Optional[list] = None,\n correlation_id: Optional[str] = None,\n ) -> Dict[str, Any]:\n \"\"\"\n Record an agent action.\n \n Args:\n agent_name: Name of the agent performing the action\n action_type: Type of action (plan | execute | evaluate | monitor)\n input_data: Input data the agent received (simplified)\n reasoning: Agent's reasoning process (structured)\n output_data: Agent's output/decision (structured)\n metadata: Additional metadata (LLM model, tokens, duration, etc.)\n related_tool_steps: List of tool execution step numbers related to this action\n correlation_id: Optional correlation ID to link related events\n \"\"\"\n self.step_counter += 1\n timestamp = datetime.datetime.now().isoformat()\n \n # Infer agent type from agent name\n agent_type = self._infer_agent_type(agent_name)\n \n action_info = {\n \"step\": self.step_counter,\n \"timestamp\": timestamp,\n \"agent_name\": agent_name,\n \"agent_type\": agent_type, \n \"action_type\": action_type,\n \"correlation_id\": correlation_id or f\"action_{self.step_counter}_{timestamp}\", \n }\n \n # Add input (with smart truncation)\n if input_data:\n action_info[\"input\"] = self._truncate_data(input_data, max_length=1000)\n \n # Add reasoning (keep structured)\n if reasoning:\n action_info[\"reasoning\"] = self._truncate_data(reasoning, max_length=2000)\n \n # Add output (keep structured)\n if output_data:\n action_info[\"output\"] = self._truncate_data(output_data, max_length=1000)\n \n # Add metadata\n if metadata:\n action_info[\"metadata\"] = metadata\n \n # Add related tool steps for correlation\n if related_tool_steps:\n action_info[\"related_tool_steps\"] = related_tool_steps\n \n # Append to JSONL file\n await self._append_to_file(action_info)\n \n logger.debug(\n f\"Recorded {action_type} action from {agent_name} (step {self.step_counter})\"\n )\n \n return action_info\n \n def _infer_agent_type(self, agent_name: str) -> str:\n name_lower = agent_name.lower()\n \n if \"host\" in name_lower:\n return \"host\"\n elif \"grounding\" in name_lower:\n return \"grounding\"\n elif \"eval\" in name_lower:\n return \"eval\"\n elif \"coordinator\" in name_lower:\n return \"coordinator\"\n else:\n return \"unknown\"\n \n def _truncate_data(self, data: Any, max_length: int) -> Any:\n if isinstance(data, str):\n if len(data) > max_length:\n return data[:max_length] + \"... [truncated]\"\n return data\n \n elif isinstance(data, dict):\n result = {}\n for key, value in data.items():\n if isinstance(value, str) and len(value) > max_length:\n result[key] = value[:max_length] + \"... [truncated]\"\n elif isinstance(value, (dict, list)):\n # Recursively truncate nested structures\n result[key] = self._truncate_data(value, max_length)\n else:\n result[key] = value\n return result\n \n elif isinstance(data, list):\n # Truncate list items\n result = []\n for item in data:\n if isinstance(item, str) and len(item) > max_length:\n result.append(item[:max_length] + \"... [truncated]\")\n elif isinstance(item, (dict, list)):\n result.append(self._truncate_data(item, max_length))\n else:\n result.append(item)\n return result\n \n else:\n return data\n \n async def _append_to_file(self, action_info: Dict[str, Any]):\n \"\"\"Append action to JSONL file.\"\"\"\n with open(self.actions_file, \"a\", encoding=\"utf-8\") as f:\n f.write(json.dumps(action_info, ensure_ascii=False))\n f.write(\"\\n\")\n \n def get_step_count(self) -> int:\n \"\"\"Get current step count.\"\"\"\n return self.step_counter\n\n\ndef load_agent_actions(trajectory_dir: str) -> list:\n \"\"\"\n Load agent actions from a trajectory directory.\n \"\"\"\n actions_file = Path(trajectory_dir) / \"agent_actions.jsonl\"\n \n if not actions_file.exists():\n logger.warning(f\"Agent actions file not found: {actions_file}\")\n return []\n \n actions = []\n try:\n with open(actions_file, \"r\", encoding=\"utf-8\") as f:\n for line in f:\n line = line.strip()\n if line:\n actions.append(json.loads(line))\n \n logger.info(f\"Loaded {len(actions)} agent actions from {actions_file}\")\n return actions\n \n except Exception as e:\n logger.error(f\"Failed to load agent actions from {actions_file}: {e}\")\n return []\n\n\ndef analyze_agent_actions(actions: list) -> Dict[str, Any]:\n \"\"\"\n Analyze agent actions and generate statistics.\n \"\"\"\n if not actions:\n return {\n \"total_actions\": 0,\n \"by_agent\": {},\n \"by_type\": {},\n }\n \n # Count by agent\n by_agent = {}\n by_type = {}\n \n for action in actions:\n agent_name = action.get(\"agent_name\", \"unknown\")\n action_type = action.get(\"action_type\", \"unknown\")\n \n by_agent[agent_name] = by_agent.get(agent_name, 0) + 1\n by_type[action_type] = by_type.get(action_type, 0) + 1\n \n return {\n \"total_actions\": len(actions),\n \"by_agent\": by_agent,\n \"by_type\": by_type,\n }\n\n\ndef format_agent_actions(actions: list, format_type: str = \"compact\") -> str:\n \"\"\"\n Format agent actions for display.\n \"\"\"\n if not actions:\n return \"No agent actions recorded\"\n \n if format_type == \"compact\":\n lines = []\n for action in actions:\n step = action.get(\"step\", \"?\")\n agent = action.get(\"agent_name\", \"?\")\n action_type = action.get(\"action_type\", \"?\")\n \n # Try to extract key info from reasoning or output\n key_info = \"\"\n if action.get(\"reasoning\"):\n thought = action[\"reasoning\"].get(\"thought\", \"\")\n if thought:\n key_info = f\": {thought[:60]}...\"\n \n lines.append(f\"Step {step}: [{agent}] {action_type}{key_info}\")\n \n return \"\\n\".join(lines)\n \n elif format_type == \"detailed\":\n lines = []\n for action in actions:\n lines.append(f\"\\n{'='*60}\")\n lines.append(f\"Step {action.get('step', '?')}: {action.get('agent_name', '?')}\")\n lines.append(f\"Type: {action.get('action_type', '?')}\")\n lines.append(f\"Time: {action.get('timestamp', '?')}\")\n \n if action.get(\"reasoning\"):\n lines.append(\"\\nReasoning:\")\n lines.append(json.dumps(action[\"reasoning\"], indent=2, ensure_ascii=False))\n \n if action.get(\"output\"):\n lines.append(\"\\nOutput:\")\n lines.append(json.dumps(action[\"output\"], indent=2, ensure_ascii=False))\n \n if action.get(\"metadata\"):\n lines.append(\"\\nMetadata:\")\n lines.append(json.dumps(action[\"metadata\"], indent=2, ensure_ascii=False))\n \n return \"\\n\".join(lines)\n \n else:\n raise ValueError(f\"Unknown format type: {format_type}\")"
},
{
"path": "anytool/recording/manager.py",
"content": "import datetime\nimport json\nimport ast\nimport types\nfrom typing import Any, Dict, List, Optional\nfrom pathlib import Path\n\nfrom anytool.utils.logging import Logger\nfrom .recorder import TrajectoryRecorder\nfrom .action_recorder import ActionRecorder\n\nlogger = Logger.get_logger(__name__)\n\n\nclass RecordingManager:\n # Global instance management (singleton pattern)\n _global_instance: Optional['RecordingManager'] = None\n \n def __init__(\n self,\n enabled: bool = True,\n task_id: str = \"\",\n log_dir: str = \"./logs/recordings\",\n backends: Optional[List[str]] = None,\n enable_screenshot: bool = True,\n enable_video: bool = False,\n enable_conversation_log: bool = True,\n auto_save_interval: int = 10,\n server_url: Optional[str] = None,\n agent_name: str = \"GroundingAgent\",\n ):\n \"\"\"\n Initialize automatic recording manager\n \n Args:\n enabled: whether to enable recording\n task_id: task ID (for naming recording directory)\n log_dir: log directory path\n backends: list of backends to record (None = all)\n (optional: \"mcp\", \"gui\", \"shell\", \"system\", \"web\")\n enable_screenshot: whether to enable screenshot (through platform.ScreenshotClient)\n enable_video: whether to enable video recording (through platform.RecordingClient)\n enable_conversation_log: whether to save LLM conversations to conversations.jsonl (default: True)\n auto_save_interval: automatic save interval (steps)\n server_url: local server address (None = read from config/environment variables)\n agent_name: name of the agent performing the recording (default: \"GroundingAgent\")\n \"\"\"\n self.enabled = enabled\n self.task_id = task_id\n self.log_dir = log_dir\n self.backends = set(backends) if backends else {\"mcp\", \"gui\", \"shell\", \"system\", \"web\"}\n self.enable_screenshot = enable_screenshot\n self.enable_video = enable_video\n self.enable_conversation_log = enable_conversation_log\n self.auto_save_interval = auto_save_interval\n self.server_url = server_url\n self.agent_name = agent_name\n \n # internal state\n self._recorder: Optional[TrajectoryRecorder] = None\n self._action_recorder: Optional[ActionRecorder] = None\n self._is_started = False\n self._step_counter = 0\n \n # registered LLM clients (for automatic recording)\n self._registered_llm_clients = []\n # Store original methods for restoration\n self._original_methods = {}\n \n # video/screenshot clients (internal management)\n self._recording_client = None\n self._screenshot_client = None\n \n # Register as global instance\n RecordingManager._global_instance = self\n\n @classmethod\n def is_recording(cls) -> bool:\n \"\"\"\n Check if there is an active recording session\n \n Returns:\n bool: True if recording is active\n \"\"\"\n return cls._global_instance is not None and cls._global_instance._is_started\n \n @classmethod\n async def record_retrieved_tools(\n cls,\n task_instruction: str,\n tools: List[Any],\n search_debug_info: Optional[Dict[str, Any]] = None,\n ):\n \"\"\"\n Record the tools retrieved for a task\n \n Args:\n task_instruction: The task instruction used for retrieval\n tools: List of retrieved tools\n search_debug_info: Debug info from search (similarity scores, LLM selections)\n \"\"\"\n instance = cls._global_instance\n if not instance or not instance._is_started or not instance._recorder:\n return\n \n # Extract tool info\n tool_info = []\n for tool in tools:\n info = {\n \"name\": getattr(tool, \"name\", str(tool)),\n }\n if hasattr(tool, \"backend_type\"):\n info[\"backend\"] = tool.backend_type.value if hasattr(tool.backend_type, \"value\") else str(tool.backend_type)\n if hasattr(tool, \"_runtime_info\") and tool._runtime_info:\n info[\"server_name\"] = tool._runtime_info.server_name\n tool_info.append(info)\n \n # Build metadata\n metadata = {\n \"instruction\": task_instruction[:500], # Truncate long instructions\n \"count\": len(tools),\n \"tools\": tool_info,\n }\n \n # Add search debug info if available\n if search_debug_info:\n metadata[\"search_debug\"] = {\n \"search_mode\": search_debug_info.get(\"search_mode\", \"\"),\n \"total_candidates\": search_debug_info.get(\"total_candidates\", 0),\n \"mcp_count\": search_debug_info.get(\"mcp_count\", 0),\n \"non_mcp_count\": search_debug_info.get(\"non_mcp_count\", 0),\n \"llm_filter\": search_debug_info.get(\"llm_filter\", {}),\n \"tool_scores\": search_debug_info.get(\"tool_scores\", []),\n }\n \n # Save to metadata\n await instance._recorder.add_metadata(\"retrieved_tools\", metadata)\n \n logger.info(f\"Recorded {len(tools)} retrieved tools (with search debug info: {search_debug_info is not None})\")\n \n @classmethod\n async def record_iteration_context(\n cls,\n iteration: int,\n messages_input: List[Dict[str, Any]],\n messages_output: List[Dict[str, Any]],\n llm_response_summary: Dict[str, Any],\n max_content_length: int = 5000,\n ):\n \"\"\"\n Record a single iteration's LLM conversation to conversations.jsonl (real-time).\n \n Args:\n iteration: Iteration number\n messages_input: Messages sent to LLM\n messages_output: Messages after LLM response \n llm_response_summary: Summary of LLM response\n max_content_length: Max length for message content truncation\n \"\"\"\n instance = cls._global_instance\n if not instance or not instance._is_started or not instance._recorder:\n return\n \n # Check if conversation recording is enabled\n if not getattr(instance, 'enable_conversation_log', True):\n return\n \n def truncate_message_content(messages: List[Dict]) -> List[Dict]:\n \"\"\"Truncate message content to avoid huge log files.\"\"\"\n result = []\n for msg in messages:\n new_msg = {\"role\": msg.get(\"role\", \"unknown\")}\n content = msg.get(\"content\", \"\")\n \n if isinstance(content, str):\n if len(content) > max_content_length:\n new_msg[\"content\"] = content[:max_content_length] + f\"... [truncated, total {len(content)} chars]\"\n else:\n new_msg[\"content\"] = content\n elif isinstance(content, list):\n # Handle multi-part content (e.g., with images)\n new_content = []\n for item in content:\n if isinstance(item, dict):\n if item.get(\"type\") == \"image\":\n new_content.append({\"type\": \"image\", \"note\": \"[image data omitted]\"})\n elif item.get(\"type\") == \"text\":\n text = item.get(\"text\", \"\")\n if len(text) > max_content_length:\n new_content.append({\n \"type\": \"text\",\n \"text\": text[:max_content_length] + f\"... [truncated, total {len(text)} chars]\"\n })\n else:\n new_content.append(item)\n else:\n new_content.append(item)\n else:\n new_content.append(item)\n new_msg[\"content\"] = new_content\n else:\n new_msg[\"content\"] = str(content)[:max_content_length]\n \n if \"tool_calls\" in msg:\n new_msg[\"tool_calls\"] = msg[\"tool_calls\"]\n \n result.append(new_msg)\n return result\n \n # Build record\n import datetime\n record = {\n \"iteration\": iteration,\n \"timestamp\": datetime.datetime.now().strftime(\"%Y-%m-%dT%H:%M:%S\"),\n \"llm_response_summary\": llm_response_summary,\n \"messages_input\": truncate_message_content(messages_input),\n \"messages_output\": truncate_message_content(messages_output),\n }\n \n # Append to conversations.jsonl (real-time)\n conv_file = instance._recorder.trajectory_dir / \"conversations.jsonl\"\n try:\n with open(conv_file, \"a\", encoding=\"utf-8\") as f:\n f.write(json.dumps(record, ensure_ascii=False))\n f.write(\"\\n\")\n except Exception as e:\n logger.debug(f\"Failed to write conversation log: {e}\")\n \n @classmethod\n async def record_tool_execution(\n cls,\n tool_name: str,\n backend: str,\n parameters: Dict[str, Any],\n result: Any,\n server_name: Optional[str] = None,\n is_success: bool = True,\n metadata: Optional[Dict[str, Any]] = None,\n ):\n \"\"\"\n Record tool execution (internal method, called by BaseTool automatically)\n \n Args:\n tool_name: Name of the tool\n backend: Backend type (gui, shell, mcp, etc.)\n parameters: Tool parameters\n result: Tool execution result (content or error message)\n server_name: Server name for MCP backend\n is_success: Whether the tool execution was successful (default: True for backward compatibility)\n metadata: Tool result metadata (e.g. intermediate_steps for GUI)\n \"\"\"\n if not cls._global_instance or not cls._global_instance._is_started:\n return\n \n instance = cls._global_instance\n \n # Check if should record this backend\n if backend not in instance.backends:\n return\n \n # Create mock tool_call and result objects for compatibility with existing _record_* methods\n class MockFunctionCall:\n def __init__(self, name, arguments):\n self.name = name\n self.arguments = arguments\n \n class MockToolCall:\n def __init__(self, name, arguments):\n self.function = MockFunctionCall(name, arguments)\n \n class MockResult:\n def __init__(self, content, is_success=True, metadata=None):\n self.content = content\n self.is_success = is_success\n self.is_error = not is_success\n self.error = content if not is_success else None\n self.metadata = metadata or {}\n \n tool_call = MockToolCall(tool_name, parameters)\n mock_result = MockResult(result, is_success=is_success, metadata=metadata)\n \n try:\n if backend == \"mcp\":\n server = server_name or \"unknown\"\n await instance._record_mcp(tool_call, mock_result, server)\n elif backend == \"gui\":\n await instance._record_gui(tool_call, mock_result)\n elif backend == \"shell\":\n await instance._record_shell(tool_call, mock_result)\n elif backend == \"system\":\n await instance._record_system(tool_call, mock_result)\n elif backend == \"web\":\n await instance._record_web(tool_call, mock_result)\n \n instance._step_counter += 1\n except Exception as e:\n logger.debug(f\"Failed to record tool execution: {e}\")\n \n @staticmethod\n def _parse_arguments(arg_data):\n \"\"\"Safely parse tool_call.function.arguments which may be JSON string.\n\n Handles:\n 1. Proper JSON strings with true/false/null\n 2. Python literal strings (produced by OpenAI) using ast.literal_eval\n 3. Already-dict objects (returned by SDK)\n \"\"\"\n if not isinstance(arg_data, str):\n return arg_data or {}\n\n # First, try JSON\n try:\n return json.loads(arg_data)\n except json.JSONDecodeError:\n pass\n\n # Fallback to Python literal\n try:\n return ast.literal_eval(arg_data)\n except Exception:\n logger.debug(\"Failed to parse arguments, returning raw string\")\n return {\"raw\": arg_data}\n \n async def start(self, task_id: Optional[str] = None):\n \"\"\"Start automatic recording\n Args:\n task_id: If provided, override the current task_id for this recording session. This allows\n external callers (e.g. Coordinator) to specify a meaningful task identifier without\n having to recreate the RecordingManager instance.\n \"\"\"\n # Allow dynamic update of task_id before recording actually starts\n if task_id:\n self.task_id = task_id\n if not self.enabled or self._is_started:\n return\n \n try:\n # check server availability (only when video or screenshot is enabled)\n if self.enable_video or self.enable_screenshot:\n await self._check_server_availability()\n \n self._recorder = TrajectoryRecorder(\n task_name=self.task_id,\n log_dir=self.log_dir,\n enable_screenshot=self.enable_screenshot,\n enable_video=self.enable_video,\n server_url=self.server_url,\n )\n \n # create action recorder for agent decision tracking\n self._action_recorder = ActionRecorder(\n trajectory_dir=Path(self._recorder.get_trajectory_dir())\n )\n \n \n # create video client (internal management)\n if self.enable_video:\n from anytool.platform import RecordingClient\n self._recording_client = RecordingClient(base_url=self.server_url)\n success = await self._recording_client.start_recording()\n if success:\n logger.info(\"Video recording started\")\n else:\n logger.warning(\"Video recording failed to start\")\n \n # create screenshot client (internal management)\n if self.enable_screenshot:\n from anytool.platform import ScreenshotClient\n self._screenshot_client = ScreenshotClient(base_url=self.server_url)\n logger.debug(\"Screenshot client ready\")\n \n # save initial metadata\n await self._recorder.add_metadata(\"task_id\", self.task_id)\n await self._recorder.add_metadata(\"backends\", list(self.backends))\n await self._recorder.add_metadata(\"start_time\", datetime.datetime.now().strftime(\"%Y-%m-%dT%H:%M:%S\"))\n\n # Capture and save initial screenshot if enabled\n if self.enable_screenshot and self._screenshot_client:\n try:\n init_shot = await self._screenshot_client.capture()\n if init_shot:\n await self._recorder.save_init_screenshot(init_shot)\n logger.debug(\"Initial screenshot saved\")\n except Exception as e:\n logger.debug(f\"Failed to capture initial screenshot: {e}\")\n \n self._is_started = True\n logger.info(f\"Recording started: {self._recorder.get_trajectory_dir()}\")\n \n except Exception as e:\n logger.error(f\"Recording failed to start: {e}\")\n raise\n \n async def _check_server_availability(self):\n \"\"\"Check if local server is available\"\"\"\n try:\n from anytool.platform import SystemInfoClient\n\n # Use context manager to ensure aiohttp session is closed, avoiding warning of unclosed session\n async with SystemInfoClient(base_url=self.server_url) as client:\n info = await client.get_system_info()\n\n if info:\n logger.info(f\"Server connected ({info.get('platform', 'unknown')})\")\n else:\n logger.warning(\"Server not responding, video/screenshot functionality unavailable\")\n \n except Exception:\n logger.warning(\"Cannot connect to server, video/screenshot functionality unavailable\")\n \n async def stop(self):\n \"\"\"Stop automatic recording\"\"\"\n if not self.enabled or not self._is_started:\n return\n \n try:\n # stop video recording and save\n if self._recording_client:\n try:\n video_path = None\n if self._recorder:\n video_path = str(Path(self._recorder.get_trajectory_dir()) / \"screen_recording.mp4\")\n \n video_bytes = await self._recording_client.end_recording(dest=video_path)\n if video_bytes and video_path:\n video_size_mb = len(video_bytes) / (1024 * 1024)\n logger.info(f\"Video recording saved: {video_path} ({video_size_mb:.2f} MB)\")\n except Exception as e:\n logger.warning(f\"Video recording failed to save: {e}\")\n\n # close RecordingClient session, avoid unclosed session warning\n try:\n if self._recording_client:\n await self._recording_client.close()\n except Exception as e:\n logger.debug(f\"Failed to close RecordingClient session: {e}\")\n \n # close screenshot client\n if self._screenshot_client:\n try:\n await self._screenshot_client.close()\n except Exception as e:\n logger.debug(f\"Screenshot client failed to close: {e}\")\n finally:\n self._screenshot_client = None\n \n # finalize trajectory recording\n if self._recorder:\n # save final metadata\n await self._recorder.add_metadata(\"end_time\", datetime.datetime.now().isoformat())\n await self._recorder.add_metadata(\"total_steps\", self._step_counter)\n \n # generate summary\n await self.generate_summary()\n \n # finalize recording\n await self._recorder.finalize()\n \n logger.info(f\"Recording completed: {self._recorder.get_trajectory_dir()}\")\n \n # Restore original methods for registered LLM clients\n for client in self._registered_llm_clients:\n client_id = id(client)\n if client_id in self._original_methods:\n try:\n # Restore original complete method\n original_method = self._original_methods[client_id]\n client.complete = original_method\n except Exception as e:\n logger.debug(f\"Failed to restore original method for LLM client: {e}\")\n \n # Clear registered clients and original methods\n self._registered_llm_clients.clear()\n self._original_methods.clear()\n \n self._is_started = False\n self._recorder = None\n self._action_recorder = None\n \n except Exception as e:\n logger.error(f\"Recording failed to stop: {e}\")\n \n def register_to_llm(self, llm_client):\n if not self.enabled:\n return\n \n # Check if already registered to avoid double-wrapping\n if id(llm_client) in self._original_methods:\n logger.warning(f\"LLM client {llm_client} is already registered, skipping\")\n return\n \n # Save original complete method for restoration\n original_complete = llm_client.complete\n self._original_methods[id(llm_client)] = original_complete\n \n # Wrap complete method\n async def wrapped_complete(self_client, *args, **kwargs):\n # Call original method\n response = await original_complete(*args, **kwargs)\n \n # Automatically record tool calls\n if response.get(\"tool_results\"):\n await self._auto_record_tool_results(response[\"tool_results\"])\n \n return response\n \n # Replace method with properly bound method\n llm_client.complete = types.MethodType(wrapped_complete, llm_client)\n self._registered_llm_clients.append(llm_client)\n \n async def _auto_record_tool_results(self, tool_results: List[Dict]):\n \"\"\"\n Internal method: automatically record tool execution results from LLM client\n \n This is called by register_to_llm() wrapper. Tool results should contain\n backend and server_name information.\n \"\"\"\n if not self._recorder or not self._is_started:\n return\n \n for tool_result in tool_results:\n # Get necessary information from tool_result\n tool_call = tool_result.get(\"tool_call\")\n result = tool_result.get(\"result\")\n backend = tool_result.get(\"backend\")\n server_name = tool_result.get(\"server_name\")\n \n if not tool_call or not result:\n logger.warning(\"Tool result missing 'tool_call' or 'result', skipping\")\n continue\n \n if not backend:\n logger.warning(\n f\"Tool result missing 'backend' field, skipping recording. \"\n f\"Tool: {tool_call.function.name}. \"\n f\"Ensure your LLM client provides backend information in tool_results.\"\n )\n continue\n \n # Extract metadata for embedding intermediate_steps (GUI)\n result_metadata = result.metadata if hasattr(result, 'metadata') else None\n \n await RecordingManager.record_tool_execution(\n tool_name=tool_call.function.name,\n backend=backend,\n parameters=self._parse_arguments(tool_call.function.arguments),\n result=result.content if hasattr(result, 'content') else str(result),\n server_name=server_name,\n is_success=result.is_success if hasattr(result, 'is_success') else True,\n metadata=result_metadata,\n )\n\n async def _record_mcp(self, tool_call, result, server: str):\n tool_name = tool_call.function.name\n parameters = self._parse_arguments(tool_call.function.arguments)\n \n command = f\"{server}.{tool_name}\"\n result_str = str(result.content) if result.is_success else str(result.error)\n result_brief = result_str[:200] + \"...\" if len(result_str) > 200 else result_str\n \n is_actual_success = result.is_success and not result_str.startswith(\"ERROR:\")\n \n step_info = await self._recorder.record_step(\n backend=\"mcp\",\n tool=tool_name,\n command=command,\n result={\n \"status\": \"success\" if is_actual_success else \"error\",\n \"output\": result_brief,\n },\n parameters=parameters,\n extra={\n \"server\": server,\n },\n auto_screenshot=self.enable_screenshot\n )\n \n # Add agent_name to step_info\n step_info[\"agent_name\"] = self.agent_name\n \n async def _record_gui(self, tool_call, result):\n tool_name = tool_call.function.name\n parameters = self._parse_arguments(tool_call.function.arguments)\n \n # Extract actual pyautogui command (from action_history)\n command = \"gui_agent\"\n if result.is_success and hasattr(result, 'metadata') and result.metadata:\n action_history = result.metadata.get(\"action_history\", [])\n if action_history:\n # Get last successful execution action\n for action in reversed(action_history):\n planned_action = action.get(\"planned_action\", {})\n execution_result = action.get(\"execution_result\", {})\n \n if planned_action.get(\"action_type\") == \"PYAUTOGUI_COMMAND\":\n cmd = planned_action.get(\"command\", \"\")\n if cmd and execution_result.get(\"status\") == \"success\":\n command = cmd\n break\n elif execution_result.get(\"status\") == \"success\":\n action_type = planned_action.get(\"action_type\", \"\")\n if action_type and action_type not in [\"WAIT\", \"DONE\", \"FAIL\"]:\n params = planned_action.get(\"parameters\", {})\n if params:\n param_str = \", \".join([f\"{k}={v}\" for k, v in list(params.items())[:2]])\n command = f\"{action_type}({param_str})\"\n else:\n command = action_type\n break\n \n result_str = str(result.content) if result.is_success else str(result.error)\n \n is_actual_success = result.is_success\n if result.is_success:\n first_200_chars = result_str[:200] if result_str else \"\"\n critical_failure_patterns = [\"Task failed\", \"CRITICAL ERROR:\", \"FATAL:\"]\n has_critical_failure = any(pattern in first_200_chars for pattern in critical_failure_patterns)\n is_actual_success = not has_critical_failure\n \n # Extract intermediate_steps from metadata for embedding in traj.jsonl\n extra = {}\n if hasattr(result, 'metadata') and result.metadata:\n intermediate_steps = result.metadata.get(\"intermediate_steps\")\n if intermediate_steps:\n extra[\"intermediate_steps\"] = intermediate_steps\n \n step_info = await self._recorder.record_step(\n backend=\"gui\",\n tool=\"gui_agent\",\n command=command,\n result={\n \"status\": \"success\" if is_actual_success else \"error\",\n \"output\": result_str,\n },\n parameters=parameters,\n auto_screenshot=self.enable_screenshot,\n extra=extra if extra else None,\n )\n \n step_info[\"agent_name\"] = self.agent_name\n \n async def _record_shell(self, tool_call, result):\n tool_name = tool_call.function.name\n parameters = self._parse_arguments(tool_call.function.arguments)\n \n task = parameters.get(\"task\", tool_name)\n exit_code = 0 if result.is_success else 1\n \n stdout = str(result.content) if result.is_success else \"\"\n stderr = str(result.error) if result.is_error else \"\"\n \n command = task \n if hasattr(result, 'metadata') and result.metadata:\n code_history = result.metadata.get(\"code_history\", [])\n if code_history:\n # Try to find the last successful execution\n found_success = False\n for code_info in reversed(code_history):\n if code_info.get(\"status\") == \"success\":\n lang = code_info.get(\"lang\", \"bash\")\n code = code_info.get(\"code\", \"\")\n # String format code block: ```lang\\ncode\\n```\n command = f\"```{lang}\\n{code}\\n```\"\n found_success = True\n break\n \n # If no successful execution found, use last code block\n if not found_success and code_history:\n last_code = code_history[-1]\n lang = last_code.get(\"lang\", \"bash\")\n code = last_code.get(\"code\", \"\")\n command = f\"```{lang}\\n{code}\\n```\"\n \n stdout_brief = stdout[:200] + \"...\" if len(stdout) > 200 else stdout\n stderr_brief = stderr[:200] + \"...\" if len(stderr) > 200 else stderr\n \n is_actual_success = result.is_success\n if result.is_success:\n first_200_chars = stdout[:200] if stdout else \"\"\n critical_failure_patterns = [\"Task failed after\", \"[TASK_FAILED:\"]\n has_critical_failure = any(pattern in first_200_chars for pattern in critical_failure_patterns)\n is_actual_success = not has_critical_failure\n \n step_info = await self._recorder.record_step(\n backend=\"shell\",\n tool=\"shell_agent\",\n command=command,\n result={\n \"status\": \"success\" if is_actual_success else \"error\",\n \"exit_code\": exit_code,\n \"stdout\": stdout_brief,\n \"stderr\": stderr_brief,\n },\n auto_screenshot=self.enable_screenshot\n )\n \n step_info[\"agent_name\"] = self.agent_name\n \n async def _record_system(self, tool_call, result):\n tool_name = tool_call.function.name\n parameters = self._parse_arguments(tool_call.function.arguments)\n \n command = tool_name\n if parameters:\n key_params = []\n for key in ['path', 'file', 'directory', 'name', 'provider', 'backend']:\n if key in parameters and parameters[key]:\n key_params.append(f\"{parameters[key]}\")\n if key_params:\n command = f\"{tool_name}({', '.join(key_params[:2])})\"\n \n result_str = str(result.content) if result.is_success else str(result.error)\n result_brief = result_str[:200] + \"...\" if len(result_str) > 200 else result_str\n \n is_actual_success = result.is_success\n if result.is_success and result_str:\n is_actual_success = not result_str.startswith(\"ERROR:\")\n \n step_info = await self._recorder.record_step(\n backend=\"system\",\n tool=tool_name,\n command=command,\n result={\n \"status\": \"success\" if is_actual_success else \"error\",\n \"output\": result_brief,\n },\n auto_screenshot=self.enable_screenshot\n )\n \n step_info[\"agent_name\"] = self.agent_name\n \n async def _record_web(self, tool_call, result):\n tool_name = tool_call.function.name\n parameters = self._parse_arguments(tool_call.function.arguments)\n \n query = parameters.get(\"query\", \"\")\n command = query if query else \"deep_research\"\n \n result_str = str(result.content) if result.is_success else str(result.error)\n \n is_actual_success = result.is_success\n if result.is_success and result_str:\n is_actual_success = not result_str.startswith(\"ERROR:\")\n \n step_info = await self._recorder.record_step(\n backend=\"web\",\n tool=\"deep_research_agent\",\n command=command,\n result={\n \"status\": \"success\" if is_actual_success else \"error\",\n \"output\": result_str, # Full output preserved for training/replay\n },\n auto_screenshot=self.enable_screenshot\n )\n \n # Add agent_name to step_info\n step_info[\"agent_name\"] = self.agent_name\n \n async def add_metadata(self, key: str, value: Any):\n if self._recorder:\n await self._recorder.add_metadata(key, value)\n \n async def save_plan(self, plan: Dict[str, Any], agent_name: str = \"GroundingAgent\"):\n \"\"\"\n Save agent plan to recording directory.\n This integrates planning information with execution trajectory.\n \n Args:\n plan: The plan data (usually containing task_updates or plan steps)\n agent_name: Name of the agent creating the plan\n \"\"\"\n if not self._recorder or not self._is_started:\n logger.warning(\"Cannot save plan: recording not started\")\n return\n \n try:\n plan_dir = Path(self._recorder.get_trajectory_dir()) / \"plans\"\n plan_dir.mkdir(exist_ok=True)\n \n timestamp = datetime.datetime.now().strftime(\"%Y%m%d_%H%M%S\")\n plan_data = {\n \"version\": timestamp,\n \"created_at\": datetime.datetime.now().isoformat(),\n \"created_by\": agent_name,\n \"plan\": plan\n }\n \n # Save versioned plan\n plan_file = plan_dir / f\"plan_{timestamp}.json\"\n with open(plan_file, 'w', encoding='utf-8') as f:\n json.dump(plan_data, f, indent=2, ensure_ascii=False)\n \n # Save current plan (latest)\n current_plan_file = plan_dir / \"current_plan.json\"\n with open(current_plan_file, 'w', encoding='utf-8') as f:\n json.dump(plan_data, f, indent=2, ensure_ascii=False)\n \n logger.debug(f\"Saved plan to recording: {plan_file.name}\")\n except Exception as e:\n logger.error(f\"Failed to save plan: {e}\")\n \n async def log_decision(\n self, \n agent_name: str, \n decision: str, \n context: Optional[Dict[str, Any]] = None\n ):\n \"\"\"\n Log agent decision with optional context.\n This provides insight into agent reasoning process.\n \n Args:\n agent_name: Name of the agent making the decision\n decision: Description of the decision\n context: Additional context information\n \"\"\"\n if not self._recorder or not self._is_started:\n logger.warning(\"Cannot log decision: recording not started\")\n return\n \n try:\n traj_dir = Path(self._recorder.get_trajectory_dir())\n log_file = traj_dir / \"decisions.log\"\n \n timestamp = datetime.datetime.now().isoformat()\n log_entry = f\"[{timestamp}] {agent_name}: {decision}\"\n if context:\n log_entry += f\"\\n Context: {json.dumps(context, ensure_ascii=False)}\"\n log_entry += \"\\n\"\n \n with open(log_file, 'a', encoding='utf-8') as f:\n f.write(log_entry)\n \n logger.debug(f\"Logged decision from {agent_name}\")\n except Exception as e:\n logger.error(f\"Failed to log decision: {e}\")\n \n async def record_agent_action(\n self,\n agent_name: str,\n action_type: str,\n input_data: Optional[Dict[str, Any]] = None,\n reasoning: Optional[Dict[str, Any]] = None,\n output_data: Optional[Dict[str, Any]] = None,\n metadata: Optional[Dict[str, Any]] = None,\n related_tool_steps: Optional[list] = None,\n correlation_id: Optional[str] = None,\n ) -> Optional[Dict[str, Any]]:\n \"\"\"\n Record an agent's action and decision-making process.\n \n Args:\n agent_name: Name of the agent performing the action\n action_type: Type of action (plan | execute | evaluate | monitor)\n input_data: Input data the agent received (simplified)\n reasoning: Agent's reasoning process (structured)\n output_data: Agent's output/decision (structured)\n metadata: Additional metadata (LLM model, tokens, duration, etc.)\n related_tool_steps: List of tool execution step numbers related to this action\n correlation_id: Optional correlation ID to link related events\n \n Returns:\n The recorded action info, or None if recording not started\n \"\"\"\n if not self._action_recorder or not self._is_started:\n logger.debug(\"Cannot record agent action: recording not started\")\n return None\n \n try:\n action_info = await self._action_recorder.record_action(\n agent_name=agent_name,\n action_type=action_type,\n input_data=input_data,\n reasoning=reasoning,\n output_data=output_data,\n metadata=metadata,\n related_tool_steps=related_tool_steps,\n correlation_id=correlation_id,\n )\n \n logger.debug(f\"Recorded agent action: {agent_name} - {action_type}\")\n return action_info\n \n except Exception as e:\n logger.error(f\"Failed to record agent action: {e}\")\n return None\n \n async def generate_summary(self) -> Dict[str, Any]:\n \"\"\"\n Generate a comprehensive summary of the recording session.\n \"\"\"\n if not self._recorder or not self._is_started:\n logger.warning(\"Cannot generate summary: recording not started\")\n return {}\n \n try:\n from .action_recorder import load_agent_actions, analyze_agent_actions\n from .utils import load_trajectory_from_jsonl, analyze_trajectory\n \n traj_dir = self._recorder.get_trajectory_dir()\n \n # Load all recorded data\n trajectory = load_trajectory_from_jsonl(f\"{traj_dir}/traj.jsonl\")\n agent_actions = load_agent_actions(traj_dir)\n \n # Analyze data\n traj_stats = analyze_trajectory(trajectory)\n action_stats = analyze_agent_actions(agent_actions)\n \n # Build summary\n summary = {\n \"task_id\": self.task_id,\n \"start_time\": self._recorder.metadata.get(\"start_time\", \"\"),\n \"end_time\": self._recorder.metadata.get(\"end_time\", \"\"),\n \"trajectory\": {\n \"total_steps\": traj_stats.get(\"total_steps\", 0),\n \"success_count\": traj_stats.get(\"success_count\", 0),\n \"success_rate\": traj_stats.get(\"success_rate\", 0),\n \"by_backend\": traj_stats.get(\"backends\", {}),\n \"by_tool\": traj_stats.get(\"tools\", {}),\n },\n \"agent_actions\": {\n \"total_actions\": action_stats.get(\"total_actions\", 0),\n \"by_agent\": action_stats.get(\"by_agent\", {}),\n \"by_type\": action_stats.get(\"by_type\", {}),\n }\n }\n \n # Save summary to file\n summary_file = Path(traj_dir) / \"summary.json\"\n with open(summary_file, 'w', encoding='utf-8') as f:\n json.dump(summary, f, indent=2, ensure_ascii=False)\n \n logger.info(f\"Generated summary: {summary_file}\")\n return summary\n \n except Exception as e:\n logger.error(f\"Failed to generate summary: {e}\")\n return {}\n \n async def __aenter__(self):\n await self.start()\n return self\n \n async def __aexit__(self, exc_type, exc_val, exc_tb):\n await self.stop()\n return False\n \n @property\n def recording_status(self) -> bool:\n return self._is_started\n \n @property\n def trajectory_dir(self) -> Optional[str]:\n if self._recorder:\n return str(self._recorder.get_trajectory_dir())\n return None\n \n @property\n def recording_client(self):\n return self._recording_client\n \n @property\n def screenshot_client(self):\n return self._screenshot_client\n \n @property\n def step_count(self) -> int:\n \"\"\"Get current step count\"\"\"\n return self._step_counter\n\n\n__all__ = [\n 'RecordingManager',\n]"
},
{
"path": "anytool/recording/recorder.py",
"content": "import datetime\nimport json\nfrom typing import Any, Dict, List, Optional\nfrom pathlib import Path\n\nfrom anytool.utils.logging import Logger\n\nlogger = Logger.get_logger(__name__)\n\n\nclass TrajectoryRecorder:\n def __init__(\n self,\n task_name: str = \"\",\n log_dir: str = \"./logs/trajectories\",\n enable_screenshot: bool = True,\n enable_video: bool = False,\n server_url: Optional[str] = None,\n ):\n \"\"\"\n Initialize trajectory recorder\n \n Args:\n task_name: task name (optional, will be saved in metadata)\n log_dir: log directory\n enable_screenshot: whether to save screenshots (through platform.ScreenshotClient)\n enable_video: whether to enable video recording (through platform.RecordingClient)\n server_url: local_server address (None = read from config/environment variables)\n \"\"\"\n timestamp = datetime.datetime.now().strftime(\"%Y%m%d_%H%M%S\")\n \n # Simplify naming rule: add prefix if task_name is provided, otherwise use timestamp only\n if task_name:\n folder_name = f\"{task_name}_{timestamp}\"\n else:\n folder_name = timestamp\n \n self.trajectory_dir = Path(log_dir) / folder_name\n self.trajectory_dir.mkdir(parents=True, exist_ok=True)\n \n # Create screenshots directory\n if enable_screenshot:\n self.screenshots_dir = self.trajectory_dir / \"screenshots\"\n self.screenshots_dir.mkdir(exist_ok=True)\n else:\n self.screenshots_dir = None\n \n # Config\n self.task_name = task_name\n self.enable_screenshot = enable_screenshot\n self.enable_video = enable_video\n self.server_url = server_url\n \n # Trajectory data\n self.steps: List[Dict] = []\n self.step_counter = 0\n \n # Metadata\n self.metadata = {\n \"task_name\": task_name,\n \"start_time\": datetime.datetime.now().strftime(\"%Y-%m-%dT%H:%M:%S\"),\n \"enable_screenshot\": enable_screenshot,\n \"enable_video\": enable_video,\n }\n \n # Video recorder (lazy initialization)\n self._video_recorder = None\n \n # Save initial metadata\n self._save_metadata()\n \n async def record_step(\n self,\n backend: str,\n tool: str,\n command: str,\n result: Optional[Dict[str, Any]] = None,\n parameters: Optional[Dict[str, Any]] = None,\n screenshot: Optional[bytes] = None,\n extra: Optional[Dict[str, Any]] = None,\n auto_screenshot: bool = False,\n ) -> Dict[str, Any]:\n \"\"\"\n Record one step operation\n \n Args:\n backend: backend type (gui/shell/mcp/web/system)\n tool: tool name (name of BaseTool)\n command: human-readable core command\n result: execution result\n parameters: tool parameters\n screenshot: screenshot bytes (if provided)\n extra: extra information (e.g. server field for MCP)\n auto_screenshot: whether to automatically capture screenshot (through platform.ScreenshotClient)\n \"\"\"\n self.step_counter += 1\n step_num = self.step_counter\n timestamp = datetime.datetime.now().strftime(\"%Y-%m-%dT%H:%M:%S\")\n\n step_info = {\n \"step\": step_num,\n \"timestamp\": timestamp,\n \"backend\": backend,\n }\n\n # MCP needs to record server (between backend and tool)\n if extra and \"server\" in extra:\n step_info[\"server\"] = extra.pop(\"server\")\n\n # General fields\n step_info[\"tool\"] = tool # BaseTool name\n step_info[\"command\"] = command # human-readable core command\n\n # parameters unified write to top level\n if parameters:\n step_info[\"parameters\"] = parameters\n elif extra and \"parameters\" in extra:\n step_info[\"parameters\"] = extra.pop(\"parameters\")\n\n # Execution result remains original\n step_info[\"result\"] = result or {}\n\n # Other extra information (e.g. coordinates/url) only added when needed\n if extra:\n step_info.update(extra)\n \n # Automatic screenshot (if enabled and no screenshot provided)\n if auto_screenshot and screenshot is None and self.enable_screenshot:\n screenshot = await self._capture_screenshot()\n \n # Save screenshot\n if screenshot and self.enable_screenshot and self.screenshots_dir:\n screenshot_filename = f\"step_{step_num:03d}.png\"\n screenshot_path = self.screenshots_dir / screenshot_filename\n with open(screenshot_path, \"wb\") as f:\n f.write(screenshot)\n step_info[\"screenshot\"] = f\"screenshots/{screenshot_filename}\"\n \n # Add to trajectory\n self.steps.append(step_info)\n \n # Save to traj.jsonl in real time\n await self._append_to_traj_file(step_info)\n \n return step_info\n \n async def _capture_screenshot(self) -> Optional[bytes]:\n \"\"\"Capture screenshot automatically through platform.ScreenshotClient\"\"\"\n try:\n from anytool.platform import ScreenshotClient\n \n # Lazy initialization screenshot client\n if not hasattr(self, '_screenshot_client'):\n try:\n self._screenshot_client = ScreenshotClient(base_url=self.server_url)\n except Exception:\n self._screenshot_client = None\n return None\n \n if self._screenshot_client is None:\n return None\n \n return await self._screenshot_client.capture()\n \n except Exception:\n return None\n \n async def save_init_screenshot(self, screenshot: bytes, filename: str = \"init.png\"):\n \"\"\"Save initial screenshot to screenshots dir and update metadata.\"\"\"\n if not (self.enable_screenshot and self.screenshots_dir and screenshot):\n return\n try:\n filepath = self.screenshots_dir / filename\n with open(filepath, \"wb\") as f:\n f.write(screenshot)\n # Update metadata\n self.metadata[\"init_screenshot\"] = f\"screenshots/{filename}\"\n self._save_metadata()\n except Exception as e:\n logger.debug(f\"Failed to save initial screenshot: {e}\")\n \n async def _append_to_traj_file(self, step_info: Dict[str, Any]):\n \"\"\"Add step to traj.jsonl file\"\"\"\n traj_file = self.trajectory_dir / \"traj.jsonl\"\n with open(traj_file, \"a\", encoding=\"utf-8\") as f:\n f.write(json.dumps(step_info, ensure_ascii=False))\n f.write(\"\\n\")\n \n def _save_metadata(self):\n \"\"\"Save metadata to metadata.json\"\"\"\n metadata_file = self.trajectory_dir / \"metadata.json\"\n with open(metadata_file, \"w\", encoding=\"utf-8\") as f:\n json.dump(self.metadata, f, indent=2, ensure_ascii=False)\n \n async def start_video_recording(self):\n \"\"\"Start video recording (through platform.RecordingClient)\"\"\"\n if not self.enable_video:\n return\n \n try:\n from anytool.recording.video import VideoRecorder\n \n video_path = self.trajectory_dir / \"recording.mp4\"\n self._video_recorder = VideoRecorder(str(video_path), base_url=self.server_url)\n \n success = await self._video_recorder.start()\n if not success:\n self._video_recorder = None\n \n except Exception as e:\n logger.warning(f\"Video recording failed to start: {e}\")\n self._video_recorder = None\n \n async def stop_video_recording(self):\n \"\"\"Stop video recording\"\"\"\n if self._video_recorder:\n try:\n await self._video_recorder.stop()\n except Exception:\n pass\n finally:\n self._video_recorder = None\n \n async def add_metadata(self, key: str, value: Any):\n \"\"\"Add metadata\"\"\"\n self.metadata[key] = value\n self._save_metadata()\n \n async def finalize(self):\n \"\"\"Finalize recording, save final information\"\"\"\n self.metadata[\"end_time\"] = datetime.datetime.now().strftime(\"%Y-%m-%dT%H:%M:%S\")\n self.metadata[\"total_steps\"] = self.step_counter\n \n # Backend statistics\n backend_counts = {}\n for step in self.steps:\n backend = step.get(\"backend\", \"unknown\")\n backend_counts[backend] = backend_counts.get(backend, 0) + 1\n self.metadata[\"backend_counts\"] = backend_counts\n \n self._save_metadata()\n\n # Close internal ScreenshotClient, avoid unclosed session warning\n await self._cleanup_screenshot_client()\n\n # Stop video recording\n await self.stop_video_recording()\n \n logger.info(f\"Recording completed: {self.trajectory_dir} (steps: {self.step_counter})\")\n \n async def _cleanup_screenshot_client(self):\n \"\"\"Cleanup screenshot client resources\"\"\"\n if hasattr(self, '_screenshot_client') and self._screenshot_client:\n try:\n await self._screenshot_client.close()\n except Exception as e:\n logger.debug(f\"Failed to close screenshot client: {e}\")\n finally:\n self._screenshot_client = None\n \n def __del__(self):\n \"\"\"Ensure resources are cleaned up even if finalize() is not called\"\"\"\n # Note: This is a safety net. Best practice is to call finalize() explicitly.\n if hasattr(self, '_video_recorder') and self._video_recorder:\n logger.warning(\n f\"TrajectoryRecorder for {self.trajectory_dir} was not finalized properly. \"\n \"Consider calling finalize() or using async context manager.\"\n )\n \n def get_trajectory_dir(self) -> str:\n \"\"\"Get trajectory directory path\"\"\"\n return str(self.trajectory_dir)\n \n async def __aenter__(self):\n \"\"\"Async context manager entry\"\"\"\n return self\n \n async def __aexit__(self, exc_type, exc_val, exc_tb):\n \"\"\"Async context manager exit - ensures finalize() is called\"\"\"\n await self.finalize()\n return False\n\nasync def record_gui_step(\n recorder: TrajectoryRecorder,\n command: str,\n task_description: str,\n result: Dict[str, Any] = None,\n screenshot: Optional[bytes] = None,\n max_steps: int = 10,\n tool: str = \"gui_agent\",\n) -> Dict[str, Any]:\n \"\"\"\n Record GUI step\n \n Args:\n recorder: recorder instance\n command: actual executed pyautogui command (e.g. \"pyautogui.moveTo(960, 540)\")\n task_description: task description\n result: execution result\n screenshot: screenshot\n max_steps: maximum number of steps\n tool: tool name\n \"\"\"\n parameters = {\n \"task_description\": task_description,\n \"max_steps\": max_steps,\n }\n \n return await recorder.record_step(\n backend=\"gui\",\n tool=tool,\n command=command,\n result=result,\n parameters=parameters,\n screenshot=screenshot,\n )\n\n\nasync def record_shell_step(\n recorder: TrajectoryRecorder,\n command: str,\n exit_code: int,\n stdout: Optional[str] = None,\n stderr: Optional[str] = None,\n screenshot: Optional[bytes] = None,\n tool: str = \"shell_agent\",\n) -> Dict[str, Any]:\n \"\"\"\n Record Shell step\n \n Args:\n recorder: recorder instance\n command: command executed\n exit_code: exit code\n stdout: standard output (simplified version, not saved completely)\n stderr: standard error (simplified version)\n screenshot: screenshot\n tool: tool name\n \"\"\"\n stdout_brief = stdout[:200] + \"...\" if stdout and len(stdout) > 200 else stdout\n stderr_brief = stderr[:200] + \"...\" if stderr and len(stderr) > 200 else stderr\n \n result = {\n \"status\": \"success\" if exit_code == 0 else \"error\",\n \"exit_code\": exit_code,\n \"stdout\": stdout_brief,\n \"stderr\": stderr_brief,\n }\n \n return await recorder.record_step(\n backend=\"shell\",\n tool=tool,\n command=command,\n result=result,\n screenshot=screenshot,\n )\n\nasync def record_mcp_step(\n recorder: TrajectoryRecorder,\n server: str,\n tool_name: str,\n parameters: Dict[str, Any],\n result: Any,\n screenshot: Optional[bytes] = None,\n) -> Dict[str, Any]:\n \"\"\"\n Record MCP step\n \n Args:\n recorder: recorder instance\n server: MCP server name\n tool_name: tool name\n parameters: tool parameters\n result: execution result\n screenshot: screenshot\n \"\"\"\n command = f\"{server}.{tool_name}\"\n \n result_str = str(result)\n result_brief = result_str[:200] + \"...\" if len(result_str) > 200 else result_str\n \n return await recorder.record_step(\n backend=\"mcp\",\n tool=tool_name,\n command=command,\n result={\"status\": \"success\", \"output\": result_brief},\n parameters=parameters,\n screenshot=screenshot,\n extra={\n \"server\": server,\n }\n )\n\n\nasync def record_web_step(\n recorder: TrajectoryRecorder,\n query: str,\n result: Dict[str, Any],\n screenshot: Optional[bytes] = None,\n tool: str = \"deep_research_agent\",\n) -> Dict[str, Any]:\n \"\"\"\n Record Web step (deep research)\n \n Args:\n recorder: recorder instance\n query: search query\n result: execution result\n screenshot: screenshot\n tool: tool name\n \"\"\"\n command = query # directly use query as command\n \n return await recorder.record_step(\n backend=\"web\",\n tool=tool,\n command=command,\n result=result,\n screenshot=screenshot,\n )"
},
{
"path": "anytool/recording/utils.py",
"content": "import json\nimport os\nfrom pathlib import Path\nfrom typing import Any, Dict, List, Optional, Tuple\nfrom anytool.utils.logging import Logger\n\nlogger = Logger.get_logger(__name__)\n\ndef load_trajectory_from_jsonl(jsonl_path: str) -> List[Dict[str, Any]]:\n trajectory = []\n \n # Check if file exists first\n if not os.path.exists(jsonl_path):\n logger.debug(f\"No trajectory file found at {jsonl_path} (this is normal for knowledge-only tasks)\")\n return []\n \n try:\n with open(jsonl_path, \"r\", encoding=\"utf-8\") as f:\n for line in f:\n line = line.strip()\n if line:\n step = json.loads(line)\n trajectory.append(step)\n \n logger.info(f\"Loaded {len(trajectory)} steps from {jsonl_path}\")\n return trajectory\n \n except Exception as e:\n logger.error(f\"Failed to load trajectory from {jsonl_path}: {e}\")\n return []\n\n\ndef load_metadata(trajectory_dir: str) -> Optional[Dict[str, Any]]:\n metadata_path = os.path.join(trajectory_dir, \"metadata.json\")\n \n try:\n with open(metadata_path, \"r\", encoding=\"utf-8\") as f:\n metadata = json.load(f)\n return metadata\n except Exception as e:\n logger.warning(f\"Failed to load metadata from {metadata_path}: {e}\")\n return None\n\n\ndef format_trajectory_for_export(\n trajectory: List[Dict[str, Any]],\n format_type: str = \"compact\"\n) -> str:\n if format_type == \"compact\":\n return _format_compact(trajectory)\n elif format_type == \"detailed\":\n return _format_detailed(trajectory)\n elif format_type == \"markdown\":\n return _format_markdown(trajectory)\n else:\n raise ValueError(f\"Unknown format type: {format_type}\")\n\n\ndef _format_compact(trajectory: List[Dict[str, Any]]) -> str:\n \"\"\"Compact format: one line per step.\"\"\"\n lines = []\n for step in trajectory:\n step_num = step.get(\"step\", \"?\")\n backend = step.get(\"backend\", \"?\")\n server = step.get(\"server\")\n tool = step.get(\"tool\", \"?\")\n result_status = \"success\" if step.get(\"result\", {}).get(\"status\") == \"success\" else \"error\"\n \n # Include server name for MCP backend\n backend_str = f\"{backend}@{server}\" if server else backend\n lines.append(f\"Step {step_num}: [{backend_str}] {tool} -> {result_status}\")\n \n return \"\\n\".join(lines)\n\n\ndef _format_detailed(trajectory: List[Dict[str, Any]]) -> str:\n \"\"\"Detailed format: multiple lines per step with parameters.\"\"\"\n lines = []\n for step in trajectory:\n step_num = step.get(\"step\", \"?\")\n timestamp = step.get(\"timestamp\", \"?\")\n backend = step.get(\"backend\", \"?\")\n server = step.get(\"server\")\n tool = step.get(\"tool\", \"?\")\n command = step.get(\"command\", \"?\")\n parameters = step.get(\"parameters\", {})\n result = step.get(\"result\", {})\n \n from anytool.utils.display import Box, BoxStyle\n \n box = Box(width=66, style=BoxStyle.ROUNDED, color='bl')\n lines.append(\"\")\n lines.append(box.top_line(0))\n lines.append(box.text_line(f\"Step {step_num} ({timestamp})\", align='center', indent=0, text_color='c'))\n lines.append(box.separator_line(0))\n lines.append(box.text_line(f\"Backend: {backend}\", indent=0))\n if server:\n lines.append(box.text_line(f\"Server: {server}\", indent=0))\n lines.append(box.text_line(f\"Tool: {tool}\", indent=0))\n lines.append(box.text_line(f\"Command: {command}\", indent=0))\n lines.append(box.separator_line(0))\n # Parameters and result can be multi-line\n param_str = json.dumps(parameters, indent=2)\n for param_line in param_str.split('\\n'):\n lines.append(box.text_line(param_line, indent=0))\n lines.append(box.separator_line(0))\n result_str = json.dumps(result, indent=2)\n for result_line in result_str.split('\\n'):\n lines.append(box.text_line(result_line, indent=0))\n lines.append(box.bottom_line(0))\n \n return \"\\n\".join(lines)\n\n\ndef _format_markdown(trajectory: List[Dict[str, Any]]) -> str:\n \"\"\"Markdown format: table format.\"\"\"\n lines = [\n \"# Trajectory\",\n \"\",\n \"| Step | Backend | Server | Tool | Status | Screenshot |\",\n \"|------|---------|--------|------|--------|------------|\"\n ]\n \n for step in trajectory:\n step_num = step.get(\"step\", \"?\")\n backend = step.get(\"backend\", \"?\")\n server = step.get(\"server\", \"-\")\n tool = step.get(\"tool\", \"?\")\n result_status = \"โ\" if step.get(\"result\", {}).get(\"status\") == \"success\" else \"โ\"\n screenshot = \"๐ท\" if step.get(\"screenshot\") else \"\"\n \n lines.append(f\"| {step_num} | {backend} | {server} | {tool} | {result_status} | {screenshot} |\")\n \n return \"\\n\".join(lines)\n\n\ndef analyze_trajectory(trajectory: List[Dict[str, Any]]) -> Dict[str, Any]:\n \"\"\"\n Analyze trajectory and return statistics.\n \"\"\"\n if not trajectory:\n return {\n \"total_steps\": 0,\n \"success_rate\": 0.0,\n \"backends\": {},\n \"action_types\": {}\n }\n \n total_steps = len(trajectory)\n success_count = 0\n backends = {}\n action_types = {}\n \n for step in trajectory:\n # Count successes\n if step.get(\"result\", {}).get(\"status\") == \"success\":\n success_count += 1\n \n # Count backends\n backend = step.get(\"backend\", \"unknown\")\n backends[backend] = backends.get(backend, 0) + 1\n \n # Count tool types\n tool = step.get(\"tool\", \"unknown\")\n action_types[tool] = action_types.get(tool, 0) + 1\n \n return {\n \"total_steps\": total_steps,\n \"success_count\": success_count,\n \"success_rate\": success_count / total_steps if total_steps > 0 else 0.0,\n \"backends\": backends,\n \"tools\": action_types \n }\n\n\ndef load_recording_session(recording_dir: str) -> Dict[str, Any]:\n \"\"\"\n Load complete recording session including trajectory, metadata, plans, and snapshots.\n \n Args:\n recording_dir: Path to recording directory\n \n Returns:\n Dictionary containing all session data:\n {\n \"trajectory\": List[Dict],\n \"metadata\": Dict,\n \"plans\": List[Dict],\n \"decisions\": List[str],\n \"statistics\": Dict\n }\n \"\"\"\n recording_path = Path(recording_dir)\n \n if not recording_path.exists():\n logger.error(f\"Recording directory not found: {recording_dir}\")\n return {}\n \n session = {\n \"trajectory\": [],\n \"metadata\": None,\n \"plans\": [],\n \"decisions\": [],\n \"statistics\": {}\n }\n \n # Load trajectory\n traj_file = recording_path / \"traj.jsonl\"\n if traj_file.exists():\n session[\"trajectory\"] = load_trajectory_from_jsonl(str(traj_file))\n session[\"statistics\"] = analyze_trajectory(session[\"trajectory\"])\n \n # Load metadata\n metadata_file = recording_path / \"metadata.json\"\n if metadata_file.exists():\n session[\"metadata\"] = load_metadata(str(recording_path))\n \n # Load plans\n plans_dir = recording_path / \"plans\"\n if plans_dir.exists():\n for plan_file in sorted(plans_dir.glob(\"plan_*.json\")):\n try:\n with open(plan_file, 'r', encoding='utf-8') as f:\n session[\"plans\"].append(json.load(f))\n except Exception as e:\n logger.warning(f\"Failed to load plan {plan_file}: {e}\")\n \n # Load decisions log\n decisions_file = recording_path / \"decisions.log\"\n if decisions_file.exists():\n try:\n with open(decisions_file, 'r', encoding='utf-8') as f:\n session[\"decisions\"] = f.readlines()\n except Exception as e:\n logger.warning(f\"Failed to load decisions: {e}\")\n \n return session\n\n\ndef filter_trajectory(\n trajectory: List[Dict[str, Any]],\n backend: Optional[str] = None,\n tool: Optional[str] = None,\n status: Optional[str] = None,\n time_range: Optional[Tuple[str, str]] = None\n) -> List[Dict[str, Any]]:\n filtered = trajectory\n \n if backend:\n filtered = [s for s in filtered if s.get(\"backend\") == backend]\n \n if tool:\n filtered = [s for s in filtered if s.get(\"tool\") == tool]\n \n if status:\n filtered = [s for s in filtered if s.get(\"result\", {}).get(\"status\") == status]\n \n if time_range:\n start_time, end_time = time_range\n filtered = [\n s for s in filtered \n if start_time <= s.get(\"timestamp\", \"\") <= end_time\n ]\n \n return filtered\n\n\ndef extract_errors(trajectory: List[Dict[str, Any]]) -> List[Dict[str, Any]]:\n return [\n step for step in trajectory\n if step.get(\"result\", {}).get(\"status\") == \"error\"\n ]\n\n\ndef generate_summary_report(recording_dir: str, output_file: Optional[str] = None) -> str:\n session = load_recording_session(recording_dir)\n \n if not session:\n return \"Error: Could not load recording session\"\n \n lines = []\n lines.append(\"# Recording Session Summary\\n\")\n \n # Metadata section\n if session[\"metadata\"]:\n lines.append(\"## Metadata\")\n metadata = session[\"metadata\"]\n lines.append(f\"- **Task ID**: {metadata.get('task_id', 'N/A')}\")\n lines.append(f\"- **Start Time**: {metadata.get('start_time', 'N/A')}\")\n lines.append(f\"- **End Time**: {metadata.get('end_time', 'N/A')}\")\n lines.append(f\"- **Total Steps**: {metadata.get('total_steps', 0)}\")\n lines.append(f\"- **Backends**: {', '.join(metadata.get('backends', []))}\")\n lines.append(\"\")\n \n # Statistics section\n if session[\"statistics\"]:\n lines.append(\"## Statistics\")\n stats = session[\"statistics\"]\n lines.append(f\"- **Total Steps**: {stats.get('total_steps', 0)}\")\n lines.append(f\"- **Success Count**: {stats.get('success_count', 0)}\")\n lines.append(f\"- **Success Rate**: {stats.get('success_rate', 0):.2%}\")\n lines.append(\"\")\n \n lines.append(\"### Backend Distribution\")\n for backend, count in stats.get('backends', {}).items():\n lines.append(f\"- {backend}: {count}\")\n lines.append(\"\")\n \n lines.append(\"### Tool Distribution\")\n for tool, count in sorted(stats.get('tools', {}).items(), key=lambda x: x[1], reverse=True):\n lines.append(f\"- {tool}: {count}\")\n lines.append(\"\")\n \n # Plans section\n if session[\"plans\"]:\n lines.append(f\"## Plans ({len(session['plans'])} total)\")\n for i, plan in enumerate(session[\"plans\"], 1):\n lines.append(f\"### Plan {i}\")\n lines.append(f\"- Created: {plan.get('created_at', 'N/A')}\")\n lines.append(f\"- Created by: {plan.get('created_by', 'N/A')}\")\n plan_data = plan.get('plan', {})\n if 'task_updates' in plan_data:\n lines.append(f\"- Tasks: {len(plan_data['task_updates'])}\")\n lines.append(\"\")\n \n # Errors section\n if session[\"trajectory\"]:\n errors = extract_errors(session[\"trajectory\"])\n if errors:\n lines.append(f\"## Errors ({len(errors)} total)\")\n for error in errors[:5]: # Show first 5 errors\n lines.append(f\"- Step {error.get('step')}: {error.get('backend')} - {error.get('tool')}\")\n error_msg = error.get('result', {}).get('output', 'No error message')\n lines.append(f\" ```\\n {error_msg[:200]}\\n ```\")\n if len(errors) > 5:\n lines.append(f\" ... and {len(errors) - 5} more errors\")\n lines.append(\"\")\n \n # Decisions section\n if session[\"decisions\"]:\n lines.append(f\"## Decisions ({len(session['decisions'])} total)\")\n for decision in session[\"decisions\"][:10]: # Show first 10 decisions\n lines.append(f\" {decision.strip()}\")\n if len(session[\"decisions\"]) > 10:\n lines.append(f\" ... and {len(session['decisions']) - 10} more decisions\")\n lines.append(\"\")\n \n report = \"\\n\".join(lines)\n \n # Save to file if requested\n if output_file:\n try:\n with open(output_file, 'w', encoding='utf-8') as f:\n f.write(report)\n logger.info(f\"Report saved to {output_file}\")\n except Exception as e:\n logger.error(f\"Failed to save report: {e}\")\n \n return report\n\n\ndef compare_recordings(recording_dir1: str, recording_dir2: str) -> Dict[str, Any]:\n session1 = load_recording_session(recording_dir1)\n session2 = load_recording_session(recording_dir2)\n \n stats1 = session1.get(\"statistics\", {})\n stats2 = session2.get(\"statistics\", {})\n \n return {\n \"session1\": {\n \"path\": recording_dir1,\n \"total_steps\": stats1.get(\"total_steps\", 0),\n \"success_rate\": stats1.get(\"success_rate\", 0),\n \"backends\": stats1.get(\"backends\", {})\n },\n \"session2\": {\n \"path\": recording_dir2,\n \"total_steps\": stats2.get(\"total_steps\", 0),\n \"success_rate\": stats2.get(\"success_rate\", 0),\n \"backends\": stats2.get(\"backends\", {})\n },\n \"differences\": {\n \"step_diff\": stats2.get(\"total_steps\", 0) - stats1.get(\"total_steps\", 0),\n \"success_rate_diff\": stats2.get(\"success_rate\", 0) - stats1.get(\"success_rate\", 0)\n }\n }"
},
{
"path": "anytool/recording/video.py",
"content": "\"\"\"\nVideo Recorder\n\nCommunicates with local_server through platform.RecordingClient\nSupports local and remote recording (through configuration LOCAL_SERVER_URL)\n\"\"\"\n\nfrom pathlib import Path\nfrom typing import Optional\n\nfrom anytool.utils.logging import Logger\nfrom anytool.platform import RecordingClient\n\nlogger = Logger.get_logger(__name__)\n\n\nclass VideoRecorder:\n def __init__(\n self,\n output_path: str,\n base_url: Optional[str] = None,\n ):\n \"\"\"\n Initialize video recorder\n \n Args:\n output_path: output video path\n base_url: local_server address (None = read from config/environment variables)\n \"\"\"\n self.output_path = Path(output_path)\n self.base_url = base_url\n self.is_recording = False\n self._client: Optional[RecordingClient] = None\n \n async def start(self):\n \"\"\"Start recording screen\"\"\"\n if self.is_recording:\n return False\n \n try:\n if self._client is None:\n self._client = RecordingClient(base_url=self.base_url)\n \n success = await self._client.start_recording()\n \n if success:\n self.is_recording = True\n logger.info(f\"Video recording started\")\n return True\n else:\n logger.warning(\"Video recording failed to start\")\n return False\n \n except Exception as e:\n logger.warning(f\"Video recording failed to start: {e}\")\n return False\n \n async def stop(self):\n \"\"\"Stop recording screen and save to local\"\"\"\n if not self.is_recording:\n return False\n \n try:\n if self._client:\n video_bytes = await self._client.end_recording(dest=str(self.output_path))\n \n if video_bytes:\n video_size_mb = len(video_bytes) / (1024 * 1024)\n self.is_recording = False\n logger.info(f\"Video recording stopped ({video_size_mb:.2f} MB)\")\n return True\n else:\n logger.warning(\"Video recording failed to stop\")\n return False\n \n except Exception as e:\n logger.warning(f\"Video recording failed to stop: {e}\")\n return False\n finally:\n if self._client:\n try:\n await self._client.close()\n except Exception:\n pass\n self._client = None\n\n\n__all__ = ['VideoRecorder']"
},
{
"path": "anytool/recording/viewer.py",
"content": "\"\"\"\nRecording Viewer\nConvenient tools for viewing and analyzing recording sessions.\n\"\"\"\n\nimport json\nfrom pathlib import Path\nfrom typing import Optional, Dict, Any, List\n\nfrom anytool.utils.logging import Logger\nfrom .utils import load_recording_session, generate_summary_report\nfrom .action_recorder import load_agent_actions, analyze_agent_actions, format_agent_actions\n\nlogger = Logger.get_logger(__name__)\n\n\nclass RecordingViewer:\n \"\"\"\n Viewer for analyzing recording sessions.\n \n Provides convenient methods to:\n - Load and display recordings\n - Analyze agent behaviors\n - Generate reports\n \"\"\"\n \n def __init__(self, recording_dir: str):\n \"\"\"\n Initialize viewer with a recording directory.\n \n Args:\n recording_dir: Path to recording directory\n \"\"\"\n self.recording_dir = Path(recording_dir)\n \n if not self.recording_dir.exists():\n raise ValueError(f\"Recording directory not found: {recording_dir}\")\n \n # Load session data\n self.session = load_recording_session(str(self.recording_dir))\n \n logger.info(f\"Loaded recording from {recording_dir}\")\n \n def show_summary(self) -> str:\n \"\"\"\n Display a summary of the recording.\n \n Returns:\n Formatted summary string\n \"\"\"\n if not self.session.get(\"metadata\"):\n return \"No metadata available\"\n \n metadata = self.session[\"metadata\"]\n stats = self.session.get(\"statistics\", {})\n \n lines = []\n lines.append(\"=\" * 70)\n lines.append(\"RECORDING SUMMARY\")\n lines.append(\"=\" * 70)\n lines.append(f\"Task ID: {metadata.get('task_id', 'N/A')}\")\n lines.append(f\"Start: {metadata.get('start_time', 'N/A')}\")\n lines.append(f\"End: {metadata.get('end_time', 'N/A')}\")\n lines.append(f\"Total Steps: {metadata.get('total_steps', 0)}\")\n lines.append(\"\")\n \n lines.append(\"Statistics:\")\n lines.append(f\" - Success Rate: {stats.get('success_rate', 0):.2%}\")\n lines.append(f\" - Success Count: {stats.get('success_count', 0)}/{stats.get('total_steps', 0)}\")\n lines.append(\"\")\n \n if stats.get(\"backends\"):\n lines.append(\"Backend Usage:\")\n for backend, count in sorted(stats[\"backends\"].items(), key=lambda x: x[1], reverse=True):\n lines.append(f\" - {backend}: {count}\")\n \n lines.append(\"=\" * 70)\n \n return \"\\n\".join(lines)\n \n def show_agent_actions(self, format_type: str = \"compact\", agent_name: Optional[str] = None) -> str:\n actions = load_agent_actions(str(self.recording_dir))\n \n if agent_name:\n actions = [a for a in actions if a.get(\"agent_name\") == agent_name]\n \n if not actions:\n return f\"No agent actions found{' for ' + agent_name if agent_name else ''}\"\n \n # Add header\n header = f\"\\nAGENT ACTIONS ({len(actions)} total)\"\n if agent_name:\n header += f\" - {agent_name}\"\n header += \"\\n\" + \"=\" * 70\n \n # Format actions\n formatted = format_agent_actions(actions, format_type)\n \n return header + \"\\n\" + formatted\n \n def analyze_agents(self) -> str:\n actions = load_agent_actions(str(self.recording_dir))\n stats = analyze_agent_actions(actions)\n \n lines = []\n lines.append(\"\\nAGENT ANALYSIS\")\n lines.append(\"=\" * 70)\n lines.append(f\"Total Actions: {stats.get('total_actions', 0)}\")\n lines.append(\"\")\n \n lines.append(\"By Agent:\")\n for agent, count in sorted(stats.get('by_agent', {}).items(), key=lambda x: x[1], reverse=True):\n percentage = (count / stats['total_actions'] * 100) if stats['total_actions'] > 0 else 0\n lines.append(f\" - {agent}: {count} ({percentage:.1f}%)\")\n lines.append(\"\")\n \n lines.append(\"By Action Type:\")\n for action_type, count in sorted(stats.get('by_type', {}).items(), key=lambda x: x[1], reverse=True):\n percentage = (count / stats['total_actions'] * 100) if stats['total_actions'] > 0 else 0\n lines.append(f\" - {action_type}: {count} ({percentage:.1f}%)\")\n \n return \"\\n\".join(lines)\n \n def generate_full_report(self, output_file: Optional[str] = None) -> str:\n return generate_summary_report(str(self.recording_dir), output_file)\n \n def export_to_json(self, output_file: str):\n with open(output_file, 'w', encoding='utf-8') as f:\n json.dump(self.session, f, indent=2, ensure_ascii=False)\n \n logger.info(f\"Exported session to {output_file}\")\n \n def show_timeline(self, max_events: int = 50) -> str:\n # Load all events\n actions = load_agent_actions(str(self.recording_dir))\n trajectory = self.session.get(\"trajectory\", [])\n \n # Combine all events with unified format\n timeline = []\n \n # Add agent actions\n for action in actions:\n timeline.append({\n \"timestamp\": action.get(\"timestamp\", \"\"),\n \"type\": \"agent_action\",\n \"agent_name\": action.get(\"agent_name\", \"\"),\n \"agent_type\": action.get(\"agent_type\", \"unknown\"),\n \"action_type\": action.get(\"action_type\", \"\"),\n \"step\": action.get(\"step\"),\n \"correlation_id\": action.get(\"correlation_id\", \"\"),\n \"description\": f\"[{action.get('agent_type', '?').upper()}] {action.get('action_type', '?')}\",\n \"related_tool_steps\": action.get(\"related_tool_steps\", []),\n })\n \n # Add tool executions\n for traj_step in trajectory:\n timeline.append({\n \"timestamp\": traj_step.get(\"timestamp\", \"\"),\n \"type\": \"tool_execution\",\n \"backend\": traj_step.get(\"backend\", \"\"),\n \"tool\": traj_step.get(\"tool\", \"\"),\n \"step\": traj_step.get(\"step\"),\n \"agent_name\": traj_step.get(\"agent_name\", \"\"),\n \"description\": f\"[TOOL:{traj_step.get('backend', '?').upper()}] {traj_step.get('tool', '?')}\",\n \"status\": traj_step.get(\"result\", {}).get(\"status\", \"\"),\n })\n \n # Sort by timestamp\n timeline.sort(key=lambda x: x.get(\"timestamp\", \"\"))\n \n # Format output\n lines = []\n lines.append(\"\\nUNIFIED TIMELINE\")\n lines.append(\"=\" * 100)\n lines.append(f\"Total events: {len(timeline)} (showing first {max_events})\")\n lines.append(\"\")\n \n for i, item in enumerate(timeline[:max_events]):\n timestamp = item.get(\"timestamp\", \"N/A\")\n time_str = timestamp.split(\"T\")[1][:8] if \"T\" in timestamp else timestamp[-8:]\n \n # Format line with type indicator\n type_marker = {\n \"agent_action\": \"๐ค\",\n \"tool_execution\": \"๐ง\"\n }.get(item.get(\"type\"), \"โข\")\n \n desc = item.get(\"description\", \"\")\n agent = item.get(\"agent_name\", \"\")\n agent_type = item.get(\"agent_type\", \"\")\n \n line = f\"{time_str} {type_marker} {desc}\"\n \n # Add agent info if available\n if agent and agent_type:\n line += f\" (by {agent}/{agent_type})\"\n elif agent:\n line += f\" (by {agent})\"\n \n lines.append(line)\n \n # Show correlations\n correlations = []\n if item.get(\"related_tool_steps\"):\n correlations.append(f\"โ tool steps: {item['related_tool_steps']}\")\n if item.get(\"related_action_step\"):\n correlations.append(f\"โ action step: {item['related_action_step']}\")\n \n if correlations:\n for corr in correlations:\n lines.append(f\" {corr}\")\n \n if len(timeline) > max_events:\n lines.append(f\"\\n... and {len(timeline) - max_events} more events\")\n \n return \"\\n\".join(lines)\n \n def show_agent_flow(self, agent_name: Optional[str] = None) -> str:\n \"\"\"\n Show the flow of a specific agent's actions and related events.\n \"\"\"\n actions = load_agent_actions(str(self.recording_dir))\n \n if agent_name:\n actions = [a for a in actions if a.get(\"agent_name\") == agent_name]\n \n lines = []\n lines.append(f\"\\nAGENT FLOW{' - ' + agent_name if agent_name else ''}\")\n lines.append(\"=\" * 100)\n \n # Sort by timestamp\n actions.sort(key=lambda x: x.get(\"timestamp\", \"\"))\n \n for action in actions:\n timestamp = action.get(\"timestamp\", \"N/A\").split(\"T\")[1][:8] if \"T\" in action.get(\"timestamp\", \"\") else \"N/A\"\n \n agent_type = action.get(\"agent_type\", \"?\").upper()\n action_type = action.get(\"action_type\", \"?\")\n step = action.get(\"step\", \"?\")\n lines.append(f\"{timestamp} [{agent_type}] Action #{step}: {action_type}\")\n \n # Show reasoning if available\n if action.get(\"reasoning\"):\n thought = action[\"reasoning\"].get(\"thought\", \"\")\n if thought:\n lines.append(f\" ๐ญ {thought[:80]}...\")\n \n # Show output\n if action.get(\"output\"):\n output = action[\"output\"]\n if isinstance(output, dict):\n for key in [\"message\", \"status\", \"evaluation\"]:\n if key in output:\n lines.append(f\" ๐ค {key}: {str(output[key])[:60]}\")\n \n lines.append(\"\")\n \n return \"\\n\".join(lines)\n\n\ndef view_recording(recording_dir: str):\n \"\"\"\n Quick interactive viewer for a recording.\n \"\"\"\n try:\n viewer = RecordingViewer(recording_dir)\n \n print(viewer.show_summary())\n print(\"\\n\")\n \n print(viewer.analyze_agents())\n print(\"\\n\")\n \n print(\"Agent Actions (compact):\")\n print(viewer.show_agent_actions(format_type=\"compact\"))\n \n except Exception as e:\n logger.error(f\"Failed to view recording: {e}\")\n print(f\"Error: {e}\")\n\n\ndef compare_recordings(recording_dir1: str, recording_dir2: str) -> str:\n \"\"\"\n Compare two recordings side by side.\n \"\"\"\n try:\n viewer1 = RecordingViewer(recording_dir1)\n viewer2 = RecordingViewer(recording_dir2)\n \n lines = []\n lines.append(\"=\" * 70)\n lines.append(\"RECORDING COMPARISON\")\n lines.append(\"=\" * 70)\n lines.append(\"\")\n \n # Compare metadata\n meta1 = viewer1.session.get(\"metadata\", {})\n meta2 = viewer2.session.get(\"metadata\", {})\n \n lines.append(\"Recording 1:\")\n lines.append(f\" Task: {meta1.get('task_id', 'N/A')}\")\n lines.append(f\" Steps: {meta1.get('total_steps', 0)}\")\n lines.append(\"\")\n \n lines.append(\"Recording 2:\")\n lines.append(f\" Task: {meta2.get('task_id', 'N/A')}\")\n lines.append(f\" Steps: {meta2.get('total_steps', 0)}\")\n lines.append(\"\")\n \n # Compare statistics\n stats1 = viewer1.session.get(\"statistics\", {})\n stats2 = viewer2.session.get(\"statistics\", {})\n \n lines.append(\"Differences:\")\n lines.append(f\" Steps: {meta2.get('total_steps', 0) - meta1.get('total_steps', 0):+d}\")\n lines.append(f\" Success Rate: {stats2.get('success_rate', 0) - stats1.get('success_rate', 0):+.2%}\")\n \n return \"\\n\".join(lines)\n \n except Exception as e:\n logger.error(f\"Failed to compare recordings: {e}\")\n return f\"Error: {e}\"\n\n\n# CLI interface\nif __name__ == \"__main__\":\n import sys\n \n if len(sys.argv) < 2:\n print(\"Usage: python -m anytool.recording.viewer \")\n sys.exit(1)\n \n recording_dir = sys.argv[1]\n view_recording(recording_dir)"
},
{
"path": "anytool/tool_layer.py",
"content": "from __future__ import annotations\n\nimport asyncio\nimport traceback\nimport uuid\nfrom dataclasses import dataclass, field\nfrom typing import Any, Dict, List, Optional\n\nfrom anytool.agents import GroundingAgent\nfrom anytool.llm import LLMClient\nfrom anytool.grounding.core.grounding_client import GroundingClient\nfrom anytool.config import get_config, load_config\nfrom anytool.config.loader import get_agent_config\nfrom anytool.recording import RecordingManager\nfrom anytool.utils.logging import Logger\n\nlogger = Logger.get_logger(__name__)\n\n\n@dataclass\nclass AnyToolConfig:\n # LLM Configuration\n llm_model: str = \"openrouter/anthropic/claude-sonnet-4.5\"\n llm_enable_thinking: bool = False\n llm_timeout: float = 120.0\n llm_max_retries: int = 3\n llm_rate_limit_delay: float = 0.0\n llm_kwargs: Dict[str, Any] = field(default_factory=dict)\n \n # Separate models for specific tasks (None = use llm_model)\n tool_retrieval_model: Optional[str] = None # Model for tool retrieval LLM filter\n visual_analysis_model: Optional[str] = None # Model for visual analysis\n \n # Grounding Configuration\n grounding_config_path: Optional[str] = None\n grounding_max_iterations: int = 20\n grounding_system_prompt: Optional[str] = None\n \n # Backend Configuration\n backend_scope: Optional[List[str]] = None # None = All backends [\"shell\", \"gui\", \"mcp\", \"web\", \"system\"]\n \n # Workspace Configuration\n workspace_dir: Optional[str] = None\n \n # Recording Configuration\n enable_recording: bool = False\n recording_backends: Optional[List[str]] = None\n recording_log_dir: str = \"./logs/recordings\"\n enable_screenshot: bool = True\n enable_video: bool = True\n enable_conversation_log: bool = True # Save LLM conversations to conversations.jsonl\n \n # Logging Configuration\n log_level: str = \"INFO\"\n log_to_file: bool = False\n log_file_path: Optional[str] = None\n \n def __post_init__(self):\n \"\"\"Validate configuration\"\"\"\n if not self.llm_model:\n raise ValueError(\"llm_model is required\")\n \n logger.debug(f\"AnyToolConfig initialized with model: {self.llm_model}\")\n\n\nclass AnyTool:\n def __init__(self, config: Optional[AnyToolConfig] = None):\n self.config = config or AnyToolConfig()\n \n self._llm_client: Optional[LLMClient] = None\n self._grounding_client: Optional[GroundingClient] = None\n self._grounding_agent: Optional[GroundingAgent] = None\n self._recording_manager: Optional[RecordingManager] = None\n \n self._initialized = False\n self._running = False\n \n logger.debug(\"AnyTool instance created\")\n \n async def initialize(self) -> None:\n if self._initialized:\n logger.warning(\"AnyTool already initialized\")\n return\n \n logger.info(\"Initializing AnyTool...\")\n \n try:\n self._llm_client = LLMClient(\n model=self.config.llm_model,\n enable_thinking=self.config.llm_enable_thinking,\n rate_limit_delay=self.config.llm_rate_limit_delay,\n max_retries=self.config.llm_max_retries,\n timeout=self.config.llm_timeout,\n **self.config.llm_kwargs\n )\n logger.info(f\"โ LLM Client: {self.config.llm_model}\")\n \n # Load grounding config\n # If custom config is provided, merge it with default configs\n # load_config supports multiple files and deep merges them (later files override earlier ones)\n if self.config.grounding_config_path:\n from anytool.config.loader import CONFIG_DIR\n from anytool.config.constants import CONFIG_GROUNDING, CONFIG_SECURITY\n # Load default configs + custom config (custom values will override defaults)\n grounding_config = load_config(\n CONFIG_DIR / CONFIG_GROUNDING,\n CONFIG_DIR / CONFIG_SECURITY,\n self.config.grounding_config_path\n )\n logger.info(f\"Merged custom grounding config: {self.config.grounding_config_path}\")\n else:\n # Load default configs only\n grounding_config = get_config()\n \n self._grounding_client = GroundingClient(config=grounding_config)\n await self._grounding_client.initialize_all_providers()\n \n backends = list(self._grounding_client.list_providers().keys())\n logger.info(f\"โ Grounding Client: {len(backends)} backends\")\n logger.debug(f\" Available backends: {[b.value for b in backends]}\")\n \n if self.config.enable_recording:\n self._recording_manager = RecordingManager(\n enabled=True,\n task_id=\"\",\n log_dir=self.config.recording_log_dir,\n backends=self.config.recording_backends,\n enable_screenshot=self.config.enable_screenshot,\n enable_video=self.config.enable_video,\n enable_conversation_log=self.config.enable_conversation_log,\n agent_name=\"AnyTool\",\n )\n # Inject recording_manager to grounding_client for GUI intermediate steps\n self._grounding_client.recording_manager = self._recording_manager\n # Register to LLM client for auto-recording tool results\n self._recording_manager.register_to_llm(self._llm_client)\n logger.info(f\"โ Recording enabled: {len(self._recording_manager.backends or [])} backends\")\n \n agent_config = get_agent_config(\"GroundingAgent\")\n if agent_config:\n # Use config file values, but command-line args (self.config) take priority\n max_iterations = agent_config.get(\"max_iterations\", self.config.grounding_max_iterations)\n # Command-line backend_scope > config file > default\n backend_scope = self.config.backend_scope or agent_config.get(\"backend_scope\") or [\"gui\", \"shell\", \"mcp\", \"web\", \"system\"]\n visual_analysis_timeout = agent_config.get(\"visual_analysis_timeout\", 30.0)\n # Update config with values from config file\n self.config.grounding_max_iterations = max_iterations\n logger.info(f\"Loaded GroundingAgent config from config_agents.json (max_iterations={max_iterations}, visual_analysis_timeout={visual_analysis_timeout}s)\")\n else:\n # Fall back to AnyToolConfig values\n max_iterations = self.config.grounding_max_iterations\n backend_scope = self.config.backend_scope or [\"gui\", \"shell\", \"mcp\", \"web\", \"system\"]\n visual_analysis_timeout = 30.0\n logger.warning(f\"config_agents.json not found, using default config (max_iterations={max_iterations})\")\n \n # Create separate LLM client for tool retrieval if configured\n tool_retrieval_llm = None\n if self.config.tool_retrieval_model:\n tool_retrieval_llm = LLMClient(\n model=self.config.tool_retrieval_model,\n timeout=self.config.llm_timeout,\n max_retries=self.config.llm_max_retries,\n )\n logger.info(f\"โ Tool retrieval LLM: {self.config.tool_retrieval_model}\")\n \n self._grounding_agent = GroundingAgent(\n name=\"AnyTool-GroundingAgent\",\n backend_scope=backend_scope,\n llm_client=self._llm_client,\n grounding_client=self._grounding_client,\n recording_manager=self._recording_manager,\n system_prompt=self.config.grounding_system_prompt,\n max_iterations=max_iterations,\n visual_analysis_timeout=visual_analysis_timeout,\n tool_retrieval_llm=tool_retrieval_llm,\n visual_analysis_model=self.config.visual_analysis_model,\n )\n logger.info(f\"โ GroundingAgent: {', '.join(backend_scope)}\")\n \n self._initialized = True\n logger.info(\"=\"*60)\n logger.info(\"AnyTool ready to use!\")\n logger.info(\"=\"*60)\n \n except Exception as e:\n logger.error(f\"Failed to initialize AnyTool: {e}\")\n await self.cleanup()\n raise\n \n async def execute(\n self,\n task: str,\n context: Optional[Dict[str, Any]] = None,\n workspace_dir: Optional[str] = None,\n max_iterations: Optional[int] = None,\n task_id: Optional[str] = None,\n ) -> Dict[str, Any]:\n \"\"\"\n Execute a task with AnyTool.\n \n Args:\n task: Task instruction\n context: Additional context\n workspace_dir: Working directory\n max_iterations: Max iterations override\n task_id: External task ID for recording/logging. If None, generates a random one.\n This allows external callers (e.g., OSWorld) to specify their own task ID\n so recordings can be easily matched with benchmark results.\n \"\"\"\n if not self._initialized:\n raise RuntimeError(\n \"AnyTool not initialized. \"\n \"Call await tool_layer.initialize() first or use async with.\"\n )\n \n if self._running:\n raise RuntimeError(\"AnyTool is already running a task.\")\n \n logger.info(\"=\"*60)\n logger.info(f\"Task: {task[:100]}...\")\n logger.info(\"=\"*60)\n \n self._running = True\n start_time = asyncio.get_event_loop().time()\n # Use external task_id if provided, otherwise generate one\n if task_id is None:\n task_id = f\"task_{uuid.uuid4().hex[:8]}\"\n logger.info(f\"Task ID: {task_id}\")\n \n try:\n execution_context = context or {}\n execution_context[\"task_id\"] = task_id\n execution_context[\"instruction\"] = task\n \n if max_iterations is not None:\n execution_context[\"max_iterations\"] = max_iterations\n \n if self._recording_manager:\n if self._recording_manager.recording_status:\n await self._recording_manager.stop()\n logger.debug(\"Stopped previous recording session\")\n \n self._recording_manager.task_id = task_id\n await self._recording_manager.start()\n logger.info(f\"Recording started: {task_id}\")\n \n if workspace_dir:\n execution_context[\"workspace_dir\"] = workspace_dir\n logger.info(f\"Workspace: {workspace_dir}\")\n elif self.config.workspace_dir:\n execution_context[\"workspace_dir\"] = self.config.workspace_dir\n logger.info(f\"Workspace: {self.config.workspace_dir}\")\n elif self._recording_manager and self._recording_manager.trajectory_dir:\n execution_context[\"workspace_dir\"] = self._recording_manager.trajectory_dir\n logger.info(f\"Workspace: {execution_context['workspace_dir']}\")\n else:\n import tempfile\n from pathlib import Path\n workspace = Path(tempfile.gettempdir()) / \"anytool_workspace\" / task_id\n workspace.mkdir(parents=True, exist_ok=True)\n execution_context[\"workspace_dir\"] = str(workspace)\n logger.info(f\"Workspace: {execution_context['workspace_dir']}\")\n \n logger.info(f\"Executing with GroundingAgent (max {max_iterations or self.config.grounding_max_iterations} iterations)...\")\n \n result = await self._grounding_agent.process(execution_context)\n \n execution_time = asyncio.get_event_loop().time() - start_time\n \n final_result = {\n **result,\n \"task_id\": task_id,\n \"execution_time\": execution_time,\n }\n \n status = result.get('status', 'unknown')\n iterations = result.get('iterations', 0)\n tool_count = len(result.get('tool_executions', []))\n \n logger.info(\"=\"*60)\n if status == \"success\":\n logger.info(\n f\"Task completed successfully! \"\n f\"({iterations} iterations, {tool_count} tool calls, {execution_time:.2f}s)\"\n )\n elif status == \"incomplete\":\n logger.warning(\n f\"Task incomplete after {iterations} iterations. \"\n f\"Consider increasing max_iterations.\"\n )\n else:\n logger.error(f\"Task failed: {result.get('error', 'Unknown error')}\")\n logger.info(\"=\"*60)\n \n return final_result\n \n except Exception as e:\n execution_time = asyncio.get_event_loop().time() - start_time\n tb = traceback.format_exc(limit=10)\n logger.error(f\"Task execution failed: {e}\", exc_info=True)\n \n return {\n \"status\": \"error\",\n \"error\": str(e),\n \"traceback\": tb,\n \"response\": f\"Task execution error: {str(e)}\",\n \"execution_time\": execution_time,\n \"task_id\": task_id,\n \"iterations\": 0,\n \"tool_executions\": [],\n }\n \n finally:\n if self._recording_manager and self._recording_manager.recording_status:\n try:\n await self._recording_manager.stop()\n logger.debug(f\"Recording stopped: {task_id}\")\n except Exception as e:\n logger.warning(f\"Failed to stop recording: {e}\")\n \n # Trigger quality evolution periodically\n await self._maybe_evolve_quality()\n \n self._running = False\n \n async def _maybe_evolve_quality(self) -> None:\n \"\"\"Trigger quality evolution based on global execution count.\"\"\"\n if not self._grounding_client or not self._grounding_client.quality_manager:\n return\n \n # Check if evolution should be triggered (every 10 global executions)\n if self._grounding_client.quality_manager.should_evolve():\n try:\n report = await self._grounding_client.evolve_quality()\n if report.get(\"recommendations\"):\n logger.info(f\"Quality evolution: {report['recommendations']}\")\n except Exception as e:\n logger.debug(f\"Quality evolution skipped: {e}\")\n \n async def cleanup(self) -> None:\n \"\"\"\n Close all sessions and release resources.\n Automatically called when using context manager.\n \"\"\"\n logger.info(\"Cleaning up AnyTool resources...\")\n \n try:\n if self._grounding_client:\n await self._grounding_client.close_all_sessions()\n logger.debug(\"All grounding sessions closed\")\n \n if self._recording_manager and self._recording_manager.recording_status:\n try:\n await self._recording_manager.stop()\n logger.debug(\"Recording manager stopped\")\n except Exception as e:\n logger.warning(f\"Failed to stop recording: {e}\")\n \n self._initialized = False\n self._running = False\n \n logger.info(\"AnyTool cleanup complete\")\n \n except Exception as e:\n logger.error(f\"Error during cleanup: {e}\", exc_info=True)\n \n def is_initialized(self) -> bool:\n return self._initialized\n \n def is_running(self) -> bool:\n return self._running\n \n def get_config(self) -> AnyToolConfig:\n return self.config\n \n def list_backends(self) -> List[str]:\n if not self._initialized:\n raise RuntimeError(\"AnyTool not initialized\")\n return [backend.value for backend in self._grounding_client.list_providers().keys()]\n \n def list_sessions(self) -> List[str]:\n if not self._initialized:\n raise RuntimeError(\"AnyTool not initialized\")\n return self._grounding_client.list_sessions()\n \n async def __aenter__(self):\n \"\"\"Context manager entry\"\"\"\n await self.initialize()\n return self\n \n async def __aexit__(self, exc_type, exc_val, exc_tb):\n \"\"\"Context manager exit\"\"\"\n await self.cleanup()\n return False\n \n def __repr__(self) -> str:\n status = \"initialized\" if self._initialized else \"not initialized\"\n if self._running:\n status = \"running\"\n backends = \", \".join(self.config.backend_scope) if self.config.backend_scope else \"all\"\n return f\"\""
},
{
"path": "anytool/utils/cli_display.py",
"content": "\"\"\"CLI Display utilities for AnyTool startup and interaction\"\"\"\n\nfrom anytool.tool_layer import AnyToolConfig\nfrom anytool.utils.display import Box, BoxStyle, colorize\n\n\nclass CLIDisplay: \n @staticmethod\n def print_banner():\n box = Box(width=70, style=BoxStyle.ROUNDED, color='c')\n \n print()\n print(box.top_line(indent=4))\n print(box.empty_line(indent=4))\n \n title = colorize(\"AnyTool\", 'c', bold=True)\n print(box.text_line(title, align='center', indent=4, text_color=''))\n \n subtitle = \"Universal Tool-Use Layer for AI Agents\"\n print(box.text_line(subtitle, align='center', indent=4, text_color='gr'))\n \n print(box.empty_line(indent=4))\n print(box.bottom_line(indent=4))\n print()\n \n @staticmethod\n def print_configuration(config: AnyToolConfig):\n box = Box(width=70, style=BoxStyle.ROUNDED, color='bl')\n \n print(box.text_line(colorize(\"โ System Configuration\", 'c', bold=True), align='center', indent=4, text_color=''))\n print(box.separator_line(indent=4))\n \n configs = [\n (\"AI Model\", config.llm_model, 'bl'),\n (\"Max Iterations\", str(config.grounding_max_iterations), 'c'),\n (\"LLM Timeout\", f\"{config.llm_timeout}s\", 'c'),\n ]\n \n for label, value, color in configs:\n line = f\" {label:20s} {colorize(value, color)}\"\n print(box.text_line(line, indent=4, text_color=''))\n \n print(box.bottom_line(indent=4))\n print()\n \n @staticmethod\n def print_initialization_progress(steps: list, show_header: bool = True):\n box = Box(width=70, style=BoxStyle.ROUNDED, color='g')\n \n if show_header:\n print(box.text_line(colorize(\"โบ Initializing Components\", 'g', bold=True), \n align='center', indent=4, text_color=''))\n print(box.separator_line(indent=4))\n \n for step, status in steps:\n if status == \"ok\":\n icon = colorize(\"โ\", 'g')\n elif status == \"error\":\n icon = colorize(\"โ\", 'rd')\n else:\n icon = colorize(\"[...]\", 'y')\n \n line = f\" {icon} {step}\"\n print(box.text_line(line, indent=4, text_color=''))\n \n print(box.bottom_line(indent=4))\n print()\n \n @staticmethod\n def print_result_summary(result: dict):\n box = Box(width=70, style=BoxStyle.ROUNDED, color='c')\n \n print()\n print(box.text_line(colorize(\"โ Execution Summary\", 'c', bold=True), \n align='center', indent=4, text_color=''))\n print(box.separator_line(indent=4))\n \n status = result.get(\"status\", \"unknown\")\n status_colors = {\n \"completed\": 'g',\n \"timeout\": 'y',\n \"error\": 'rd',\n \"max_iterations_reached\": 'y',\n }\n status_color = status_colors.get(status, 'gr')\n status_display = colorize(status.upper(), status_color, bold=True)\n \n exec_time = result.get('execution_time', 0)\n result_lines = [\n f\" Status: {status_display}\",\n f\" Execution Time: {colorize(f'{exec_time:.2f}s', 'c')}\",\n f\" Iterations: {colorize(str(result.get('iterations', 0)), 'y')}\",\n f\" Completed Tasks: {colorize(str(result.get('completed_tasks', 0)), 'g')}\",\n ]\n \n if result.get('evaluation_results'):\n result_lines.append(f\" Evaluations: {colorize(str(len(result['evaluation_results'])), 'bl')}\")\n \n for line in result_lines:\n print(box.text_line(line, indent=4, text_color=''))\n \n print(box.bottom_line(indent=4))\n print()\n \n # Print user response (the actual answer/result)\n if result.get('user_response'):\n response_box = Box(width=70, style=BoxStyle.ROUNDED, color='g')\n print(response_box.text_line(colorize(\"โ Result\", 'g', bold=True), \n align='center', indent=4, text_color=''))\n print(response_box.separator_line(indent=4))\n \n user_response = result['user_response']\n for line in user_response.split('\\n'):\n if line.strip():\n display_line = f\" {line.strip()}\"\n print(response_box.text_line(display_line, indent=4, text_color=''))\n \n print(response_box.bottom_line(indent=4))\n print()\n \n @staticmethod\n def print_interactive_header():\n box = Box(width=70, style=BoxStyle.ROUNDED, color='c')\n \n print(box.text_line(colorize(\"โจ Interactive Mode\", 'c', bold=True), \n align='center', indent=4, text_color=''))\n print(box.separator_line(indent=4))\n \n help_lines = [\n \"\",\n colorize(\" Ready to execute your tasks!\", 'g'),\n \"\",\n colorize(\" Available Commands:\", 'c', bold=True),\n \" \" + colorize(\"status\", 'bl') + \" โ View system status\",\n \" \" + colorize(\"help\", 'bl') + \" โ Show available commands\",\n \" \" + colorize(\"quit\", 'bl') + \" โ Exit interactive mode\",\n \"\",\n colorize(\" โธ Enter your task description below:\", 'gr'),\n \"\",\n ]\n \n for line in help_lines:\n print(box.text_line(line, indent=4, text_color=''))\n \n print(box.bottom_line(indent=4))\n print()\n \n @staticmethod\n def print_task_header(query: str, title: str = \"โถ Executing Task\"):\n box = Box(width=70, style=BoxStyle.ROUNDED, color='g')\n print()\n print(box.text_line(colorize(title, 'g', bold=True), align='center', indent=4, text_color=''))\n print(box.separator_line(indent=4))\n print(box.text_line(\"\", indent=4, text_color=''))\n print(box.text_line(f\" {query}\", indent=4, text_color=''))\n print(box.text_line(\"\", indent=4, text_color=''))\n print(box.bottom_line(indent=4))\n \n @staticmethod\n def print_system_ready():\n box = Box(width=70, style=BoxStyle.ROUNDED, color='g')\n print(box.text_line(colorize(\"โ System Ready\", 'g', bold=True), \n align='center', indent=4, text_color=''))\n print(box.separator_line(indent=4))\n print(box.text_line(\"\", indent=4, text_color=''))\n print(box.text_line(colorize(\" Real-time UI will display:\", 'c'), indent=4, text_color=''))\n print(box.text_line(\" ยง Agent activities and status\", indent=4, text_color=''))\n print(box.text_line(\" โ Grounding backend operations\", indent=4, text_color=''))\n print(box.text_line(\" โ Execution logs\", indent=4, text_color=''))\n print(box.text_line(\"\", indent=4, text_color=''))\n print(box.bottom_line(indent=4))\n print()\n \n @staticmethod\n def print_status(agent):\n box = Box(width=70, style=BoxStyle.ROUNDED, color='bl')\n print()\n print(box.text_line(colorize(\"System Status\", 'bl', bold=True), \n align='center', indent=4, text_color=''))\n print(box.separator_line(indent=4))\n \n status = agent.get_status()\n status_lines = [\n f\"Initialized: {colorize('Yes' if status['initialized'] else 'No', 'g' if status['initialized'] else 'rd')}\",\n f\"Running: {colorize('Yes' if status['running'] else 'No', 'y' if status['running'] else 'g')}\",\n ]\n \n if \"agents\" in status:\n status_lines.append(f\"Agents: {colorize(', '.join(status['agents']), 'c')}\")\n \n for line in status_lines:\n print(box.text_line(line, indent=4, text_color=''))\n \n print(box.bottom_line(indent=4))\n print()\n \n @staticmethod\n def print_help():\n box = Box(width=70, style=BoxStyle.ROUNDED, color='y')\n print()\n print(box.text_line(colorize(\"Available Commands\", 'y', bold=True), \n align='center', indent=4, text_color=''))\n print(box.separator_line(indent=4))\n \n help_items = [\n (colorize(\"status\", 'c'), \"Show system status\"),\n (colorize(\"help\", 'c'), \"Show this help message\"),\n (colorize(\"quit/exit\", 'c'), \"Exit interactive mode\"),\n (\"\", \"\"),\n (colorize(\"Other input\", 'gr'), \"Execute as task\"),\n ]\n \n for cmd, desc in help_items:\n if cmd:\n print(box.text_line(f\" {cmd:20s} {desc}\", indent=4, text_color=''))\n else:\n print(box.separator_line(indent=4))\n \n print(box.bottom_line(indent=4))\n print()"
},
{
"path": "anytool/utils/display.py",
"content": "from typing import Optional, List\nfrom enum import Enum\nimport re\n\n\nclass Colors:\n RESET = \"\\033[0m\"\n BOLD = \"\\033[1m\"\n DIM = \"\\033[2m\"\n \n RED = \"\\033[91m\"\n GREEN = \"\\033[92m\"\n YELLOW = \"\\033[93m\"\n BLUE = \"\\033[94m\"\n MAGENTA = \"\\033[95m\"\n CYAN = \"\\033[96m\"\n WHITE = \"\\033[97m\"\n GRAY = \"\\033[90m\"\n \n GREEN_SOFT = '\\033[38;5;78m'\n BLUE_SOFT = '\\033[38;5;39m'\n CYAN_SOFT = '\\033[38;5;51m'\n YELLOW_SOFT = '\\033[38;5;222m'\n RED_SOFT = '\\033[38;5;204m'\n MAGENTA_SOFT = '\\033[38;5;141m'\n GRAY_SOFT = '\\033[38;5;246m'\n\n\nclass BoxStyle(Enum):\n ROUNDED = \"rounded\" # Rounded corner box โญโโฎโฐโฏ\n SQUARE = \"square\" # Square corner box โโโโโ\n DOUBLE = \"double\" # Double line box โโโโโ\n SIMPLE = \"simple\" # Simple box ===\n\n\nBOX_CHARS = {\n BoxStyle.ROUNDED: {\n 'tl': 'โญ', 'tr': 'โฎ', 'bl': 'โฐ', 'br': 'โฏ',\n 'h': 'โ', 'v': 'โ'\n },\n BoxStyle.SQUARE: {\n 'tl': 'โ', 'tr': 'โ', 'bl': 'โ', 'br': 'โ',\n 'h': 'โ', 'v': 'โ'\n },\n BoxStyle.DOUBLE: {\n 'tl': 'โ', 'tr': 'โ', 'bl': 'โ', 'br': 'โ',\n 'h': 'โ', 'v': 'โ'\n },\n}\n\n\ndef strip_ansi(text: str) -> str:\n \"\"\"\n Strip ANSI color codes from text\n \n Args:\n text: Text with potential ANSI codes\n \n Returns:\n Clean text without ANSI codes\n \"\"\"\n ansi_escape = re.compile(r'\\x1B(?:[@-Z\\\\-_]|\\[[0-?]*[ -/]*[@-~])')\n return ansi_escape.sub('', text)\n\n\ndef colorize(text: str, color: str = '', bold: bool = False) -> str:\n try:\n color_map = {\n 'r': Colors.RESET,\n 'b': Colors.BOLD,\n 'd': Colors.DIM,\n 'g': Colors.GREEN_SOFT,\n 'bl': Colors.BLUE_SOFT,\n 'c': Colors.CYAN_SOFT,\n 'y': Colors.YELLOW_SOFT,\n 'rd': Colors.RED_SOFT,\n 'm': Colors.MAGENTA_SOFT,\n 'gr': Colors.GRAY_SOFT,\n }\n \n prefix = Colors.BOLD if bold else ''\n code = color_map.get(color, color)\n return f\"{prefix}{code}{text}{Colors.RESET}\"\n except:\n return text\n\n\nclass Box:\n def __init__(self, \n width: int = 68,\n style: BoxStyle = BoxStyle.ROUNDED,\n color: str = 'bl',\n padding: int = 2):\n \n self.width = width\n self.style = style\n self.color = color\n self.padding = padding\n self.chars = BOX_CHARS.get(style, BOX_CHARS[BoxStyle.ROUNDED])\n \n def top_line(self, indent: int = 2) -> str:\n indent_str = \" \" * indent\n if self.style == BoxStyle.SIMPLE:\n return colorize(indent_str + \"=\" * self.width, self.color)\n return colorize(\n indent_str + self.chars['tl'] + self.chars['h'] * self.width + self.chars['tr'],\n self.color\n )\n \n def bottom_line(self, indent: int = 2) -> str:\n indent_str = \" \" * indent\n if self.style == BoxStyle.SIMPLE:\n return colorize(indent_str + \"=\" * self.width, self.color)\n return colorize(\n indent_str + self.chars['bl'] + self.chars['h'] * self.width + self.chars['br'],\n self.color\n )\n \n def separator_line(self, indent: int = 2) -> str:\n indent_str = \" \" * indent\n if self.style == BoxStyle.SIMPLE:\n return colorize(indent_str + \"-\" * self.width, self.color)\n return colorize(indent_str + \" \" + self.chars['h'] * self.width, self.color)\n \n def empty_line(self, indent: int = 2) -> str:\n indent_str = \" \" * indent\n if self.style == BoxStyle.SIMPLE:\n return \"\"\n return colorize(\n indent_str + self.chars['v'] + \" \" * self.width + self.chars['v'],\n self.color\n )\n \n def text_line(self, text: str, align: str = 'left', indent: int = 2, text_color: str = '') -> str:\n indent_str = \" \" * indent\n content_width = self.width - 2 * self.padding\n \n # Strip ANSI codes to get actual display length\n clean_text = strip_ansi(text)\n text_len = len(clean_text)\n \n # Use original text (may contain colors) or apply new color\n display_text = colorize(text, text_color) if text_color else text\n \n if align == 'center':\n left_pad = (content_width - text_len) // 2\n right_pad = content_width - text_len - left_pad\n content = \" \" * left_pad + display_text + \" \" * right_pad\n elif align == 'right':\n left_pad = content_width - text_len\n content = \" \" * left_pad + display_text\n else: # left\n right_pad = content_width - text_len\n content = display_text + \" \" * right_pad\n \n if self.style == BoxStyle.SIMPLE:\n return indent_str + \" \" * self.padding + content\n \n padding_str = \" \" * self.padding\n return colorize(indent_str + self.chars['v'], self.color) + \\\n padding_str + content + padding_str + \\\n colorize(self.chars['v'], self.color)\n \n def build(self, \n title: Optional[str] = None,\n lines: List[str] = None,\n footer: Optional[str] = None,\n indent: int = 2) -> str:\n \n result = []\n \n result.append(self.top_line(indent))\n \n if title:\n result.append(self.empty_line(indent))\n result.append(self.text_line(title, align='center', indent=indent, text_color='c'))\n result.append(self.empty_line(indent))\n \n if lines:\n for line in lines:\n result.append(self.text_line(line, indent=indent))\n \n if footer:\n result.append(self.empty_line(indent))\n result.append(self.text_line(footer, align='center', indent=indent, text_color='gr'))\n \n result.append(self.bottom_line(indent))\n \n return \"\\n\".join(result)\n\n\ndef print_box(title: Optional[str] = None,\n lines: List[str] = None,\n footer: Optional[str] = None,\n width: int = 68,\n style: BoxStyle = BoxStyle.ROUNDED,\n color: str = 'bl',\n indent: int = 2):\n \n box = Box(width=width, style=style, color=color)\n print(box.build(title=title, lines=lines, footer=footer, indent=indent))\n\n\ndef print_banner(title: str,\n subtitle: Optional[str] = None,\n width: int = 66,\n style: BoxStyle = BoxStyle.ROUNDED,\n color: str = 'bl',\n indent: int = 2):\n \n box = Box(width=width, style=style, color=color)\n print()\n print(box.top_line(indent))\n print(box.empty_line(indent))\n print(box.text_line(title, align='center', indent=indent, text_color='c'))\n if subtitle:\n print(box.text_line(subtitle, align='center', indent=indent, text_color='gr'))\n print(box.empty_line(indent))\n print(box.bottom_line(indent))\n print()\n\n\ndef print_section(title: str,\n content: List[str],\n color: str = 'c',\n indent: int = 2):\n indent_str = \" \" * indent\n print(f\"\\n{indent_str}{colorize('- ' + title, color, bold=True)}\")\n for line in content:\n print(f\"{indent_str} {line}\")\n\n\ndef print_separator(width: int = 68, color: str = 'bl', indent: int = 2):\n indent_str = \" \" * indent\n print(colorize(indent_str + \"โ\" * width, color))"
},
{
"path": "anytool/utils/logging.py",
"content": "import logging\nimport os\nimport sys\nimport threading\nimport json\nfrom pathlib import Path\nfrom datetime import datetime\nfrom typing import Optional\nfrom colorama import init\n\ninit(autoreset=True)\n\n\ndef _load_log_level_from_config() -> int:\n \"\"\"\n Load log_level from config_grounding.json and convert to ANYTOOL_DEBUG value.\n Returns: 0 (WARNING), 1 (INFO), or 2 (DEBUG)\n \"\"\"\n try:\n config_path = Path(__file__).parent.parent / \"config\" / \"config_grounding.json\"\n if config_path.exists():\n with open(config_path, 'r', encoding='utf-8') as f:\n config = json.load(f)\n log_level = config.get(\"log_level\", \"INFO\").upper()\n \n # Convert log level string to ANYTOOL_DEBUG value\n level_map = {\n \"DEBUG\": 2,\n \"INFO\": 1,\n \"WARNING\": 0,\n \"ERROR\": 0,\n \"CRITICAL\": 0\n }\n return level_map.get(log_level, 1) # Default to INFO\n except Exception:\n # If any error occurs, silently return default INFO level\n pass\n return 1 # Default to INFO\n\n\n# 0=WARNING, 1=INFO, 2=DEBUG; can be overridden by set_debug / environment variable\n# Load from config_grounding.json to ensure consistency\nANYTOOL_DEBUG = _load_log_level_from_config()\n\n# Default log directory and file pattern\n# Use absolute path to anytool/logs directory\nDEFAULT_LOG_DIR = os.path.join(os.path.dirname(__file__), \"..\", \"..\", \"logs\")\nDEFAULT_LOG_FILE_PATTERN = \"anytool_{timestamp}.log\"\n\n\nclass FlushFileHandler(logging.FileHandler):\n \"\"\"File handler that flushes after each emit for real-time logging\"\"\"\n \n def emit(self, record):\n super().emit(record)\n self.flush() # Immediately flush to disk\n\n\nclass ColoredFormatter(logging.Formatter):\n COLORS = {\n 'DEBUG': '\\033[1;36m', # Bold cyan\n 'INFO': '\\033[1;32m', # Bold green\n 'WARNING': '\\033[1;33m', # Bold yellow\n 'ERROR': '\\033[1;31m', # Bold red\n 'CRITICAL': '\\033[1;35m', # Bold magenta\n 'RESET': '\\033[0m',\n }\n\n def format(self, record: logging.LogRecord) -> str:\n formatted = super().format(record)\n \n level_color = self.COLORS.get(record.levelname, self.COLORS[\"RESET\"])\n colored_line = f\"{level_color}{formatted}{self.COLORS['RESET']}\"\n \n return colored_line\n\n\nclass Logger:\n \"\"\"\n Thread-safe logger facade that:\n 1. Configures handlers only once (lazy initialization).\n 2. Ensures all subsequent loggers obtained via ``Logger.get_logger()``\n inherit the configured handlers.\n 3. Dynamically adapts log levels according to ``ANYTOOL_DEBUG``.\n \"\"\"\n\n _ROOT_NAME = \"anytool\" # Package root name\n # Standard format: time with milliseconds | level | file:line number | message\n _LOG_FORMAT = (\n \"%(asctime)s.%(msecs)03d [%(levelname)-8s] %(filename)s:%(lineno)d - %(message)s\"\n )\n\n _lock = threading.Lock()\n _configured = False\n _registered: dict[str, logging.Logger] = {}\n \n @staticmethod\n def _get_default_log_file() -> str:\n \"\"\"Generate default log file path with timestamp (to seconds)\n \n Log files are organized by the running script name:\n - logs//anytool_2025-10-24_15-30-00.log\n \"\"\"\n # Get the name of the main script\n script_name = \"anytool\" # Default name\n try:\n import __main__\n if hasattr(__main__, \"__file__\") and __main__.__file__:\n # Extract script name without extension\n script_path = os.path.basename(__main__.__file__)\n script_name = os.path.splitext(script_path)[0]\n except Exception:\n # If can't get script name, use default\n pass\n \n # Create log directory: logs//\n log_dir = os.path.join(DEFAULT_LOG_DIR, script_name)\n \n timestamp = datetime.now().strftime(\"%Y-%m-%d_%H-%M-%S\")\n filename = DEFAULT_LOG_FILE_PATTERN.format(timestamp=timestamp)\n return os.path.abspath(os.path.join(log_dir, filename))\n\n @classmethod\n def get_logger(cls, name: Optional[str] = None) -> logging.Logger:\n \"\"\"Return a logger with *name* (defaults to ``anytool``).\n The first call triggers :meth:`configure` automatically.\"\"\"\n if name is None:\n name = cls._ROOT_NAME\n\n # Check if configuration is needed to avoid recursive calls.\n need_config = False\n with cls._lock:\n logger = cls._registered.get(name)\n if logger is None:\n logger = logging.getLogger(name)\n logger.propagate = True\n cls._registered[name] = logger\n if not cls._configured:\n need_config = True\n\n if need_config:\n cls.configure()\n return logger\n\n @classmethod\n def configure(\n cls,\n *,\n level: Optional[int] = None,\n fmt: Optional[str] = None,\n log_to_console: bool = True,\n log_to_file: Optional[str] = \"auto\",\n use_colors: bool = True,\n force_color: bool = False,\n force: bool = False,\n attach_to_root: bool = False,\n ) -> None:\n \"\"\"\n Configure the logging system. Usually called automatically\n on first use; pass ``force=True`` to reconfigure explicitly.\n\n Args:\n level: Log level (DEBUG, INFO, WARNING, ERROR, CRITICAL)\n fmt: Log format string\n log_to_console: Whether to output to console\n log_to_file: Log file path (\"auto\" auto-generate by date, None disable, or specify path)\n use_colors: Whether to use colors on console\n force_color: Force use of colors (even if not supported)\n force: Whether to force reconfiguration\n attach_to_root: Whether to attach to root logger\n\n If *attach_to_root* is ``True``, handlers are attached to the *root*\n logger (``\"\"``). This makes every loggerโregardless of its nameโ\n inherit the handlers (handy for standalone scripts) but will also\n surface logs from third-party libraries. Choose with care.\n \"\"\"\n with cls._lock:\n if cls._configured and not force:\n # Already configured and no need to force reconfiguration, only update level.\n if level is not None:\n cls._update_level(level)\n return\n\n resolved_level = cls._resolve_level(level)\n fmt_str = fmt or cls._LOG_FORMAT\n\n # Handle log_to_file parameter\n actual_log_file = None\n if log_to_file == \"auto\":\n actual_log_file = cls._get_default_log_file()\n elif log_to_file is not None:\n actual_log_file = log_to_file\n\n # Select the logger to attach handlers to (root logger or anytool).\n target_logger = (\n logging.getLogger() if attach_to_root else logging.getLogger(cls._ROOT_NAME)\n )\n target_logger.setLevel(resolved_level)\n\n # Clean up old handlers.\n for h in target_logger.handlers[:]:\n target_logger.removeHandler(h)\n\n # Construct Formatter\n date_fmt = \"%Y-%m-%d %H:%M:%S\"\n color_supported = force_color or (use_colors and cls._stdout_supports_color())\n console_formatter = (\n ColoredFormatter(fmt_str, datefmt=date_fmt) if color_supported \n else logging.Formatter(fmt_str, datefmt=date_fmt)\n )\n file_formatter = logging.Formatter(fmt_str, datefmt=date_fmt)\n\n # Console Handler\n if log_to_console:\n ch = logging.StreamHandler(sys.stdout)\n ch.setLevel(resolved_level)\n ch.setFormatter(console_formatter)\n target_logger.addHandler(ch)\n\n # File Handler (with real-time flush)\n if actual_log_file:\n dir_path = os.path.dirname(actual_log_file)\n if dir_path:\n os.makedirs(dir_path, exist_ok=True)\n fh = FlushFileHandler(actual_log_file, encoding=\"utf-8\")\n fh.setLevel(resolved_level)\n fh.setFormatter(file_formatter)\n target_logger.addHandler(fh)\n \n # Record log file location\n if not cls._configured:\n print(f\"Log file enabled: {actual_log_file}\")\n\n cls._configured = True\n\n @classmethod\n def set_debug(cls, debug_level: int = 2) -> None:\n \"\"\"Dynamically switch debug level: 0 = WARNING, 1 = INFO, 2 = DEBUG.\"\"\"\n global ANYTOOL_DEBUG\n ANYTOOL_DEBUG = max(0, min(debug_level, 2))\n cls._update_level(cls._resolve_level(None))\n\n @classmethod\n def add_file_handler(\n cls, \n filepath: str, \n logger_name: Optional[str] = None\n ) -> None:\n \"\"\"\n Append a file handler to the given (default ``anytool``) logger.\n \n Args:\n filepath: Log file path\n logger_name: Log logger name\n \"\"\"\n logger = cls.get_logger(logger_name or cls._ROOT_NAME)\n\n dir_path = os.path.dirname(filepath)\n if dir_path:\n os.makedirs(dir_path, exist_ok=True)\n\n fh = FlushFileHandler(filepath, encoding=\"utf-8\")\n fh.setLevel(logger.level)\n fh.setFormatter(logging.Formatter(cls._LOG_FORMAT, datefmt=\"%Y-%m-%d %H:%M:%S\"))\n logger.addHandler(fh)\n\n @classmethod\n def reset_configuration(cls) -> None:\n \"\"\"Remove all handlers and clear registered loggers.\"\"\"\n with cls._lock:\n for lg in cls._registered.values():\n for h in lg.handlers[:]:\n lg.removeHandler(h)\n cls._registered.clear()\n cls._configured = False\n\n @staticmethod\n def _stdout_supports_color() -> bool:\n return sys.stdout.isatty() and not os.getenv(\"NO_COLOR\")\n\n @classmethod\n def _resolve_level(cls, level: Optional[int]) -> int:\n if level is not None:\n # Allow passing logging.INFO / \"INFO\" / 20 etc.\n return getattr(logging, str(level).upper(), level)\n return {2: logging.DEBUG, 1: logging.INFO}.get(ANYTOOL_DEBUG, logging.WARNING)\n\n @classmethod\n def _update_level(cls, level: int) -> None:\n for lg in cls._registered.values():\n lg.setLevel(level)\n for h in lg.handlers:\n h.setLevel(level)\n\n\n# Adjust debug level automatically according to the\n# ``ANYTOOL_DEBUG`` (preferred) or legacy ``DEBUG`` environment variable.\n_env_debug = os.getenv(\"ANYTOOL_DEBUG\") or os.getenv(\"DEBUG\")\nif _env_debug is not None:\n try:\n Logger.set_debug(int(_env_debug))\n except ValueError:\n # When not a number, use common format: DEBUG=1/true\n Logger.set_debug(2 if _env_debug.strip().lower() in {\"1\", \"true\", \"yes\"} else 0)\n\n# Initialize logger system, attach to root so all loggers inherit the configuration\n# This ensures any logger obtained via Logger.get_logger() will work correctly\nLogger.configure(attach_to_root=True)\n\n# Get anytool logger for internal logging\nlogger = Logger.get_logger()\nlogger.debug(\"AnyTool logging initialized\")"
},
{
"path": "anytool/utils/telemetry/__init__.py",
"content": ""
},
{
"path": "anytool/utils/telemetry/events.py",
"content": "from abc import ABC, abstractmethod\nfrom dataclasses import dataclass\nfrom typing import Any\n\n\nclass BaseTelemetryEvent(ABC):\n \"\"\"Base class for all telemetry events\"\"\"\n\n @property\n @abstractmethod\n def name(self) -> str:\n \"\"\"Event name for tracking\"\"\"\n pass\n\n @property\n @abstractmethod\n def properties(self) -> dict[str, Any]:\n \"\"\"Event properties to send with the event\"\"\"\n pass\n\n\n@dataclass\nclass MCPAgentExecutionEvent(BaseTelemetryEvent):\n \"\"\"Comprehensive event for tracking complete MCP agent execution\"\"\"\n\n # Execution method and context\n execution_method: str # \"run\" or \"astream\"\n query: str # The actual user query\n success: bool\n\n # Agent configuration\n model_provider: str\n model_name: str\n server_count: int\n server_identifiers: list[dict[str, str]]\n total_tools_available: int\n tools_available_names: list[str]\n max_steps_configured: int\n memory_enabled: bool\n use_server_manager: bool\n\n # Execution PARAMETERS\n max_steps_used: int | None\n manage_connector: bool\n external_history_used: bool\n\n # Execution results\n steps_taken: int | None = None\n tools_used_count: int | None = None\n tools_used_names: list[str] | None = None\n response: str | None = None # The actual response\n execution_time_ms: int | None = None\n error_type: str | None = None\n\n # Context\n conversation_history_length: int | None = None\n\n @property\n def name(self) -> str:\n return \"mcp_agent_execution\"\n\n @property\n def properties(self) -> dict[str, Any]:\n return {\n # Core execution info\n \"execution_method\": self.execution_method,\n \"query\": self.query,\n \"query_length\": len(self.query),\n \"success\": self.success,\n # Agent configuration\n \"model_provider\": self.model_provider,\n \"model_name\": self.model_name,\n \"server_count\": self.server_count,\n \"server_identifiers\": self.server_identifiers,\n \"total_tools_available\": self.total_tools_available,\n \"tools_available_names\": self.tools_available_names,\n \"max_steps_configured\": self.max_steps_configured,\n \"memory_enabled\": self.memory_enabled,\n \"use_server_manager\": self.use_server_manager,\n # Execution parameters (always include, even if None)\n \"max_steps_used\": self.max_steps_used,\n \"manage_connector\": self.manage_connector,\n \"external_history_used\": self.external_history_used,\n # Execution results (always include, even if None)\n \"steps_taken\": self.steps_taken,\n \"tools_used_count\": self.tools_used_count,\n \"tools_used_names\": self.tools_used_names,\n \"response\": self.response,\n \"response_length\": len(self.response) if self.response else None,\n \"execution_time_ms\": self.execution_time_ms,\n \"error_type\": self.error_type,\n \"conversation_history_length\": self.conversation_history_length,\n }"
},
{
"path": "anytool/utils/telemetry/telemetry.py",
"content": "import logging\nimport os\nimport platform\nimport uuid\nfrom collections.abc import Callable\nfrom functools import wraps\nfrom pathlib import Path\nfrom typing import Any\n\nfrom posthog import Posthog\nfrom scarf import ScarfEventLogger\n\nfrom mcp_use.logging import MCP_USE_DEBUG\nfrom mcp_use.telemetry.events import (\n BaseTelemetryEvent,\n MCPAgentExecutionEvent,\n)\nfrom mcp_use.telemetry.utils import get_package_version\n\nlogger = logging.getLogger(__name__)\n\n\ndef singleton(cls):\n \"\"\"A decorator that implements the singleton pattern for a class.\"\"\"\n instance = [None]\n \n def wrapper(*args, **kwargs):\n if instance[0] is None:\n instance[0] = cls(*args, **kwargs)\n return instance[0]\n \n return wrapper\n\ndef requires_telemetry(func: Callable) -> Callable:\n \"\"\"Decorator that skips function execution if telemetry is disabled\"\"\"\n\n @wraps(func)\n def wrapper(self, *args, **kwargs):\n if not self._posthog_client and not self._scarf_client:\n return None\n return func(self, *args, **kwargs)\n\n return wrapper\n\n\ndef get_cache_home() -> Path:\n \"\"\"Get platform-appropriate cache directory.\"\"\"\n # XDG_CACHE_HOME for Linux and manually set envs\n env_var: str | None = os.getenv(\"XDG_CACHE_HOME\")\n if env_var and (path := Path(env_var)).is_absolute():\n return path\n\n system = platform.system()\n if system == \"Windows\":\n appdata = os.getenv(\"LOCALAPPDATA\") or os.getenv(\"APPDATA\")\n if appdata:\n return Path(appdata)\n return Path.home() / \"AppData\" / \"Local\"\n elif system == \"Darwin\": # macOS\n return Path.home() / \"Library\" / \"Caches\"\n else: # Linux or other Unix\n return Path.home() / \".cache\"\n\n\n@singleton\nclass Telemetry:\n \"\"\"\n Service for capturing anonymized telemetry data via PostHog and Scarf.\n If the environment variable `MCP_USE_ANONYMIZED_TELEMETRY=false`, telemetry will be disabled.\n \"\"\"\n\n USER_ID_PATH = str(get_cache_home() / \"mcp_use_3\" / \"telemetry_user_id\")\n VERSION_DOWNLOAD_PATH = str(get_cache_home() / \"mcp_use\" / \"download_version\")\n PROJECT_API_KEY = \"phc_lyTtbYwvkdSbrcMQNPiKiiRWrrM1seyKIMjycSvItEI\"\n HOST = \"https://eu.i.posthog.com\"\n SCARF_GATEWAY_URL = \"https://mcpuse.gateway.scarf.sh/events\"\n UNKNOWN_USER_ID = \"UNKNOWN_USER_ID\"\n\n _curr_user_id = None\n\n def __init__(self):\n telemetry_disabled = os.getenv(\"MCP_USE_ANONYMIZED_TELEMETRY\", \"true\").lower() == \"false\"\n\n if telemetry_disabled:\n self._posthog_client = None\n self._scarf_client = None\n logger.debug(\"Telemetry disabled\")\n else:\n logger.info(\"Anonymized telemetry enabled. Set MCP_USE_ANONYMIZED_TELEMETRY=false to disable.\")\n\n # Initialize PostHog\n try:\n self._posthog_client = Posthog(\n project_api_key=self.PROJECT_API_KEY,\n host=self.HOST,\n disable_geoip=False,\n enable_exception_autocapture=True,\n )\n\n # Silence posthog's logging unless debug mode (level 2)\n if MCP_USE_DEBUG < 2:\n posthog_logger = logging.getLogger(\"posthog\")\n posthog_logger.disabled = True\n\n except Exception as e:\n logger.warning(f\"Failed to initialize PostHog telemetry: {e}\")\n self._posthog_client = None\n\n # Initialize Scarf\n try:\n self._scarf_client = ScarfEventLogger(\n endpoint_url=self.SCARF_GATEWAY_URL,\n timeout=3.0,\n verbose=MCP_USE_DEBUG >= 2,\n )\n\n # Silence scarf's logging unless debug mode (level 2)\n if MCP_USE_DEBUG < 2:\n scarf_logger = logging.getLogger(\"scarf\")\n scarf_logger.disabled = True\n\n except Exception as e:\n logger.warning(f\"Failed to initialize Scarf telemetry: {e}\")\n self._scarf_client = None\n\n @property\n def user_id(self) -> str:\n \"\"\"Get or create a persistent anonymous user ID\"\"\"\n if self._curr_user_id:\n return self._curr_user_id\n\n try:\n is_first_time = not os.path.exists(self.USER_ID_PATH)\n\n if is_first_time:\n logger.debug(f\"Creating user ID path: {self.USER_ID_PATH}\")\n os.makedirs(os.path.dirname(self.USER_ID_PATH), exist_ok=True)\n with open(self.USER_ID_PATH, \"w\") as f:\n new_user_id = str(uuid.uuid4())\n f.write(new_user_id)\n self._curr_user_id = new_user_id\n\n logger.debug(f\"User ID path created: {self.USER_ID_PATH}\")\n else:\n with open(self.USER_ID_PATH) as f:\n self._curr_user_id = f.read().strip()\n\n # Always check for version-based download tracking\n self.track_package_download(\n {\n \"triggered_by\": \"user_id_property\",\n }\n )\n except Exception as e:\n logger.debug(f\"Failed to get/create user ID: {e}\")\n self._curr_user_id = self.UNKNOWN_USER_ID\n\n return self._curr_user_id\n\n @requires_telemetry\n def capture(self, event: BaseTelemetryEvent) -> None:\n \"\"\"Capture a telemetry event\"\"\"\n # Send to PostHog\n if self._posthog_client:\n try:\n # Add package version to all events\n properties = event.properties.copy()\n properties[\"mcp_use_version\"] = get_package_version()\n\n self._posthog_client.capture(distinct_id=self.user_id, event=event.name, properties=properties)\n except Exception as e:\n logger.debug(f\"Failed to track PostHog event {event.name}: {e}\")\n\n # Send to Scarf\n if self._scarf_client:\n try:\n # Add package version and user_id to all events\n properties = {}\n properties[\"mcp_use_version\"] = get_package_version()\n properties[\"user_id\"] = self.user_id\n properties[\"event\"] = event.name\n\n # Convert complex types to simple types for Scarf compatibility\n self._scarf_client.log_event(properties=properties)\n except Exception as e:\n logger.debug(f\"Failed to track Scarf event {event.name}: {e}\")\n\n @requires_telemetry\n def track_package_download(self, properties: dict[str, Any] | None = None) -> None:\n \"\"\"Track package download event specifically for Scarf analytics\"\"\"\n if self._scarf_client:\n try:\n current_version = get_package_version()\n should_track = False\n first_download = False\n\n # Check if version file exists\n if not os.path.exists(self.VERSION_DOWNLOAD_PATH):\n # First download\n should_track = True\n first_download = True\n\n # Create directory and save version\n os.makedirs(os.path.dirname(self.VERSION_DOWNLOAD_PATH), exist_ok=True)\n with open(self.VERSION_DOWNLOAD_PATH, \"w\") as f:\n f.write(current_version)\n else:\n # Read saved version\n with open(self.VERSION_DOWNLOAD_PATH) as f:\n saved_version = f.read().strip()\n\n # Compare versions (simple string comparison for now)\n if current_version > saved_version:\n should_track = True\n first_download = False\n\n # Update saved version\n with open(self.VERSION_DOWNLOAD_PATH, \"w\") as f:\n f.write(current_version)\n\n if should_track:\n logger.debug(f\"Tracking package download event with properties: {properties}\")\n # Add package version and user_id to event\n event_properties = (properties or {}).copy()\n event_properties[\"mcp_use_version\"] = current_version\n event_properties[\"user_id\"] = self.user_id\n event_properties[\"event\"] = \"package_download\"\n event_properties[\"first_download\"] = first_download\n\n # Convert complex types to simple types for Scarf compatibility\n self._scarf_client.log_event(properties=event_properties)\n except Exception as e:\n logger.debug(f\"Failed to track Scarf package_download event: {e}\")\n\n @requires_telemetry\n def track_agent_execution(\n self,\n execution_method: str,\n query: str,\n success: bool,\n model_provider: str,\n model_name: str,\n server_count: int,\n server_identifiers: list[dict[str, str]],\n total_tools_available: int,\n tools_available_names: list[str],\n max_steps_configured: int,\n memory_enabled: bool,\n use_server_manager: bool,\n max_steps_used: int | None,\n manage_connector: bool,\n external_history_used: bool,\n steps_taken: int | None = None,\n tools_used_count: int | None = None,\n tools_used_names: list[str] | None = None,\n response: str | None = None,\n execution_time_ms: int | None = None,\n error_type: str | None = None,\n conversation_history_length: int | None = None,\n ) -> None:\n \"\"\"Track comprehensive agent execution\"\"\"\n event = MCPAgentExecutionEvent(\n execution_method=execution_method,\n query=query,\n success=success,\n model_provider=model_provider,\n model_name=model_name,\n server_count=server_count,\n server_identifiers=server_identifiers,\n total_tools_available=total_tools_available,\n tools_available_names=tools_available_names,\n max_steps_configured=max_steps_configured,\n memory_enabled=memory_enabled,\n use_server_manager=use_server_manager,\n max_steps_used=max_steps_used,\n manage_connector=manage_connector,\n external_history_used=external_history_used,\n steps_taken=steps_taken,\n tools_used_count=tools_used_count,\n tools_used_names=tools_used_names,\n response=response,\n execution_time_ms=execution_time_ms,\n error_type=error_type,\n conversation_history_length=conversation_history_length,\n )\n self.capture(event)\n\n @requires_telemetry\n def flush(self) -> None:\n \"\"\"Flush any queued telemetry events\"\"\"\n # Flush PostHog\n if self._posthog_client:\n try:\n self._posthog_client.flush()\n logger.debug(\"PostHog client telemetry queue flushed\")\n except Exception as e:\n logger.debug(f\"Failed to flush PostHog client: {e}\")\n\n # Scarf events are sent immediately, no flush needed\n if self._scarf_client:\n logger.debug(\"Scarf telemetry events sent immediately (no flush needed)\")\n\n @requires_telemetry\n def shutdown(self) -> None:\n \"\"\"Shutdown telemetry clients and flush remaining events\"\"\"\n # Shutdown PostHog\n if self._posthog_client:\n try:\n self._posthog_client.shutdown()\n logger.debug(\"PostHog client shutdown successfully\")\n except Exception as e:\n logger.debug(f\"Error shutting down PostHog client: {e}\")\n\n # Scarf doesn't require explicit shutdown\n if self._scarf_client:\n logger.debug(\"Scarf telemetry client shutdown (no action needed)\")\n"
},
{
"path": "anytool/utils/telemetry/utils.py",
"content": "\"\"\"\nUtility functions for extracting model information from LangChain LLMs.\n\nThis module provides utilities to extract provider and model information\nfrom LangChain language models for telemetry purposes.\n\"\"\"\n\nimport importlib.metadata\n\nfrom langchain_core.language_models.base import BaseLanguageModel\n\n\ndef get_package_version() -> str:\n \"\"\"Get the current mcp-use package version.\"\"\"\n try:\n return importlib.metadata.version(\"mcp-use\")\n except importlib.metadata.PackageNotFoundError:\n return \"unknown\"\n\n\ndef get_model_provider(llm: BaseLanguageModel) -> str:\n \"\"\"Extract the model provider from LangChain LLM using BaseChatModel standards.\"\"\"\n # Use LangChain's standard _llm_type property for identification\n return getattr(llm, \"_llm_type\", llm.__class__.__name__.lower())\n\n\ndef get_model_name(llm: BaseLanguageModel) -> str:\n \"\"\"Extract the model name from LangChain LLM using BaseChatModel standards.\"\"\"\n # First try _identifying_params which may contain model info\n if hasattr(llm, \"_identifying_params\"):\n identifying_params = llm._identifying_params\n if isinstance(identifying_params, dict):\n # Common keys that contain model names\n for key in [\"model\", \"model_name\", \"model_id\", \"deployment_name\"]:\n if key in identifying_params:\n return str(identifying_params[key])\n\n # Fallback to direct model attributes\n return getattr(llm, \"model\", getattr(llm, \"model_name\", llm.__class__.__name__))\n\n\ndef extract_model_info(llm: BaseLanguageModel) -> tuple[str, str]:\n \"\"\"Extract both provider and model name from LangChain LLM.\n\n Returns:\n Tuple of (provider, model_name)\n \"\"\"\n return get_model_provider(llm), get_model_name(llm)\n"
},
{
"path": "anytool/utils/ui.py",
"content": "\"\"\"\nAnyTool Terminal UI System\n\nProvides real-time CLI visualization for AnyTool execution flow.\nDisplays agent activities, grounding backends, and detailed logs.\n\nUses native ANSI colors and custom box drawing for a clean, lightweight interface.\n\"\"\"\n\nfrom typing import Optional, Dict, Any, List, Tuple\nfrom datetime import datetime\nfrom enum import Enum\nimport asyncio\nimport sys\nimport shutil\n\nfrom anytool.utils.display import Box, BoxStyle, colorize\n\n\nclass AgentStatus(Enum):\n \"\"\"Agent execution status\"\"\"\n IDLE = \"idle\"\n THINKING = \"thinking\"\n EXECUTING = \"executing\"\n WAITING = \"waiting\"\n\n\nclass AnyToolUI:\n \"\"\"\n AnyTool Terminal UI\n \n Provides real-time visualization of:\n - Agent activities and status\n - Grounding backend operations\n - Execution logs\n - System metrics\n \n Design Philosophy:\n - Lightweight and fast (no heavy dependencies)\n - Clean ANSI-based rendering\n - Minimal CPU overhead\n - Easy to customize\n \"\"\"\n \n def __init__(self, enable_live: bool = True, compact: bool = False):\n \"\"\"\n Initialize UI\n \n Args:\n enable_live: Whether to enable live display updates\n compact: Use compact layout (for smaller terminals)\n \"\"\"\n self.enable_live = enable_live\n self.compact = compact\n \n # Terminal dimensions\n self.term_width, self.term_height = self._get_terminal_size()\n \n # State tracking\n self.agent_status: Dict[str, AgentStatus] = {}\n self.agent_activities: Dict[str, List[str]] = {}\n self.grounding_operations: List[Dict[str, Any]] = []\n self.grounding_backends: List[Dict[str, Any]] = [] # Backend info (type, servers, etc.)\n self.log_buffer: List[Tuple[str, str, datetime]] = [] # (message, level, timestamp)\n \n # Metrics\n self.metrics: Dict[str, Any] = {\n \"start_time\": None,\n \"iterations\": 0,\n \"completed_tasks\": 0,\n \"llm_calls\": 0,\n \"grounding_calls\": 0,\n }\n \n # Live display state\n self._live_running = False\n self._live_task: Optional[asyncio.Task] = None\n self._last_render: List[str] = []\n \n def _get_terminal_size(self) -> Tuple[int, int]:\n \"\"\"Get terminal size\"\"\"\n try:\n size = shutil.get_terminal_size((80, 24))\n return size.columns, size.lines\n except:\n return 80, 24\n \n def _clear_screen(self):\n \"\"\"Clear screen\"\"\"\n if self.enable_live:\n # Clear entire screen and move cursor to top-left\n sys.stdout.write('\\033[2J\\033[H')\n sys.stdout.flush()\n \n def _move_cursor_home(self):\n \"\"\"Move cursor to home position\"\"\"\n sys.stdout.write('\\033[H')\n sys.stdout.flush()\n \n def _hide_cursor(self):\n \"\"\"Hide cursor\"\"\"\n sys.stdout.write('\\033[?25l')\n sys.stdout.flush()\n \n def _show_cursor(self):\n \"\"\"Show cursor\"\"\"\n sys.stdout.write('\\033[?25h')\n sys.stdout.flush()\n \n # Banner and Startup\n def print_banner(self):\n \"\"\"Print startup banner\"\"\"\n box = Box(width=70, style=BoxStyle.ROUNDED, color='c')\n \n print()\n print(box.top_line(indent=4))\n print(box.empty_line(indent=4))\n \n # Title\n title = colorize(\"AnyTool\", 'c', bold=True)\n print(box.text_line(title, align='center', indent=4, text_color=''))\n \n # Subtitle\n subtitle = \"Universal Tool-Use Layer for AI Agents\"\n print(box.text_line(subtitle, align='center', indent=4, text_color='gr'))\n \n print(box.empty_line(indent=4))\n print(box.bottom_line(indent=4))\n print()\n \n def print_initialization(self, steps: List[Tuple[str, str]]):\n \"\"\"\n Print initialization steps\n \n Args:\n steps: List of (component_name, status) tuples\n \"\"\"\n box = Box(width=70, style=BoxStyle.ROUNDED, color='bl')\n \n print(box.text_line(\"Initializing Components\", align='center', indent=4, text_color='c'))\n print(box.separator_line(indent=4))\n \n for component, status in steps:\n icon = colorize(\"โ\", 'g') if status == \"ok\" else colorize(\"โ\", 'rd')\n line = f\"{icon} {component}\"\n print(box.text_line(line, indent=4))\n \n print(box.bottom_line(indent=4))\n print()\n \n async def start_live_display(self):\n \"\"\"Start live display\"\"\"\n if not self.enable_live or self._live_running:\n return\n \n self._live_running = True\n self.metrics[\"start_time\"] = datetime.now()\n self._clear_screen()\n self._hide_cursor()\n \n # Start update loop\n self._live_task = asyncio.create_task(self._live_update_loop())\n \n async def stop_live_display(self):\n \"\"\"Stop live display\"\"\"\n if not self._live_running:\n return\n \n self._live_running = False\n \n if self._live_task:\n self._live_task.cancel()\n try:\n await self._live_task\n except asyncio.CancelledError:\n pass\n \n self._show_cursor()\n print() # Add newline after live display\n \n async def _live_update_loop(self):\n \"\"\"Live update loop\"\"\"\n while self._live_running:\n try:\n self.render()\n await asyncio.sleep(2.0)\n except asyncio.CancelledError:\n break\n except Exception as e:\n print(f\"UI render error: {e}\")\n \n def render(self):\n \"\"\"Render entire UI\"\"\"\n if not self.enable_live or not self._live_running:\n return\n \n # Clear and redraw\n self._clear_screen()\n \n lines = []\n \n # Header\n lines.extend(self._render_header())\n lines.append(\"\")\n \n # Stack all panels vertically\n lines.extend(self._render_agents())\n lines.append(\"\")\n lines.extend(self._render_grounding())\n lines.append(\"\")\n lines.extend(self._render_logs())\n \n output = \"\\n\".join(lines)\n sys.stdout.write(output)\n sys.stdout.flush()\n \n def update_display(self):\n \"\"\"Update display (alias for render())\"\"\"\n self.render()\n \n def _render_header(self) -> List[str]:\n \"\"\"Render header section\"\"\"\n lines = []\n \n # Calculate elapsed time\n elapsed = \"0s\"\n if self.metrics[\"start_time\"]:\n delta = datetime.now() - self.metrics[\"start_time\"]\n minutes = delta.seconds // 60\n seconds = delta.seconds % 60\n if minutes > 0:\n elapsed = f\"{minutes}m{seconds}s\"\n else:\n elapsed = f\"{seconds}s\"\n \n status_text = (\n f\"โถ {colorize('RUNNING', 'g')} | \"\n f\"Time: {colorize(elapsed, 'c')} | \"\n f\"Iter: {colorize(str(self.metrics['iterations']), 'y')} | \"\n f\"Tasks: {colorize(str(self.metrics['completed_tasks']), 'g')} | \"\n f\"LLM: {colorize(str(self.metrics['llm_calls']), 'bl')} | \"\n f\"Grounding: {colorize(str(self.metrics['grounding_calls']), 'm')}\"\n )\n \n lines.append(\" \" + status_text)\n lines.append(\" \" + \"โ\" * 60)\n \n return lines\n \n def _render_agents(self) -> List[str]:\n \"\"\"Render agents section\"\"\"\n lines = []\n \n lines.append(\" \" + colorize(\"ยง Agents\", 'c', bold=True))\n \n # Agent info\n agents = [\n (\"GroundingAgent\", 'c', self.agent_status.get(\"GroundingAgent\", AgentStatus.IDLE)),\n ]\n \n for agent_name, color, status in agents:\n # Status icon\n status_icons = {\n AgentStatus.IDLE: \"โ\",\n AgentStatus.THINKING: \"โ\",\n AgentStatus.EXECUTING: \"โ\",\n AgentStatus.WAITING: \"โท\",\n }\n icon = status_icons.get(status, \"โ\")\n \n # Recent activity\n activities = self.agent_activities.get(agent_name, [])\n activity = activities[-1][:40] if activities else \"idle\"\n \n # Format line\n line = f\" {colorize(icon, 'y')} {colorize(agent_name, color):<20s} {activity}\"\n lines.append(line)\n \n return lines\n \n \n def _render_grounding(self) -> List[str]:\n \"\"\"Render grounding operations section\"\"\"\n lines = []\n \n lines.append(\" \" + colorize(\"โ Grounding Backends\", 'c', bold=True))\n \n # Show backend types and servers\n if self.grounding_backends:\n for backend_info in self.grounding_backends:\n backend_name = backend_info.get(\"name\", \"unknown\")\n backend_type = backend_info.get(\"type\", \"unknown\")\n servers = backend_info.get(\"servers\", [])\n \n # Backend type icon\n type_icons = {\n \"gui\": \"โ \", \n \"shell\": \"$\", \n \"mcp\": \"โ\", \n \"system\": \"โ\", \n \"web\": \"โ\", \n }\n icon = type_icons.get(backend_type, \"โ\")\n \n # Format backend line\n if backend_type == \"mcp\" and servers:\n servers_str = \", \".join([s[:15] for s in servers])\n line = f\" {icon} {colorize(backend_name, 'y')} ({backend_type}): {colorize(servers_str, 'gr')}\"\n else:\n line = f\" {icon} {colorize(backend_name, 'y')} ({backend_type})\"\n \n lines.append(line)\n \n # Show last 3 operations\n recent_ops = self.grounding_operations[-3:] if self.grounding_operations else []\n \n if recent_ops:\n lines.append(\" \" + colorize(\"Recent Operations:\", 'gr'))\n for op in recent_ops:\n backend = op.get(\"backend\", \"unknown\")\n action = op.get(\"action\", \"unknown\")[:40]\n status = op.get(\"status\", \"pending\")\n \n # Status icon\n if status == \"success\":\n icon = colorize(\"โ\", 'g')\n elif status == \"pending\":\n icon = colorize(\"โณ\", 'y')\n else:\n icon = colorize(\"โ\", 'rd')\n \n line = f\" {icon} {colorize(backend, 'bl')}: {action}\"\n lines.append(line)\n \n return lines\n \n def _render_logs(self) -> List[str]:\n \"\"\"Render logs section\"\"\"\n lines = []\n \n lines.append(\" \" + colorize(\"โ Recent Events\", 'c', bold=True))\n \n # Show last 5 logs\n recent_logs = self.log_buffer[-5:] if self.log_buffer else []\n \n if recent_logs:\n for message, level, timestamp in recent_logs:\n time_str = timestamp.strftime(\"%H:%M:%S\")\n \n # Truncate long messages\n msg_display = message[:55]\n \n log_line = f\" {colorize(time_str, 'gr')} | {msg_display}\"\n lines.append(log_line)\n \n return lines\n \n \n def update_agent_status(self, agent_name: str, status: AgentStatus):\n \"\"\"Update agent status\"\"\"\n self.agent_status[agent_name] = status\n \n def add_agent_activity(self, agent_name: str, activity: str):\n \"\"\"Add agent activity\"\"\"\n if agent_name not in self.agent_activities:\n self.agent_activities[agent_name] = []\n \n self.agent_activities[agent_name].append(activity)\n \n # Keep only last 10 activities\n if len(self.agent_activities[agent_name]) > 10:\n self.agent_activities[agent_name] = self.agent_activities[agent_name][-10:]\n \n def update_grounding_backends(self, backends: List[Dict[str, Any]]):\n \"\"\"\n Update grounding backends information\n \n Args:\n backends: List of backend info dicts with keys:\n - name: backend name\n - type: backend type (gui, shell, mcp, system, web)\n - servers: list of server names (for mcp)\n \"\"\"\n self.grounding_backends = backends\n \n def add_grounding_operation(self, backend: str, action: str, status: str = \"pending\"):\n \"\"\"Add grounding operation\"\"\"\n self.grounding_operations.append({\n \"backend\": backend,\n \"action\": action,\n \"status\": status,\n \"timestamp\": datetime.now(),\n })\n \n self.metrics[\"grounding_calls\"] += 1\n \n def add_log(self, message: str, level: str = \"info\"):\n \"\"\"Add log message\"\"\"\n self.log_buffer.append((message, level, datetime.now()))\n \n # Keep only last 100 logs\n if len(self.log_buffer) > 100:\n self.log_buffer = self.log_buffer[-100:]\n \n def update_metrics(self, **kwargs):\n \"\"\"Update metrics\"\"\"\n self.metrics.update(kwargs)\n \n def print_summary(self, result: Dict[str, Any]):\n \"\"\"Print execution summary\"\"\"\n box = Box(width=70, style=BoxStyle.ROUNDED, color='c')\n \n print()\n print(box.text_line(colorize(\"โ Execution Summary\", 'c', bold=True), align='center', indent=4, text_color=''))\n print(box.separator_line(indent=4))\n \n # Status\n status = result.get(\"status\", \"unknown\")\n status_display = {\n \"completed\": colorize(\"COMPLETED\", 'g', bold=True),\n \"timeout\": colorize(\"TIMEOUT\", 'y', bold=True),\n \"error\": colorize(\"ERROR\", 'rd', bold=True),\n }\n status_text = status_display.get(status, status)\n \n print(box.text_line(f\" Status: {status_text}\", indent=4, text_color=''))\n print(box.text_line(f\" Execution Time: {colorize(f'{result.get('execution_time', 0):.2f}s', 'c')}\", indent=4, text_color=''))\n print(box.text_line(f\" Iterations: {colorize(str(result.get('iterations', 0)), 'y')}\", indent=4, text_color=''))\n print(box.text_line(f\" Completed Tasks: {colorize(str(result.get('completed_tasks', 0)), 'g')}\", indent=4, text_color=''))\n \n if result.get('evaluation_results'):\n print(box.text_line(f\" Evaluations: {colorize(str(len(result['evaluation_results'])), 'bl')}\", indent=4, text_color=''))\n \n print(box.bottom_line(indent=4))\n print()\n\n\ndef create_ui(enable_live: bool = True, compact: bool = False) -> AnyToolUI:\n \"\"\"\n Create AnyTool UI instance\n \n Args:\n enable_live: Whether to enable live display updates\n compact: Use compact layout for smaller terminals\n \"\"\"\n return AnyToolUI(enable_live=enable_live, compact=compact)"
},
{
"path": "anytool/utils/ui_integration.py",
"content": "\"\"\"\nAnyTool UI Integration\n\nIntegrates the UI system with AnyTool core components.\nProvides hooks and callbacks to update UI in real-time.\n\"\"\"\n\nimport asyncio\nfrom typing import Optional\n\nfrom anytool.utils.ui import AnyToolUI, AgentStatus\nfrom anytool.utils.logging import Logger\n\nlogger = Logger.get_logger(__name__)\n\n\nclass UIIntegration:\n \"\"\"\n UI Integration for AnyTool\n \n Connects AnyTool components with the UI system to provide real-time\n visualization of agent activities and execution flow.\n \"\"\"\n \n def __init__(self, ui: AnyToolUI):\n \"\"\"\n Initialize UI integration\n \n Args:\n ui: AnyToolUI instance\n \"\"\"\n self.ui = ui\n self._update_task: Optional[asyncio.Task] = None\n self._running = False\n \n # Tracked components\n self._llm_client = None\n self._grounding_client = None\n \n def attach_llm_client(self, llm_client):\n \"\"\"\n Attach LLM client\n \n Args:\n llm_client: LLMClient instance\n \"\"\"\n self._llm_client = llm_client\n logger.debug(\"UI attached to LLMClient\")\n \n def attach_grounding_client(self, grounding_client):\n \"\"\"\n Attach grounding client\n \n Args:\n grounding_client: GroundingClient instance\n \"\"\"\n self._grounding_client = grounding_client\n logger.debug(\"UI attached to GroundingClient\")\n \n async def start_monitoring(self, poll_interval: float = 0.5):\n \"\"\"\n Start monitoring and updating UI\n \n Args:\n poll_interval: Update interval in seconds\n \"\"\"\n if self._running:\n logger.warning(\"UI monitoring already running\")\n return\n \n self._running = True\n \n # Immediately update UI once before starting the loop\n await self._update_ui()\n \n self._update_task = asyncio.create_task(\n self._monitor_loop(poll_interval)\n )\n logger.info(\"UI monitoring started\")\n \n async def stop_monitoring(self):\n \"\"\"Stop monitoring\"\"\"\n if not self._running:\n return\n \n self._running = False\n \n if self._update_task:\n self._update_task.cancel()\n try:\n await self._update_task\n except asyncio.CancelledError:\n pass\n \n logger.info(\"UI monitoring stopped\")\n \n async def _monitor_loop(self, poll_interval: float):\n \"\"\"\n Main monitoring loop\n \n Args:\n poll_interval: Update interval in seconds\n \"\"\"\n while self._running:\n try:\n await self._update_ui()\n await asyncio.sleep(poll_interval)\n except asyncio.CancelledError:\n break\n except Exception as e:\n logger.error(f\"UI update error: {e}\", exc_info=True)\n \n async def _update_ui(self):\n \"\"\"Update UI with current state\"\"\"\n # Update grounding backends info\n if self._grounding_client:\n backends = []\n try:\n # Get list of providers\n providers = self._grounding_client.list_providers()\n \n for backend_type, provider in providers.items():\n backend_name = backend_type.value if hasattr(backend_type, 'value') else str(backend_type)\n \n backend_info = {\n \"name\": backend_name,\n \"type\": backend_name, # gui, shell, mcp, system, web\n \"servers\": []\n }\n \n # For MCP provider, get server names\n if backend_name == \"mcp\":\n try:\n # Try to get MCP sessions from provider\n if hasattr(provider, '_sessions'):\n backend_info[\"servers\"] = list(provider._sessions.keys())\n except Exception:\n pass\n \n backends.append(backend_info)\n \n self.ui.update_grounding_backends(backends)\n except Exception as e:\n logger.debug(f\"Failed to update grounding backends: {e}\")\n \n # Refresh display\n self.ui.update_display()\n \n # Event handlers - to be called by agents\n \n def on_agent_start(self, agent_name: str, activity: str):\n \"\"\"\n Called when agent starts an activity\n \n Args:\n agent_name: Agent name\n activity: Activity description\n \"\"\"\n self.ui.update_agent_status(agent_name, AgentStatus.EXECUTING)\n self.ui.add_agent_activity(agent_name, activity)\n self.ui.add_log(f\"{agent_name}: {activity}\", level=\"info\")\n \n def on_agent_thinking(self, agent_name: str):\n \"\"\"\n Called when agent is thinking\n \n Args:\n agent_name: Agent name\n \"\"\"\n self.ui.update_agent_status(agent_name, AgentStatus.THINKING)\n \n def on_agent_complete(self, agent_name: str, result: str = \"\"):\n \"\"\"\n Called when agent completes an activity\n \n Args:\n agent_name: Agent name\n result: Result description\n \"\"\"\n self.ui.update_agent_status(agent_name, AgentStatus.IDLE)\n if result:\n self.ui.add_log(f\"{agent_name}: {result}\", level=\"success\")\n \n def on_llm_call(self, model: str, prompt_length: int):\n \"\"\"\n Called when LLM is called\n \n Args:\n model: Model name\n prompt_length: Prompt length\n \"\"\"\n self.ui.update_metrics(\n llm_calls=self.ui.metrics.get(\"llm_calls\", 0) + 1\n )\n self.ui.add_log(f\"LLM call: {model} (prompt: {prompt_length} chars)\", level=\"debug\")\n \n def on_grounding_call(self, backend: str, action: str):\n \"\"\"\n Called when grounding backend is called\n \n Args:\n backend: Backend name\n action: Action description\n \"\"\"\n self.ui.add_grounding_operation(backend, action, status=\"pending\")\n self.ui.add_log(f\"Grounding [{backend}]: {action}\", level=\"info\")\n \n def on_grounding_complete(self, backend: str, action: str, success: bool):\n \"\"\"\n Called when grounding operation completes\n \n Args:\n backend: Backend name\n action: Action description\n success: Whether operation succeeded\n \"\"\"\n status = \"success\" if success else \"error\"\n \n # Update last operation status\n for op in reversed(self.ui.grounding_operations):\n if op[\"backend\"] == backend and op[\"action\"] == action and op[\"status\"] == \"pending\":\n op[\"status\"] = status\n break\n \n level = \"success\" if success else \"error\"\n result = \"โ\" if success else \"โ\"\n self.ui.add_log(f\"Grounding [{backend}]: {action} {result}\", level=level)\n \n \n def on_iteration(self, iteration: int):\n \"\"\"\n Called on each iteration\n \n Args:\n iteration: Iteration number\n \"\"\"\n self.ui.update_metrics(iterations=iteration)\n \n def on_error(self, message: str):\n \"\"\"\n Called when an error occurs\n \n Args:\n message: Error message\n \"\"\"\n self.ui.add_log(f\"ERROR: {message}\", level=\"error\")\n\n\nclass UILoggingHandler:\n \"\"\"\n Logging handler that forwards logs to UI\n \"\"\"\n \n def __init__(self, ui: AnyToolUI):\n \"\"\"\n Initialize logging handler\n \n Args:\n ui: AnyToolUI instance\n \"\"\"\n self.ui = ui\n \n def emit(self, record):\n \"\"\"\n Emit a log record to UI\n \n Args:\n record: Log record\n \"\"\"\n level_map = {\n \"DEBUG\": \"debug\",\n \"INFO\": \"info\",\n \"WARNING\": \"warning\",\n \"ERROR\": \"error\",\n \"CRITICAL\": \"error\",\n }\n \n level = level_map.get(record.levelname, \"info\")\n message = record.getMessage()\n \n # Filter out noisy logs\n if any(skip in message.lower() for skip in [\"processing card\", \"workflow poll\"]):\n return\n \n self.ui.add_log(message, level=level)\n\n\ndef create_integration(ui: AnyToolUI) -> UIIntegration:\n \"\"\"\n Create UI integration instance\n \n Args:\n ui: AnyToolUI instance\n \n Returns:\n UIIntegration instance\n \"\"\"\n return UIIntegration(ui)"
},
{
"path": "pyproject.toml",
"content": "[build-system]\nrequires = [\"setuptools>=68.0\", \"wheel\"]\nbuild-backend = \"setuptools.build_meta\"\n\n[project]\nname = \"anytool\"\nversion = \"0.1.0\"\ndescription = \"AnyTool: Universal Tool-Use Layer for AI Agents\"\nreadme = \"README.md\"\nrequires-python = \">=3.10\"\nlicense = {text = \"MIT\"}\nauthors = [\n {name = \"lingruixu@HKUDS\", email = \"lingruixu.db@gmail.com\"}\n]\n\ndependencies = [\n \"litellm>=1.70.0\",\n \"python-dotenv>=1.0.0\",\n \"openai>=1.0.0\",\n \"jsonschema>=4.25.0\",\n \"mcp>=1.0.0\",\n \"anthropic>=0.71.0\",\n \"pillow>=12.0.0\",\n \"flask>=3.1.0\",\n \"pyautogui>=0.9.54\",\n \"pydantic>=2.12.0\",\n \"requests>=2.32.0\",\n]\n\n[project.optional-dependencies]\nmacos = [\n \"pyobjc-core>=12.0\",\n \"pyobjc-framework-cocoa>=12.0\",\n \"pyobjc-framework-quartz>=12.0\",\n \"atomacos>=3.2.0\",\n]\n\nlinux = [\n \"python-xlib>=0.33\",\n \"pyatspi>=2.38.0\",\n \"numpy>=1.24.0\",\n]\n\nwindows = [\n \"pywinauto>=0.6.8\",\n \"pywin32>=306\",\n \"PyGetWindow>=0.0.9\",\n]\n\ndev = [\n \"pytest>=7.0.0\",\n \"pytest-asyncio>=0.21.0\",\n \"black>=23.0.0\",\n \"flake8>=6.0.0\",\n \"mypy>=1.0.0\",\n]\n\nall = [\n \"anytool[macos,linux,windows,dev]\",\n]\n\n[project.urls]\nRepository = \"https://github.com/HKUDS/AnyTool\"\n\"Bug Tracker\" = \"https://github.com/HKUDS/AnyTool/issues\"\n\n[project.scripts]\nanytool = \"anytool.__main__:run_main\"\nanytool-server = \"anytool.local_server.main:main\"\n\n[tool.setuptools]\npackages = {find = {where = [\".\"], include = [\"anytool*\"]}}\n\n[tool.setuptools.package-data]\nanytool = [\n \"config/*.json\",\n \"config/*.json.example\",\n \"local_server/config.json\",\n \"local_server/README.md\",\n]\n"
},
{
"path": "requirements.txt",
"content": "# AnyTool core dependencies\nlitellm>=1.70.0\npython-dotenv>=1.0.0\nopenai>=1.0.0\njsonschema>=4.25.0\nmcp>=1.0.0\nanthropic>=0.71.0\npillow>=12.0.0\ncolorama\n\n# Local server dependencies (cross-platform)\nflask>=3.1.0\npyautogui>=0.9.54\npydantic>=2.12.0\nrequests>=2.32.0\n\n# # macOS-specific dependencies (local server)\n# pyobjc-core>=12.0; sys_platform == 'darwin'\n# pyobjc-framework-cocoa>=12.0; sys_platform == 'darwin'\n# pyobjc-framework-quartz>=12.0; sys_platform == 'darwin'\n# atomacos>=3.2.0; sys_platform == 'darwin'\n\n# # Linux-specific dependencies (local server)\n# python-xlib>=0.33; sys_platform == 'linux'\n# pyatspi>=2.38.0; sys_platform == 'linux'\n# numpy>=1.24.0; sys_platform == 'linux'\n\n# # Windows-specific dependencies (local server)\n# pywinauto>=0.6.8; sys_platform == 'win32'\n# pywin32>=306; sys_platform == 'win32'\n# PyGetWindow>=0.0.9; sys_platform == 'win32'"
}
]