Showing preview only (1,130K chars total). Download the full file or copy to clipboard to get everything.
Repository: HKUDS/AnyTool
Branch: main
Commit: 506430fec133
Files: 132
Total size: 1.1 MB
Directory structure:
gitextract_sqfmt1l8/
├── .gitignore
├── COMMUNICATION.md
├── LICENSE
├── README.md
├── anytool/
│ ├── __init__.py
│ ├── __main__.py
│ ├── agents/
│ │ ├── __init__.py
│ │ ├── base.py
│ │ └── grounding_agent.py
│ ├── config/
│ │ ├── __init__.py
│ │ ├── config_agents.json
│ │ ├── config_dev.json.example
│ │ ├── config_grounding.json
│ │ ├── config_mcp.json.example
│ │ ├── config_security.json
│ │ ├── constants.py
│ │ ├── grounding.py
│ │ ├── loader.py
│ │ └── utils.py
│ ├── grounding/
│ │ ├── backends/
│ │ │ ├── __init__.py
│ │ │ ├── gui/
│ │ │ │ ├── __init__.py
│ │ │ │ ├── anthropic_client.py
│ │ │ │ ├── anthropic_utils.py
│ │ │ │ ├── config.py
│ │ │ │ ├── provider.py
│ │ │ │ ├── session.py
│ │ │ │ ├── tool.py
│ │ │ │ └── transport/
│ │ │ │ ├── actions.py
│ │ │ │ ├── connector.py
│ │ │ │ └── local_connector.py
│ │ │ ├── mcp/
│ │ │ │ ├── __init__.py
│ │ │ │ ├── client.py
│ │ │ │ ├── config.py
│ │ │ │ ├── installer.py
│ │ │ │ ├── provider.py
│ │ │ │ ├── session.py
│ │ │ │ ├── tool_cache.py
│ │ │ │ ├── tool_converter.py
│ │ │ │ └── transport/
│ │ │ │ ├── connectors/
│ │ │ │ │ ├── __init__.py
│ │ │ │ │ ├── base.py
│ │ │ │ │ ├── http.py
│ │ │ │ │ ├── sandbox.py
│ │ │ │ │ ├── stdio.py
│ │ │ │ │ ├── utils.py
│ │ │ │ │ └── websocket.py
│ │ │ │ └── task_managers/
│ │ │ │ ├── __init__.py
│ │ │ │ ├── sse.py
│ │ │ │ ├── stdio.py
│ │ │ │ ├── streamable_http.py
│ │ │ │ └── websocket.py
│ │ │ ├── shell/
│ │ │ │ ├── __init__.py
│ │ │ │ ├── provider.py
│ │ │ │ ├── session.py
│ │ │ │ └── transport/
│ │ │ │ ├── connector.py
│ │ │ │ └── local_connector.py
│ │ │ └── web/
│ │ │ ├── __init__.py
│ │ │ ├── provider.py
│ │ │ └── session.py
│ │ └── core/
│ │ ├── exceptions.py
│ │ ├── grounding_client.py
│ │ ├── provider.py
│ │ ├── quality/
│ │ │ ├── __init__.py
│ │ │ ├── manager.py
│ │ │ ├── store.py
│ │ │ └── types.py
│ │ ├── search_tools.py
│ │ ├── security/
│ │ │ ├── __init__.py
│ │ │ ├── e2b_sandbox.py
│ │ │ ├── policies.py
│ │ │ └── sandbox.py
│ │ ├── session.py
│ │ ├── system/
│ │ │ ├── __init__.py
│ │ │ ├── provider.py
│ │ │ └── tool.py
│ │ ├── tool/
│ │ │ ├── __init__.py
│ │ │ ├── base.py
│ │ │ ├── local_tool.py
│ │ │ └── remote_tool.py
│ │ ├── transport/
│ │ │ ├── connectors/
│ │ │ │ ├── __init__.py
│ │ │ │ ├── aiohttp_connector.py
│ │ │ │ └── base.py
│ │ │ └── task_managers/
│ │ │ ├── __init__.py
│ │ │ ├── aiohttp_connection_manager.py
│ │ │ ├── async_ctx.py
│ │ │ ├── base.py
│ │ │ ├── noop.py
│ │ │ └── placeholder.py
│ │ └── types.py
│ ├── llm/
│ │ ├── __init__.py
│ │ └── client.py
│ ├── local_server/
│ │ ├── README.md
│ │ ├── __init__.py
│ │ ├── config.json
│ │ ├── feature_checker.py
│ │ ├── health_checker.py
│ │ ├── main.py
│ │ ├── platform_adapters/
│ │ │ ├── __init__.py
│ │ │ ├── linux_adapter.py
│ │ │ ├── macos_adapter.py
│ │ │ ├── pyxcursor.py
│ │ │ └── windows_adapter.py
│ │ ├── requirements.txt
│ │ ├── run.sh
│ │ └── utils/
│ │ ├── __init__.py
│ │ ├── accessibility.py
│ │ └── screenshot.py
│ ├── platform/
│ │ ├── __init__.py
│ │ ├── config.py
│ │ ├── recording.py
│ │ ├── screenshot.py
│ │ └── system_info.py
│ ├── prompts/
│ │ ├── __init__.py
│ │ └── grounding_agent_prompts.py
│ ├── recording/
│ │ ├── __init__.py
│ │ ├── action_recorder.py
│ │ ├── manager.py
│ │ ├── recorder.py
│ │ ├── utils.py
│ │ ├── video.py
│ │ └── viewer.py
│ ├── tool_layer.py
│ └── utils/
│ ├── cli_display.py
│ ├── display.py
│ ├── logging.py
│ ├── telemetry/
│ │ ├── __init__.py
│ │ ├── events.py
│ │ ├── telemetry.py
│ │ └── utils.py
│ ├── ui.py
│ └── ui_integration.py
├── pyproject.toml
└── requirements.txt
================================================
FILE CONTENTS
================================================
================================================
FILE: .gitignore
================================================
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class
# OS files
.DS_Store
.DS_Store?
._*
.Spotlight-V100
.Trashes
ehthumbs.db
Thumbs.db
Desktop.ini
# IDE files
.vscode/
.idea/
.pytest_cache/
# Distribution / packaging
dist/
build/
*.egg-info/
*.egg
# Environment files
.env
# MCP files
anytool/config/config_mcp.json
# Logs
logs/
# Embedding cache
.anytool/
embedding_cache/
tool_quality/
# MCP tool cache
mcp_tool_cache.json
mcp_tool_cache_sanitized.json
# Config files
anytool/config/config_dev.json
# LLM keys
anytool/llm/remote_client/
# Local server temp files
anytool/local_server/temp/
examples/
================================================
FILE: COMMUNICATION.md
================================================
We provide QR codes for joining the HKUDS discussion groups on **WeChat** and **Feishu**.
You can join by scanning the QR codes below:
<img src="https://github.com/HKUDS/.github/blob/main/profile/QR.png" alt="WeChat QR Code" width="400"/>
================================================
FILE: LICENSE
================================================
MIT License
Copyright (c) 2025 HKUDS
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
================================================
FILE: README.md
================================================
<div align="center">
<picture>
<img src="assets/AnyTool_logo.png" width="800px" style="border: none; box-shadow: none;" alt="AnyTool Logo">
</picture>
## AnyTool: Universal Tool-Use Layer for AI Agents
### ✨ **One Line of Code to Supercharge any Agent with <br>Fast, Scalable and Powerful Tool Use** ✨
[](https://github.com/HKUDS/AnyTool/)
[](https://www.python.org/)
[](https://opensource.org/licenses/MIT/)
[](./COMMUNICATION.md)
[](./COMMUNICATION.md)
| ⚡ **Fast - Lightning Tool Retrieval** | 📈 **Self-Evolving Tool Orchestration** | ⚡ **Universal Tool Automation** |
</div>
## 🎯 What is AnyTool?
AnyTool is a **Universal Tool-Use Layer** that transforms how AI agents interact with tools. It solves three fundamental challenges that prevent reliable agent automation: **overwhelming tool contexts**, **unreliable community tools**, and **limited capability coverage** -- delivering the first truly intelligent tool orchestration system for production AI agents.
## 💡 Research Highlights
⚡ **Fast - Lightning Tool Retrieval**
- **Smart Context Management**: Progressive tool filtering delivers exact tools in milliseconds through multi-stage pipeline, eliminating context pollution while maintaining speed.
- **Zero-Waste Processing**: Pre-computed embeddings and lazy initialization eliminate redundant processing - tools are instantly ready across all executions.
📈 **Scalable - Self-Evolving Tool Orchestration**
- **Adaptive MCP Tool Selection**: Smart caching and selective re-indexing maintain constant performance from 10 to 10,000 tools with optimal resource usage.
- **Self-Evolving Tool Optimization**: System continuously improves through persistent memory, becoming more efficient as your tool ecosystem expands.
🌍 **Powerful - Universal Tool Automation**
- **Quality-Aware Selection**: Built-in reliability tracking and safety controls deliver production-ready automation through persistent learning and execution safeguards.
- **Universal Tool-Use Capability**: Multi-backend architecture seamlessly extends beyond web APIs to system operations, GUI automation, and deep research through unified interface.
## ⚡ Easy-to-Use and Effortless Integration
One line to get intelligent tool orchestration. Zero-config setup transforms complex multi-tool workflows into a single API call.
```python
from anytool import AnyTool
# One line to get intelligent tool orchestration
async with AnyTool() as tool_layer:
result = await tool_layer.execute(
"Research trending AI coding tools from GitHub and tech news, "
"collect their features and user feedback, analyze adoption patterns, "
"then create a comparison report with insights"
)
```
---
## 📋 Table of Contents
- [🎯 Quick Start](#-quick-start)
- [🚀 Technical Innovation & Implementation](#-technical-implementation)
- [🔧 Configuration Guide](#-configuration-guide)
- [📖 Code Structure](#-code-structure)
- [🔗 Related Projects](#-related-projects)
---
## 🎯 Quick Start
### 1. Environment Setup
```bash
# Clone repository
git clone https://github.com/HKUDS/AnyTool.git
cd AnyTool
# Create and activate conda environment (includes ffmpeg for video recording)
conda create -n anytool python=3.12 ffmpeg -c conda-forge -y
conda activate anytool
# Install dependencies
pip install -r requirements.txt
```
> [!NOTE]
> Create a `.env` file and add your API keys (refer to `anytool/.env.example`).
### 2. Execution Mode: Local vs Server
AnyTool's Shell and GUI backends support two execution modes. You can configure the mode in `anytool/config/config_grounding.json`:
```jsonc
{
"shell": { "mode": "local", ... }, // or "server"
"gui": { "mode": "local", ... } // or "server"
}
```
#### Local Mode (Default — no server needed)
In **local mode**, Shell and GUI operations are executed directly in-process via `subprocess` / `asyncio`. This is the simplest setup — **no local server required**. Just use AnyTool as normal, see [Quick Integration](#3-quick-integration) for usage examples.
> [!TIP]
> **Use local mode when** you are running AnyTool on the same machine you want to control (your own laptop / desktop). This is the recommended mode for most users.
#### Server Mode (for remote VMs / isolation)
In **server mode**, Shell and GUI operations are sent over HTTP to a running `local_server` Flask service. This is required when:
- **Controlling a remote VM** — the agent runs on your host, while the server runs inside the VM.
- **Process isolation / sandboxing** — you want script execution in a separate process for security or stability.
- **Multi-machine deployments** — the agent and the execution environment are on different machines.
To use server mode, set `"mode": "server"` in `config_grounding.json`, then install platform-specific dependencies and start the server:
> [!IMPORTANT]
> **Platform-specific setup required**: Different operating systems need different dependencies for desktop control. Please install the required dependencies for your OS before starting the local server:
<details>
<summary><b>macOS Setup</b></summary>
```bash
# Install macOS-specific dependencies
pip install pyobjc-core pyobjc-framework-cocoa pyobjc-framework-quartz atomacos
```
**Permissions Required**: macOS will automatically prompt for permissions when you first run the local server. Grant the following:
- **Accessibility** (for GUI control)
- **Screen Recording** (for screenshots and video capture)
> If prompts don't appear, manually grant permissions in System Settings → Privacy & Security.
</details>
<details>
<summary><b>Linux Setup</b></summary>
```bash
# Install Linux-specific dependencies
pip install python-xlib pyatspi numpy
# Install system packages
sudo apt install at-spi2-core python3-tk scrot
```
> [!NOTE]
> **Optional dependencies:**
> - Accessibility: `pyatspi` + `at-spi2-core`
> - Window management: `wmctrl`
> - Cursor in screenshots: `libx11-dev` + `libxfixes-dev`
</details>
<details>
<summary><b>Windows Setup</b></summary>
```bash
# Install Windows-specific dependencies
pip install pywinauto pywin32 PyGetWindow
```
</details>
After installing the platform-specific dependencies, start the local server:
```bash
python -m anytool.local_server.main
```
> [!NOTE]
> See [`anytool/local_server/README.md`](anytool/local_server/README.md) for complete API documentation and advanced configuration.
#### Mode Comparison
| | Local Mode (`"local"`) | Server Mode (`"server"`) |
|---|---|---|
| **Setup** | Zero — just run your agent | Start `local_server` first |
| **Use case** | Same-machine development | Remote VMs, sandboxing, multi-machine |
| **Shell execution** | `asyncio.subprocess` in-process | HTTP → Flask → `subprocess` |
| **GUI execution** | Direct pyautogui / ScreenshotHelper | HTTP → Flask → pyautogui |
| **Dependencies** | Only core AnyTool | Core + Flask + platform deps |
| **Network** | None required | HTTP between agent ↔ server |
### 3. Quick Integration
AnyTool is a **plug-and-play Universal Tool-Use Layer** for any AI agent. The task passed to `execute()` can come from your agent's planning module, user input, or any workflow system.
```python
import asyncio
from anytool import AnyTool
from anytool.tool_layer import AnyToolConfig
async def main():
config = AnyToolConfig(
enable_recording=True,
recording_backends=["gui", "shell", "mcp", "web"],
enable_screenshot=True,
enable_video=True,
)
async with AnyTool(config=config) as tool_layer:
result = await tool_layer.execute(
"Research trending AI coding tools from GitHub and tech news, "
"collect their features and user feedback, analyze adoption patterns, "
"then create a comparison report with insights"
)
print(result["response"])
asyncio.run(main())
```
> [!TIP]
> **MCP Server Configuration**: For tasks requiring specific tools, add relevant MCP servers to `anytool/config/config_mcp.json`. Unsure which servers to add? Simply add all potentially useful ones, AnyTool's Smart Tool RAG will automatically select the appropriate tools for your task. See [MCP Configuration](#mcp-configuration) for details.
---
## Technical Innovation & Implementation
### 🧩 Challenge 1: MCP Tool Context Overload
**The Problem**. Current MCP agents suffer from a fundamental design flaw: they load ALL configured servers and tools at every execution step, creating an overwhelming action space, creates three critical issues:
- ⚡ **Slow Performance with Massive Context Loading**<br>
Complete tool set from all pre-configured servers loaded simultaneously at every step, degrading execution speed
- 🎯 **Poor Accuracy from Blind Tool Setup**<br>
Users cannot preview tools before connecting, leading to over-setup "just in case" and confusing tool selection
- 💸 **Resource Waste with No Memory**<br>
Same tools reloaded at every execution step with no caching, causing redundant loading
### ✅ AnyTool's Solution: Tool Context Management Framework
**Motivation**: "Load Everything" → "Retrieve What's Needed"<br>
**Improvement**: Faster tool selection, cleaner context, and efficient resource usage through smart retrieval and memory.
#### **Technical Innovation**:<br>
**🎯 Multi-Stage Tool Retrieval Pipeline**
- **Progressive MCP Tool Filtering**: server selection → tool name matching → tool semantic search → LLM ranking
- **Reduces MCP Tool Search Space**: Each stage narrows down candidate tools for optimizing precision and speed
**💾 Long-Term Tool Memory**
- **Save Once, Use Forever**: Pre-compute tool embeddings once and save them to disk for instant reuse
- **Zero Waste Processing**: No more redundant processing - tools are ready to use immediately across all execution steps
**🧠 Adaptive Tool Selection**
- **Adaptive MCP Tool Ranking**: LLM-based tool selection refinement triggered only when MCP tool results are large or ambiguous
- **Tool Selection Efficiency**: Balances MCP tool accuracy with computational efficiency
**🚀 On-Demand Resource Management**
- **Lazy MCP Server Startup**: MCP server initialization triggered only when specific tools are needed
- **Selective Tool Updates**: Incremental re-indexing of only changed MCP tools, not the entire tool set
---
### 🚨 Challenge 2: MCP Tool Quality Issues
**The Problem**. Current MCP servers suffer from community contribution challenges that create three critical issues:
- 🔍 **Poor Tool Descriptions**<br>
Misleading claims, non-existent advertised tools, and vague capability specifications lead to wrong tool selection.
- 📊 **No Reliability Signals**<br>
Cannot assess MCP tool quality before use, causing blind selection decisions.
- ⚠️ **Security and Safety Gaps**<br>
Unvetted community tools may execute dangerous operations without proper safeguards.
### ✅ **AnyTool Solution: Self-Contained Quality Management**
**Motivation**: "Blind Tool Trust" → "Smart Quality Assessment"<br>
**Improvement**: Reliable tool selection, safe execution, and autonomous recovery through quality tracking and safety controls.
#### **Technical Innovation:**<br>
**🎯 Quality-Aware Tool Selection**
- **Description Quality Check**: LLM-based evaluation of MCP tool description clarity and completeness.
- **Performance-Based Ranking**: Track call/success rates for each MCP tool in persistent memory to prioritize reliable options.
**💾 Learning-Based Tool Memory**
- **Track Tool Performance**: Remember which MCP tools work well and which fail over time.
- **Smart Tool Prioritization**: Automatically rank tools based on past success rates and description quality.
**🛡️ Safety-First Execution**
- **Block Dangerous Operations**: Prevent arbitrary code execution and require user approval for sensitive MCP tool operations.
- **Execution Safeguards**: Built-in safety controls for all MCP tool executions.
**🚀 Self-Healing Tool Management**
- **Autonomous Tool Switching**: Switch failed MCP tools locally without restarting expensive planning loops.
- **Local Failure Recovery**: Automatically switch to alternative MCP tools on failure without escalating to upper-level agents.
---
### 🔄 Challenge 3: Limited MCP Capability Scope
**The Problem**. Current MCP ecosystem focuses primarily on Web APIs and online services, creating significant automation gaps that prevent comprehensive task completion:
- **🖥️ Missing System Operations**<br>
No native support for file manipulation, process management, or command execution on local systems.
- **🖱️ No Desktop Automation**<br>
Cannot control GUI applications that lack APIs, limiting automation to web-only scenarios.
- **📊 Incomplete Tool Coverage**<br>
Limited server categories in community and incomplete tool sets within existing servers create workflow bottlenecks.
### ✅ AnyTool Solution: Universal Capability Extension<br>(MCP + System Commands + GUI Control ≈ Universal Task Completion)
**Motivation**: "Web-Only MCP" → "Universal Task Completion"<br>
**Improvement**: Complete automation coverage through multi-backend architecture that seamlessly extends MCP capabilities beyond web APIs.
**🏗️ Multi-Backend Architecture**
- **MCP Backend**: Community servers for Web APIs and online services
- **Shell Backend**: Bash/Python execution for system-level operations and file management
- **GUI Backend**: Pixel-level automation for any visual application without API requirements
- **Web Backend**: Deep web research and data extraction capabilities
**💡 Self-Evolving Capability Discovery**
- **Intelligent Gap Detection**: Planning agent identifies when MCP tools are insufficient for task requirements
- **Automatic Backend Selection**: Shell/GUI backends automatically fill capability gaps without manual intervention
- **Dynamic Capability Expansion**: Previously impossible tasks become achievable through backend combination
**🎭 Unified Tool Orchestration**
- **Uniform Tool Schema**: All backends expose identical interface for seamless agent tool selection
- **Transparent Backend Switching**: Agents select optimal tools across backend types without knowing implementation details
- **Intelligent Tool Routing**: Automatic routing to the most appropriate backend based on task requirements
**🚀 Seamless Integration Layer**
- **Single Tool Interface**: Unified API that abstracts away backend complexity from AI agents.
- **Cross-Backend Coordination**: Enable complex workflows that span multiple backend capabilities.
- **Consistent Safety Controls**: Apply security and safety measures uniformly across all backend types.
---
## 🔧 Configuration Guide
### Configuration Overview
AnyTool uses a layered configuration system:
- **`config_dev.json`** (highest priority): Local development overrides. Overrides all other configurations.
- **`config_agents.json`**: Agent definitions and backend access control
- **`config_mcp.json`**: MCP server registry
- **`config_grounding.json`**: Backend-specific settings and Smart Tool RAG configuration
- **`config_security.json`**: Security policies with runtime user confirmation for sensitive operations
---
### Agent Configuration
**Path**: `anytool/config/config_agents.json`
**Purpose**: Define agent roles, control backend access scope, and set execution limits to prevent infinite loops.
**Example configuration**:
```json
{
"agents": [
{
"name": "GroundingAgent",
"class_name": "GroundingAgent",
"backend_scope": ["gui", "shell", "mcp", "system", "web"],
"max_iterations": 20
}
]
}
```
**Key Fields**:
| Field | Description | Options/Example |
|-------|-------------|-----------------|
| `backend_scope` | Accessible backends | `[]` or any combination of `["gui", "shell", "mcp", "system", "web"]` |
| `max_iterations` | Maximum execution cycles | Any integer (e.g., `15`, `20`, `50`) or `null` (unlimited) |
---
### MCP Configuration
**Path**: `anytool/config/config_mcp.json` (copy from `config_mcp.json.example`)
**Purpose**: Register MCP servers with connection details. AnyTool automatically discovers tools from all registered servers and makes them available through Smart Tool RAG.
**Example configuration**:
```json
{
"mcpServers": {
"github": {
"command": "npx",
"args": ["-y", "@modelcontextprotocol/server-github"],
"env": {
"GITHUB_PERSONAL_ACCESS_TOKEN": "${GITHUB_TOKEN}"
}
}
}
}
```
---
<details>
<summary><b>Runtime Configuration (AnyToolConfig)</b></summary>
### Runtime Configuration (AnyToolConfig)
**Complete example**:
```python
from anytool import AnyTool
from anytool.tool_layer import AnyToolConfig
config = AnyToolConfig(
# LLM Configuration
llm_model="anthropic/claude-sonnet-4-5",
llm_enable_thinking=False,
llm_timeout=120.0,
llm_max_retries=3,
llm_rate_limit_delay=0.0,
llm_kwargs={}, # Additional LiteLLM parameters
# Separate models for specific tasks (None = use llm_model)
tool_retrieval_model=None, # Model for tool retrieval LLM filter
visual_analysis_model=None, # Model for visual analysis
# Grounding Configuration
grounding_config_path=None, # Path to custom config file
grounding_max_iterations=20,
grounding_system_prompt=None, # Custom system prompt
# Backend Configuration
backend_scope=["gui", "shell", "mcp", "web", "system"],
# Workspace Configuration
workspace_dir=None, # Auto-create temp dir if None
# Recording Configuration
enable_recording=True,
recording_backends=["gui", "shell", "mcp"],
recording_log_dir="./logs/recordings",
enable_screenshot=True,
enable_video=True,
enable_conversation_log=True, # Save LLM conversations to conversations.jsonl
# Logging Configuration
log_level="INFO",
log_to_file=False,
log_file_path=None,
)
async with AnyTool(config=config) as tool_layer:
result = await tool_layer.execute("Your task here")
# Or with external task_id for benchmark integration:
# result = await tool_layer.execute("Your task", task_id="my-task-001")
```
</details>
---
<details>
<summary><b>Other Configuration Files</b></summary>
### Backend Configuration
**Path**: `anytool/config/config_grounding.json`
**Purpose**: Configure backend-specific behaviors, timeouts, Smart Tool RAG system for efficient tool selection, and Tool Quality Tracking for self-evolving tool intelligence.
**Key Fields**:
| Backend | Field | Description | Options/Default |
|---------|-------|-------------|-----------------|
| **shell** | `timeout` | Command timeout (seconds) | Any integer (default: `60`) |
| | `conda_env` | Auto-activate conda environment | Environment name or `null` (default: `"anytool"`) |
| | `working_dir` | Working directory for command execution | Any valid path (default: current directory) |
| | `default_shell` | Shell to use | `"/bin/bash"`, `"/bin/zsh"`, etc. |
| **gui** | `timeout` | Operation timeout (seconds) | Any integer (default: `90`) |
| | `screenshot_on_error` | Capture screenshot on failure | `true` or `false` (default: `true`) |
| | `driver_type` | GUI automation driver | `"pyautogui"` or other supported drivers |
| **mcp** | `timeout` | Request timeout (seconds) | Any integer (default: `30`) |
| | `sandbox` | Run in E2B sandbox | `true` or `false` (default: `false`) |
| | `eager_sessions` | Pre-connect all servers at startup | `true` or `false` (default: `false`, lazy connection) |
| **tool_search** | `search_mode` | Tool retrieval strategy | `"semantic"`, `"hybrid"` (semantic + LLM filter), or `"llm"` (default: `"hybrid"`) |
| | `max_tools` | Maximum tools to return from search | Any integer (default: `40`) |
| | `enable_llm_filter` | Enable LLM-based tool pre-filtering | `true` or `false` (default: `true`) |
| | `llm_filter_threshold` | Enable LLM filter when tools exceed this count | Any integer (default: `50`) |
| | `enable_cache_persistence` | Persist embedding cache to disk | `true` or `false` (default: `true`) |
| **tool_quality** | `enabled` | Enable tool quality tracking | `true` or `false` (default: `true`) |
| | `enable_persistence` | Persist quality data to disk | `true` or `false` (default: `true`) |
| | `cache_dir` | Directory for quality cache | Path string (default: `.anytool/tool_quality` in project directory) |
| | `auto_evaluate_descriptions` | Automatically evaluate tool descriptions using LLM | `true` or `false` (default: `true`) |
| | `enable_quality_ranking` | Incorporate quality scores in tool ranking | `true` or `false` (default: `true`) |
| | `evolve_interval` | Trigger self-evolution every N tool executions | Any integer 1-100 (default: `5`) |
---
### Security Configuration
**Path**: `anytool/config/config_security.json`
**Purpose**: Define security policies with command filtering and access control.
**Key Fields**:
| Section | Field | Description | Options |
|---------|-------|-------------|---------|
| **global** | `allow_shell_commands` | Enable shell command execution | `true` or `false` (default: `true`) |
| | `allow_network_access` | Enable network operations | `true` or `false` (default: `true`) |
| | `allow_file_access` | Enable file system operations | `true` or `false` (default: `true`) |
| | `blocked_commands` | Platform-specific command blacklist | Object with `common`, `linux`, `darwin`, `windows` arrays |
| | `sandbox_enabled` | Enable sandboxing for all operations | `true` or `false` (default: `false`) |
| **backend** | `shell`, `mcp`, `gui`, `web` | Per-backend security overrides | Same fields as global, backend-specific |
**Example blocked commands**: `rm -rf`, `shutdown`, `reboot`, `mkfs`, `dd`, `format`, `iptables`
**Behavior**:
- Blocked commands are **rejected automatically**
- Sandbox mode isolates operations in secure environments (E2B sandbox for MCP)
---
### Developer Configuration
**Path**: `anytool/config/config_dev.json` (copy from `config_dev.json.example`)
**Loading Priority**: `config_grounding.json` → `config_security.json` → `config_dev.json` (dev.json overrides the former ones)
</details>
---
## 📖 Code Structure
### 📖 Quick Overview
> **Legend**: ⚡ Core modules | 🔧 Supporting modules
```
AnyTool/
├── anytool/
│ ├── __init__.py # Package exports
│ ├── __main__.py # CLI entry point (python -m anytool)
│ ├── tool_layer.py # AnyTool main class
│ │
│ ├── ⚡ agents/ # Agent System
│ ├── ⚡ grounding/ # Unified Backend System
│ │ ├── core/ # Core abstractions
│ │ └── backends/ # Backend implementations
│ │ ├── shell/ # Shell command execution
│ │ ├── gui/ # Anthropic Computer Use
│ │ ├── mcp/ # Model Context Protocol
│ │ └── web/ # Web search & browsing
│ │
│ ├── 🔧 prompts/ # Prompt Templates
│ ├── 🔧 llm/ # LLM Integration
│ ├── 🔧 config/ # Configuration System
│ ├── 🔧 local_server/ # GUI Backend Server
│ ├── 🔧 recording/ # Execution Recording
│ ├── 🔧 platform/ # Platform Integration
│ └── 🔧 utils/ # Utilities
│
├── .anytool/ # Runtime cache
│ ├── embedding_cache/ # Tool embeddings for Smart Tool RAG
│ └── tool_quality/ # Persistent tool quality tracking data
│
├── logs/ # Execution logs
│
├── requirements.txt # Python dependencies
├── pyproject.toml # Package configuration
└── README.md
```
---
### 📂 Detailed Module Structure
<details open>
<summary><b>⚡ agents/</b> - Agent System</summary>
```
agents/
├── __init__.py
├── base.py # Base agent class with common functionality
└── grounding_agent.py # Execution Agent (tool calling & iteration control)
```
**Key Responsibilities**: Task execution with intelligent tool selection and iteration control.
</details>
<details open>
<summary><b>⚡ grounding/</b> - Unified Backend System (Core Integration Layer)</summary>
**Key Responsibilities**: Unified tool abstraction, backend routing, session pooling, Smart Tool RAG, and Self-Evolving Quality Tracking*.
#### Core Abstractions
```
grounding/core/
├── grounding_client.py # Unified interface across all backends
├── provider.py # Abstract provider base class
├── session.py # Session lifecycle management
├── search_tools.py # Smart Tool RAG for semantic search
├── exceptions.py # Custom exception definitions
├── types.py # Shared type definitions
│
├── tool/ # Tool abstraction layer
│ ├── base.py # Tool base class
│ ├── local_tool.py # Local tool implementation
│ └── remote_tool.py # Remote tool implementation
│
├── quality/ # Self-evolving tool quality tracking
│ ├── manager.py # Quality manager with adaptive ranking
│ ├── store.py # Persistent quality data storage
│ └── types.py # Quality record data types
│
├── security/ # Security & sandboxing 🔧
│ ├── policies.py # Security policy enforcement
│ ├── sandbox.py # Sandbox abstraction
│ └── e2b_sandbox.py # E2B sandbox integration
│
├── system/ # System-level provider
│ ├── provider.py
│ └── tool.py
│
└── transport/ # Transport layer abstractions 🔧
├── connectors/
│ ├── base.py
│ └── aiohttp_connector.py
└── task_managers/
├── base.py
├── async_ctx.py
├── aiohttp_connection_manager.py
└── placeholder.py
```
#### Backend Implementations
<details>
<summary><b>Shell Backend</b> - Command execution via local server</summary>
```
backends/shell/
├── provider.py # Shell provider implementation
├── session.py # Shell session management
└── transport/
└── connector.py # HTTP connector to local server
```
</details>
<details>
<summary><b>GUI Backend</b> - Anthropic Computer Use integration</summary>
```
backends/gui/
├── provider.py # GUI provider implementation
├── session.py # GUI session management
├── tool.py # GUI-specific tools
├── anthropic_client.py # Anthropic API client wrapper
├── anthropic_utils.py # Utility functions
├── config.py # GUI configuration
└── transport/
├── connector.py # Computer Use API connector
└── actions.py # Action execution logic
```
</details>
<details>
<summary><b>MCP Backend</b> - Model Context Protocol servers</summary>
```
backends/mcp/
├── provider.py # MCP provider implementation
├── session.py # MCP session management
├── client.py # MCP client
├── config.py # MCP configuration loader
├── installer.py # MCP server installer
├── tool_converter.py # Convert MCP tools to unified format
├── tool_cache.py # MCP tool cache for offline tool discovery
└── transport/
├── connectors/ # Multiple transport types
│ ├── base.py
│ ├── stdio.py # Standard I/O connector
│ ├── http.py # HTTP connector
│ ├── websocket.py # WebSocket connector
│ ├── sandbox.py # Sandboxed connector
│ └── utils.py
└── task_managers/ # Protocol-specific managers
├── stdio.py
├── sse.py
├── streamable_http.py
└── websocket.py
```
</details>
<details>
<summary><b>Web Backend</b> - Search and browsing</summary>
```
backends/web/
├── provider.py # Web provider implementation
└── session.py # Web session management
```
</details>
</details>
<details>
<summary><b>🔧 prompts/</b> - Prompt Templates</summary>
```
prompts/
├── __init__.py
└── grounding_agent_prompts.py # Grounding agent system & tool selection prompts
```
</details>
<details>
<summary><b>🔧 llm/</b> - LLM Integration</summary>
```
llm/
├── __init__.py
└── client.py # LiteLLM wrapper with retry logic
```
</details>
<details>
<summary><b>🔧 config/</b> - Configuration System</summary>
```
config/
├── __init__.py
├── loader.py # Configuration file loader
├── constants.py # System constants
├── grounding.py # Grounding configuration dataclasses
├── utils.py # Configuration utilities
│
├── config_grounding.json # Backend-specific settings
├── config_agents.json # Agent configurations
├── config_mcp.json.example # MCP server definitions (copy to config_mcp.json)
├── config_security.json # Security policies
└── config_dev.json.example # Development config template
```
</details>
<details>
<summary><b>🔧 local_server/</b> - GUI Backend Server</summary>
```
local_server/
├── __init__.py
├── main.py # Flask application entry point
├── config.json # Server configuration
├── feature_checker.py # Platform feature detection
├── health_checker.py # Server health monitoring
├── platform_adapters/ # OS-specific implementations
│ ├── macos_adapter.py # macOS automation (atomacos, pyobjc)
│ ├── linux_adapter.py # Linux automation (pyatspi, xlib)
│ ├── windows_adapter.py # Windows automation (pywinauto)
│ └── pyxcursor.py # Custom cursor handling
├── utils/
│ ├── accessibility.py # Accessibility tree utilities
│ └── screenshot.py # Screenshot capture
└── README.md
```
**Purpose**: Lightweight Flask service enabling computer control (GUI, Shell, Files, Screen capture).
</details>
<details>
<summary><b>🔧 recording/</b> - Execution Recording</summary>
```
recording/
├── __init__.py
├── recorder.py # Main recording manager
├── manager.py # Recording lifecycle management
├── action_recorder.py # Action-level logging
├── video.py # Video capture integration
├── viewer.py # Trajectory viewer and analyzer
└── utils.py # Recording utilities
```
**Purpose**: Execution audit with trajectory recording and video capture.
</details>
<details>
<summary><b>🔧 platform/</b> - Platform Integration</summary>
```
platform/
├── __init__.py
├── config.py # Platform-specific configuration
├── recording.py # Recording integration
├── screenshot.py # Screenshot utilities
└── system_info.py # System information gathering
```
</details>
<details>
<summary><b>🔧 utils/</b> - Shared Utilities</summary>
```
utils/
├── logging.py # Structured logging system
├── ui.py # Terminal UI components
├── display.py # Display formatting utilities
├── cli_display.py # CLI-specific display
├── ui_integration.py # UI integration helpers
└── telemetry/ # Usage analytics (opt-in)
├── __init__.py
├── events.py
├── telemetry.py
└── utils.py
```
</details>
<details>
<summary><b>📊 logs/</b> - Execution Logs & Recordings</summary>
```
logs/
├── <script_name>/ # Main application logs
│ └── anytool_YYYY-MM-DD_HH-MM-SS.log # Timestamped log files
│
└── recordings/ # Execution recordings
└── task_<id>/ # Individual recording session
├── trajectory.json # Complete execution trajectory
├── screenshots/ # Visual execution record (GUI backend)
│ ├── tool_<name>_<timestamp>.png
│ ├── tool_<name>_<timestamp>.png
│ └── ... # Sequential screenshots
├── workspace/ # Task workspace
│ └── [generated files] # Files created during execution
└── screen_recording.mp4 # Video recording (if enabled)
```
**Recording Control**: Enable via `AnyToolConfig(enable_recording=True)`, filter backends with `recording_backends=["gui", "shell", ...]`
</details>
---
## 🔗 Related Projects
AnyTool builds upon excellent open-source projects, we sincerely thank their authors and contributors:
- **[OSWorld](https://github.com/xlang-ai/OSWorld)**: Comprehensive benchmark for evaluating computer-use agents across diverse operating system tasks.
- **[mcp-use](https://github.com/mcp-use/mcp-use)**: Platform that simplifies MCP agent development with client SDKs.
---
<div align="center">
**🌟 If this project helps you, please give us a Star!**
**🤖 Empower AI Agent with intelligent tool orchestration!**
</div>
---
<p align="center">
<em> ❤️ Thanks for visiting ✨ AnyTool!</em><br><br>
<img src="https://visitor-badge.laobi.icu/badge?page_id=HKUDS.AnyTool&style=for-the-badge&color=00d4ff" alt="Views">
</p>
================================================
FILE: anytool/__init__.py
================================================
from importlib import import_module as _imp
from typing import Dict as _Dict, Any as _Any, TYPE_CHECKING as _TYPE_CHECKING
if _TYPE_CHECKING:
from anytool.tool_layer import AnyTool as AnyTool, AnyToolConfig as AnyToolConfig
from anytool.agents import GroundingAgent as GroundingAgent
from anytool.llm import LLMClient as LLMClient
from anytool.recording import RecordingManager as RecordingManager
__version__ = "0.1.0"
__all__ = [
# Version
"__version__",
# Main API
"AnyTool",
"AnyToolConfig",
# Core Components
"GroundingAgent",
"GroundingClient",
"LLMClient",
"BaseTool",
"ToolResult",
"BackendType",
# Recording System
"RecordingManager",
"RecordingViewer",
]
# Map attribute → sub-module that provides it
_attr_to_module: _Dict[str, str] = {
# Main API
"AnyTool": "anytool.tool_layer",
"AnyToolConfig": "anytool.tool_layer",
# Core Components
"GroundingAgent": "anytool.agents",
"GroundingClient": "anytool.grounding.core.grounding_client",
"LLMClient": "anytool.llm",
"BaseTool": "anytool.grounding.core.tool.base",
"ToolResult": "anytool.grounding.core.types",
"BackendType": "anytool.grounding.core.types",
# Recording System
"RecordingManager": "anytool.recording",
"RecordingViewer": "anytool.recording.viewer",
}
def __getattr__(name: str) -> _Any:
"""Dynamically import sub-modules on first attribute access.
This keeps the *initial* package import lightweight and avoids raising
`ModuleNotFoundError` for optional / heavy dependencies until the
corresponding functionality is explicitly used.
"""
if name not in _attr_to_module:
raise AttributeError(f"module 'anytool' has no attribute '{name}'")
module_name = _attr_to_module[name]
module = _imp(module_name)
value = getattr(module, name)
globals()[name] = value
return value
def __dir__():
return sorted(list(globals().keys()) + list(_attr_to_module.keys()))
================================================
FILE: anytool/__main__.py
================================================
import asyncio
import argparse
import sys
import logging
from typing import Optional
from anytool.tool_layer import AnyTool, AnyToolConfig
from anytool.utils.logging import Logger
from anytool.utils.ui import create_ui, AnyToolUI
from anytool.utils.ui_integration import UIIntegration
from anytool.utils.cli_display import CLIDisplay
from anytool.utils.display import colorize
logger = Logger.get_logger(__name__)
class UIManager:
def __init__(self, ui: Optional[AnyToolUI], ui_integration: Optional[UIIntegration]):
self.ui = ui
self.ui_integration = ui_integration
self._original_log_levels = {}
async def start_live_display(self):
if not self.ui or not self.ui_integration:
return
print()
print(colorize(" ▣ Starting real-time visualization...", 'c'))
print()
await asyncio.sleep(1)
self._suppress_logs()
await self.ui.start_live_display()
await self.ui_integration.start_monitoring(poll_interval=2.0)
async def stop_live_display(self):
if not self.ui or not self.ui_integration:
return
await self.ui_integration.stop_monitoring()
await self.ui.stop_live_display()
self._restore_logs()
def print_summary(self, result: dict):
if self.ui:
self.ui.print_summary(result)
else:
CLIDisplay.print_result_summary(result)
def _suppress_logs(self):
log_names = ["anytool", "anytool.grounding", "anytool.agents"]
for name in log_names:
log = logging.getLogger(name)
self._original_log_levels[name] = log.level
log.setLevel(logging.CRITICAL)
def _restore_logs(self):
for name, level in self._original_log_levels.items():
logging.getLogger(name).setLevel(level)
self._original_log_levels.clear()
async def _execute_task(anytool: AnyTool, query: str, ui_manager: UIManager):
await ui_manager.start_live_display()
result = await anytool.execute(query)
await ui_manager.stop_live_display()
ui_manager.print_summary(result)
return result
async def interactive_mode(anytool: AnyTool, ui_manager: UIManager):
CLIDisplay.print_interactive_header()
while True:
try:
prompt = colorize(">>> ", 'c', bold=True)
query = input(f"\n{prompt}").strip()
if not query:
continue
if query.lower() in ['exit', 'quit', 'q']:
print("\nExiting...")
break
if query.lower() == 'status':
_print_status(anytool)
continue
if query.lower() == 'help':
CLIDisplay.print_help()
continue
CLIDisplay.print_task_header(query)
await _execute_task(anytool, query, ui_manager)
except KeyboardInterrupt:
print("\n\nInterrupt signal detected, exiting...")
break
except Exception as e:
logger.error(f"Error: {e}", exc_info=True)
print(f"\nError: {e}")
async def single_query_mode(anytool: AnyTool, query: str, ui_manager: UIManager):
CLIDisplay.print_task_header(query, title="▶ Single Query Execution")
await _execute_task(anytool, query, ui_manager)
def _print_status(anytool: AnyTool):
"""Print system status"""
from anytool.utils.display import Box, BoxStyle
box = Box(width=70, style=BoxStyle.ROUNDED, color='bl')
print()
print(box.text_line(colorize("System Status", 'bl', bold=True),
align='center', indent=4, text_color=''))
print(box.separator_line(indent=4))
status_lines = [
f"Initialized: {colorize('Yes' if anytool.is_initialized() else 'No', 'g' if anytool.is_initialized() else 'rd')}",
f"Running: {colorize('Yes' if anytool.is_running() else 'No', 'y' if anytool.is_running() else 'g')}",
f"Model: {colorize(anytool.config.llm_model, 'c')}",
]
if anytool.is_initialized():
backends = anytool.list_backends()
status_lines.append(f"Backends: {colorize(', '.join(backends), 'c')}")
sessions = anytool.list_sessions()
status_lines.append(f"Active Sessions: {colorize(str(len(sessions)), 'y')}")
for line in status_lines:
print(box.text_line(f" {line}", indent=4, text_color=''))
print(box.bottom_line(indent=4))
print()
def _create_argument_parser() -> argparse.ArgumentParser:
"""Create command-line argument parser"""
parser = argparse.ArgumentParser(
description='AnyTool - Universal Tool-Use Layer for AI Agents',
formatter_class=argparse.RawDescriptionHelpFormatter,
)
w
# Subcommands
subparsers = parser.add_subparsers(dest='command', help='Available commands')
# refresh-cache subcommand
cache_parser = subparsers.add_parser(
'refresh-cache',
help='Refresh MCP tool cache (starts all servers once)'
)
cache_parser.add_argument(
'--config', '-c', type=str,
help='MCP configuration file path'
)
# Basic arguments (for run mode)
parser.add_argument('--config', '-c', type=str, help='Configuration file path (JSON format)')
parser.add_argument('--query', '-q', type=str, help='Single query mode: execute query directly')
# LLM arguments
parser.add_argument('--model', '-m', type=str, help='LLM model name')
# Logging arguments
parser.add_argument('--log-level', type=str, choices=['DEBUG', 'INFO', 'WARNING', 'ERROR'], help='Log level')
# Execution arguments
parser.add_argument('--max-iterations', type=int, help='Maximum iteration count')
parser.add_argument('--timeout', type=float, help='LLM API call timeout (seconds)')
# UI arguments
parser.add_argument('--interactive', '-i', action='store_true', help='Force interactive mode')
parser.add_argument('--no-ui', action='store_true', help='Disable visualization UI')
parser.add_argument('--ui-compact', action='store_true', help='Use compact UI layout')
return parser
async def refresh_mcp_cache(config_path: Optional[str] = None):
"""Refresh MCP tool cache by starting servers one by one and saving tool metadata."""
from anytool.grounding.backends.mcp import MCPProvider, get_tool_cache
from anytool.grounding.core.types import SessionConfig, BackendType
from anytool.config import load_config, get_config
print("Refreshing MCP tool cache...")
print("Servers will be started one by one (start -> get tools -> close).")
print()
# Load config
if config_path:
config = load_config(config_path)
else:
config = get_config()
# Get MCP config
mcp_config = getattr(config, 'mcp', None) or {}
if hasattr(mcp_config, 'model_dump'):
mcp_config = mcp_config.model_dump()
# Skip dependency checks for refresh-cache (servers are pre-validated)
mcp_config["check_dependencies"] = False
# Create provider
provider = MCPProvider(config=mcp_config)
await provider.initialize()
servers = provider.list_servers()
total = len(servers)
print(f"Found {total} MCP servers configured")
print()
cache = get_tool_cache()
cache.set_server_order(servers) # Preserve config order when saving
total_tools = 0
success_count = 0
skipped_count = 0
failed_servers = []
# Load existing cache to skip already processed servers
existing_cache = cache.get_all_tools()
# Timeout for each server (in seconds)
SERVER_TIMEOUT = 60
# Process servers one by one
for i, server_name in enumerate(servers, 1):
# Skip if already cached (resume support)
if server_name in existing_cache:
cached_tools = existing_cache[server_name]
total_tools += len(cached_tools)
skipped_count += 1
print(f"[{i}/{total}] {server_name}... ⏭ cached ({len(cached_tools)} tools)")
continue
print(f"[{i}/{total}] {server_name}...", end=" ", flush=True)
session_id = f"mcp-{server_name}"
try:
# Create session and get tools with timeout protection
async with asyncio.timeout(SERVER_TIMEOUT):
# Create session for this server
cfg = SessionConfig(
session_name=session_id,
backend_type=BackendType.MCP,
connection_params={"server": server_name},
)
session = await provider.create_session(cfg)
# Get tools from this server
tools = await session.list_tools()
# Convert to metadata format
tool_metadata = []
for tool in tools:
tool_metadata.append({
"name": tool.schema.name,
"description": tool.schema.description or "",
"parameters": tool.schema.parameters or {},
})
# Save to cache (incremental)
cache.save_server(server_name, tool_metadata)
# Close session immediately to free resources
await provider.close_session(session_id)
total_tools += len(tools)
success_count += 1
print(f"✓ {len(tools)} tools")
except asyncio.TimeoutError:
error_msg = f"Timeout after {SERVER_TIMEOUT}s"
failed_servers.append((server_name, error_msg))
print(f"✗ {error_msg}")
# Save failed server info to cache
cache.save_failed_server(server_name, error_msg)
# Try to close session if it was created
try:
await provider.close_session(session_id)
except Exception:
pass
except Exception as e:
error_msg = str(e)
failed_servers.append((server_name, error_msg))
print(f"✗ {error_msg[:50]}")
# Save failed server info to cache
cache.save_failed_server(server_name, error_msg)
# Try to close session if it was created
try:
await provider.close_session(session_id)
except Exception:
pass
print()
print(f"{'='*50}")
print(f"✓ Collected {total_tools} tools from {success_count + skipped_count}/{total} servers")
if skipped_count > 0:
print(f" (skipped {skipped_count} cached, processed {success_count} new)")
print(f"✓ Cache saved to: {cache.cache_path}")
if failed_servers:
print(f"✗ Failed servers ({len(failed_servers)}):")
for name, err in failed_servers[:10]:
print(f" - {name}: {err[:60]}")
if len(failed_servers) > 10:
print(f" ... and {len(failed_servers) - 10} more (see cache file for details)")
print()
print("Done! Future list_tools() calls will use cache (no server startup).")
def _load_config(args) -> AnyToolConfig:
"""Load configuration"""
cli_overrides = {}
if args.model:
cli_overrides['llm_model'] = args.model
if args.max_iterations is not None:
cli_overrides['grounding_max_iterations'] = args.max_iterations
if args.timeout is not None:
cli_overrides['llm_timeout'] = args.timeout
if args.log_level:
cli_overrides['log_level'] = args.log_level
try:
# Load from config file if provided
if args.config:
import json
with open(args.config, 'r', encoding='utf-8') as f:
config_dict = json.load(f)
# Apply CLI overrides
config_dict.update(cli_overrides)
config = AnyToolConfig(**config_dict)
print(f"✓ Loaded from config file: {args.config}")
else:
# Use default config + CLI overrides
config = AnyToolConfig(**cli_overrides)
print("✓ Using default configuration")
if cli_overrides:
print(f"✓ CLI overrides: {', '.join(cli_overrides.keys())}")
if args.log_level:
Logger.set_level(args.log_level)
return config
except Exception as e:
logger.error(f"Failed to load configuration: {e}")
sys.exit(1)
def _setup_ui(args) -> tuple[Optional[AnyToolUI], Optional[UIIntegration]]:
if args.no_ui:
CLIDisplay.print_banner()
return None, None
ui = create_ui(enable_live=True, compact=args.ui_compact)
ui.print_banner()
ui_integration = UIIntegration(ui)
return ui, ui_integration
async def _initialize_anytool(config: AnyToolConfig, args) -> AnyTool:
anytool = AnyTool(config)
init_steps = [("Initializing AnyTool...", "loading")]
CLIDisplay.print_initialization_progress(init_steps, show_header=False)
if not args.config:
original_log_level = Logger.get_logger("anytool").level
for log_name in ["anytool", "anytool.grounding", "anytool.agents"]:
Logger.get_logger(log_name).setLevel(logging.WARNING)
await anytool.initialize()
# Restore log level
if not args.config:
for log_name in ["anytool", "anytool.grounding", "anytool.agents"]:
Logger.get_logger(log_name).setLevel(original_log_level)
# Print initialization results
backends = anytool.list_backends()
init_steps = [
("LLM Client", "ok"),
(f"Grounding Backends ({len(backends)} available)", "ok"),
("Grounding Agent", "ok"),
]
if config.enable_recording:
init_steps.append(("Recording Manager", "ok"))
CLIDisplay.print_initialization_progress(init_steps, show_header=True)
return anytool
async def main():
parser = _create_argument_parser()
args = parser.parse_args()
# Handle subcommands
if args.command == 'refresh-cache':
await refresh_mcp_cache(args.config)
return 0
# Load configuration
config = _load_config(args)
# Setup UI
ui, ui_integration = _setup_ui(args)
# Print configuration
CLIDisplay.print_configuration(config)
anytool = None
try:
# Initialize AnyTool
anytool = await _initialize_anytool(config, args)
# Connect UI (if enabled)
if ui_integration:
ui_integration.attach_llm_client(anytool._llm_client)
ui_integration.attach_grounding_client(anytool._grounding_client)
CLIDisplay.print_system_ready()
ui_manager = UIManager(ui, ui_integration)
# Run appropriate mode
if args.query:
await single_query_mode(anytool, args.query, ui_manager)
else:
await interactive_mode(anytool, ui_manager)
except KeyboardInterrupt:
print("\n\nInterrupt signal detected")
except Exception as e:
logger.error(f"Error: {e}", exc_info=True)
print(f"\nError: {e}")
return 1
finally:
if anytool:
print("\nCleaning up resources...")
await anytool.cleanup()
print("\nGoodbye!")
return 0
def run_main():
"""Run main function"""
try:
exit_code = asyncio.run(main())
sys.exit(exit_code)
except KeyboardInterrupt:
print("\n\nProgram interrupted")
sys.exit(0)
if __name__ == "__main__":
run_main()
================================================
FILE: anytool/agents/__init__.py
================================================
from anytool.agents.base import BaseAgent, AgentStatus, AgentRegistry
from anytool.agents.grounding_agent import GroundingAgent
__all__ = [
"BaseAgent",
"AgentStatus",
"AgentRegistry",
"GroundingAgent",
]
================================================
FILE: anytool/agents/base.py
================================================
from __future__ import annotations
import json
from abc import ABC, abstractmethod
from typing import TYPE_CHECKING, Dict, List, Optional, Type, Any
from anytool.utils.logging import Logger
if TYPE_CHECKING:
from anytool.llm import LLMClient
from anytool.grounding.core.grounding_client import GroundingClient
from anytool.recording import RecordingManager
logger = Logger.get_logger(__name__)
class BaseAgent(ABC):
def __init__(
self,
name: str,
backend_scope: Optional[List[str]] = None,
llm_client: Optional[LLMClient] = None,
grounding_client: Optional[GroundingClient] = None,
recording_manager: Optional[RecordingManager] = None,
) -> None:
"""
Initialize the BaseAgent.
Args:
name: Unique name for the agent
backend_scope: List of backend types this agent can access (e.g., ["gui", "shell", "mcp", "web", "system"])
llm_client: LLM client for agent reasoning (optional, can be set later)
grounding_client: Reference to GroundingClient for tool execution
recording_manager: RecordingManager for recording execution
"""
self._name = name
self._grounding_client: Optional[GroundingClient] = grounding_client
self._backend_scope = backend_scope or []
self._llm_client = llm_client
self._recording_manager: Optional[RecordingManager] = recording_manager
self._step = 0
self._status = AgentStatus.ACTIVE
self._register_self()
logger.info(f"Initialized {self.__class__.__name__}: {name}")
@property
def name(self) -> str:
return self._name
@property
def grounding_client(self) -> Optional[GroundingClient]:
"""Get the grounding client."""
return self._grounding_client
@property
def backend_scope(self) -> List[str]:
return self._backend_scope
@property
def llm_client(self) -> Optional[LLMClient]:
return self._llm_client
@llm_client.setter
def llm_client(self, client: LLMClient) -> None:
self._llm_client = client
@property
def recording_manager(self) -> Optional[RecordingManager]:
"""Get the recording manager."""
return self._recording_manager
@property
def step(self) -> int:
return self._step
@property
def status(self) -> str:
return self._status
@abstractmethod
async def process(self, context: Dict[str, Any]) -> Dict[str, Any]:
pass
@abstractmethod
def construct_messages(self, context: Dict[str, Any]) -> List[Dict[str, Any]]:
"""
Construct messages for LLM reasoning.
Context must contain 'instruction' key.
"""
pass
async def get_llm_response(
self,
messages: List[Dict[str, Any]],
tools: Optional[List] = None,
**kwargs
) -> Dict[str, Any]:
if not self._llm_client:
raise ValueError(f"LLM client not initialized for agent {self.name}")
try:
response = await self._llm_client.complete(
messages=messages,
tools=tools,
**kwargs
)
return response
except Exception as e:
logger.error(f"{self.name}: LLM call failed: {e}", exc_info=True)
raise
def response_to_dict(self, response: str) -> Dict[str, Any]:
try:
if response.strip().startswith("```json") or response.strip().startswith("```"):
lines = response.strip().split('\n')
if lines and lines[0].startswith('```'):
lines = lines[1:]
end_idx = len(lines)
for i, line in enumerate(lines):
if line.strip() == '```':
end_idx = i
break
response = '\n'.join(lines[:end_idx])
return json.loads(response)
except json.JSONDecodeError as e:
# If parsing fails, try to find and extract just the JSON object/array
if "Extra data" in str(e):
try:
decoder = json.JSONDecoder()
obj, idx = decoder.raw_decode(response)
logger.warning(
f"{self.name}: Successfully extracted JSON but found extra text after position {idx}. "
f"Extra text: {response[idx:idx+100]}..."
)
return obj
except Exception as e2:
logger.error(f"{self.name}: Failed to extract JSON even with raw_decode: {e2}")
logger.error(f"{self.name}: Failed to parse response: {e}")
logger.error(f"{self.name}: Response content: {response[:500]}")
return {"error": "Failed to parse response", "raw": response}
def increment_step(self) -> None:
self._step += 1
@classmethod
def _register_self(cls) -> None:
"""Register the agent class in the registry upon instantiation."""
# Get the actual instance class, not BaseAgent
if cls.__name__ != "BaseAgent" and cls.__name__ not in AgentRegistry._registry:
AgentRegistry.register(cls.__name__, cls)
def __repr__(self) -> str:
return f"<{self.__class__.__name__}(name={self.name}, step={self.step}, status={self.status})>"
class AgentStatus:
"""Constants for agent status."""
ACTIVE = "active"
IDLE = "idle"
WAITING = "waiting"
class AgentRegistry:
"""
Registry for managing agent classes.
Allows dynamic registration and retrieval of agent types.
"""
_registry: Dict[str, Type[BaseAgent]] = {}
@classmethod
def register(cls, name: str, agent_cls: Type[BaseAgent]) -> None:
if name in cls._registry:
logger.warning(f"Agent class '{name}' already registered, overwriting")
cls._registry[name] = agent_cls
logger.debug(f"Registered agent class: {name}")
@classmethod
def get_cls(cls, name: str) -> Type[BaseAgent]:
if name not in cls._registry:
raise ValueError(f"No agent class registered under '{name}'")
return cls._registry[name]
@classmethod
def list_registered(cls) -> List[str]:
return list(cls._registry.keys())
@classmethod
def clear(cls) -> None:
cls._registry.clear()
logger.debug("Agent registry cleared")
================================================
FILE: anytool/agents/grounding_agent.py
================================================
from __future__ import annotations
import copy
import json
from typing import TYPE_CHECKING, Any, Dict, List, Optional
from anytool.agents.base import BaseAgent
from anytool.grounding.core.types import BackendType, ToolResult
from anytool.platform.screenshot import ScreenshotClient
from anytool.prompts import GroundingAgentPrompts
from anytool.utils.logging import Logger
if TYPE_CHECKING:
from anytool.llm import LLMClient
from anytool.grounding.core.grounding_client import GroundingClient
from anytool.recording import RecordingManager
logger = Logger.get_logger(__name__)
class GroundingAgent(BaseAgent):
def __init__(
self,
name: str = "GroundingAgent",
backend_scope: Optional[List[str]] = None,
llm_client: Optional[LLMClient] = None,
grounding_client: Optional[GroundingClient] = None,
recording_manager: Optional[RecordingManager] = None,
system_prompt: Optional[str] = None,
max_iterations: int = 15,
visual_analysis_timeout: float = 30.0,
tool_retrieval_llm: Optional[LLMClient] = None,
visual_analysis_model: Optional[str] = None,
) -> None:
"""
Initialize the Grounding Agent.
Args:
name: Agent name
backend_scope: List of backends this agent can access (None = all available)
llm_client: LLM client for reasoning
grounding_client: GroundingClient for tool execution
recording_manager: RecordingManager for recording execution
system_prompt: Custom system prompt
max_iterations: Maximum LLM reasoning iterations for self-correction
visual_analysis_timeout: Timeout for visual analysis LLM calls in seconds
tool_retrieval_llm: LLM client for tool retrieval filter (None = use llm_client)
visual_analysis_model: Model name for visual analysis (None = use llm_client.model)
"""
super().__init__(
name=name,
backend_scope=backend_scope or ["gui", "shell", "mcp", "web", "system"],
llm_client=llm_client,
grounding_client=grounding_client,
recording_manager=recording_manager
)
self._system_prompt = system_prompt or self._default_system_prompt()
self._max_iterations = max_iterations
self._visual_analysis_timeout = visual_analysis_timeout
self._tool_retrieval_llm = tool_retrieval_llm
self._visual_analysis_model = visual_analysis_model
logger.info(f"Grounding Agent initialized: {name}")
logger.info(f"Backend scope: {self._backend_scope}")
logger.info(f"Max iterations: {self._max_iterations}")
logger.info(f"Visual analysis timeout: {self._visual_analysis_timeout}s")
if tool_retrieval_llm:
logger.info(f"Tool retrieval model: {tool_retrieval_llm.model}")
if visual_analysis_model:
logger.info(f"Visual analysis model: {visual_analysis_model}")
def _truncate_messages(
self,
messages: List[Dict[str, Any]],
keep_recent: int = 8,
max_tokens_estimate: int = 120000
) -> List[Dict[str, Any]]:
if len(messages) <= keep_recent + 2: # +2 for system and initial user
return messages
total_text = json.dumps(messages, ensure_ascii=False)
estimated_tokens = len(total_text) // 4
if estimated_tokens < max_tokens_estimate:
return messages
logger.info(f"Truncating message history: {len(messages)} messages, "
f"~{estimated_tokens:,} tokens -> keeping recent {keep_recent} rounds")
system_messages = []
user_instruction = None
conversation_messages = []
for msg in messages:
role = msg.get("role")
if role == "system":
system_messages.append(msg)
elif role == "user" and user_instruction is None:
user_instruction = msg
else:
conversation_messages.append(msg)
recent_messages = conversation_messages[-(keep_recent * 2):] if conversation_messages else []
truncated = system_messages.copy()
if user_instruction:
truncated.append(user_instruction)
truncated.extend(recent_messages)
logger.info(f"After truncation: {len(truncated)} messages, "
f"~{len(json.dumps(truncated, ensure_ascii=False))//4:,} tokens (estimated)")
return truncated
async def process(self, context: Dict[str, Any]) -> Dict[str, Any]:
"""
Process a task execution request with multi-round iteration control.
"""
instruction = context.get("instruction", "")
if not instruction:
logger.error("Grounding Agent: No instruction provided")
return {"error": "No instruction provided", "status": "error"}
# Store current instruction for visual analysis context
self._current_instruction = instruction
logger.info(f"Grounding Agent: Processing instruction at step {self.step}")
# Exist workspace files check
workspace_info = await self._check_workspace_artifacts(context)
if workspace_info["has_files"]:
context["workspace_artifacts"] = workspace_info
logger.info(f"Workspace has {len(workspace_info['files'])} existing files: {workspace_info['files']}")
# Get available tools (auto-search with cap)
tools = await self._get_available_tools(instruction)
# Get search debug info (similarity scores, LLM selections)
search_debug_info = None
if self.grounding_client:
search_debug_info = self.grounding_client.get_last_search_debug_info()
# Build retrieved tools list for return value
retrieved_tools_list = []
for tool in tools:
tool_info = {
"name": getattr(tool, "name", str(tool)),
"description": getattr(tool, "description", ""),
}
if hasattr(tool, "backend_type"):
tool_info["backend"] = tool.backend_type.value if hasattr(tool.backend_type, "value") else str(tool.backend_type)
if hasattr(tool, "_runtime_info") and tool._runtime_info:
tool_info["server_name"] = tool._runtime_info.server_name
# Add similarity score if available
if search_debug_info and search_debug_info.get("tool_scores"):
for score_info in search_debug_info["tool_scores"]:
if score_info["name"] == tool_info["name"]:
tool_info["similarity_score"] = score_info["score"]
break
retrieved_tools_list.append(tool_info)
# Record retrieved tools
if self._recording_manager:
from anytool.recording import RecordingManager
await RecordingManager.record_retrieved_tools(
task_instruction=instruction,
tools=tools,
search_debug_info=search_debug_info,
)
# Initialize iteration state
max_iterations = context.get("max_iterations", self._max_iterations)
current_iteration = 0
all_tool_results = []
iteration_contexts = []
consecutive_empty_responses = 0 # Track consecutive empty LLM responses
MAX_CONSECUTIVE_EMPTY = 5 # Exit after this many empty responses
# Build initial messages
messages = self.construct_messages(context)
try:
while current_iteration < max_iterations:
current_iteration += 1
logger.info(f"Grounding Agent: Iteration {current_iteration}/{max_iterations}")
# Truncate message history to prevent context length issues
# Start truncating after 5 iterations to keep context manageable
if current_iteration >= 5:
messages = self._truncate_messages(
messages,
keep_recent=8, # 保留最近8轮对话
max_tokens_estimate=120000 # Claude Sonnet 4.5 上下文限制是200K，保守使用120K
)
messages_input_snapshot = copy.deepcopy(messages)
# [DISABLED] Iteration summary generation
# Tool results (including visual analysis) are already in context,
# LLM can make decisions directly without separate summary.
# To re-enable, uncomment below and pass iteration_summary_prompt to complete()
# iteration_summary_prompt = GroundingAgentPrompts.iteration_summary(
# instruction=instruction,
# iteration=current_iteration,
# max_iterations=max_iterations
# ) if context.get("auto_execute", True) else None
# Call LLMClient for single round
# LLM will decide whether to call tools or finish with <COMPLETE>
llm_response = await self._llm_client.complete(
messages=messages,
tools=tools if context.get("auto_execute", True) else None,
execute_tools=context.get("auto_execute", True),
summary_prompt=None, # Disabled
tool_result_callback=self._visual_analysis_callback
)
# Update messages with LLM response
messages = llm_response["messages"]
# Collect tool results
tool_results_this_iteration = llm_response.get("tool_results", [])
if tool_results_this_iteration:
all_tool_results.extend(tool_results_this_iteration)
# [DISABLED] Iteration summary logging
# llm_summary = llm_response.get("iteration_summary")
# if llm_summary:
# logger.info(f"Iteration {current_iteration} summary: {llm_summary[:150]}...")
assistant_message = llm_response.get("message", {})
assistant_content = assistant_message.get("content", "")
has_tool_calls = llm_response.get('has_tool_calls', False)
logger.info(f"Iteration {current_iteration} - Has tool calls: {has_tool_calls}, "
f"Tool results: {len(tool_results_this_iteration)}, "
f"Content length: {len(assistant_content)} chars")
if len(assistant_content) > 0:
logger.info(f"Iteration {current_iteration} - Assistant content preview: {repr(assistant_content[:300])}")
consecutive_empty_responses = 0 # Reset counter on valid response
else:
if not has_tool_calls:
consecutive_empty_responses += 1
logger.warning(f"Iteration {current_iteration} - NO tool calls and NO content "
f"(empty response {consecutive_empty_responses}/{MAX_CONSECUTIVE_EMPTY})")
if consecutive_empty_responses >= MAX_CONSECUTIVE_EMPTY:
logger.error(f"Exiting due to {MAX_CONSECUTIVE_EMPTY} consecutive empty LLM responses. "
"This may indicate API issues, rate limiting, or context too long.")
break
else:
consecutive_empty_responses = 0 # Reset if we have tool calls
# Snapshot messages after LLM call (accumulated context)
messages_output_snapshot = copy.deepcopy(messages)
# Record iteration context
iteration_context = {
"iteration": current_iteration,
"messages_input": messages_input_snapshot,
"messages_output": messages_output_snapshot,
"llm_response_summary": {
"assistant_content": assistant_content,
"has_tool_calls": has_tool_calls,
# "iteration_summary": llm_summary, # Disabled with iteration summary
"tool_calls_count": len(tool_results_this_iteration),
},
}
iteration_contexts.append(iteration_context)
# Real-time save to conversations.jsonl
from anytool.recording import RecordingManager
await RecordingManager.record_iteration_context(
iteration=current_iteration,
messages_input=messages_input_snapshot,
messages_output=messages_output_snapshot,
llm_response_summary=iteration_context["llm_response_summary"],
)
# Check for completion token in assistant content
# [DISABLED] Also check in iteration summary when enabled
# is_complete = (
# GroundingAgentPrompts.TASK_COMPLETE in assistant_content or
# (llm_summary and GroundingAgentPrompts.TASK_COMPLETE in llm_summary)
# )
is_complete = GroundingAgentPrompts.TASK_COMPLETE in assistant_content
if is_complete:
# Task is complete - LLM generated completion token
logger.info(f"Task completed at iteration {current_iteration} (found {GroundingAgentPrompts.TASK_COMPLETE})")
break
else:
# LLM didn't generate <COMPLETE>, continue to next iteration
if tool_results_this_iteration:
logger.debug(f"Task in progress, LLM called {len(tool_results_this_iteration)} tools")
else:
logger.debug(f"Task in progress, LLM did not generate <COMPLETE>")
# Remove previous iteration guidance to avoid accumulation
messages = [
msg for msg in messages
if not (msg.get("role") == "system" and "Iteration" in msg.get("content", "") and "complete" in msg.get("content", ""))
]
guidance_msg = {
"role": "system",
"content": f"Iteration {current_iteration} complete. "
f"Check if task is finished - if yes, output {GroundingAgentPrompts.TASK_COMPLETE}. "
f"If not, continue with next action."
}
messages.append(guidance_msg)
# [DISABLED] Full iteration feedback with summary
# self._remove_previous_guidance(messages)
# feedback_msg = self._build_iteration_feedback(
# iteration=current_iteration,
# llm_summary=llm_summary,
# add_guidance=True
# )
# if feedback_msg:
# messages.append(feedback_msg)
# logger.debug(f"Added iteration {current_iteration} feedback with guidance")
continue
# Build final result
result = await self._build_final_result(
instruction=instruction,
messages=messages,
all_tool_results=all_tool_results,
iterations=current_iteration,
max_iterations=max_iterations,
iteration_contexts=iteration_contexts,
retrieved_tools_list=retrieved_tools_list,
search_debug_info=search_debug_info,
)
# Record agent action to recording manager
if self._recording_manager:
await self._record_agent_execution(result, instruction)
# Increment step
self.increment_step()
logger.info(f"Grounding Agent: Execution completed with status: {result.get('status')}")
return result
except Exception as e:
logger.error(f"Grounding Agent: Execution failed: {e}")
result = {
"error": str(e),
"status": "error",
"instruction": instruction,
"iteration": current_iteration
}
self.increment_step()
return result
def _default_system_prompt(self) -> str:
"""Default system prompt for the grounding agent."""
return GroundingAgentPrompts.SYSTEM_PROMPT
def construct_messages(
self,
context: Dict[str, Any]
) -> List[Dict[str, Any]]:
messages = [{"role": "system", "content": self._system_prompt}]
# Get instruction from context
instruction = context.get("instruction", "")
if not instruction:
raise ValueError("context must contain 'instruction' field")
# Add workspace directory
workspace_dir = context.get("workspace_dir")
if workspace_dir:
messages.append({
"role": "system",
"content": GroundingAgentPrompts.workspace_directory(workspace_dir)
})
# Add workspace artifacts information
workspace_artifacts = context.get("workspace_artifacts")
if workspace_artifacts and workspace_artifacts.get("has_files"):
files = workspace_artifacts.get("files", [])
matching_files = workspace_artifacts.get("matching_files", [])
recent_files = workspace_artifacts.get("recent_files", [])
if matching_files:
artifact_msg = GroundingAgentPrompts.workspace_matching_files(matching_files)
elif len(recent_files) >= 2:
artifact_msg = GroundingAgentPrompts.workspace_recent_files(
total_files=len(files),
recent_files=recent_files
)
else:
artifact_msg = GroundingAgentPrompts.workspace_file_list(files)
messages.append({
"role": "system",
"content": artifact_msg
})
# User instruction
messages.append({"role": "user", "content": instruction})
return messages
async def _get_available_tools(self, task_description: Optional[str]) -> List:
"""
Retrieve tools with auto-search + cap to control prompt bloat.
Falls back to returning all tools if search fails.
"""
grounding_client = self.grounding_client
if not grounding_client:
return []
backends = [BackendType(name) for name in self._backend_scope]
try:
# Use dedicated tool retrieval LLM if configured, otherwise use main LLM
retrieval_llm = self._tool_retrieval_llm or self._llm_client
tools = await grounding_client.get_tools_with_auto_search(
task_description=task_description,
backend=backends,
use_cache=True,
llm_callable=retrieval_llm,
)
logger.info(
f"GroundingAgent selected {len(tools)} tools (auto-search) from {len(backends)} backends"
)
return tools
except Exception as e:
logger.warning(f"Auto-search tools failed, falling back to full list: {e}")
# Fallback: fetch all tools (previous behaviour)
all_tools = []
for backend_name in self._backend_scope:
try:
backend_type = BackendType(backend_name)
tools = await grounding_client.list_tools(backend=backend_type)
all_tools.extend(tools)
logger.debug(f"Retrieved {len(tools)} tools from backend: {backend_name}")
except Exception as e:
logger.debug(f"Could not get tools from {backend_name}: {e}")
logger.info(
f"GroundingAgent fallback retrieved {len(all_tools)} tools from {len(self._backend_scope)} backends"
)
return all_tools
async def _visual_analysis_callback(
self,
result: ToolResult,
tool_name: str,
tool_call: Dict,
backend: str
) -> ToolResult:
"""
Callback for LLMClient to handle visual analysis after tool execution.
"""
# 1. Check if LLM requested to skip visual analysis
skip_visual_analysis = False
try:
arguments = tool_call.function.arguments
if isinstance(arguments, str):
args = json.loads(arguments.strip() or "{}")
else:
args = arguments
if isinstance(args, dict) and args.get("skip_visual_analysis"):
skip_visual_analysis = True
logger.info(f"Visual analysis skipped for {tool_name} (meta-parameter set by LLM)")
except Exception as e:
logger.debug(f"Could not parse tool arguments: {e}")
# 2. If skip requested, return original result
if skip_visual_analysis:
return result
# 3. Check if this backend needs visual analysis
if backend != "gui":
return result
# 4. Check if tool has visual data
metadata = getattr(result, 'metadata', None)
has_screenshots = metadata and (metadata.get("screenshot") or metadata.get("screenshots"))
# 5. If no visual data, try to capture a screenshot
if not has_screenshots:
try:
logger.info(f"No visual data from {tool_name}, capturing screenshot...")
screenshot_client = ScreenshotClient()
screenshot_bytes = await screenshot_client.capture()
if screenshot_bytes:
# Add screenshot to result metadata
if metadata is None:
result.metadata = {}
metadata = result.metadata
metadata["screenshot"] = screenshot_bytes
has_screenshots = True
logger.info(f"Screenshot captured for visual analysis")
else:
logger.warning("Failed to capture screenshot")
except Exception as e:
logger.warning(f"Error capturing screenshot: {e}")
# 6. If still no screenshots, return original result
if not has_screenshots:
logger.debug(f"No visual data available for {tool_name}")
return result
# 7. Perform visual analysis
return await self._enhance_result_with_visual_context(result, tool_name)
async def _enhance_result_with_visual_context(
self,
result: ToolResult,
tool_name: str
) -> ToolResult:
"""
Enhance tool result with visual analysis for grounding agent workflows.
"""
import asyncio
import base64
import litellm
try:
metadata = getattr(result, 'metadata', None)
if not metadata:
return result
# Collect all screenshots
screenshots_bytes = []
# Check for multiple screenshots first
if metadata.get("screenshots"):
screenshots_list = metadata["screenshots"]
if isinstance(screenshots_list, list):
screenshots_bytes = [s for s in screenshots_list if s]
# Fall back to single screenshot
elif metadata.get("screenshot"):
screenshots_bytes = [metadata["screenshot"]]
if not screenshots_bytes:
return result
# Select key screenshots if there are too many
selected_screenshots = self._select_key_screenshots(screenshots_bytes, max_count=3)
# Convert to base64
visual_b64_list = []
for visual_data in selected_screenshots:
if isinstance(visual_data, bytes):
visual_b64_list.append(base64.b64encode(visual_data).decode('utf-8'))
else:
visual_b64_list.append(visual_data) # Already base64
# Build prompt based on number of screenshots
num_screenshots = len(visual_b64_list)
prompt = GroundingAgentPrompts.visual_analysis(
tool_name=tool_name,
num_screenshots=num_screenshots,
task_description=getattr(self, '_current_instruction', '')
)
# Build content with text prompt + all images
content = [{"type": "text", "text": prompt}]
for visual_b64 in visual_b64_list:
content.append({
"type": "image_url",
"image_url": {
"url": f"data:image/png;base64,{visual_b64}"
}
})
# Use dedicated visual analysis model if configured, otherwise use main LLM model
visual_model = self._visual_analysis_model or (self._llm_client.model if self._llm_client else "openrouter/anthropic/claude-sonnet-4.5")
response = await asyncio.wait_for(
litellm.acompletion(
model=visual_model,
messages=[{
"role": "user",
"content": content
}],
timeout=self._visual_analysis_timeout
),
timeout=self._visual_analysis_timeout + 5
)
analysis = response.choices[0].message.content.strip()
# Inject visual analysis into content
original_content = result.content or "(no text output)"
enhanced_content = f"{original_content}\n\n**Visual content**: {analysis}"
# Create enhanced result
enhanced_result = ToolResult(
status=result.status,
content=enhanced_content,
error=result.error,
metadata={**metadata, "visual_analyzed": True, "visual_analysis": analysis},
execution_time=result.execution_time
)
logger.info(f"Enhanced {tool_name} result with visual analysis ({num_screenshots} screenshot(s))")
return enhanced_result
except asyncio.TimeoutError:
logger.warning(f"Visual analysis timed out for {tool_name}, returning original result")
return result
except Exception as e:
logger.warning(f"Failed to analyze visual content for {tool_name}: {e}")
return result
def _select_key_screenshots(
self,
screenshots: List[bytes],
max_count: int = 3
) -> List[bytes]:
"""
Select key screenshots if there are too many.
"""
if len(screenshots) <= max_count:
return screenshots
selected_indices = set()
# Always include last (final state)
selected_indices.add(len(screenshots) - 1)
# If room, include first (initial state)
if max_count >= 2:
selected_indices.add(0)
# Fill remaining slots with evenly spaced middle screenshots
remaining_slots = max_count - len(selected_indices)
if remaining_slots > 0:
# Calculate spacing
available_indices = [
i for i in range(1, len(screenshots) - 1)
if i not in selected_indices
]
if available_indices:
step = max(1, len(available_indices) // (remaining_slots + 1))
for i in range(remaining_slots):
idx = min((i + 1) * step, len(available_indices) - 1)
if idx < len(available_indices):
selected_indices.add(available_indices[idx])
# Return screenshots in original order
selected = [screenshots[i] for i in sorted(selected_indices)]
logger.debug(
f"Selected {len(selected)} screenshots at indices {sorted(selected_indices)} "
f"from total of {len(screenshots)}"
)
return selected
def _get_workspace_path(self, context: Dict[str, Any]) -> Optional[str]:
"""
Get workspace directory path from context.
"""
return context.get("workspace_dir")
def _scan_workspace_files(
self,
workspace_path: str,
recent_threshold: int = 600 # seconds
) -> Dict[str, Any]:
"""
Scan workspace directory and collect file information.
Args:
workspace_path: Path to workspace directory
recent_threshold: Threshold in seconds for recent files
Returns:
Dictionary with file information:
- files: List of all filenames
- file_details: Dict mapping filename to file info (size, modified, age_seconds)
- recent_files: List of recently modified filenames
"""
import os
import time
result = {
"files": [],
"file_details": {},
"recent_files": []
}
if not workspace_path or not os.path.exists(workspace_path):
return result
# Recording system files to exclude from workspace scanning
excluded_files = {"metadata.json", "traj.jsonl"}
try:
current_time = time.time()
for filename in os.listdir(workspace_path):
filepath = os.path.join(workspace_path, filename)
if os.path.isfile(filepath) and filename not in excluded_files:
result["files"].append(filename)
# Get file stats
stat = os.stat(filepath)
file_info = {
"size": stat.st_size,
"modified": stat.st_mtime,
"age_seconds": current_time - stat.st_mtime
}
result["file_details"][filename] = file_info
# Track recently created/modified files
if file_info["age_seconds"] < recent_threshold:
result["recent_files"].append(filename)
result["files"] = sorted(result["files"])
except Exception as e:
logger.debug(f"Error scanning workspace files: {e}")
return result
async def _check_workspace_artifacts(self, context: Dict[str, Any]) -> Dict[str, Any]:
"""
Check workspace directory for existing artifacts that might be relevant to the task.
Enhanced to detect if task might already be completed.
"""
import re
workspace_info = {"has_files": False, "files": [], "file_details": {}, "recent_files": []}
try:
# Get workspace path
workspace_path = self._get_workspace_path(context)
# Scan workspace files
scan_result = self._scan_workspace_files(workspace_path, recent_threshold=600)
if scan_result["files"]:
workspace_info["has_files"] = True
workspace_info["files"] = scan_result["files"]
workspace_info["file_details"] = scan_result["file_details"]
workspace_info["recent_files"] = scan_result["recent_files"]
logger.info(f"Grounding Agent: Found {len(scan_result['files'])} existing files in workspace "
f"({len(scan_result['recent_files'])} recent)")
# Check if instruction mentions specific filenames
instruction = context.get("instruction", "")
if instruction:
# Look for potential file references in instruction
potential_outputs = []
# Match common file patterns: filename.ext, "filename", 'filename'
file_patterns = re.findall(r'["\']?([a-zA-Z0-9_\-]+\.[a-zA-Z0-9]+)["\']?', instruction)
for pattern in file_patterns:
if pattern in scan_result["files"]:
potential_outputs.append(pattern)
if potential_outputs:
workspace_info["matching_files"] = potential_outputs
logger.info(f"Grounding Agent: Found {len(potential_outputs)} files matching task: {potential_outputs}")
except Exception as e:
logger.debug(f"Could not check workspace artifacts: {e}")
return workspace_info
def _build_iteration_feedback(
self,
iteration: int,
llm_summary: Optional[str] = None,
add_guidance: bool = True
) -> Optional[Dict[str, str]]:
"""
Build feedback message to add to next iteration.
"""
if not llm_summary:
return None
feedback_content = GroundingAgentPrompts.iteration_feedback(
iteration=iteration,
llm_summary=llm_summary,
add_guidance=add_guidance
)
return {
"role": "system",
"content": feedback_content
}
def _remove_previous_guidance(self, messages: List[Dict[str, Any]]) -> None:
"""
Remove guidance section from previous iteration feedback messages.
"""
for msg in messages:
if msg.get("role") == "system":
content = msg.get("content", "")
# Check if this is an iteration feedback message with guidance
if "## Iteration" in content and "Summary" in content and "---" in content:
# Remove everything from "---" onwards (the guidance part)
summary_only = content.split("---")[0].strip()
msg["content"] = summary_only
async def _generate_final_summary(
self,
instruction: str,
messages: List[Dict],
iterations: int
) -> tuple[str, bool, List[Dict]]:
"""
Generate final summary across all iterations for reporting to upper layer.
Returns:
tuple[str, bool, List[Dict]]: (summary_text, success_flag, context_used)
- summary_text: The generated summary or error message
- success_flag: True if summary was generated successfully, False otherwise
- context_used: The cleaned messages used for generating summary
"""
final_summary_prompt = {
"role": "user",
"content": GroundingAgentPrompts.final_summary(
instruction=instruction,
iterations=iterations
)
}
clean_messages = []
for msg in messages:
# Skip tool result messages
if msg.get("role") == "tool":
continue
# Copy message and remove tool_calls if present
clean_msg = msg.copy()
if "tool_calls" in clean_msg:
del clean_msg["tool_calls"]
clean_messages.append(clean_msg)
clean_messages.append(final_summary_prompt)
# Save context for return
context_for_return = copy.deepcopy(clean_messages)
try:
# Call LLMClient to generate final summary (without tools)
summary_response = await self._llm_client.complete(
messages=clean_messages,
tools=None,
execute_tools=False
)
final_summary = summary_response.get("message", {}).get("content", "")
if final_summary:
logger.info(f"Generated final summary: {final_summary[:200]}...")
return final_summary, True, context_for_return
else:
logger.warning("LLM returned empty final summary")
return f"Task completed after {iterations} iteration(s). Check execution history for details.", True, context_for_return
except Exception as e:
logger.error(f"Error generating final summary: {e}")
return f"Task completed after {iterations} iteration(s), but failed to generate summary: {str(e)}", False, context_for_return
async def _build_final_result(
self,
instruction: str,
messages: List[Dict],
all_tool_results: List[Dict],
iterations: int,
max_iterations: int,
iteration_contexts: List[Dict] = None,
retrieved_tools_list: List[Dict] = None,
search_debug_info: Dict[str, Any] = None,
) -> Dict[str, Any]:
"""
Build final execution result.
Args:
instruction: Original instruction
messages: Complete conversation history (including all iteration summaries)
all_tool_results: All tool execution results
iterations: Number of iterations performed
max_iterations: Maximum allowed iterations
iteration_contexts: Context snapshots for each iteration
retrieved_tools_list: List of tools retrieved for this task
search_debug_info: Debug info from tool search (similarity scores, LLM selections)
"""
is_complete = self._check_task_completion(messages)
tool_executions = self._format_tool_executions(all_tool_results)
result = {
"instruction": instruction,
"step": self.step,
"iterations": iterations,
"tool_executions": tool_executions,
"messages": messages,
"iteration_contexts": iteration_contexts or [],
"retrieved_tools_list": retrieved_tools_list or [],
"search_debug_info": search_debug_info,
"keep_session": True
}
if is_complete:
logger.info("Task completed with <COMPLETE> marker")
# Use LLM's own completion response directly (no extra LLM call needed)
# LLM already generates a summary before outputting <COMPLETE>
last_response = self._extract_last_assistant_message(messages)
# Remove the <COMPLETE> token from response for cleaner output
result["response"] = last_response.replace(GroundingAgentPrompts.TASK_COMPLETE, "").strip()
result["status"] = "success"
# [DISABLED] Extra LLM call to generate final summary
# final_summary, summary_success, final_summary_context = await self._generate_final_summary(
# instruction=instruction,
# messages=messages,
# iterations=iterations
# )
# result["response"] = final_summary
# result["final_summary_context"] = final_summary_context
else:
result["response"] = self._extract_last_assistant_message(messages)
result["status"] = "incomplete"
result["warning"] = (
f"Task reached max iterations ({max_iterations}) without completion. "
f"This may indicate the task needs more steps or clarification."
)
return result
def _format_tool_executions(self, all_tool_results: List[Dict]) -> List[Dict]:
executions = []
for tr in all_tool_results:
tool_result_obj = tr.get("result")
tool_call = tr.get("tool_call")
status = "unknown"
if hasattr(tool_result_obj, 'status'):
status_obj = tool_result_obj.status
status = getattr(status_obj, 'value', status_obj)
# Extract tool_name and arguments from tool_call object (litellm format)
tool_name = "unknown"
arguments = {}
if tool_call is not None:
if hasattr(tool_call, 'function'):
# tool_call is an object with .function attribute
tool_name = getattr(tool_call.function, 'name', 'unknown')
args_raw = getattr(tool_call.function, 'arguments', '{}')
if isinstance(args_raw, str):
try:
arguments = json.loads(args_raw) if args_raw.strip() else {}
except json.JSONDecodeError:
arguments = {}
else:
arguments = args_raw if isinstance(args_raw, dict) else {}
elif isinstance(tool_call, dict):
# Fallback: tool_call is a dict
func = tool_call.get("function", {})
tool_name = func.get("name", "unknown")
args_raw = func.get("arguments", "{}")
if isinstance(args_raw, str):
try:
arguments = json.loads(args_raw) if args_raw.strip() else {}
except json.JSONDecodeError:
arguments = {}
else:
arguments = args_raw if isinstance(args_raw, dict) else {}
executions.append({
"tool_name": tool_name,
"arguments": arguments,
"backend": tr.get("backend"),
"server_name": tr.get("server_name"),
"status": status,
"content": tool_result_obj.content if hasattr(tool_result_obj, 'content') else None,
"error": tool_result_obj.error if hasattr(tool_result_obj, 'error') else None,
"execution_time": tool_result_obj.execution_time if hasattr(tool_result_obj, 'execution_time') else None,
"metadata": tool_result_obj.metadata if hasattr(tool_result_obj, 'metadata') else {},
})
return executions
def _check_task_completion(self, messages: List[Dict]) -> bool:
for msg in reversed(messages):
if msg.get("role") == "assistant":
content = msg.get("content", "")
return GroundingAgentPrompts.TASK_COMPLETE in content
return False
def _extract_last_assistant_message(self, messages: List[Dict]) -> str:
for msg in reversed(messages):
if msg.get("role") == "assistant":
return msg.get("content", "")
return ""
async def _record_agent_execution(
self,
result: Dict[str, Any],
instruction: str
) -> None:
"""
Record agent execution to recording manager.
Args:
result: Execution result
instruction: Original instruction
"""
if not self._recording_manager:
return
# Extract tool execution summary
tool_summary = []
if result.get("tool_executions"):
for exec_info in result["tool_executions"]:
tool_summary.append({
"tool": exec_info.get("tool_name", "unknown"),
"backend": exec_info.get("backend", "unknown"),
"status": exec_info.get("status", "unknown"),
})
await self._recording_manager.record_agent_action(
agent_name=self.name,
action_type="execute",
input_data={"instruction": instruction},
reasoning={
"response": result.get("response", ""),
"tools_selected": tool_summary,
},
output_data={
"status": result.get("status", "unknown"),
"iterations": result.get("iterations", 0),
"num_tool_executions": len(result.get("tool_executions", [])),
},
metadata={
"step": self.step,
"instruction": instruction,
}
)
================================================
FILE: anytool/config/__init__.py
================================================
from .grounding import *
from .loader import *
from .constants import *
from .utils import *
from . import constants
__all__ = [
# Grounding Config
"BackendConfig",
"ShellConfig",
"WebConfig",
"MCPConfig",
"GUIConfig",
"ToolSearchConfig",
"SessionConfig",
"SecurityPolicy",
"GroundingConfig",
# Loader
"CONFIG_DIR",
"load_config",
"get_config",
"reset_config",
"save_config",
"load_agents_config",
"get_agent_config",
# Utils
"get_config_value",
"load_json_file",
"save_json_file",
] + constants.__all__
================================================
FILE: anytool/config/config_agents.json
================================================
{
"agents": [
{
"name": "GroundingAgent",
"class_name": "GroundingAgent",
"backend_scope": ["gui", "shell", "mcp", "system", "web"],
"max_iterations": 15,
"visual_analysis_timeout": 60.0
}
]
}
================================================
FILE: anytool/config/config_dev.json.example
================================================
{
"comment": "[Optional] Loading grounding.json → security.json → dev.json (dev.json overrides the former ones)",
"debug": true,
"log_level": "DEBUG",
"security_policies": {
"global": {
"blocked_commands": []
}
}
}
================================================
FILE: anytool/config/config_grounding.json
================================================
{
"shell": {
"mode": "local",
"timeout": 60,
"max_retries": 3,
"retry_interval": 3.0,
"default_shell": "/bin/bash",
"working_dir": null,
"env": {},
"conda_env": null,
"default_port": 5000
},
"mcp": {
"timeout": 30,
"max_retries": 3,
"retry_interval": 2.0,
"sandbox": false,
"auto_initialize": true,
"eager_sessions": false,
"sse_read_timeout": 300.0,
"check_dependencies": true,
"auto_install": true
},
"gui": {
"mode": "local",
"timeout": 90,
"max_retries": 3,
"retry_interval": 5.0,
"driver_type": "pyautogui",
"failsafe": false,
"screenshot_on_error": true,
"pkgs_prefix": "import pyautogui; import time; pyautogui.FAILSAFE = {failsafe}; {command}"
},
"tool_search": {
"embedding_model": "BAAI/bge-small-en-v1.5",
"max_tools": 40,
"search_mode": "hybrid",
"enable_llm_filter": true,
"llm_filter_threshold": 50,
"enable_cache_persistence": true,
"cache_dir": null
},
"tool_quality": {
"enabled": true,
"enable_persistence": true,
"cache_dir": null,
"auto_evaluate_descriptions": true,
"enable_quality_ranking": true,
"evolve_interval": 5
},
"tool_cache_ttl": 600,
"tool_cache_maxsize": 500,
"debug": false,
"log_level": "INFO",
"enabled_backends": [
{
"name": "shell",
"provider_cls": "anytool.grounding.backends.shell.ShellProvider"
},
{
"name": "web",
"provider_cls": "anytool.grounding.backends.web.WebProvider"
},
{
"name": "mcp",
"provider_cls": "anytool.grounding.backends.mcp.MCPProvider"
},
{
"name": "gui",
"provider_cls": "anytool.grounding.backends.gui.GUIProvider"
}
],
"_comment_system_backend": "Note: 'system' backend is automatically registered and always available. It provides meta-level tools for querying system state. Do not add it to enabled_backends as it requires special initialization."
}
================================================
FILE: anytool/config/config_mcp.json.example
================================================
================================================
FILE: anytool/config/config_security.json
================================================
{
"security_policies": {
"global": {
"allow_shell_commands": true,
"allow_network_access": true,
"allow_file_access": true,
"blocked_commands": {
"common": ["rm", "-rf", "shutdown", "reboot", "poweroff", "halt"],
"linux": ["mkfs", "dd", "iptables", "systemctl", "init", "kill", "-9", "pkill"],
"darwin": ["diskutil", "dd", "pfctl", "launchctl", "killall"],
"windows": ["del", "format", "rd", "rmdir", "/s", "/q", "taskkill", "/f"]
},
"sandbox_enabled": false
},
"backend": {
"shell": {
"allow_shell_commands": true,
"allow_file_access": true,
"blocked_commands": {
"common": ["rm", "-rf", "shutdown", "reboot", "poweroff", "halt"],
"linux": [
"mkfs", "mkfs.ext4", "mkfs.xfs",
"dd",
"iptables", "ip6tables", "nftables",
"systemctl", "service",
"fdisk", "parted", "gdisk",
"mount", "umount",
"chmod", "777",
"chown", "root",
"passwd",
"useradd", "userdel", "usermod",
"kill", "-9", "pkill", "killall"
],
"darwin": [
"diskutil",
"dd",
"pfctl",
"launchctl",
"dscl",
"chmod", "777",
"chown", "root",
"passwd",
"killall",
"pmset"
],
"windows": [
"del", "erase",
"format",
"rd", "rmdir", "/s", "/q",
"diskpart",
"reg", "delete",
"net", "user",
"taskkill", "/f",
"wmic"
]
},
"sandbox_enabled": false
},
"mcp": {
"sandbox_enabled": false
},
"web": {
"allow_network_access": true,
"allowed_domains": []
}
}
}
}
================================================
FILE: anytool/config/constants.py
================================================
from pathlib import Path
CONFIG_GROUNDING = "config_grounding.json"
CONFIG_SECURITY = "config_security.json"
CONFIG_MCP = "config_mcp.json"
CONFIG_DEV = "config_dev.json"
CONFIG_AGENTS = "config_agents.json"
LOG_LEVELS = ["DEBUG", "INFO", "WARNING", "ERROR", "CRITICAL"]
# Project root directory (AnyTool/)
PROJECT_ROOT = Path(__file__).parent.parent.parent
__all__ = [
"CONFIG_GROUNDING",
"CONFIG_SECURITY",
"CONFIG_MCP",
"CONFIG_DEV",
"CONFIG_AGENTS",
"LOG_LEVELS",
"PROJECT_ROOT",
]
================================================
FILE: anytool/config/grounding.py
================================================
from typing import Dict, Optional, Any, List, Literal
try:
from pydantic import BaseModel, Field, field_validator
PYDANTIC_V2 = True
except ImportError:
from pydantic import BaseModel, Field, validator as field_validator
PYDANTIC_V2 = False
from anytool.grounding.core.types import (
SessionConfig,
SecurityPolicy,
BackendType
)
from .constants import LOG_LEVELS
class ConfigMixin:
"""Mixin to add utility methods for config access"""
def get_value(self, key: str, default=None):
"""
Safely get config value, works with both dict and Pydantic models.
Args:
key: Configuration key
default: Default value if key not found
"""
if isinstance(self, dict):
return self.get(key, default)
else:
return getattr(self, key, default)
class BackendConfig(BaseModel, ConfigMixin):
"""Base backend configuration"""
enabled: bool = Field(True, description="Whether the backend is enabled")
timeout: int = Field(30, ge=1, le=300, description="Timeout in seconds")
max_retries: int = Field(3, ge=0, le=10, description="Maximum retry attempts")
class ShellConfig(BackendConfig):
"""
Shell backend configuration
Attributes:
enabled: Whether shell backend is enabled
mode: Execution mode - "local" runs scripts in-process via subprocess,
"server" connects to a running local_server via HTTP
timeout: Default timeout for shell operations (seconds)
max_retries: Maximum number of retry attempts for failed operations
retry_interval: Wait time between retries (seconds)
default_shell: Path to default shell executable
working_dir: Default working directory for bash scripts
env: Default environment variables for shell operations
conda_env: Conda environment name to activate before execution (optional)
default_port: Default port for shell server connection (only used in server mode)
"""
mode: Literal["local", "server"] = Field("local", description="Execution mode: 'local' (in-process subprocess) or 'server' (HTTP local_server)")
retry_interval: float = Field(3.0, ge=0.1, le=60.0, description="Wait time between retries in seconds")
default_shell: str = Field("/bin/bash", description="Default shell path")
working_dir: Optional[str] = Field(None, description="Default working directory for bash scripts")
env: Dict[str, str] = Field(default_factory=dict, description="Default environment variables")
conda_env: Optional[str] = Field(None, description="Conda environment name to activate (e.g., 'myenv')")
default_port: int = Field(5000, ge=1, le=65535, description="Default port for shell server")
@field_validator('default_shell')
@classmethod
def validate_shell(cls, v):
if not v or not isinstance(v, str):
raise ValueError("Shell path must be a non-empty string")
return v
@field_validator('working_dir')
@classmethod
def validate_working_dir(cls, v):
if v is not None and not isinstance(v, str):
raise ValueError("Working directory must be a string")
return v
class WebConfig(BackendConfig):
"""
Web backend configuration - AI Deep Research
Attributes:
enabled: Whether web backend is enabled
timeout: Default timeout for web operations (seconds)
max_retries: Maximum number of retry attempts
Note:
All web-specific parameters (API key, base URL) are loaded from
environment variables or use default values in WebSession:
- OPENROUTER_API_KEY: API key for deep research (required)
- Deep research base URL defaults to "https://openrouter.ai/api/v1"
"""
pass
class MCPConfig(BackendConfig):
"""MCP backend configuration"""
sandbox: bool = Field(False, description="Whether to enable sandbox")
auto_initialize: bool = Field(True, description="Whether to auto initialize")
eager_sessions: bool = Field(False, description="Whether to eagerly create sessions for all servers on initialization")
retry_interval: float = Field(2.0, ge=0.1, le=60.0, description="Wait time between retries in seconds")
servers: Dict[str, Dict[str, Any]] = Field(default_factory=dict, description="MCP servers configuration, loaded from config_mcp.json")
sse_read_timeout: float = Field(300.0, ge=1.0, le=3600.0, description="SSE read timeout in seconds for HTTP/Sandbox connectors")
class GUIConfig(BackendConfig):
"""
GUI backend configuration
Attributes:
mode: Execution mode - "local" runs GUI operations in-process,
"server" connects to a running local_server via HTTP
"""
mode: Literal["local", "server"] = Field("local", description="Execution mode: 'local' (in-process) or 'server' (HTTP local_server)")
retry_interval: float = Field(5.0, ge=0.1, le=60.0, description="Wait time between retries in seconds")
driver_type: str = Field("pyautogui", description="GUI driver type")
failsafe: bool = Field(False, description="Whether to enable pyautogui failsafe mode")
screenshot_on_error: bool = Field(True, description="Whether to capture screenshot on error")
pkgs_prefix: str = Field(
"import pyautogui; import time; pyautogui.FAILSAFE = {failsafe}; {command}",
description="Python command prefix for pyautogui setup"
)
class ToolSearchConfig(BaseModel):
"""Tool search and ranking configuration"""
embedding_model: str = Field(
"BAAI/bge-small-en-v1.5",
description="Embedding model name for semantic search"
)
max_tools: int = Field(
20,
ge=1,
le=1000,
description="Maximum number of tools to return from search"
)
search_mode: str = Field(
"hybrid",
description="Default search mode: semantic, keyword, or hybrid"
)
enable_llm_filter: bool = Field(
True,
description="Whether to use LLM for backend/server filtering"
)
llm_filter_threshold: int = Field(
50,
ge=1,
le=1000,
description="Only apply LLM filter when tool count exceeds this threshold"
)
enable_cache_persistence: bool = Field(
False,
description="Whether to persist embeddings to disk"
)
cache_dir: Optional[str] = Field(
None,
description="Directory for embedding cache. None means use default <project_root>/.anytool/embedding_cache"
)
@field_validator('search_mode')
@classmethod
def validate_search_mode(cls, v):
valid_modes = ['semantic', 'keyword', 'hybrid']
if v.lower() not in valid_modes:
raise ValueError(f"Search mode must be one of {valid_modes}, got: {v}")
return v.lower()
class ToolQualityConfig(BaseModel):
"""Tool quality tracking configuration"""
enabled: bool = Field(
True,
description="Whether to enable tool quality tracking"
)
enable_persistence: bool = Field(
True,
description="Whether to persist quality data to disk"
)
cache_dir: Optional[str] = Field(
None,
description="Directory for quality cache. None means use default <project_root>/.anytool/tool_quality"
)
auto_evaluate_descriptions: bool = Field(
True,
description="Whether to automatically evaluate tool descriptions using LLM"
)
enable_quality_ranking: bool = Field(
True,
description="Whether to incorporate quality scores in tool ranking"
)
evolve_interval: int = Field(
5,
ge=1,
le=100,
description="Trigger quality evolution every N tool executions"
)
class GroundingConfig(BaseModel):
"""
Main configuration for Grounding module.
Contains configuration for all grounding backends and grounding-level settings.
Note: Local server connection uses defaults or environment variables (LOCAL_SERVER_URL).
"""
# Backend configurations
shell: ShellConfig = Field(default_factory=ShellConfig)
web: WebConfig = Field(default_factory=WebConfig)
mcp: MCPConfig = Field(default_factory=MCPConfig)
gui: GUIConfig = Field(default_factory=GUIConfig)
system: BackendConfig = Field(default_factory=BackendConfig)
# Grounding-level settings
tool_search: ToolSearchConfig = Field(default_factory=ToolSearchConfig)
tool_quality: ToolQualityConfig = Field(default_factory=ToolQualityConfig)
enabled_backends: List[Dict[str, str]] = Field(
default_factory=list,
description="List of enabled backends, each item: {'name': str, 'provider_cls': str}"
)
session_defaults: SessionConfig = Field(
default_factory=lambda: SessionConfig(
session_name="",
backend_type=BackendType.SHELL,
timeout=30,
auto_reconnect=True,
health_check_interval=30
)
)
tool_cache_ttl: int = Field(
300,
ge=1,
le=3600,
description="Tool cache time-to-live in seconds"
)
tool_cache_maxsize: int = Field(
300,
ge=1,
le=10000,
description="Maximum number of tool cache entries"
)
debug: bool = Field(False, description="Debug mode")
log_level: str = Field("INFO", description="Log level")
security_policies: Dict[str, Any] = Field(default_factory=dict)
@field_validator('log_level')
@classmethod
def validate_log_level(cls, v):
if v.upper() not in LOG_LEVELS:
raise ValueError(f"Log level must be one of {LOG_LEVELS}, got: {v}")
return v.upper()
def get_backend_config(self, backend_type: str) -> BackendConfig:
"""Get configuration for specified backend"""
name = backend_type.lower()
if not hasattr(self, name):
from anytool.utils.logging import Logger
logger = Logger.get_logger(__name__)
logger.warning(f"Unknown backend type: {backend_type}")
return BackendConfig()
return getattr(self, name)
def get_security_policy(self, backend_type: str) -> SecurityPolicy:
global_policy = self.security_policies.get("global", {})
backend_policy = self.security_policies.get("backend", {}).get(backend_type.lower(), {})
merged_policy = {**global_policy, **backend_policy}
return SecurityPolicy.from_dict(merged_policy)
__all__ = [
"BackendConfig",
"ShellConfig",
"WebConfig",
"MCPConfig",
"GUIConfig",
"ToolSearchConfig",
"ToolQualityConfig",
"GroundingConfig",
]
================================================
FILE: anytool/config/loader.py
================================================
import threading
from pathlib import Path
from typing import Union, Iterable, Dict, Any, Optional
from .grounding import GroundingConfig
from .constants import (
CONFIG_GROUNDING,
CONFIG_SECURITY,
CONFIG_DEV,
CONFIG_MCP,
CONFIG_AGENTS
)
from anytool.utils.logging import Logger
from .utils import load_json_file, save_json_file as save_json
logger = Logger.get_logger(__name__)
CONFIG_DIR = Path(__file__).parent
# Global configuration singleton
_config: GroundingConfig | None = None
_config_lock = threading.RLock() # Use RLock to support recursive locking
def _deep_merge_dict(base: dict, update: dict) -> dict:
"""Deep merge two dictionaries, update's values will override base's values"""
result = base.copy()
for key, value in update.items():
if key in result and isinstance(result[key], dict) and isinstance(value, dict):
result[key] = _deep_merge_dict(result[key], value)
else:
result[key] = value
return result
def _load_json_file(path: Path) -> Dict[str, Any]:
"""Load single JSON configuration file.
This function wraps the generic load_json_file and adds global configuration specific error handling and logging.
"""
if not path.exists():
logger.debug(f"Configuration file does not exist, skipping: {path}")
return {}
try:
data = load_json_file(path)
logger.info(f"Loaded configuration file: {path}")
return data
except Exception as e:
logger.warning(f"Failed to load configuration file {path}: {e}")
return {}
def _load_multiple_files(paths: Iterable[Path]) -> Dict[str, Any]:
"""Load configuration from multiple files"""
merged = {}
for path in paths:
data = _load_json_file(path)
if data:
merged = _deep_merge_dict(merged, data)
return merged
def load_config(*config_paths: Union[str, Path]) -> GroundingConfig:
"""
Load configuration files
"""
global _config
with _config_lock:
if config_paths:
paths = [Path(p) for p in config_paths]
else:
paths = [
CONFIG_DIR / CONFIG_GROUNDING,
CONFIG_DIR / CONFIG_SECURITY,
CONFIG_DIR / CONFIG_DEV, # Optional: development environment configuration
]
# Load and merge configuration
raw_data = _load_multiple_files(paths)
# Load MCP configuration (separate processing)
# Check if mcpServers already provided in merged custom configs
has_custom_mcp_servers = "mcpServers" in raw_data
if has_custom_mcp_servers:
# Use mcpServers from custom config
if "mcp" not in raw_data:
raw_data["mcp"] = {}
raw_data["mcp"]["servers"] = raw_data.pop("mcpServers")
logger.debug(f"Using custom MCP servers from provided config ({len(raw_data['mcp']['servers'])} servers)")
else:
# Load default MCP servers from config_mcp.json
mcp_data = _load_json_file(CONFIG_DIR / CONFIG_MCP)
if mcp_data and "mcpServers" in mcp_data:
if "mcp" not in raw_data:
raw_data["mcp"] = {}
raw_data["mcp"]["servers"] = mcp_data["mcpServers"]
logger.debug(f"Loaded MCP servers from default config_mcp.json ({len(raw_data['mcp']['servers'])} servers)")
# Validate and create configuration object
try:
_config = GroundingConfig.model_validate(raw_data)
except Exception as e:
logger.error(f"Validation failed, using default configuration: {e}")
_config = GroundingConfig()
# Adjust log level according to configuration
if _config.debug:
Logger.set_debug(2)
elif _config.log_level:
try:
Logger.configure(level=_config.log_level)
except Exception as e:
logger.warning(f"Failed to set log level {_config.log_level}: {e}")
return _config
def get_config() -> GroundingConfig:
"""
Get global configuration instance.
Usage:
- Get configuration in Provider: get_config().get_backend_config('shell')
- Get security policy in Tool: get_config().get_security_policy('shell')
"""
global _config
if _config is None:
with _config_lock:
if _config is None:
load_config()
return _config
def reset_config() -> None:
"""Reset configuration (for testing)"""
global _config
with _config_lock:
_config = None
def save_config(config: GroundingConfig, path: Union[str, Path]) -> None:
save_json(config.model_dump(), path)
logger.info(f"Configuration saved to: {path}")
def load_agents_config() -> Dict[str, Any]:
agents_config_path = CONFIG_DIR / CONFIG_AGENTS
return _load_json_file(agents_config_path)
def get_agent_config(agent_name: str) -> Optional[Dict[str, Any]]:
"""
Get the configuration of the specified agent
"""
agents_config = load_agents_config()
if "agents" not in agents_config:
logger.warning(f"No 'agents' key found in {CONFIG_AGENTS}")
return None
for agent_cfg in agents_config.get("agents", []):
if agent_cfg.get("name") == agent_name:
return agent_cfg
logger.warning(f"Agent '{agent_name}' not found in {CONFIG_AGENTS}")
return None
__all__ = [
"CONFIG_DIR",
"load_config",
"get_config",
"reset_config",
"save_config",
"load_agents_config",
"get_agent_config"
]
================================================
FILE: anytool/config/utils.py
================================================
import json
from pathlib import Path
from typing import Any
def get_config_value(config: Any, key: str, default=None):
if isinstance(config, dict):
return config.get(key, default)
else:
return getattr(config, key, default)
def load_json_file(filepath: str | Path) -> dict[str, Any]:
filepath = Path(filepath) if isinstance(filepath, str) else filepath
with open(filepath, 'r', encoding='utf-8') as f:
return json.load(f)
def save_json_file(data: dict[str, Any], filepath: str | Path, indent: int = 2) -> None:
filepath = Path(filepath) if isinstance(filepath, str) else filepath
# Ensure directory exists
filepath.parent.mkdir(parents=True, exist_ok=True)
with open(filepath, 'w', encoding='utf-8') as f:
json.dump(data, f, indent=indent, ensure_ascii=False)
__all__ = ["get_config_value", "load_json_file", "save_json_file"]
================================================
FILE: anytool/grounding/backends/__init__.py
================================================
# Use lazy imports to avoid loading all backends unconditionally
def _lazy_import_provider(provider_name: str):
"""Lazy import provider class"""
if provider_name == 'mcp':
from .mcp.provider import MCPProvider
return MCPProvider
elif provider_name == 'shell':
from .shell.provider import ShellProvider
return ShellProvider
elif provider_name == 'web':
from .web.provider import WebProvider
return WebProvider
elif provider_name == 'gui':
from .gui.provider import GUIProvider
return GUIProvider
else:
raise ImportError(f"Unknown provider: {provider_name}")
class _ProviderRegistry:
"""Lazy provider registry"""
def __getitem__(self, key):
return _lazy_import_provider(key)
def __contains__(self, key):
return key in ['mcp', 'shell', 'web', 'gui']
BACKEND_PROVIDERS = _ProviderRegistry()
__all__ = [
'BACKEND_PROVIDERS',
'_lazy_import_provider'
]
================================================
FILE: anytool/grounding/backends/gui/__init__.py
================================================
from .provider import GUIProvider
from .session import GUISession
from .transport.connector import GUIConnector
from .transport.local_connector import LocalGUIConnector
try:
from .anthropic_client import AnthropicGUIClient
from . import anthropic_utils
_anthropic_available = True
except ImportError:
_anthropic_available = False
__all__ = [
# Core Provider and Session
"GUIProvider",
"GUISession",
# Transport layer
"GUIConnector",
"LocalGUIConnector",
]
# Add Anthropic modules to exports if available
if _anthropic_available:
__all__.extend(["AnthropicGUIClient", "anthropic_utils"])
================================================
FILE: anytool/grounding/backends/gui/anthropic_client.py
================================================
import base64
import os
import time
from typing import Any, Dict, Optional, Tuple, List
from anytool.utils.logging import Logger
from PIL import Image
import io
logger = Logger.get_logger(__name__)
try:
from anthropic import (
Anthropic,
AnthropicBedrock,
AnthropicVertex,
APIError,
APIResponseValidationError,
APIStatusError,
)
from anthropic.types.beta import (
BetaMessageParam,
BetaTextBlockParam,
)
ANTHROPIC_AVAILABLE = True
except ImportError:
logger.warning("Anthropic SDK not available. Install with: pip install anthropic")
ANTHROPIC_AVAILABLE = False
# Import utility functions
from .anthropic_utils import (
APIProvider,
PROVIDER_TO_DEFAULT_MODEL_NAME,
COMPUTER_USE_BETA_FLAG,
PROMPT_CACHING_BETA_FLAG,
get_system_prompt,
inject_prompt_caching,
maybe_filter_to_n_most_recent_images,
response_to_params,
)
# API retry configuration
API_RETRY_TIMES = 10
API_RETRY_INTERVAL = 5 # seconds
class AnthropicGUIClient:
"""
Anthropic LLM Client for GUI operations.
Uses Claude Sonnet 4.5 with computer-use-2025-01-24 API.
Features:
- Vision-based screen understanding
- Automatic screenshot resizing (configurable display size)
- Coordinate scaling between display and actual screen
"""
def __init__(
self,
model: str = "claude-sonnet-4-5",
platform: str = "Ubuntu",
api_key: Optional[str] = None,
provider: str = "anthropic",
max_tokens: int = 4096,
screen_size: Tuple[int, int] = (1920, 1080),
display_size: Tuple[int, int] = (1024, 768), # Computer use display size
pyautogui_size: Optional[Tuple[int, int]] = None, # PyAutoGUI working size
only_n_most_recent_images: int = 3,
enable_prompt_caching: bool = True,
backup_api_key: Optional[str] = None,
):
"""
Initialize Anthropic GUI Client for Claude Sonnet 4.5.
Args:
model: Model name (only "claude-sonnet-4-5" supported)
platform: Platform type (Ubuntu, Windows, or macOS)
api_key: Anthropic API key (defaults to ANTHROPIC_API_KEY env var)
provider: API provider (only "anthropic" supported)
max_tokens: Maximum tokens for response
screen_size: Actual screenshot resolution (width, height) - physical pixels
display_size: Display size for computer use tool (width, height)
Screenshots will be resized to this size before sending to API
pyautogui_size: PyAutoGUI working size (logical pixels). If None, assumed same as screen_size.
On Retina/HiDPI displays, this may be screen_size / 2
only_n_most_recent_images: Number of recent screenshots to keep in history
enable_prompt_caching: Whether to enable prompt caching for cost optimization
backup_api_key: Backup API key (defaults to ANTHROPIC_API_KEY_BACKUP env var)
"""
if not ANTHROPIC_AVAILABLE:
raise RuntimeError("Anthropic SDK not installed. Install with: pip install anthropic")
# Only support claude-sonnet-4-5
if model != "claude-sonnet-4-5":
logger.warning(f"Model '{model}' not supported. Using 'claude-sonnet-4-5'")
model = "claude-sonnet-4-5"
self.model = model
self.platform = platform
self.api_key = api_key or os.environ.get("ANTHROPIC_API_KEY")
if not self.api_key:
raise ValueError("Anthropic API key not provided. Set ANTHROPIC_API_KEY env var or pass api_key parameter")
# Backup API key for failover
self.backup_api_key = backup_api_key or os.environ.get("ANTHROPIC_API_KEY_BACKUP")
# Only support anthropic provider
if provider != "anthropic":
logger.warning(f"Provider '{provider}' not supported. Using 'anthropic'")
provider = "anthropic"
self.provider = APIProvider(provider)
self.max_tokens = max_tokens
self.screen_size = screen_size
self.display_size = display_size
self.pyautogui_size = pyautogui_size or screen_size # Default to screen_size if not specified
self.only_n_most_recent_images = only_n_most_recent_images
self.enable_prompt_caching = enable_prompt_caching
# Message history
self.messages: List[BetaMessageParam] = []
# Calculate resize factor for coordinate scaling
# Step 1: LLM coordinates (display_size) -> Physical pixels (screen_size)
# Step 2: Physical pixels -> PyAutoGUI logical pixels (pyautogui_size)
self.resize_factor = (
self.pyautogui_size[0] / display_size[0], # x scale factor
self.pyautogui_size[1] / display_size[1] # y scale factor
)
logger.info(
f"Initialized AnthropicGUIClient:\n"
f" Model: {model}\n"
f" Platform: {platform}\n"
f" Screen Size (physical): {screen_size}\n"
f" PyAutoGUI Size (logical): {self.pyautogui_size}\n"
f" Display Size (LLM): {display_size}\n"
f" Resize Factor (LLM->PyAutoGUI): {self.resize_factor}\n"
f" Prompt Caching: {enable_prompt_caching}"
)
def _create_client(self, api_key: Optional[str] = None):
"""Create Anthropic client (only supports anthropic provider)."""
key = api_key or self.api_key
return Anthropic(api_key=key, max_retries=4)
def _resize_screenshot(self, screenshot_bytes: bytes) -> bytes:
"""
Resize screenshot to display size for Computer Use API.
For computer-use-2025-01-24, the screenshot must be resized to the
display_width_px x display_height_px specified in the tool definition.
"""
screenshot_image = Image.open(io.BytesIO(screenshot_bytes))
resized_image = screenshot_image.resize(self.display_size, Image.Resampling.LANCZOS)
output_buffer = io.BytesIO()
resized_image.save(output_buffer, format='PNG')
return output_buffer.getvalue()
def _scale_coordinates(self, x: int, y: int) -> Tuple[int, int]:
"""
Scale coordinates from display size to actual screen size.
The API returns coordinates in display_size (e.g., 1024x768).
We need to scale them to actual screen_size (e.g., 1920x1080) for execution.
Args:
x, y: Coordinates in display size space
Returns:
Scaled coordinates in actual screen size space
"""
scaled_x = int(x * self.resize_factor[0])
scaled_y = int(y * self.resize_factor[1])
return scaled_x, scaled_y
async def plan_action(
self,
task_description: str,
screenshot: bytes,
action_history: List[Dict[str, Any]] = None,
) -> Tuple[Optional[str], List[str]]:
"""
Plan next action based on task and current screenshot.
Includes prompt caching, error handling, and backup API key support.
Args:
task_description: Task to accomplish
screenshot: Current screenshot (PNG bytes)
action_history: Previous actions (for context)
Returns:
Tuple of (reasoning, list of pyautogui commands)
"""
# Resize screenshot
resized_screenshot = self._resize_screenshot(screenshot)
screenshot_b64 = base64.b64encode(resized_screenshot).decode('utf-8')
# Initialize messages with first task + screenshot
if not self.messages:
# IMPORTANT: Image should come BEFORE text for better model understanding
# This matches OSWorld's implementation which has proven effectiveness
self.messages.append({
"role": "user",
"content": [
{
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": screenshot_b64,
},
},
{"type": "text", "text": task_description},
]
})
# Filter images BEFORE adding new screenshot to control message size
# This is critical to avoid exceeding the 25MB API limit
image_truncation_threshold = 10
if self.only_n_most_recent_images and len(self.messages) > 1:
# Reserve 1 slot for the screenshot we're about to add
maybe_filter_to_n_most_recent_images(
self.messages,
max(1, self.only_n_most_recent_images - 1),
min_removal_threshold=1, # More aggressive filtering
)
# Add tool result from previous action if exists
if self.messages and self.messages[-1]["role"] == "assistant":
last_content = self.messages[-1]["content"]
if isinstance(last_content, list) and any(
block.get("type") == "tool_use" for block in last_content
):
tool_use_id = next(
block["id"] for block in last_content
if block.get("type") == "tool_use"
)
self._add_tool_result(tool_use_id, "Success", resized_screenshot)
# Define tools and betas for claude-sonnet-4-5 with computer-use-2025-01-24
tools = [{
'name': 'computer',
'type': 'computer_20250124',
'display_width_px': self.display_size[0],
'display_height_px': self.display_size[1],
'display_number': 1
}]
betas = [COMPUTER_USE_BETA_FLAG]
# Prepare system prompt with optional caching
system = BetaTextBlockParam(
type="text",
text=get_system_prompt(self.platform)
)
# Enable prompt caching if supported and enabled
if self.enable_prompt_caching:
betas.append(PROMPT_CACHING_BETA_FLAG)
inject_prompt_caching(self.messages)
system["cache_control"] = {"type": "ephemeral"} # type: ignore
# Model name - use claude-sonnet-4-5 directly
model_name = "claude-sonnet-4-5"
# Enable thinking for complex computer use tasks
extra_body = {"thinking": {"type": "enabled", "budget_tokens": 2048}}
# Log request details for debugging
# Count current images in messages
total_images = sum(
1
for message in self.messages
for item in (message.get("content", []) if isinstance(message.get("content"), list) else [])
if isinstance(item, dict) and item.get("type") == "image"
)
tool_result_images = sum(
1
for message in self.messages
for item in (message.get("content", []) if isinstance(message.get("content"), list) else [])
if isinstance(item, dict) and item.get("type") == "tool_result"
for content in item.get("content", [])
if isinstance(content, dict) and content.get("type") == "image"
)
logger.info(
f"Anthropic API request:\n"
f" Model: {model_name}\n"
f" Display Size: {self.display_size}\n"
f" Betas: {betas}\n"
f" Images: {total_images} ({tool_result_images} in tool_results)\n"
f" Messages: {len(self.messages)}"
)
# Try API call with retry and backup
client = self._create_client()
response = None
try:
# Retry loop with automatic image count reduction on 25MB error
for attempt in range(API_RETRY_TIMES):
try:
response = client.beta.messages.create(
max_tokens=self.max_tokens,
messages=self.messages,
model=model_name,
system=[system],
tools=tools,
betas=betas,
extra_body=extra_body
)
logger.info(f"API call succeeded on attempt {attempt + 1}")
break
except (APIError, APIStatusError, APIResponseValidationError) as e:
error_msg = str(e)
logger.warning(f"Anthropic API error (attempt {attempt+1}/{API_RETRY_TIMES}): {error_msg}")
# Handle 25MB payload limit error (including HTTP 413)
if ("25000000" in error_msg or
"Member must have length less than or equal to" in error_msg or
"request_too_large" in error_msg or
"413" in str(e)):
logger.warning("Detected 25MB limit error, reducing image count")
current_count = self.only_n_most_recent_images
new_count = max(1, current_count // 2)
self.only_n_most_recent_images = new_count
maybe_filter_to_n_most_recent_images(
self.messages,
new_count,
min_removal_threshold=1, # Aggressive filtering when hitting limit
)
logger.info(f"Image count reduced from {current_count} to {new_count}")
if attempt < API_RETRY_TIMES - 1:
time.sleep(API_RETRY_INTERVAL)
else:
raise
except (APIError, APIStatusError, APIResponseValidationError) as e:
logger.error(f"Primary API key failed: {e}")
# Try backup API key if available
if self.backup_api_key:
logger.warning("Retrying with backup API key...")
try:
backup_client = self._create_client(self.backup_api_key)
response = backup_client.beta.messages.create(
max_tokens=self.max_tokens,
messages=self.messages,
model=model_name,
system=[system],
tools=tools,
betas=betas,
extra_body=extra_body
)
logger.info("Successfully used backup API key")
except Exception as backup_e:
logger.error(f"Backup API key also failed: {backup_e}")
return None, ["FAIL"]
else:
return None, ["FAIL"]
except Exception as e:
logger.error(f"Unexpected error: {e}")
return None, ["FAIL"]
if not response:
return None, ["FAIL"]
# Parse response using utility function
response_params = response_to_params(response)
# Extract reasoning and commands
reasoning = ""
commands = []
for block in response_params:
block_type = block.get("type")
if block_type == "text":
reasoning = block.get("text", "")
elif block_type == "thinking":
reasoning = block.get("thinking", "")
elif block_type == "tool_use":
tool_input = block.get("input", {})
command = self._parse_computer_tool_use(tool_input)
if command:
commands.append(command)
else:
logger.warning(f"Failed to parse tool_use: {tool_input}")
# Store assistant response
self.messages.append({
"role": "assistant",
"content": response_params
})
logger.info(f"Parsed {len(commands)} commands from response")
return reasoning, commands
def _add_tool_result(
self,
tool_use_id: str,
result: str,
screenshot_bytes: Optional[bytes] = None
):
"""
Add tool result to message history.
IMPORTANT: Put screenshot BEFORE text for consistency with initial message.
"""
# Build content list with image first (if provided), then text
content_list = []
# Add screenshot first if provided (consistent with initial message ordering)
if screenshot_bytes is not None:
screenshot_b64 = base64.b64encode(screenshot_bytes).decode('utf-8')
content_list.append({
"type": "image",
"source": {
"type": "base64",
"media_type": "image/png",
"data": screenshot_b64
}
})
# Then add text result
content_list.append({"type": "text", "text": result})
tool_result_content = [{
"type": "tool_result",
"tool_use_id": tool_use_id,
"content": content_list
}]
self.messages.append({
"role": "user",
"content": tool_result_content
})
def _parse_computer_tool_use(self, tool_input: Dict[str, Any]) -> Optional[str]:
"""
Parse Anthropic computer tool use to pyautogui command.
Args:
tool_input: Tool input from Anthropic (action, coordinate, text, etc.)
Returns:
PyAutoGUI command string or control command (DONE, FAIL)
"""
action = tool_input.get("action")
if not action:
return None
# Action conversion
action_conversion = {
"left click": "click",
"right click": "right_click"
}
action = action_conversion.get(action, action)
text = tool_input.get("text")
coordinate = tool_input.get("coordinate")
scroll_direction = tool_input.get("scroll_direction")
scroll_amount = tool_input.get("scroll_amount", 5)
# Scale coordinates to actual screen size
if coordinate:
coordinate = self._scale_coordinates(coordinate[0], coordinate[1])
# Build commands
command = ""
if action == "mouse_move":
if coordinate:
x, y = coordinate
command = f"pyautogui.moveTo({x}, {y}, duration=0.5)"
elif action in ("left_click", "click"):
if coordinate:
x, y = coordinate
command = f"pyautogui.click({x}, {y})"
else:
command = "pyautogui.click()"
elif action == "right_click":
if coordinate:
x, y = coordinate
command = f"pyautogui.rightClick({x}, {y})"
else:
command = "pyautogui.rightClick()"
elif action == "double_click":
if coordinate:
x, y = coordinate
command = f"pyautogui.doubleClick({x}, {y})"
else:
command = "pyautogui.doubleClick()"
elif action == "middle_click":
if coordinate:
x, y = coordinate
command = f"pyautogui.middleClick({x}, {y})"
else:
command = "pyautogui.middleClick()"
elif action == "left_click_drag":
if coordinate:
x, y = coordinate
command = f"pyautogui.dragTo({x}, {y}, duration=0.5)"
elif action == "key":
if text:
keys = text.split('+')
# Key conversion
key_conversion = {
"page_down": "pagedown",
"page_up": "pageup",
"super_l": "win",
"super": "command",
"escape": "esc"
}
converted_keys = [key_conversion.get(k.strip().lower(), k.strip().lower()) for k in keys]
# Press and release keys
for key in converted_keys:
command += f"pyautogui.keyDown('{key}'); "
for key in reversed(converted_keys):
command += f"pyautogui.keyUp('{key}'); "
# Remove trailing semicolon and space
command = command.rstrip('; ')
elif action == "type":
if text:
command = f"pyautogui.typewrite({repr(text)}, interval=0.01)"
elif action == "scroll":
if scroll_direction in ("up", "down"):
scroll_value = scroll_amount if scroll_direction == "up" else -scroll_amount
if coordinate:
x, y = coordinate
command = f"pyautogui.scroll({scroll_value}, {x}, {y})"
else:
command = f"pyautogui.scroll({scroll_value})"
elif scroll_direction in ("left", "right"):
scroll_value = scroll_amount if scroll_direction == "right" else -scroll_amount
if coordinate:
x, y = coordinate
command = f"pyautogui.hscroll({scroll_value}, {x}, {y})"
else:
command = f"pyautogui.hscroll({scroll_value})"
elif action == "screenshot":
# Screenshot is automatically handled by the system
# Return special marker to indicate no action needed
return "SCREENSHOT"
elif action == "wait":
# Wait for specified duration
duration = tool_input.get("duration", 1)
command = f"pyautogui.sleep({duration})"
elif action == "done":
return "DONE"
elif action == "fail":
return "FAIL"
return command if command else None
def reset(self):
"""Reset message history."""
self.messages = []
logger.info("Reset AnthropicGUIClient message history")
================================================
FILE: anytool/grounding/backends/gui/anthropic_utils.py
================================================
from typing import List, cast
from enum import Enum
from datetime import datetime
from anytool.utils.logging import Logger
logger = Logger.get_logger(__name__)
try:
from anthropic.types.beta import (
BetaCacheControlEphemeralParam,
BetaContentBlockParam,
BetaImageBlockParam,
BetaMessage,
BetaMessageParam,
BetaTextBlock,
BetaTextBlockParam,
BetaToolResultBlockParam,
BetaToolUseBlockParam,
)
ANTHROPIC_AVAILABLE = True
except ImportError:
ANTHROPIC_AVAILABLE = False
# Beta flags
# For claude-sonnet-4-5 with computer-use-2025-01-24
COMPUTER_USE_BETA_FLAG = "computer-use-2025-01-24"
PROMPT_CACHING_BETA_FLAG = "prompt-caching-2024-07-31"
class APIProvider(Enum):
"""API Provider enumeration"""
ANTHROPIC = "anthropic"
# BEDROCK = "bedrock"
# VERTEX = "vertex"
# Provider to model name mapping (simplified for claude-sonnet-4-5 only)
PROVIDER_TO_DEFAULT_MODEL_NAME: dict = {
(APIProvider.ANTHROPIC, "claude-sonnet-4-5"): "claude-sonnet-4-5",
# (APIProvider.BEDROCK, "claude-sonnet-4-5"): "us.anthropic.claude-sonnet-4-5-v1:0",
# (APIProvider.VERTEX, "claude-sonnet-4-5"): "claude-sonnet-4-5-v1",
}
def get_system_prompt(platform: str = "Ubuntu") -> str:
"""
Get system prompt based on platform.
Args:
platform: Platform type (Ubuntu, Windows, macOS, or Darwin)
Returns:
System prompt string
"""
# Normalize platform name
platform_lower = platform.lower()
if platform_lower in ["windows", "win32"]:
return f"""<SYSTEM_CAPABILITY>
* You are utilising a Windows virtual machine using x86_64 architecture with internet access.
* You can use the computer tool to interact with the desktop: take screenshots, click, type, and control applications.
* To accomplish tasks, you MUST use the computer tool to see the screen and take actions.
* To open browser, please just click on the Chrome icon. Note, Chrome is what is installed on your system.
* When viewing a page it can be helpful to zoom out so that you can see everything on the page. Either that, or make sure you scroll down to see everything before deciding something isn't available.
* DO NOT ask users for clarification during task execution. DO NOT stop to request more information from users. Always take action using available tools.
* When using your computer function calls, they take a while to run and send back to you. Where possible/feasible, try to chain multiple of these calls all into one function calls request.
* The current date is {datetime.today().strftime('%A, %B %d, %Y')}.
* Home directory of this Windows system is 'C:\\Users\\user'.
* When you want to open some applications on Windows, please use Double Click on it instead of clicking once.
* After each action, the system will provide you with a new screenshot showing the result.
* Continue taking actions until the task is complete.
</SYSTEM_CAPABILITY>"""
elif platform_lower in ["macos", "darwin", "mac"]:
return f"""<SYSTEM_CAPABILITY>
* You are utilising a macOS system with internet access.
* You can use the computer tool to interact with the desktop: take screenshots, click, type, and control applications.
* To accomplish tasks, you MUST use the computer tool to see the screen and take actions.
* To open browser, please just click on the Chrome icon. Note, Chrome is what is installed on your system.
* When viewing a page it can be helpful to zoom out so that you can see everything on the page. Either that, or make sure you scroll down to see everything before deciding something isn't available.
* DO NOT ask users for clarification during task execution. DO NOT stop to request more information from users. Always take action using available tools.
* When using your computer function calls, they take a while to run and send back to you. Where possible/feasible, try to chain multiple of these calls all into one function calls request.
* The current date is {datetime.today().strftime('%A, %B %d, %Y')}.
* Home directory of this macOS system is typically '/Users/[username]' or can be accessed via '~'.
* On macOS, use Command (⌘) key combinations instead of Ctrl (e.g., Command+C for copy).
* After each action, the system will provide you with a new screenshot showing the result.
* Continue taking actions until the task is complete.
* When the task is completed, simply describe what you've done in your response WITHOUT using the tool again.
</SYSTEM_CAPABILITY>"""
else: # Ubuntu/Linux
return f"""<SYSTEM_CAPABILITY>
* You are utilising an Ubuntu virtual machine using x86_64 architecture with internet access.
* You can use the computer tool to interact with the desktop: take screenshots, click, type, and control applications.
* To accomplish tasks, you MUST use the computer tool to see the screen and take actions.
* To open browser, please just click on the Chrome icon. Note, Chrome is what is installed on your system.
* When viewing a page it can be helpful to zoom out so that you can see everything on the page. Either that, or make sure you scroll down to see everything before deciding something isn't available.
* DO NOT ask users for clarification during task execution. DO NOT stop to request more information from users. Always take action using available tools.
* When using your computer function calls, they take a while to run and send back to you. Where possible/feasible, try to chain multiple of these calls all into one function calls request.
* The current date is {datetime.today().strftime('%A, %B %d, %Y')}.
* Home directory of this Ubuntu system is '/home/user'.
* After each action, the system will provide you with a new screenshot showing the result.
* Continue taking actions until the task is complete.
</SYSTEM_CAPABILITY>"""
def inject_prompt_caching(messages: List[BetaMessageParam]) -> None:
"""
Set cache breakpoints for the 3 most recent turns.
One cache breakpoint is left for tools/system prompt, to be shared across sessions.
Args:
messages: Message history (modified in place)
"""
if not ANTHROPIC_AVAILABLE:
return
breakpoints_remaining = 3
for message in reversed(messages):
if message["role"] == "user" and isinstance(
content := message["content"], list
):
if breakpoints_remaining:
breakpoints_remaining -= 1
# Use type ignore to bypass TypedDict check until SDK types are updated
content[-1]["cache_control"] = BetaCacheControlEphemeralParam( # type: ignore
{"type": "ephemeral"}
)
else:
content[-1].pop("cache_control", None)
# we'll only ever have one extra turn per loop
break
def maybe_filter_to_n_most_recent_images(
messages: List[BetaMessageParam],
images_to_keep: int,
min_removal_threshold: int,
) -> None:
"""
With the assumption that images are screenshots that are of diminishing value as
the conversation progresses, remove all but the final `images_to_keep` tool_result
images in place, with a chunk of min_removal_threshold to reduce the amount we
break the implicit prompt cache.
Args:
messages: Message history (modified in place)
images_to_keep: Number of recent images to keep
min_removal_threshold: Minimum number of images to remove at once (for cache efficiency)
"""
if not ANTHROPIC_AVAILABLE or images_to_keep is None:
return
tool_result_blocks = cast(
list[BetaToolResultBlockParam],
[
item
for message in messages
for item in (
message["content"] if isinstance(message["content"], list) else []
)
if isinstance(item, dict) and item.get("type") == "tool_result"
],
)
total_images = sum(
1
for tool_result in tool_result_blocks
for content in tool_result.get("content", [])
if isinstance(content, dict) and content.get("type") == "image"
)
images_to_remove = total_images - images_to_keep
# for better cache behavior, we want to remove in chunks
images_to_remove -= images_to_remove % min_removal_threshold
for tool_result in tool_result_blocks:
if isinstance(tool_result.get("content"), list):
new_content = []
for content in tool_result.get("content", []):
if isinstance(content, dict) and content.get("type") == "image":
if images_to_remove > 0:
images_to_remove -= 1
continue
new_content.append(content)
tool_result["content"] = new_content
def response_to_params(response: BetaMessage) -> List[BetaContentBlockParam]:
"""
Convert Anthropic response to parameter list.
Handles both text blocks, tool use blocks, and thinking blocks.
Args:
response: Anthropic API response
Returns:
List of content blocks
"""
if not ANTHROPIC_AVAILABLE:
return []
res: List[BetaContentBlockParam] = []
if response.content:
for block in response.content:
# Check block type using type attribute
# Note: type may be a string or enum, so convert to string for comparison
block_type = str(getattr(block, "type", ""))
if block_type == "text":
# Regular text block
if isinstance(block, BetaTextBlock) and block.text:
res.append(BetaTextBlockParam(type="text", text=block.text))
elif block_type == "thinking":
# Thinking block (for Claude 4 and Sonnet 3.7)
thinking_block = {
"type": "thinking",
"thinking": getattr(block, "thinking", ""),
}
if hasattr(block, "signature"):
thinking_block["signature"] = getattr(block, "signature", None)
res.append(cast(BetaContentBlockParam, thinking_block))
elif block_type == "tool_use":
# Tool use block - only include required fields to avoid API errors
# (e.g., 'caller' field is not permitted by Anthropic API)
tool_use_dict = {
"type": "tool_use",
"id": block.id,
"name": block.name,
"input": block.input,
}
res.append(cast(BetaToolUseBlockParam, tool_use_dict))
else:
# Unknown block type - try to handle generically
try:
res.append(cast(BetaContentBlockParam, block.model_dump()))
except Exception as e:
logger.warning(f"Failed to parse block type {block_type}: {e}")
return res
else:
return []
================================================
FILE: anytool/grounding/backends/gui/config.py
================================================
from typing import Dict, Any, Optional
import os
import platform as platform_module
from anytool.utils.logging import Logger
logger = Logger.get_logger(__name__)
def build_llm_config(user_config: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
"""
Build complete LLM configuration with auto-detection and environment variables.
Auto-detects:
- API key from environment variables (ANTHROPIC_API_KEY)
- Platform from system (macOS/Windows/Ubuntu)
- Provider defaults to 'anthropic'
User-provided config values will override auto-detected values.
Args:
user_config: User-provided configuration (optional)
Returns:
Complete LLM configuration dict
Example:
>>> # Auto-detect everything
>>> config = build_llm_config()
>>> # Override specific values
>>> config = build_llm_config({
... "model": "claude-3-5-sonnet-20241022",
... "max_tokens": 8192
... })
"""
if user_config is None:
user_config = {}
# Auto-detect platform
system = platform_module.system()
if system == "Darwin":
detected_platform = "macOS"
elif system == "Windows":
detected_platform = "Windows"
else: # Linux
detected_platform = "Ubuntu"
# Auto-detect API key from environment
api_key = os.environ.get("ANTHROPIC_API_KEY")
if not api_key:
logger.warning(
"ANTHROPIC_API_KEY not found in environment. "
"Please set it: export ANTHROPIC_API_KEY='your-key'"
)
# Build configuration with precedence: user_config > auto-detected > defaults
config = {
"type": user_config.get("type", "anthropic"),
"model": user_config.get("model", "claude-sonnet-4-5"),
"platform": user_config.get("platform", detected_platform),
"api_key": user_config.get("api_key", api_key),
"provider": user_config.get("provider", "anthropic"),
"max_tokens": user_config.get("max_tokens", 4096),
"only_n_most_recent_images": user_config.get("only_n_most_recent_images", 3),
"enable_prompt_caching": user_config.get("enable_prompt_caching", True),
}
# Optional: screen_size (will be auto-detected from screenshot later)
if "screen_size" in user_config:
config["screen_size"] = user_config["screen_size"]
logger.info(f"Built LLM config - Platform: {config['platform']}, Model: {config['model']}")
if config["api_key"]:
logger.info(f"API key loaded: {config['api_key'][:10]}...")
return config
================================================
FILE: anytool/grounding/backends/gui/provider.py
================================================
from typing import Dict, Any, Union
from anytool.grounding.core.types import BackendType, SessionConfig
from anytool.grounding.core.provider import Provider
from anytool.grounding.core.session import BaseSession
from anytool.config import get_config
from anytool.config.utils import get_config_value
from anytool.platform import get_local_server_config
from anytool.utils.logging import Logger
from .transport.connector import GUIConnector
from .transport.local_connector import LocalGUIConnector
from .session import GUISession
logger = Logger.get_logger(__name__)
class GUIProvider(Provider):
"""
Provider for GUI desktop environment.
Manages communication with desktop_env through HTTP API or local in-process execution.
Supports two modes:
- "local": Execute GUI operations directly in-process (no server needed)
- "server": Connect to a running local_server via HTTP API
Supports automatic default session creation:
- If no session exists, a default session will be created on first use
- Default session uses configuration from config file or environment
"""
DEFAULT_SID = BackendType.GUI.value
def __init__(self, config: Dict[str, Any] = None):
"""
Initialize GUI provider.
Args:
config: Provider configuration
"""
super().__init__(BackendType.GUI, config)
self.connectors: Dict[str, Union[GUIConnector, LocalGUIConnector]] = {}
async def initialize(self) -> None:
"""
Initialize the provider and create default session.
"""
if not self.is_initialized:
logger.info("Initializing GUI provider")
# Auto-create default session
await self.create_session(SessionConfig(
session_name=self.DEFAULT_SID,
backend_type=BackendType.GUI,
connection_params={}
))
self.is_initialized = True
async def create_session(self, session_config: SessionConfig) -> BaseSession:
"""
Create GUI session.
Args:
session_config: Session configuration
Returns:
GUISession instance
"""
# Load GUI backend configuration
gui_config = get_config().get_backend_config("gui")
# Determine execution mode: "local" or "server"
mode = getattr(gui_config, "mode", "local")
# Extract connection parameters
conn_params = session_config.connection_params
timeout = get_config_value(conn_params, 'timeout', gui_config.timeout)
retry_times = get_config_value(conn_params, 'retry_times', gui_config.max_retries)
retry_interval = get_config_value(conn_params, 'retry_interval', gui_config.retry_interval)
# Build pkgs_prefix with failsafe setting
failsafe_str = "True" if gui_config.failsafe else "False"
pkgs_prefix = get_config_value(
conn_params,
'pkgs_prefix',
gui_config.pkgs_prefix.format(failsafe=failsafe_str, command="{command}")
)
if mode == "local":
# ---------- LOCAL MODE ----------
logger.info("GUI backend using LOCAL mode (no server required)")
connector = LocalGUIConnector(
timeout=timeout,
retry_times=retry_times,
retry_interval=retry_interval,
pkgs_prefix=pkgs_prefix,
)
else:
# ---------- SERVER MODE ----------
logger.info("GUI backend using SERVER mode (connecting to local_server)")
local_server_config = get_local_server_config()
vm_ip = get_config_value(conn_params, 'vm_ip', local_server_config['host'])
server_port = get_config_value(conn_params, 'server_port', local_server_config['port'])
connector = GUIConnector(
vm_ip=vm_ip,
server_port=server_port,
timeout=timeout,
retry_times=retry_times,
retry_interval=retry_interval,
pkgs_prefix=pkgs_prefix,
)
# Create session
session = GUISession(
connector=connector,
session_id=session_config.session_name,
backend_type=BackendType.GUI,
config=session_config,
)
# Store connector and session
self.connectors[session_config.session_name] = connector
self._sessions[session_config.session_name] = session
logger.info(f"Created GUI session: {session_config.session_name} (mode={mode})")
return session
async def close_session(self, session_name: str) -> None:
"""
Close GUI session.
Args:
session_name: Name of the session to close
"""
if session_name in self._sessions:
session = self._sessions[session_name]
await session.disconnect()
del self._sessions[session_name]
if session_name in self.connectors:
connector = self.connectors[session_name]
await connector.disconnect()
del self.connectors[session_name]
logger.info(f"Closed GUI session: {session_name}")
================================================
FILE: anytool/grounding/backends/gui/session.py
================================================
from typing import Dict, Any, Union
import os
from anytool.grounding.core.session import BaseSession
from anytool.grounding.core.types import BackendType, SessionStatus, SessionConfig
from anytool.utils.logging import Logger
from .transport.connector import GUIConnector
from .transport.local_connector import LocalGUIConnector
from .tool import GUIAgentTool
from .config import build_llm_config
logger = Logger.get_logger(__name__)
class GUISession(BaseSession):
"""
Session for GUI desktop environment.
Manages connection and tools for GUI automation.
"""
def __init__(
self,
connector: Union[GUIConnector, LocalGUIConnector],
session_id: str,
backend_type: BackendType.GUI,
config: SessionConfig,
auto_connect: bool = True,
auto_initialize: bool = True,
):
"""
Initialize GUI session.
Args:
connector: GUI HTTP connector
session_id: Unique session identifier
backend_type: Backend type (GUI)
config: Session configuration
auto_connect: Auto-connect on context enter
auto_initialize: Auto-initialize on context enter
"""
super().__init__(
connector=connector,
session_id=session_id,
backend_type=backend_type,
auto_connect=auto_connect,
auto_initialize=auto_initialize,
)
self.config = config
self.gui_connector = connector
async def initialize(self) -> Dict[str, Any]:
"""
Initialize session: connect and discover tools.
Returns:
Session information dict
"""
logger.info(f"Initializing GUI session: {self.session_id}")
# Ensure connected
if not self.connector.is_connected:
await self.connect()
# Create LLM client if configured
llm_client = None
user_llm_config = self.config.connection_params.get("llm_config")
# Build complete LLM config with auto-detection
# If user provides llm_config, merge with auto-detected values
# If user doesn't provide llm_config, try to auto-build one if ANTHROPIC_API_KEY exists
if user_llm_config or os.environ.get("ANTHROPIC_API_KEY"):
llm_config = build_llm_config(user_llm_config)
if llm_config.get("type") == "anthropic":
# Check if API key is available
if not llm_config.get("api_key"):
logger.warning(
"Anthropic API key not found. Skipping LLM client initialization. "
"Set ANTHROPIC_API_KEY environment variable or provide api_key in llm_config."
)
else:
try:
from .anthropic_client import AnthropicGUIClient
# Detect actual screen size from screenshot (most accurate)
# PyAutoGUI may report logical resolution, but we need the actual screenshot size
try:
screenshot_bytes = await self.gui_connector.get_screenshot()
if screenshot_bytes:
from PIL import Image
import io
img = Image.open(io.BytesIO(screenshot_bytes))
actual_screen_size = img.size
logger.info(f"Auto-detected screen size from screenshot: {actual_screen_size}")
screen_size = actual_screen_size
else:
raise RuntimeError("Could not get screenshot")
except Exception as e:
# Fallback to pyautogui detection
actual_screen_size = await self.gui_connector.get_screen_size()
if actual_screen_size:
logger.info(f"Auto-detected screen size from pyautogui: {actual_screen_size}")
screen_size = actual_screen_size
else:
# Final fallback to configured value
screen_size = llm_config.get("screen_size", (1920, 1080))
logger.warning(f"Could not auto-detect screen size, using configured: {screen_size}")
# Detect PyAutoGUI working size (logical pixels)
pyautogui_size = await self.gui_connector.get_screen_size()
if pyautogui_size:
logger.info(f"PyAutoGUI working size (logical): {pyautogui_size}")
else:
# If we can't detect PyAutoGUI size, assume it's the same as screen size
pyautogui_size = screen_size
logger.warning(f"Could not detect PyAutoGUI size, assuming same as screen: {pyautogui_size}")
llm_client = AnthropicGUIClient(
model=llm_config["model"],
platform=llm_config["platform"],
api_key=llm_config["api_key"],
provider=llm_config["provider"],
screen_size=screen_size,
pyautogui_size=pyautogui_size,
max_tokens=llm_config["max_tokens"],
only_n_most_recent_images=llm_config["only_n_most_recent_images"],
)
logger.info(
f"Initialized Anthropic LLM client - "
f"Model: {llm_config['model']}, Platform: {llm_config['platform']}"
)
except Exception as e:
logger.warning(f"Failed to initialize Anthropic client: {e}")
# Get recording_manager from connection_params if available
recording_manager = self.config.connection_params.get("recording_manager")
# Create GUI Agent Tool
self.tools = [
GUIAgentTool(
connector=self.gui_connector,
llm_client=llm_client,
recording_manager=recording_manager
)
]
logger.info(f"Initialized GUI session with {len(self.tools)} tool(s)")
# Return session info
session_info = {
"session_id": self.session_id,
"backend_type": self.backend_type.value,
"vm_ip": self.gui_connector.vm_ip,
"server_port": self.gui_connector.server_port,
"num_tools": len(self.tools),
"tools": [tool.name for tool in self.tools],
"llm_client": "anthropic" if llm_client else "none",
}
return session_info
async def connect(self) -> None:
"""Connect to GUI desktop environment"""
if self.connector.is_connected:
return
self.status = SessionStatus.CONNECTING
logger.info(f"Connecting to desktop_env at {self.gui_connector.base_url}")
await self.connector.connect()
self.status = SessionStatus.CONNECTED
logger.info("Connected to desktop environment")
async def disconnect(self) -> None:
"""Disconnect from GUI desktop environment"""
if not self.connector.is_connected:
return
logger.info("Disconnecting from desktop environment")
await self.connector.disconnect()
self.status = SessionStatus.DISCONNECTED
logger.info("Disconnected from desktop environment")
@property
def is_connected(self) -> bool:
"""Check if session is connected"""
return self.connector.is_connected
================================================
FILE: anytool/grounding/backends/gui/tool.py
================================================
import base64
from typing import Any, Dict
from anytool.grounding.core.tool.base import BaseTool
from anytool.grounding.core.types import BackendType, ToolResult, ToolStatus
from .transport.connector import GUIConnector
from .transport.actions import ACTION_SPACE, KEYBOARD_KEYS
from anytool.utils.logging import Logger
logger = Logger.get_logger(__name__)
class GUIAgentTool(BaseTool):
"""
LLM-powered GUI Agent Tool.
This tool acts as an intelligent agent that:
- Takes a task description as input
- Observes the desktop via screenshot
- Uses LLM/VLM to understand and plan actions
- Outputs action space commands
- Executes actions through the connector
"""
_name = "gui_agent"
_description = """Vision-based GUI automation agent for tasks requiring graphical interface interaction.
Use this tool when the task involves:
- Operating desktop applications with graphical interfaces (browsers, editors, design tools, etc.)
- Tasks that require visual understanding of UI elements, layouts, or content
- Multi-step workflows that need click, drag, type, or other GUI interactions
- Scenarios where programmatic APIs or command-line tools are unavailable or insufficient
The agent observes screen state through screenshots, uses vision-language models to understand
the interface, plans appropriate actions, and executes GUI operations autonomously.
IMPORTANT - max_steps Parameter Guidelines:
- Simple tasks (1-2 actions): 15-20 steps
- Medium tasks (3-5 actions): 25-35 steps
- Complex tasks (6+ actions, like web navigation): 35-50 steps
- When uncertain, prefer larger values (35+) to avoid premature termination
- Default is 25, but increase for multi-step workflows
Input:
- task_description: Natural language task description
- max_steps: Maximum actions (default 25, increase for complex tasks)
Output: Task execution results with action history and completion status
"""
backend_type = BackendType.GUI
def __init__(self, connector: GUIConnector, llm_client=None, recording_manager=None, **kwargs):
"""
Initialize GUI Agent Tool.
Args:
connector: GUI connector for communication with desktop_env
llm_client: LLM/VLM client for vision-based planning (optional)
recording_manager: RecordingManager for recording intermediate steps (optional)
**kwargs: Additional arguments for BaseTool
"""
super().__init__(**kwargs)
self.connector = connector
self.llm_client = llm_client # Will be injected later
self.recording_manager = recording_manager # For recording intermediate steps
self.action_history = [] # Track executed actions
async def _arun(
self,
task_description: str,
max_steps: int = 50,
) -> ToolResult:
"""
Execute a GUI automation task using LLM planning.
This is the main entry point that:
1. Gets current screenshot
2. Uses LLM to plan next action based on task and screenshot
3. Executes the planned action
4. Repeats until task is complete or max_steps reached
Args:
task_description: Natural language description of the task
max_steps: Maximum number of actions to execute (default 25)
Recommended values based on task complexity:
- Simple (1-2 actions): 15-20
- Medium (3-5 actions): 25-35
- Complex (6+ actions, web navigation, multi-app): 35-50
When in doubt, use higher values to avoid premature termination
Returns:
ToolResult with task execution status
"""
if not task_description:
return ToolResult(
status=ToolStatus.ERROR,
error="task_description is required"
)
logger.info(f"Starting GUI task: {task_description}")
self.action_history = []
# Execute task with LLM planning loop
try:
result = await self._execute_task_with_planning(
task_description=task_description,
max_steps=max_steps,
)
return result
except Exception as e:
logger.error(f"Task execution failed: {e}")
return ToolResult(
status=ToolStatus.ERROR,
error=str(e),
metadata={
"task_description": task_description,
"actions_executed": len(self.action_history),
"action_history": self.action_history,
}
)
async def _execute_task_with_planning(
self,
task_description: str,
max_steps: int,
) -> ToolResult:
"""
Execute task with LLM-based planning loop.
Planning loop:
1. Observe: Get screenshot
2. Plan: LLM decides next action
3. Execute: Perform the action
4. Verify: Check if task is complete
5. Repeat until done or max_steps
Args:
task_description: Task to complete
max_steps: Maximum planning iterations
Returns:
ToolResult with execution details
"""
# Collect all screenshots for visual analysis
all_screenshots = []
# Collect intermediate steps
intermediate_steps = []
for step in range(max_steps):
logger.info(f"Planning step {step + 1}/{max_steps}")
# Step 1: Observe current state
screenshot = await self.connector.get_screenshot()
if not screenshot:
return ToolResult(
status=ToolStatus.ERROR,
error="Failed to get screenshot for planning",
metadata={"step": step, "action_history": self.action_history}
)
# Collect screenshot for visual analysis
all_screenshots.append(screenshot)
# Step 2: Plan next action using LLM
planned_action = await self._plan_next_action(
task_description=task_description,
screenshot=screenshot,
action_history=self.action_history,
)
# Check if task is complete
if planned_action["action_type"] == "DONE":
logger.info("Task marked as complete by LLM")
reasoning = planned_action.get("reasoning", "Task completed successfully")
intermediate_steps.append({
"step_number": step + 1,
"action": "DONE",
"reasoning": reasoning,
"status": "done",
})
return ToolResult(
status=ToolStatus.SUCCESS,
content=f"Task completed: {task_description}\n\nFinal state: {reasoning}",
metadata={
"steps_taken": step + 1,
"action_history": self.action_history,
"screenshots": all_screenshots,
"intermediate_steps": intermediate_steps,
"final_reasoning": reasoning,
}
)
# Check if task failed
if planned_action["action_type"] == "FAIL":
logger.warning("Task marked as failed by LLM")
reason = planned_action.get("reason", "Task cannot be completed")
intermediate_steps.append({
"step_number": step + 1,
"action": "FAIL",
"reasoning": planned_action.get("reasoning", ""),
"status": "failed",
})
return ToolResult(
status=ToolStatus.ERROR,
error=reason,
metadata={
"steps_taken": step + 1,
"action_history": self.action_history,
"screenshots": all_screenshots,
"intermediate_steps": intermediate_steps,
}
)
# Check if action is WAIT (screenshot observation, continue to next step)
if planned_action["action_type"] == "WAIT":
logger.info("Screenshot observation step, continuing planning loop")
intermediate_steps.append({
"step_number": step + 1,
"action": "WAIT",
"reasoning": planned_action.get("reasoning", ""),
"status": "observation",
})
continue
# Step 3: Execute the planned action
execution_result = await self._execute_planned_action(planned_action)
# Record action in history
self.action_history.append({
"step": step + 1,
"planned_action": planned_action,
"execution_result": execution_result,
})
intermediate_steps.append({
"step_number": step + 1,
"action": planned_action.get("action_type", "unknown"),
"reasoning": planned_action.get("reasoning", ""),
"status": execution_result.get("status", "unknown"),
})
# Check execution result
if execution_result.get("status") != "success":
logger.warning(f"Action execution failed: {execution_result.get('error')}")
# Continue to next iteration for retry planning
# Max steps reached
return ToolResult(
status=ToolStatus.ERROR,
error=f"Task incomplete after {max_steps} steps",
metadata={
"task_description": task_description,
"steps_taken": max_steps,
"action_history": self.action_history,
"screenshots": all_screenshots,
"intermediate_steps": intermediate_steps,
}
)
async def _plan_next_action(
self,
task_description: str,
screenshot: bytes,
action_history: list,
) -> Dict[str, Any]:
"""
Use LLM/VLM to plan the next action.
This method sends:
- Task description
- Current screenshot (vision input)
- Action history (context)
- Available ACTION_SPACE
And gets back a structured action plan.
Args:
task_description: The task to accomplish
screenshot: Current desktop screenshot (PNG/JPEG bytes)
action_history: Previously executed actions
Returns:
Dict with action_type and parameters
"""
if self.llm_client is None:
# Fallback: Simple heuristic or manual mode
logger.warning("No LLM client configured, using fallback mode")
return {
"action_type": "FAIL",
"reason": "LLM client not configured"
}
# Check if using Anthropic client
try:
from .anthropic_client import AnthropicGUIClient
is_anthropic = isinstance(self.llm_client, AnthropicGUIClient)
except ImportError:
is_anthropic = False
if is_anthropic:
# Use Anthropic client
try:
reasoning, commands = await self.llm_client.plan_action(
task_description=task_description,
screenshot=screenshot,
action_history=action_history,
)
if commands == ["FAIL"]:
return {
"action_type": "FAIL",
"reason": "Anthropic planning failed"
}
if commands == ["DONE"]:
return {
"action_type": "DONE",
"reasoning": reasoning
}
if commands == ["SCREENSHOT"]:
# Screenshot is automatically handled by system
# Continue to next planning step
logger.info("LLM requested screenshot (observation step)")
return {
"action_type": "WAIT",
"reasoning": reasoning or "Observing screen state"
}
# If no commands but has reasoning, task is complete
# (Anthropic returns text-only when task is done)
if not commands and reasoning:
logger.info("LLM returned text-only response, interpreting as task completion")
return {
"action_type": "DONE",
"reasoning": reasoning
}
# No commands and no reasoning = error
if not commands:
return {
"action_type": "FAIL",
"reason": "No commands generated and no completion message"
}
# Return first command (Anthropic returns pyautogui commands directly)
return {
"action_type": "PYAUTOGUI_COMMAND",
"command": commands[0],
"reasoning": reasoning
}
except Exception as e:
logger.error(f"Anthropic planning failed: {e}")
return {
"action_type": "FAIL",
"reason": f"Planning error: {str(e)}"
}
# Generic LLM client (for future integration with other LLMs)
# Encode screenshot to base64 for LLM
screenshot_b64 = base64.b64encode(screenshot).decode('utf-8')
# Prepare prompt for LLM
prompt = self._build_planning_prompt(
task_description=task_description,
action_history=action_history,
)
# Call LLM with vision input
try:
response = await self.llm_client.plan_action(
prompt=prompt,
image_base64=screenshot_b64,
action_space=ACTION_SPACE,
keyboard_keys=KEYBOARD_KEYS,
)
# Parse LLM response to action dict
action = self._parse_llm_response(response)
logger.info(f"LLM planned action: {action['action_type']}")
return action
except Exception as e:
logger.error(f"LLM planning failed: {e}")
return {
"action_type": "FAIL",
"reason": f"Planning error: {str(e)}"
}
def _build_planning_prompt(
self,
task_description: str,
action_history: list,
) -> str:
"""
Build prompt for LLM planning.
Args:
t
gitextract_sqfmt1l8/ ├── .gitignore ├── COMMUNICATION.md ├── LICENSE ├── README.md ├── anytool/ │ ├── __init__.py │ ├── __main__.py │ ├── agents/ │ │ ├── __init__.py │ │ ├── base.py │ │ └── grounding_agent.py │ ├── config/ │ │ ├── __init__.py │ │ ├── config_agents.json │ │ ├── config_dev.json.example │ │ ├── config_grounding.json │ │ ├── config_mcp.json.example │ │ ├── config_security.json │ │ ├── constants.py │ │ ├── grounding.py │ │ ├── loader.py │ │ └── utils.py │ ├── grounding/ │ │ ├── backends/ │ │ │ ├── __init__.py │ │ │ ├── gui/ │ │ │ │ ├── __init__.py │ │ │ │ ├── anthropic_client.py │ │ │ │ ├── anthropic_utils.py │ │ │ │ ├── config.py │ │ │ │ ├── provider.py │ │ │ │ ├── session.py │ │ │ │ ├── tool.py │ │ │ │ └── transport/ │ │ │ │ ├── actions.py │ │ │ │ ├── connector.py │ │ │ │ └── local_connector.py │ │ │ ├── mcp/ │ │ │ │ ├── __init__.py │ │ │ │ ├── client.py │ │ │ │ ├── config.py │ │ │ │ ├── installer.py │ │ │ │ ├── provider.py │ │ │ │ ├── session.py │ │ │ │ ├── tool_cache.py │ │ │ │ ├── tool_converter.py │ │ │ │ └── transport/ │ │ │ │ ├── connectors/ │ │ │ │ │ ├── __init__.py │ │ │ │ │ ├── base.py │ │ │ │ │ ├── http.py │ │ │ │ │ ├── sandbox.py │ │ │ │ │ ├── stdio.py │ │ │ │ │ ├── utils.py │ │ │ │ │ └── websocket.py │ │ │ │ └── task_managers/ │ │ │ │ ├── __init__.py │ │ │ │ ├── sse.py │ │ │ │ ├── stdio.py │ │ │ │ ├── streamable_http.py │ │ │ │ └── websocket.py │ │ │ ├── shell/ │ │ │ │ ├── __init__.py │ │ │ │ ├── provider.py │ │ │ │ ├── session.py │ │ │ │ └── transport/ │ │ │ │ ├── connector.py │ │ │ │ └── local_connector.py │ │ │ └── web/ │ │ │ ├── __init__.py │ │ │ ├── provider.py │ │ │ └── session.py │ │ └── core/ │ │ ├── exceptions.py │ │ ├── grounding_client.py │ │ ├── provider.py │ │ ├── quality/ │ │ │ ├── __init__.py │ │ │ ├── manager.py │ │ │ ├── store.py │ │ │ └── types.py │ │ ├── search_tools.py │ │ ├── security/ │ │ │ ├── __init__.py │ │ │ ├── e2b_sandbox.py │ │ │ ├── policies.py │ │ │ └── sandbox.py │ │ ├── session.py │ │ ├── system/ │ │ │ ├── __init__.py │ │ │ ├── provider.py │ │ │ └── tool.py │ │ ├── tool/ │ │ │ ├── __init__.py │ │ │ ├── base.py │ │ │ ├── local_tool.py │ │ │ └── remote_tool.py │ │ ├── transport/ │ │ │ ├── connectors/ │ │ │ │ ├── __init__.py │ │ │ │ ├── aiohttp_connector.py │ │ │ │ └── base.py │ │ │ └── task_managers/ │ │ │ ├── __init__.py │ │ │ ├── aiohttp_connection_manager.py │ │ │ ├── async_ctx.py │ │ │ ├── base.py │ │ │ ├── noop.py │ │ │ └── placeholder.py │ │ └── types.py │ ├── llm/ │ │ ├── __init__.py │ │ └── client.py │ ├── local_server/ │ │ ├── README.md │ │ ├── __init__.py │ │ ├── config.json │ │ ├── feature_checker.py │ │ ├── health_checker.py │ │ ├── main.py │ │ ├── platform_adapters/ │ │ │ ├── __init__.py │ │ │ ├── linux_adapter.py │ │ │ ├── macos_adapter.py │ │ │ ├── pyxcursor.py │ │ │ └── windows_adapter.py │ │ ├── requirements.txt │ │ ├── run.sh │ │ └── utils/ │ │ ├── __init__.py │ │ ├── accessibility.py │ │ └── screenshot.py │ ├── platform/ │ │ ├── __init__.py │ │ ├── config.py │ │ ├── recording.py │ │ ├── screenshot.py │ │ └── system_info.py │ ├── prompts/ │ │ ├── __init__.py │ │ └── grounding_agent_prompts.py │ ├── recording/ │ │ ├── __init__.py │ │ ├── action_recorder.py │ │ ├── manager.py │ │ ├── recorder.py │ │ ├── utils.py │ │ ├── video.py │ │ └── viewer.py │ ├── tool_layer.py │ └── utils/ │ ├── cli_display.py │ ├── display.py │ ├── logging.py │ ├── telemetry/ │ │ ├── __init__.py │ │ ├── events.py │ │ ├── telemetry.py │ │ └── utils.py │ ├── ui.py │ └── ui_integration.py ├── pyproject.toml └── requirements.txt
SYMBOL INDEX (1125 symbols across 96 files)
FILE: anytool/__init__.py
function __getattr__ (line 53) | def __getattr__(name: str) -> _Any:
function __dir__ (line 70) | def __dir__():
FILE: anytool/__main__.py
class UIManager (line 17) | class UIManager:
method __init__ (line 18) | def __init__(self, ui: Optional[AnyToolUI], ui_integration: Optional[U...
method start_live_display (line 23) | async def start_live_display(self):
method stop_live_display (line 37) | async def stop_live_display(self):
method print_summary (line 46) | def print_summary(self, result: dict):
method _suppress_logs (line 52) | def _suppress_logs(self):
method _restore_logs (line 59) | def _restore_logs(self):
function _execute_task (line 65) | async def _execute_task(anytool: AnyTool, query: str, ui_manager: UIMana...
function interactive_mode (line 73) | async def interactive_mode(anytool: AnyTool, ui_manager: UIManager):
function single_query_mode (line 107) | async def single_query_mode(anytool: AnyTool, query: str, ui_manager: UI...
function _print_status (line 112) | def _print_status(anytool: AnyTool):
function _create_argument_parser (line 142) | def _create_argument_parser() -> argparse.ArgumentParser:
function refresh_mcp_cache (line 184) | async def refresh_mcp_cache(config_path: Optional[str] = None):
function _load_config (line 322) | def _load_config(args) -> AnyToolConfig:
function _setup_ui (line 364) | def _setup_ui(args) -> tuple[Optional[AnyToolUI], Optional[UIIntegration]]:
function _initialize_anytool (line 375) | async def _initialize_anytool(config: AnyToolConfig, args) -> AnyTool:
function main (line 409) | async def main():
function run_main (line 462) | def run_main():
FILE: anytool/agents/base.py
class BaseAgent (line 17) | class BaseAgent(ABC):
method __init__ (line 18) | def __init__(
method name (line 48) | def name(self) -> str:
method grounding_client (line 52) | def grounding_client(self) -> Optional[GroundingClient]:
method backend_scope (line 57) | def backend_scope(self) -> List[str]:
method llm_client (line 61) | def llm_client(self) -> Optional[LLMClient]:
method llm_client (line 65) | def llm_client(self, client: LLMClient) -> None:
method recording_manager (line 69) | def recording_manager(self) -> Optional[RecordingManager]:
method step (line 74) | def step(self) -> int:
method status (line 78) | def status(self) -> str:
method process (line 82) | async def process(self, context: Dict[str, Any]) -> Dict[str, Any]:
method construct_messages (line 86) | def construct_messages(self, context: Dict[str, Any]) -> List[Dict[str...
method get_llm_response (line 93) | async def get_llm_response(
method response_to_dict (line 113) | def response_to_dict(self, response: str) -> Dict[str, Any]:
method increment_step (line 145) | def increment_step(self) -> None:
method _register_self (line 149) | def _register_self(cls) -> None:
method __repr__ (line 155) | def __repr__(self) -> str:
class AgentStatus (line 159) | class AgentStatus:
class AgentRegistry (line 166) | class AgentRegistry:
method register (line 175) | def register(cls, name: str, agent_cls: Type[BaseAgent]) -> None:
method get_cls (line 182) | def get_cls(cls, name: str) -> Type[BaseAgent]:
method list_registered (line 188) | def list_registered(cls) -> List[str]:
method clear (line 192) | def clear(cls) -> None:
FILE: anytool/agents/grounding_agent.py
class GroundingAgent (line 21) | class GroundingAgent(BaseAgent):
method __init__ (line 22) | def __init__(
method _truncate_messages (line 73) | def _truncate_messages(
method process (line 116) | async def process(self, context: Dict[str, Any]) -> Dict[str, Any]:
method _default_system_prompt (line 364) | def _default_system_prompt(self) -> str:
method construct_messages (line 368) | def construct_messages(
method _get_available_tools (line 414) | async def _get_available_tools(self, task_description: Optional[str]) ...
method _visual_analysis_callback (line 457) | async def _visual_analysis_callback(
method _enhance_result_with_visual_context (line 522) | async def _enhance_result_with_visual_context(
method _select_key_screenshots (line 623) | def _select_key_screenshots(
method _get_workspace_path (line 669) | def _get_workspace_path(self, context: Dict[str, Any]) -> Optional[str]:
method _scan_workspace_files (line 675) | def _scan_workspace_files(
method _check_workspace_artifacts (line 736) | async def _check_workspace_artifacts(self, context: Dict[str, Any]) ->...
method _build_iteration_feedback (line 781) | def _build_iteration_feedback(
method _remove_previous_guidance (line 804) | def _remove_previous_guidance(self, messages: List[Dict[str, Any]]) ->...
method _generate_final_summary (line 817) | async def _generate_final_summary(
method _build_final_result (line 878) | async def _build_final_result(
method _format_tool_executions (line 945) | def _format_tool_executions(self, all_tool_results: List[Dict]) -> Lis...
method _check_task_completion (line 997) | def _check_task_completion(self, messages: List[Dict]) -> bool:
method _extract_last_assistant_message (line 1004) | def _extract_last_assistant_message(self, messages: List[Dict]) -> str:
method _record_agent_execution (line 1010) | async def _record_agent_execution(
FILE: anytool/config/grounding.py
class ConfigMixin (line 17) | class ConfigMixin:
method get_value (line 20) | def get_value(self, key: str, default=None):
class BackendConfig (line 34) | class BackendConfig(BaseModel, ConfigMixin):
class ShellConfig (line 41) | class ShellConfig(BackendConfig):
method validate_shell (line 68) | def validate_shell(cls, v):
method validate_working_dir (line 75) | def validate_working_dir(cls, v):
class WebConfig (line 80) | class WebConfig(BackendConfig):
class MCPConfig (line 98) | class MCPConfig(BackendConfig):
class GUIConfig (line 108) | class GUIConfig(BackendConfig):
class ToolSearchConfig (line 127) | class ToolSearchConfig(BaseModel):
method validate_search_mode (line 164) | def validate_search_mode(cls, v):
class ToolQualityConfig (line 171) | class ToolQualityConfig(BaseModel):
class GroundingConfig (line 201) | class GroundingConfig(BaseModel):
method validate_log_level (line 253) | def validate_log_level(cls, v):
method get_backend_config (line 258) | def get_backend_config(self, backend_type: str) -> BackendConfig:
method get_security_policy (line 268) | def get_security_policy(self, backend_type: str) -> SecurityPolicy:
FILE: anytool/config/loader.py
function _deep_merge_dict (line 26) | def _deep_merge_dict(base: dict, update: dict) -> dict:
function _load_json_file (line 36) | def _load_json_file(path: Path) -> Dict[str, Any]:
function _load_multiple_files (line 53) | def _load_multiple_files(paths: Iterable[Path]) -> Dict[str, Any]:
function load_config (line 62) | def load_config(*config_paths: Union[str, Path]) -> GroundingConfig:
function get_config (line 118) | def get_config() -> GroundingConfig:
function reset_config (line 135) | def reset_config() -> None:
function save_config (line 141) | def save_config(config: GroundingConfig, path: Union[str, Path]) -> None:
function load_agents_config (line 146) | def load_agents_config() -> Dict[str, Any]:
function get_agent_config (line 151) | def get_agent_config(agent_name: str) -> Optional[Dict[str, Any]]:
FILE: anytool/config/utils.py
function get_config_value (line 6) | def get_config_value(config: Any, key: str, default=None):
function load_json_file (line 13) | def load_json_file(filepath: str | Path) -> dict[str, Any]:
function save_json_file (line 20) | def save_json_file(data: dict[str, Any], filepath: str | Path, indent: i...
FILE: anytool/grounding/backends/__init__.py
function _lazy_import_provider (line 3) | def _lazy_import_provider(provider_name: str):
class _ProviderRegistry (line 21) | class _ProviderRegistry:
method __getitem__ (line 23) | def __getitem__(self, key):
method __contains__ (line 26) | def __contains__(self, key):
FILE: anytool/grounding/backends/gui/anthropic_client.py
class AnthropicGUIClient (line 46) | class AnthropicGUIClient:
method __init__ (line 57) | def __init__(
method _create_client (line 141) | def _create_client(self, api_key: Optional[str] = None):
method _resize_screenshot (line 146) | def _resize_screenshot(self, screenshot_bytes: bytes) -> bytes:
method _scale_coordinates (line 160) | def _scale_coordinates(self, x: int, y: int) -> Tuple[int, int]:
method plan_action (line 177) | async def plan_action(
method _add_tool_result (line 403) | def _add_tool_result(
method _parse_computer_tool_use (line 442) | def _parse_computer_tool_use(self, tool_input: Dict[str, Any]) -> Opti...
method reset (line 572) | def reset(self):
FILE: anytool/grounding/backends/gui/anthropic_utils.py
class APIProvider (line 31) | class APIProvider(Enum):
function get_system_prompt (line 46) | def get_system_prompt(platform: str = "Ubuntu") -> str:
function inject_prompt_caching (line 106) | def inject_prompt_caching(messages: List[BetaMessageParam]) -> None:
function maybe_filter_to_n_most_recent_images (line 134) | def maybe_filter_to_n_most_recent_images(
function response_to_params (line 188) | def response_to_params(response: BetaMessage) -> List[BetaContentBlockPa...
FILE: anytool/grounding/backends/gui/config.py
function build_llm_config (line 9) | def build_llm_config(user_config: Optional[Dict[str, Any]] = None) -> Di...
FILE: anytool/grounding/backends/gui/provider.py
class GUIProvider (line 16) | class GUIProvider(Provider):
method __init__ (line 32) | def __init__(self, config: Dict[str, Any] = None):
method initialize (line 42) | async def initialize(self) -> None:
method create_session (line 56) | async def create_session(self, session_config: SessionConfig) -> BaseS...
method close_session (line 126) | async def close_session(self, session_name: str) -> None:
FILE: anytool/grounding/backends/gui/session.py
class GUISession (line 14) | class GUISession(BaseSession):
method __init__ (line 20) | def __init__(
method initialize (line 50) | async def initialize(self) -> Dict[str, Any]:
method connect (line 161) | async def connect(self) -> None:
method disconnect (line 174) | async def disconnect(self) -> None:
method is_connected (line 186) | def is_connected(self) -> bool:
FILE: anytool/grounding/backends/gui/tool.py
class GUIAgentTool (line 12) | class GUIAgentTool(BaseTool):
method __init__ (line 52) | def __init__(self, connector: GUIConnector, llm_client=None, recording...
method _arun (line 68) | async def _arun(
method _execute_task_with_planning (line 123) | async def _execute_task_with_planning(
method _plan_next_action (line 265) | async def _plan_next_action(
method _build_planning_prompt (line 397) | def _build_planning_prompt(
method _parse_llm_response (line 448) | def _parse_llm_response(self, response: str) -> Dict[str, Any]:
method _execute_planned_action (line 477) | async def _execute_planned_action(
method execute_action (line 531) | async def execute_action(
method get_screenshot (line 561) | async def get_screenshot(self) -> ToolResult:
method _record_intermediate_step (line 576) | async def _record_intermediate_step(
method _format_action_command (line 674) | def _format_action_command(self, planned_action: Dict[str, Any]) -> str:
FILE: anytool/grounding/backends/gui/transport/actions.py
function build_pyautogui_command (line 149) | def build_pyautogui_command(action_type: str, parameters: Dict[str, Any]...
FILE: anytool/grounding/backends/gui/transport/connector.py
class GUIConnector (line 11) | class GUIConnector(AioHttpConnector):
method __init__ (line 17) | def __init__(
method _retry_invoke (line 47) | async def _retry_invoke(
method _is_valid_image_response (line 94) | def _is_valid_image_response(content_type: str, data: Optional[bytes])...
method _fix_pyautogui_less_than_bug (line 110) | def _fix_pyautogui_less_than_bug(command: str) -> str:
method get_screen_size (line 178) | async def get_screen_size(self) -> Optional[tuple[int, int]]:
method get_screenshot (line 204) | async def get_screenshot(self) -> Optional[bytes]:
method execute_python_command (line 229) | async def execute_python_command(self, command: str) -> Optional[Dict[...
method execute_action (line 255) | async def execute_action(self, action_type: str, parameters: Dict[str,...
method get_accessibility_tree (line 325) | async def get_accessibility_tree(self, max_depth: int = 5) -> Optional...
method get_cursor_position (line 349) | async def get_cursor_position(self) -> Optional[tuple[int, int]]:
method invoke (line 367) | async def invoke(self, name: str, params: dict[str, Any]) -> Any:
FILE: anytool/grounding/backends/gui/transport/local_connector.py
class LocalGUIConnector (line 29) | class LocalGUIConnector(BaseConnector[Any]):
method __init__ (line 38) | def __init__(
method _get_screenshot_helper (line 60) | def _get_screenshot_helper(self):
method _get_accessibility_helper (line 66) | def _get_accessibility_helper(self):
method connect (line 76) | async def connect(self) -> None:
method _retry_invoke (line 87) | async def _retry_invoke(
method _fix_pyautogui_less_than_bug (line 127) | def _fix_pyautogui_less_than_bug(command: str) -> str:
method _is_valid_image_response (line 172) | def _is_valid_image_response(content_type: str, data: Optional[bytes])...
method get_screen_size (line 187) | async def get_screen_size(self) -> Optional[tuple[int, int]]:
method get_screenshot (line 206) | async def get_screenshot(self) -> Optional[bytes]:
method execute_python_command (line 227) | async def execute_python_command(self, command: str) -> Optional[Dict[...
method execute_action (line 258) | async def execute_action(
method get_accessibility_tree (line 319) | async def get_accessibility_tree(
method get_cursor_position (line 333) | async def get_cursor_position(self) -> Optional[tuple[int, int]]:
method invoke (line 349) | async def invoke(self, name: str, params: dict[str, Any]) -> Any:
method request (line 360) | async def request(self, *args: Any, **kwargs: Any) -> Any:
FILE: anytool/grounding/backends/mcp/client.py
class MCPClient (line 22) | class MCPClient:
method __init__ (line 29) | def __init__(
method _get_mcp_servers (line 80) | def _get_mcp_servers(self) -> dict[str, Any]:
method from_dict (line 94) | def from_dict(
method from_config_file (line 120) | def from_config_file(
method add_server (line 140) | def add_server(
method remove_server (line 158) | def remove_server(self, name: str) -> None:
method get_server_names (line 180) | def get_server_names(self) -> list[str]:
method save_config (line 188) | def save_config(self, filepath: str) -> None:
method create_session (line 196) | async def create_session(self, server_name: str, auto_initialize: bool...
method create_all_sessions (line 292) | async def create_all_sessions(
method get_session (line 324) | def get_session(self, server_name: str) -> MCPSession:
method get_all_active_sessions (line 342) | def get_all_active_sessions(self) -> dict[str, MCPSession]:
method close_session (line 350) | async def close_session(self, server_name: str) -> None:
method close_all_sessions (line 387) | async def close_all_sessions(self) -> None:
FILE: anytool/grounding/backends/mcp/config.py
function create_connector_from_config (line 29) | async def create_connector_from_config(
FILE: anytool/grounding/backends/mcp/installer.py
class MCPDependencyError (line 15) | class MCPDependencyError(RuntimeError):
class MCPCommandNotFoundError (line 20) | class MCPCommandNotFoundError(MCPDependencyError):
class MCPInstallationCancelledError (line 25) | class MCPInstallationCancelledError(MCPDependencyError):
class MCPInstallationFailedError (line 30) | class MCPInstallationFailedError(MCPDependencyError):
class Colors (line 35) | class Colors:
class MCPInstallerManager (line 47) | class MCPInstallerManager:
method __init__ (line 54) | def __init__(self, prompt: PromptFunc | None = None, auto_install: boo...
method _default_cli_prompt (line 68) | async def _default_cli_prompt(self, message: str) -> bool:
method _ask_user (line 92) | async def _ask_user(self, message: str) -> bool:
method _check_command_available (line 106) | def _check_command_available(self, command: str) -> bool:
method _check_package_installed (line 117) | async def _check_package_installed(self, command: str, args: List[str]...
method _extract_npm_package (line 163) | def _extract_npm_package(self, args: List[str]) -> Optional[str]:
method _extract_python_package (line 197) | def _extract_python_package(self, args: List[str]) -> Optional[str]:
method _extract_uv_package (line 234) | def _extract_uv_package(self, args: List[str]) -> Optional[str]:
method _check_npm_package (line 256) | async def _check_npm_package(self, package_name: str) -> bool:
method _check_python_package (line 279) | async def _check_python_package(self, package_name: str) -> bool:
method _check_uv_pip_package (line 326) | async def _check_uv_pip_package(self, package_name: str) -> bool:
method _install_package (line 368) | async def _install_package(self, command: str, args: List[str], use_su...
method _get_install_command (line 513) | def _get_install_command(self, command: str, args: List[str]) -> Optio...
method ensure_dependencies (line 539) | async def ensure_dependencies(
method _ensure_dependencies_impl (line 564) | async def _ensure_dependencies_impl(
function get_global_installer (line 687) | def get_global_installer() -> MCPInstallerManager:
function set_global_installer (line 694) | def set_global_installer(installer: MCPInstallerManager) -> None:
FILE: anytool/grounding/backends/mcp/provider.py
class MCPProvider (line 23) | class MCPProvider(Provider[MCPSession]):
method __init__ (line 31) | def __init__(self, config: Dict | None = None, installer: Optional[MCP...
method initialize (line 83) | async def initialize(self) -> None:
method list_servers (line 111) | def list_servers(self) -> List[str]:
method create_session (line 119) | async def create_session(self, session_config: SessionConfig) -> MCPSe...
method close_session (line 164) | async def close_session(self, session_name: str) -> None:
method list_tools (line 200) | async def list_tools(self, session_name: str | None = None, use_cache:...
method _load_tools_from_cache (line 248) | def _load_tools_from_cache(self) -> List[BaseTool]:
method _sanitize_and_save_cache (line 273) | def _sanitize_and_save_cache(
method _build_tools_from_cache (line 299) | def _build_tools_from_cache(
method _list_tools_live (line 330) | async def _list_tools_live(self) -> List[BaseTool]:
method _save_tools_to_cache (line 401) | async def _save_tools_to_cache(self, tools: List[BaseTool]) -> None:
method ensure_server_session (line 419) | async def ensure_server_session(self, server_name: str) -> Optional[MC...
method _lazy_create (line 442) | async def _lazy_create(self, server: str) -> None:
FILE: anytool/grounding/backends/mcp/session.py
class MCPSession (line 19) | class MCPSession(BaseSession):
method __init__ (line 26) | def __init__(
method initialize (line 50) | async def initialize(self) -> Dict[str, Any]:
FILE: anytool/grounding/backends/mcp/tool_cache.py
class MCPToolCache (line 18) | class MCPToolCache:
method __init__ (line 23) | def __init__(self, cache_path: Optional[Path] = None, sanitized_cache_...
method set_server_order (line 30) | def set_server_order(self, order: List[str]):
method _reorder_servers (line 34) | def _reorder_servers(self, servers: Dict[str, List[Dict]]) -> Dict[str...
method _ensure_dir (line 50) | def _ensure_dir(self):
method load (line 54) | def load(self) -> Dict[str, Any]:
method save (line 73) | def save(self, servers: Dict[str, List[Dict]]):
method save_server (line 97) | def save_server(self, server_name: str, tools: List[Dict]):
method get_server_tools (line 126) | def get_server_tools(self, server_name: str) -> Optional[List[Dict]]:
method get_all_tools (line 131) | def get_all_tools(self) -> Dict[str, List[Dict]]:
method has_cache (line 136) | def has_cache(self) -> bool:
method clear (line 141) | def clear(self):
method save_failed_server (line 148) | def save_failed_server(self, server_name: str, error: str):
method get_failed_servers (line 178) | def get_failed_servers(self) -> Dict[str, Dict]:
method load_sanitized (line 183) | def load_sanitized(self) -> Dict[str, Any]:
method save_sanitized (line 202) | def save_sanitized(self, servers: Dict[str, List[Dict]]):
method get_all_sanitized_tools (line 226) | def get_all_sanitized_tools(self) -> Dict[str, List[Dict]]:
method has_sanitized_cache (line 231) | def has_sanitized_cache(self) -> bool:
method clear_sanitized (line 236) | def clear_sanitized(self):
function get_tool_cache (line 248) | def get_tool_cache() -> MCPToolCache:
FILE: anytool/grounding/backends/mcp/tool_converter.py
function _sanitize_mcp_schema (line 19) | def _sanitize_mcp_schema(params: Dict[str, Any]) -> Dict[str, Any]:
function _deep_sanitize (line 54) | def _deep_sanitize(schema: Dict[str, Any]) -> Dict[str, Any]:
function convert_mcp_tool_to_base_tool (line 149) | def convert_mcp_tool_to_base_tool(
FILE: anytool/grounding/backends/mcp/transport/connectors/base.py
class MCPBaseConnector (line 26) | class MCPBaseConnector(BaseConnector[ClientSession]):
method __init__ (line 32) | def __init__(
method public_identifier (line 56) | def public_identifier(self) -> str:
method _get_streams_from_connection (line 60) | async def _get_streams_from_connection(self):
method _after_connect (line 66) | async def _after_connect(self) -> None:
method _before_disconnect (line 89) | async def _before_disconnect(self) -> None:
method _cleanup_on_connect_failure (line 113) | async def _cleanup_on_connect_failure(self) -> None:
method initialize (line 127) | async def initialize(self) -> dict[str, Any]:
method tools (line 169) | def tools(self) -> list[Tool]:
method resources (line 176) | def resources(self) -> list[Resource]:
method prompts (line 183) | def prompts(self) -> list[Prompt]:
method is_connected (line 190) | def is_connected(self) -> bool:
method _ensure_connected (line 218) | async def _ensure_connected(self) -> None:
method call_tool (line 240) | async def call_tool(self, name: str, arguments: dict[str, Any]) -> Cal...
method list_tools (line 295) | async def list_tools(self) -> list[Tool]:
method list_resources (line 309) | async def list_resources(self) -> list[Resource]:
method read_resource (line 322) | async def read_resource(self, uri: str) -> ReadResourceResult:
method list_prompts (line 331) | async def list_prompts(self) -> list[Prompt]:
method get_prompt (line 344) | async def get_prompt(self, name: str, arguments: dict[str, Any] | None...
method request (line 353) | async def request(self, method: str, params: dict[str, Any] | None = N...
method invoke (line 361) | async def invoke(self, name: str, params: dict[str, Any]) -> Any:
FILE: anytool/grounding/backends/mcp/transport/connectors/http.py
class HttpConnector (line 33) | class HttpConnector(MCPBaseConnector):
method __init__ (line 40) | def __init__(
method connect (line 85) | async def connect(self) -> None:
method disconnect (line 113) | async def disconnect(self) -> None:
method _before_connect (line 132) | async def _before_connect(self) -> None:
method _try_jsonrpc_connection (line 298) | async def _try_jsonrpc_connection(self) -> None:
method _after_connect (line 327) | async def _after_connect(self) -> None:
method _before_disconnect (line 346) | async def _before_disconnect(self) -> None:
method public_identifier (line 361) | def public_identifier(self) -> str:
method _next_jsonrpc_id (line 369) | def _next_jsonrpc_id(self) -> int:
method _send_jsonrpc_request (line 374) | async def _send_jsonrpc_request(
method _parse_tools_from_json (line 455) | def _parse_tools_from_json(self, tools_data: List[Dict]) -> List[Tool]:
method _parse_resources_from_json (line 470) | def _parse_resources_from_json(self, resources_data: List[Dict]) -> Li...
method _parse_prompts_from_json (line 486) | def _parse_prompts_from_json(self, prompts_data: List[Dict]) -> List[P...
method initialize (line 505) | async def initialize(self) -> Dict[str, Any]:
method is_connected (line 564) | def is_connected(self) -> bool:
method _ensure_connected (line 570) | async def _ensure_connected(self) -> None:
method list_tools (line 578) | async def list_tools(self) -> List[Tool]:
method call_tool (line 592) | async def call_tool(self, name: str, arguments: Dict[str, Any]) -> Cal...
method list_resources (line 631) | async def list_resources(self) -> List[Resource]:
method read_resource (line 645) | async def read_resource(self, uri: str) -> ReadResourceResult:
method list_prompts (line 654) | async def list_prompts(self) -> List[Prompt]:
method get_prompt (line 668) | async def get_prompt(self, name: str, arguments: Dict[str, Any] | None...
method request (line 680) | async def request(self, method: str, params: Dict[str, Any] | None = N...
method invoke (line 688) | async def invoke(self, name: str, params: Dict[str, Any]) -> Any:
FILE: anytool/grounding/backends/mcp/transport/connectors/sandbox.py
class SandboxConnector (line 23) | class SandboxConnector(MCPBaseConnector):
method __init__ (line 31) | def __init__(
method _handle_stdout (line 88) | def _handle_stdout(self, data: str) -> None:
method _handle_stderr (line 93) | def _handle_stderr(self, data: str) -> None:
method wait_for_server_response (line 98) | async def wait_for_server_response(self, base_url: str, timeout: int =...
method _before_connect (line 151) | async def _before_connect(self) -> None:
method _after_connect (line 198) | async def _after_connect(self) -> None:
method _before_disconnect (line 203) | async def _before_disconnect(self) -> None:
method _cleanup_on_connect_failure (line 226) | async def _cleanup_on_connect_failure(self) -> None:
method sandbox (line 244) | def sandbox(self) -> BaseSandbox:
method public_identifier (line 249) | def public_identifier(self) -> str:
FILE: anytool/grounding/backends/mcp/transport/connectors/stdio.py
class StdioConnector (line 19) | class StdioConnector(MCPBaseConnector):
method __init__ (line 27) | def __init__(
method _before_connect (line 64) | async def _before_connect(self) -> None:
method _after_connect (line 68) | async def _after_connect(self) -> None:
method public_identifier (line 75) | def public_identifier(self) -> dict[str, str]:
FILE: anytool/grounding/backends/mcp/transport/connectors/utils.py
function is_stdio_server (line 4) | def is_stdio_server(server_config: dict[str, Any]) -> bool:
FILE: anytool/grounding/backends/mcp/transport/connectors/websocket.py
class WebSocketConnector (line 24) | class WebSocketConnector(MCPBaseConnector):
method __init__ (line 31) | def __init__(
method _get_streams_from_connection (line 60) | async def _get_streams_from_connection(self):
method _after_connect (line 64) | async def _after_connect(self) -> None:
method _receive_messages (line 78) | async def _receive_messages(self) -> None:
method _before_disconnect (line 108) | async def _before_disconnect(self) -> None:
method _cleanup_on_connect_failure (line 145) | async def _cleanup_on_connect_failure(self) -> None:
method _send_request (line 169) | async def _send_request(self, method: str, params: dict[str, Any] | No...
method initialize (line 195) | async def initialize(self) -> dict[str, Any]:
method list_tools (line 207) | async def list_tools(self) -> list[dict[str, Any]]:
method tools (line 214) | def tools(self) -> list[Tool]:
method call_tool (line 220) | async def call_tool(self, name: str, arguments: dict[str, Any]) -> Any:
method list_resources (line 225) | async def list_resources(self) -> list[dict[str, Any]]:
method read_resource (line 231) | async def read_resource(self, uri: str) -> tuple[bytes, str]:
method request (line 237) | async def request(self, method: str, params: dict[str, Any] | None = N...
method public_identifier (line 243) | def public_identifier(self) -> str:
FILE: anytool/grounding/backends/mcp/transport/task_managers/sse.py
class SseConnectionManager (line 18) | class SseConnectionManager(AsyncContextConnectionManager[Tuple[Any, Any]...
method __init__ (line 26) | def __init__(
FILE: anytool/grounding/backends/mcp/transport/task_managers/stdio.py
class FilteredStderrWrapper (line 25) | class FilteredStderrWrapper(io.TextIOBase):
method __init__ (line 32) | def __init__(self, wrapped_stream: TextIO):
method write (line 45) | def write(self, s: str) -> int:
method _process_line (line 64) | def _process_line(self, line: str):
method _is_harmless_error (line 181) | def _is_harmless_error(self) -> bool:
method flush (line 206) | def flush(self):
method fileno (line 220) | def fileno(self) -> int:
method closed (line 227) | def closed(self) -> bool:
class StdioConnectionManager (line 232) | class StdioConnectionManager(AsyncContextConnectionManager[Tuple[Any, An...
method __init__ (line 243) | def __init__(
method _establish_connection (line 269) | async def _establish_connection(self) -> Tuple[Any, Any]:
method _close_connection (line 301) | async def _close_connection(self) -> None:
method _suppress_mcp_json_errors (line 323) | def _suppress_mcp_json_errors(self):
method _restore_mcp_logging (line 342) | def _restore_mcp_logging(self):
FILE: anytool/grounding/backends/mcp/transport/task_managers/streamable_http.py
function _make_shim (line 21) | def _make_shim():
class StreamableHttpConnectionManager (line 77) | class StreamableHttpConnectionManager(
method __init__ (line 86) | def __init__(
FILE: anytool/grounding/backends/mcp/transport/task_managers/websocket.py
class WebSocketConnectionManager (line 16) | class WebSocketConnectionManager(
method __init__ (line 20) | def __init__(self, url: str, headers: dict[str, str] | None = None):
FILE: anytool/grounding/backends/shell/provider.py
class ShellProvider (line 14) | class ShellProvider(Provider[ShellSession]):
method __init__ (line 18) | def __init__(self, config: dict | None = None):
method _setup_security_policy (line 22) | def _setup_security_policy(self, config: dict | None = None):
method initialize (line 41) | async def initialize(self) -> None:
method create_session (line 50) | async def create_session(self, session_config: SessionConfig) -> Shell...
method close_session (line 99) | async def close_session(self, session_id: str) -> None:
FILE: anytool/grounding/backends/shell/session.py
class ShellSession (line 15) | class ShellSession(BaseSession):
method __init__ (line 18) | def __init__(
method initialize (line 35) | async def initialize(self):
class PythonScriptTool (line 45) | class PythonScriptTool(BaseTool):
method __init__ (line 49) | def __init__(self, session: "ShellSession", default_working_dir: str =...
method _arun (line 56) | async def _arun(self, code: str, timeout: int = 90, working_dir: str |...
class BashScriptTool (line 69) | class BashScriptTool(BaseTool):
method __init__ (line 73) | def __init__(self, session: "ShellSession", default_working_dir: str =...
method _arun (line 80) | async def _arun(self, script: str, timeout: int = 30, working_dir: str...
class ShellAgentTool (line 93) | class ShellAgentTool(BaseTool):
method __init__ (line 112) | def __init__(
method _get_system_info (line 135) | async def _get_system_info(self):
method _arun (line 192) | async def _arun(self, task: str, timeout: int = 300):
method _execute_code_from_response (line 353) | async def _execute_code_from_response(self, response: str):
method _generate_feedback (line 399) | def _generate_feedback(self, result: str, iteration: int, last_error: ...
method _extract_output (line 413) | def _extract_output(self, result):
method _check_task_status (line 432) | def _check_task_status(self, response: str, execution_result: str, las...
FILE: anytool/grounding/backends/shell/transport/connector.py
class ShellConnector (line 11) | class ShellConnector(AioHttpConnector):
method __init__ (line 19) | def __init__(
method _retry_invoke (line 34) | async def _retry_invoke(
method run_python_script (line 96) | async def run_python_script(
method run_bash_script (line 145) | async def run_bash_script(
FILE: anytool/grounding/backends/shell/transport/local_connector.py
function _get_conda_activation_prefix (line 32) | def _get_conda_activation_prefix(conda_env: str | None) -> str:
function _wrap_script_with_conda (line 61) | def _wrap_script_with_conda(script: str, conda_env: str | None) -> str:
class LocalShellConnector (line 97) | class LocalShellConnector(BaseConnector[Any]):
method __init__ (line 106) | def __init__(
method connect (line 125) | async def connect(self) -> None:
method _run_subprocess (line 136) | async def _run_subprocess(
method _run_shell_command (line 191) | async def _run_shell_command(
method run_python_script (line 248) | async def run_python_script(
method run_bash_script (line 321) | async def run_bash_script(
method invoke (line 384) | async def invoke(self, name: str, params: dict[str, Any]) -> Any:
method request (line 406) | async def request(self, *args: Any, **kwargs: Any) -> Any:
FILE: anytool/grounding/backends/web/provider.py
class WebProvider (line 10) | class WebProvider(Provider[WebSession]):
method __init__ (line 14) | def __init__(self, config: Dict[str, Any] = None):
method initialize (line 17) | async def initialize(self) -> None:
method create_session (line 29) | async def create_session(self, session_config: SessionConfig) -> WebSe...
method close_session (line 50) | async def close_session(self, session_name: str) -> None:
FILE: anytool/grounding/backends/web/session.py
class WebConnector (line 22) | class WebConnector(BaseConnector):
method __init__ (line 23) | def __init__(self, api_key: str, base_url: str):
method connect (line 29) | async def connect(self) -> None:
method disconnect (line 51) | async def disconnect(self) -> None:
method is_connected (line 60) | def is_connected(self) -> bool:
method invoke (line 63) | async def invoke(self, name: str, params: dict) -> Any:
method request (line 70) | async def request(self, *args: Any, **kwargs: Any) -> Any:
class WebSession (line 74) | class WebSession(BaseSession):
method __init__ (line 78) | def __init__(
method web_connector (line 104) | def web_connector(self) -> WebConnector:
method initialize (line 107) | async def initialize(self) -> Dict[str, Any]:
class DeepResearchTool (line 141) | class DeepResearchTool(BaseTool):
method __init__ (line 185) | def __init__(
method _arun (line 193) | async def _arun(self, query: str) -> str:
FILE: anytool/grounding/core/exceptions.py
class ErrorCode (line 8) | class ErrorCode(str, Enum):
class GroundingError (line 27) | class GroundingError(Exception):
method __init__ (line 45) | def __init__(
method to_dict (line 59) | def to_dict(self) -> Dict[str, Any]:
method __str__ (line 68) | def __str__(self) -> str:
method __repr__ (line 71) | def __repr__(self) -> str:
FILE: anytool/grounding/core/grounding_client.py
class GroundingClient (line 19) | class GroundingClient:
method __init__ (line 23) | def __init__(self, config: Optional[GroundingConfig] = None, recording...
method _register_providers_from_config (line 59) | def _register_providers_from_config(self) -> None:
method _register_system_provider (line 102) | def _register_system_provider(self) -> None:
method _init_quality_manager (line 116) | def _init_quality_manager(self):
method quality_manager (line 153) | def quality_manager(self):
method get_quality_report (line 158) | def get_quality_report(self) -> Dict[str, Any]:
method evolve_quality (line 166) | async def evolve_quality(self) -> Dict[str, Any]:
method get_tool_insights (line 184) | def get_tool_insights(self, tool: BaseTool) -> Dict[str, Any]:
method register_provider (line 192) | def register_provider(self, provider: Provider) -> None:
method get_provider (line 195) | def get_provider(self, backend: BackendType) -> Provider:
method list_providers (line 198) | def list_providers(self) -> Dict[BackendType, Provider]:
method recording_manager (line 202) | def recording_manager(self):
method recording_manager (line 207) | def recording_manager(self, manager):
method initialize_all_providers (line 215) | async def initialize_all_providers(self) -> None:
method create_session (line 219) | async def create_session(
method list_sessions (line 293) | def list_sessions(self) -> List[str]:
method close_session (line 296) | async def close_session(self, name: str) -> None:
method close_all_sessions (line 320) | async def close_all_sessions(self) -> None:
method ensure_session (line 324) | async def ensure_session(self, backend: BackendType, server: str | Non...
method get_session_info (line 330) | def get_session_info(self, name: str) -> SessionInfo:
method get_session (line 336) | def get_session(self, name: str) -> BaseSession:
method _fetch_tools (line 343) | async def _fetch_tools(
method list_tools (line 449) | async def list_tools(
method list_backend_tools (line 514) | async def list_backend_tools(
method list_session_tools (line 521) | async def list_session_tools(
method list_all_backend_tools (line 531) | async def list_all_backend_tools(
method search_tools (line 542) | async def search_tools(
method get_last_search_debug_info (line 615) | def get_last_search_debug_info(self) -> Optional[Dict[str, Any]]:
method get_tools_with_auto_search (line 625) | async def get_tools_with_auto_search(
method invoke_tool (line 738) | async def invoke_tool(
FILE: anytool/grounding/core/provider.py
class Provider (line 18) | class Provider(ABC, Generic[TSession]):
method __init__ (line 20) | def __init__(self, backend_type: BackendType, config: Dict[str, Any] =...
method _setup_security_policy (line 30) | def _setup_security_policy(self, config: dict | None = None):
method ensure_initialized (line 34) | async def ensure_initialized(self) -> None:
method initialize (line 42) | async def initialize(self) -> None:
method create_session (line 49) | async def create_session(self, session_config: SessionConfig) -> TSess...
method close_session (line 54) | async def close_session(self, session_name: str) -> None:
method list_sessions (line 58) | def list_sessions(self) -> List[str]:
method get_session (line 62) | def get_session(self, session_name: str) -> Optional[TSession]:
method close_all_sessions (line 66) | async def close_all_sessions(self) -> None:
method __repr__ (line 77) | def __repr__(self) -> str:
method list_tools (line 83) | async def list_tools(self, session_name: Optional[str] = None) -> List...
method call_tool (line 100) | async def call_tool(
class ProviderRegistry (line 131) | class ProviderRegistry:
method __init__ (line 135) | def __init__(self) -> None:
method register (line 138) | def register(self, provider: "Provider") -> None:
method get (line 142) | def get(self, backend: BackendType) -> "Provider":
method list (line 147) | def list(self) -> dict[BackendType, "Provider"]:
FILE: anytool/grounding/core/quality/__init__.py
function get_quality_manager (line 9) | def get_quality_manager() -> "ToolQualityManager | None":
function set_quality_manager (line 14) | def set_quality_manager(manager: "ToolQualityManager") -> None:
FILE: anytool/grounding/core/quality/manager.py
class ToolQualityManager (line 31) | class ToolQualityManager:
method __init__ (line 43) | def __init__(
method get_tool_key (line 77) | def get_tool_key(self, tool: "BaseTool") -> str:
method _compute_description_hash (line 90) | def _compute_description_hash(self, tool: "BaseTool") -> str:
method get_record (line 95) | def get_record(self, tool: "BaseTool") -> ToolQualityRecord:
method get_quality_score (line 111) | def get_quality_score(self, tool: "BaseTool") -> float:
method record_execution (line 116) | async def record_execution(
method evaluate_description (line 151) | async def evaluate_description(
method adjust_ranking (line 327) | def adjust_ranking(
method get_penalty (line 350) | def get_penalty(self, tool: "BaseTool") -> float:
method check_changes (line 355) | def check_changes(self, tools: List["BaseTool"]) -> Dict[str, str]:
method save (line 385) | async def save(self) -> None:
method clear_cache (line 396) | def clear_cache(self) -> None:
method get_stats (line 402) | def get_stats(self) -> Dict:
method get_top_tools (line 429) | def get_top_tools(
method get_problematic_tools (line 452) | def get_problematic_tools(
method get_quality_report (line 470) | def get_quality_report(self) -> Dict:
method _generate_recommendations (line 539) | def _generate_recommendations(
method compute_adaptive_quality_weight (line 573) | def compute_adaptive_quality_weight(self) -> float:
method should_reevaluate_description (line 604) | def should_reevaluate_description(self, tool: "BaseTool") -> bool:
method evolve (line 634) | async def evolve(self, tools: List["BaseTool"]) -> Dict:
method should_evolve (line 690) | def should_evolve(self) -> bool:
method get_tool_insights (line 694) | def get_tool_insights(self, tool: "BaseTool") -> Dict:
FILE: anytool/grounding/core/quality/store.py
class QualityStore (line 17) | class QualityStore:
method __init__ (line 29) | def __init__(self, cache_dir: Optional[Path] = None):
method load_all (line 43) | def load_all(self) -> tuple[Dict[str, ToolQualityRecord], int]:
method save_all (line 76) | async def save_all(self, records: Dict[str, ToolQualityRecord], global...
method save_record (line 102) | async def save_record(self, record: ToolQualityRecord, all_records: Di...
method clear (line 107) | def clear(self) -> None:
FILE: anytool/grounding/core/quality/types.py
class ExecutionRecord (line 11) | class ExecutionRecord:
class DescriptionQuality (line 20) | class DescriptionQuality:
method overall_score (line 28) | def overall_score(self) -> float:
class ToolQualityRecord (line 34) | class ToolQualityRecord:
method success_rate (line 69) | def success_rate(self) -> float:
method avg_execution_time_ms (line 76) | def avg_execution_time_ms(self) -> float:
method recent_success_rate (line 83) | def recent_success_rate(self) -> float:
method consecutive_failures (line 91) | def consecutive_failures(self) -> int:
method penalty (line 102) | def penalty(self) -> float:
method quality_score (line 137) | def quality_score(self) -> float:
method add_execution (line 144) | def add_execution(self, record: ExecutionRecord) -> None:
method to_dict (line 160) | def to_dict(self) -> Dict[str, Any]:
method from_dict (line 191) | def from_dict(cls, data: Dict[str, Any]) -> "ToolQualityRecord":
FILE: anytool/grounding/core/search_tools.py
class SearchMode (line 25) | class SearchMode(str, Enum):
class ToolRanker (line 31) | class ToolRanker:
method __init__ (line 38) | def __init__(
method _get_cache_key (line 100) | def _get_cache_key(self, tool: BaseTool) -> Tuple[str, str, str]:
method _get_cache_file_path (line 114) | def _get_cache_file_path(self) -> Path:
method _load_persistent_cache (line 120) | def _load_persistent_cache(self) -> None:
method _rebuild_text_index (line 151) | def _rebuild_text_index(self) -> None:
method _save_persistent_cache (line 161) | def _save_persistent_cache(self) -> None:
method rank (line 194) | def rank(
method _tokenize (line 210) | def _tokenize(text: str) -> list[str]:
method _keyword_search (line 215) | def _keyword_search(
method _ensure_model (line 256) | def _ensure_model(self) -> bool:
method _init_remote_embedding (line 265) | def _init_remote_embedding(self) -> bool:
method _init_local_embedding (line 289) | def _init_local_embedding(self) -> bool:
method _get_embedding (line 311) | def _get_embedding(self, tool: BaseTool) -> Optional[np.ndarray]:
method _set_embedding (line 324) | def _set_embedding(self, tool: BaseTool, embedding: np.ndarray) -> None:
method _semantic_search (line 345) | def _semantic_search(
method _hybrid_search (line 404) | def _hybrid_search(
method get_cache_stats (line 427) | def get_cache_stats(self) -> Dict[str, Any]:
method clear_cache (line 465) | def clear_cache(self, backend: Optional[str] = None, server: Optional[...
class SearchDebugInfo (line 511) | class SearchDebugInfo:
method __init__ (line 514) | def __init__(self):
method to_dict (line 534) | def to_dict(self) -> Dict[str, Any]:
class SearchCoordinator (line 553) | class SearchCoordinator(BaseTool):
method get_parameters_schema (line 565) | def get_parameters_schema(cls) -> Dict[str, Any]:
method __init__ (line 574) | def __init__(
method _arun (line 649) | async def _arun(
method _record_tool_scores (line 823) | def _record_tool_scores(
method _populate_selected_tools (line 842) | def _populate_selected_tools(
method _llm_filter_with_planning (line 864) | async def _llm_filter_with_planning(
method _generate_search_query (line 1001) | async def _generate_search_query(self, task_prompt: str) -> str:
method _log_search_results (line 1014) | def _log_search_results(self, all_tools: list[BaseTool], filtered_tool...
method _format_tool_list (line 1069) | def _format_tool_list(tools: list[BaseTool]) -> str:
method _format_ranked (line 1074) | def _format_ranked(results: list[tuple[BaseTool, float]], mode: Search...
method _run (line 1080) | def _run(self, *args, **kwargs):
method get_embedding_cache_stats (line 1083) | def get_embedding_cache_stats(self) -> Dict[str, Any]:
method clear_embedding_cache (line 1091) | def clear_embedding_cache(self, backend: Optional[str] = None, server:...
method get_last_search_debug_info (line 1103) | def get_last_search_debug_info(self) -> Optional[Dict[str, Any]]:
FILE: anytool/grounding/core/security/e2b_sandbox.py
class E2BSandbox (line 40) | class E2BSandbox(BaseSandbox):
method __init__ (line 43) | def __init__(self, options: SandboxOptions):
method start (line 76) | async def start(self) -> bool:
method stop (line 101) | async def stop(self) -> None:
method execute_safe (line 135) | async def execute_safe(self, command: str, **kwargs) -> Any:
method get_connector (line 179) | def get_connector(self) -> Any:
method get_host (line 187) | def get_host(self, port: int) -> str:
method sandbox (line 205) | def sandbox(self) -> Any:
method process (line 210) | def process(self) -> Any:
FILE: anytool/grounding/core/security/policies.py
class Colors (line 10) | class Colors:
class SecurityPolicyManager (line 21) | class SecurityPolicyManager:
method __init__ (line 22) | def __init__(self, prompt: PromptFunc | None = None):
method _default_cli_prompt (line 27) | async def _default_cli_prompt(self, message: str) -> bool:
method set_global_policy (line 51) | def set_global_policy(self, policy: SecurityPolicy) -> None:
method set_backend_policy (line 54) | def set_backend_policy(self, backend_type: BackendType, policy: Securi...
method get_policy (line 57) | def get_policy(self, backend_type: BackendType) -> SecurityPolicy:
method _ask_user (line 67) | async def _ask_user(self, message: str) -> bool:
method check_command_allowed (line 76) | async def check_command_allowed(self, backend_type: BackendType, comma...
method check_domain_allowed (line 149) | async def check_domain_allowed(self, backend_type: BackendType, domain...
FILE: anytool/grounding/core/security/sandbox.py
class BaseSandbox (line 7) | class BaseSandbox(ABC):
method __init__ (line 8) | def __init__(self, options: SandboxOptions):
method start (line 13) | async def start(self) -> bool:
method stop (line 18) | async def stop(self) -> None:
method execute_safe (line 23) | async def execute_safe(self, command: str, **kwargs) -> Any:
method get_connector (line 27) | def get_connector(self) -> Any:
method is_active (line 31) | def is_active(self) -> bool:
class SandboxManager (line 35) | class SandboxManager:
method __init__ (line 36) | def __init__(self):
method register_sandbox (line 39) | def register_sandbox(self, backend_type: BackendType, sandbox: BaseSan...
method get_sandbox (line 42) | def get_sandbox(self, backend_type: BackendType) -> Optional[BaseSandb...
method start_all (line 45) | async def start_all(self) -> None:
method stop_all (line 49) | async def stop_all(self) -> None:
FILE: anytool/grounding/core/session.py
class BaseSession (line 13) | class BaseSession(ABC):
method __init__ (line 17) | def __init__(
method __aenter__ (line 38) | async def __aenter__(self) -> "BaseSession":
method __aexit__ (line 45) | async def __aexit__(self, exc_type, exc_val, exc_tb) -> None:
method connect (line 55) | async def connect(self) -> None:
method disconnect (line 62) | async def disconnect(self) -> None:
method is_connected (line 69) | def is_connected(self) -> bool:
method initialize (line 73) | async def initialize(self) -> Dict[str, Any]:
method list_tools (line 82) | async def list_tools(self) -> List[BaseTool]:
method call_tool (line 90) | async def call_tool(self, tool_name: str, parameters=None) -> ToolResult:
method _touch (line 106) | def _touch(self):
method info (line 110) | def info(self) -> SessionInfo:
FILE: anytool/grounding/core/system/provider.py
class SystemProvider (line 9) | class SystemProvider(Provider):
method __init__ (line 13) | def __init__(self, client: GroundingClient):
method initialize (line 18) | async def initialize(self):
method create_session (line 21) | async def create_session(self, session_config: SessionConfig):
method list_tools (line 27) | async def list_tools(self, session_name: str | None = None):
method call_tool (line 30) | async def call_tool(
method close_session (line 44) | async def close_session(self, session_name: str) -> None:
FILE: anytool/grounding/core/system/tool.py
class _BaseSystemTool (line 6) | class _BaseSystemTool(LocalTool):
method __init__ (line 9) | def __init__(self, client: GroundingClient):
method client (line 14) | def client(self) -> GroundingClient:
class ListProvidersTool (line 18) | class ListProvidersTool(_BaseSystemTool):
method _arun (line 22) | async def _arun(self) -> ToolResult:
class ListBackendToolsTool (line 30) | class ListBackendToolsTool(_BaseSystemTool):
method _arun (line 34) | async def _arun(self, backend: str) -> ToolResult:
class ListSessionToolsTool (line 48) | class ListSessionToolsTool(_BaseSystemTool):
method _arun (line 52) | async def _arun(self, session_id: str) -> ToolResult:
class ListAllBackendToolsTool (line 61) | class ListAllBackendToolsTool(_BaseSystemTool):
method _arun (line 65) | async def _arun(self, use_cache: bool = False) -> ToolResult:
FILE: anytool/grounding/core/tool/base.py
class ToolRuntimeInfo (line 23) | class ToolRuntimeInfo:
method __init__ (line 25) | def __init__(
method __repr__ (line 37) | def __repr__(self):
class BaseTool (line 41) | class BaseTool(ABC):
method __init__ (line 46) | def __init__(self,
method name (line 64) | def name(self) -> str:
method description (line 69) | def description(self) -> str:
method get_parameters_schema (line 75) | def get_parameters_schema(cls) -> Dict[str, Any]:
method validate_parameters (line 117) | def validate_parameters(self, params: Dict[str, Any]) -> None:
method run (line 127) | def run(self, **kwargs):
method __call__ (line 134) | def __call__(self, **kwargs):
method __acall__ (line 137) | async def __acall__(self, **kwargs):
method arun (line 140) | async def arun(self, **kwargs) -> ToolResult:
method _arun (line 162) | async def _arun(self, **kwargs): ...
method bind_runtime_info (line 164) | def bind_runtime_info(
method runtime_info (line 190) | def runtime_info(self) -> Optional['ToolRuntimeInfo']:
method is_bound (line 195) | def is_bound(self) -> bool:
method invoke (line 199) | async def invoke(
method _wrap_result (line 221) | def _wrap_result(self, obj: Any, elapsed: float) -> ToolResult:
method _auto_record_execution (line 236) | async def _auto_record_execution(
method _record_to_quality_manager (line 288) | async def _record_to_quality_manager(
method _run (line 305) | def _run(self, **kwargs):
method __repr__ (line 308) | def __repr__(self):
method __init_subclass__ (line 314) | def __init_subclass__(cls, **kwargs):
FILE: anytool/grounding/core/tool/local_tool.py
class LocalTool (line 10) | class LocalTool(BaseTool):
method _run (line 11) | def _run(self, **kwargs):
method _dispatch_run (line 14) | async def _dispatch_run(self, **kwargs) -> Any:
method _arun (line 28) | async def _arun(self, **kwargs):
FILE: anytool/grounding/core/tool/remote_tool.py
class RemoteTool (line 14) | class RemoteTool(BaseTool):
method __init__ (line 17) | def __init__(
method _arun (line 31) | async def _arun(self, **kwargs):
FILE: anytool/grounding/core/transport/connectors/aiohttp_connector.py
class AioHttpConnector (line 13) | class AioHttpConnector(BaseConnector[aiohttp.ClientSession]):
method __init__ (line 16) | def __init__(self, base_url: str, **session_kw):
method connect (line 21) | async def connect(self) -> None:
method _request (line 31) | async def _request(
method get_json (line 56) | async def get_json(self, path: str, **kw) -> Any:
method get_bytes (line 63) | async def get_bytes(self, path: str, **kw) -> bytes:
method post_json (line 68) | async def post_json(
method request (line 100) | async def request(self, method: str, path: str, **kw) -> aiohttp.Clien...
method invoke (line 103) | async def invoke(self, name: str, params: dict[str, Any]) -> Any:
method _handle_other_json (line 138) | async def _handle_other_json(self, method: str, path: str, params: dic...
FILE: anytool/grounding/core/transport/connectors/base.py
class BaseConnector (line 19) | class BaseConnector(ABC, Generic[T]):
method __init__ (line 26) | def __init__(self, connection_manager: BaseConnectionManager[T]):
method connect (line 32) | async def connect(self) -> None:
method disconnect (line 54) | async def disconnect(self) -> None:
method _before_connect (line 75) | async def _before_connect(self) -> None:
method _after_connect (line 79) | async def _after_connect(self) -> None:
method _cleanup_on_connect_failure (line 83) | async def _cleanup_on_connect_failure(self) -> None:
method _before_disconnect (line 92) | async def _before_disconnect(self) -> None:
method _after_disconnect (line 96) | async def _after_disconnect(self) -> None:
method is_connected (line 101) | def is_connected(self) -> bool:
method _to_json_compatible (line 106) | def _to_json_compatible(obj: Any) -> Any:
method _parse_as (line 116) | def _parse_as(data: Any, model_cls: "Type[BaseModel] | None" = None) -...
method invoke (line 128) | async def invoke(self, name: str, params: dict[str, Any]) -> Any:
method request (line 136) | async def request(self, *args: Any, **kwargs: Any) -> Any:
FILE: anytool/grounding/core/transport/task_managers/aiohttp_connection_manager.py
class AioHttpConnectionManager (line 14) | class AioHttpConnectionManager(
method __init__ (line 19) | def __init__(
method _establish_connection (line 36) | async def _establish_connection(self) -> aiohttp.ClientSession:
method _close_connection (line 42) | async def _close_connection(self) -> None:
FILE: anytool/grounding/core/transport/task_managers/async_ctx.py
class _BaseExceptionGroup (line 14) | class _BaseExceptionGroup(Exception):
class AsyncContextConnectionManager (line 21) | class AsyncContextConnectionManager(Generic[T, P], BaseConnectionManager...
method __init__ (line 22) | def __init__(self,
method _establish_connection (line 32) | async def _establish_connection(self) -> T:
method _close_connection (line 69) | async def _close_connection(self) -> None:
FILE: anytool/grounding/core/transport/task_managers/base.py
class BaseConnectionManager (line 17) | class BaseConnectionManager(Generic[T], ABC):
method __init__ (line 24) | def __init__(self):
method _establish_connection (line 34) | async def _establish_connection(self) -> T:
method _close_connection (line 49) | async def _close_connection(self) -> None:
method start (line 58) | async def start(self, timeout: float | None = None) -> T:
method stop (line 127) | async def stop(self, timeout: float = 5.0) -> None:
method get_streams (line 154) | def get_streams(self) -> T | None:
method _connection_task (line 162) | async def _connection_task(self) -> None:
FILE: anytool/grounding/core/transport/task_managers/noop.py
class NoOpConnectionManager (line 12) | class NoOpConnectionManager(BaseConnectionManager[Any]):
method _establish_connection (line 19) | async def _establish_connection(self) -> Any:
method _close_connection (line 23) | async def _close_connection(self) -> None:
FILE: anytool/grounding/core/transport/task_managers/placeholder.py
class PlaceholderConnectionManager (line 5) | class PlaceholderConnectionManager(BaseConnectionManager[Any]):
method _establish_connection (line 12) | async def _establish_connection(self) -> Any:
method _close_connection (line 16) | async def _close_connection(self) -> None:
FILE: anytool/grounding/core/types.py
class BackendType (line 15) | class BackendType(str, Enum):
class ToolStatus (line 24) | class ToolStatus(str, Enum):
class SessionStatus (line 29) | class SessionStatus(str, Enum):
class BaseEntity (line 43) | class BaseEntity(BaseModel):
class JsonRpcBase (line 48) | class JsonRpcBase(BaseEntity):
class RpcMessage (line 52) | class RpcMessage(JsonRpcBase, Generic[MethodT, RequestParamsT]):
class Request (line 57) | class Request(RpcMessage[MethodT, RequestParamsT]):
class Notification (line 61) | class Notification(RpcMessage[MethodT, NotificationParamsT]):
class Result (line 65) | class Result(JsonRpcBase):
class ErrorData (line 69) | class ErrorData(BaseEntity):
class ToolResult (line 75) | class ToolResult(Result):
method is_success (line 83) | def is_success(self) -> bool: return self.status == ToolStatus.SUCCESS
method is_error (line 86) | def is_error(self) -> bool: return self.status == ToolStatus.ERROR
class SecurityPolicy (line 89) | class SecurityPolicy(BaseEntity):
method from_dict (line 98) | def from_dict(cls, data: Dict) -> "SecurityPolicy":
method check (line 155) | def check(self, *, command: str | None = None, domain: str | None = No...
method find_dangerous_tokens (line 181) | def find_dangerous_tokens(self, command: str) -> List[str]:
class ToolSchema (line 203) | class ToolSchema(BaseEntity):
method validate_parameters (line 214) | def validate_parameters(self, params: Dict[str, Any], *, raise_exc: bo...
method is_allowed (line 237) | def is_allowed(self, *, command: str | None = None, domain: str | None...
class SessionConfig (line 242) | class SessionConfig(BaseEntity):
class SessionInfo (line 254) | class SessionInfo(SessionConfig):
class SandboxOptions (line 260) | class SandboxOptions(BaseEntity):
class ClientMessage (line 276) | class ClientMessage(
FILE: anytool/llm/client.py
function _sanitize_schema (line 22) | def _sanitize_schema(params: Dict) -> Dict:
function _schema_to_openai (line 71) | def _schema_to_openai(schema: ToolSchema) -> ChatCompletionToolParam:
function _prepare_tools_for_llmclient (line 94) | def _prepare_tools_for_llmclient(
function _summarize_tool_result (line 153) | async def _summarize_tool_result(
function _tool_result_to_message_async (line 208) | async def _tool_result_to_message_async(
function _execute_tool_call (line 261) | async def _execute_tool_call(
class LLMClient (line 317) | class LLMClient:
method __init__ (line 319) | def __init__(
method _rate_limit (line 357) | async def _rate_limit(self):
method _call_with_retry (line 370) | async def _call_with_retry(self, **completion_kwargs):
method complete (line 440) | async def complete(
method format_messages_to_text (line 683) | def format_messages_to_text(messages: List[Dict]) -> str:
FILE: anytool/local_server/feature_checker.py
class FeatureChecker (line 12) | class FeatureChecker:
method __init__ (line 13) | def __init__(self, platform_adapter=None, accessibility_helper=None):
method check_screenshot_available (line 19) | def check_screenshot_available(self, use_cache: bool = True) -> bool:
method check_shell_available (line 43) | def check_shell_available(self, use_cache: bool = True) -> bool:
method check_python_available (line 74) | def check_python_available(self, use_cache: bool = True) -> bool:
method check_file_ops_available (line 109) | def check_file_ops_available(self, use_cache: bool = True) -> bool:
method check_window_mgmt_available (line 136) | def check_window_mgmt_available(self, use_cache: bool = True) -> bool:
method check_recording_available (line 167) | def check_recording_available(self, use_cache: bool = True) -> bool:
method check_accessibility_available (line 191) | def check_accessibility_available(self, use_cache: bool = True) -> bool:
method check_platform_adapter_available (line 211) | def check_platform_adapter_available(self, use_cache: bool = True) -> ...
method check_all_features (line 220) | def check_all_features(self, use_cache: bool = True) -> Dict[str, bool]:
method clear_cache (line 240) | def clear_cache(self):
method get_feature_report (line 244) | def get_feature_report(self) -> Dict[str, Any]:
FILE: anytool/local_server/health_checker.py
class HealthStatus (line 13) | class HealthStatus:
method __init__ (line 15) | def __init__(self, feature_available: bool, endpoint_available: Option...
method fully_available (line 22) | def fully_available(self) -> bool:
method __str__ (line 26) | def __str__(self):
class HealthChecker (line 37) | class HealthChecker:
method __init__ (line 40) | def __init__(self, feature_checker: FeatureChecker,
method _get_test_file_path (line 63) | def _get_test_file_path(self, filename: str) -> str:
method _register_temp_file (line 69) | def _register_temp_file(self, filepath: str):
method cleanup_temp_files (line 74) | def cleanup_temp_files(self):
method check_screenshot (line 103) | def check_screenshot(self) -> Tuple[bool, str]:
method check_cursor_position (line 140) | def check_cursor_position(self) -> Tuple[bool, str]:
method check_screen_size (line 157) | def check_screen_size(self) -> Tuple[bool, str]:
method check_shell_command (line 174) | def check_shell_command(self) -> Tuple[bool, str]:
method check_python_execution (line 200) | def check_python_execution(self) -> Tuple[bool, str]:
method check_bash_script (line 227) | def check_bash_script(self) -> Tuple[bool, str]:
method check_file_operations (line 252) | def check_file_operations(self) -> Tuple[bool, str]:
method check_desktop_path (line 276) | def check_desktop_path(self) -> Tuple[bool, str]:
method check_window_management (line 294) | def check_window_management(self) -> Tuple[bool, str]:
method check_recording (line 314) | def check_recording(self) -> Tuple[bool, str]:
method check_accessibility (line 357) | def check_accessibility(self) -> Tuple[bool, str]:
method check_health_endpoint (line 380) | def check_health_endpoint(self) -> Tuple[bool, str]:
method check_platform_info (line 392) | def check_platform_info(self) -> Tuple[bool, str]:
method check_all (line 404) | def check_all(self, test_endpoints: bool = True) -> Dict[str, HealthSt...
method print_results (line 460) | def print_results(self, results: Dict[str, HealthStatus] = None,
method get_summary (line 568) | def get_summary(self) -> dict:
method get_simple_features_dict (line 584) | def get_simple_features_dict(self) -> Dict[str, bool]:
FILE: anytool/local_server/main.py
function get_conda_activation_prefix (line 51) | def get_conda_activation_prefix(conda_env: str = None) -> str:
function wrap_script_with_conda (line 111) | def wrap_script_with_conda(script: str, conda_env: str = None) -> str:
function health_check (line 158) | def health_check():
function get_platform (line 177) | def get_platform():
function execute_command (line 193) | def execute_command():
function execute_command_with_verification (line 249) | def execute_command_with_verification():
function _get_machine_architecture (line 368) | def _get_machine_architecture() -> str:
function launch_app (line 381) | def launch_app():
function run_python (line 418) | def run_python():
function run_bash_script (line 519) | def run_bash_script():
function capture_screen_with_cursor (line 597) | def capture_screen_with_cursor():
function get_cursor_position (line 619) | def get_cursor_position():
function get_screen_size (line 628) | def get_screen_size():
function get_accessibility_tree (line 638) | def get_accessibility_tree():
function list_directory (line 653) | def list_directory():
function file_operation (line 683) | def file_operation():
function get_desktop_path (line 720) | def get_desktop_path():
function activate_window (line 735) | def activate_window():
function close_window (line 762) | def close_window():
function get_window_size (line 789) | def get_window_size():
function set_wallpaper (line 803) | def set_wallpaper():
function start_recording (line 829) | def start_recording():
function end_recording (line 880) | def end_recording():
function get_terminal_output (line 941) | def get_terminal_output():
function upload_file (line 965) | def upload_file():
function download_file (line 997) | def download_file():
function open_file (line 1017) | def open_file():
function print_banner (line 1047) | def print_banner(host: str = "127.0.0.1", port: int = 5000, debug: bool ...
function run_health_check_async (line 1071) | def run_health_check_async():
function run_server (line 1089) | def run_server(host: str = "127.0.0.1", port: int = 5000, debug: bool = ...
function main (line 1111) | def main():
FILE: anytool/local_server/platform_adapters/__init__.py
function get_platform_adapter (line 31) | def get_platform_adapter() -> Optional[Any]:
FILE: anytool/local_server/platform_adapters/linux_adapter.py
class LinuxAdapter (line 20) | class LinuxAdapter:
method __init__ (line 22) | def __init__(self):
method capture_screenshot_with_cursor (line 27) | def capture_screenshot_with_cursor(self, output_path: str) -> bool:
method activate_window (line 67) | def activate_window(self, window_name: str, strict: bool = False, by_c...
method close_window (line 98) | def close_window(self, window_name: str, strict: bool = False, by_clas...
method get_accessibility_tree (line 129) | def get_accessibility_tree(self, max_depth: int = 10, max_width: int =...
method _serialize_atspi_element (line 164) | def _serialize_atspi_element(
method get_screen_size (line 255) | def get_screen_size(self) -> Dict[str, int]:
method list_windows (line 279) | def list_windows(self) -> List[Dict[str, Any]]:
method get_terminal_output (line 315) | def get_terminal_output(self) -> Optional[str]:
method _find_terminals (line 348) | def _find_terminals(self, element) -> List[Accessible]:
method set_wallpaper (line 363) | def set_wallpaper(self, image_path: str) -> Dict[str, Any]:
method get_system_info (line 395) | def get_system_info(self) -> Dict[str, Any]:
method start_recording (line 436) | def start_recording(self, output_path: str) -> Dict[str, Any]:
method stop_recording (line 507) | def stop_recording(self, process) -> Dict[str, Any]:
method get_running_applications (line 539) | def get_running_applications(self) -> List[Dict[str, str]]:
FILE: anytool/local_server/platform_adapters/macos_adapter.py
class MacOSAdapter (line 18) | class MacOSAdapter:
method __init__ (line 19) | def __init__(self):
method capture_screenshot_with_cursor (line 27) | def capture_screenshot_with_cursor(self, output_path: str) -> bool:
method activate_window (line 46) | def activate_window(self, window_name: str, strict: bool = False) -> D...
method close_window (line 111) | def close_window(self, window_name: str, strict: bool = False) -> Dict...
method get_accessibility_tree (line 154) | def get_accessibility_tree(self, max_depth: int = 10) -> Dict[str, Any]:
method _serialize_ax_element (line 210) | def _serialize_ax_element(self, element, depth: int = 0, max_depth: in...
method get_running_applications (line 297) | def get_running_applications(self) -> List[Dict[str, str]]:
method set_wallpaper (line 324) | def set_wallpaper(self, image_path: str) -> Dict[str, Any]:
method get_system_info (line 357) | def get_system_info(self) -> Dict[str, Any]:
method _detect_screen_device (line 393) | def _detect_screen_device(self) -> str:
method start_recording (line 435) | def start_recording(self, output_path: str) -> Dict[str, Any]:
method stop_recording (line 536) | def stop_recording(self, process) -> Dict[str, Any]:
method list_windows (line 593) | def list_windows(self) -> List[Dict[str, Any]]:
method get_terminal_output (line 651) | def get_terminal_output(self) -> Optional[str]:
FILE: anytool/local_server/platform_adapters/pyxcursor.py
class XFixesCursorImage (line 14) | class XFixesCursorImage(ctypes.Structure):
class Display (line 42) | class Display(ctypes.Structure):
class Xcursor (line 46) | class Xcursor:
method __init__ (line 49) | def __init__(self, display=None):
method argbdata_to_pixdata (line 81) | def argbdata_to_pixdata(self, data, len):
method getCursorImageData (line 102) | def getCursorImageData(self):
method getCursorImageArray (line 112) | def getCursorImageArray(self):
method getCursorImageArrayFast (line 125) | def getCursorImageArrayFast(self):
method saveImage (line 137) | def saveImage(self, imgarray, text):
FILE: anytool/local_server/platform_adapters/windows_adapter.py
class WindowsAdapter (line 21) | class WindowsAdapter:
method __init__ (line 24) | def __init__(self):
method capture_screenshot_with_cursor (line 29) | def capture_screenshot_with_cursor(self, output_path: str) -> bool:
method _get_cursor (line 70) | def _get_cursor(self) -> tuple:
method activate_window (line 114) | def activate_window(self, window_name: str, strict: bool = False) -> D...
method close_window (line 155) | def close_window(self, window_name: str, strict: bool = False) -> Dict...
method get_accessibility_tree (line 195) | def get_accessibility_tree(self, max_depth: int = 10, max_width: int =...
method _serialize_uia_element (line 231) | def _serialize_uia_element(
method list_windows (line 335) | def list_windows(self) -> List[Dict[str, Any]]:
method set_wallpaper (line 366) | def set_wallpaper(self, image_path: str) -> Dict[str, Any]:
method get_system_info (line 399) | def get_system_info(self) -> Dict[str, Any]:
method start_recording (line 424) | def start_recording(self, output_path: str) -> Dict[str, Any]:
method stop_recording (line 491) | def stop_recording(self, process) -> Dict[str, Any]:
method get_running_applications (line 525) | def get_running_applications(self) -> List[Dict[str, str]]:
method get_screen_size (line 575) | def get_screen_size(self) -> Dict[str, int]:
method get_terminal_output (line 591) | def get_terminal_output(self) -> Optional[str]:
FILE: anytool/local_server/utils/accessibility.py
class AccessibilityHelper (line 10) | class AccessibilityHelper:
method __init__ (line 11) | def __init__(self):
method get_tree (line 28) | def get_tree(self, max_depth: int = 10) -> Dict[str, Any]:
method is_available (line 44) | def is_available(self) -> bool:
method find_element_by_name (line 47) | def find_element_by_name(self, tree: Dict[str, Any], name: str) -> Opt...
method find_element_by_role (line 53) | def find_element_by_role(self, tree: Dict[str, Any], role: str) -> Opt...
method _search_tree (line 59) | def _search_tree(self, node: Dict[str, Any], key: str, value: str) -> ...
method flatten_tree (line 76) | def flatten_tree(self, tree: Dict[str, Any]) -> list:
method _flatten_node (line 84) | def _flatten_node(self, node: Dict[str, Any], result: list):
method get_visible_elements (line 98) | def get_visible_elements(self, tree: Dict[str, Any]) -> list:
method get_clickable_elements (line 115) | def get_clickable_elements(self, tree: Dict[str, Any]) -> list:
method get_statistics (line 131) | def get_statistics(self, tree: Dict[str, Any]) -> Dict[str, Any]:
FILE: anytool/local_server/utils/screenshot.py
class ScreenshotHelper (line 13) | class ScreenshotHelper:
method __init__ (line 14) | def __init__(self):
method capture (line 31) | def capture(self, output_path: str, with_cursor: bool = True) -> bool:
method capture_region (line 50) | def capture_region(
method get_screen_size (line 83) | def get_screen_size(self) -> Tuple[int, int]:
method get_cursor_position (line 97) | def get_cursor_position(self) -> Tuple[int, int]:
method capture_to_base64 (line 111) | def capture_to_base64(self, with_cursor: bool = True) -> Optional[str]:
method compare_screenshots (line 149) | def compare_screenshots(self, path1: str, path2: str) -> float:
method annotate_screenshot (line 196) | def annotate_screenshot(
FILE: anytool/platform/config.py
function get_local_server_config (line 8) | def get_local_server_config() -> Dict[str, Any]:
function get_client_base_url (line 65) | def get_client_base_url() -> str:
FILE: anytool/platform/recording.py
class RecordingClient (line 9) | class RecordingClient:
method __init__ (line 18) | def __init__(
method _get_session (line 39) | async def _get_session(self) -> aiohttp.ClientSession:
method start_recording (line 47) | async def start_recording(self, auto_cleanup: bool = True) -> bool:
method end_recording (line 89) | async def end_recording(self, dest: Optional[str] = None) -> Optional[...
method close (line 123) | async def close(self):
method __aenter__ (line 132) | async def __aenter__(self):
method __aexit__ (line 136) | async def __aexit__(self, exc_type, exc_val, exc_tb):
class RecordingContextManager (line 142) | class RecordingContextManager:
method __init__ (line 144) | def __init__(
method __aenter__ (line 172) | async def __aenter__(self) -> RecordingClient:
method __aexit__ (line 183) | async def __aexit__(self, exc_type, exc_val, exc_tb):
FILE: anytool/platform/screenshot.py
class ScreenshotClient (line 19) | class ScreenshotClient:
method __init__ (line 21) | def __init__(
method _get_session (line 44) | async def _get_session(self) -> aiohttp.ClientSession:
method _is_valid_image_response (line 53) | def _is_valid_image_response(content_type: str, data: Optional[bytes])...
method capture (line 81) | async def capture(self) -> Optional[bytes]:
method capture_to_file (line 117) | async def capture_to_file(self, output_path: str) -> bool:
method get_screen_size (line 132) | async def get_screen_size(self) -> tuple[int, int]:
method close (line 160) | async def close(self):
method __aenter__ (line 166) | async def __aenter__(self):
method __aexit__ (line 170) | async def __aexit__(self, exc_type, exc_val, exc_tb):
class AutoScreenshotWrapper (line 176) | class AutoScreenshotWrapper:
method __init__ (line 196) | def __init__(
method __getattr__ (line 217) | def __getattr__(self, name):
method _capture_and_notify (line 221) | async def _capture_and_notify(self):
method execute (line 233) | async def execute(self, *args, **kwargs):
method _arun (line 245) | async def _arun(self, *args, **kwargs):
method enable (line 257) | def enable(self):
method disable (line 261) | def disable(self):
FILE: anytool/platform/system_info.py
class SystemInfoClient (line 9) | class SystemInfoClient:
method __init__ (line 17) | def __init__(
method _get_session (line 39) | async def _get_session(self) -> aiohttp.ClientSession:
method get_system_info (line 47) | async def get_system_info(self, use_cache: bool = True) -> Optional[Di...
method get_screen_size (line 90) | async def get_screen_size(self) -> Optional[Dict[str, int]]:
method get_cursor_position (line 118) | async def get_cursor_position(self) -> Optional[Dict[str, int]]:
method clear_cache (line 145) | def clear_cache(self):
method close (line 150) | async def close(self):
method __aenter__ (line 156) | async def __aenter__(self):
method __aexit__ (line 160) | async def __aexit__(self, exc_type, exc_val, exc_tb):
function get_system_info (line 165) | async def get_system_info(base_url: Optional[str] = None) -> Optional[Di...
function get_screen_size (line 170) | async def get_screen_size(base_url: Optional[str] = None) -> Optional[Di...
FILE: anytool/prompts/grounding_agent_prompts.py
class GroundingAgentPrompts (line 4) | class GroundingAgentPrompts:
method iteration_summary (line 68) | def iteration_summary(
method visual_analysis (line 108) | def visual_analysis(
method final_summary (line 149) | def final_summary(
method workspace_directory (line 180) | def workspace_directory(workspace_dir: str) -> str:
method workspace_matching_files (line 200) | def workspace_matching_files(matching_files: List[str]) -> str:
method workspace_recent_files (line 211) | def workspace_recent_files(total_files: int, recent_files: List[str]) ...
method workspace_file_list (line 221) | def workspace_file_list(files: List[str]) -> str:
method iteration_feedback (line 231) | def iteration_feedback(
FILE: anytool/recording/action_recorder.py
class ActionRecorder (line 18) | class ActionRecorder:
method __init__ (line 28) | def __init__(self, trajectory_dir: Path):
method record_action (line 42) | async def record_action(
method _infer_agent_type (line 110) | def _infer_agent_type(self, agent_name: str) -> str:
method _truncate_data (line 124) | def _truncate_data(self, data: Any, max_length: int) -> Any:
method _append_to_file (line 157) | async def _append_to_file(self, action_info: Dict[str, Any]):
method get_step_count (line 163) | def get_step_count(self) -> int:
function load_agent_actions (line 168) | def load_agent_actions(trajectory_dir: str) -> list:
function analyze_agent_actions (line 194) | def analyze_agent_actions(actions: list) -> Dict[str, Any]:
function format_agent_actions (line 223) | def format_agent_actions(actions: list, format_type: str = "compact") ->...
FILE: anytool/recording/manager.py
class RecordingManager (line 15) | class RecordingManager:
method __init__ (line 19) | def __init__(
method is_recording (line 78) | def is_recording(cls) -> bool:
method record_retrieved_tools (line 88) | async def record_retrieved_tools(
method record_iteration_context (line 142) | async def record_iteration_context(
method record_tool_execution (line 230) | async def record_tool_execution(
method _parse_arguments (line 300) | def _parse_arguments(arg_data):
method start (line 324) | async def start(self, task_id: Optional[str] = None):
method _check_server_availability (line 394) | async def _check_server_availability(self):
method stop (line 411) | async def stop(self):
method register_to_llm (line 483) | def register_to_llm(self, llm_client):
method _auto_record_tool_results (line 511) | async def _auto_record_tool_results(self, tool_results: List[Dict]):
method _record_mcp (line 553) | async def _record_mcp(self, tool_call, result, server: str):
method _record_gui (line 581) | async def _record_gui(self, tool_call, result):
method _record_shell (line 642) | async def _record_shell(self, tool_call, result):
method _record_system (line 699) | async def _record_system(self, tool_call, result):
method _record_web (line 732) | async def _record_web(self, tool_call, result):
method add_metadata (line 759) | async def add_metadata(self, key: str, value: Any):
method save_plan (line 763) | async def save_plan(self, plan: Dict[str, Any], agent_name: str = "Gro...
method log_decision (line 802) | async def log_decision(
method record_agent_action (line 838) | async def record_agent_action(
method generate_summary (line 888) | async def generate_summary(self) -> Dict[str, Any]:
method __aenter__ (line 941) | async def __aenter__(self):
method __aexit__ (line 945) | async def __aexit__(self, exc_type, exc_val, exc_tb):
method recording_status (line 950) | def recording_status(self) -> bool:
method trajectory_dir (line 954) | def trajectory_dir(self) -> Optional[str]:
method recording_client (line 960) | def recording_client(self):
method screenshot_client (line 964) | def screenshot_client(self):
method step_count (line 968) | def step_count(self) -> int:
FILE: anytool/recording/recorder.py
class TrajectoryRecorder (line 11) | class TrajectoryRecorder:
method __init__ (line 12) | def __init__(
method record_step (line 72) | async def record_step(
method _capture_screenshot (line 147) | async def _capture_screenshot(self) -> Optional[bytes]:
method save_init_screenshot (line 168) | async def save_init_screenshot(self, screenshot: bytes, filename: str ...
method _append_to_traj_file (line 182) | async def _append_to_traj_file(self, step_info: Dict[str, Any]):
method _save_metadata (line 189) | def _save_metadata(self):
method start_video_recording (line 195) | async def start_video_recording(self):
method stop_video_recording (line 214) | async def stop_video_recording(self):
method add_metadata (line 224) | async def add_metadata(self, key: str, value: Any):
method finalize (line 229) | async def finalize(self):
method _cleanup_screenshot_client (line 251) | async def _cleanup_screenshot_client(self):
method __del__ (line 261) | def __del__(self):
method get_trajectory_dir (line 270) | def get_trajectory_dir(self) -> str:
method __aenter__ (line 274) | async def __aenter__(self):
method __aexit__ (line 278) | async def __aexit__(self, exc_type, exc_val, exc_tb):
function record_gui_step (line 283) | async def record_gui_step(
function record_shell_step (line 319) | async def record_shell_step(
function record_mcp_step (line 358) | async def record_mcp_step(
function record_web_step (line 395) | async def record_web_step(
FILE: anytool/recording/utils.py
function load_trajectory_from_jsonl (line 9) | def load_trajectory_from_jsonl(jsonl_path: str) -> List[Dict[str, Any]]:
function load_metadata (line 33) | def load_metadata(trajectory_dir: str) -> Optional[Dict[str, Any]]:
function format_trajectory_for_export (line 45) | def format_trajectory_for_export(
function _format_compact (line 59) | def _format_compact(trajectory: List[Dict[str, Any]]) -> str:
function _format_detailed (line 76) | def _format_detailed(trajectory: List[Dict[str, Any]]) -> str:
function _format_markdown (line 115) | def _format_markdown(trajectory: List[Dict[str, Any]]) -> str:
function analyze_trajectory (line 137) | def analyze_trajectory(trajectory: List[Dict[str, Any]]) -> Dict[str, Any]:
function load_recording_session (line 176) | def load_recording_session(recording_dir: str) -> Dict[str, Any]:
function filter_trajectory (line 240) | def filter_trajectory(
function extract_errors (line 268) | def extract_errors(trajectory: List[Dict[str, Any]]) -> List[Dict[str, A...
function generate_summary_report (line 275) | def generate_summary_report(recording_dir: str, output_file: Optional[st...
function compare_recordings (line 362) | def compare_recordings(recording_dir1: str, recording_dir2: str) -> Dict...
FILE: anytool/recording/video.py
class VideoRecorder (line 17) | class VideoRecorder:
method __init__ (line 18) | def __init__(
method start (line 35) | async def start(self):
method stop (line 58) | async def stop(self):
FILE: anytool/recording/viewer.py
class RecordingViewer (line 17) | class RecordingViewer:
method __init__ (line 27) | def __init__(self, recording_dir: str):
method show_summary (line 44) | def show_summary(self) -> str:
method show_agent_actions (line 81) | def show_agent_actions(self, format_type: str = "compact", agent_name:...
method analyze_agents (line 101) | def analyze_agents(self) -> str:
method generate_full_report (line 124) | def generate_full_report(self, output_file: Optional[str] = None) -> str:
method export_to_json (line 127) | def export_to_json(self, output_file: str):
method show_timeline (line 133) | def show_timeline(self, max_events: int = 50) -> str:
method show_agent_flow (line 218) | def show_agent_flow(self, agent_name: Optional[str] = None) -> str:
function view_recording (line 261) | def view_recording(recording_dir: str):
function compare_recordings (line 282) | def compare_recordings(recording_dir1: str, recording_dir2: str) -> str:
FILE: anytool/tool_layer.py
class AnyToolConfig (line 21) | class AnyToolConfig:
method __post_init__ (line 58) | def __post_init__(self):
class AnyTool (line 66) | class AnyTool:
method __init__ (line 67) | def __init__(self, config: Optional[AnyToolConfig] = None):
method initialize (line 80) | async def initialize(self) -> None:
method execute (line 190) | async def execute(
method _maybe_evolve_quality (line 326) | async def _maybe_evolve_quality(self) -> None:
method cleanup (line 340) | async def cleanup(self) -> None:
method is_initialized (line 367) | def is_initialized(self) -> bool:
method is_running (line 370) | def is_running(self) -> bool:
method get_config (line 373) | def get_config(self) -> AnyToolConfig:
method list_backends (line 376) | def list_backends(self) -> List[str]:
method list_sessions (line 381) | def list_sessions(self) -> List[str]:
method __aenter__ (line 386) | async def __aenter__(self):
method __aexit__ (line 391) | async def __aexit__(self, exc_type, exc_val, exc_tb):
method __repr__ (line 396) | def __repr__(self) -> str:
FILE: anytool/utils/cli_display.py
class CLIDisplay (line 7) | class CLIDisplay:
method print_banner (line 9) | def print_banner():
method print_configuration (line 27) | def print_configuration(config: AnyToolConfig):
method print_initialization_progress (line 47) | def print_initialization_progress(steps: list, show_header: bool = True):
method print_result_summary (line 70) | def print_result_summary(result: dict):
method print_interactive_header (line 122) | def print_interactive_header():
method print_task_header (line 149) | def print_task_header(query: str, title: str = "▶ Executing Task"):
method print_system_ready (line 160) | def print_system_ready():
method print_status (line 175) | def print_status(agent):
method print_help (line 198) | def print_help():
FILE: anytool/utils/display.py
class Colors (line 6) | class Colors:
class BoxStyle (line 29) | class BoxStyle(Enum):
function strip_ansi (line 52) | def strip_ansi(text: str) -> str:
function colorize (line 66) | def colorize(text: str, color: str = '', bold: bool = False) -> str:
class Box (line 88) | class Box:
method __init__ (line 89) | def __init__(self,
method top_line (line 101) | def top_line(self, indent: int = 2) -> str:
method bottom_line (line 110) | def bottom_line(self, indent: int = 2) -> str:
method separator_line (line 119) | def separator_line(self, indent: int = 2) -> str:
method empty_line (line 125) | def empty_line(self, indent: int = 2) -> str:
method text_line (line 134) | def text_line(self, text: str, align: str = 'left', indent: int = 2, t...
method build (line 164) | def build(self,
function print_box (line 192) | def print_box(title: Optional[str] = None,
function print_banner (line 204) | def print_banner(title: str,
function print_section (line 223) | def print_section(title: str,
function print_separator (line 233) | def print_separator(width: int = 68, color: str = 'bl', indent: int = 2):
FILE: anytool/utils/logging.py
function _load_log_level_from_config (line 14) | def _load_log_level_from_config() -> int:
class FlushFileHandler (line 51) | class FlushFileHandler(logging.FileHandler):
method emit (line 54) | def emit(self, record):
class ColoredFormatter (line 59) | class ColoredFormatter(logging.Formatter):
method format (line 69) | def format(self, record: logging.LogRecord) -> str:
class Logger (line 78) | class Logger:
method _get_default_log_file (line 98) | def _get_default_log_file() -> str:
method get_logger (line 124) | def get_logger(cls, name: Optional[str] = None) -> logging.Logger:
method configure (line 146) | def configure(
method set_debug (line 237) | def set_debug(cls, debug_level: int = 2) -> None:
method add_file_handler (line 244) | def add_file_handler(
method reset_configuration (line 268) | def reset_configuration(cls) -> None:
method _stdout_supports_color (line 278) | def _stdout_supports_color() -> bool:
method _resolve_level (line 282) | def _resolve_level(cls, level: Optional[int]) -> int:
method _update_level (line 289) | def _update_level(cls, level: int) -> None:
FILE: anytool/utils/telemetry/events.py
class BaseTelemetryEvent (line 6) | class BaseTelemetryEvent(ABC):
method name (line 11) | def name(self) -> str:
method properties (line 17) | def properties(self) -> dict[str, Any]:
class MCPAgentExecutionEvent (line 23) | class MCPAgentExecutionEvent(BaseTelemetryEvent):
method name (line 59) | def name(self) -> str:
method properties (line 63) | def properties(self) -> dict[str, Any]:
FILE: anytool/utils/telemetry/telemetry.py
function singleton (line 23) | def singleton(cls):
function requires_telemetry (line 34) | def requires_telemetry(func: Callable) -> Callable:
function get_cache_home (line 46) | def get_cache_home() -> Path:
class Telemetry (line 66) | class Telemetry:
method __init__ (line 81) | def __init__(self):
method user_id (line 127) | def user_id(self) -> str:
method capture (line 161) | def capture(self, event: BaseTelemetryEvent) -> None:
method track_package_download (line 189) | def track_package_download(self, properties: dict[str, Any] | None = N...
method track_agent_execution (line 236) | def track_agent_execution(
method flush (line 289) | def flush(self) -> None:
method shutdown (line 304) | def shutdown(self) -> None:
FILE: anytool/utils/telemetry/utils.py
function get_package_version (line 13) | def get_package_version() -> str:
function get_model_provider (line 21) | def get_model_provider(llm: BaseLanguageModel) -> str:
function get_model_name (line 27) | def get_model_name(llm: BaseLanguageModel) -> str:
function extract_model_info (line 42) | def extract_model_info(llm: BaseLanguageModel) -> tuple[str, str]:
FILE: anytool/utils/ui.py
class AgentStatus (line 20) | class AgentStatus(Enum):
class AnyToolUI (line 28) | class AnyToolUI:
method __init__ (line 45) | def __init__(self, enable_live: bool = True, compact: bool = False):
method _get_terminal_size (line 80) | def _get_terminal_size(self) -> Tuple[int, int]:
method _clear_screen (line 88) | def _clear_screen(self):
method _move_cursor_home (line 95) | def _move_cursor_home(self):
method _hide_cursor (line 100) | def _hide_cursor(self):
method _show_cursor (line 105) | def _show_cursor(self):
method print_banner (line 111) | def print_banner(self):
method print_initialization (line 131) | def print_initialization(self, steps: List[Tuple[str, str]]):
method start_live_display (line 151) | async def start_live_display(self):
method stop_live_display (line 164) | async def stop_live_display(self):
method _live_update_loop (line 181) | async def _live_update_loop(self):
method render (line 192) | def render(self):
method update_display (line 217) | def update_display(self):
method _render_header (line 221) | def _render_header(self) -> List[str]:
method _render_agents (line 250) | def _render_agents(self) -> List[str]:
method _render_grounding (line 282) | def _render_grounding(self) -> List[str]:
method _render_logs (line 337) | def _render_logs(self) -> List[str]:
method update_agent_status (line 359) | def update_agent_status(self, agent_name: str, status: AgentStatus):
method add_agent_activity (line 363) | def add_agent_activity(self, agent_name: str, activity: str):
method update_grounding_backends (line 374) | def update_grounding_backends(self, backends: List[Dict[str, Any]]):
method add_grounding_operation (line 386) | def add_grounding_operation(self, backend: str, action: str, status: s...
method add_log (line 397) | def add_log(self, message: str, level: str = "info"):
method update_metrics (line 405) | def update_metrics(self, **kwargs):
method print_summary (line 409) | def print_summary(self, result: Dict[str, Any]):
function create_ui (line 438) | def create_ui(enable_live: bool = True, compact: bool = False) -> AnyToo...
FILE: anytool/utils/ui_integration.py
class UIIntegration (line 17) | class UIIntegration:
method __init__ (line 25) | def __init__(self, ui: AnyToolUI):
method attach_llm_client (line 40) | def attach_llm_client(self, llm_client):
method attach_grounding_client (line 50) | def attach_grounding_client(self, grounding_client):
method start_monitoring (line 60) | async def start_monitoring(self, poll_interval: float = 0.5):
method stop_monitoring (line 81) | async def stop_monitoring(self):
method _monitor_loop (line 97) | async def _monitor_loop(self, poll_interval: float):
method _update_ui (line 113) | async def _update_ui(self):
method on_agent_start (line 151) | def on_agent_start(self, agent_name: str, activity: str):
method on_agent_thinking (line 163) | def on_agent_thinking(self, agent_name: str):
method on_agent_complete (line 172) | def on_agent_complete(self, agent_name: str, result: str = ""):
method on_llm_call (line 184) | def on_llm_call(self, model: str, prompt_length: int):
method on_grounding_call (line 197) | def on_grounding_call(self, backend: str, action: str):
method on_grounding_complete (line 208) | def on_grounding_complete(self, backend: str, action: str, success: bo...
method on_iteration (line 230) | def on_iteration(self, iteration: int):
method on_error (line 239) | def on_error(self, message: str):
class UILoggingHandler (line 249) | class UILoggingHandler:
method __init__ (line 254) | def __init__(self, ui: AnyToolUI):
method emit (line 263) | def emit(self, record):
function create_integration (line 288) | def create_integration(ui: AnyToolUI) -> UIIntegration:
Condensed preview — 132 files, each showing path, character count, and a content snippet. Download the .json file or copy for the full structured content (1,160K chars).
[
{
"path": ".gitignore",
"chars": 635,
"preview": "# Byte-compiled / optimized / DLL files\n__pycache__/\n*.py[cod]\n*$py.class\n\n# OS files\n.DS_Store\n.DS_Store?\n._*\n.Spotligh"
},
{
"path": "COMMUNICATION.md",
"chars": 240,
"preview": "We provide QR codes for joining the HKUDS discussion groups on **WeChat** and **Feishu**.\n\nYou can join by scanning the "
},
{
"path": "LICENSE",
"chars": 1062,
"preview": "MIT License\n\nCopyright (c) 2025 HKUDS\n\nPermission is hereby granted, free of charge, to any person obtaining a copy\nof t"
},
{
"path": "README.md",
"chars": 34219,
"preview": "<div align=\"center\">\n\n<picture>\n <img src=\"assets/AnyTool_logo.png\" width=\"800px\" style=\"border: none; box-shadow: no"
},
{
"path": "anytool/__init__.py",
"chars": 2028,
"preview": "from importlib import import_module as _imp\nfrom typing import Dict as _Dict, Any as _Any, TYPE_CHECKING as _TYPE_CHECKI"
},
{
"path": "anytool/__main__.py",
"chars": 15871,
"preview": "import asyncio\nimport argparse\nimport sys\nimport logging\nfrom typing import Optional\n\nfrom anytool.tool_layer import Any"
},
{
"path": "anytool/agents/__init__.py",
"chars": 221,
"preview": "from anytool.agents.base import BaseAgent, AgentStatus, AgentRegistry\nfrom anytool.agents.grounding_agent import Groundi"
},
{
"path": "anytool/agents/base.py",
"chars": 6572,
"preview": "from __future__ import annotations\n\nimport json\nfrom abc import ABC, abstractmethod\nfrom typing import TYPE_CHECKING, Di"
},
{
"path": "anytool/agents/grounding_agent.py",
"chars": 45423,
"preview": "from __future__ import annotations\n\nimport copy\nimport json\nfrom typing import TYPE_CHECKING, Any, Dict, List, Optional\n"
},
{
"path": "anytool/config/__init__.py",
"chars": 603,
"preview": "from .grounding import *\nfrom .loader import *\nfrom .constants import * \nfrom .utils import *\nfrom . import constants\n\n_"
},
{
"path": "anytool/config/config_agents.json",
"chars": 234,
"preview": "{\n \"agents\": [\n {\n \"name\": \"GroundingAgent\",\n \"class_name\": \"GroundingAgent\",\n \"backend_scope\": [\"gui"
},
{
"path": "anytool/config/config_dev.json.example",
"chars": 244,
"preview": "{\n \"comment\": \"[Optional] Loading grounding.json → security.json → dev.json (dev.json overrides the former ones)\",\n \n "
},
{
"path": "anytool/config/config_grounding.json",
"chars": 1993,
"preview": "{\n \"shell\": {\n \"mode\": \"local\",\n \"timeout\": 60,\n \"max_retries\": 3,\n \"retry_interval\": 3.0,\n \"default_she"
},
{
"path": "anytool/config/config_mcp.json.example",
"chars": 0,
"preview": ""
},
{
"path": "anytool/config/config_security.json",
"chars": 1906,
"preview": "{\n \"security_policies\": {\n \"global\": {\n \"allow_shell_commands\": true,\n \"allow_network_access\": true,\n "
},
{
"path": "anytool/config/constants.py",
"chars": 518,
"preview": "from pathlib import Path\n\nCONFIG_GROUNDING = \"config_grounding.json\"\nCONFIG_SECURITY = \"config_security.json\"\nCONFIG_MCP"
},
{
"path": "anytool/config/grounding.py",
"chars": 10763,
"preview": "from typing import Dict, Optional, Any, List, Literal\ntry:\n from pydantic import BaseModel, Field, field_validator\n "
},
{
"path": "anytool/config/loader.py",
"chars": 5699,
"preview": "import threading\nfrom pathlib import Path\nfrom typing import Union, Iterable, Dict, Any, Optional\n\nfrom .grounding impor"
},
{
"path": "anytool/config/utils.py",
"chars": 916,
"preview": "import json\nfrom pathlib import Path\nfrom typing import Any\n\n\ndef get_config_value(config: Any, key: str, default=None):"
},
{
"path": "anytool/grounding/backends/__init__.py",
"chars": 987,
"preview": "# Use lazy imports to avoid loading all backends unconditionally\n\ndef _lazy_import_provider(provider_name: str):\n \"\"\""
},
{
"path": "anytool/grounding/backends/gui/__init__.py",
"chars": 638,
"preview": "from .provider import GUIProvider\nfrom .session import GUISession\nfrom .transport.connector import GUIConnector\nfrom .tr"
},
{
"path": "anytool/grounding/backends/gui/anthropic_client.py",
"chars": 22672,
"preview": "import base64\nimport os\nimport time\nfrom typing import Any, Dict, Optional, Tuple, List\nfrom anytool.utils.logging impor"
},
{
"path": "anytool/grounding/backends/gui/anthropic_utils.py",
"chars": 11072,
"preview": "from typing import List, cast\nfrom enum import Enum\nfrom datetime import datetime\nfrom anytool.utils.logging import Logg"
},
{
"path": "anytool/grounding/backends/gui/config.py",
"chars": 2642,
"preview": "from typing import Dict, Any, Optional\nimport os\nimport platform as platform_module\nfrom anytool.utils.logging import Lo"
},
{
"path": "anytool/grounding/backends/gui/provider.py",
"chars": 5393,
"preview": "from typing import Dict, Any, Union\nfrom anytool.grounding.core.types import BackendType, SessionConfig\nfrom anytool.gro"
},
{
"path": "anytool/grounding/backends/gui/session.py",
"chars": 8154,
"preview": "from typing import Dict, Any, Union\nimport os\nfrom anytool.grounding.core.session import BaseSession\nfrom anytool.ground"
},
{
"path": "anytool/grounding/backends/gui/tool.py",
"chars": 26943,
"preview": "import base64\nfrom typing import Any, Dict\nfrom anytool.grounding.core.tool.base import BaseTool\nfrom anytool.grounding."
},
{
"path": "anytool/grounding/backends/gui/transport/actions.py",
"chars": 8424,
"preview": "\"\"\"\nGUI Action Space Definitions.\n\"\"\"\nfrom typing import Dict, Any\n\n# Screen resolution constants\nX_MAX = 1920\nY_MAX = 1"
},
{
"path": "anytool/grounding/backends/gui/transport/connector.py",
"chars": 14686,
"preview": "import asyncio\nimport re\nfrom typing import Any, Dict, Optional\nfrom anytool.grounding.core.transport.connectors import "
},
{
"path": "anytool/grounding/backends/gui/transport/local_connector.py",
"chars": 14305,
"preview": "\"\"\"\nLocal GUI Connector — execute GUI operations directly in-process.\n\nThis connector has the **same public API** as GUI"
},
{
"path": "anytool/grounding/backends/mcp/__init__.py",
"chars": 1178,
"preview": "\"\"\"\nMCP Backend for AnyTool Grounding.\n\nThis module provides the MCP (Model Context Protocol) backend implementation\nfor"
},
{
"path": "anytool/grounding/backends/mcp/client.py",
"chars": 16439,
"preview": "\"\"\"\nClient for managing MCP servers and sessions.\n\nThis module provides a high-level client that manages MCP servers, co"
},
{
"path": "anytool/grounding/backends/mcp/config.py",
"chars": 5221,
"preview": "\"\"\"\nConfiguration loader for MCP session.\n\nThis module provides functionality to load MCP configuration from JSON files."
},
{
"path": "anytool/grounding/backends/mcp/installer.py",
"chars": 29734,
"preview": "import asyncio\nimport sys\nimport shutil\nfrom typing import Callable, Awaitable, Optional, Dict, List\nfrom anytool.utils."
},
{
"path": "anytool/grounding/backends/mcp/provider.py",
"chars": 19751,
"preview": "\"\"\"\nMCP Provider implementation.\n\nThis module provides a provider for managing MCP server sessions.\n\"\"\"\nimport asyncio\nf"
},
{
"path": "anytool/grounding/backends/mcp/session.py",
"chars": 2613,
"preview": "\"\"\"\nSession manager for MCP connections.\n\nThis module provides a session manager for MCP connections,\nwhich handles auth"
},
{
"path": "anytool/grounding/backends/mcp/tool_cache.py",
"chars": 9443,
"preview": "import json\nfrom pathlib import Path\nfrom datetime import datetime\nfrom typing import Any, Dict, List, Optional\n\nfrom an"
},
{
"path": "anytool/grounding/backends/mcp/tool_converter.py",
"chars": 7286,
"preview": "\"\"\"\nTool converter for MCP.\n\nThis module provides utilities to convert MCP tools to BaseTool instances.\n\"\"\"\n\nimport copy"
},
{
"path": "anytool/grounding/backends/mcp/transport/connectors/__init__.py",
"chars": 542,
"preview": "\"\"\"\nConnectors for various MCP transports.\n\nThis module provides interfaces for connecting to MCP implementations\nthroug"
},
{
"path": "anytool/grounding/backends/mcp/transport/connectors/base.py",
"chars": 14557,
"preview": "\"\"\"\nBase connector for MCP implementations.\n\nThis module provides the base connector interface that all MCP connectors m"
},
{
"path": "anytool/grounding/backends/mcp/transport/connectors/http.py",
"chars": 28765,
"preview": "\"\"\"\nHTTP connector for MCP implementations.\n\nThis module provides a connector for communicating with MCP implementations"
},
{
"path": "anytool/grounding/backends/mcp/transport/connectors/sandbox.py",
"chars": 9947,
"preview": "\"\"\"\nSandbox connector for MCP implementations.\n\nThis module provides a connector for communicating with MCP implementati"
},
{
"path": "anytool/grounding/backends/mcp/transport/connectors/stdio.py",
"chars": 2928,
"preview": "\"\"\"\nStdIO connector for MCP implementations.\n\nThis module provides a connector for communicating with MCP implementation"
},
{
"path": "anytool/grounding/backends/mcp/transport/connectors/utils.py",
"chars": 365,
"preview": "from typing import Any\n\n\ndef is_stdio_server(server_config: dict[str, Any]) -> bool:\n \"\"\"Check if the server configur"
},
{
"path": "anytool/grounding/backends/mcp/transport/connectors/websocket.py",
"chars": 9564,
"preview": "\"\"\"\nWebSocket connector for MCP implementations.\n\nThis module provides a connector for communicating with MCP implementa"
},
{
"path": "anytool/grounding/backends/mcp/transport/task_managers/__init__.py",
"chars": 495,
"preview": "\"\"\"\nConnectors for various MCP transports.\n\nThis module provides interfaces for connecting to MCP implementations\nthroug"
},
{
"path": "anytool/grounding/backends/mcp/transport/task_managers/sse.py",
"chars": 1580,
"preview": "\"\"\"\nSSE connection management for MCP implementations.\n\nThis module provides a connection manager for SSE-based MCP conn"
},
{
"path": "anytool/grounding/backends/mcp/transport/task_managers/stdio.py",
"chars": 14909,
"preview": "\"\"\"\nStdIO connection management for MCP implementations.\n\nThis module provides a connection manager for stdio-based MCP "
},
{
"path": "anytool/grounding/backends/mcp/transport/task_managers/streamable_http.py",
"chars": 3807,
"preview": "\"\"\"\nStreamable HTTP connection management for MCP implementations.\n\nThis module provides a connection manager for stream"
},
{
"path": "anytool/grounding/backends/mcp/transport/task_managers/websocket.py",
"chars": 925,
"preview": "\"\"\"\nWebSocket connection management for MCP implementations.\n\nThis module provides a connection manager for WebSocket-ba"
},
{
"path": "anytool/grounding/backends/shell/__init__.py",
"chars": 281,
"preview": "from .provider import ShellProvider\nfrom .session import ShellSession\nfrom .transport.connector import ShellConnector\nfr"
},
{
"path": "anytool/grounding/backends/shell/provider.py",
"chars": 4358,
"preview": "from anytool.grounding.core.provider import Provider\nfrom anytool.grounding.core.types import BackendType, SessionConfig"
},
{
"path": "anytool/grounding/backends/shell/session.py",
"chars": 21128,
"preview": "import re\nfrom typing import Union\nfrom anytool.grounding.core.types import BackendType\nfrom anytool.grounding.core.sess"
},
{
"path": "anytool/grounding/backends/shell/transport/connector.py",
"chars": 7735,
"preview": "import asyncio\nfrom typing import Any, Optional, Dict\n\nfrom anytool.grounding.core.transport.connectors import AioHttpCo"
},
{
"path": "anytool/grounding/backends/shell/transport/local_connector.py",
"chars": 15254,
"preview": "\"\"\"\nLocal Shell Connector — execute Python / Bash scripts directly via subprocess.\n\nThis connector has the **same public"
},
{
"path": "anytool/grounding/backends/web/__init__.py",
"chars": 116,
"preview": "from .provider import WebProvider\nfrom .session import WebSession\n\n__all__ = [\n \"WebProvider\",\n \"WebSession\"\n]"
},
{
"path": "anytool/grounding/backends/web/provider.py",
"chars": 2030,
"preview": "from typing import Dict, Any\nfrom anytool.grounding.core.types import BackendType, SessionConfig\nfrom anytool.grounding."
},
{
"path": "anytool/grounding/backends/web/session.py",
"chars": 8643,
"preview": "import os\nfrom typing import Dict, Any, Optional\nfrom anytool.grounding.core.session import BaseSession\nfrom anytool.gro"
},
{
"path": "anytool/grounding/core/exceptions.py",
"chars": 1866,
"preview": "\"\"\"\nUnified exception & error-code definitions for the grounding framework\n\"\"\"\nfrom enum import Enum, auto\nfrom typing i"
},
{
"path": "anytool/grounding/core/grounding_client.py",
"chars": 36828,
"preview": "import asyncio\nimport time\nfrom collections import OrderedDict\nfrom datetime import datetime\nfrom typing import Any, Dic"
},
{
"path": "anytool/grounding/core/provider.py",
"chars": 5463,
"preview": "\"\"\"\nprovider is to manage sessions of a backend, if the backend is mcp, then provider will manage sessions through serve"
},
{
"path": "anytool/grounding/core/quality/__init__.py",
"chars": 727,
"preview": "from .types import ToolQualityRecord, ExecutionRecord, DescriptionQuality\nfrom .manager import ToolQualityManager\nfrom ."
},
{
"path": "anytool/grounding/core/quality/manager.py",
"chars": 26212,
"preview": "\"\"\"\nTool Quality Manager\n\nCore API (called by main flow):\n- record_execution(): Called by BaseTool after execution\n- adj"
},
{
"path": "anytool/grounding/core/quality/store.py",
"chars": 4223,
"preview": "\"\"\"\nPersistent storage for tool quality data.\n\"\"\"\n\nimport json\nimport asyncio\nfrom pathlib import Path\nfrom typing impor"
},
{
"path": "anytool/grounding/core/quality/types.py",
"chars": 7933,
"preview": "\"\"\"\nData types for tool quality tracking.\n\"\"\"\n\nfrom dataclasses import dataclass, field\nfrom datetime import datetime\nfr"
},
{
"path": "anytool/grounding/core/search_tools.py",
"chars": 45957,
"preview": "from anytool.grounding.core.tool.base import BaseTool\nimport re\nimport os\nimport numpy as np\nimport httpx\nfrom typing im"
},
{
"path": "anytool/grounding/core/security/__init__.py",
"chars": 454,
"preview": "from .sandbox import BaseSandbox, SandboxManager\nfrom .policies import SecurityPolicyManager, SecurityPolicy\n\n# Try to i"
},
{
"path": "anytool/grounding/core/security/e2b_sandbox.py",
"chars": 7264,
"preview": "\"\"\"\nE2B Sandbox implementation.\n\nThis module provides a concrete implementation of BaseSandbox using E2B.\n\"\"\"\n\nimport os"
},
{
"path": "anytool/grounding/core/security/policies.py",
"chars": 6066,
"preview": "import asyncio\nimport sys\nfrom typing import Callable, Awaitable, Dict, Optional\nfrom ..types import SecurityPolicy, Bac"
},
{
"path": "anytool/grounding/core/security/sandbox.py",
"chars": 1378,
"preview": "from typing import Any, Dict, Optional\nfrom abc import ABC, abstractmethod\n\nfrom ..types import SandboxOptions, BackendT"
},
{
"path": "anytool/grounding/core/session.py",
"chars": 3959,
"preview": "from abc import ABC, abstractmethod\nfrom typing import Any, Dict, List\nfrom datetime import datetime\n\nfrom .tool import "
},
{
"path": "anytool/grounding/core/system/__init__.py",
"chars": 124,
"preview": "from .provider import SystemProvider\nfrom .tool import SYSTEM_TOOLS\n\n__all__ = [\n \"SystemProvider\",\n \"SYSTEM_TOOLS"
},
{
"path": "anytool/grounding/core/system/provider.py",
"chars": 1493,
"preview": "from typing import List, Dict, Any\nfrom ..provider import Provider\nfrom ..types import BackendType, SessionConfig\nfrom ."
},
{
"path": "anytool/grounding/core/system/tool.py",
"chars": 2517,
"preview": "from ..tool.local_tool import LocalTool\nfrom ..types import BackendType, ToolResult, ToolStatus\nfrom ..grounding_client "
},
{
"path": "anytool/grounding/core/tool/__init__.py",
"chars": 147,
"preview": "from .base import BaseTool\nfrom .local_tool import LocalTool\nfrom .remote_tool import RemoteTool\n\n__all__ = [\"BaseTool\","
},
{
"path": "anytool/grounding/core/tool/base.py",
"chars": 12316,
"preview": "\"\"\"\nBaseTool.\nAll pre-defined grounding atomic operations will inherit this tool class.\nRemoteTool needs to pass in conn"
},
{
"path": "anytool/grounding/core/tool/local_tool.py",
"chars": 894,
"preview": "\"\"\"\nLocalTool.\nExecutes entirely inside this Python process.\n\"\"\"\nimport asyncio\nfrom typing import Any\nfrom .base import"
},
{
"path": "anytool/grounding/core/tool/remote_tool.py",
"chars": 3287,
"preview": "\"\"\"\nRemoteTool.\nWrapper around a connector that calls a remote tool.\n\"\"\"\nfrom typing import Optional\nfrom anytool.utils."
},
{
"path": "anytool/grounding/core/transport/connectors/__init__.py",
"chars": 140,
"preview": "from .base import BaseConnector\nfrom .aiohttp_connector import AioHttpConnector\n\n__all__ = [\n \"BaseConnector\", \n \""
},
{
"path": "anytool/grounding/core/transport/connectors/aiohttp_connector.py",
"chars": 5221,
"preview": "from typing import Any\nfrom yarl import URL\nimport aiohttp\n\nfrom ..task_managers import AioHttpConnectionManager\nfrom .b"
},
{
"path": "anytool/grounding/core/transport/connectors/base.py",
"chars": 4909,
"preview": "\"\"\"\nBase connector abstraction.\n\nA connector is a very thin wrapper-class that owns a *connection manager*\n(e.g. AioHttp"
},
{
"path": "anytool/grounding/core/transport/task_managers/__init__.py",
"chars": 430,
"preview": "from .base import BaseConnectionManager\nfrom .aiohttp_connection_manager import AioHttpConnectionManager\nfrom .async_ctx"
},
{
"path": "anytool/grounding/core/transport/task_managers/aiohttp_connection_manager.py",
"chars": 1926,
"preview": "\"\"\"\nLong-lived aiohttp ClientSession manager based on AsyncContextConnectionManager.\n\nIt keeps a single ClientSession op"
},
{
"path": "anytool/grounding/core/transport/task_managers/async_ctx.py",
"chars": 5264,
"preview": "\"\"\"\nGeneric connection manager based on an *async context manager*.\nGive it any factory that returns an async–context-ma"
},
{
"path": "anytool/grounding/core/transport/task_managers/base.py",
"chars": 8170,
"preview": "\"\"\"\nBase connection manager for all backend connectors.\n\nThis module provides an abstract base class for different types"
},
{
"path": "anytool/grounding/core/transport/task_managers/noop.py",
"chars": 789,
"preview": "\"\"\"No-op connection manager for local (in-process) connectors.\n\nLocal connectors execute commands directly via subproces"
},
{
"path": "anytool/grounding/core/transport/task_managers/placeholder.py",
"chars": 655,
"preview": "from typing import Any\nfrom .base import BaseConnectionManager\n\n\nclass PlaceholderConnectionManager(BaseConnectionManage"
},
{
"path": "anytool/grounding/core/types.py",
"chars": 8820,
"preview": "from enum import Enum\nfrom datetime import datetime\nfrom typing import Any, Dict, Generic, List, TypeVar, Optional\nimpor"
},
{
"path": "anytool/llm/__init__.py",
"chars": 29,
"preview": "from .client import LLMClient"
},
{
"path": "anytool/llm/client.py",
"chars": 28724,
"preview": "import litellm\nimport json\nimport asyncio\nimport time\nfrom typing import List, Sequence, Union, Dict, Optional\nfrom dote"
},
{
"path": "anytool/local_server/README.md",
"chars": 2635,
"preview": "# AnyTool Local Server (Desktop Version)\n\n## 1. Introduction\n\nThe AnyTool Local Server is a **lightweight, cross-platfor"
},
{
"path": "anytool/local_server/__init__.py",
"chars": 66,
"preview": "from .main import app, run_server\n\n__all__ = [\"app\", \"run_server\"]"
},
{
"path": "anytool/local_server/config.json",
"chars": 106,
"preview": "{\n \"server\": {\n \"host\": \"127.0.0.1\",\n \"port\": 5000,\n \"debug\": false,\n \"threaded\": true\n }\n}\n"
},
{
"path": "anytool/local_server/feature_checker.py",
"chars": 10174,
"preview": "import platform\nimport subprocess\nimport tempfile\nfrom typing import Dict, Any\n\nfrom anytool.utils.logging import Logger"
},
{
"path": "anytool/local_server/health_checker.py",
"chars": 22376,
"preview": "import requests\nimport os\nfrom pathlib import Path\nfrom typing import Dict, Tuple, Optional\nfrom anytool.utils.logging i"
},
{
"path": "anytool/local_server/main.py",
"chars": 40182,
"preview": "import os\nimport platform\nimport shlex\nimport subprocess\nimport signal\nimport time\nimport json\nimport uuid\nfrom datetime"
},
{
"path": "anytool/local_server/platform_adapters/__init__.py",
"chars": 1058,
"preview": "import platform\nfrom typing import Optional, Any\n\nplatform_name = platform.system()\n\nif platform_name == \"Darwin\":\n t"
},
{
"path": "anytool/local_server/platform_adapters/linux_adapter.py",
"chars": 20515,
"preview": "import subprocess\nimport os\nfrom typing import Dict, Any, Optional, List\nfrom anytool.utils.logging import Logger\nfrom P"
},
{
"path": "anytool/local_server/platform_adapters/macos_adapter.py",
"chars": 26861,
"preview": "import subprocess\nimport os\nfrom typing import Dict, Any, Optional, List\nfrom anytool.utils.logging import Logger\n\ntry:\n"
},
{
"path": "anytool/local_server/platform_adapters/pyxcursor.py",
"chars": 4550,
"preview": "import os\nimport ctypes\nimport ctypes.util\nimport numpy as np\n\n# A helper function to convert data from Xlib to byte arr"
},
{
"path": "anytool/local_server/platform_adapters/windows_adapter.py",
"chars": 22263,
"preview": "import os\nimport ctypes\nimport subprocess\nfrom typing import Dict, Any, Optional, List\nfrom anytool.utils.logging import"
},
{
"path": "anytool/local_server/requirements.txt",
"chars": 719,
"preview": "# Local server dependencies (cross-platform)\nflask>=3.1.0\npyautogui>=0.9.54\npydantic>=2.12.0\nrequests>=2.32.0\n\n# # macOS"
},
{
"path": "anytool/local_server/run.sh",
"chars": 630,
"preview": "#!/bin/bash\nSCRIPT_DIR=\"$( cd \"$( dirname \"${BASH_SOURCE[0]}\" )\" && pwd )\"\nPROJECT_ROOT=\"$( cd \"$SCRIPT_DIR/../..\" && pw"
},
{
"path": "anytool/local_server/utils/__init__.py",
"chars": 142,
"preview": "from .accessibility import AccessibilityHelper\nfrom .screenshot import ScreenshotHelper\n\n__all__ = [\"AccessibilityHelper"
},
{
"path": "anytool/local_server/utils/accessibility.py",
"chars": 5179,
"preview": "import platform\nfrom anytool.utils.logging import Logger\nfrom typing import Dict, Any, Optional\n\nlogger = Logger.get_log"
},
{
"path": "anytool/local_server/utils/screenshot.py",
"chars": 8372,
"preview": "import platform\nimport os\nimport logging\nfrom typing import Optional, Tuple\nfrom PIL import Image\nimport pyautogui\n\nlogg"
},
{
"path": "anytool/platform/__init__.py",
"chars": 605,
"preview": "from .system_info import SystemInfoClient, get_system_info, get_screen_size\nfrom .recording import RecordingClient, Reco"
},
{
"path": "anytool/platform/config.py",
"chars": 2725,
"preview": "import os\nimport json\nfrom typing import Dict, Any\nfrom anytool.utils.logging import Logger\n\nlogger = Logger.get_logger("
},
{
"path": "anytool/platform/recording.py",
"chars": 7420,
"preview": "import aiohttp\nfrom typing import Optional\nfrom anytool.utils.logging import Logger\nfrom .config import get_client_base_"
},
{
"path": "anytool/platform/screenshot.py",
"chars": 8743,
"preview": "\"\"\"\nScreenshot client for capturing screens via HTTP API.\n\nThis module provides a screenshot client that captures screen"
},
{
"path": "anytool/platform/system_info.py",
"chars": 6086,
"preview": "import aiohttp\nfrom typing import Optional, Dict, Any\nfrom anytool.utils.logging import Logger\nfrom .config import get_c"
},
{
"path": "anytool/prompts/__init__.py",
"chars": 110,
"preview": "from anytool.prompts.grounding_agent_prompts import GroundingAgentPrompts\n\n__all__ = [\"GroundingAgentPrompts\"]"
},
{
"path": "anytool/prompts/grounding_agent_prompts.py",
"chars": 10320,
"preview": "from typing import List\n\n\nclass GroundingAgentPrompts:\n \n TASK_COMPLETE = \"<COMPLETE>\"\n \n SYSTEM_PROMPT = f\""
},
{
"path": "anytool/recording/__init__.py",
"chars": 1297,
"preview": "\"\"\"\n RecordingManager\n ├── internal management of platform.RecordingClient\n ├── internal management of plat"
},
{
"path": "anytool/recording/action_recorder.py",
"chars": 9157,
"preview": "\"\"\"\nAgent Action Recorder\n\nRecords agent decision-making processes, reasoning, and outputs.\nFocuses on high-level agent "
},
{
"path": "anytool/recording/manager.py",
"chars": 40491,
"preview": "import datetime\nimport json\nimport ast\nimport types\nfrom typing import Any, Dict, List, Optional\nfrom pathlib import Pat"
},
{
"path": "anytool/recording/recorder.py",
"chars": 14074,
"preview": "import datetime\nimport json\nfrom typing import Any, Dict, List, Optional\nfrom pathlib import Path\n\nfrom anytool.utils.lo"
},
{
"path": "anytool/recording/utils.py",
"chars": 13607,
"preview": "import json\nimport os\nfrom pathlib import Path\nfrom typing import Any, Dict, List, Optional, Tuple\nfrom anytool.utils.lo"
},
{
"path": "anytool/recording/video.py",
"chars": 2649,
"preview": "\"\"\"\nVideo Recorder\n\nCommunicates with local_server through platform.RecordingClient\nSupports local and remote recording "
},
{
"path": "anytool/recording/viewer.py",
"chars": 12254,
"preview": "\"\"\"\nRecording Viewer\nConvenient tools for viewing and analyzing recording sessions.\n\"\"\"\n\nimport json\nfrom pathlib import"
},
{
"path": "anytool/tool_layer.py",
"chars": 17300,
"preview": "from __future__ import annotations\n\nimport asyncio\nimport traceback\nimport uuid\nfrom dataclasses import dataclass, field"
},
{
"path": "anytool/utils/cli_display.py",
"chars": 8888,
"preview": "\"\"\"CLI Display utilities for AnyTool startup and interaction\"\"\"\n\nfrom anytool.tool_layer import AnyToolConfig\nfrom anyto"
},
{
"path": "anytool/utils/display.py",
"chars": 7471,
"preview": "from typing import Optional, List\nfrom enum import Enum\nimport re\n\n\nclass Colors:\n RESET = \"\\033[0m\"\n BOLD = \"\\033"
},
{
"path": "anytool/utils/logging.py",
"chars": 11446,
"preview": "import logging\nimport os\nimport sys\nimport threading\nimport json\nfrom pathlib import Path\nfrom datetime import datetime\n"
},
{
"path": "anytool/utils/telemetry/__init__.py",
"chars": 0,
"preview": ""
},
{
"path": "anytool/utils/telemetry/events.py",
"chars": 3112,
"preview": "from abc import ABC, abstractmethod\nfrom dataclasses import dataclass\nfrom typing import Any\n\n\nclass BaseTelemetryEvent("
},
{
"path": "anytool/utils/telemetry/telemetry.py",
"chars": 11973,
"preview": "import logging\nimport os\nimport platform\nimport uuid\nfrom collections.abc import Callable\nfrom functools import wraps\nfr"
},
{
"path": "anytool/utils/telemetry/utils.py",
"chars": 1769,
"preview": "\"\"\"\nUtility functions for extracting model information from LangChain LLMs.\n\nThis module provides utilities to extract p"
},
{
"path": "anytool/utils/ui.py",
"chars": 15140,
"preview": "\"\"\"\nAnyTool Terminal UI System\n\nProvides real-time CLI visualization for AnyTool execution flow.\nDisplays agent activiti"
},
{
"path": "anytool/utils/ui_integration.py",
"chars": 8812,
"preview": "\"\"\"\nAnyTool UI Integration\n\nIntegrates the UI system with AnyTool core components.\nProvides hooks and callbacks to updat"
},
{
"path": "pyproject.toml",
"chars": 1607,
"preview": "[build-system]\nrequires = [\"setuptools>=68.0\", \"wheel\"]\nbuild-backend = \"setuptools.build_meta\"\n\n[project]\nname = \"anyto"
},
{
"path": "requirements.txt",
"chars": 871,
"preview": "# AnyTool core dependencies\nlitellm>=1.70.0\npython-dotenv>=1.0.0\nopenai>=1.0.0\njsonschema>=4.25.0\nmcp>=1.0.0\nanthropic>="
}
]
About this extraction
This page contains the full source code of the HKUDS/AnyTool GitHub repository, extracted and formatted as plain text for AI agents and large language models (LLMs). The extraction includes 132 files (1.1 MB), approximately 230.3k tokens, and a symbol index with 1125 extracted functions, classes, methods, constants, and types. Use this with OpenClaw, Claude, ChatGPT, Cursor, Windsurf, or any other AI tool that accepts text input. You can copy the full output to your clipboard or download it as a .txt file.
Extracted by GitExtract — free GitHub repo to text converter for AI. Built by Nikandr Surkov.